With various AI applications popping up like flowers, the construction and optimization of knowledge bases have become crucial. In this field, Dify and Ragflow, as two highly regarded tools, each show their unique charm, attracting the attention of developers and enterprises.
Dify, with its excellent visual orchestration capabilities, has greatly lowered the development threshold, allowing non-technical personnel to easily build intelligent applications. Therefore, it has an advantage in development efficiency and has become the first choice for many companies. Ragflow, on the other hand, focuses on the deep optimization of knowledge bases. It excels in retrieval accuracy and recall rates, providing reliable solutions for scenarios with extremely high requirements for knowledge precision.
What exactly are the differences between these two tools in terms of functions, features, and applicable scenarios? How to make the right choice based on actual needs? Today, let's analyze the knowledge bases of Dify and Ragflow in depth, uncover their mysteries, and provide a strong decision-making basis for your AI development journey.
1. Initial Impressions of Dify and Ragflow Knowledge Bases
1.1 Introduction to Dify Knowledge Base
Dify is a highly regarded low-code LLM application development platform designed to lower the development threshold so that more people without technical backgrounds can participate in the development of AI applications. Just like building with blocks, through visual workflow orchestration, users can easily complete complex logic designs by simply dragging and dropping components, quickly building production-level generative AI applications.
In practical applications, Dify's advantages have been fully demonstrated. An e-commerce platform used Dify to successfully build an intelligent customer service system in just 3 days. This system can not only respond quickly to user inquiries but also accurately provide services such as order queries and return/exchange policy generation according to user needs, greatly improving user experience and operational efficiency.
1.2 Introduction to Ragflow Knowledge Base
Ragflow is a RAG engine focused on complex document parsing and high-precision retrieval, with unique advantages in handling unstructured data. It is like a professional document analyst, capable of deeply understanding data in various complex formats. Whether it is PDF, scanned copies, tables, or images, it can accurately identify titles, paragraphs, images, and other elements, and even supports visual editing, making document processing more efficient and precise.
In legal contract review scenarios, Ragflow can automatically extract key fields from clauses, with an accuracy rate leading the industry. This advantage makes it shine in professional fields, providing enterprises with reliable knowledge retrieval and analysis services.
Ragflow uses a multi-path recall mechanism and fusion reranking technology, acting like an experienced librarian who can accurately find the most relevant parts to user questions from a vast amount of knowledge. This significantly reduces "hallucination" problems in AI answers, making answers more accurate and reliable. At the same time, it can also provide key citation snapshots and original source links for answers, allowing users to see the source of the answer at a glance, enhancing the credibility of the answer.
2. Principle Analysis: Digging Deep into the Roots of Differences
2.1 Principle of Dify Knowledge Base
The principle of Dify's knowledge base is like a meticulously built intelligent factory. From the processing of raw materials (data) to the generation of products (answers), every link has been carefully designed and optimized. Dify's knowledge base encapsulation is relatively more complete than Ragflow's. The process of documents entering the vector database after being vectorized through chunking settings belongs to the knowledge base, and keywords are automatically formed in the knowledge base.
The key settings for retrieval are also set inside the knowledge base, which means that for an AI assistant, in a single Q&A session, the retrieval method in each knowledge base is different. The retrieval method of the knowledge base can be set when data enters the knowledge base, or it can be set in the subsequent knowledge base settings. Dify does not have as many types for document chunking, only two modes: General and Parent-Child.
2.2 Principle of Ragflow Knowledge Base
The principle of Ragflow's knowledge base focuses on deep understanding of documents and high-precision retrieval. Every step contains precise grasp and efficient processing of knowledge. Its knowledge base mainly completes the process from data chunking, vectorization, storage in the database, to constructing automatic keywords, automatic questions, and knowledge graphs.
Here, in the data chunking and vectorization process, it can set different chunking and vectorization methods according to the document type:
- General: Continuous chunking of files based on preset block markers. Supports MD, MDX, DOCX, XLSX, XLS, PPT, PDF, TXT, JPEG, JPG, PNG, TIF, GIF, CSV, JSON, EML, HTML.
- Q&A: Excel and CSV/TXT files suitable for Q&A.
- Resume: DOCX, PDF, TXT suitable for resumes.
- Manual: PDF suitable for manuals.
- Table: XLSX, XLS, CSV/TXT suitable for tables.
- Paper: PDF suitable for papers.
- Book: DOCX, PDF, TXT suitable for books.
- Laws: DOCX, PDF, TXT suitable for legal documents.
- Presentation: PDF, PPTX suitable for presentations.
- Picture: JPEG, JPG, PNG, TIF, GIF suitable for pictures.
- One: Each document is chunked completely (as a whole).
- Tag: The function of the knowledge base is as a tag set for other knowledge. Tag datasets do not participate in retrieval but participate in the data vectorization process.
Deep document understanding technology is one of Ragflow's core competencies. It can deeply parse unstructured data in various complex formats like a professional document analyst. RAGFlow provides multiple chunking templates to facilitate chunking of files with different layouts and ensure semantic integrity. In the chunking method, you can choose the default template suitable for your file layout and format.
In fact, when documents are stored in the knowledge base, chunking is extremely important. The integrity of chunking determines the quality of retrieved data later.
Text chunking is an important step for Ragflow to process data. It adopts different chunking strategies according to the type and characteristics of the document to divide the document into text blocks of appropriate size.
In the storage link, Ragflow stores the vectorized text blocks into the vector database, while retaining information such as the original text corresponding to the text blocks and their positions in the original file, facilitating user viewing and tracing.
In the retrieval stage, Ragflow adopts a multi-path recall mechanism and fusion reranking technology, which can accurately find the parts most relevant to user questions from massive knowledge. It combines multiple retrieval strategies, such as vector retrieval and keyword retrieval, to improve knowledge coverage. Then, through fusion reranking technology, the retrieval results are optimized and sorted to ensure that the answers presented to users are the most relevant and accurate.
3. Chunking Mode Comparison: Who is Better?
3.1 Detailed Explanation of Dify Chunking Modes
Dify mainly provides two chunking modes: General and Parent-Child. These two modes are like two different keys, each adapting to different types of document "locks".
General Mode is Dify's default segmentation method, like a master key, suitable for most common scenarios. It uses coarse-grained segmentation to split document content into independent segments, each of which can be retrieved individually. In this mode, segmentation rules are very flexible, defaulting to segmentation by paragraph (\n), but if users have special needs, they can also use regular expressions to customize segmentation rules, such as segmentation by sentence. The segment length can also be adjusted according to needs. The default character count per segment does not exceed 500 Tokens, and the maximum can be set to 4000 Tokens. A certain overlap can also be set between adjacent segments, recommended to be 10%-25% of the segment length to improve recall effects. General mode also provides basic text preprocessing functions, such as replacing continuous spaces, newlines, and tabs, deleting URLs and email addresses, etc. For documents with clear structures and strong paragraph independence, such as Frequently Asked Questions (FAQ) and product manuals, General mode can quickly and accurately locate relevant content, providing efficient retrieval services for users.
Parent-Child Mode is an upgrade based on the General mode, adopting a two-layer segmentation structure, like a precise combination lock, more suitable for handling documents with complex structures and strong context dependencies. In this mode, the Parent-chunk is a larger text unit, such as a paragraph or full text, used to provide rich context information. The Child-chunk is a small unit within the Parent-chunk, such as a single sentence, used to improve retrieval precision. The system first precisely matches user questions through Child-chunks to find the most relevant small blocks of information, and then sends the corresponding Parent-chunk to the LLM together to provide a complete context background. In application scenarios such as contract review and policy interpretation that require "clause-by-clause analysis", Parent-Child mode can leverage its unique advantages to ensure the accuracy and completeness of answers.
3.2 Detailed Explanation of Ragflow Chunking Modes
Ragflow provides a variety of chunking templates, like a treasure chest filled with various professional tools, capable of meeting the chunking needs of different types of documents.
- General Chunking: The most commonly used chunking method, supporting a very wide range of document formats. This chunking method is like a general cutting tool. The system uses visual detection models to split continuous text into multiple fragments, and then merges these continuous fragments into chunks with a Token count not exceeding the specified "Token count", capable of adapting to the basic structure of most documents.
- Q&A Chunking: Specifically used to handle files in Q&A format, supporting Excel and CSV/TXT file formats.
- Resume Chunking: Mainly used to handle resume documents, supporting DOCX, PDF, and TXT file formats.
- Manual Chunking: Suitable for manual documents, only supporting PDF format.
- Table Chunking: Used to handle table documents, supporting EXCEL and CSV/TXT format files.
- Paper Chunking: Specifically used to handle paper documents, only supporting PDF files.
- Tag Knowledge Base: Acts as a tag set for other knowledge.
Comparing the chunking modes of Dify and Ragflow, it can be found that Ragflow has a larger number of chunking modes, covering a wider range of document types and application scenarios. Although Dify has relatively fewer chunking modes, the General and Parent-Child modes can also meet the processing needs of most common documents, and are relatively simple to configure and use.
If your document type is relatively single and belongs to common structured documents, Dify's chunking modes may be sufficient to meet your needs; but if you need to handle various complex formats of documents, or have higher requirements for chunking granularity and adaptability, then Ragflow's multiple chunking templates will undoubtedly provide stronger support.
4. Retrieval Mechanism Comparison: Battle of Accuracy and Recall
4.1 Dify Retrieval Mechanism
Dify's retrieval mechanism provides multiple flexible and powerful retrieval strategies to meet knowledge acquisition needs in different scenarios. Dify has three retrieval modes: Vector Retrieval, Full-Text Retrieval, and Hybrid Retrieval.
Vector Retrieval is one of Dify's core retrieval methods. By generating query embeddings and querying text segments most similar to their vector representations, it calculates the vector difference between the question and the content to determine which chunk fits the question. Dify also provides a series of parameter settings. TopK is used to screen text fragments with the highest similarity to the user's question. The system will dynamically adjust the number of fragments based on the context window size of the selected model, with a default value of 3. The Score threshold is used to set the similarity threshold for text fragment screening. Only text fragments exceeding the set score will be recalled. The system turns off this setting by default. After opening, the default value is 0.5. In addition, users can configure the Rerank model API key on the "Model Provider" page to perform semantic reranking on the recalled document results after semantic retrieval.
Full-Text Retrieval is like a meticulous keyword search expert. It indexes all vocabularies in the document, allowing users to query any vocabulary and return text fragments containing these vocabularies. This retrieval method is very effective in scenarios that require exact matching of keywords.
Hybrid Retrieval cleverly blends the advantages of vector retrieval and full-text retrieval, like an all-around knowledge hunter, executing both retrieval methods simultaneously and applying a reranking step to select the best result matching the user's question from the two types of query results.
4.2 Ragflow Retrieval Mechanism
Ragflow's retrieval mechanism is like a precise intelligent navigation system, setting retrieval parameters through various ways and using advanced technical means to ensure the most precise information is obtained from the knowledge base.
In AI assistants, visual orchestration components, and APIs, Ragflow provides users with functions to flexibly set retrieval parameters.
Ragflow adopts a Multi-path Recall Mechanism, which is like opening multiple search paths simultaneously, combining multiple retrieval strategies to greatly improve knowledge coverage. It comprehensively uses multiple methods such as vector retrieval and keyword retrieval to search the knowledge base from different angles.
Fusion Reranking Technology is another core highlight of Ragflow's retrieval mechanism. It acts like a strict quality gatekeeper, optimizing and sorting retrieval results. After obtaining a large number of candidate results from multi-path recall, fusion reranking technology will reorder them based on factors such as relevance and importance to the user's question, placing the results that best meet user needs at the front.
4.3 Comparison of Retrieval Effects
To intuitively understand the differences in retrieval effects between Dify and Ragflow, we conducted a series of actual tests. In the tests, from the test data, Ragflow indeed showed certain advantages in retrieval accuracy and recall rate. When dealing with complex questions, Ragflow's multi-path recall mechanism and fusion reranking technology can search the knowledge base more comprehensively, find more relevant information, and sort this information more accurately, thereby providing more precise answers.
In terms of recall rate, Ragflow's multiple chunking modes and flexible retrieval strategies allow it to more effectively recall relevant text blocks from the knowledge base. For some questions with vague semantics or involving multiple knowledge points, Ragflow can find more potential relevant content through different retrieval paths and chunking matches, while Dify may miss some relevant information in some cases due to relatively fewer chunking modes.
This does not mean that Dify's retrieval effect is poor. Dify's retrieval mechanism can still meet user needs in many scenarios, especially when there is a high requirement for retrieval speed or when questions are relatively simple and clear. Dify's multiple retrieval strategies and parameter settings can quickly provide accurate answers.
5. Selection Suggestions: How to Choose Based on Needs
5.1 Choose Based on Document Complexity
If your document structure is simple and the format is single, such as common FAQ documents and simple product introduction documents, Dify's General and Parent-Child mode chunking is sufficient to meet the needs. Its simple chunking modes and fast processing speed can help you efficiently build a knowledge base and achieve quick retrieval.
But if your document format is complex and contains a large amount of unstructured data, such as legal contracts, product manuals, and operation manuals, Ragflow's multiple chunking templates will be a better choice. It can perform precise chunking based on the characteristics of the document, ensuring accurate extraction of key information even in complex documents, providing users with high-quality retrieval services.
5.2 Choose Based on Development Cycle
When you are facing tight time constraints and need to quickly build an AI application and go live in a short time, Dify's low-code development and visual orchestration advantages become prominent. You don't need to invest a lot of time in complex development work. You can quickly build a fully functional AI assistant by simply dragging and dropping and configuring, meeting urgent business needs.
If there is an extremely high requirement for the accuracy and comprehensiveness of knowledge, and the development cycle is relatively loose, Ragflow is the better choice. Although it may smooth slightly in development efficiency, through deep understanding of documents and high-precision retrieval, it can provide users with the most accurate and comprehensive answers, ensuring the reliability of AI applications in professional fields.
5.3 Choose Based on Team Technical Capabilities
If your team's technical capabilities are weak, lacking professional developers, or members have no coding foundation, Dify's low-code platform allows non-technical personnel to get started easily. Through the visual operation interface, they can quickly create and deploy AI applications, lowering the technical threshold and improving team work efficiency.
If your team has strong technical strength and has customized development needs, Ragflow provides you with more possibilities. Its open-source nature and rich customization options allow technical teams to deeply customize and optimize the knowledge base according to specific business needs, achieving more advanced functions and performance improvements.
5.4 Choose Based on Application Scenarios
Dify, with its low-code development and visual orchestration advantages, is suitable for application scenarios such as product recommendations, order status queries, and return/exchange process guidance. Dify is also suitable as a content generation assistant.
Ragflow's advantage lies in its deep understanding of complex documents and high-precision retrieval capabilities, which can meet the strict requirements for knowledge accuracy and integrity in professional fields. In the legal field, many law firms use Ragflow to process a large number of legal documents, such as contract reviews and case analyses. In the medical industry, hospitals use Ragflow to analyze patient medical records and medical reports. In the financial field, Ragflow can help analysts process complex documents such as financial statements and market research reports.
Summary
Choose which component to use based on the document types currently being processed, application scenarios, development time cycle, and manpower intensity invested. We hope this article can help you with your AI development tool selection.