Skip to main content

RAG Optimization

Diagnose and improve RAG quality across indexing, retrieval, and generation stages

RAG (Retrieval Augmented Generation) quality depends on three stages:
  1. Indexing: Document parsing, chunking, and vectorization
  2. Retrieval: Matching and recalling relevant chunks from the index
  3. Generation: The LLM generates answers based on retrieved content and the query
Each stage can introduce quality issues. This guide follows a diagnose → fix pattern to help you locate problems and apply targeted solutions.

Step 1: Establish a Baseline

Before optimizing, you need a quantifiable evaluation standard to measure whether changes are effective.

Build an Evaluation Set

Prepare at least 100 test cases, each containing a question and expected result. Cover the following types:
TypeExample
Factual"What is the warranty period for Product X?"
Comparative"Compare the main differences between Product X and Product Y"
Tutorial"How to install Product X?"
Analytical"Why has Product X's sales been increasing over the past 3 months?"

Run a Baseline Test

Use the QA service debug panel or the chat API to run the full evaluation set. Record the retrieved chunks and answers for each case. This report serves as the reference baseline for all subsequent optimizations.

Step 2: Diagnose and Improve

Analyze underperforming cases from the baseline report and apply the corresponding solutions below.

2.1 Retrieval Failure: Relevant Knowledge Not Found

The knowledge base contains relevant content, but retrieval fails to match it.
If the knowledge base lacks relevant information, the model cannot answer the question. Check whether the knowledge base covers the topics in your evaluation cases and supplement missing content.
Document format issues may cause key content to be lost during parsing:
  • Ensure clear heading hierarchy and content structure
  • Remove watermarks
  • Avoid complex tables with merged or cross-page cells
  • Prefer Markdown format; convert PDF/DOCX to Markdown before importing
If queries are primarily in English or another language, source documents should use the same language. For technical terms, consider multilingual annotation.
Standardize different expressions for the same entity. For example, unify "ML", "Machine Learning", and "机器学习" to a single form.
You can use an LLM to help standardize terminology. For long documents, split into parts and process sequentially.
In multi-turn conversations, user input may be incomplete or lack context. Query rewriting automatically completes the query based on conversation history, improving retrieval matching.Example: User inputs Phone X1, system rewrites to Please provide specifications for all versions of Phone X1 currently on sale.Enable the Query Rewriting toggle in the per-knowledge-base configuration of Knowledge Retrieval or Knowledge QA.

2.2 Retrieval Failure: Irrelevant Results Returned

The knowledge base has relevant content, but retrieval returns unrelated chunks.
Typical problem: The knowledge base contains multiple categories of documents, and results from category B appear when searching category A.Solution: Add tags to documents and filter by tags during retrieval.Setup:
  • During import: Add tags in step 3 of Upload Data
  • After import: Edit tags from the "More" menu in the document list
Usage:
  • Set Tag Filter in per-knowledge-base configuration of retrieval or QA services
  • Specify tags in API request parameters
Typical problem: Multiple documents with similar structure (e.g., all containing a "Feature Overview" section) cause retrieval to return content from the wrong document.Solution: Define document metadata so the system performs structured search before vector retrieval, precisely locating the target document.How it works:
  1. Extract metadata from the query (e.g., {"key": "name", "value": "Phone X1"})
  2. Filter chunks containing that metadata
  3. Perform vector retrieval on the filtered results
Impact: Without metadata, unrelated chunks (40-41% similarity) may rank above the target chunk (37%). After enabling metadata, the target chunk's similarity rises to 65% and ranks first.Click "Meta Info" in the document list to view and configure metadata.

2.3 Incomplete Chunks

Improper chunking strategies lead to the following issues:
IssueSymptomImpact
Chunks too shortA single chunk contains only one sentence, missing contextCannot match relevant content during retrieval
Chunks too longA single chunk mixes multiple topicsReturns irrelevant information during recall
Semantic truncationChunks are forcibly cut in the middle of a sentenceRetrieved content is incomplete
Smart chunking adaptively selects chunk boundaries based on semantic relevance rather than fixed lengths:
  1. The system first divides the document into paragraphs using built-in sentence markers
  2. Then determines chunk boundaries based on semantic relevance between paragraphs
Suitable for most scenarios. Select during knowledge base creation or data import.
Automatic chunking may still produce unexpected truncations or parsing errors (e.g., spaces parsed as %20). Spot-check chunk quality after import.View chunks in the Chunk Details tab of the knowledge base detail page. Edit and fix any issues directly. Fixed content takes effect immediately for retrieval.
Fixes only affect indexed chunks, not source documents. Re-importing requires re-checking.
For edge cases that automatic chunking cannot handle:
  • Add chunk: Supplement missing key information or additional context. Select the parent document, max 6000 characters
  • Delete chunk: Remove chunks with incorrect or duplicate content without affecting other chunks from the same document
See Document Management — Chunk Management.

2.4 Poor Reranking

Chunks are retrieved but ranking is suboptimal, leading to poor final results.
The similarity threshold filters out chunks scoring below the threshold after reranking.
  • Too high: May discard all relevant chunks, leaving the model without sufficient information
  • Too low: May include many low-relevance chunks, introducing noise
There is no universal best threshold. Test iteratively using the debug panel. Start from 0.2.Configuration: Similarity Threshold (0.01~1.0) in per-knowledge-base configuration of Knowledge Retrieval or Knowledge QA.
Max recall count determines how many chunks are returned. For complex questions requiring summarization, enumeration, or comparison, providing more quality chunks (e.g., K=20) usually works better than just 5.However, too many chunks may exceed the model's input length limit, causing content truncation.Configuration: Max Recall Count (1~20) in per-knowledge-base configuration.
The reranking model reorders initially recalled chunks. Available modes:
ModeUse Case
QA mode (default)User asks a complete question, expects an answer from chunks
Similarity modeQuery and chunks share similar phrasing, e.g., finding historical records
Custom advanced modeSpecial ranking rules needed, e.g., "prioritize content from the last year"

2.5 Model Comprehension Issues

Correct chunks are retrieved, but the model generates inaccurate answers.
Typical problem: Answers appear to be crudely assembled; the model fails to understand the relationship between knowledge and the question.Different models vary in instruction following, knowledge comprehension, and long-text handling:
ScenarioRecommended Model
Simple information lookup and summarySmaller models (e.g., qwen3-flash)
Complex logical reasoningLarger models (e.g., qwen3-max)
Queries requiring many chunksModels with larger context windows
Switch the Model in the QA service configuration page.
Typical problem: Results don't follow requirements or lack completeness.Three common optimization methods:Method 1: Constrain output content
If the provided information is insufficient to answer the question,
clearly state "Based on available information, I cannot answer this question."
Never fabricate answers.
Method 2: Add few-shot examplesInclude input-output examples in the prompt to guide the model's format and style.Method 3: Separate knowledge contentClearly separate instructions and knowledge base content with delimiter markers so the model can better understand the structure.
Typical problem: Answers include the model's general knowledge rather than strictly using the knowledge base.Enable the Rejection toggle in the QA service configuration. When retrieval results are insufficient, the model refuses to answer and returns a custom response instead of fabricating an answer.Configuration: Rejection toggle in Knowledge QA.
Typical problem: Same question produces different answers each time, or answers are too similar when variety is desired.Adjust the temperature parameter:
  • Higher values: More diverse output, suitable for creative writing and brainstorming
  • Lower values: More deterministic output, suitable for factual queries and technical documentation
Click the settings icon next to the model in the QA service configuration page.

2.6 Multi-Knowledge-Base Optimization

When binding multiple knowledge bases, these strategies can improve joint retrieval quality:
StrategyDescription
Set knowledge base weights/priorityAssign higher weights or rankings to core knowledge bases so their results dominate the final ranking
Enable knowledge base routingThe system automatically determines which bases to query based on intent, reducing interference from irrelevant bases. Configure in Knowledge Retrieval — Global Config
Choose reranking model modeQA mode for Q&A scenarios; Similarity mode for finding semantically similar passages; Custom advanced mode for special ranking needs

Step 3: Continuous Iteration

After each configuration change:
  1. Re-evaluate: Rerun the same evaluation set to quantify improvements
  2. Compare: Check against the baseline report to confirm which issues were resolved and whether new issues were introduced
  3. Expand coverage: Add newly discovered bad cases to the evaluation set
Optimization is an iterative process, typically requiring 2-3 rounds of adjustments. Start with indexing and retrieval before optimizing generation.
Getting Started
Playground
Data Ingestion
Knowledge Service
Cookbook
Integration
Settings, Billing & Limits