Building a RAG system that demos well takes an afternoon. Building one that answers reliably across a real document corpus takes considerably longer, because the difficulty sits in retrieval rather than generation. If the right passage does not reach the model, no amount of prompt refinement produces a correct answer. This guide covers the architecture decisions that determine retrieval quality, how to evaluate the components separately, and the failure modes that appear once real users ask questions you did not anticipate.
Preparing and Chunking the Corpus
Chunking is the most consequential decision in a RAG system and the one most often made by accepting a default. Split too small and passages lose the context that makes them meaningful. Split too large and retrieval returns mostly irrelevant text that dilutes the useful portion. The right approach depends on document structure, and treating a policy manual, a set of support articles, and a contract library identically produces poor results on at least two of them.
Chunking Along Document Structure
Split at natural boundaries such as sections and headings rather than at fixed character counts. Structural splitting keeps semantically complete units together.
Preserving Context in Each Chunk
Include the document title and section path within the chunk. A passage retrieved without indication of where it came from produces answers lacking necessary qualification.
Handling Tables and Structured Content
Tables fragment badly under naive splitting. Extract and represent them separately, since a partial table is frequently worse than no table at all.
Managing Document Versions
Superseded documents retrieved alongside current ones produce contradictory answers. Version metadata and filtering are essential in any corpus that changes.
Building a Repeatable Ingestion Pipeline
Corpora change continuously. Automated ingestion through data engineering capability keeps the index current rather than accurate only at launch.
Improving Retrieval Beyond Basic Similarity
Vector similarity alone underperforms on real corpora, particularly for queries containing exact identifiers, product codes, or specific terminology where semantic similarity is the wrong measure. Production systems generally combine several retrieval methods and rerank the results, because each method fails differently and the combination covers more query types than any single approach. This is where most of the achievable quality improvement sits.
Combining Keyword and Vector Search
Hybrid retrieval catches both exact matches and semantic relevance. Queries containing identifiers or rare terms perform poorly under pure vector search.
Reranking Retrieved Candidates
Retrieve generously, then rerank with a more precise model. This two-stage approach consistently outperforms retrieving a small set directly.
Filtering on Metadata
Restricting by date, document type, department, or permission before ranking reduces the candidate pool and eliminates entire classes of irrelevant result.
Handling Multi-Part Questions
Queries containing several distinct questions retrieve poorly as one unit. Decompose them and retrieve separately, then compose the answer from both sets.
Enforcing Access Control at Retrieval
Users must not receive passages from documents they cannot access. Apply permission filtering during retrieval, since restricting output afterwards is unreliable.
Evaluating Retrieval and Generation Separately
Most teams evaluate the final answer and stop, which conflates two failure modes requiring different fixes. If the correct passage was never retrieved, prompt engineering cannot help. If it was retrieved and the answer is still wrong, retrieval is not the problem. Measuring the stages independently tells you which to work on, and this single practice separates systems that improve steadily from those that plateau.
Building a Question and Answer Test Set
Assemble real questions with known correct source passages. This is tedious and is the foundation for every subsequent measurement.
Measuring Retrieval Independently
Check whether the correct passage appears in retrieved results, regardless of the final answer. Retrieval failures and generation failures need entirely different remedies.
Assessing Groundedness of Answers
Verify that responses follow from retrieved content rather than from model knowledge. Ungrounded answers are the hardest failure for users to detect.
Testing Out-of-Scope Questions
Include questions the corpus cannot answer. The system should decline rather than construct a plausible response from loosely related passages.
Monitoring Quality After Deployment
Real queries differ from test sets. Ongoing observation through MLOps services surfaces the question types you did not anticipate.
Handling the Failure Modes That Appear in Production
Systems that perform well in testing fail in production in recognisable ways. Users ask questions the corpus cannot answer, and the system answers anyway. Retrieved passages contradict each other because outdated documents remain indexed. Answers are correct but unverifiable because no source was cited. Each of these is addressable, but only if anticipated, since users encountering them tend to stop trusting the system rather than reporting the specific fault.
Declining When Coverage Is Absent
Instruct and test for refusal when retrieved content does not support an answer. Confident responses to unanswerable questions destroy trust quickly.
Citing Sources for Every Answer
Show which documents informed the response. This makes answers verifiable and makes errors diagnosable, since you can see which passage caused them.
Resolving Contradictory Sources
Where documents conflict, surface the conflict rather than selecting arbitrarily. Silent selection between contradictory sources is the most dangerous behaviour available.
Keeping the Corpus Current
Answer quality tracks content quality. Assign ownership for document accuracy, since retrieval faithfully reproduces outdated information with full confidence.
Managing Inference Cost at Volume
Reranking and generation costs accumulate. Reviewing AI cost optimisation approaches keeps unit economics viable as usage grows.
Frequently Asked Questions
What is the hardest part of building a RAG system?
Retrieval, not generation. If the correct passage never reaches the model, no prompt refinement produces a correct answer. Most quality improvement comes from chunking, hybrid search, and reranking rather than from generation-side work.
How should I chunk documents?
Along natural structural boundaries such as sections and headings rather than fixed character counts, preserving document title and section context within each chunk. The right approach varies by document type, so uniform settings across a mixed corpus underperform.
Is vector search enough on its own?
Usually not. Pure vector search handles semantic similarity but performs poorly on exact identifiers, product codes, and rare terminology. Combining keyword and vector retrieval, then reranking the candidates, covers considerably more query types.
How do I evaluate a RAG system?
Measure retrieval and generation separately using a test set of real questions with known source passages. Conflating them hides which stage is failing, and the two require entirely different remedies.
Why does my system give confident wrong answers?
Usually because it answers questions the corpus does not cover, or because outdated documents remain indexed alongside current ones. Test refusal behaviour explicitly and apply version filtering so superseded content is not retrieved.
Should answers cite their sources?
Yes. Citation makes answers verifiable for users and makes errors diagnosable for you, since you can trace a wrong answer to the passage that caused it. In regulated contexts it is frequently a requirement.
