Advanced RAG
Explore reranking, query rewriting, hybrid retrieval, and techniques for difficult knowledge tasks.
When basic retrieval is not enough
Simple top-k vector search works well for direct questions over clean prose, but harder tasks may involve multiple documents, exact identifiers, long conversations, tables, or ambiguous terminology. Advanced RAG adds stages that improve query interpretation, candidate recall, evidence ordering, or answer verification. Each stage increases cost and creates another possible failure. Add one only when an evaluation set shows a specific gap. A complicated pipeline without a measured problem is difficult to operate and may make a reliable baseline worse.
Query rewriting
A rewrite can turn a conversational question such as “What about contractors?” into a standalone query that includes the earlier subject. It can also expand abbreviations or produce several subqueries for a multi-part request. Preserve the original question and constraints, and validate that the rewrite did not add an unsupported assumption. For sensitive searches, use deterministic expansion or user confirmation when ambiguity affects permissions or scope. Log both forms so a missed result can be traced to the original wording or to the rewrite stage.
question = "What exceptions apply to contractor leave?"
vector_hits = vector_store.search(question, top_k=20)
keyword_hits = keyword_index.search(question, top_k=20)
candidate_ids = {hit["id"] for hit in vector_hits + keyword_hits}
candidates = [vector_store.get(item_id) for item_id in candidate_ids]
scores = reranker.rank(
query=question,
documents=[item["text"] for item in candidates],
)
for score, item in sorted(
zip(scores, candidates), key=lambda pair: pair[0], reverse=True
)[:5]:
print(score, item["source"], item["text"])Multi-query and decomposition
A complex question may need separate searches for definitions, dates, exceptions, and supporting evidence. Query decomposition can retrieve each part independently before combining results. The application should retain which subquestion produced each chunk and detect when one part has no support. Do not let a planner drop constraints while splitting the task. Set limits on the number of subqueries and total candidates, because decomposition can multiply latency and token usage. For simple questions, bypassing this stage is often faster and just as accurate.
Reranking evidence
A reranker examines the question and candidate passages together and predicts which passages are most useful. It can resolve cases where embedding similarity favors a broad topical match over a precise answer. Retrieve a wider candidate set first, then rerank only that bounded set. Measure whether the reranker improves evidence recall, answer quality, and tail latency on your data. Preserve the initial and final rankings in logs. A reranker cannot recover a document that was never retrieved, and it may introduce its own language, domain, or bias failures.
Parent and hierarchical retrieval
Small chunks are precise, but they may lack surrounding definitions or exceptions. Parent retrieval stores small searchable units while returning a larger parent section or document window after a match. Hierarchical systems can first identify a document, then a section, then a paragraph. These approaches improve context when structure is strong, but they can return too much irrelevant text. Cap the expanded window, preserve section labels, and evaluate whether the added context improves answer completeness without crowding out stronger evidence from other sources.
Handling tables and relationships
Questions about tables, timelines, product variants, or linked policies require more than plain semantic similarity. Store structured rows or fields when possible, preserve headers with every value, and consider deterministic filters or calculations before generation. For a question spanning documents, retain entity and document relationships so retrieval can follow an approved graph rather than guessing from text alone. Let the model explain the result after trusted computation. This reduces errors where a model reads a visually arranged table as if it were ordinary prose.
Conflict and answerability
Advanced systems should detect when sources disagree, are outside their effective dates, or fail to cover the question. Rank authoritative and current sources using trusted metadata, but do not hide a meaningful conflict. The answer can explain which source is newer, cite both, and ask the user to confirm the applicable scope. If evidence is absent, produce a clear abstention with a useful next step. Confidence should reflect evidence quality and coverage, not merely the model’s fluent wording or a high similarity score.
Evaluating advanced designs
Compare each advanced technique with a simpler baseline on the same fixed questions and source versions. Track retrieval recall, reciprocal rank, evidence completeness, citation precision, answer faithfulness, answer correctness, abstention quality, latency, and cost. Include adversarial queries, stale documents, access-control boundaries, and no-answer cases. Review failures by stage and keep examples in regression tests. The best design is the one that improves the product’s measured outcomes within its reliability and budget limits, not the one with the most retrieval components.
Worked example: a two-part policy question
For “What is the contractor leave allowance, and who approves an exception?” a single vector query may retrieve the allowance but miss the approval workflow. Decomposition can create two bounded subqueries, retrieve evidence for each, and require the final response to cite both. If the approval policy is absent, the system should answer the allowance and explicitly mark the second part unresolved. This is better than allowing a planner to invent a complete response from one partially relevant passage.
Code walkthrough
The sample merges vector and keyword candidate IDs, then sends the bounded set to a reranker. The merge improves recall, while reranking spends a more expensive model call only on candidates already retrieved. In production, preserve source metadata through the merge, cap candidate count, deduplicate by document version, and apply authorization before reranking. Sort scores together with their records rather than losing the mapping. Compare this pipeline against vector-only retrieval so added complexity has measurable justification.
Practical exercise
Build a ten-question evaluation set with direct lookups, paraphrases, exact identifiers, a multi-part question, a table question, a stale-policy question, and two questions whose answers are absent. Run vector-only, hybrid, and hybrid-plus-reranking configurations. Record top-five evidence, final answer, citations, latency, and cost. For every improvement, identify whether it came from better recall, better ordering, or a better prompt. Keep any regression as a permanent test before changing the pipeline again.
Sources and further reading
These primary or specialist references informed the concepts in this guide. Product details can change, so verify current documentation before implementation.