Home / AI & GenAI

AI & GenAIHow-toIntermediate

Optimization: tuning the RAG retrieval budget - top-k, reranking depth and how much context to send

Sending 20 chunks "just in case" makes answers slower, more expensive and sometimes worse. Measure retrieval quality, tune how many candidates you retrieve, how many you rerank and how many you send to the model, deduplicate and order context sensibly, and pick the smallest context that keeps answer quality.

Deepak Bhardwaj2 min read

Applies toRAG applications on any vector store or search service (Azure AI Search, OpenSearch, pgvector, Qdrant, Bedrock Knowledge Bases) with optional rerankers (semantic ranker, cross-encoders, hosted rerank models)

Three numbers to tune

Number Meaning Effect
k_retrieve Candidates fetched from the index Higher → better recall, slower search
k_rerank Candidates scored by the reranker Higher → better ordering, more reranker cost/latency
k_context Chunks sent to the LLM Higher → more input tokens, risk of distracting the model

A common shape: retrieve 50 → rerank 50 → send 5–8. Your data decides the right values.

Build an evaluation set

  • 100–300 real questions with the chunk(s) or document(s) that should be used.
  • Metrics: recall@k for retrieval, answer correctness/groundedness for the final answer (see the LLM-as-a-judge post).

Run a sweep

k_retrieve k_rerank k_context Recall@context Answer score Input tokens p95 latency
20 20 10 measure measure measure measure
50 50 5 measure measure measure measure
50 50 8 measure measure measure measure
100 50 5 measure measure measure measure

Pick the cheapest configuration within tolerance of the best answer score.

Make each chunk count

Technique Benefit
Reranker score threshold Drop weak chunks instead of always sending k
Deduplicate near-identical chunks Fewer wasted tokens
Merge adjacent chunks from the same document Coherent context, fewer headers
Put the most relevant chunks first (and/or last) Models attend better to the edges of long contexts
Trim chunk metadata to what's needed for citations Fewer tokens 
ranked = rerank(question, candidates)[:k_rerank]
context = [c for c in ranked if c.score >= 0.35][:k_context]
context = dedupe(context, similarity=0.95)

Dynamic budgets

  • Simple factual questions: fewer chunks.
  • Comparison or summary questions: more chunks, or a map-reduce pattern.
  • Classify the question type cheaply and set k accordingly.

Tip

Track average input tokens per answer alongside answer quality. When someone raises k_context to fix one bad answer, the cost change is visible immediately.

Checklist

  • Evaluation set with expected sources
  • k_retrieve / k_rerank / k_context swept
  • Score threshold and deduplication applied
  • Context ordering considered
  • Dynamic budget by question type where useful
  • Tokens and quality tracked together

Related fixes