Advanced RAG retrieval: query rewriting, multi-query, HyDE and question decomposition, measured

Users do not type good search queries. They write "and what about the second one?", "it's still failing", or three questions in one sentence. Embedding that text directly retrieves poorly. Query transformation is the cheapest place to win recall back.

#Measure first

Build a retrieval evaluation set: 50-200 questions, each with the IDs of the chunks or documents that should be retrieved.

def recall_at_k(eval_set, retrieve, k=5):
    hits = 0
    for item in eval_set:
        got = {r["doc_id"] for r in retrieve(item["question"], k)}
        hits += bool(got & set(item["relevant"]))
    return hits / len(eval_set)

Record recall@k, MRR and p95 latency for the baseline. Every technique below adds LLM calls; keep it only if it pays for itself.

#Technique 1: standalone-question rewriting (conversations)

Follow-up questions depend on history. Rewrite them into a self-contained query:

REWRITE = """Rewrite the user's last message as a standalone search query using the conversation.
Keep product names, error codes and versions exactly. Output only the query."""

def standalone(history, last_msg):
    convo = "\n".join(f"{m['role']}: {m['content']}" for m in history[-6:])
    r = llm.chat.completions.create(model=SMALL_MODEL, temperature=0, max_tokens=80,
        messages=[{"role": "system", "content": REWRITE},
                  {"role": "user", "content": f"{convo}\nuser: {last_msg}"}])
    return r.choices[0].message.content.strip()

"Does it support PITR?" after a conversation about Azure Database for PostgreSQL becomes "Azure Database for PostgreSQL Flexible Server point-in-time restore support". This is usually the single biggest win for chat interfaces. Use a small, fast model.

#Technique 2: multi-query with reciprocal rank fusion

Generate several phrasings, retrieve for each, fuse rankings:

MULTI = """Write 4 different search queries that would find documents answering the question.
Vary terminology (e.g. error text, product names, symptoms). One per line, no numbering."""

def multi_query(q):
    r = llm.chat.completions.create(model=SMALL_MODEL, temperature=0.4, max_tokens=200,
        messages=[{"role": "system", "content": MULTI}, {"role": "user", "content": q}])
    return [q] + [l.strip() for l in r.choices[0].message.content.splitlines() if l.strip()][:4]

def rrf(result_lists, k=60):
    scores = {}
    for results in result_lists:
        for rank, r in enumerate(results):
            scores[r["doc_id"]] = scores.get(r["doc_id"], 0) + 1 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

def retrieve_multi(q, k=5):
    lists = [retrieve(v, k=20) for v in multi_query(q)]     # run in parallel in production
    return rrf(lists)[:k]

Run the retrievals concurrently; latency then grows by roughly one LLM call, not five searches. RRF needs no score normalisation, which is why it is the standard fusion method (it is also used for hybrid keyword + vector search).

#Technique 3: HyDE (hypothetical document embeddings)

Ask the model to write a plausible answer, then embed that instead of the question. Answers look more like documents than questions do.

def hyde_retrieve(q, k=5):
    r = llm.chat.completions.create(model=SMALL_MODEL, temperature=0.2, max_tokens=200,
        messages=[{"role": "system", "content": "Write a short technical passage that answers the question. It may be wrong; it is only used for search."},
                  {"role": "user", "content": q}])
    return retrieve(r.choices[0].message.content, k)

Warning

HyDE can steer retrieval toward the model's own wrong assumptions, especially on internal topics the model has never seen (your product names, internal tools). Combine it with the original query (multi-query + RRF) rather than replacing it, and check it on your eval set.

#Technique 4: decomposition for multi-part questions

"Compare backup retention and DR options for Azure SQL and RDS" needs evidence on four topics. Split it:

DECOMP = """Split the question into 2-4 independent sub-questions needed to answer it. One per line.
If it is already a single question, return it unchanged."""

def decompose(q):
    r = llm.chat.completions.create(model=SMALL_MODEL, temperature=0, max_tokens=150,
        messages=[{"role": "system", "content": DECOMP}, {"role": "user", "content": q}])
    return [l.strip() for l in r.choices[0].message.content.splitlines() if l.strip()]

def retrieve_decomposed(q, per_sub=3):
    seen, out = set(), []
    for sub in decompose(q):
        for r in retrieve(sub, per_sub):
            if r["doc_id"] not in seen:
                seen.add(r["doc_id"]); out.append(r)
    return out

Give the generator the sub-questions too, so it structures the answer around them.

#Keep exact terms exact

Rewriting models sometimes "fix" error codes or rename products. Protect them:

• Extract tokens matching patterns (error codes like ORA-\d+, 0x[0-9A-F]+, CVE IDs, version numbers) and require them in every rewritten query.
• Always include the original query among the variants.
• Use hybrid search so exact tokens still match on the keyword side.

#Add a reranker after fusion

Query expansion raises recall but adds noise. A cross-encoder or hosted reranker (Azure AI Search semantic ranker, Cohere Rerank, bge-reranker) reorders the fused top 30-50 down to the 5 you send to the LLM. Recall from expansion plus precision from reranking is the usual production combination.

#Routing: do not transform everything

A small classifier prompt (or rules) picks the path, so simple queries stay fast.

#Report the results

Fill it with your numbers. Keep the variants that move recall meaningfully within your latency budget.

#Prevent it next time

• Retrieval eval set in CI; fail changes that drop recall@k.
• Log the original and transformed queries for every request, so bad rewrites are visible.
• Cache rewrites for repeated questions.