Applies toRAG chatbots on Azure OpenAI, Amazon Bedrock, OCI Generative AI, or self-hosted models
When a customer says "the bot gave a wrong answer", you need the full trace for that request: the question, the retrieved chunks, the final prompt and the answer. Without it you're guessing.
1. Where did it go wrong?
| Finding in the trace | Type | Fix area |
|---|---|---|
| The right document wasn't retrieved at all | Retrieval miss | Search, chunking, embeddings |
| The right chunk was retrieved, but the answer contradicts it | Generation failure | Prompt, model, temperature |
| Retrieved chunks are outdated or conflicting | Content problem | Source data, versioning |
| No relevant chunk exists, and the model answered anyway | Missing abstention | Prompt and guardrails |
2. Fix retrieval misses
- Use hybrid search (keyword + vector). Product codes, error numbers and KB IDs ("AADSTS50076") match much better with keywords.
- Add a reranker and pass the top 3–5 reranked chunks, not 20 raw ones.
- Chunk by document structure (headings, sections), keep the title and section path in each chunk.
- Rewrite the user's question into a standalone query in multi-turn chats ("what about the second one?").
3. Fix generation failures
A grounding prompt that works well in support scenarios:
Answer using ONLY the sources below. Each source has an id.
- Cite the source id in square brackets after each claim, e.g. [KB-1042].
- If the sources don't contain the answer, say "I don't have that information"
and suggest contacting support. Don't use outside knowledge.
- Never invent commands, URLs, version numbers or prices.
Sources:
{chunks}
Also:
- Temperature 0–0.3 for factual support answers.
- Put sources before the question, and keep the context focused. Long, noisy context increases errors.
- Validate citations in code: every cited id must be one of the retrieved chunks. Drop or flag answers that cite nothing.
4. Abstain when retrieval is weak
If the best reranker score is below a threshold, don't call the model with weak context. Return a "not sure, here are related articles" response or hand over to a human.
5. Measure it
Track on a fixed test set of real questions with known answers:
- Groundedness (are claims supported by the sources?)
- Answer relevance and context recall
- The abstention rate (too high is also a problem)
Run it on every change to prompts, chunking, models or the index.
Tip
Add a "Was this helpful? 👍 / 👎" button that stores the trace id. Reviewing the thumbs-down traces each week is the fastest way to find what to fix next.