Symptoms
- Customers or agents report the assistant quoted a wrong price, refund rule, eligibility or policy.
- Screenshots on social media or complaints arrive before monitoring notices.
1. Contain first
Pick the smallest action that stops new wrong answers:
| Option | When |
|---|---|
| Block the topic (route "pricing/refund" intents to a fixed message or human) | Problem limited to one topic |
| Roll back the last prompt, model or index change | Started right after a release |
| Switch the assistant to "search results only" mode (links, no generated text) | Unsure of the scope |
| Turn the assistant off and show a contact page | Legal or safety exposure |
A topic block can be a simple pre-check before the LLM call:
BLOCKED = ("refund", "price", "pricing", "cancellation fee")
if any(w in user_msg.lower() for w in BLOCKED):
return FIXED_REPLY_WITH_LINK_TO_POLICY_PAGE
Keep the containment in place until the fix is verified, and tell customer support what the assistant now does.
2. Find the cause
Look at the traces for a few bad answers: the prompt, retrieved chunks and the response.
| Finding | Cause |
|---|---|
| Retrieved chunk has the wrong value | Outdated or wrong source document in the index |
| Correct chunk retrieved, answer still wrong | Model ignored or misread context (prompt, model change) |
| No relevant chunk retrieved | Retrieval gap; model filled it from training data |
| Answer mixes two products | Chunking or metadata filter problem |
Check what changed recently: prompt version, model version or deployment, index rebuild, new documents.
3. Measure the impact
You need to know how many customers got a wrong answer. Search the conversation logs for the period:
-- example against a conversation log table
SELECT conversation_id, user_id, created_at, LEFT(answer, 200)
FROM assistant_messages
WHERE created_at >= '2026-10-25'
AND (answer ILIKE '%refund within 90 days%' OR answer ILIKE '%no cancellation fee%')
ORDER BY created_at;
If you store trace IDs, use your observability tool's search instead. If you don't keep conversation logs, note that in the incident; you will have to rely on complaints and estimates.
Warning
Conversation logs contain personal data. Access them under your data protection process, and limit the export to the fields needed for impact analysis.
4. Fix
- Wrong source document: correct or remove it, re-index, and verify the old chunk is gone (search the index for the wrong value).
- Prompt/model regression: roll back to the last good version; re-run your evaluation set before re-releasing.
- Retrieval gap: add the authoritative document, and instruct the model to say "I don't know, here is the policy page" when context lacks the answer.
Add the failing questions to the evaluation set with the correct expected answers, so the same failure blocks future releases.
5. Notify and correct
With the business, legal and support:
- Decide whether to contact affected customers and whether to honour the wrong answer (some jurisdictions and tribunals have held companies to their chatbot's statements).
- Prepare a support script for incoming questions.
- Record the decision and the customer list in the incident.
6. Re-enable carefully
Remove containment only after:
- The fixed assistant passes the evaluation set, including the new cases.
- A human has reviewed answers to the 20-30 most common questions in the affected topic.
- Monitoring is in place (below).
Prevent it next time
- Ground high-risk topics (price, policy, legal) only in approved documents, and show citations.
- Keep an evaluation set with exact-value checks; run it on every prompt, model and index change.
- Sample and review live answers daily for high-risk topics.
- Give users a "report wrong answer" button that pages the owning team.