Symptoms
- Spike in "I couldn't find that" answers, or answers citing unrelated documents.
- User thumbs-down rate jumps the day after an ingestion change.
- Nothing changed in the model or prompt.
Step 1: confirm it's retrieval, not generation
Take 10–20 failing questions and look at what was retrieved:
for q in failing_questions:
hits = search(q, top=5)
print(q)
for h in hits:
print(f" {h.score:.3f} {h.source} {h.text[:80]!r}")
If the right passages aren't in the top results, it's retrieval. If they are, but the answer is still wrong, look at the prompt/model instead.
Step 2: what changed?
| Change | How it breaks retrieval | Check |
|---|---|---|
| Chunk size / overlap | Answers split across chunks; headings lost | Compare chunk counts and average length old vs new |
| Embedding model or version | Query and document vectors from different models don't match | Index metadata, ingestion config; query embedder config |
| Text extraction (PDF parser, OCR) | Tables/text missing or garbled | Inspect raw chunk text for a known document |
| Filters / security fields | Documents filtered out for everyone | Count docs per source/allowed_groups value |
| Partial ingestion | Indexer failed halfway; only part of the corpus | Document counts vs source |
| Analyzer / language settings (hybrid search) | Keyword side stops matching | Index schema diff |
| Semantic ranker / reranker turned off or changed | Worse ordering | Query config |
Quick checks:
print(old_index.count(), new_index.count())
print(index_metadata["embedding_model"], query_config["embedding_model"])
Azure AI Search: compare index definitions and document counts:
curl -s "https://srch-prod.search.windows.net/indexes/kb-v12/stats?api-version=2024-07-01" -H "api-key: $KEY"
curl -s "https://srch-prod.search.windows.net/indexes/kb-v11/stats?api-version=2024-07-01" -H "api-key: $KEY"
Step 3: roll back
If you build each version into a new index and point the app at it through an alias or config value, rollback is a pointer change:
# Azure AI Search index alias (where available), or switch the app's index name in config
curl -X PUT "https://srch-prod.search.windows.net/aliases/kb-current?api-version=2024-07-01" \
-H "Content-Type: application/json" -H "api-key: $ADMIN_KEY" \
-d '{"name":"kb-current","indexes":["kb-v11"]}'
Check whether aliases are GA or preview for your API version. A config value holding the index name works everywhere.
If you rebuilt in place (same index), there's no quick rollback: restore from a snapshot if your vector store supports it, or re-run the previous pipeline version into a new index. That's the lesson for next time.
Step 4: fix forward
- Restore the previous chunking/extraction settings, or fix the new ones and re-test.
- If the embedding model changed deliberately, re-embed everything and the query side together (see the embedding migration post).
- Re-run ingestion for missing documents.
Step 5: add a retrieval gate
Before switching the alias to a new index, run an automatic check:
| Metric | Gate example |
|---|---|
| Document count | Within ±2% of source system count |
| Recall@5 on a labelled question set | Not more than 3 points below the current index |
| Zero-result rate on the set | Not higher than current |
| Spot check | 10 known questions return their known source in top 3 |
def recall_at_k(index, qa_pairs, k=5):
hit = sum(any(h.doc_id == expected for h in index.search(q, top=k)) for q, expected in qa_pairs)
return hit / len(qa_pairs)
Tip
A labelled set of 100–200 real questions with their correct source documents is the single most useful asset for RAG operations. Build it from support tickets and user feedback.
Prevent it next time
- Blue/green indexes with an alias or config pointer; never rebuild in place.
- Retrieval gate in the ingestion pipeline.
- Index metadata (embedding model, chunking version, pipeline commit) stored with each index.
- Monitoring on zero-result rate and negative feedback per day.