Symptoms
- Users say the assistant doesn't know about a policy or product released yesterday.
- Deleted or replaced documents are still being cited.
- No errors shown to users, because retrieval still "works".
1. Measure freshness
Compare the newest source change with the newest indexed document:
-- source system or document catalogue
SELECT MAX(modified_at) FROM documents WHERE status = 'published';
Then the newest indexed chunk. This needs an indexed_at (or source modified_at) field stored with every chunk; if you don't have one, add it now. For pgvector:
SELECT MAX(indexed_at) FROM chunks;
For Azure AI Search, with indexed_at marked sortable:
GET https://<search>.search.windows.net/indexes/docs/docs?api-version=2024-07-01&search=*&$orderby=indexed_at desc&$top=1&$select=id,indexed_at
api-key: <query-key>
The gap between the two is your freshness lag. Agree with the business what lag is acceptable (minutes, hours).
2. Find the stalled stage
A typical pipeline: change detection → queue → parse/chunk → embed → upsert → (delete handling).
| Stage | What to check |
|---|---|
| Change detection | Webhook subscription expired, delta token lost, crawler schedule disabled |
| Queue | Depth growing, dead-letter queue filling |
| Parse/chunk | Worker crash-looping on one bad file (poison message) |
| Embed | 429s from the embedding API, expired key, quota moved |
| Upsert | Vector store full, schema mismatch after an index change, auth |
| Deletes | Only upserts handled; deletes never propagated |
kubectl get pods -n rag -l app=ingest-worker
kubectl logs -n rag deploy/ingest-worker --tail=200 | grep -iE "error|exception|429|timeout"
For Azure AI Search indexers:
GET https://<search>.search.windows.net/indexers/docs-indexer/status?api-version=2024-07-01
api-key: <admin-key>
Look at lastResult.status, errors and warnings.
3. Fix the common causes
- Poison document: move it to the dead-letter queue and let the rest flow; fix the parser later.
- Expired subscription (for example Microsoft Graph change notifications expire and must be renewed): recreate it and run a delta sync from the last known point.
- Embedding API failures: restore the key or quota; see the quota runbook.
- Index schema change: a new field or a different vector dimension makes upserts fail. Align the pipeline with the index definition.
4. Backfill safely
Re-process documents changed since the last good indexing time. Make upserts idempotent by using stable chunk IDs (document ID + chunk number + content hash), so re-running doesn't create duplicates:
chunk_id = f"{doc_id}:{i}:{hashlib.sha256(text.encode()).hexdigest()[:12]}"
Delete chunks of documents that were removed or replaced in the source during the gap.
Warning
A full re-index during business hours can push the embedding API and vector store into throttling and slow down live queries. Backfill incrementally, rate-limited, starting with the most-used sources.
5. Tell users
If answers may be outdated for a known period, show a notice in the assistant until freshness is back within target.
Prevent it next time
- Alert on freshness lag (newest source change vs newest indexed item), not just on worker errors.
- Alert on queue depth and dead-letter count.
- Renew change-notification subscriptions automatically and alert on renewal failure.
- Handle deletes as first-class events and test them.