Running a RAG pipeline in production: an operations checklist

A RAG system can fail in more places than a typical API. Retrieval can return the wrong chunks while the LLM still writes a confident answer, and nothing raises an error. This checklist covers the places where problems hide.

#Ingestion

• Freshness SLA for each source, such as "SharePoint changes are searchable within 1 hour". Alert on the lag, not only on job failures.
• Deletes are propagated. Removed or permission-revoked documents must leave the index too, or the system leaks data.
• Count documents and chunks per source each run. A sudden 40% drop usually means a connector or parser broke.
• Store the source URL, last-modified time and ACL with every chunk.

#Chunking and embeddings

• Version the embedding model in index metadata. Vectors from different models can't be compared, so changing the model means a full re-embed into a new index, then an alias swap.
• Record the chunk size and overlap you use, and the reason. Re-evaluate them when document types change.
• Watch embedding API latency and 429s during bulk re-indexing (see the throttling post), and use batch endpoints.

#Retrieval quality

• Keep a golden set of 50–200 real questions with known relevant documents.
• Track recall@k and MRR on it nightly, and after every index or config change.
• Prefer hybrid search (keyword + vector) plus a reranker. Product codes, error numbers and names are often missed by vector search alone.
• Know your index parameters (HNSW M, ef_search, or IVF nprobe). Some engines fix them when the index is built. Measure recall, don't assume it.

def recall_at_k(results, golden, k=5):
    """results: {qid: [doc_ids ranked]}, golden: {qid: set(relevant_doc_ids)}"""
    hits = [len(set(results[q][:k]) & golden[q]) / len(golden[q]) for q in golden if golden[q]]
    return sum(hits) / len(hits)

#Generation quality

• Measure groundedness: is every claim supported by a retrieved chunk? An LLM judge works when you calibrate it against human labels.
• Measure the refusal rate. "I don't know" is the right answer when retrieval finds nothing, so track how often it happens.
• Show citations to users, and log which chunks each answer cited.
• Pin model versions, and run the golden set before you switch versions.

#Security

• Security trimming at query time. Filter by the user's groups in the search query. Never rely on the LLM to hide documents.
• Treat retrieved content as untrusted input. Indirect prompt injection in a document ("ignore previous instructions...") is a real attack. Keep tools and actions behind explicit allow-lists.
• Redact secrets and personal data during ingestion, not only at output.
• Set log retention for prompts and answers in line with your data classification.

#Observability

For each request, log a trace containing:

Tip

Each week, read 20 random 👎 answers and add the ones that represent real problems to the golden set. That's how the set stays representative of real traffic.

#Runbook: "the bot is giving wrong answers"

1. Pull the trace. Was the right document retrieved?
2. No: check ingestion (is the document indexed and recent?), then the query rewrite, then search parameters and filters.
3. Yes, but the answer is still wrong: the prompt or model is the issue. Check the context order, truncation, and whether the chunk was cut off at the token limit.
4. Add the case to the golden set so it doesn't come back.