Applies toAny LLM application (chat, RAG, extraction, agents) on hosted or self-hosted models; evaluation tooling of your choice
Define quality per use case
| Use case | Quality dimensions |
|---|---|
| RAG assistant | Correctness, groundedness (supported by sources), citation accuracy, refusal when unknown |
| Extraction | Field-level accuracy, schema validity |
| Classification/routing | Accuracy, confusion between important classes |
| Agent | Task success rate, steps, cost, unsafe actions |
| Summarisation | Coverage of key points, no invented facts, length |
Add safety dimensions for all: harmful content, data leakage, prompt injection resistance.
Offline evaluation
| Element | Guidance |
|---|---|
| Test set source | Real (anonymised) user requests plus edge and adversarial cases |
| Size | Start with 100–300; grow with every incident and bug |
| Labels | Expected answer, key facts, or rubric per case |
| Graders | Rules → judge model → human sample (see the LLM-as-a-judge and eval cost posts) |
| Versioning | Test set, prompts and rubrics in Git |
Release gates
| Change | Gate |
|---|---|
| Prompt edit | Smoke + regression suite pass, no drop beyond tolerance |
| Model/version change | Full suite + human review sample |
| Retrieval change (chunking, embeddings, k) | Retrieval metrics + answer quality |
| New tool for an agent | Scenario tests including misuse |
See the LLMOps foundation post for wiring gates into CI/CD.
Online evaluation
| Signal | Use |
|---|---|
| Thumbs up/down, comments | Quick quality signal; biased but useful for trends |
| Escalations/handoffs to humans | Failure indicator |
| Sampled conversations graded weekly | Unbiased quality measure |
| Guardrail triggers | Safety trends |
| A/B tests | Compare variants on real traffic where risk allows |
Close the loop
- Every bad production case becomes a test case.
- Monthly quality report per use case with trends.
- Owners per use case accountable for quality metrics.
Tip
Start small: 50 real cases with clear expected answers, checked on every change, beats a sophisticated framework you'll set up "later".
Checklist
- Quality dimensions per use case defined
- Offline test set from real traffic, versioned
- Graded by rules, judges and humans
- Release gates in CI/CD
- Online signals and sampled reviews
- Production failures fed back as tests