Home / AI & GenAI

AI & GenAIHow-toIntermediate

Evaluating LLM answers with LLM-as-a-judge: a practical setup

Build a test set, write a clear grading rubric, calibrate the judge against human labels, and run evaluations in CI so prompt and model changes don't silently reduce quality.

Deepak Bhardwaj2 min read

Applies toAny LLM application; frameworks such as Azure AI Foundry evaluations, Bedrock model evaluation, Ragas, promptfoo, DeepEval

"It looks better" isn't a release criterion. An automated judge lets you compare versions on hundreds of cases in minutes.

1. Build the test set

  • 100–300 real questions from tickets, chats or search logs, not invented ones.
  • For each: the expected answer or key facts, and the source document id (for RAG).
  • Include hard cases: ambiguous questions, questions with no answer in the KB, multi-step questions, other languages.
  • Keep it in version control.

2. Write a narrow rubric

One judge prompt per criterion works better than one prompt scoring everything.

You are grading a customer support answer for FACTUAL CORRECTNESS.

Question: {question}
Reference facts: {reference}
Answer to grade: {answer}

Score:
1 = contradicts the reference or contains a wrong command/value
2 = partly correct, missing an important fact
3 = correct and complete with respect to the reference

Return JSON: {"score": <1-3>, "reason": "<one sentence>"}

Other useful criteria: groundedness (claims supported by the retrieved sources), relevance, appropriate refusal (says "I don't know" when it should), tone/policy.

3. Reduce judge bias

  • Use a small discrete scale (pass/fail or 1–3), not 1–10.
  • For A/B comparisons, run both orders (A vs B and B vs A). Judges favour the first answer.
  • Judges favour longer answers. State in the rubric that length isn't a merit.
  • Use a strong model as judge, ideally not the same one that produced the answers.
  • Temperature 0, and parse the JSON strictly.

4. Calibrate against humans

Have two people label 50–100 answers with the same rubric. Measure agreement between the judge and humans (simple % agreement or Cohen's kappa). If agreement is low, the rubric is unclear: fix it before trusting the scores.

5. Run it in CI

# promptfoo-style example
tests: file://tests/support_questions.yaml
assert:
  - type: llm-rubric
    value: "Answer is factually consistent with the reference and gives no invented commands"
thresholds:
  pass_rate: 0.9

Fail the pipeline when the pass rate drops more than a set margin below the current production version.

6. Keep humans in the loop

Review a sample of judge decisions every week, especially failures. Add new production failure cases to the test set.

Tip

Track cost and latency next to quality. A change that adds 2% quality and doubles the cost per answer should be a conscious decision, not a surprise on the invoice.

Related fixes