Home / AI & GenAI

AI & GenAIHow-toIntermediate

Optimization: running LLM evaluations without a huge bill - tiered suites, smaller judges, caching and batch

Evaluation is essential for safe releases, but running every test with a large judge model on every commit gets expensive. Split suites into fast and full tiers, use rule-based checks before judge models, calibrate smaller judges against a strong one, cache results for unchanged cases and run big suites through batch APIs.

Deepak Bhardwaj2 min read

Applies toLLM evaluation pipelines in CI/CD (custom, promptfoo-style tools, Azure AI Foundry evaluations, Bedrock evaluations, open-source eval frameworks)

Tiered suites

Tier When Size Checks
Smoke Every commit / PR 20–50 cases Rules + small judge
Regression Before merge to main or nightly 200–500 cases Rules + calibrated judge
Full Before release, model change Thousands Strong judge, human sample

Cheap checks first

Many quality checks don't need an LLM:

Check Method
Valid JSON / schema Parser and schema validation
Required citation present Regex / structure check
Forbidden content (secrets, competitor names, PII) Pattern matching, classifiers
Exact answers (IDs, numbers) String/number comparison
Retrieval recall Compare retrieved chunk IDs to expected

Run the judge only on cases that pass cheap checks or need semantic grading.

Smaller judges, calibrated

  1. Grade a few hundred cases with a strong judge (and some human labels).
  2. Grade the same cases with a smaller judge.
  3. Measure agreement. If it's high enough for the tier, use the smaller judge there.
  4. Re-calibrate when prompts or rubrics change.

See the LLM-as-a-judge post for rubric design.

Reuse and cache

  • Cache judge results keyed by (case ID, app output hash, rubric version, judge model). If the app output didn't change, don't re-judge.
  • Prompt caching for the judge's long rubric (see the prompt caching post).

Batch for big suites

Full suites aren't latency sensitive. Submit them through batch APIs at lower prices (check current pricing) and collect results when ready.

Track evaluation cost

Metric Use
Cost per run per tier Budget and trend
Judge calls avoided by cache Value of caching
Time per tier Developer feedback loop

Tip

Keep the smoke tier under a few minutes and a small cost. Developers then run it on every change, which catches more problems than a perfect suite nobody waits for.

Checklist

  • Smoke, regression and full tiers defined
  • Rule-based checks before judges
  • Smaller judges calibrated against strong judge/humans
  • Results cached by output hash and rubric version
  • Full suites via batch APIs
  • Evaluation cost tracked

Related fixes