Applies toAny LLM application using hosted APIs (Azure OpenAI, Bedrock, Anthropic, OpenAI, Gemini) or self-hosted models; AI gateways with routing features
Three approaches
| Approach | How | Fits |
|---|---|---|
| Static by task | Each feature uses a fixed model chosen by evaluation (classification → small, complex reasoning → large) | Clear, separate tasks |
| Router | A classifier (rules, small model, or embedding similarity) picks the model per request | Mixed traffic in one feature |
| Cascade | Try small model; escalate to large if confidence/validation fails | Most requests easy, some hard |
Static by task: start here
| Task | Usually fine with |
|---|---|
| Classification, routing, extraction to JSON | Small/fast models |
| Summaries of short texts | Small–medium |
| RAG answers over good retrieval | Medium |
| Multi-step reasoning, code generation, agents | Large |
Confirm with your own evaluation set; don't assume.
Router sketch
def choose_model(request):
if request.task in ("classify", "extract"):
return "small-model"
if len(request.context_tokens) > 20000 or request.needs_tools:
return "large-model"
score = difficulty_classifier(request.text) # small model or heuristic
return "large-model" if score > 0.7 else "medium-model"
Cascade sketch
answer = call("small-model", prompt)
if not passes_checks(answer): # schema valid, cites sources, self-rated confidence, etc.
answer = call("large-model", prompt)
Checks must be cheap and reliable: JSON schema validation, required citations present, or a verifier model.
Prove quality
- Build an evaluation set (a few hundred real requests with expected outcomes).
- Measure quality, cost and latency for: large-only, small-only, routed/cascaded.
- Accept routing only if quality stays within an agreed tolerance.
- Re-run when models or prompts change (see the LLMOps foundation post).
Track in production
| Metric | Why |
|---|---|
| Share of traffic per model | Is routing behaving as expected? |
| Escalation rate (cascade) | Too high → small model not good enough; cost may rise |
| Quality signals per model (feedback, eval samples) | Catch silent quality loss |
| Cost per request / per feature | The point of the exercise |
Tip
A gateway (see the AI gateway foundation post) is a good place for routing rules, so you can change models per feature without redeploying applications.
Checklist
- Tasks mapped to model sizes with evaluation
- Router or cascade only where traffic is mixed
- Cheap, reliable escalation checks
- Quality, cost, latency compared on eval set
- Per-model traffic and escalation rate monitored