Applies toHosted models (Azure OpenAI / Foundry, Amazon Bedrock, Anthropic, OpenAI, Google Gemini, OCI Generative AI) and open-weight models served on your own infrastructure
Requirements first
| Requirement | Questions |
|---|---|
| Task | Chat, RAG answers, extraction, classification, code, agents, vision? |
| Quality bar | What does "good enough" mean, measured how? |
| Latency | TTFT and total time targets? |
| Volume and cost | Requests per day, tokens per request, budget? |
| Data handling | Can data leave your tenant/region? Contractual and regulatory limits? |
| Deployment | Must it run in a specific cloud, region, on-premises, or offline? |
| Context length | Longest inputs you need? |
| Tool use / structured output | Needed and reliable? |
Hosted vs open-weight
| Hosted API | Open-weight self-hosted | |
|---|---|---|
| Quality at the top end | Usually strongest frontier models | Strong and improving; size limits on your hardware |
| Operations | Provider runs it | You run GPUs, serving, scaling, patching |
| Data control | Provider terms, regional options | Full control |
| Cost shape | Per token | GPU capacity (fixed), cheaper at high steady volume |
| Customisation | Fine-tuning options vary | Full fine-tuning, quantization, any serving stack |
See the hosted to self-hosted transformation post for the migration path.
Shortlist and evaluate
- Pick 3–5 candidates across sizes (small, medium, large) that meet the hard constraints.
- Build a test set: 100–500 real examples with expected outputs or grading rubrics.
- Run all candidates with the same prompts (adapt minimally per model).
- Measure quality (rules + judge + human sample), latency, cost per 1,000 requests.
| Model | Quality score | TTFT p95 | Cost / 1k requests | Notes |
|---|---|---|---|---|
| Candidate A (large) | ||||
| Candidate B (medium) | ||||
| Candidate C (small/open) |
Choose the smallest model that meets the quality bar (see the model routing post for mixing models).
Plan for change
- Models are retired and replaced regularly (see the model retirement Sev1 post).
- Keep prompts, evaluation sets and a gateway abstraction so switching models is a tested, routine change.
- Pin model versions in production; test new versions before switching.
Tip
Spend more time on the evaluation set than on reading model announcements. A good test set makes every future model decision fast.
Checklist
- Requirements and hard constraints written
- Hosted vs open-weight considered
- Candidates evaluated on your own test set
- Quality, latency and cost compared
- Smallest adequate model chosen
- Version pinning and switch process in place