Applies toGenAI apps on Azure OpenAI / AI Foundry, Amazon Bedrock, OCI Generative AI or self-hosted models; any framework (LangChain, Semantic Kernel, LlamaIndex, plain SDK)
Typical POC gaps
| POC | Production needs |
|---|---|
API key in a .env file | Managed identity / workload identity, secrets in a vault |
| Public endpoints | Private endpoints or controlled egress |
| "Looks good" on 10 questions | Evaluation set with targets, re-run on every change |
print() debugging | Traces, metrics, logs with PII rules |
| No limits | Token budgets, rate limits, kill switches |
| One region, one model | Fallback model/region, retry policy |
| The person who built it | A named owning team, on-call and runbook |
Close them in roughly this order. Each step makes the next one easier.
1. Freeze scope and success criteria
Write one page: users, tasks the app does (and refuses), data sources, success metric (e.g. "deflects 25% of L1 tickets with ≥ 85% correct answers"). This decides how much of the rest you need.
2. Identity, secrets and network
- Replace keys with managed identity (Azure), IAM roles (AWS) or resource principals (OCI).
- Private endpoints for the model service, vector store and storage; DNS tested from the app subnet.
- Users authenticate through your IdP; pass user identity to retrieval for permission trimming.
3. Gateway
Put an AI gateway (APIM, API Gateway, or an internal proxy) between the app and models: per-app limits, logging, model routing, fallback and a central switch-off point.
4. Evaluation
- Turn POC demo questions into a proper set of 100–300 cases, including out-of-scope and adversarial ones.
- Metrics: correctness, groundedness, format validity, refusal correctness.
- Gate in CI: a prompt, model or retrieval change that drops below target doesn't deploy.
5. Observability
- Tracing per request: retrieval results, prompt version, model, tokens, latency, tool calls (OpenTelemetry GenAI conventions).
- Dashboards: TTFT, error rate by type (429, 5xx, content filter), tokens and cost per day, feedback rate.
- Log retention and redaction agreed with security (prompts and outputs can contain personal data).
6. Safety
- Content filtering on input and output; prompt injection defences on retrieved and user content.
- Tool permissions least-privilege; human approval for actions that change things.
- A documented safe mode and kill switch, tested once before go-live.
7. Cost controls
- Token budgets per app and per user; alerts at 50/80/100% of monthly budget.
- Right-size the model: test whether a smaller model meets the evaluation bar for each step.
- Caching where answers repeat.
8. Resilience
| Failure | Response |
|---|---|
| 429 / quota | Retry with backoff, then secondary deployment |
| Region/model outage | Fallback deployment or model, tested |
| Vector store down | Degraded answer ("search unavailable") instead of hallucinating |
| Model retirement | Version pinning + calendar of retirement dates |
9. Ownership and operations
- Owning team, on-call rota, runbook (top 5 failure modes and fixes).
- Change process for prompts and models (they are code).
- Feedback loop: weekly review of thumbs-down and "don't know" answers.
Go-live gate
- Scope and success metric agreed
- No keys in code; private networking verified
- Traffic through the gateway with limits
- Evaluation targets met and gated in CI
- Traces, dashboards, alerts live
- Safety controls and kill switch tested
- Budgets and alerts set
- Fallback tested
- Owner, runbook and on-call in place
Tip
Don't rewrite the POC from scratch by default. Most of its prompt and retrieval logic survives; what changes is everything around it. Rewrites usually lose the tuning that made the demo work.