Applies toOrganisations using Azure OpenAI / Foundry, Amazon Bedrock, OCI Generative AI or self-hosted models across several teams; AI gateways such as Azure API Management or open-source LLM proxies
Platform building blocks
| Block | Purpose |
|---|---|
| Gateway | Single entry point: auth, routing, quotas, logging (see the AI gateway foundation post) |
| Model catalog | Approved models per data classification and use, with versions |
| Capacity | Shared deployments (PTU and pay-as-you-go), regions, fallback |
| Identity | Per-application identity (managed identity/IAM role), no shared keys |
| Quotas | Tokens per minute/day per team or app |
| Guardrails | Content filters, PII redaction, prompt injection checks, output validation |
| Observability | Usage, latency, errors, cost per team |
| Onboarding | Request form, risk tier, approval, provisioned access |
Shared vs dedicated deployments
| Shared deployments behind gateway | Dedicated per team | |
|---|---|---|
| Utilisation | High (pooled) | Often low |
| Isolation | Logical (quotas) | Strong |
| Fits | Most teams | Regulated workloads, large steady users with PTU needs |
Quotas and fairness
<!-- APIM: token limit per subscription (team/app) -->
<llm-token-limit counter-key="@(context.Subscription.Id)"
tokens-per-minute="20000"
estimate-prompt-tokens="true"
remaining-tokens-header-name="x-remaining-tokens" />
Use the policy names supported by your APIM version (azure-openai-token-limit / llm-token-limit). Set higher quotas for production apps than for experimentation.
Showback
- Emit token usage per team/app from the gateway (
llm-emit-token-metricor logs). - Monthly report per team: tokens, cost estimate, top models (see the data platform cost allocation post for the method).
- Showback first; chargeback once numbers are trusted.
Governance link
- Each app registered with a use-case risk tier (see the responsible AI governance post).
- Higher tiers get stricter guardrails, logging and review.
- Kill switches at gateway, app and tool levels (see the kill switch series).
Operating the platform
| Task | Owner |
|---|---|
| Capacity planning, quotas, model lifecycle | Platform team |
| Prompts, evaluation, app behaviour | App teams |
| Policies, risk tiers | Governance group |
| Incidents | Shared runbooks with clear handoffs |
Tip
Make the platform the easiest path: self-service onboarding in a day, good docs and sample code. Teams bypass platforms that are slower than getting their own keys.
Checklist
- Gateway, catalog and identity model in place
- Shared vs dedicated capacity decided
- Per-team quotas enforced
- Usage showback running
- Guardrails by risk tier
- Self-service onboarding path