Applies toAzure API Management (GenAI gateway capabilities) with Azure OpenAI / Azure AI Foundry; the same pattern applies to Kong, LiteLLM, Portkey and Envoy AI Gateway
Why a gateway first
Without one, every team gets its own API keys, quotas are fought over, nobody knows who spends what, and switching models means changing every app. With a gateway:
| Concern | Gateway feature |
|---|---|
| Authentication | Apps call the gateway with Entra ID tokens or subscription keys; the gateway calls models with managed identity (no model keys in apps) |
| Fair use | Token limits per team or app (tokens per minute, monthly quotas) |
| Cost visibility | Token metrics emitted with dimensions (team, app, model) |
| Resilience | Backend pools with priority and weight, plus circuit breakers (see the HA post) |
| Safety | Content safety checks, logging prompts and responses where policy allows |
| Speed and cost | Semantic caching of similar prompts |
1. Import the model API
In APIM: APIs → Add API → Azure OpenAI (or Azure AI Foundry / OpenAI-compatible). Select the resource, and APIM creates the API and a backend. Enable the managed identity of APIM and give it the Cognitive Services OpenAI User role on the model resource.
2. Core inbound policy
<policies>
<inbound>
<base />
<validate-azure-ad-token tenant-id="{{tenant-id}}">
<client-application-ids><application-id>{{allowed-app-id}}</application-id></client-application-ids>
</validate-azure-ad-token>
<authentication-managed-identity resource="https://cognitiveservices.azure.com" />
<azure-openai-token-limit counter-key="@(context.Subscription.Id)"
tokens-per-minute="20000" estimate-prompt-tokens="true"
remaining-tokens-header-name="x-remaining-tokens" />
<azure-openai-emit-token-metric namespace="genai">
<dimension name="Subscription" value="@(context.Subscription.Id)" />
<dimension name="API" value="@(context.Api.Name)" />
<dimension name="Deployment" value="@(context.Request.MatchedParameters["deployment-id"])" />
</azure-openai-emit-token-metric>
<set-backend-service backend-id="aoai-pool" />
</inbound>
<backend><base /></backend>
<outbound><base /></outbound>
</policies>
For non-Azure OpenAI backends, use llm-token-limit and llm-emit-token-metric instead.
3. Products and subscriptions per team
- One product per tier (for example
genai-standardandgenai-premium), with different token limits. - One subscription per app, which gives you per-app metrics and the ability to revoke one app.
4. Logging
- Application Insights for metrics, latency and errors.
- If policy allows, log prompts and completions to a protected Log Analytics workspace, with retention set and access restricted. Mask personal data first.
5. Network
- APIM in a VNet (internal mode or private endpoint). Model resources accessed through private endpoints with public access disabled.
- Apps reach only APIM; nothing reaches the model endpoints directly.
6. Developer experience
- Publish the API in the developer portal with examples.
- Apps use the standard OpenAI SDK with the base URL pointed at the gateway, so no special client is needed.
Tip
Make the gateway mandatory from day one, with an Azure Policy that denies public network access on AI resources. Bolting a gateway on after ten teams have shipped is a migration project.