Sev1: a batch or embedding job used up the LLM quota and production chat is getting 429s

#Symptoms

• Production chat users see "busy" errors; logs show HTTP 429 with Retry-After.
• Request volume from users is normal.
• Someone started a re-index, evaluation or bulk summarisation job.

#1. Confirm who is using the quota

Azure OpenAI metrics, split by deployment and status code (metric names vary slightly; check what your resource shows):

az monitor metrics list --resource <aoai-resource-id> \
  --metric "AzureOpenAIRequests" --dimension ModelDeploymentName StatusCode \
  --interval PT1M --offset 1h -o table

If you log calls through a gateway, group by caller:

ApiManagementGatewayLogs
| where TimeGenerated > ago(1h)
| summarize requests=count(), throttled=countif(ResponseCode == 429) by ApimSubscriptionId, bin(TimeGenerated, 5m)
| order by requests desc

#2. Stop or slow the batch job

The fastest fix is to stop the job, or cut its concurrency:

kubectl scale deployment reembed-worker -n ai --replicas=0

If the job must continue, cap its rate (one worker, smaller batches) and add backoff that respects Retry-After.

Tip

Bulk work that doesn't need an answer in seconds belongs on a batch API (for example Azure OpenAI Global Batch or Bedrock batch inference), which uses separate quota and typically costs less.

#3. Give production its capacity back

Azure OpenAI quota is assigned per deployment from a per-region, per-model pool. Move capacity from the batch deployment to production, or raise production if pool quota is free:

az cognitiveservices account deployment list -g rg-ai -n aoai-prod \
  --query "[].{name:name, model:properties.model.name, sku:sku.name, capacity:sku.capacity}" -o table

# Reduce the batch deployment, raise production (capacity units are thousands of TPM for Standard)
az cognitiveservices account deployment create -g rg-ai -n aoai-prod \
  --deployment-name embed-batch --model-name text-embedding-3-large --model-version "1" \
  --model-format OpenAI --sku-name Standard --sku-capacity 50

az cognitiveservices account deployment create -g rg-ai -n aoai-prod \
  --deployment-name gpt-chat --model-name <model> --model-version <version> \
  --model-format OpenAI --sku-name GlobalStandard --sku-capacity 400

Running deployment create on an existing deployment name updates its capacity. Use the exact model name and version the deployment already has.

#4. If both workloads share one deployment

That is the root problem. Short-term, add a per-caller limit at the gateway. In Azure API Management:

<azure-openai-token-limit counter-key="@(context.Subscription.Id)"
    tokens-per-minute="20000" estimate-prompt-tokens="true" remaining-tokens-header-name="x-remaining-tokens" />

Give the batch job its own subscription key with a low limit and production a higher one.

#5. Verify

Watch 429s drop for the production deployment and check end-user error rates. Communicate to the batch job owner when and how they can resume.

#Prevent it next time

• Separate deployments (or separate resources) for production, batch and experiments.
• Per-caller token limits at the gateway; batch jobs on batch APIs.
• Alert on 429 rate per deployment and on TPM utilisation above 80%.
• A change process for large one-off jobs: announce, schedule off-peak, rate-limit.