A cost kill switch for GenAI: stopping runaway token spend automatically on Azure and AWS

#Why budgets alone are too slow

Cloud billing data arrives with a delay of hours. A runaway agent can spend thousands before the first budget alert fires. Use layers:

#Layer 1: limits in the app

• Always set max_tokens (or max_completion_tokens) on every call.
• Agents: maximum steps per run, maximum tool calls, maximum total tokens per run; stop and report when reached.

#Layer 2: gateway token limits (Azure APIM)

<inbound>
  <base />
  <llm-token-limit counter-key="@(context.Subscription.Id)"
                   tokens-per-minute="20000"
                   estimate-prompt-tokens="true"
                   remaining-tokens-header-name="x-remaining-tokens" />
</inbound>

Older APIM versions use azure-openai-token-limit. Check which policy your tier and version support. Optional quota attributes allow daily or monthly token caps per subscription.

#Layer 3: metric alert → automated switch

Azure OpenAI emits token metrics (for example ProcessedPromptTokens and GeneratedTokens; check the metric names on your resource). Create an alert on an unusual rate:

az monitor metrics alert create -g rg-ai -n "aoai-token-spike" \
  --scopes <openai-resource-id> \
  --condition "total GeneratedTokens > 2000000" \
  --window-size 15m --evaluation-frequency 5m \
  --action <action-group-id>

The action group calls an Automation runbook or Function that pulls a switch, for example sets the APIM named value ai-kill-<app> to on, or suspends the subscription of the heaviest caller. Notify on-call at the same time.

Warning

Automatic shutoff on a metric can stop a legitimate busy day. Start with alert-only, learn normal peaks for a few weeks, then automate the switch with a threshold well above them.

#Layer 4: budgets with actions

#Azure

Create a budget on the resource group or subscription that hosts the AI resources, with an action group at, say, 90% and 100% of the monthly amount. The action group runs automation that:

1. Turns off non-critical AI features (feature flags).
2. Reduces deployment capacity, or suspends low-priority APIM subscriptions.

#AWS

AWS Budgets actions can apply an IAM policy or SCP automatically when a threshold is crossed. A deny policy for Bedrock invocation:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Deny",
    "Action": ["bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream", "bedrock:Converse", "bedrock:ConverseStream"],
    "Resource": "*"
  }]
}

Attach it via a budget action to the application roles (or as an SCP on the account), with approval set to automatic or manual as you prefer. For faster reaction, use CloudWatch metrics for Bedrock (InputTokenCount, OutputTokenCount per model) with an alarm that triggers a Lambda to attach the same policy.

#Find who is spending

• Azure: APIM logs with subscription and token counts, or the llm-emit-token-metric / azure-openai-emit-token-metric policy with dimensions for app and user.
• AWS: Bedrock model invocation logging to CloudWatch Logs or S3, grouped by role/caller.

#Restore

After the cause is fixed (loop, leaked key, bad prompt):

1. Raise limits back in steps, not all at once.
2. Remove deny policies / unsuspend subscriptions.
3. Record the incident cost and the reaction time of each layer.

#Checklist

• max_tokens and agent step caps everywhere
• Gateway token limits per app
• Token-rate alerts with a tested automated switch
• Budgets with actions on AI resource scopes
• Per-app token attribution in logs