What a kill switch is (and isn't)
A kill switch is a deliberate, pre-built control that turns off part of an AI system, on purpose and in seconds to minutes, without a code deployment.
| Not the same as | Why |
|---|---|
| Circuit breaker | Trips automatically on errors to protect availability. A kill switch is a decision, often made because the system works but does something harmful |
| Fallback model | Keeps the feature running on another model. A kill switch may stop the feature altogether |
| Rollback | Returns to a previous version. Useful, but slower and only helps if the old version was safe |
You need a kill switch when an AI feature gives harmful or wrong answers at scale, an agent takes destructive actions, a RAG index exposes data people shouldn't see, costs run away, or a key leaks.
The six levels
| Level | What it stops | Where it lives | Typical speed |
|---|---|---|---|
| 1. Tool | One agent tool (e.g. delete_ticket, run_sql) | Tool registry / MCP server config | Seconds |
| 2. Feature | One AI feature in one app (e.g. "AI summary") | Feature flag service | Seconds to 1 min |
| 3. Agent / workflow | Running and new agent executions | Agent runtime: stop flag checked each step | Seconds |
| 4. Model / prompt version | A specific model deployment or prompt | Gateway routing, model alias, prompt registry | Minutes |
| 5. Tenant / user | One abusive user, customer or app | Gateway subscription, app blocklist | Seconds |
| 6. Platform | All AI traffic for an app, team or the whole company | AI gateway policy, identity (disable keys/roles), network | Minutes |
Design from the bottom up: the smallest switch that solves the problem keeps the most value running.
Design principles
Fail closed, but degrade gracefully
When a switch is on, users should see a clear, safe response ("AI summaries are temporarily unavailable"), not a stack trace or an empty screen. Decide the degraded behaviour for each feature in advance:
| Feature | Degraded mode |
|---|---|
| Chat assistant | Message plus link to human support |
| Summaries | Hide the panel |
| Agent that acts | Queue requests for human review |
| Search with AI answers | Plain keyword results only |
No deployment needed
A kill switch that needs a build and release is not a kill switch. Store the state outside the code: a feature flag service, a gateway named value, a config table. The app checks it on every request or caches it for a few seconds at most.
Check at the right point
- Check before each model call and before each tool execution, not only at the start of a session. Agents run for minutes.
- For streaming responses, check between chunks for long generations if stopping mid-answer matters.
Two independent paths
The application-level switch (feature flag) can fail if the app is the thing that's broken. Keep a second, independent switch outside the app: the gateway or identity layer. Neither path should depend on the AI system itself.
Who can pull it
Pre-agree who may pull each level (on-call engineer, product owner, security) without asking first. Write it down. A kill switch that needs three approvals in a crisis is too slow.
Make it visible and audited
Every flip is logged with who, when, why and ticket number, and announced in the incident channel. Dashboards show current switch states.
A minimal reference setup
User → App (feature flag check) → AI gateway (APIM / proxy: tenant + global switch, model routing)
→ Model endpoint (identity: roles/keys can be disabled)
Agent runtime → stop flag per run + tool registry (per-tool enable) → Tools
The kill switch register
Keep one table per AI system in your runbook:
| Switch | Level | How to pull | Who | User impact | Restore |
|---|---|---|---|---|---|
ff-ai-summary | Feature | az appconfig feature disable ... | On-call | Summary panel hidden | Enable flag |
apim nv ai-global-off | Platform | Set named value to on | On-call + incident lead | All AI features return 503 | Set off |
tool:refund_payment | Tool | Disable in tool registry | On-call | Agent can't refund; hands off | Re-enable after review |
Tip
Test every switch in a drill before you need it (see the kill switch drill checklist). Many teams find their switch is cached for an hour or bypassed by a background job.
Prevent the need for it
Kill switches are a last line of defence. Evaluation gates, guardrails, least-privilege tools and budgets reduce how often you need them. Build both.