Checklist: testing AI kill switches with regular drills before you need them

#Why drill

Common findings from a first kill switch drill:

• The flag is cached in the app for 60 minutes.
• A queue worker or scheduled job calls the model without checking the flag.
• The on-call engineer doesn't have rights to change the flag store.
• The degraded message shows a raw error to users.
• Restoring the switch doesn't restore the feature (state stuck, cache not refreshed).

#Before the drill

• Kill switch register up to date: each switch, level, command, owner, expected user impact
• Drill scope chosen: one feature, one agent tool, one tenant, or the gateway switch
• Environment: start in staging; run production drills in a low-traffic window with stakeholder approval
• Synthetic traffic or test users ready to show the effect
• Observers: someone timing, someone watching user-facing behaviour, someone watching logs
• Rollback plan if the switch can't be restored

#During the drill

For each switch:

• Note the time; flip the switch using only the runbook (no tribal knowledge)
• Measure time until new requests are stopped/degraded on all instances
• Check in-flight work: long agent runs, streaming responses, batch jobs
• Check every caller path: web/API, background workers, scheduled jobs, other apps sharing the model
• Check user experience: message text, no stack traces, links work
• Check the bypass: can anyone still reach the model directly (old key, direct endpoint, cached client)?
• Check monitoring: did dashboards and alerts show the change? Was the flip logged with who and why?
• Restore and measure time until normal service
• Confirm restore is complete: caches refreshed, autoscalers and schedules re-enabled

#Results table

#After the drill

• Each issue has an owner and due date
• Runbook updated with exact commands and measured timings
• Permissions fixed (who could not flip what)
• Next drill scheduled. Quarterly for critical switches, plus after major releases or architecture changes

#Scenarios to rotate through

1. Harmful output reported → safe mode level 2.
2. Agent loop writing to the ticket system → agent read-only, then stop runs.
3. Leaked model key → endpoint cut-off on a staging resource.
4. Restricted documents in RAG answers → source blocklist + purge in staging index.
5. Token spend spike → automated tripwire fires (use a lowered threshold in staging).
6. Provider outage → confirm switches still work when the model endpoint is down.

Tip

Do one drill where the person who built the switch is not allowed to help. That's the real test of the runbook.

#Production drill safety

• Use the smallest scope (one tenant or a canary percentage) when drilling in production.
• Announce it to support teams, so user reports during the drill aren't escalated as incidents.
• Have the restore command ready in the same terminal before flipping.