Home / AI & GenAI

AI & GenAIHow-toAdvanced

Automated tripwires: signals that should pull a GenAI kill switch without waiting for a human

Some failures move faster than an on-call engineer can respond. Pick the signals worth automating (harmful-output rate, blocked-content spikes, write-tool bursts, token rate, groundedness drop), set thresholds from baselines, and wire alerts to a switch with a human confirmation step.

Deepak Bhardwaj4 min read

Applies toLLM applications with telemetry in Azure Monitor / Application Insights, CloudWatch or Prometheus; switch targets such as feature flags, APIM named values or agent stop flags

When automation is worth it

Automate the switch when... Keep it human when...
Harm grows with every minute (agent writes, data exposure, spend) Impact is reputational and needs judgement
The signal is precise (few false positives) The signal is noisy (quality scores on small samples)
The switch is small and reversible (one tool, one tenant, safe mode) The switch is big (whole platform off)

A common middle path: automatic small switch + page a human to decide on anything bigger.

Signals that make good tripwires

Signal Source Example tripwire Switch
Write-tool call burst Agent tool logs > 3× normal writes in 5 min for one agent Agent read-only mode
Same tool, same target repeated Agent tool logs > 20 identical calls in one run Stop that run
Content filter blocks on output Model provider filter results / your output classifier Output blocks > 2% of responses over 15 min Safe mode level 2
Restricted source retrieved for unauthorised user Retrieval logs vs ACL check Any occurrence Block that source
Token rate per app Gateway or provider metrics > 5× the 7-day p95 for 15 min App switch at gateway
Groundedness / eval score drop Online evaluation sample Rolling score below threshold for 1 h Page + suggest safe mode
Prompt injection detections Input shield (e.g. Prompt Shields) Spike from one tenant Block tenant

Baselines first

Run every tripwire in alert-only mode for 2–4 weeks. Record normal daily and weekly peaks, then set thresholds above them. Revisit after releases that change traffic patterns.

Example: write-tool burst → read-only mode (Azure)

Log each tool call as a custom event in Application Insights with agent, tool, risk properties. Then create a log search alert:

customEvents
| where timestamp > ago(5m) and name == "tool_call"
| extend agent = tostring(customDimensions.agent), risk = tostring(customDimensions.risk)
| where risk in ("write", "irreversible")
| summarize writes = count() by agent
| where writes > 50

az monitor scheduled-query create -g rg-ai -n "agent-write-burst" \
  --scopes <app-insights-id> \
  --condition "count 'WriteBurst' > 0" \
  --condition-query WriteBurst="customEvents | where timestamp > ago(5m) and name == 'tool_call' | extend risk = tostring(customDimensions.risk) | where risk in ('write','irreversible') | summarize writes = count() by agent=tostring(customDimensions.agent) | where writes > 50" \
  --evaluation-frequency 5m --window-size 5m --severity 1 \
  --action-groups <action-group-id>

CLI syntax for scheduled queries changes between versions; creating the rule in the portal and exporting it as Bicep is often easier.

The action group calls a Function that sets the agent's mode:

import os
from azure.appconfiguration import AzureAppConfigurationClient, ConfigurationSetting
from azure.identity import DefaultAzureCredential

def main(alert: dict):
    agent = alert["data"]["alertContext"].get("dimensions", [{}])[0].get("value", "unknown")
    client = AzureAppConfigurationClient(os.environ["APPCONFIG_ENDPOINT"], DefaultAzureCredential())
    client.set_configuration_setting(ConfigurationSetting(key=f"agent:{agent}:mode", value="read_only"))
    notify_incident_channel(f"Tripwire: {agent} switched to read_only after write burst. Confirm or revert.")

The alert payload shape depends on the alert type (common alert schema). Log it once and adapt the parsing.

Guard the guard

  • Rate-limit the automation: if a tripwire fires 10 times in an hour, stop and page instead of flapping.
  • Never automatically switch back on. Restoring is always a human decision after review.
  • Fail safe: if the automation itself errors, page on-call.
  • Log every automatic flip with the triggering alert ID.

Measure

  • Time from first bad event to switch (target: minutes).
  • False-positive trips per month (target: low, and each one reviewed).
  • Incidents where a tripwire should have fired but didn't. Add those signals.

Warning

Content-based tripwires on small samples are noisy. Use rates over a window with a minimum count (for example, at least 200 responses), not single events, unless the signal is a hard rule like "restricted document retrieved".

Checklist

  • Each tripwire: signal, baseline, threshold, switch, owner
  • Alert-only period completed
  • Automation flips small, reversible switches only
  • Human confirmation and manual restore
  • Flapping protection and failure paging

Related fixes