A feature-level kill switch for LLM features with Azure App Configuration feature flags

#Goal

Each AI feature (chat, summaries, drafting replies) has its own flag. Turning the flag off:

• stops new model calls for that feature within seconds,
• shows users a clear degraded message,
• needs no build, no deployment and no restart.

#1. Create the flags

az appconfig feature set   --name appcs-ops-prod --feature ai-chat        --yes
az appconfig feature set   --name appcs-ops-prod --feature ai-summaries   --yes
az appconfig feature enable --name appcs-ops-prod --feature ai-chat       --yes
az appconfig feature enable --name appcs-ops-prod --feature ai-summaries  --yes

Name flags after the feature, not the model. You want to switch off "AI summaries", whichever model powers them.

Also create a sentinel key that the app watches to know when to refresh:

az appconfig kv set --name appcs-ops-prod --key Sentinel --value 1 --yes

#2. Python: load flags with a short refresh interval

from azure.appconfiguration.provider import load, WatchKey
from featuremanagement import FeatureManager
from azure.identity import DefaultAzureCredential

config = load(
    endpoint="https://appcs-ops-prod.azconfig.io",
    credential=DefaultAzureCredential(),
    feature_flag_enabled=True,
    feature_flag_refresh_enabled=True,
    refresh_on=[WatchKey("Sentinel")],
    refresh_interval=15,          # seconds; keep it short for kill switches
)
features = FeatureManager(config)

def summarise(ticket_text: str) -> dict:
    config.refresh()              # cheap: only calls the service when the interval has passed
    if not features.is_enabled("ai-summaries"):
        return {"summary": None, "notice": "AI summaries are temporarily unavailable."}
    return {"summary": call_llm(ticket_text)}

Check names and parameters against the provider version you install; the package APIs have changed between major versions.

#3. .NET equivalent

builder.Configuration.AddAzureAppConfiguration(o =>
{
    o.Connect(new Uri("https://appcs-ops-prod.azconfig.io"), new DefaultAzureCredential())
     .UseFeatureFlags(ff => ff.SetRefreshInterval(TimeSpan.FromSeconds(15)));
});
builder.Services.AddAzureAppConfiguration();
builder.Services.AddFeatureManagement();
// app.UseAzureAppConfiguration();  // refresh middleware

if (!await featureManager.IsEnabledAsync("ai-summaries"))
    return Results.Ok(new { notice = "AI summaries are temporarily unavailable." });

#4. Pull the switch

az appconfig feature disable --name appcs-ops-prod --feature ai-summaries --yes
az appconfig kv set --name appcs-ops-prod --key Sentinel --value "$(date +%s)" --yes   # nudge refresh

Within the refresh interval, every instance stops calling the model for that feature. Restore with feature enable.

#5. Background jobs too

Kill switches often fail because a batch job or queue worker also calls the model and doesn't check the flag. Every code path that calls the LLM for this feature must check it:

• API handlers
• Queue/Functions workers (check per message, not at start-up)
• Scheduled jobs (check before each batch)

Search the codebase for every call to your LLM client and confirm a flag check precedes it.

#6. Permissions

Keep the flag store in the same region as the app, and consider a replica for resilience; if the store is unreachable, the provider keeps the last loaded values. Decide whether that is acceptable for your risk.

#7. Make flips visible

• Enable diagnostic settings on the App Configuration store, so writes are logged with the caller's identity.
• Post flips to the incident channel (an Event Grid subscription on Microsoft.AppConfiguration.KeyValueModified can trigger a webhook).

Warning

Don't cache flag values in process for longer than your agreed reaction time. A one-hour cache turns a kill switch into a one-hour delay.

#Checklist

• One flag per AI feature, named by feature
• Refresh interval ≤ 30 seconds; sentinel key
• Every LLM call path checks the flag (including workers)
• Degraded response agreed with product owner
• On-call can flip; flips are logged and announced
• Tested in a drill