The rollback switch: returning to the last good model or prompt version in one step

#The idea

Your answer quality depends on a release bundle:

model deployment + model version + system prompt version + retrieval settings + guardrail config

If any part is hard-coded, rolling back means a code deployment. Instead, keep a named pointer to the active bundle and a pointer to the last known-good one.

#1. Version everything

On Azure OpenAI, set model deployments so they don't auto-upgrade to new model versions on their own, so the version only changes when you change it. Check the deployment's version upgrade option.

#2. The bundle pointer

{
  "assistant-support": {
    "active":     { "deployment": "gpt-4o-2025-08", "prompt": "support_system_v8", "retrieval": "retrieval_v4", "guardrail": "strict_v2" },
    "last_good":  { "deployment": "gpt-4o-2024-11", "prompt": "support_system_v7", "retrieval": "retrieval_v3", "guardrail": "strict_v2" }
  }
}

Store it in a config service (App Configuration key, Parameter Store, a database row). The app reads it per request (with a short cache) and loads prompt/retrieval by ID.

#3. The rollback switch

# Azure App Configuration example: copy last_good into active
LAST=$(az appconfig kv show --name appcs-ai --key "bundle:assistant-support:last_good" --query value -o tsv)
az appconfig kv set --name appcs-ai --key "bundle:assistant-support:active" --value "$LAST" --yes

The same in one line for the runbook, or as a button in your internal admin tool that writes an audit record.

#4. Release process that keeps "last good" honest

1. New bundle passes offline evaluation (golden set, safety tests).
2. Canary: route a small share of traffic to the new bundle (by user hash) and compare online metrics.
3. Promote: last_good ← current active; active ← new bundle.
4. Keep the old deployment and prompt files available for at least one release cycle. Don't delete the model deployment the rollback points to.

import hashlib

def pick_bundle(user_id: str, cfg: dict) -> dict:
    canary = cfg.get("canary")
    bucket = int(hashlib.sha256(user_id.encode()).hexdigest(), 16) % 100   # stable per user
    if canary and bucket < canary["percent"]:
        return canary["bundle"]
    return cfg["active"]

Use a stable hash like this rather than Python's built-in hash(), which is randomised per process, so users would flip between versions.

#5. When rollback isn't enough

#6. Prove it works

In a drill: change active to a deliberately different prompt in staging, run the rollback command, and confirm responses use the last_good prompt within your target time (log the bundle IDs on every response; this is also invaluable in incidents).

log.info("llm_response", extra={"bundle": bundle, "deployment": bundle["deployment"], "prompt": bundle["prompt"]})

Warning

Model retirements make "last good" disappear. Track provider retirement dates for every deployment the rollback depends on.

#Checklist

• Model, prompt, retrieval and guardrail versioned and loaded by ID
• active and last_good pointers in config
• Rollback command/button in runbook, audited
• Canary and promotion update last_good
• Bundle IDs logged on every response