Kill switches for self-hosted models on Kubernetes: ingress, network policy, scale to zero

#Levers, from least to most disruptive

Keep the model warm (levers 1–2) when you might restore quickly. Scale to zero when the model itself is the problem or to stop cost.

#1. Block at the ingress

NGINX Ingress: return 503 for the model's host or path with a server snippet or by pointing the ingress at a "maintenance" service. Cleanest is a second backend:

kubectl -n llm patch ingress llm-api --type=json \
  -p='[{"op":"replace","path":"/spec/rules/0/http/paths/0/backend/service/name","value":"maintenance-503"}]'

maintenance-503 is a tiny deployment that returns a JSON 503 with a message. If you use an API gateway (Kong, APIM, Envoy Gateway), use its route disable instead.

#2. NetworkPolicy deny

Blocks all inbound traffic to the model pods except from the namespaces you list (for example, the evaluation namespace you'll use to investigate):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: llm-kill-switch
  namespace: llm
spec:
  podSelector:
    matchLabels: { app: vllm-llama }
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: llm-eval }

kubectl apply -f llm-kill-switch.yaml     # pull
kubectl delete -f llm-kill-switch.yaml    # restore

Requires a CNI that enforces NetworkPolicy (Azure CNI with Azure/Calico/Cilium policy, Calico, Cilium).

#3. Scale to zero

kubectl -n llm scale deployment vllm-llama --replicas=0

Also stop anything that scales it back up:

• HPA: an HPA scales the deployment back to its minReplicas. Delete it for the duration (kubectl -n llm delete hpa vllm-llama) and recreate it from Git afterwards.
• KEDA: pause the ScaledObject with the annotation autoscaling.keda.sh/paused-replicas: "0".
• GitOps (Argo CD, Flux): suspend sync for the app, or Git will scale it back within minutes:

argocd app set llm-serving --sync-policy none
flux suspend kustomization llm-serving

KServe: set minReplicas: 0 and remove traffic, or delete the InferenceService.

#4. Remove the model

When the model weights themselves must not be served (licence issue, withdrawn model, poisoned weights):

• Delete the deployment/InferenceService.
• Remove the weights from the model cache PVC and the registry/bucket path the pods download from, or move them to a quarantine location with no read access.
• Block the model ID in your deployment pipeline.

#Clients need to cope

Callers should get a clear error, not hang:

• Gateway or maintenance-503 returns JSON with a code like model_disabled.
• Client timeouts short enough not to pile up requests.
• Optional fallback to another model, if that's allowed for the incident type.

#Verify

kubectl -n llm get deploy,pods,netpol,ingress
curl -s -o /dev/null -w "%{http_code}\n" https://llm.internal.example/v1/models   # expect 503 or timeout

vLLM exposes Prometheus metrics. Request rate (vllm:request_success_total and running requests) should drop to zero after the switch. Metric names vary by version.

#Restore safely

1. Fix or replace the model / config.
2. Scale to 1 replica, test from the eval namespace (NetworkPolicy still on).
3. Remove the NetworkPolicy, then restore ingress.
4. Re-enable autoscaling and GitOps sync.

Tip

Put levers 1 and 2 in your GitOps repo as ready-to-apply manifests (disabled by default). In an incident you merge one PR or run one command, rather than writing YAML under pressure.

#Checklist

• Maintenance backend deployed and tested
• NetworkPolicy manifest ready; CNI enforces it
• Autoscaler and GitOps pause steps documented
• Model quarantine procedure for weights
• Drill timed for each lever