Highly available self-hosted LLM inference on Kubernetes

#1. Basics

• ≥ 2 replicas of each model, on different nodes and zones.
• A GPU node pool per GPU type, with taints so only GPU workloads land there.
• An internal load balancer or gateway in front, with health-based routing.

#2. Deployment (excerpt)

apiVersion: apps/v1
kind: Deployment
metadata: { name: llm-chat, namespace: inference }
spec:
  replicas: 3
  strategy: { type: RollingUpdate, rollingUpdate: { maxUnavailable: 0, maxSurge: 1 } }
  selector: { matchLabels: { app: llm-chat } }
  template:
    metadata: { labels: { app: llm-chat } }
    spec:
      tolerations: [ { key: nvidia.com/gpu, operator: Exists, effect: NoSchedule } ]
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector: { matchLabels: { app: llm-chat } }
      terminationGracePeriodSeconds: 120
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.x.y        # pin a tested tag
          args: ["--model", "/models/qwen2.5-14b-instruct-awq", "--served-model-name", "chat-default",
                 "--max-model-len", "16384", "--gpu-memory-utilization", "0.90", "--enable-prefix-caching"]
          ports: [ { containerPort: 8000 } ]
          resources: { limits: { nvidia.com/gpu: 1 } }
          startupProbe:
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 10
            failureThreshold: 90                # up to 15 min to load the model
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 5
          livenessProbe:
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 15
            failureThreshold: 4
          volumeMounts: [ { name: models, mountPath: /models, readOnly: true }, { name: shm, mountPath: /dev/shm } ]
      volumes:
        - name: models
          persistentVolumeClaim: { claimName: models-ro }
        - name: shm
          emptyDir: { medium: Memory, sizeLimit: 8Gi }

• Startup probe with a long allowance: loading model weights can take minutes. Without it, the liveness probe kills the pod while it's still loading.
• maxUnavailable: 0 and maxSurge: 1 keep full capacity during rollouts (you need one spare GPU node).
• terminationGracePeriodSeconds lets in-flight generations finish.

#3. PodDisruptionBudget

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: llm-chat, namespace: inference }
spec:
  minAvailable: 2
  selector: { matchLabels: { app: llm-chat } }

This keeps node upgrades and drains from taking down too many replicas at once.

#4. Autoscaling on the right signal

GPU utilisation is a poor signal for LLMs. Scale on queue depth or concurrent requests instead, with KEDA and Prometheus:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: llm-chat, namespace: inference }
spec:
  scaleTargetRef: { name: llm-chat }
  minReplicaCount: 3
  maxReplicaCount: 8
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: sum(vllm:num_requests_waiting{app="llm-chat"})
        threshold: "5"

Pair this with the cluster autoscaler or node auto-provisioning for GPU nodes, and remember that GPU nodes take minutes to start. Keep minReplicaCount high enough for normal peaks.

#5. Fast model loading

• Weights on a shared read-only volume (Azure Files/NFS, FSx for Lustre, OCI File Storage) or baked into a node image or cache, not downloaded from Hugging Face at every start.
• Pre-pull large images on GPU nodes (a DaemonSet or a node image with the image cached).

#6. Routing

• A Kubernetes Service plus an internal ingress or gateway. For better efficiency, use a gateway that routes by queue length or prefix-cache affinity (Gateway API Inference Extension, or vLLM production-stack routers).
• Put the AI gateway in front for authentication and quotas.

#7. Failure tests

Tip

Size for N+1: with 3 replicas carrying peak load, losing one puts 50% more load on each of the other two. Either run 4, or accept slower responses during a failure, and agree that with the service owner up front.