Autoscaling self-hosted LLMs on Kubernetes: KEDA on queue depth and KV cache, plus GPU node autoscaling

A vLLM pod at full load can show 15% CPU. The GPU is the bottleneck, and GPU utilisation is a poor scaling signal too (it is "busy" with one request or with fifty). Scale on what users feel: requests waiting and cache pressure.

#Signals that work

Metric names change between vLLM versions; check /metrics on your build:

kubectl -n llm port-forward deploy/llama-8b 8000:8000 &
curl -s localhost:8000/metrics | grep -E "^vllm:(num_requests|gpu_cache|kv_cache)"

#Scrape the metrics

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata: { name: vllm, namespace: llm }
spec:
  selector: { matchLabels: { app: llama-8b } }
  podMetricsEndpoints: [{ port: http, path: /metrics, interval: 15s }]

#KEDA ScaledObject

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: llama-8b, namespace: llm }
spec:
  scaleTargetRef: { name: llama-8b }
  minReplicaCount: 2
  maxReplicaCount: 12
  pollingInterval: 15
  cooldownPeriod: 600
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:   { stabilizationWindowSeconds: 0,   policies: [{ type: Pods, value: 2, periodSeconds: 60 }] }
        scaleDown: { stabilizationWindowSeconds: 900, policies: [{ type: Pods, value: 1, periodSeconds: 300 }] }
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-operated.monitoring:9090
      query: sum(vllm:num_requests_waiting{namespace="llm", app="llama-8b"})
      threshold: "4"
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-operated.monitoring:9090
      query: avg(vllm:gpu_cache_usage_perc{namespace="llm", app="llama-8b"}) * 100
      threshold: "80"

• With threshold: "4" on waiting requests, KEDA targets about 4 waiting requests per replica.
• Scale up quickly, scale down slowly: model pods are expensive to start, and traffic is bursty.
• The cache trigger catches long-context load that does not queue yet but is about to preempt.

Warning

Scaling to zero sounds attractive for GPU cost, but the first request then waits for node provisioning, image pull and weight loading, often several minutes. Use minReplicaCount of at least 1 (2 for HA) for user-facing models, and scale-to-zero only for batch or internal endpoints with relaxed latency.

#GPU nodes must follow

Pending pods trigger node scale-up. Make it fast and predictable:

• Dedicated GPU node pool with taints (nvidia.com/gpu=present:NoSchedule) and the model pods tolerating them.
• Cluster autoscaler or Karpenter with GPU instance types allowed and quota available in the region.
• Pod resource request nvidia.com/gpu: 1 (or more for tensor parallel).

Karpenter NodePool sketch (EKS):

apiVersion: karpenter.sh/v1
kind: NodePool
metadata: { name: gpu-inference }
spec:
  template:
    spec:
      requirements:
      - { key: karpenter.k8s.aws/instance-family, operator: In, values: ["g6e", "g6", "p5"] }
      - { key: karpenter.sh/capacity-type, operator: In, values: ["on-demand"] }
      taints: [{ key: nvidia.com/gpu, value: "present", effect: NoSchedule }]
      nodeClassRef: { group: karpenter.k8s.aws, kind: EC2NodeClass, name: gpu }
  limits: { "nvidia.com/gpu": 32 }
  disruption: { consolidationPolicy: WhenEmpty, consolidateAfter: 30m }

WhenEmpty avoids Karpenter evicting busy model pods to consolidate.

#Shrink cold start

Overprovisioning with a placeholder:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: { name: overprovisioning }
value: -10
globalDefault: false
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: gpu-placeholder, namespace: llm }
spec:
  replicas: 1
  selector: { matchLabels: { app: gpu-placeholder } }
  template:
    metadata: { labels: { app: gpu-placeholder } }
    spec:
      priorityClassName: overprovisioning
      tolerations: [{ key: nvidia.com/gpu, operator: Exists, effect: NoSchedule }]
      containers:
      - name: pause
        image: registry.k8s.io/pause:3.9
        resources: { limits: { nvidia.com/gpu: 1 } }

The placeholder holds one GPU node warm; a real model pod preempts it instantly, and the autoscaler replaces the spare.

#Readiness and traffic

• Readiness probe must pass only after weights are loaded (/health on vLLM returns OK when ready). Otherwise the load balancer sends traffic to a pod still loading.
• Set a generous startupProbe so slow loads are not killed.
• Graceful shutdown: terminationGracePeriodSeconds long enough to finish in-flight generations.

startupProbe:   { httpGet: { path: /health, port: 8000 }, periodSeconds: 10, failureThreshold: 60 }
readinessProbe: { httpGet: { path: /health, port: 8000 }, periodSeconds: 5 }

#Test it

Replay a traffic ramp (2x and 5x normal) and record: time from queue growth to new replica Ready, TTFT p95 during the ramp, and whether scale-down causes errors. Tune thresholds and windows from the data.

#Prevent it next time

• Alert when maxReplicaCount is reached or GPU quota blocks node provisioning.
• Keep warm capacity sized to the largest burst you can tolerate during cold start.
• Re-check metric names after model server upgrades; a renamed metric silently breaks scaling.