LLM pods take 20 minutes to become ready: slow model weight downloads and loading on Kubernetes, and how to cut cold starts

#Symptoms

• A new GPU pod takes 10–30 minutes from scheduling to Ready; autoscaling reacts far too late for traffic spikes.
• Pods are killed and restarted during startup because liveness probes fail while weights load.
• Every restart downloads the model again from Hugging Face or object storage.
• Node scale-up adds another 5–10 minutes before the pod is even scheduled.

#Step 1: measure where the time goes

Break startup into phases using pod events and server logs:

kubectl get events -n llm --field-selector involvedObject.name=<pod> --sort-by=.lastTimestamp
kubectl logs -n llm <pod> | grep -iE "download|loading|weights|took|graph|ready"

Fix the biggest phase first.

#Step 2: stop downloading on every start

Never pull from the internet at pod start in production.

Pre-download once into a cache volume:

export HF_HUB_ENABLE_HF_TRANSFER=1     # faster parallel downloads (needs the hf_transfer package)
huggingface-cli download meta-llama/Llama-3.1-70B-Instruct \
  --local-dir /models/llama-3.1-70b-instruct --revision <pinned-commit>

Then point the server at the local path and set HF_HUB_OFFLINE=1 so it never tries the network.

#Step 3: load faster

• Safetensors format: memory-mappable and much faster to load than pickle .bin files. Convert if your weights aren't safetensors.
• Streaming loaders: vLLM supports alternative load formats such as the Run:ai Model Streamer (--load-format runai_streamer), which reads tensors concurrently from local disk or S3-compatible storage; and tensorizer-based loading. Check your vLLM version's docs for supported formats and options.

vllm serve /models/llama-3.1-70b-instruct --tensor-parallel-size 4 \
  --load-format runai_streamer

• Storage throughput: loading 140 GB at 500 MB/s takes ~5 minutes; at 3 GB/s about 50 seconds. Choose storage by throughput, not just capacity (Premium/Ultra disks, high-throughput file shares, local NVMe).
• Fewer, larger reads: many small shards on a slow network file system load slower than a few large ones.

#Step 4: smaller images, pre-pulled

• Use slim serving images; don't install training libraries.
• Pre-pull images on GPU node pools (a DaemonSet that pulls the image, or node image caching features of your cloud), so new pods skip the pull.
• Keep the image tag stable between releases where possible; every new tag is a full pull on every node.

#Step 5: probes that match reality

Use a startupProbe so Kubernetes doesn't kill the pod while it loads:

startupProbe:
  httpGet: { path: /health, port: 8000 }
  periodSeconds: 10
  failureThreshold: 120        # up to 20 minutes to start
readinessProbe:
  httpGet: { path: /health, port: 8000 }
  periodSeconds: 5
livenessProbe:
  httpGet: { path: /health, port: 8000 }
  periodSeconds: 10
  failureThreshold: 6

Liveness only starts after the startup probe succeeds.

#Step 6: warm-up time

• CUDA graph capture and kernel compilation take time on first start. Some engines can cache compilation artefacts on disk; put that cache on the same persistent volume as the weights.
• --enforce-eager in vLLM skips CUDA graph capture (faster start, slower decoding); only for development or very spiky workloads where start time matters more than throughput.

#Step 7: capacity strategy

Even a fast start is minutes, not seconds. Plan for it:

• Keep a minimum number of warm replicas sized for normal peaks.
• Scale on leading indicators (queue depth, waiting requests) rather than GPU utilisation alone.
• Keep spare GPU nodes (overprovisioning with low-priority placeholder pods) so pods don't also wait for node provisioning.

Tip

Track "time to ready" for every new pod as a metric. It's the number that tells you how late your autoscaling will always be.

#Decision table

#Prevent it next time

• Pin model revisions and stage them into your own storage as part of the release.
• Benchmark startup end to end for every new model size before production.
• Use startup probes on every model server.
• Size minimum replicas from realistic cold-start times.