KServe InferenceService stuck not Ready: storage-initializer failures, Knative revision deadlines, probes and GPU scheduling

#Symptom

kubectl get isvc shows READY as False or Unknown, URL is empty or points at an old revision, and clients get 404, 503 or connection resets. A large LLM is often involved, because the default timeouts and disk sizes in KServe and Knative were set for small models.

Work top-down: conditions first, then the pod, then the layer the conditions point to.

#Check 1: read the conditions

NS=llm; ISVC=llama-chat

kubectl get isvc "$ISVC" -n "$NS"

kubectl get isvc "$ISVC" -n "$NS" -o jsonpath=\
'{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

The message field is the most useful single line KServe gives you. Map it to a layer:

Condition names and reasons vary by KServe version; check your build's output rather than matching strings in alerts.

#Check 2: the storage-initializer

KServe injects an init container named storage-initializer that copies the model from storageUri into /mnt/models before the server starts.

kubectl get pods -n "$NS" -l serving.kserve.io/inferenceservice="$ISVC"
POD=$(kubectl get pods -n "$NS" -l serving.kserve.io/inferenceservice="$ISVC" -o name | head -1)

kubectl logs -n "$NS" "$POD" -c storage-initializer --tail=50
kubectl describe -n "$NS" "$POD" | sed -n '/Init Containers/,/Containers:/p'

kubectl get configmap inferenceservice-config -n kserve -o jsonpath='{.data.storageInitializer}'

Tip

For models over ~20 GB, stop downloading on every pod start. A ReadWriteMany PVC populated once, or a modelcar image cached on the node, cuts cold start from many minutes to seconds and removes the init container as a failure point. Check that your KServe version has modelcar enabled before you use oci://.

#Check 3: Knative revision never becomes ready (Serverless mode)

In Serverless mode each spec change creates a Knative revision. If the revision is not ready inside the progress deadline, Knative marks it failed and the ISVC reports RevisionMissing or keeps serving the previous revision.

kubectl get ksvc -n "$NS"
kubectl get revision -n "$NS" -l serving.knative.dev/service="${ISVC}-predictor"
kubectl describe revision -n "$NS" <revision-name> | sed -n '/Conditions/,$p'

The Knative service name is <isvc>-predictor on recent versions and <isvc>-predictor-default on older ones.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llama-chat
  namespace: llm
  annotations:
    serving.knative.dev/progress-deadline: "1800s"
    autoscaling.knative.dev/min-scale: "1"
spec:
  predictor:
    minReplicas: 1
    model:
      modelFormat:
        name: huggingface
      storageUri: pvc://models-pvc/llama-3.1-8b-instruct
      resources:
        limits:
          nvidia.com/gpu: "1"
          memory: 48Gi
        requests:
          cpu: "6"
          memory: 48Gi

Warning

With minReplicas: 0, every scale-from-zero runs the full model download and load path. For LLMs this means minutes of 503s or held requests. Keep at least one replica for anything user-facing.

Many teams running LLMs move to RawDeployment mode (called Standard on newer releases). It removes Knative revisions and the queue-proxy sidecar, so a plain Deployment and HPA or KEDA handle scaling. You lose scale-to-zero and revision-based canaries.

kubectl annotate isvc "$ISVC" -n "$NS" serving.kserve.io/deploymentMode=RawDeployment --overwrite

Check whether your version accepts the mode change on an existing ISVC; some versions need the resource deleted and recreated.

#Check 4: running but never ready

The model server is up but its readiness probe fails, or it crash-loops after loading.

kubectl logs -n "$NS" "$POD" -c kserve-container --tail=100
kubectl logs -n "$NS" "$POD" -c kserve-container --previous --tail=100
kubectl get pod -n "$NS" "${POD#pod/}" -o jsonpath='{.status.containerStatuses[*].restartCount}'

Pass runtime flags as args on the predictor. With the Hugging Face runtime and the vLLM backend, engine flags pass through, but which flags are accepted depends on the runtime image version.

#Check 5: pending on GPUs

kubectl describe pod -n "$NS" "$POD" | sed -n '/Events/,$p'
kubectl get nodes -l nvidia.com/gpu.present=true \
  -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\\.com/gpu

Insufficient nvidia.com/gpu means every GPU is allocated or the cluster autoscaler can't add a node. untolerated taint means the predictor is missing the GPU node pool toleration. Add tolerations and a nodeSelector under spec.predictor.

A subtle case: during a Serverless rollout, the new revision needs its own GPU while the old one still holds one. On a fully packed pool the new revision stays Pending until the progress deadline expires, so each update fails. Keep one spare GPU of headroom, or use RawDeployment with maxSurge: 0 and accept a short capacity dip.

#Check 6: ingress

kubectl get configmap inferenceservice-config -n kserve -o jsonpath='{.data.ingress}'
kubectl get virtualservice,httproute -n "$NS" 2>/dev/null

IngressReady=False with ReconcileError usually means the ingress config expects Istio while the cluster runs something else, or the ingress gateway named there doesn't exist. Fix the ConfigMap, then restart the KServe controller.

#Prevent it next time

• Stage weights on a PVC or modelcar image, not a per-pod download.
• Set progress-deadline, startupProbe and minReplicas from the measured cold-start time of each model, plus a margin.
• Keep one spare GPU per node pool used for rollouts, or use RawDeployment for large models.
• Alert on an ISVC that stays not Ready for more than 15 minutes, using kube-state-metrics custom resource state or a simple CronJob that reads .status.conditions.
• Record image digest, storageUri and runtime version for every ISVC so you can diff a failing rollout against the last good one.