Home / AI & GenAI

AI & GenAITroubleshootingAdvanced

Triton Inference Server model won't load - fixing repository layout, config.pbtxt, backends and GPU memory

When Triton reports a model as UNAVAILABLE, the cause is usually the repository layout, a config.pbtxt mismatch, a missing backend or framework version, or not enough GPU memory. Read the server log and the repository index, then fix the layer that failed.

Deepak Bhardwaj5 min read

Applies toNVIDIA Triton Inference Server 2.x (NGC containers); ONNX Runtime, TensorRT, PyTorch, Python and vLLM backends; Docker and Kubernetes

Symptoms

  • curl localhost:8000/v2/health/ready returns a non-200 status. Triton isn't ready while any model failed to load (with the default settings).
  • The startup log ends with a table showing UNAVAILABLE: ... next to a model.
  • Clients get Request for unknown model or model is not ready.

1. Read the status

# server and model readiness
curl -s -o /dev/null -w "%{http_code}\n" localhost:8000/v2/health/ready
curl -s localhost:8000/v2/models/<model>/ready -o /dev/null -w "%{http_code}\n"

# every model in the repository with state and reason
curl -s -X POST localhost:8000/v2/repository/index | jq

The reason field in the index usually says exactly what failed. The full error is in the server log:

docker logs triton 2>&1 | grep -E "UNAVAILABLE|error|failed" | head -30
# Kubernetes
kubectl logs deploy/triton -n inference | grep -E "UNAVAILABLE|error|failed"

Start Triton with more detail when needed: --log-verbose=1.

2. Repository layout

Triton expects this structure:

model_repository/
└── resnet50/
    ├── config.pbtxt
    └── 1/
        └── model.onnx
Mistake Error you see
No numeric version folder failed to load ... no version is available
Wrong file name (resnet.onnx instead of model.onnx) File not found for the backend
Folder name differs from name: in config.pbtxt Name mismatch error
Extra files or folders at the top level Treated as models; may fail to load

Default file names by backend: model.onnx, model.plan (TensorRT), model.pt (PyTorch), model.savedmodel/ (TensorFlow) and model.py (Python). You can override them with default_model_filename.

3. config.pbtxt

name: "resnet50"
backend: "onnxruntime"
max_batch_size: 16
input [
  { name: "input", data_type: TYPE_FP32, dims: [ 3, 224, 224 ] }
]
output [
  { name: "output", data_type: TYPE_FP32, dims: [ 1000 ] }
]
instance_group [ { count: 1, kind: KIND_GPU, gpus: [ 0 ] } ]
dynamic_batching { max_queue_delay_microseconds: 2000 }

Common mismatches:

Issue Fix
Input/output names differ from the model Inspect the model (for example Netron, or polygraphy inspect model model.onnx) and use its names
dims include the batch dimension while max_batch_size > 0 Leave out the batch dimension; Triton adds it
Model has no batch dimension but max_batch_size > 0 Set max_batch_size: 0
Wrong data type (FP16 vs FP32) Match the model's tensor types
Variable dimensions Use -1 for those dims

For ONNX, TensorRT and some other backends, Triton can auto-complete a missing or partial config from the model. Start with a minimal config, then read what Triton generated:

curl -s localhost:8000/v2/models/resnet50/config | jq

4. Backend and version problems

Error Cause Fix
unable to find backend / backend library not found The backend isn't in this container image Use the NGC image that includes it (for example the -py3 full image, or a vLLM/TensorRT-LLM image)
TensorRT engine fails to deserialise Engine built with a different TensorRT version or GPU architecture Rebuild the .plan file with the same TensorRT version and GPU type as the server
PyTorch model load error TorchScript saved with a newer PyTorch Export with a compatible version; use TorchScript (torch.jit.save), not a pickled model
Python backend import errors Missing packages in the stub environment Package an execution environment (conda-pack) and set EXECUTION_ENV_PATH

Tip

TensorRT engines are tied to the GPU model and the TensorRT version. Build them in CI using the same Triton container version you deploy, on the same GPU type.

5. GPU memory

nvidia-smi --query-gpu=index,name,memory.used,memory.total --format=csv
  • CUDA out of memory during load: too many instances (instance_group count), too many models on one GPU, or a large LLM. Lower the instance count, spread models across GPUs, or use a bigger or quantised model.
  • Several models on the same GPU each reserve memory. Use --model-control-mode=explicit and load only the models you need:
tritonserver --model-repository=/models --model-control-mode=explicit --load-model=resnet50
curl -X POST localhost:8000/v2/repository/models/bert/load

6. Kubernetes specifics

  • Readiness probe on /v2/health/ready, liveness on /v2/health/live. Give the readiness probe a long enough start-up window (startupProbe) for large models.
  • Model repository from a PVC or object storage (s3://, gs://, as:// paths with credentials). Check that the pod's identity can read it.
  • --exit-on-error=false keeps the server up when one model fails, which is useful in shared deployments, as long as you alert on per-model readiness.

Checklist

  • Repository index shows the failed model and reason
  • Layout: model folder, numeric version, default file name
  • config.pbtxt names, dims, types and batch settings match the model
  • Backend present in the image; engine/framework versions match
  • GPU memory fits all instances; explicit model control if crowded
  • Probes and alerts on per-model readiness

Related fixes