Triton Inference Server dynamic batching: tune queue delay, batch sizes and instances with metrics, not guesses

#When this matters

Your embedding or reranker model on Triton has low GPU utilization at moderate traffic, or p99 latency jumps under load. Dynamic batching combines individual requests into batches on the server, which raises throughput a lot for models that batch well.

Tip

For generative LLMs, don't use plain dynamic batching. Use a backend with in-flight (continuous) batching such as the TensorRT-LLM backend, or a serving engine like vLLM. Dynamic batching suits fixed-shape, single-pass models: embeddings, rerankers, classifiers, vision.

#Step 1: enable dynamic batching

In the model's config.pbtxt:

name: "bge_embed"
platform: "onnxruntime_onnx"
max_batch_size: 64

dynamic_batching {
  max_queue_delay_microseconds: 2000
}

instance_group [
  { count: 1, kind: KIND_GPU }
]

• max_batch_size must be > 0 and supported by the model's first dimension.
• max_queue_delay_microseconds is how long Triton may hold a request waiting for more to form a bigger batch. It is the key latency/throughput knob.
• preferred_batch_size is optional; only set it if the model is much faster at specific sizes (common with TensorRT engines built for fixed sizes).

#Step 2: measure a baseline with perf_analyzer

perf_analyzer -m bge_embed -u localhost:8001 -i grpc \
  --concurrency-range 1:64:8 --shape input_ids:256 --shape attention_mask:256 \
  --percentile=95 -f bge_embed_baseline.csv

Run it once with dynamic batching off and once on. You want a table of concurrency vs throughput (infer/s) vs p95 latency.

#Step 3: read the server metrics

Triton exposes Prometheus metrics on port 8002 (/metrics). The useful ones per model:

Derived values (use rate() over 5 minutes in PromQL):

# Average batch size actually executed
rate(nv_inference_count{model="bge_embed"}[5m]) / rate(nv_inference_exec_count{model="bge_embed"}[5m])

# Average queue time per request (microseconds)
rate(nv_inference_queue_duration_us{model="bge_embed"}[5m]) / rate(nv_inference_request_success{model="bge_embed"}[5m])

How to read them:

#Step 4: tune in small steps

1. Start with max_queue_delay_microseconds 1000-5000 (1-5 ms). For latency-sensitive online traffic stay low; for batch/offline embedding go higher.
2. Try instance_group { count: 2, kind: KIND_GPU }. Two instances can overlap data transfer and compute. Check GPU memory first.
3. Re-run perf_analyzer after each change. Keep the change only if p95 at your target concurrency improves.

For a more systematic sweep, Triton Model Analyzer can search configurations automatically:

model-analyzer profile --model-repository /models --profile-models bge_embed \
  --output-model-repository-path /tmp/ma_out

#Step 5: protect latency under overload

Without limits, a burst fills the queue and every request gets slow. Bound the queue and reject quickly instead:

dynamic_batching {
  max_queue_delay_microseconds: 2000
  default_queue_policy {
    max_queue_size: 256
    timeout_action: REJECT
    default_timeout_microseconds: 200000
  }
}

Clients then get a fast error they can retry against another replica, which is better than a 10-second response. Pair this with autoscaling on queue time, not only on GPU utilization.

#Client side matters too

• Clients must send concurrent requests (async gRPC, multiple workers). A single synchronous client never forms batches.
• If the client already sends batches of 32, server batching adds less; check avg batch size vs client batch size.
• Keep input shapes consistent (pad/truncate to fixed lengths, or bucket by length) so requests can be batched together. For variable-length inputs, check your backend's ragged batching support.

#Prevent it next time

• Dashboard per model: request rate, avg batch size, queue time, compute time, GPU utilization.
• Store the perf_analyzer results with each config change in Git, next to config.pbtxt.
• Related: Triton model load failures, choosing an LLM serving stack.