Applies tovLLM OpenAI-compatible server (V1 engine, roughly 0.10 and later); flag defaults and metric names vary by version
How to use this sheet
This sheet is for engineers who already run vLLM and need the exact flag, endpoint or metric quickly during tuning or an incident. It groups the settings by what they control. Defaults move between releases, so check your build before you copy a value:
vllm --version
vllm serve --help=all | less # full flag list for your installed version
Warning
Some flags are renamed, made the default or removed between minor releases (for example, chunked prefill and prefix caching are on by default in recent V1 builds). Pin the vLLM image tag in production and diff
--helpoutput before you upgrade.
Memory and context flags
| Flag | What it controls | Notes |
|---|---|---|
--gpu-memory-utilization 0.90 | Fraction of each GPU's VRAM vLLM may take (weights + activations + KV cache) | Lower it if other processes share the GPU. Raising it gives more KV cache, not faster kernels |
--max-model-len 16384 | Longest prompt + output accepted per request | Startup fails if KV cache can't fit one sequence of this length |
--kv-cache-dtype fp8 | KV cache precision | Roughly halves KV memory compared with FP16. Check quality on your eval set |
--quantization <method> | Weight quantization | Usually detected from the model config. Set it only if you have to |
--swap-space 4 | CPU swap space per GPU in GiB | Name and behaviour vary by version |
--cpu-offload-gb 0 | Offloads part of the weights to CPU RAM | Costs latency. Use it only as a last resort |
Check the KV cache capacity vLLM computed at startup. It tells you more than any calculation you do yourself:
kubectl logs deploy/vllm | grep -iE "KV cache|maximum concurrency|num_gpu_blocks"
Batching and scheduling flags
| Flag | What it controls | Trade-off |
|---|---|---|
--max-num-seqs 256 | Max sequences scheduled together | Higher means more throughput and more KV pressure and preemption |
--max-num-batched-tokens 8192 | Token budget per scheduler step | Higher helps prefill throughput. Lower protects decode latency (inter-token latency) |
--enable-chunked-prefill | Splits long prompts across steps | Stops one long prompt from stalling decodes |
--enable-prefix-caching | Reuses KV blocks for shared prefixes | Big win for long, stable system prompts and RAG templates |
--scheduling-policy priority | Default is fcfs | With priority, a request's priority field is used. Lower values are served first |
--max-num-partial-prefills | How many long prefills run at once | Only in some versions. Check --help |
Send a priority with the OpenAI SDK through extra_body:
client.chat.completions.create(
model="llama-70b",
messages=[{"role": "user", "content": "Summarise incident INC-4411"}],
extra_body={"priority": -10}, # lower = earlier; default 0
)
Parallelism flags
| Flag | Use when |
|---|---|
--tensor-parallel-size 4 | The model doesn't fit on one GPU. Keep TP inside one NVLink domain |
--pipeline-parallel-size 2 | Splitting across nodes, or across GPUs without fast interconnect |
--data-parallel-size 2 | Several replicas in one server process (recent versions). Often simpler to run separate pods |
--distributed-executor-backend mp|ray | Single node: mp. Multi-node: ray |
API, security and model naming flags
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--revision <commit-sha> \
--served-model-name llama-8b \
--host 0.0.0.0 --port 8000 \
--api-key "$VLLM_API_KEY" \
--generation-config vllm \
--max-model-len 16384 \
--gpu-memory-utilization 0.90
| Flag | Why it matters |
|---|---|
--served-model-name | Stable name for clients. Change the weights without changing app config |
--revision | Pins the Hugging Face commit so a re-download can't swap the weights |
--api-key | Bearer token on /v1/*. It's one shared key, so put a gateway in front for per-team auth |
--generation-config vllm | Ignore the model repo's generation_config.json defaults (temperature, top_p). The default auto applies them |
--chat-template <file> | Override a missing or broken chat template |
--enable-auto-tool-choice --tool-call-parser <name> | Tool calling. The parser must match the model family |
--enable-lora --lora-modules name=path --max-lora-rank 64 | Multi-LoRA serving |
--trust-remote-code | Runs Python code from the model repo. Avoid it unless you have reviewed that code |
Endpoints to probe
H="Authorization: Bearer $VLLM_API_KEY"
curl -s localhost:8000/health -o /dev/null -w '%{http_code}\n' # 200 when the engine is alive
curl -s localhost:8000/version
curl -s -H "$H" localhost:8000/v1/models | jq '.data[] | {id, max_model_len}'
curl -s -H "$H" localhost:8000/tokenize \
-H 'Content-Type: application/json' \
-d '{"model":"llama-8b","prompt":"How many tokens is this?"}' | jq '.count'
curl -s localhost:8000/metrics | grep -E '^vllm:' | cut -d'{' -f1 | sort -u
Tip
/healthonly tells you the engine process is up. For a readiness probe that proves the model can generate, send a one-token request instead:max_tokens: 1on a short prompt with a 10-second timeout. Use/healthfor liveness only, or a slow request may get a healthy pod restarted.
Use /tokenize to check prompt sizes against max_model_len before a batch job. Its token counts match the server's tokenizer exactly. A local tiktoken estimate does not.
Metrics to watch
Metric names vary by version. List them with the /metrics command above.
| Metric | Read it as |
|---|---|
vllm:num_requests_running / vllm:num_requests_waiting | If waiting stays above 0 while running is at max-num-seqs, you need more capacity or a lower max_tokens |
vllm:gpu_cache_usage_perc or vllm:kv_cache_usage_perc | Sustained > 0.95 means preemptions are coming |
vllm:num_preemptions_total | Rising means the KV cache is too small for your concurrency |
vllm:time_to_first_token_seconds | Queueing + prefill. Goes up with long prompts and queue depth |
vllm:inter_token_latency_seconds or vllm:time_per_output_token_seconds | Decode speed users feel. Goes up with large prefill batches |
vllm:prefix_cache_hits_total / vllm:prefix_cache_queries_total | Prefix cache hit ratio. Low means your prompts don't share a stable prefix |
PromQL for p95 TTFT and prefix cache hit ratio:
histogram_quantile(0.95, sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m])))
sum(rate(vllm:prefix_cache_hits_total[5m])) / sum(rate(vllm:prefix_cache_queries_total[5m]))
Benchmarks with vllm bench serve
Run the benchmark from a separate pod or machine against the deployed endpoint, not on the GPU node:
vllm bench serve \
--backend openai-chat --endpoint /v1/chat/completions \
--base-url http://vllm.inference.svc:8000 \
--model llama-8b --tokenizer meta-llama/Llama-3.1-8B-Instruct \
--dataset-name random --random-input-len 2000 --random-output-len 300 \
--num-prompts 500 --max-concurrency 32 --request-rate inf \
--save-result --result-dir ./bench
--request-ratesets how fast requests start.--max-concurrencycaps how many run at once. Sweep concurrency (8, 16, 32, 64) and plot TTFT p95 against output tokens per second. The knee in that curve is your real per-replica capacity.- Match
--random-input-lenand output length to production percentiles, not averages. Long-tail prompts are what break TTFT. - How the random dataset lengths are sampled has changed between versions, so read
vllm bench serve --helpfor your build.
Quick symptom map
| Symptom | Likely cause | First fix |
|---|---|---|
| Startup error that KV cache can't hold max seq len | --max-model-len too large for leftover VRAM | Lower --max-model-len, use --kv-cache-dtype fp8 or add TP |
| TTFT spikes while throughput looks fine | Queueing or long prefills | Check num_requests_waiting. Enable chunked prefill. Scale out |
| Decode stutters when long prompts arrive | Prefill steps crowding out decode | Lower --max-num-batched-tokens |
| Answers change after redeploy with no code change | Repo generation_config.json changed, or revision not pinned | --revision <sha>, --generation-config vllm |
| 404 model not found | Client uses repo id, server uses --served-model-name | Align names. Check /v1/models |
Prevent it next time
- Keep the full
vllm servecommand in version control, with every flag set explicitly, including the ones that match today's defaults. - Save
vllm bench serveresults for each release and compare before you promote a new image. - Alert on waiting requests, KV cache usage and preemption rate, not on GPU utilization.