Options at a glance
| Scheme | Memory for weights vs BF16 | Quality risk | Hardware notes |
|---|---|---|---|
| FP8 (W8A8) | ~½ | Usually small | Native FP8 compute on Ada, Hopper, Blackwell; Ampere runs weight-only FP8 kernels |
| INT8 (W8A8) | ~½ | Small | Wide support |
| INT4 weight-only (W4A16, AWQ/GPTQ) | ~¼ | Moderate, task-dependent | Wide support; best for memory-bound, low-batch serving |
| FP8 KV cache | KV cache ~½ | Small to moderate | More concurrent sequences / longer contexts |
Check the vLLM docs for the current support matrix on your GPU generation. Kernel support changes often.
1. Baseline at BF16
Record on your hardware with your prompts: GPUs needed, max concurrent sequences at your context length, TTFT p95, output tokens/s, and evaluation score.
2. Quickest path: dynamic FP8 in vLLM
vllm serve meta-llama/Llama-3.1-70B-Instruct --quantization fp8 \
--tensor-parallel-size 2 --max-model-len 16384 --kv-cache-dtype fp8
Weights are quantized at load time, with no calibration. Good for a first test; a pre-quantized checkpoint loads faster and gives you a fixed, evaluated artefact.
3. Produce a checkpoint with llm-compressor
from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot # older releases: llmcompressor.transformers
from llmcompressor.modifiers.quantization import QuantizationModifier
MODEL = "meta-llama/Llama-3.1-8B-Instruct"
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype="auto")
tok = AutoTokenizer.from_pretrained(MODEL)
recipe = QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"])
oneshot(model=model, recipe=recipe)
model.save_pretrained("Llama-3.1-8B-Instruct-FP8-Dynamic")
tok.save_pretrained("Llama-3.1-8B-Instruct-FP8-Dynamic")
INT4 schemes (W4A16 with GPTQ) need a calibration dataset. Use a few hundred samples that look like your production prompts, not generic web text. Many popular models also have published FP8/AWQ/GPTQ checkpoints; check who produced them and evaluate before trusting them.
4. Evaluate quality
| Check | Pass condition (example) |
|---|---|
| Task evaluation set | Within 1–2 points of BF16 |
| Structured output validity | No drop |
| Long-context tasks | No drop (quantization errors show up here first) |
| Non-English / code, if used | Within tolerance |
| Spot review of 50 answers | No new failure pattern |
INT4 often holds up on extraction and classification but can lose more on reasoning and long-form generation. Decide per workload.
5. Load test
Use vLLM's benchmark tooling or your own load generator with realistic prompt/output lengths:
| Variant | GPUs | Max concurrency at SLA | TTFT p95 | Tokens/s | Eval score |
|---|---|---|---|---|---|
| BF16 | measure | measure | measure | measure | measure |
| FP8 + FP8 KV | |||||
| W4A16 |
The win is usually one of: same model on fewer GPUs, more concurrency on the same GPUs, or a larger model in the same footprint.
6. Roll out
- Deploy the quantized model as a new deployment behind the gateway.
- Canary 10% of traffic; watch feedback, eval drift, format errors.
- Shift fully; keep BF16 config available for rollback for a few weeks.
- Record scheme, tool versions and calibration data with the model artefact.
Warning
Changing quantization is a model change. Run the full evaluation gate, not just a smoke test. Small numeric differences can flip borderline classifications or break strict JSON outputs.
Checklist
- BF16 baseline (capacity, latency, quality)
- Scheme matched to GPU generation
- Checkpoint produced or vetted, calibration data representative
- Evaluation within tolerance per workload
- Load test shows the expected gain
- Canary rollout with rollback path