1. Estimate demand in tokens
peak TPM = peak concurrent users × requests per user per minute × (prompt tokens + output tokens)
Example: 400 concurrent users × 0.5 requests/min × (3,000 prompt + 500 output tokens) = 700,000 tokens per minute at peak.
Also capture:
- Output tokens separately. They drive latency and GPU time more than input tokens do.
- The peak-to-average ratio (business hours, month-end).
- Growth over 6–12 months.
2. Managed models: quota vs provisioned
| Option | Good for | Watch out |
|---|---|---|
| Pay-as-you-go (standard, global standard) | Variable or low volume, pilots | TPM/RPM quotas per region and subscription; 429s under load; shared capacity |
| Provisioned (Azure PTU, Bedrock provisioned throughput, OCI dedicated AI clusters) | Steady high volume, latency SLOs | Paid whether used or not; sized in units; needs a commitment |
| Batch APIs | Offline jobs | Lower cost; results in hours |
Common pattern: provisioned for the baseline, with spillover to pay-as-you-go through the gateway (see the AI gateway post).
Use the provider's capacity calculator with your real prompt and output token sizes. Throughput per unit differs by model and by input/output mix.
3. Self-hosted GPU sizing
Memory:
weights (GB) ≈ parameters (billions) × bytes per parameter
FP16/BF16 = 2, FP8 = 1, INT4 = ~0.5
KV cache (GB) ≈ 2 × layers × kv_heads × head_dim × bytes × context tokens × concurrent sequences / 1e9
A 70B model in FP16 needs about 140 GB for weights alone, so 2× 80 GB GPUs is the minimum, plus KV cache. In FP8 it's about 70 GB, which fits on one 141 GB H200 with room for KV cache.
Throughput: benchmark your model, quantisation and prompt mix with your serving engine (vLLM, TGI, TensorRT-LLM). Measure tokens per second per GPU at your latency target (for example p95 time-to-first-token < 1 s). Then:
GPUs needed = peak output tokens per second ÷ measured tokens per second per GPU × 1.3 (headroom)
4. Cloud GPU quotas and reservations
- Request GPU VM family quota (per region) weeks ahead. Large GPUs are often capacity-constrained.
- Use capacity reservations, or reserved or savings plans, for steady production.
- Have a second region or a second GPU type approved as a fallback.
- Spot or preemptible GPUs only for batch or interruptible work.
5. Monitor headroom
| Metric | Action threshold |
|---|---|
| TPM used vs quota (per deployment) | > 70% at peak: request more quota or add a deployment |
| PTU utilisation | > 80% sustained: add units; < 40%: reduce |
| 429 rate | > 1% of requests |
| GPU utilisation / KV cache use (self-hosted) | > 85% at peak |
| p95 time-to-first-token / tokens per second | Above or below the SLO |
6. Review cadence
Monthly capacity review: actual peak versus forecast, upcoming launches, model changes (bigger context means more tokens), and commitment renewals.
Tip
Long system prompts and large RAG contexts multiply every request. Cutting retrieved context from 8,000 to 3,000 tokens can cut the capacity you need by more than half, at no quality cost if retrieval is good.