Applies toAzure OpenAI / Azure AI Foundry provisioned deployments (PTU), Amazon Bedrock Provisioned Throughput; same thinking for reserved GPU capacity for self-hosted models
Measure utilisation
In Azure Monitor for the Azure OpenAI resource, check Provisioned-managed utilization (per deployment) by hour over several weeks.
| Pattern | Meaning |
|---|---|
| Mostly < 40%, short peaks | Over-provisioned for baseline |
| Often at 100% with 429s | Under-provisioned or peaks not offloaded |
| Busy business hours, idle nights | Batch work could fill nights |
Size for the baseline, not the peak
PTU tier ≈ steady business-hours load (use the provider's capacity calculator with your prompt/completion token sizes)
Peaks → pay-as-you-go (Standard/Global Standard) deployments
Use the current capacity calculator: throughput per PTU depends on the model and on input/output token mix.
Spillover to pay-as-you-go
| Option | How |
|---|---|
| Built-in spillover (where available) | Configure a Standard deployment to receive overflow from the provisioned deployment; check feature availability |
| Gateway routing | APIM backend pool with priority: PTU first, Standard second; on 429 retry to next (see the HA endpoints post) |
| Application fallback | Catch 429 from PTU and retry on a Standard deployment |
Don't retry repeatedly against a saturated PTU deployment; route away immediately.
Fill the valleys
- Move bulk jobs (summaries, classification backfills, evaluations) to off-peak hours on the PTU deployment.
- Or move them to batch APIs at lower prices (see the batch transformation post) and keep PTU for interactive traffic.
Reduce tokens per request
Every token saved increases effective PTU capacity:
- Prompt caching (see the prompt caching post).
- Shorter outputs via
max_tokensand format instructions. - Smaller models for simple tasks (see the model routing post).
Review cadence
- Monthly: utilisation percentiles, 429 count, spillover volume and cost.
- Before reservation renewals: re-size based on the last 3 months.
Warning
Monthly or yearly PTU reservations can't simply be cancelled. Base commitments on measured baseline usage, and check current reservation terms before buying.
Checklist
- Hourly utilisation measured over weeks
- PTUs sized for baseline with calculator
- Spillover or gateway fallback for peaks
- Batch work moved off-peak or to batch APIs
- Token reduction applied
- Monthly utilisation review