Applies toAzure OpenAI / OpenAI (automatic prompt caching), Anthropic Claude (cache_control breakpoints, including on Bedrock and Vertex), Amazon Bedrock prompt caching for supported models, Gemini context caching; self-hosted vLLM prefix caching
How it works
The provider stores the computed state for a prompt prefix. A later request that starts with exactly the same tokens reuses it: cached input tokens are billed at a lower rate (check current pricing) and processed faster.
Rules vary by provider:
| Provider | How you enable it | Notes to check |
|---|---|---|
| OpenAI / Azure OpenAI | Automatic for prompts above a minimum length on supported models | Exact-prefix match; usage shows cached tokens |
| Anthropic Claude | Mark cache breakpoints with cache_control | Cache writes may cost more than normal input; reads much less; lifetime options |
| Amazon Bedrock | Cache checkpoints for supported models | Model support list |
| Gemini | Explicit context caches and implicit caching | Storage charge per time for explicit caches |
| vLLM (self-hosted) | --enable-prefix-caching | Saves GPU compute, not a billing discount |
Order the prompt for caching
[1] System instructions ← static
[2] Tool definitions ← static
[3] Long reference material ← static or rarely changing (policies, schemas, few-shot examples)
[4] Conversation history ← grows, but earlier turns stay identical
[5] Retrieved chunks for this turn ← variable
[6] User's current question ← variable
Things that break the cache: a timestamp or request ID at the top, re-ordered tool definitions, whitespace changes, per-user text in the system prompt.
Anthropic example
resp = client.messages.create(
model="claude-sonnet-5-5",
max_tokens=1024,
system=[
{"type": "text", "text": LONG_STATIC_INSTRUCTIONS,
"cache_control": {"type": "ephemeral"}}
],
messages=[{"role": "user", "content": question}],
)
print(resp.usage) # cache_creation_input_tokens, cache_read_input_tokens
Check it's working
| Provider | Field |
|---|---|
| OpenAI / Azure OpenAI | usage.prompt_tokens_details.cached_tokens |
| Anthropic | usage.cache_read_input_tokens / cache_creation_input_tokens |
Log these per request and track cache hit ratio per feature. A sudden drop after a release usually means something changed at the top of the prompt.
When it pays off most
- Long system prompts or tool lists sent on every call.
- Multi-turn chats (history reused).
- Document Q&A where many questions hit the same long document.
- Agents calling the model many times with the same instructions.
Tip
Move anything per-request (user name, date, request ID) from the system prompt into the final user message. That one change often turns a 0% cache hit rate into a high one.
Checklist
- Provider caching rules checked (minimum length, lifetime, pricing)
- Prompt ordered static → variable
- No volatile data in the prefix
- Cache usage fields logged
- Hit ratio tracked per feature and after releases