Support bots get the same questions again and again ("how do I reset my VPN password?"). A semantic cache answers repeats in milliseconds without a model call.
1. How it works
- Embed the incoming question.
- Search the cache for a previous question with similarity ≥ threshold, within the same scope.
- Hit → return the stored answer. Miss → call the model, then store question, embedding and answer with a TTL.
2. Choosing the threshold
Too low and you return the answer to a different question:
| Question A | Question B | Same answer? |
|---|---|---|
| "How do I reset my VPN password?" | "VPN password reset steps" | Yes |
| "How do I reset my VPN password?" | "How do I reset my email password?" | No, but very similar embeddings |
Build a labelled set of question pairs (same / different answer), compute similarities with your embedding model, and pick the threshold with close to zero false hits. Thresholds are model-specific; don't copy a number from a blog.
3. Scope the cache key
The cached answer must be valid for the new user. Include in the key or filter:
- Tenant / customer
- User permissions or role (answers built from documents only some users can see)
- Language
- Product and version
- Prompt and model version (bump it to invalidate everything after a prompt change)
Warning
A shared cache across tenants or permission levels can leak confidential answers to the wrong user. Scope first, optimise later.
4. What not to cache
- Personal or account-specific questions ("what's the status of my order?").
- Answers that used tool calls with live data.
- Time-sensitive content (outages, prices) unless the TTL is short.
5. Invalidation
- TTL on every entry (hours to days, depending on how often content changes).
- When a source document changes, delete cache entries that cited it. Store the source ids with each cached answer.
6. Provider prompt caching is different (use both)
Many providers cache a repeated prompt prefix (long system prompt, tool definitions, static documents) and bill it at a lower rate with lower latency. It doesn't skip the model call, but it is safe because the answer is still generated fresh. Put stable content first in the prompt to benefit.
7. Measure
- Hit rate, and cost saved per day.
- Sample cache hits for human review: was the returned answer right for the new question?
- Thumbs-down rate on cached vs fresh answers.
Tip
Start with an exact-match cache on the normalised question (lowercase, trimmed). It's risk-free, and in many support bots it already catches a big share of repeats before you add semantic matching.