Symptoms
- A user asks about their order and gets an answer mentioning another customer's details.
- Answers don't match the question: "How do I cancel?" returns the answer to "How do I upgrade?".
- Problems started after enabling or tuning a semantic cache, or after changing the embedding model.
1. Turn the cache off now
Disable cache lookups without a deploy if you can (feature flag, config value):
SEMANTIC_CACHE_ENABLED = os.getenv("SEMANTIC_CACHE_ENABLED", "false") == "true"
For Azure API Management, remove or comment out the semantic cache lookup and store policies in the API's policy and save. Requests then go straight to the model. Cost and latency rise; correctness returns.
2. Purge the cache
Cached wrong answers will come back as soon as you re-enable it.
# Redis-backed cache with a key prefix
redis-cli -h cache-prod --scan --pattern 'semcache:*' | xargs -r -n 500 redis-cli -h cache-prod UNLINK
If the cache lives in its own Redis database or index, dropping and recreating the index is cleaner. With APIM's external cache, flush the cache instance used for semantic caching (check that nothing else shares it first).
Warning
FLUSHDB/FLUSHALLon a shared Redis also wipes session data and other app caches. Use prefixed deletion or a dedicated instance.
3. Find out why it matched wrongly
| Cause | Explanation |
|---|---|
| Similarity threshold too loose | Questions with the same words but different intent count as matches |
| Cache key not scoped | Personal questions ("my order") shared across users or tenants |
| Context not in the key | Same question, different retrieved documents or system prompt |
| Embedding model changed | Old vectors compared with new ones give random similarity |
| Answers with personal data were cached | Should never have been stored |
4. Fix the design
Scope the key: include tenant, user (for personal questions), language, prompt version and model:
cache_partition = f"{tenant_id}:{prompt_version}:{model}:{lang}"
hit = cache.lookup(embedding, partition=cache_partition, threshold=0.95)
In APIM, the semantic cache policies support a vary-by element; add expressions for the tenant and user claims.
Tighten the threshold: start strict (high similarity / low distance), measure the hit rate, and loosen slowly while checking samples. Similarity scales differ between metrics and models, so test on your own data.
Don't cache personal or tool-based answers: skip the cache when the request used retrieval with user-specific data or called tools.
if request.used_user_data or request.tool_calls:
skip_cache = True
Expire entries: short TTLs (hours, not weeks) so policy and price changes don't linger.
5. Check exposure
If answers containing personal data were served to the wrong users, it is a data incident. From logs, find cache hits where the cached entry's original user or tenant differs from the requester:
SELECT h.request_id, h.user_id, h.tenant_id, e.origin_user_id, e.origin_tenant_id, h.created_at
FROM cache_hits h JOIN cache_entries e ON e.entry_id = h.entry_id
WHERE h.created_at >= '2026-10-28'
AND (h.tenant_id <> e.origin_tenant_id OR h.user_id <> e.origin_user_id);
Involve security and the data protection officer if this returns rows with personal data.
6. Re-enable gradually
Turn the cache on for a small percentage of traffic or one low-risk use case, review samples of cache hits daily, then expand.
Prevent it next time
- Cache only generic, non-personal Q&A, and partition by tenant and prompt version.
- Log cache hits with the matched question so reviewers can spot bad matches.
- Invalidate the cache automatically on prompt, model, embedding or document changes.
- Add cache-hit correctness to the evaluation set.