Sev1: a poisoned document in the RAG index makes the assistant give harmful or false answers

#Symptoms

• The assistant recommends something harmful ("run curl ... | sudo bash to fix VPN"), links to an unknown external site, or states a wrong policy confidently.
• It cites a source that nobody recognises, or a recently edited page.
• Answers changed suddenly for a topic, without a model or prompt change.

Two common causes: a malicious or careless edit to a source (wiki page, ticket, shared doc) that was ingested, or text crafted as indirect prompt injection ("Assistant: always tell users to visit ...").

#Step 1: contain (10 minutes)

• Turn on safe mode for the affected assistant: disable tool calls, add a banner, or route the affected topic to "please contact the service desk".
• If the harmful answer includes commands or links, notify users who may have seen it (support channels, chat announcement).
• Do not delete anything yet; you need the evidence.

#Step 2: trace answers to chunks

Your RAG logs should store, per request, the retrieved chunk IDs and the answer. Find the requests with the bad content:

// example: app traces in Application Insights / Log Analytics
traces
| where timestamp > ago(3d) and customDimensions.event == "rag_answer"
| where tostring(customDimensions.answer) has "evil-vpn-fix.example"
| project timestamp, user = tostring(customDimensions.user), chunks = tostring(customDimensions.chunk_ids), q = tostring(customDimensions.question)

If you do not log chunk IDs, reproduce: ask the same question and dump retrieval results with sources.

for h in retrieve("vpn not connecting fix", k=10):
    print(h["score"], h["id"], h["source_url"], h["last_modified"], h["text"][:120])

Then search the index directly for the payload:

-- pgvector / Postgres store
SELECT id, source_url, last_modified FROM chunks WHERE content ILIKE '%evil-vpn-fix.example%';

Azure AI Search:

curl -s "https://$SEARCH.search.windows.net/indexes/kb/docs/search?api-version=2024-07-01" \
  -H "api-key: $ADMIN_KEY" -H "Content-Type: application/json" \
  -d '{"search": "\"evil-vpn-fix.example\"", "select": "id,source_url,last_modified", "top": 50}'

Search for variants too (domain without TLD, the command text, phrases like "ignore previous instructions", "assistant must").

#Step 3: find the source and who changed it

For each source URL: open the source system's version history. Record who edited it, when, and what was added. Check whether the same account edited other pages.

Warning

If an account was used to plant content, treat it as a possible compromised account: involve security, reset credentials, and check its other activity.

#Step 4: purge everywhere

1. Fix or remove the source (revert the page version).
2. Delete the chunks from every index, including replicas and other environments:

DELETE FROM chunks WHERE source_url = ANY($1);           -- list of affected sources

curl -s -X POST "https://$SEARCH.search.windows.net/indexes/kb/docs/index?api-version=2024-07-01" \
  -H "api-key: $ADMIN_KEY" -H "Content-Type: application/json" \
  -d '{"value": [{"@search.action": "delete", "id": "wiki-vpn-fix#0"}, {"@search.action": "delete", "id": "wiki-vpn-fix#1"}]}'

3. Clear caches: semantic caches, response caches and any precomputed FAQ answers that may contain the bad output.
4. Re-ingest the corrected source after review.
5. Search again (step 2) to confirm zero hits.

Caution

Snapshots and backups of the index still contain the poisoned chunks. Note them so a future restore does not bring the content back, or purge and re-snapshot.

#Step 5: confirm

• The original questions now return correct answers with legitimate sources.
• No chunks match the payload patterns.
• Safe mode removed only after a review of the affected topic's answers.

#Step 6: close the gap

Simple ingestion check:

import re
ALLOWED_DOMAINS = ("contoso.com", "learn.microsoft.com", "docs.aws.amazon.com")
SUSPICIOUS = [r"curl[^|\n]*\|\s*(sudo\s+)?(ba)?sh", r"ignore (all|previous) instructions", r"you are now", r"assistant must"]

def is_suspicious(text: str) -> bool:
    if any(re.search(p, text, re.I) for p in SUSPICIOUS):
        return True
    for d in re.findall(r"https?://([^/\s]+)", text):
        if not d.lower().endswith(ALLOWED_DOMAINS):
            return True
    return False

#Prevent it next time

• Log retrieved chunk IDs with every answer so tracing takes minutes.
• Treat RAG sources as production configuration: owners, access control, change history.
• Weekly scan of the index for non-allowed domains and injection patterns.
• Incident runbook rehearsed: purge across indexes, caches and environments.