Sev1: the RAG assistant is showing documents people shouldn't see. Pulling the data source

#Symptom

• A user reports the assistant quoted a salary file, a legal memo or a customer contract.
• Citations point to a library or folder that should be restricted.
• Often after adding a new data source, changing an indexer or migrating the index.

#Step 1: stop retrieval from the source (minutes)

The quickest switch is at query time, without touching the index. Every chunk should carry a source (or library) field. Exclude it with a filter driven by config:

BLOCKED_SOURCES = flags.get_list("rag-blocked-sources")   # e.g. ["sharepoint:HR", "share:Finance"]

def retrieve(query, user):
    filt = " and ".join(f"source ne '{s}'" for s in BLOCKED_SOURCES) or None
    return search_client.search(search_text=query, vector_queries=[vq(query)],
                                filter=combine(filt, user_security_filter(user)), top=8)

Azure AI Search OData filter example: source ne 'sharepoint:HR'. For many values, not search.in(source, 'sharepoint:HR,share:Finance', ',').

If your chunks don't have a source field, the fallback switch is safe mode (retrieval-only or handoff) for the whole assistant, or the feature flag off.

#Step 2: stop new data arriving

Disable or pause the indexer / ingestion job for that source so it doesn't keep adding documents:

• Azure AI Search indexer: remove its schedule (portal: Indexers → the indexer → Settings → Schedule: Once), or delete the indexer if its definition is kept in source control. Deleting an indexer doesn't delete indexed documents.
• Custom ingestion pipeline: disable the job or pause the queue consumer for that source.
• Bedrock Knowledge Base: don't run a sync until the source is fixed or removed.

Don't use "reset" here. It clears change tracking, and the next run re-ingests everything.

#Step 3: purge from the index

Delete the chunks by source:

results = search_client.search(search_text="*", filter="source eq 'sharepoint:HR'", select=["id"], top=1000)
ids = [{"id": r["id"]} for r in results]
while ids:
    search_client.delete_documents(documents=ids)
    results = search_client.search(search_text="*", filter="source eq 'sharepoint:HR'", select=["id"], top=1000)
    ids = [{"id": r["id"]} for r in results]

For other stores: DELETE FROM chunks WHERE source = 'sharepoint:HR' (pgvector), or a filter-based delete (Qdrant delete with a filter on payload). For Bedrock Knowledge Bases, remove the data source or the documents and sync.

#Step 4: purge caches and copies

Leaked content may also be in:

• Semantic / response caches (clear entries, or flush the whole cache).
• Conversation history stores (assistant answers quoting the documents).
• Logs and traces that store prompts and retrieved chunks.
• Evaluation datasets built from production traffic.

#Step 5: assess exposure

From retrieval logs (store user, query, retrieved doc ids per request):

AppTraces
| where TimeGenerated > ago(30d) and Message has "retrieval"
| extend user = tostring(Properties.user), docs = tostring(Properties.doc_ids)
| where docs has "sharepoint:HR"
| summarize hits = count(), first = min(TimeGenerated), last = max(TimeGenerated) by user

Share the list with data protection / privacy. Depending on the content, this may be a reportable data breach. That decision belongs to them, not the on-call engineer.

#Step 6: fix the cause

def user_security_filter(user):
    groups = ",".join(user.group_ids)
    return f"allowed_groups/any(g: search.in(g, '{groups}', ','))"

#Step 7: re-add the source

Only after permission trimming is in place and tested with users from inside and outside the allowed groups. Remove it from rag-blocked-sources last.

Caution

Hiding the citation is not a fix. If the chunk is retrieved, the model can still repeat its content.

#Prevent it next time

• Every chunk has source and allowed_groups fields from day one.
• Query-time blocklist flag ready before go-live.
• Data owner approval for each new source; test with a low-privilege user.