RAG citations point to the wrong document or page: fix chunk IDs, metadata and citation checking

#Symptom

Users report that the answer is right, but the citation links to:

• a different document than the one containing the fact,
• the right document but the wrong page or section,
• a document that no longer exists (after a re-index),

or the answer cites [3] when only two sources were shown. Trust in the assistant drops fast once people notice.

#Where citations come from

A citation passes through several steps, and each one can break it:

1. Parse the document (PDF, HTML, Word) and record page/section.
2. Chunk the text and copy metadata to each chunk.
3. Index chunks with an ID.
4. Retrieve and rerank the top chunks.
5. Build the prompt with numbered sources.
6. Model answers and cites numbers.
7. Map numbers back to links in the UI.

Test each step separately before blaming the model.

#Cause 1: metadata lost or wrong during chunking

Splitters often create new chunk objects and drop the parent's metadata, or keep only source but not page. Merged chunks spanning two pages keep the first page only.

Check a sample of indexed chunks directly in the vector store:

for hit in index.search("refund policy", top=5):
    print(hit["id"], hit.get("doc_id"), hit.get("page"), hit.get("section"), hit["content"][:80])

If page or doc_id is missing or the same for every chunk, fix the ingestion. Carry metadata explicitly:

def chunk_document(doc_id, pages, size=800, overlap=100):
    chunks = []
    for page_no, text in pages:            # page_no as shown to users (1-based)
        start = 0
        while start < len(text):
            piece = text[start:start + size]
            chunks.append({"doc_id": doc_id, "page": page_no, "offset": start, "content": piece})
            start += size - overlap
    return chunks

Page numbers off by one are very common: many PDF libraries are 0-based, and users expect 1-based or the printed page label. Decide which one you show and convert once, at parse time.

#Cause 2: unstable or colliding chunk IDs

If chunk IDs are sequential (chunk_0, chunk_1 ...) per run, re-indexing one document can overwrite another document's chunks, or old IDs point to new text. Citations stored in chat history then link to the wrong content.

Use deterministic IDs derived from the document and position:

import hashlib
def chunk_id(doc_id: str, version: str, page: int, offset: int) -> str:
    return hashlib.sha1(f"{doc_id}|{version}|{page}|{offset}".encode()).hexdigest()

Include a document version (content hash or modified time). When a document changes, delete its old chunks by doc_id before inserting new ones, so nothing stale remains. See RAG quality regression after re-index.

#Cause 3: numbering changes between retrieval and prompt

The pipeline numbers sources after retrieval, then a reranker reorders or filters them, and the UI maps [2] using the original order. Or duplicates are removed after numbering.

Rule: number sources once, at the moment you build the prompt, and keep that exact list to map citations back:

def build_context(chunks):
    sources = []
    blocks = []
    for n, c in enumerate(chunks, start=1):
        sources.append({"n": n, "doc_id": c["doc_id"], "page": c["page"], "url": c["url"]})
        blocks.append(f"[{n}] (doc: {c['title']}, page {c['page']})\n{c['content']}")
    return "\n\n".join(blocks), sources   # store `sources` with the answer

#Cause 4: the prompt makes citing ambiguous

• Sources without clear boundaries or labels.
• No instruction on format, so the model writes "according to the policy document" instead of [2].
• Too many near-duplicate chunks; the model picks any one of them.

Give a strict instruction: "Cite sources with their number in square brackets, like [2]. Only cite a source if the sentence is supported by that source's text. If no source supports it, say you don't know." Remove near-duplicate chunks before building the prompt.

#Cause 5: the model cites but the source doesn't support it

Even with a clean pipeline, models sometimes cite the wrong number. Check automatically after generation:

import re
def check_citations(answer, sources_by_n, chunks_by_n, min_overlap=0.3):
    problems = []
    for sentence in re.split(r"(?<=[.!?])\s+", answer):
        for n in map(int, re.findall(r"\[(\d+)\]", sentence)):
            if n not in sources_by_n:
                problems.append(("unknown_source", n, sentence)); continue
            words = set(re.findall(r"\w{4,}", sentence.lower()))
            src = set(re.findall(r"\w{4,}", chunks_by_n[n]["content"].lower()))
            if words and len(words & src) / len(words) < min_overlap:
                problems.append(("weak_support", n, sentence))
    return problems

Word overlap is a crude check, but it catches unknown numbers and obvious mismatches cheaply. For higher accuracy, use an LLM-as-judge or NLI model on the flagged sentences only.

On failure: drop the bad citation, regenerate with fewer sources, or show the answer with a "sources could not be verified" note.

#Evaluate it

Build a small test set (30-50 questions) with the expected document and page, and measure:

• Citation precision: cited sources that actually support the sentence.
• Citation recall: answers that cite the expected document.
• Dead links: cited chunk IDs that no longer exist in the index.

Run it in CI after every ingestion or prompt change.

#Prevent it next time

• Deterministic chunk IDs with document version; delete-then-insert per document.
• Number sources once and persist the mapping with each answer.
• Automatic citation check in production, with the failure rate on your dashboard.
• Related: retrieval budget, top-k and reranking.