RAG keeps answering from outdated documents: metadata filters, time decay and supersession

#Symptom

Users report that the assistant quotes an old version of a policy, or a runbook for a server that's been retired. The new document is in the index. Both versions are just very similar, so embeddings can't tell them apart.

Semantic similarity knows about meaning, not time. "Laptop refresh policy 2023" and "Laptop refresh policy 2026" look almost identical to it.

This is different from an index that's stopped updating. There, the new document is missing. Here, both are present, and the wrong one wins.

#Step 1: confirm it's a retrieval problem

Log what was retrieved, with dates, for a failing question:

def debug_retrieval(question, k=10):
    for rank, hit in enumerate(search(question, k=k), start=1):
        print(rank, round(hit.score, 3), hit.metadata.get("doc_id"),
              hit.metadata.get("version"), hit.metadata.get("updated_at"),
              hit.text[:70])

#Step 2: add the metadata

You can't filter on what you didn't store. Each chunk needs this information from the source:

Use the date from the source system. The date that you indexed it is misleading: a full re-index would make every document look brand new. A chunk's file date can be wrong too, since a copy or a migration resets it.

Tip

If your source has no reliable dates, that's what to fix first. Ask the document owners, or pull the dates from the system of record. Guessing from a filename won't hold up for long.

#Step 3: retire superseded versions

For content with clear versions, this is the best fix, and no ranking tricks are needed: old versions shouldn't be searchable by default.

When a new version arrives, in a single job:

1. Insert the new chunks, with status = 'current'.
2. Mark the old chunks as superseded, or delete them.
3. Verify the counts.

With pgvector:

BEGIN;

INSERT INTO chunks (doc_id, version, status, source_updated_at, text, embedding)
SELECT doc_id, version, 'current', source_updated_at, text, embedding
FROM staging_chunks
WHERE doc_id = 'hr-laptop-policy';

UPDATE chunks
SET status = 'superseded', effective_to = now()
WHERE doc_id = 'hr-laptop-policy'
  AND version < (SELECT max(version) FROM staging_chunks WHERE doc_id = 'hr-laptop-policy')
  AND status = 'current';

COMMIT;

Then filter on it in every query:

SELECT id, doc_id, version, text, 1 - (embedding <=> $1) AS score
FROM chunks
WHERE status = 'current'
ORDER BY embedding <=> $1
LIMIT 20;

Add an index on status, and check that filtering doesn't reduce your result count. See too few results with filters.

With a Qdrant filter, the same idea looks like this:

from qdrant_client import models

hits = client.query_points(
    collection_name="docs",
    query=query_vector,
    query_filter=models.Filter(must=[
        models.FieldCondition(key="status", match=models.MatchValue(value="current")),
    ]),
    limit=20,
).points

Create a payload index on the field.

Deleted documents need handling as well. Compare source and index IDs regularly, and remove any that have gone from the source. Otherwise a deleted document stays searchable for good. The incremental indexing post covers change detection.

#Step 4: handle dated questions

Users do sometimes want the old version: "What was the travel policy in 2024?" Don't lose that ability. Keep superseded chunks where audit needs matter, and filter by effective date when the question names one:

SELECT id, doc_id, version, text
FROM chunks
WHERE effective_from <= '2024-06-30'
  AND (effective_to IS NULL OR effective_to > '2024-06-30')
ORDER BY embedding <=> $1
LIMIT 20;

Extract the date from the question, using a small rule or an LLM call that returns structured output. If there's no date, use status = 'current'. For more on rewriting questions, see query rewriting.

#Step 5: use time decay only where it fits

Some content has no version, but gets less useful with age, such as incident notes, tickets and announcements. Here, a recency boost makes sense.

Fetch more candidates than you need, then re-score them:

import math
from datetime import datetime, timezone

HALF_LIFE_DAYS = {"incident": 90, "announcement": 30}

def freshness(updated_at, content_type, now=None):
    half_life = HALF_LIFE_DAYS.get(content_type)
    if half_life is None:
        return 1.0                       # no decay for this type
    now = now or datetime.now(timezone.utc)
    age_days = max((now - updated_at).total_seconds() / 86400, 0)
    return 0.5 ** (age_days / half_life)

def rescore(hits, weight=0.2, k=5):
    scored = []
    for h in hits:
        f = freshness(h.updated_at, h.content_type)
        scored.append((h.score * ((1 - weight) + weight * f), h))
    scored.sort(key=lambda x: x[0], reverse=True)
    return [h for _, h in scored[:k]]

The weight keeps relevance in charge. At 0.2, the oldest document loses at most 20% of its score. So a very relevant old document can still beat a slightly relevant new one.

Warning

Don't use decay as a fix for versioning. It only lowers an old document's rank, and it can still appear. And applying it to everything hides old content that's still correct, such as a core security standard that hasn't needed changes. Scores from different retrievers aren't comparable either, so apply it to a normalized score or after a reranker.

#Step 6: help the model choose

Give the model dates and versions, and tell it what to do when sources conflict:

def build_context(hits):
    parts = []
    for i, h in enumerate(hits, start=1):
        parts.append(
            f"[{i}] {h.title} | version {h.version} | "
            f"updated {h.updated_at:%Y-%m-%d} | status {h.status}\n{h.text}"
        )
    return "\n\n".join(parts)

SYSTEM = """Answer only from the sources below.
If sources conflict, prefer the one with status 'current' and the latest date.
Mention the date of the source that you used.
If the only relevant source is old or superseded, say so clearly.
If the sources don't answer the question, say that you don't know."""

Showing a date in the answer also lets users spot a problem. Make sure citations point to the right chunk.

#Step 7: measure it

Add time-sensitive cases to your golden set. Each one names the document version that's correct:

CASES = [
    {"q": "How often are laptops refreshed?", "must": ("hr-laptop-policy", 3), "must_not": [("hr-laptop-policy", 2)]},
    {"q": "What was the laptop policy in 2024?", "must": ("hr-laptop-policy", 2), "must_not": []},
]

def stale_rate(cases, k=5):
    stale = 0
    for c in cases:
        hits = search(c["q"], k=k)
        found = {(h.metadata["doc_id"], h.metadata["version"]) for h in hits}
        if any(x in found for x in c["must_not"]) or c["must"] not in found:
            stale += 1
    return stale / len(cases)

Track this figure for each release, along with overall recall, so you can see if a change hurts general relevance. Also report how old your index is:

SELECT status,
       COUNT(*) AS chunks,
       COUNT(DISTINCT doc_id) AS docs,
       MIN(source_updated_at) AS oldest,
       MAX(source_updated_at) AS newest
FROM chunks
GROUP BY status;

#Clean up the source too

Technology can only do so much with messy content. Two "final" versions with no dates will defeat any retriever.

• Give each document an owner and a review date.
• Remove duplicates before ingestion.
• Mark drafts, and don't index them by default.
• Keep access rules in the same metadata. An old file can also have the wrong permissions.

#Prevent it next time

• Require doc_id, version, status and a source date at ingestion, and reject anything without them.
• Alert when a source hasn't been updated for longer than expected.
• Run the stale cases in CI after each re-index. See quality regression after reindex.
• Add this to your RAG production checklist.