Re-embedding a vector index when the embedding model retires - a zero-downtime switch with dual indexes and a recall check

#Why you can't do it gradually

An embedding model maps text into its own vector space. A query embedded with model B compared against documents embedded with model A gives meaningless similarity scores. So:

• You can't re-embed "a bit at a time" in the same index and keep searching it.
• The query path and the document vectors must switch at the same moment.
• Dimensions often change too (for example 1024 to 1536), which a fixed-size vector field can't hold.

Current example: on Microsoft Foundry, Cohere-embed-v3-english and Cohere-embed-v3-multilingual are deprecated on 15 November 2026 and retire on 15 February 2027, with Cohere embed v5 models named as replacements.

#1. Plan

Rough backfill time = chunks / (requests per minute x chunks per request). Request a quota increase early if needed.

#2. Build a second index

Create kb-v2 with the new vector field size and the same metadata fields. Don't touch kb-v1, which keeps serving queries.

Backfill job outline:

import itertools, time

def batches(it, n):
    it = iter(it)
    while (b := list(itertools.islice(it, n))):
        yield b

for batch in batches(read_chunks_from_source(), 64):
    vectors = embed_v2([c.text for c in batch])          # new model, input_type set if the API needs it
    upsert("kb-v2", [dict(id=c.id, vector=v, **c.meta) for c, v in zip(batch, vectors)])
    checkpoint(batch[-1].id)                              # resume after failures

Some embedding APIs distinguish document and query inputs (for example input_type of search_document vs search_query). Use the right one for each side.

#3. Keep it in sync

While the backfill runs, new and changed documents must reach both indexes. Simplest: make the ingestion pipeline write to v1 and v2 (dual write) from the start of the backfill. Then re-run the backfill for anything changed since it started (by updated_at).

#4. Compare quality before switching

Use a retrieval test set: real questions with the chunk IDs that should be found.

def recall_at_k(index, embed, tests, k=5):
    hits = 0
    for q, expected in tests:
        ids = [r.id for r in search(index, embed(q), k)]
        hits += any(i in expected for i in ids)
    return hits / len(tests)

print("v1", recall_at_k("kb-v1", embed_v1_query, tests))
print("v2", recall_at_k("kb-v2", embed_v2_query, tests))

Also check: latency, hybrid search behaviour (if you combine keyword and vector), and any score thresholds. Similarity score ranges change between models, so a cut-off like "ignore results below 0.75" must be re-tuned.

#5. Switch

Switch the query path (embedding model and index name) together, in one config change:

• Azure AI Search: point the app at kb-v2, or use an index alias if you've set one up.
• pgvector: swap a view or table name inside a transaction.
• Qdrant/Milvus/OpenSearch: switch the collection alias.

Keep v1 for a rollback window (a week is typical), then delete it and stop dual writes.

#6. Watch after the switch

• User feedback signals and answer quality metrics.
• "No results" rate and average similarity scores.
• Embedding call errors and latency for the new model.

#Checklist

• Source text available for every chunk
• New index with correct dimensions and metadata fields
• Quota sized for the backfill; resumable job
• Dual writes during the backfill
• Recall compared on a test set; score thresholds re-tuned
• Query model and index switched together; rollback window kept
• Done well before the old model's retirement date

#Sources

• Microsoft Learn: Model retirement schedule (Cohere embed v3)