The rule
Vectors from different embedding models live in different spaces. A query embedded with model B compared against documents embedded with model A gives meaningless scores, even if the dimensions happen to match. Also, the same model name with a new version can change vectors. Pin the exact version.
Typical triggers: a model is retired by the provider, a better or cheaper model is released, or multilingual support is needed.
1. Plan
- Count the documents and chunks, and estimate cost and time: chunks × tokens per chunk ÷ embedding throughput (respect rate limits).
- Check the new dimension and supported distance metric. The index schema probably changes (for example 1536 → 3072 or 1024).
- Decide on reduced dimensions if the model supports it (some models allow shorter vectors with small quality loss).
2. Build a parallel index (blue-green)
index: kb-v1 (model A, live) ← alias "kb" points here
index: kb-v2 (model B, new) ← being filled
- Re-embed from the source text, not from old vectors.
- Store
embedding_modelandembedding_versionas fields on each record. - Make the job resumable (track the last processed id) and idempotent.
3. Keep both in sync during the backfill
New and updated documents must go to both indexes until switch-over. The simplest way: the ingestion pipeline writes to v1 and v2 when a flag is on.
4. Evaluate before switching
Run the same labelled question set against v1 and v2:
| Metric | v1 | v2 |
|---|---|---|
| Recall@5 | 0.82 | 0.89 |
| MRR | 0.71 | 0.78 |
| p95 query latency | 45 ms | 60 ms |
Also run end-to-end answer quality checks. Retrieval gains don't always become better answers.
5. Switch
- Deploy the query service to embed queries with model B and query
kb-v2, behind a feature flag or alias switch, at the same moment. - Watch the metrics: zero-result rate, click-through or thumbs-up, latency, errors.
- Keep v1 and model A access for a rollback window (a week or two).
Warning
The most common migration bug: the index switched but one service (a batch job, a second app) still embeds queries with the old model. Search the codebase and config for the old model name before the switch.
6. Clean up
Remove dual writes, delete kb-v1, and update the runbook with the new model version and its retirement date.
Tip
Put the embedding model name in the index name or metadata. When someone asks "which model built this index?" six months later, the answer shouldn't require archaeology.