Symptoms
kubectl -n vectordb get pods
# qdrant-0 0/1 CrashLoopBackOff 7 ...
kubectl -n vectordb describe pod qdrant-0 | grep -A3 "Last State"
# Reason: OOMKilled
RAG queries time out or fail; ingestion jobs error out.
Why it happens
HNSW indexes and (by default) raw vectors are kept in RAM. Memory needed grows with:
vectors × dimensions × bytes per value (4 for float32) + HNSW graph (roughly vectors × m × links) + payload indexes + overhead
Example: 20 million × 1,536 dims × 4 bytes ≈ 123 GB for raw vectors alone. A corpus that doubled, a switch to a larger embedding model (768 → 3,072 dims) or a bulk re-index (old and new segments in memory at once) all push past the limit.
Step 1: stop the bleeding
- Pause ingestion (stop the indexing job or queue consumer). Bulk loads during recovery cause repeat OOMs.
- Raise the memory limit if nodes have room:
kubectl -n vectordb patch statefulset qdrant --type=json \
-p='[{"op":"replace","path":"/spec/template/spec/containers/0/resources/limits/memory","value":"96Gi"},
{"op":"replace","path":"/spec/template/spec/containers/0/resources/requests/memory","value":"96Gi"}]'
(Prefer changing the Helm values and redeploying, so the fix survives the next upgrade.) If nodes are too small, add a larger node pool and move the StatefulSet with a node selector.
- Let replicas come back and load data from disk. Watch logs until collections are reported green/ready.
Step 2: reduce memory per vector
| Option | Engine examples | Effect |
|---|---|---|
| Vectors on disk (memmap), keep index in RAM | Qdrant on_disk: true for vectors | Big RAM reduction; slower if disk is slow |
| Index on disk | Qdrant hnsw_config.on_disk, Milvus DiskANN | More RAM saving; latency cost |
| Scalar quantization (int8) | Qdrant, Milvus, Weaviate | ~4× smaller in RAM; small recall loss, rescoring helps |
| Binary / product quantization | Several engines | Larger saving; test recall carefully |
| Smaller embeddings | Models with reducible dimensions (e.g. Matryoshka-style) | Fewer dims; requires re-embedding |
Qdrant example (update collection config):
curl -X PATCH http://qdrant:6333/collections/kb \
-H "Content-Type: application/json" \
-d '{"vectors": {"": {"on_disk": true}},
"quantization_config": {"scalar": {"type": "int8", "always_ram": true}}}'
Field names and which settings can change on an existing collection depend on the version; check the docs and test on a copy first.
Step 3: ingest safely
- Batch size and parallelism limits on upserts.
- For full rebuilds, build into a new collection on separate capacity, then switch an alias. Don't rebuild in place on the serving cluster.
- Optimizer/indexing thresholds so the engine indexes in the background without huge memory peaks.
Step 4: size and monitor
- Track vector count and memory per pod; alert at 75% of the limit.
- Capacity plan: memory per million vectors for your dimension and settings, measured on your data. Write the number down.
- Set requests = limits for vector DB pods (guaranteed QoS), so the kernel doesn't pick them first under node pressure.
Warning
Raising memory limits without node capacity just moves the pod to Pending. Check node allocatable memory before patching.
Prevent it next time
- Growth forecast for the corpus and embedding dimensions.
- Quantization/on-disk settings chosen at design time, tested for recall.
- Blue/green collections for rebuilds, with aliases.
- Ingestion throttling and alerts on memory headroom.