Sev1: OpenSearch k-NN cluster is red and vector search is down — shard allocation and k-NN memory circuit breaker

#Symptoms

• Vector queries fail with errors about unavailable shards, or k-NN searches return errors mentioning the circuit breaker.
• Cluster health red (primary shards missing) or yellow with high latency.
• Started after a node loss, a big bulk load or an index with more vectors than planned.

#1. Cluster health and unassigned shards

curl -s -u admin:$PW https://os-prod:9200/_cluster/health?pretty
curl -s -u admin:$PW "https://os-prod:9200/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason,node" | grep -v STARTED
curl -s -u admin:$PW https://os-prod:9200/_cluster/allocation/explain?pretty

Retry failed allocations after fixing the cause:

curl -s -u admin:$PW -X POST "https://os-prod:9200/_cluster/reroute?retry_failed=true&pretty"

#2. k-NN native memory and the circuit breaker

HNSW graphs for faiss/nmslib live in native memory outside the JVM heap. When their total exceeds the limit, the k-NN circuit breaker trips and searches fail.

curl -s -u admin:$PW "https://os-prod:9200/_plugins/_knn/stats?pretty" \
  | grep -E "circuit_breaker_triggered|graph_memory_usage|graph_memory_usage_percentage|cache_capacity_reached|eviction_count"
curl -s -u admin:$PW "https://os-prod:9200/_cluster/settings?include_defaults=true&pretty" | grep -i "knn.memory"

circuit_breaker_triggered: true with high graph memory usage confirms it.

Immediate options:

• Clear graph cache for indexes not needed right now (old index versions, test indexes):

curl -s -u admin:$PW -X POST "https://os-prod:9200/_plugins/_knn/clear_cache/rag-docs-v1?pretty"

• Close or delete superseded indexes after a re-index (a common cause: v1 and v2 both loaded).
• Raise the limit carefully if the nodes have free RAM outside the heap:

curl -s -u admin:$PW -X PUT https://os-prod:9200/_cluster/settings -H 'Content-Type: application/json' \
  -d '{"persistent":{"knn.memory.circuit_breaker.limit":"60%"}}'

Warning

The limit is a share of memory outside the JVM heap. Raising it on nodes that are already short of RAM leads to OS-level OOM kills, which is worse. Check node memory first (_nodes/stats/os).

#3. Reduce memory per vector

Long-term options, depending on version and engine:

• Fewer dimensions (an embedding model with smaller output, or one that supports shortening).
• Quantization (faiss scalar/product quantization, or the disk-based vector mode in newer releases) to cut memory per vector.
• Lower HNSW m for new indexes.
• More data nodes, or memory-optimised instance types.

Check the k-NN documentation for your OpenSearch version; available options change between releases.

#4. Warm up after recovery

After restarts, the first query on each segment loads graphs from disk, so latency is high. Pre-load the active index:

curl -s -u admin:$PW "https://os-prod:9200/_plugins/_knn/warmup/rag-docs-v2?pretty"

#5. Protect the RAG app meanwhile

• Fall back to keyword (BM25) search on the same index if available; it doesn't need graph memory.
• Show a "search is degraded" message rather than letting the LLM answer without context.

#Prevent it next time

• Plan native memory per node: vectors × dimensions × bytes per value × HNSW overhead, with headroom for re-indexing (two versions at once).
• Alert on graph_memory_usage_percentage, circuit breaker triggers, cluster status and disk watermarks.
• At least one replica for production vector indexes.
• Delete or close old index versions as part of the re-index runbook.