Weaviate replication and consistency: replication factor, QUORUM vs ONE, repairs and what happens when a node goes down

#Symptoms

• During a rolling restart, imports fail with errors about not enough replicas or a consistency level that can't be met.
• The same query returns slightly different results on repeated calls.
• An object deleted a minute ago appears in some searches.
• After a node was down for hours, its data seems stale.

#Concepts

Weaviate uses leaderless replication for data: any replica can accept a write, coordinated by the node that received the request. Schema changes go through Raft consensus (1.25+).

#Step 1: check the collection's replication

kb = client.collections.get("KbChunk")
cfg = kb.config.get()
print(cfg.replication_config)

curl -s http://weaviate:8080/v1/nodes?output=verbose -H "Authorization: Bearer $KEY" | \
  jq '.nodes[] | {name, status, shards: [.shards[]? | {class, name, objectCount}]}'

If factor is 1, there is no redundancy: a node restart makes that node's shards unavailable. That's the first fix.

#Step 2: set replication

from weaviate.classes.config import Configure

client.collections.create(
    "KbChunk",
    replication_config=Configure.replication(factor=3, async_enabled=True),
    # ...
)

Changing the replication factor on an existing collection is possible in newer versions (with replica movement/scale operations); on older versions you create a new collection and re-import. Check the docs for your version before planning.

You need at least as many nodes as the replication factor.

#Step 3: choose consistency levels

Defaults are QUORUM for both reads and writes in the clients. Set per request or per collection handle:

from weaviate.classes.config import ConsistencyLevel

kb_q = kb.with_consistency_level(ConsistencyLevel.QUORUM)
kb_one = kb.with_consistency_level(ConsistencyLevel.ONE)

For RAG workloads, slightly stale reads are usually fine; failing imports during maintenance are not. Common choice: writes QUORUM, reads ONE for search, QUORUM for "read your own upload" flows.

#Step 4: node down - what you'll see

With RF=3 and one node down:

With RF=2 and one node down, QUORUM (which is 2 of 2) fails. RF=2 only gives availability with ONE. Use RF=3 for production.

#Step 5: inconsistent results

Replicas can diverge when a write succeeded on some replicas but not all (ONE or QUORUM writes during a node outage). Repairs:

• Read repair: a QUORUM/ALL read that sees different versions repairs the stale replica. ONE reads don't trigger it.
• Async replication (1.26+, async_enabled=True): compares replicas in the background (hash trees) and syncs differences, so a node that was down catches up without relying on reads.

If a node was down for a long time and you don't have async replication:

• Enable it on the collection if your version supports it.
• Or run a controlled full read at ALL over the affected data (heavy) to trigger repairs.
• Or rebuild that node's replicas.

Deletes: a delete on a node-down cluster leaves the stale replica holding the object until repair. Weaviate keeps deletion markers (tombstones) to resolve this; the deletion strategy (deletion_strategy in replication config: NoAutomatedResolution, DeleteOnConflict, TimeBasedResolution) decides how conflicts between a delete and a write are settled. Pick explicitly.

#Step 6: rolling restarts and upgrades

• Restart one pod at a time and wait for /v1/.well-known/ready and /v1/nodes to show the node HEALTHY before the next.
• Set a PodDisruptionBudget with maxUnavailable: 1.
• Pause large imports during upgrades, or make the importer retry with backoff.
• Spread pods across zones so a zone outage removes at most one replica per shard.

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector: { matchLabels: { app: weaviate } }

#Step 7: monitor

• /v1/nodes status per node (HEALTHY, UNHEALTHY, UNAVAILABLE) and object count per shard.
• Prometheus metrics (PROMETHEUS_MONITORING_ENABLED=true) for request latency, errors and async replication activity.
• Compare object counts between replicas of the same shard; a large gap means replicas are diverging.

#Decision table

#Prevent it next time

• RF=3 across three zones for production collections.
• Explicit consistency levels in code for imports and for reads.
• Async replication enabled where available.
• Rolling restart runbook with health checks between pods.