How Milvus depends on other services
| Dependency | Holds | If it's down |
|---|---|---|
| etcd | Metadata: collections, segments, coordinator state | Milvus can't start; running nodes lose sessions and restart |
| Object storage (MinIO, S3, Azure Blob, GCS) | Segment data, index files, logs | Can't flush, load or build indexes; start fails on bucket checks |
| Message queue / WAL (Pulsar, Kafka, RocksMQ in standalone, Woodpecker in newer versions) | Write-ahead log of inserts and deletes | Inserts fail; nodes can't subscribe to channels |
Milvus waits for these at start and exits if it can't reach them in time, so the Milvus pod is where you see the error, but rarely where the cause is.
Step 1: what is failing?
Kubernetes:
kubectl get pods -n milvus -o wide
kubectl describe pod -n milvus <milvus-pod> | sed -n '/Events/,$p'
kubectl logs -n milvus <milvus-pod> --previous | grep -iE "error|fail|etcd|minio|pulsar|kafka|timeout" | tail -40
Docker Compose standalone:
docker compose ps
docker logs milvus-standalone 2>&1 | grep -iE "error|fail|etcd|minio" | tail -40
curl -s http://localhost:9091/healthz # "OK" when Milvus is healthy
Typical log lines and what they point at:
| Log text | Dependency |
|---|---|
failed to connect to etcd, context deadline exceeded with etcd endpoints | etcd |
etcdserver: mvcc: database space exceeded | etcd disk quota |
bucket does not exist, Access Denied, SignatureDoesNotMatch | Object storage |
pulsar ... connection refused, Kafka ... broker not available | Message queue |
find no available rootcoord / datacoord | Coordinator not up yet; usually a dependency underneath |
Step 2: etcd
kubectl get pods -n milvus -l app.kubernetes.io/name=etcd
kubectl exec -n milvus <etcd-pod> -- etcdctl endpoint health --cluster
kubectl exec -n milvus <etcd-pod> -- etcdctl endpoint status --cluster -w table
| Problem | Fix |
|---|---|
| etcd pod Pending | PVC can't bind (StorageClass, zone); fix storage |
database space exceeded | Compact and defragment, then clear the alarm (below); set auto-compaction |
Slow disk (apply request took too long warnings) | Put etcd on SSD-backed storage; etcd is very sensitive to fsync latency |
| Quorum lost (2 of 3 members down) | Restore members; never scale a 3-node etcd to 2 |
Recover from a full etcd quota:
REV=$(etcdctl endpoint status -w json | grep -o '"revision":[0-9]*' | head -1 | cut -d: -f2)
etcdctl compaction "$REV"
etcdctl defrag --cluster
etcdctl alarm disarm
The Milvus Helm chart sets etcd auto-compaction; check that your values haven't removed it.
Step 3: object storage
kubectl get pods -n milvus -l app=minio
kubectl logs -n milvus <minio-pod> | tail -20
Check the Milvus config that points at it (minio.address, minio.bucketName, minio.rootPath, minio.useSSL, credentials):
kubectl get configmap -n milvus <release>-milvus -o yaml | grep -A12 "minio:"
| Problem | Fix |
|---|---|
| Bucket missing | Create it, or let Milvus create it (needs permission) |
| Access denied on S3/Blob | IAM role / workload identity / key lacks list, get, put, delete on the bucket |
| MinIO disk full | Expand PVC; check for old segment files after drops (garbage collection) |
| TLS mismatch | useSSL true/false must match the endpoint |
Using cloud storage with useIAM | Pod identity configured and the service account annotated |
Step 4: message queue
Pulsar (default for distributed in many versions):
kubectl get pods -n milvus | grep -E "pulsar|bookie|zookeeper|broker"
kubectl logs -n milvus <pulsar-broker-pod> | tail -30
Pulsar has its own dependencies (ZooKeeper, BookKeeper). A full BookKeeper disk or a crashed ZooKeeper takes Milvus writes down with it. Check bookie PVC usage first.
Kafka: check broker health and that kafka.brokerList in the Milvus config is reachable from the Milvus pods.
Note
Newer Milvus releases add Woodpecker as a built-in write-ahead log option, which removes the separate Pulsar/Kafka cluster for many deployments. If you're designing a new install, check which WAL options your target version supports.
Step 5: Milvus itself
When all dependencies are healthy:
| Symptom | Cause | Fix |
|---|---|---|
| OOMKilled querynode | Loaded collections exceed memory | Release collections, add query nodes, use mmap or DiskANN |
| Version mismatch errors after upgrade | Mixed component versions | Finish the rolling upgrade; follow the version's upgrade path |
| Port 19530 refused but pod running | Still starting; liveness too aggressive | Give startup more time; check readiness |
| Proxy can't reach coordinators | NetworkPolicy, DNS | Test service DNS from the proxy pod |
Step 6: after recovery
from pymilvus import MilvusClient
client = MilvusClient(uri="http://milvus.example.internal:19530", token="root:<password>")
print(client.list_collections())
print(client.get_load_state(collection_name="kb_chunks"))
Collections that were loaded before may need loading again (see the separate post on "collection not loaded" errors).
Decision table
| First failing component | Look at |
|---|---|
| etcd unhealthy | Disk speed, quota, quorum |
| MinIO / S3 errors | Bucket, credentials, TLS, disk |
| Pulsar / Kafka errors | Bookie/ZooKeeper disks, broker list |
| All healthy, Milvus fails | Memory, versions, networking |
Prevent it next time
- Put etcd on fast SSD storage and alert on its DB size and fsync latency.
- Monitor object storage capacity and Pulsar bookie disks, not only Milvus pods.
- Keep dependency endpoints and credentials in one reviewed Helm values file.
- Test a full cluster restart in staging after every upgrade.