Learning: Milvus advanced - explicit schemas, HNSW tuning and recall, partition keys for multi-tenancy, hybrid BM25 search, security, Kubernetes, monitoring and backups

#About this session

#Module 1: architecture you need to operate it

Data path in one paragraph: inserts go through the proxy into the write-ahead log, land in growing segments, are flushed as sealed segments to object storage, indexes are built per sealed segment, and query nodes load sealed segments (with indexes) plus growing data into memory to serve searches.

Operational consequences:

• If etcd, object storage or the WAL is unhealthy, Milvus is unhealthy. See Milvus won't start: etcd, MinIO and message queue.
• Search capacity = query node memory and CPU. Ingest capacity = data/streaming nodes and WAL.
• Component names and how many processes exist differ between 2.4, 2.5 and 2.6; check your version's architecture page.

#Module 2: an explicit schema

Dynamic fields are handy, but production collections should define fields, types and lengths:

from pymilvus import MilvusClient, DataType

client = MilvusClient(uri="http://localhost:19530", token="root:Milvus")

schema = MilvusClient.create_schema(auto_id=False, enable_dynamic_field=False)
schema.add_field("chunk_id", DataType.VARCHAR, is_primary=True, max_length=128)
schema.add_field("tenant", DataType.VARCHAR, max_length=64, is_partition_key=True)
schema.add_field("source", DataType.VARCHAR, max_length=512)
schema.add_field("category", DataType.VARCHAR, max_length=64)
schema.add_field("updated_at", DataType.INT64)
schema.add_field("text", DataType.VARCHAR, max_length=8192, enable_analyzer=True)   # for BM25 (Module 6)
schema.add_field("dense", DataType.FLOAT_VECTOR, dim=384)
schema.add_field("sparse", DataType.SPARSE_FLOAT_VECTOR)                            # BM25 output

Design rules:

#Module 3: indexes, metrics and search parameters

idx = client.prepare_index_params()
idx.add_index(field_name="dense", index_type="HNSW", metric_type="COSINE",
              params={"M": 16, "efConstruction": 200})
idx.add_index(field_name="category", index_type="INVERTED")       # speeds up scalar filters
idx.add_index(field_name="updated_at", index_type="STL_SORT")     # numeric range filters
idx.add_index(field_name="sparse", index_type="SPARSE_INVERTED_INDEX", metric_type="BM25")

(Create the collection in Module 6 once the BM25 function is added to the schema.)

Search-time parameters per index:

client.search("kb", data=[qvec], anns_field="dense", limit=10,
              search_params={"metric_type": "COSINE", "params": {"ef": 64}})   # HNSW: ef >= limit
# IVF: {"params": {"nprobe": 16}}

Details for every index: Milvus index types cheat sheet.

#Module 4: measure recall

Use a FLAT (exact) collection as ground truth and compare:

import numpy as np, time
from pymilvus import MilvusClient, DataType

rng = np.random.default_rng(1)
N, D = 100_000, 384
X = rng.normal(size=(N, D)).astype("float32"); X /= np.linalg.norm(X, axis=1, keepdims=True)

def make(name, index_type, params):
    if client.has_collection(name):
        client.drop_collection(name)
    s = MilvusClient.create_schema(auto_id=False)
    s.add_field("id", DataType.INT64, is_primary=True)
    s.add_field("v", DataType.FLOAT_VECTOR, dim=D)
    ip = client.prepare_index_params()
    ip.add_index(field_name="v", index_type=index_type, metric_type="COSINE", params=params)
    client.create_collection(name, schema=s, index_params=ip)
    for i in range(0, N, 10_000):
        client.insert(name, [{"id": int(j), "v": X[j].tolist()} for j in range(i, min(i + 10_000, N))])
    client.flush(name)
    client.load_collection(name)

make("truth", "FLAT", {})
make("hnsw", "HNSW", {"M": 16, "efConstruction": 200})

Q = X[rng.choice(N, 200, replace=False)] + rng.normal(scale=0.05, size=(200, D)).astype("float32")
Q /= np.linalg.norm(Q, axis=1, keepdims=True)
truth = client.search("truth", data=Q.tolist(), limit=10, consistency_level="Strong")

for ef in (16, 64, 256):
    t0 = time.perf_counter()
    got = client.search("hnsw", data=Q.tolist(), limit=10, consistency_level="Strong",
                        search_params={"params": {"ef": ef}})
    ms = (time.perf_counter() - t0) * 1000 / len(Q)
    r = np.mean([len({h["id"] for h in t} & {h["id"] for h in g}) / 10 for t, g in zip(truth, got)])
    print(f"ef={ef:<4} recall@10={r:.3f} avg_ms/query={ms:.1f}")

Random vectors are a pessimistic case; repeat with your real embeddings and real questions before choosing parameters. Tune ef first (no rebuild), then M/efConstruction (rebuild).

#Module 5: consistency levels

client.search("kb", data=[qvec], limit=5, consistency_level="Bounded")

For RAG, Bounded is usually fine. Use Strong in tests that insert and immediately search, or you'll see "missing" rows that are just not visible yet.

#Module 6: hybrid search (dense + BM25)

Dense vectors capture meaning; BM25 catches exact terms like error codes (ORA-00257, 0x80070005). Milvus 2.5+ can compute BM25 sparse vectors from text with a built-in function.

from pymilvus import Function, FunctionType, AnnSearchRequest, RRFRanker

schema.add_function(Function(
    name="text_bm25", function_type=FunctionType.BM25,
    input_field_names=["text"], output_field_names=["sparse"],
))

client.create_collection("kb", schema=schema, index_params=idx)

Insert only the text and dense vector; Milvus fills sparse:

from sentence_transformers import SentenceTransformer
m = SentenceTransformer("all-MiniLM-L6-v2")

rows = [
  {"chunk_id": "ora-rb:0", "tenant": "ops", "source": "oracle-runbook", "category": "databases",
   "updated_at": 1761900000, "text": "ORA-00257 archiver error: archive destination full, free FRA space."},
  {"chunk_id": "pg-rb:0", "tenant": "ops", "source": "pg-runbook", "category": "databases",
   "updated_at": 1761900000, "text": "PostgreSQL WAL filling the disk because of an inactive replication slot."},
  {"chunk_id": "hr-1:0", "tenant": "hr", "source": "hr-policy", "category": "policy",
   "updated_at": 1761900000, "text": "Annual leave requests must be approved by your manager."},
]
for r in rows:
    r["dense"] = m.encode(r["text"], normalize_embeddings=True).tolist()
client.insert("kb", rows)

Hybrid query with Reciprocal Rank Fusion:

q = "ORA-00257 database hung"
dense_req = AnnSearchRequest(data=[m.encode(q, normalize_embeddings=True).tolist()], anns_field="dense",
                             param={"metric_type": "COSINE", "params": {"ef": 64}}, limit=10,
                             expr='tenant == "ops"')
sparse_req = AnnSearchRequest(data=[q], anns_field="sparse", param={"metric_type": "BM25"}, limit=10,
                              expr='tenant == "ops"')
res = client.hybrid_search("kb", reqs=[dense_req, sparse_req], ranker=RRFRanker(60), limit=5,
                           output_fields=["chunk_id", "text"])
for h in res[0]:
    print(round(h["distance"], 4), h["entity"]["chunk_id"])

WeightedRanker(0.7, 0.3) is the alternative when you want to weight one side. Check parameter names against your pymilvus version; hybrid APIs were refined across 2.4-2.6.

#Module 7: multi-tenancy with partition keys

With tenant as the partition key, Milvus hashes tenants into a fixed set of partitions and prunes them when the filter includes the tenant:

client.search("kb", data=[qvec], anns_field="dense", limit=5,
              filter='tenant == "ops"', output_fields=["text"])

Caution

With partition keys, a missing tenant filter returns other tenants' data. Enforce the filter in one server-side function that every query path uses, and test it.

#Module 8: authentication and RBAC

Turn on authentication in milvus.yaml (or Helm values):

common:
  security:
    authorizationEnabled: true

Restart, then change the root password and create least-privilege users:

admin = MilvusClient(uri="http://localhost:19530", token="root:Milvus")
admin.update_password(user_name="root", old_password="Milvus", new_password="<strong-password>")

admin = MilvusClient(uri="http://localhost:19530", token="root:<strong-password>")
admin.create_user(user_name="rag_app", password="<app-password>")
admin.create_role(role_name="kb_reader")
for p in ("Search", "Query", "Load", "DescribeCollection"):
    admin.grant_privilege(role_name="kb_reader", object_type="Collection", privilege=p, object_name="kb")
admin.grant_role(user_name="rag_app", role_name="kb_reader")

app = MilvusClient(uri="http://localhost:19530", token="rag_app:<app-password>")
app.search("kb", data=[qvec], anns_field="dense", limit=3)          # allowed
# app.drop_collection("kb")                                           # denied

Newer versions add privilege groups and grant_privilege_v2; the names above work across 2.4-2.6 but check your version. Also:

• Enable TLS on the proxy (tls settings in milvus.yaml) or terminate TLS at an ingress.
• Keep etcd, MinIO and the message queue on a private network with their own credentials changed from defaults.

#Module 9: deploy on Kubernetes with Helm

helm repo add zilliztech https://zilliztech.github.io/milvus-helm/
helm repo update
helm show values zilliztech/milvus > values-default.yaml

A starting values.yaml for a small distributed cluster (keys vary by chart version; compare with values-default.yaml):

cluster:
  enabled: true
queryNode:
  replicas: 2
  resources:
    requests: { cpu: "4", memory: 16Gi }
    limits:   { memory: 16Gi }
dataNode:
  replicas: 2
proxy:
  replicas: 2
etcd:
  replicaCount: 3
minio:
  mode: distributed          # or disable MinIO and point externalS3 at S3 / Azure Blob
metrics:
  serviceMonitor:
    enabled: true
extraConfigFiles:
  user.yaml: |+
    common:
      security:
        authorizationEnabled: true

helm install milvus zilliztech/milvus -n milvus --create-namespace -f values.yaml
kubectl get pods -n milvus

The Milvus Operator is the other supported route and handles upgrades and dependencies declaratively. For managed object storage, use the chart's external S3 settings and an IAM role or workload identity instead of static keys.

#Module 10: monitoring

Each component exposes Prometheus metrics on port 9091 (/metrics) and health on /healthz.

curl -s http://localhost:9091/healthz
curl -s http://localhost:9091/metrics | grep -E "^milvus_proxy_(req_count|sq_latency)" | head

Metric names change between versions; start from the Grafana dashboard JSON published in the Milvus repository for your version.

#Module 11: memory, compaction and data lifecycle

Memory sizing for HNSW, roughly: vectors × dim × 4 bytes × (1.5–2 for graph and overhead), plus scalar fields and growing data, per replica. Options when it doesn't fit: IVF_SQ8/PQ, DiskANN, mmap for fields or indexes (trades latency for memory), more query nodes.

Deletes and compaction: deletes write tombstones; space and speed recover only after compaction merges segments. Heavy update/delete workloads need compaction to keep up. See Milvus deletes don't free space.

client.compact("kb")   # manual compaction, for maintenance windows

Loading: load collections at startup, release unused ones, and use replica_number for read scaling and HA:

client.load_collection("kb", replica_number=2)

#Module 12: backups and upgrades

• Use milvus-backup to back up collections (metadata from etcd + data files in object storage) and restore with a suffix for verification. Runbook: Milvus backup and restore.
• Practise restore into a separate collection, then compare counts and run your recall/query tests.
• Upgrade one minor version at a time following the release notes; back up first; upgrade staging and run the same tests.

#Exercises

1. Build the kb collection with the explicit schema, BM25 function and indexes; load 1,000 real chunks.
2. Run the recall test with your real embeddings; choose ef for recall ≥ 0.95 and note p95 latency.
3. Compare dense-only, BM25-only and hybrid results for ten questions containing error codes.
4. Prove tenant isolation: write a test that fails if any search without a tenant filter is possible from your API layer.
5. Enable authentication, rotate the root password, create rag_app with read-only privileges, and show that drop fails.
6. Draft Helm values for your environment and list the dependencies you would run as managed services.
7. Back up kb with milvus-backup, restore with a suffix and verify.

#Knowledge check

1. Which three external dependencies does a distributed Milvus need, and what does each store?
2. Why use a VARCHAR business key as primary key in RAG collections?
3. Which HNSW parameter can you change at search time, and which need a rebuild?
4. When would you use Strong consistency?
5. What does BM25 add to dense vector search, and how are the two result lists combined?
6. What is the main risk of the partition-key multi-tenancy pattern?
7. Which setting enables authentication, and what is the first thing to do after enabling it?
8. Name three signals to monitor and why.
9. Why do deletes not immediately free space or restore speed?

#Summary

• Know the architecture: proxy, workers and three dependencies; most outages start in a dependency or in query node memory.
• Design explicit schemas with stable keys, the right index per field and a measured ef.
• Use partition keys for many tenants, but enforce tenant filters centrally.
• Hybrid dense + BM25 search fixes exact-term misses in RAG.
• Turn on auth and RBAC, deploy with Helm or the Operator, monitor, compact, back up and practise restores.

#Further reading

• Learning: Milvus for beginners
• Milvus index types cheat sheet
• Milvus won't start: etcd, MinIO and message queue
• Milvus backup and restore runbook
• Learning: Chroma advanced
• Milvus documentation: milvus.io/docs