Learning: Chroma advanced - server deployment, HNSW tuning and recall testing, batching, tenants and databases, backups, upgrades and monitoring

#About this session

Note

Chroma's single-node server is a good fit for small and medium workloads (thousands to a few million vectors per collection, moderate query rates). The final module covers how to tell when you have outgrown it.

#Module 1: from embedded to server

Any app with more than one worker, or more than one app, needs the server.

#Module 2: run the server with Docker

docker-compose.yml:

services:
  chroma:
    image: chromadb/chroma:1.0.0          # pin a version you have tested; check Docker Hub for current tags
    container_name: chroma
    ports:
      - "127.0.0.1:8000:8000"             # local only; expose through a proxy later
    volumes:
      - chroma-data:/data                 # 1.x stores data in /data (0.x used /chroma/chroma)
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "curl -sf http://localhost:8000/api/v2/heartbeat || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 3
volumes:
  chroma-data:

docker compose up -d
docker compose ps
curl -s http://127.0.0.1:8000/api/v2/heartbeat

If the image has no curl, change the health check to a Python one-liner or check from outside the container; confirm with docker inspect chroma --format '{{.State.Health.Status}}'.

Without Docker, the CLI runs the same server:

pip install chromadb
chroma run --path ./chroma-server-data --host 127.0.0.1 --port 8000

Connect from Python:

import chromadb
client = chromadb.HttpClient(host="127.0.0.1", port=8000)
print(client.heartbeat(), client.get_version())

Tip

Keep the client library version close to the server version. A big mismatch (for example a 0.5 client against a 1.x server) fails with confusing API errors.

#Module 3: HNSW parameters and what you can change later

Create a tuned collection:

col = client.get_or_create_collection(
    name="kb_tuned",
    configuration={"hnsw": {"space": "cosine", "max_neighbors": 32,
                            "ef_construction": 200, "ef_search": 128}},
)
print(col.configuration)   # confirm what was actually applied

Warning

In some versions, col.modify(configuration={"hnsw": {"ef_search": ...}}) is accepted but doesn't change search behaviour for already-built indexes. Always measure after a change (Module 4). If it didn't help, set the value at creation and rebuild. Details: Chroma returns poor matches.

#Module 4: measure recall instead of guessing

Recall@k = the share of the true k nearest neighbours that the index returns. Compute the truth with exact search in NumPy and compare.

import numpy as np, chromadb

client = chromadb.HttpClient(host="127.0.0.1", port=8000)
rng = np.random.default_rng(7)

# Lab data: replace with real embeddings from your model for meaningful numbers
N, D = 50_000, 384
base = rng.normal(size=(N, D)).astype("float32")
base /= np.linalg.norm(base, axis=1, keepdims=True)
ids = [f"v{i}" for i in range(N)]

def load(name, cfg):
    try:
        client.delete_collection(name)
    except Exception:
        pass
    col = client.create_collection(name, configuration={"hnsw": cfg})
    step = 5000
    for s in range(0, N, step):
        col.add(ids=ids[s:s + step], embeddings=base[s:s + step].tolist())
    return col

def recall(col, k=10, queries=200):
    q = base[rng.choice(N, queries, replace=False)] + rng.normal(scale=0.05, size=(queries, D)).astype("float32")
    q /= np.linalg.norm(q, axis=1, keepdims=True)
    truth = np.argsort(-(q @ base.T), axis=1)[:, :k]            # exact cosine top-k
    got = col.query(query_embeddings=q.tolist(), n_results=k)["ids"]
    hits = sum(len({f"v{j}" for j in t} & set(g)) for t, g in zip(truth, got))
    return hits / (queries * k)

for cfg in [{"space": "cosine"},
            {"space": "cosine", "max_neighbors": 32, "ef_construction": 200, "ef_search": 128}]:
    col = load("recall_test", cfg)
    print(cfg, "recall@10 =", round(recall(col), 3))

Random data is a hard case for HNSW (no structure), so recall looks worse than on real embeddings. Rerun with your own embeddings and a sample of real questions to get numbers you can trust. Record latency at the same time (time.perf_counter() around queries).

Rules of thumb from measurement, not guesses:

• Raise ef_search first (cheapest), then max_neighbors and ef_construction (rebuild needed).
• Recall around 0.95+ is a common target for RAG; check answer quality, not just recall.

#Module 5: load data in batches

Large add calls fail or time out. Chroma reports the largest batch it accepts:

max_b = client.get_max_batch_size()
print("max batch:", max_b)

def batched_upsert(col, ids, docs, metas, embs=None, size=None):
    size = min(size or 1000, max_b)
    for s in range(0, len(ids), size):
        kw = dict(ids=ids[s:s + size], documents=docs[s:s + size], metadatas=metas[s:s + size])
        if embs is not None:
            kw["embeddings"] = embs[s:s + size]
        col.upsert(**kw)

Production habits:

• Compute embeddings outside Chroma (in your pipeline, with retries and rate limits) and pass embeddings=. The server then doesn't run a model, and you control the model version.
• Use stable ids (source-id:chunk-no) and upsert, so re-runs don't duplicate.
• Delete chunks of removed or shortened documents (col.delete(where={"source": "x"}) before re-adding is the simplest pattern).
• Store source, updated_at and embedding_model in metadata for audits and freshness checks.

#Module 6: tenants and databases

Chroma has a three-level namespace: tenant → database → collection. Use it to separate teams or environments on one server.

import chromadb
from chromadb import AdminClient, HttpClient
from chromadb.config import Settings

admin = AdminClient(Settings(chroma_api_impl="chromadb.api.fastapi.FastAPI",
                             chroma_server_host="127.0.0.1", chroma_server_http_port=8000))
admin.create_tenant("team_ops")
admin.create_database("prod", tenant="team_ops")
admin.create_tenant("team_hr")
admin.create_database("prod", tenant="team_hr")

ops = HttpClient(host="127.0.0.1", port=8000, tenant="team_ops", database="prod")
hr = HttpClient(host="127.0.0.1", port=8000, tenant="team_hr", database="prod")
ops.get_or_create_collection("kb", configuration={"hnsw": {"space": "cosine"}})
print([c.name for c in hr.list_collections()])   # [] - HR can't see ops collections

The exact AdminClient construction differs between versions; check the client reference for yours.

Caution

Tenants and databases are namespaces, not security boundaries on their own. Any client that can reach the server and knows a tenant name can use it, unless authentication and authorization are enforced in front of Chroma. For sensitive data, prefer separate servers or enforce access in a proxy or your API layer.

#Module 7: secure the server

Minimum for anything beyond a laptop:

1. Never expose port 8000 to the internet. Bind to localhost or a private network.
2. Put a reverse proxy in front for TLS and authentication. Nginx example with a bearer token:

server {
    listen 443 ssl;
    server_name chroma.internal.example.com;
    ssl_certificate     /etc/ssl/chroma.crt;
    ssl_certificate_key /etc/ssl/chroma.key;

    location / {
        if ($http_authorization != "Bearer CHANGE_ME_LONG_RANDOM_TOKEN") { return 401; }
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        client_max_body_size 50m;
    }
}

Client:

client = chromadb.HttpClient(host="chroma.internal.example.com", port=443, ssl=True,
                             headers={"Authorization": "Bearer <token>"})

3. Chroma 0.5/0.6 had built-in token/basic auth (CHROMA_SERVER_AUTHN_PROVIDER, CHROMA_SERVER_AUTHN_CREDENTIALS). For 1.x, check your version's documentation; the proxy pattern works with any version.
4. Better still, let only your backend API talk to Chroma, and enforce per-user access (document-level filters) there.

#Module 8: back up, restore and upgrade

#Backup

The data lives in the volume (/data in 1.x images). For a consistent copy, stop writes briefly:

docker compose stop chroma
docker run --rm -v chroma_chroma-data:/data -v "$PWD":/backup alpine \
  tar czf /backup/chroma-$(date +%Y%m%d-%H%M).tgz -C /data .
docker compose start chroma

(The volume name gets the Compose project prefix; check with docker volume ls.) Cloud disk snapshots of a stopped server work too.

#Restore drill

docker compose down
docker volume rm chroma_chroma-data
docker volume create chroma_chroma-data
docker run --rm -v chroma_chroma-data:/data -v "$PWD":/backup alpine \
  tar xzf /backup/chroma-20261031-0200.tgz -C /data
docker compose up -d

Then check collection counts and run a few known queries. A backup is only proven when this drill works.

#Export-import: the version-independent backup

Because embeddings are stored, you can export everything through the API and re-load it into any version:

import json
col = client.get_collection("kb_tuned")
data = col.get(include=["documents", "metadatas", "embeddings"])
with open("kb_tuned.json", "w") as f:
    json.dump({"ids": data["ids"], "documents": data["documents"], "metadatas": data["metadatas"],
               "embeddings": [list(map(float, e)) for e in data["embeddings"]]}, f)

For large collections, page with limit and offset.

#Upgrades

• Pin versions (image tag and chromadb client).
• Read release notes for storage migrations; major versions have changed the on-disk format.
• Back up, upgrade a copy first, run your queries and recall test, then upgrade production.

Full procedure: Upgrading Chroma without losing your collections.

#Module 9: monitor it

A small probe you can run from cron or your monitoring system:

import time, chromadb, sys
client = chromadb.HttpClient(host="127.0.0.1", port=8000)
try:
    t0 = time.perf_counter()
    client.heartbeat()
    col = client.get_collection("kb_tuned")
    n = col.count()
    probe = col.get(limit=1, include=["embeddings"])["embeddings"][0]   # a real stored vector
    col.query(query_embeddings=[list(map(float, probe))], n_results=5)
    ms = (time.perf_counter() - t0) * 1000
    print(f"ok count={n} probe_ms={ms:.0f}")
    sys.exit(0 if ms < 1000 else 1)
except Exception as e:
    print("FAIL", e); sys.exit(2)

Also watch:

#Module 10: limits and when to move on

Consider a distributed vector database (for example Milvus, or a managed service) when you need:

• High availability with automatic failover (single-node Chroma has none; plan restores instead).
• Many millions of vectors with memory beyond one host.
• Very high query rates or strict multi-tenant security.

Until then, a well-run single Chroma server with backups is simple and effective.

#Exercises

1. Deploy the Compose file, add a proxy with TLS and a token, and connect from Python through the proxy.
2. Run the recall test with your real embeddings for three configurations. Record recall@10 and p95 latency.
3. Write a batch loader that reads a folder of text files, chunks them, embeds them outside Chroma and upserts with stable ids. Run it twice and confirm the count doesn't double.
4. Create two tenants and prove that a client for one can't list the other's collections. Then explain why that alone isn't security.
5. Do a full backup and restore drill and time it.
6. Export a collection to JSON and import it into a different Chroma version.

#Knowledge check

1. Why must multi-worker apps use the Chroma server instead of PersistentClient?
2. Which HNSW settings can't be changed after a collection is created?
3. How do you measure recall@k, and why not trust the default settings?
4. Why compute embeddings outside Chroma in production?
5. Are tenants and databases a security boundary? What do you add?
6. What are two ways to back up Chroma, and which survives version changes?
7. Name three signals you would monitor.
8. When would you move from Chroma to a distributed vector database?

#Summary

• Run Chroma as a server for shared use; pin versions, health-check it and keep it off the public network.
• Choose HNSW settings at creation and prove them with a recall test on real data.
• Load in batches with external embeddings, stable ids and upserts.
• Tenants and databases organise data; security needs a proxy or API layer.
• Back up, practise restores and upgrades on copies, and monitor what users feel.

#Further reading

• Learning: Chroma for beginners
• Moving Chroma to a server: Docker, persistence, access control and health checks
• Chroma "database is locked" with several workers
• Upgrading Chroma without losing your collections
• Learning: Milvus advanced
• Chroma documentation: docs.trychroma.com