Chroma returns poor matches: default L2 distance, HNSW settings fixed at creation, and rebuilding a collection correctly

#Symptoms

• Questions whose answer is clearly in the corpus retrieve unrelated chunks.
• distances in query results are large numbers (e.g. 0.8–1.6) where you expected cosine similarities near 1.
• Recall drops as the collection grows past a few hundred thousand items.
• You changed ef_search (or hnsw:search_ef) on an existing collection and results didn't change.

#Cause 1: the collection uses L2, not cosine

Chroma's default distance space is L2 (squared Euclidean). Many people assume cosine. For normalised embeddings, L2 and cosine give the same ranking, so results look fine. For embeddings that aren't normalised, rankings differ, and thresholds written for cosine are wrong.

Check what the collection uses:

import chromadb
client = chromadb.PersistentClient(path="./chroma")    # or HttpClient(host=..., port=8000)
col = client.get_collection("kb_chunks")
print(col.metadata)                                    # 0.x: {"hnsw:space": "cosine"} if set
print(getattr(col, "configuration", None))             # 1.x: configuration with hnsw/spann settings

Using the wrong conversion in the application (e.g. treating L2 distances as cosine distances and filtering distance < 0.3) throws good results away.

#Cause 2: HNSW parameters set at creation, low defaults

HNSW parameters in Chroma are set when the collection is created:

space, max_neighbors and ef_construction can't be changed afterwards. Chroma's docs describe ef_search as modifiable, but in practice we've seen versions where modifying it on an existing collection didn't change query results. Don't rely on it: test whether a change takes effect on your version (Step 4), and if not, rebuild.

#Step 1: measure recall, don't guess

Compare Chroma's results with an exact search done in NumPy on a sample:

import numpy as np

data = col.get(include=["embeddings"], limit=20000)          # sample; for big sets page with offset
ids, embs = data["ids"], np.array(data["embeddings"], dtype=np.float32)
embs_n = embs / np.linalg.norm(embs, axis=1, keepdims=True)

def recall_at_k(queries, k=10):
    total = 0.0
    for q in queries:
        qn = q / np.linalg.norm(q)
        exact = set(np.array(ids)[np.argsort(-embs_n @ qn)[:k]])
        got = set(col.query(query_embeddings=[q.tolist()], n_results=k)["ids"][0])
        total += len(exact & got) / k
    return total / len(queries)

Run the comparison on a collection holding only the sampled items, or compare against full exact search, so both sides search the same set. Use 100+ real query embeddings. Recall below ~0.9 at k=10 is a problem for RAG.

#Step 2: rebuild into a correctly configured collection

Chroma 1.x:

new = client.create_collection(
    name="kb_chunks_v2",
    configuration={
        "hnsw": {
            "space": "cosine",
            "max_neighbors": 32,
            "ef_construction": 200,
            "ef_search": 128,
        }
    },
    embedding_function=None,             # we pass embeddings explicitly
)

Chroma 0.5/0.6 metadata style:

new = client.create_collection(
    name="kb_chunks_v2",
    metadata={"hnsw:space": "cosine", "hnsw:M": 32,
              "hnsw:construction_ef": 200, "hnsw:search_ef": 128},
)

Copy data in pages, reusing the stored embeddings so nothing is re-embedded:

old = client.get_collection("kb_chunks")
offset, page = 0, 1000
while True:
    batch = old.get(include=["embeddings", "documents", "metadatas"], limit=page, offset=offset)
    if not batch["ids"]:
        break
    new.add(ids=batch["ids"], embeddings=batch["embeddings"],
            documents=batch["documents"], metadatas=batch["metadatas"])
    offset += page
print(old.count(), new.count())

Then point the application at kb_chunks_v2 (keep the collection name in configuration), re-run the recall check and delete the old collection after a few days.

#Step 3: check the embedding side too

The second one is common: data ingested with OpenAI embeddings, queries sent as query_texts, embedded by Chroma's default local model. Dimensions may even match by coincidence for some models; results are then garbage without any error.

#Step 4: test whether a query-time change applies

before = recall_at_k(sample_queries)
col.modify(configuration={"hnsw": {"ef_search": 256}})      # 1.x syntax
col = client.get_collection("kb_chunks")                    # re-open
after = recall_at_k(sample_queries)
print(before, after)

If after doesn't change, treat ef_search as fixed for your version and rebuild with the value you need.

#Decision table

#Prevent it next time

• Always set space explicitly when creating a collection.
• Store the embedding model name in collection metadata and check it at startup.
• Run a recall check in CI on a sample collection.
• Keep collection names in config so rebuild-and-switch is a config change.