Learning: Chroma vector database for beginners - embeddings, collections, similarity search, metadata filters and a mini RAG lookup

#About this session

By the end you will be able to store text in Chroma and find the most relevant pieces for a question, which is the retrieval half of every RAG (retrieval-augmented generation) application. The advanced session takes this to a server deployment with tuning, multi-tenancy, backups and monitoring.

#Module 1: the ideas you need first

#Why a vector database?

A normal database finds rows that match exactly: WHERE title LIKE '%disk full%'. That fails when the user asks "my server ran out of space" and the document says "volume at 100%". The words differ, the meaning is the same.

A vector database finds rows that are similar in meaning. It does this with embeddings.

#Embeddings

An embedding model turns a piece of text into a list of numbers (a vector), for example 384 numbers. Texts with similar meaning get vectors that are close together.

The first two are close; the third is far away.

#Distance

"Close" is measured with a distance function:

Smaller distance = more similar. You choose the metric when you create a collection, and you can't change it later (see Module 7).

#Chroma's building blocks

#Approximate search

To be fast, Chroma uses an HNSW index (a graph of neighbours). It returns the approximately nearest items very quickly instead of comparing your query with every item. For learning-sized data the results are effectively exact.

#Module 2: install Chroma

Create a virtual environment and install the client:

python -m venv .venv
# Windows
.venv\Scripts\activate
# Linux / macOS
source .venv/bin/activate

pip install chromadb
python -c "import chromadb; print(chromadb.__version__)"

Note

The first time you add documents without your own embedding function, Chroma downloads a small default model (all-MiniLM-L6-v2, about 80 MB) and runs it locally with ONNX Runtime. You need internet access once. No API key is required.

#Module 3: your first collection

Create lab1.py:

import chromadb

client = chromadb.PersistentClient(path="./chroma-lab")   # data saved in this folder
print(client.heartbeat())                                 # a number = Chroma is alive

col = client.get_or_create_collection(
    name="runbooks",
    configuration={"hnsw": {"space": "cosine"}},          # 1.x syntax
)
# Chroma 0.5/0.6: metadata={"hnsw:space": "cosine"} instead of configuration

print(col.name, col.count())

python lab1.py

You now have a folder chroma-lab containing a SQLite file and index files. That folder is the database.

#Module 4: add documents

Create lab2_add.py:

import chromadb

client = chromadb.PersistentClient(path="./chroma-lab")
col = client.get_or_create_collection("runbooks", configuration={"hnsw": {"space": "cosine"}})

col.add(
    ids=["rb1", "rb2", "rb3", "rb4", "rb5", "rb6"],
    documents=[
        "SQL Server transaction log is full, error 9002. Take a log backup and check log_reuse_wait_desc.",
        "PostgreSQL refuses connections: too many clients already. Use a pooler such as PgBouncer.",
        "Linux root filesystem at 100 percent. Find large files with du and clear old logs.",
        "Azure VM will not boot after a kernel update. Use the serial console and the repair VM.",
        "Kubernetes pods stuck in ImagePullBackOff because the registry credentials expired.",
        "Reset a forgotten Windows local administrator password with the VM access extension.",
    ],
    metadatas=[
        {"category": "databases", "severity": 1, "engine": "sql-server"},
        {"category": "databases", "severity": 2, "engine": "postgresql"},
        {"category": "linux", "severity": 1},
        {"category": "azure", "severity": 1},
        {"category": "kubernetes", "severity": 2},
        {"category": "windows", "severity": 3},
    ],
)
print("items:", col.count())

Rules worth knowing:

• ids must be unique strings. Adding an existing id is ignored or raises an error depending on the version; use upsert (Module 6) when you mean "insert or replace".
• Metadata values must be simple types: string, integer, float or boolean. Lists are supported in recent 1.x versions; check your version before relying on them.
• If you pass documents only, Chroma creates the embeddings. You can also pass your own embeddings=[[...], ...].

#Module 5: search by meaning

Create lab3_query.py:

import chromadb

client = chromadb.PersistentClient(path="./chroma-lab")
col = client.get_collection("runbooks")

res = col.query(
    query_texts=["my disk ran out of space"],
    n_results=3,
    include=["documents", "metadatas", "distances"],
)

for doc, meta, dist in zip(res["documents"][0], res["metadatas"][0], res["distances"][0]):
    print(f"{dist:.3f}  {meta['category']:<11} {doc[:70]}")

Expected output (numbers will differ):

0.38  linux       Linux root filesystem at 100 percent. Find large files with du an
0.62  databases   SQL Server transaction log is full, error 9002. Take a log backup
0.71  azure       Azure VM will not boot after a kernel update. Use the serial cons

The query never mentions "filesystem" or "100 percent", yet the right runbook comes first. That is semantic search.

The result is a dictionary of lists of lists: one inner list per query text. You can send several questions at once:

res = col.query(query_texts=["database won't accept connections", "forgot admin password"], n_results=1)
print(res["ids"])   # [['rb2'], ['rb6']]

#Module 6: filters, get, update and delete

#Metadata filters with where

res = col.query(
    query_texts=["database problem"],
    n_results=3,
    where={"category": "databases"},
)

res = col.query(
    query_texts=["urgent outage"],
    n_results=5,
    where={"$and": [{"severity": {"$lte": 1}}, {"category": {"$in": ["linux", "azure"]}}]},
)

#Text filters with where_document

res = col.query(query_texts=["fix it"], n_results=3, where_document={"$contains": "PgBouncer"})

#Get items without searching

print(col.get(ids=["rb1"]))
print(col.get(where={"severity": 1}, include=["metadatas"]))
print(col.peek(limit=2))

#Update, upsert and delete

col.update(ids=["rb5"], metadatas=[{"category": "kubernetes", "severity": 1}])

col.upsert(
    ids=["rb7"],
    documents=["Redis OOM command not allowed: maxmemory reached. Check big keys and eviction policy."],
    metadatas=[{"category": "databases", "severity": 2, "engine": "redis"}],
)

col.delete(ids=["rb6"])
col.delete(where={"category": "windows"})
print(col.count())

When you update a document, Chroma re-computes its embedding automatically.

#Module 7: choose the embedding function and distance deliberately

Two settings decide the quality of your search, and both are fixed per collection.

Distance: set at creation. To change it, create a new collection and copy the data:

old = client.get_collection("runbooks")
data = old.get(include=["documents", "metadatas", "embeddings"])
new = client.create_collection("runbooks_l2", configuration={"hnsw": {"space": "l2"}})
new.add(ids=data["ids"], documents=data["documents"], metadatas=data["metadatas"], embeddings=data["embeddings"])

Embedding function: the same model must be used for adding and querying. Mixing models gives meaningless distances. To use a different local model:

from chromadb.utils import embedding_functions

ef = embedding_functions.SentenceTransformerEmbeddingFunction(model_name="all-mpnet-base-v2")
col2 = client.get_or_create_collection("runbooks_mpnet", embedding_function=ef,
                                       configuration={"hnsw": {"space": "cosine"}})

(Requires pip install sentence-transformers.) Hosted options such as OpenAIEmbeddingFunction exist too; they need an API key and send your text to that provider.

Warning

If you open an existing collection later with a different embedding_function, recent versions raise an error or warn; older versions silently produce bad results. Always create and open a collection with the same embedding function.

#Module 8: a mini RAG retrieval step

RAG = find relevant text, then give it to an LLM with the question. Here is the retrieval part, ready to plug into any LLM call:

import chromadb

client = chromadb.PersistentClient(path="./chroma-lab")
col = client.get_collection("runbooks")

def retrieve(question: str, k: int = 3, max_distance: float = 0.7, category: str | None = None):
    where = {"category": category} if category else None
    res = col.query(query_texts=[question], n_results=k, where=where,
                    include=["documents", "metadatas", "distances"])
    hits = []
    for doc, meta, dist in zip(res["documents"][0], res["metadatas"][0], res["distances"][0]):
        if dist <= max_distance:            # drop weak matches
            hits.append({"text": doc, "meta": meta, "distance": round(dist, 3)})
    return hits

def build_prompt(question: str) -> str:
    hits = retrieve(question)
    if not hits:
        return f"Say you don't have a runbook for this. Question: {question}"
    context = "\n".join(f"- {h['text']}" for h in hits)
    return f"Answer using only these runbook notes:\n{context}\n\nQuestion: {question}"

print(build_prompt("postgres says too many clients"))

The max_distance cut-off is important: without it, the assistant always gets some context, even when nothing relevant exists, which encourages made-up answers. Pick the value by looking at distances for good and bad matches on your own data.

#Chunking real documents

Real runbooks are long. Split them into chunks of a few hundred words with a small overlap, and store the source in metadata:

def chunk(text, size=800, overlap=100):
    out, start = [], 0
    while start < len(text):
        out.append(text[start:start + size])
        start += size - overlap
    return out

doc = open("sql-server-runbook.txt", encoding="utf-8").read()
parts = chunk(doc)
col.upsert(
    ids=[f"sqlrb-{i}" for i in range(len(parts))],
    documents=parts,
    metadatas=[{"category": "databases", "source": "sql-server-runbook.txt", "chunk": i} for i in range(len(parts))],
)

Stable ids (source + chunk number) mean re-running the load replaces chunks instead of duplicating them.

#Module 9: housekeeping

print(client.list_collections())
client.delete_collection("runbooks_l2")

To back up a learning database, stop your script and copy the chroma-lab folder. To start again, delete the folder.

Caution

PersistentClient is meant for one process. Two scripts or app workers writing to the same folder can cause database is locked errors or lost writes. For anything shared, run Chroma as a server (advanced session). See Chroma "database is locked" with several workers.

#Exercises

1. Add ten runbook snippets from your own job, with category, severity and owner metadata.
2. Write five realistic questions. For each, record the top result and its distance. Which ones failed, and why?
3. Find a distance threshold that keeps good matches and drops irrelevant ones for your data.
4. Create the same collection with l2 and with cosine. Do the rankings change?
5. Switch to all-mpnet-base-v2 in a new collection and compare results on your five questions.
6. Write a function that returns only severity-1 runbooks for a question, and test it.

#Knowledge check

1. What does an embedding model produce, and why are similar texts close together?
2. Which distance metric is Chroma's default, and which is usual for text embeddings?
3. Can you change a collection's distance metric after creation? What do you do instead?
4. What is the difference between add, update and upsert?
5. What is the difference between where and where_document?
6. Why must the same embedding function be used for adding and querying?
7. Why add a distance threshold to a RAG retrieval function?
8. Why shouldn't several app workers share one PersistentClient folder?

#Summary

• Vector databases find text by meaning using embeddings and a distance metric.
• A Chroma collection fixes its embedding function and distance at creation; choose cosine for text.
• add, query, get, update, upsert and delete cover daily work; where and where_document narrow results.
• A RAG retrieval step is a filtered query with a distance cut-off, over well-chunked documents with stable ids.
• PersistentClient is for single-process learning and small apps; servers come next.

#Further reading

• Chroma cheat sheet: clients, collections, filters and housekeeping
• Chroma returns poor matches: distance and HNSW settings fixed at creation
• Learning: Chroma advanced
• Chroma documentation: docs.trychroma.com