Home / AI & GenAI

AI & GenAIHow-toBeginner

RAG basics: how retrieval augmented generation works, with a minimal pipeline you can read in one sitting

The four steps of RAG (chunk, embed, retrieve, generate) explained with a small Python example over your own runbooks, plus the prompt that keeps answers grounded, how to show sources, and the first checks when answers are wrong.

Deepak Bhardwaj4 min read

Applies toAny LLM and embedding API (examples use Azure OpenAI with the openai SDK 1.x); concepts apply to Bedrock Knowledge Bases, Azure AI Search, LangChain and LlamaIndex

An LLM knows what was in its training data, not your runbooks, tickets or policies. Retrieval augmented generation (RAG) fixes that by looking up relevant passages from your documents and putting them into the prompt, so the model answers from them.

The four steps

Indexing (once, then on change)       Answering (every question)
1. Split documents into chunks        3. Embed the question, retrieve top-k chunks
2. Embed chunks, store vectors        4. Send question + chunks to the LLM, return answer + sources

Step 1: chunk

import pathlib, re

def chunk(text, size=800, overlap=150):
    words = text.split()
    step = size - overlap
    return [" ".join(words[i:i + size]) for i in range(0, max(len(words) - overlap, 1), step)]

chunks = []
for path in pathlib.Path("runbooks").glob("*.md"):
    text = path.read_text(encoding="utf-8")
    for n, c in enumerate(chunk(text, size=250, overlap=40)):
        chunks.append({"id": f"{path.stem}#{n}", "source": path.name, "text": c})
print(len(chunks), "chunks")

Chunks of a few hundred words with some overlap are a reasonable start. Splitting on headings (##) usually works better than fixed sizes for Markdown runbooks.

Step 2: embed and store

import numpy as np
from openai import AzureOpenAI
client = AzureOpenAI(azure_endpoint="https://my-aoai.openai.azure.com", api_key=KEY, api_version="2024-10-21")

def embed(texts):
    resp = client.embeddings.create(model="text-embedding-3-small-prod", input=texts)
    v = np.array([d.embedding for d in resp.data])
    return v / np.linalg.norm(v, axis=1, keepdims=True)

vecs = np.vstack([embed([c["text"] for c in chunks[i:i + 64]]) for i in range(0, len(chunks), 64)])

For a few thousand chunks a NumPy array is enough. Production systems use a vector database or search service.

Step 3: retrieve

def retrieve(question, k=4):
    q = embed([question])[0]
    scores = vecs @ q
    top = np.argsort(-scores)[:k]
    return [dict(chunks[i], score=float(scores[i])) for i in top]

Step 4: generate a grounded answer

The prompt is what keeps the model honest:

SYSTEM = """You answer questions for IT support engineers using ONLY the provided context.
If the context does not contain the answer, say "I don't know based on the runbooks" and stop.
Cite sources as [source] after each statement that uses them. Be concise and give commands exactly as written."""

def answer(question):
    hits = retrieve(question)
    context = "\n\n".join(f"[{h['source']}]\n{h['text']}" for h in hits)
    r = client.chat.completions.create(
        model="gpt-4o-mini-prod",
        messages=[{"role": "system", "content": SYSTEM},
                  {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}],
        temperature=0.1, max_tokens=500)
    return r.choices[0].message.content, [h["source"] for h in hits]

text, sources = answer("How do I free space when the root disk on a Linux VM is full?")
print(text); print("Sources:", sources)

Tip

Always show the sources to the user. It lets them check the answer, and it makes wrong retrievals obvious during testing.

When answers are wrong: check retrieval first

Most bad RAG answers are retrieval problems, not model problems.

for h in retrieve("root disk full linux"):
    print(round(h["score"], 3), h["id"], h["text"][:100])
Symptom Likely cause First fix
Right document exists, not retrieved Chunk too big or too small; question uses different terms Re-chunk by headings; add hybrid keyword search
Retrieved but answer ignores it Too many irrelevant chunks; weak prompt Lower k, add a reranker, tighten the system prompt
Confident wrong answer No relevant chunk; model fills the gap "I don't know" instruction; minimum score threshold
Outdated answer Index not refreshed Re-index changed documents on a schedule
Exact error codes not found Pure vector search misses exact tokens Hybrid search (BM25 + vector)

Keep it safe

  • Permissions: only retrieve documents the asking user may read. Filter by metadata (group, tenant) at retrieval time.
  • Sensitive data: do not index secrets, passwords or personal data that should not appear in answers.
  • Prompt injection: retrieved text can contain instructions ("ignore previous instructions"). Treat context as data; the system prompt should say so, and tools the model can call should be limited.

Measure before you tune

Collect 30-50 real questions with expected answers or expected source documents. After any change (chunking, model, k), run them and compare:

  • Did the right source appear in the top-k? (retrieval hit rate)
  • Is the answer correct and supported by the sources? (manual review or LLM-as-judge)

Prevent it next time

  • Store source, section, last-updated date and access group with every chunk.
  • Re-index on document change, not by hand.
  • Keep a fixed evaluation set and run it on every change.

Related fixes