Symptoms
| Symptom | Likely chunking problem |
|---|---|
| Answers miss steps 4–6 of a procedure | Chunks too small: the procedure was split across chunks and only part was retrieved |
| Answers mention the right topic but wrong details | Chunks too large: one chunk covers many topics, so the embedding is vague |
| Table answers are wrong | Tables split in the middle, or flattened into text without headers |
| "According to the document…" but which product? | Chunks lost their context (title, product, version) |
1. Start from structure, not a fixed character count
- Split by headings (Markdown
#, HTML<h2>, Word heading styles, PDF layout). - If a section is still too big, split by paragraphs, then sentences.
- Keep lists, code blocks and tables whole where possible.
Good starting points: 300–800 tokens per chunk with 10–15% overlap for prose. Procedures and runbooks often work better as one chunk per section.
2. Add context to every chunk
Prepend a short header before embedding and storing:
Document: VPN Troubleshooting Guide (v3.2, 2026)
Section: Site-to-site tunnel > Phase 2 failures
---
<chunk text>
Store product, version, language and date as metadata fields too, so you can filter (product = 'Gateway X' AND version >= 3).
3. Consider parent-child retrieval
Embed small chunks (precise matching), but send the parent section to the model (complete context). Many frameworks support this ("small-to-big", parent document retriever).
4. Test it, don't guess
Build 50–100 real questions with the document section that answers each one. For each chunking setting, measure:
- Recall@k: is the right section in the top k results?
- MRR: how high does it rank?
def recall_at_k(results, expected, k=5):
hits = sum(1 for q in results if expected[q] in results[q][:k])
return hits / len(results)
Change one thing at a time: chunk size, overlap, headers, embedding model.
5. Re-index carefully
Changing chunking means re-embedding everything. Build a new index next to the old one, run the test set against both, then switch the alias.
Tip
Look at your chunks. Export 20 random chunks and read them. If a human can't tell what a chunk is about without the rest of the document, the model can't either.