Applies toLLM applications on hosted or self-hosted models; RAG with any search/vector store; supervised fine-tuning and LoRA; long-context models with prompt caching
What does the model lack?
| Gap | Best fit |
|---|---|
| Facts that change or are private (policies, tickets, product data) | RAG |
| Behaviour or format (tone, structure, domain jargon, classification labels) | Prompting first, then fine-tuning |
| Small, stable knowledge that fits in the context window | Long context (put it in the prompt) with prompt caching |
| Speed/cost for a narrow repeated task | Fine-tune a smaller model (see the LoRA transformation post) |
Fine-tuning is poor at teaching new facts reliably and can't respect per-user permissions.
Compare
| RAG | Fine-tuning | Long context | |
|---|---|---|---|
| Knowledge freshness | Update the index | Retrain | Update the prompt |
| Access control | Filter at retrieval per user | None per user | Per request selection |
| Citations | Natural | Hard | Possible |
| Cost per request | Retrieval + context tokens | Lower (smaller model, shorter prompts) | High input tokens (reduced by caching) |
| Build effort | Ingestion pipeline, index, evaluation | Training data, training runs, evaluation | Lowest |
| Failure mode | Retrieval misses | Confident but outdated/wrong | Model overlooks details in long input |
Decision flow
- Can good prompting with examples meet the quality bar? → Stop there.
- Does the knowledge fit comfortably in the context and change rarely? → Long context + prompt caching.
- Is the knowledge large, changing, or permission-sensitive? → RAG.
- Is the remaining gap about format/behaviour or cost at high volume? → Fine-tune.
Combining
| Combination | Use |
|---|---|
| RAG + fine-tuned model | Fine-tune for output format/domain style; RAG for facts |
| RAG + long context | Retrieve whole relevant documents rather than tiny chunks |
| Fine-tuned small model as router/classifier + large model for answers | Cost control |
Evaluate each option
Use the same evaluation set across approaches: correctness, groundedness (for RAG), format compliance, latency and cost. Choose by evidence, not preference.
Warning
Never fine-tune on data some users aren't allowed to see. The model can reveal it to anyone. Keep permission-sensitive knowledge in RAG with retrieval-time filtering.
Checklist
- Gap identified: facts, behaviour or cost
- Prompting baseline measured first
- Knowledge size, change rate and permissions assessed
- Approach (or combination) chosen
- Evaluated on the same test set