When fine-tuning is the right tool
| Problem | Better fix |
|---|---|
| Model lacks your facts / documents | RAG, not fine-tuning |
| Output format or style inconsistent | Structured output first; fine-tune if still unstable |
| Long prompt with many rules + examples, high volume | Fine-tune a smaller model |
| Domain classification / extraction at scale | Fine-tune a smaller model |
| Needs strong reasoning on new problems | Larger model; fine-tuning rarely helps |
LoRA (Low-Rank Adaptation) trains small adapter matrices instead of all weights, so it fits on one or a few GPUs and adapters are megabytes, not gigabytes.
1. Baseline
Record the current setup: model, prompt length, tokens per request, latency p95, cost per 1,000 requests, and score on your evaluation set. The fine-tuned model must beat or match this.
2. Build the training set
- Source: logged production inputs with approved outputs (human-reviewed, or large-model outputs that passed your evaluator).
- Size: a few hundred to a few thousand high-quality examples is often enough for a narrow task; quality beats volume.
- Remove PII or confirm it's allowed in training; deduplicate; hold out 10–20% for evaluation.
- Format as chat messages, with a short system prompt (the rules are now learned):
{"messages": [{"role": "system", "content": "Classify the ticket."},
{"role": "user", "content": "VPN drops every 10 minutes since the update..."},
{"role": "assistant", "content": "{\"category\":\"Network\",\"group\":\"Network-Ops\"}"}]}
3. Train with PEFT + TRL
from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer
ds = load_dataset("json", data_files={"train": "train.jsonl", "eval": "eval.jsonl"})
peft_cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, task_type="CAUSAL_LM",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"])
cfg = SFTConfig(output_dir="out/ticket-lora", num_train_epochs=3, per_device_train_batch_size=4,
gradient_accumulation_steps=4, learning_rate=2e-4, bf16=True,
eval_strategy="epoch", save_strategy="epoch", logging_steps=20)
trainer = SFTTrainer(model="Qwen/Qwen2.5-7B-Instruct", args=cfg, train_dataset=ds["train"],
eval_dataset=ds["eval"], peft_config=peft_cfg)
trainer.train()
trainer.save_model("out/ticket-lora/final")
TRL/PEFT argument names change between versions; check against the version you install. QLoRA (4-bit base + LoRA) reduces GPU memory further. Watch eval loss for overfitting: if it rises while train loss falls, stop earlier or add data.
4. Evaluate against the baseline
| Variant | Task score | p95 latency | Cost / 1k requests |
|---|---|---|---|
| Large model + long prompt (baseline) | measure | measure | measure |
| Small base model + long prompt | measure | measure | measure |
| Small model + LoRA + short prompt | measure | measure | measure |
Also check regressions: refusals, odd outputs on out-of-distribution inputs, safety behaviour.
5. Serve adapters
vLLM can serve the base model with one or more LoRA adapters:
vllm serve Qwen/Qwen2.5-7B-Instruct --enable-lora \
--lora-modules ticket-classifier=/models/adapters/ticket-lora-v3 \
--max-lora-rank 16
Clients select the adapter with model="ticket-classifier". Several adapters can share one base deployment.
6. Operate
- Version adapters with their training data snapshot and evaluation score.
- Re-train when categories, products or input styles change; schedule a monthly drift check.
- Keep the baseline prompt path as a fallback behind the gateway.
Caution
Training on unreviewed model outputs bakes in their mistakes. Review a sample of the training set by hand. Ten wrong labels in a 500-example set are visible in the results.
Checklist
- Fine-tuning is the right fix (not RAG or structured output)
- Baseline measured
- Reviewed, de-duplicated, PII-checked training set
- LoRA trained, eval loss watched
- Beats baseline on quality, latency, cost
- Adapter served, versioned, fallback kept