What carries over, what's new
| MLOps practice | Same for LLM apps? | Change |
|---|---|---|
| Model registry with versions | Partly | Most apps call hosted models; register prompts, configs and app versions too |
| Training pipelines | Rarely | Replaced by prompt/retrieval changes; fine-tuning when used |
| Offline metrics (accuracy, AUC) | No | Evaluation sets + LLM/human judges for correctness, groundedness |
| Data drift monitoring | Partly | Query/topic drift, retrieval quality, feedback rates |
| Inference monitoring | Yes | Add tokens, TTFT, cost per request, safety blocks |
| Approval gates | Yes | Gate on evaluation scores instead of accuracy |
1. Version what actually changes
An LLM app release is a combination of:
app version = prompt template(s) + model/deployment + retrieval config (index, chunking, top-k)
+ tool definitions + guardrail config
Store all of it in Git and register each release as an artefact/model version with these as tagged parameters. MLflow's prompt registry (in recent versions) or plain versioned files both work. Pick one and enforce it.
2. Evaluation datasets as first-class assets
- Curate datasets per app: question, expected answer or key facts, expected source, category.
- Version them like training data; they grow from production feedback.
- Run evaluation in the same pipeline engine you use for training jobs.
MLflow example (API names differ between MLflow 2 and 3; check your version):
import mlflow
with mlflow.start_run(run_name="kb-assistant-v14"):
mlflow.log_params({"prompt_version": "v14", "deployment": "chat-main", "top_k": 8})
results = mlflow.evaluate( # MLflow 3 also provides mlflow.genai.evaluate with scorers
data=eval_df, model_type="question-answering",
predictions="answer", targets="expected",
extra_metrics=[faithfulness_metric, relevance_metric])
print(results.metrics)
3. Tracing
Add request tracing (MLflow Tracing, OpenTelemetry GenAI conventions, or your APM) so each production request has: app version, retrieved documents, model calls, tool calls, tokens and latency. These traces become new evaluation cases when users flag bad answers.
4. One release gate
PR merged → build → run eval suite on candidate app version
→ compare with current production version
→ pass: register + deploy to staging → canary 10% → full
→ fail: block, post scores on the PR
Gate criteria example: correctness ≥ current − 1 point, groundedness ≥ 0.95, no new safety failures, cost per request ≤ +10%.
5. Monitoring additions
| Signal | Why |
|---|---|
| Tokens and cost per request, per app version | Prompt changes quietly raise cost |
| Feedback (thumbs) rate per version | Early quality signal |
| Retrieval empty/low-score rate | Index or content problems |
| Content filter / guardrail block rate | Safety tuning or abuse |
| Model version in use | Catches silent provider updates and retirements |
6. Roles
Reuse existing ML platform ownership for pipelines, registry and monitoring. Add app owners responsible for evaluation datasets and prompt reviews. They know what "correct" means for their domain.
Tip
Start with one app and one gate. A single enforced evaluation gate in CI changes behaviour more than a full LLMOps platform that nobody is required to use.
Checklist
- App release = versioned prompt + model + retrieval + tools + guardrails
- Evaluation datasets versioned and growing from feedback
- Tracing in production
- Release gate on evaluation scores
- Monitoring includes tokens, cost, feedback, retrieval and safety signals