LLMOps foundation: prompt versioning, evaluation gates and safe releases

#1. What counts as a "release"

Any change to these can change behaviour, so each one goes through the same pipeline:

• The prompt template or system prompt.
• The model, or model version (for example a new snapshot).
• Parameters (temperature, max tokens).
• Retrieval settings (chunking, top-k, filters, embedding model).
• Tools and their descriptions.

#2. Prompts in Git

prompts/
  support-assistant/
    system.md
    config.yaml        # model, version, temperature, top_k
    CHANGELOG.md
evals/
  support-assistant/
    golden.jsonl       # question, expected facts, must-not-say

Load prompts by version at runtime (from the repo build, or a prompt registry), never pasted into code.

#3. Golden evaluation set

• 50–300 real questions: common ones, edge cases, past failures, and adversarial ones (prompt injection, out of scope).
• For each one: expected key facts or a reference answer, and what it must not say.
• Kept up to date: every production incident adds a test case.

#4. Automated evaluations in CI

# promptfooconfig.yaml (excerpt)
prompts: [file://prompts/support-assistant/system.md]
providers:
  - id: azure:chat:gpt-4o-prod
tests: file://evals/support-assistant/golden.yaml
defaultTest:
  assert:
    - type: llm-rubric
      value: "Answer is grounded in the provided context and cites a source"
    - type: not-contains
      value: "As an AI language model"

npx promptfoo eval -c promptfooconfig.yaml --output results.json

The pull request fails if a gate fails. Evaluation results are attached to the PR.

#5. Staged rollout

1. Deploy to staging and run the full evaluation.
2. Shadow traffic or an internal pilot group.
3. Canary (5–10% of users) behind a feature flag, comparing feedback and error rates.
4. Full rollout. Keep the old version available for instant rollback (a flag flip, not a redeploy).

#6. Production observability

• Trace each request: prompt version, model version, retrieved documents, tokens, latency, and the user's feedback (see the OpenTelemetry tracing post).
• Sample production conversations, score them with the same evaluators, and track the trend.

#7. Model version upgrades

Providers retire model versions on a schedule. Track the retirement dates. Run the golden set against the new version weeks before the deadline, adjust prompts, then roll out as above.

Tip

Start small: 30 good test cases running on every pull request beat 1,000 cases that nobody runs. Grow the set from real failures.