Transformation: from manual ticket triage to LLM-assisted classification and routing

#The target

New ticket → LLM: category, service, priority suggestion, assignment group, summary, confidence
          → high confidence: auto-route   |   low confidence: human triage queue (with suggestions)

#1. Learn from history

Export 6–12 months of closed tickets: description, final category, assignment group, priority, reassignment count.

• Labels: use the final values, not the initial ones (initial triage is what you're improving).
• Clean up: merge near-duplicate categories; drop categories with very few tickets into "Other".
• Reassignment rate today: your baseline. Tickets bounced between groups are the cost you're cutting.

#2. Define the output schema

{
  "type": "object",
  "properties": {
    "category": {"type": "string", "enum": ["Access", "Hardware", "Network", "Email", "Application", "Other"]},
    "assignment_group": {"type": "string", "enum": ["SD-L1", "Network-Ops", "Identity", "EUC", "App-Support"]},
    "priority_suggestion": {"type": "integer", "minimum": 1, "maximum": 4},
    "summary": {"type": "string"},
    "confidence": {"type": "string", "enum": ["high", "medium", "low"]},
    "reason": {"type": "string"}
  },
  "required": ["category", "assignment_group", "priority_suggestion", "summary", "confidence", "reason"],
  "additionalProperties": false
}

Use schema-enforced output (see the structured output post).

#3. Prompt with definitions and examples

Include short definitions for each category and group ("Identity: account lockouts, MFA, group membership, SSO errors") and 1–2 examples each. Retrieval of similar past tickets (few-shot from a vector index of history) often improves accuracy more than a longer prompt.

#4. Measure before using it

Run on a held-out set of 500–1,000 historical tickets:

Model-reported confidence isn't calibrated. Check the actual accuracy of the "high" bucket, or use agreement between two runs/models as a confidence signal.

#5. Roll out in stages

1. Shadow: classify silently; compare with human triage for 2–4 weeks.
2. Suggest: show suggestions to triage agents; measure acceptance rate.
3. Auto-route high confidence for categories with proven accuracy; humans handle the rest.
4. Expand categories as data shows they're safe.

Never auto-set P1/P2 priority from the model alone. Suggest it, and have rules or humans confirm it.

#6. Integrate with the ITSM tool

• Trigger: ticket created (webhook / business rule) → queue → classifier service → update ticket fields via API.
• Write the model's reason and confidence into a work note for transparency.
• Idempotency: don't re-classify on every update; only on creation or explicit request.

#7. Operate

• Weekly: accuracy on a sample, reassignment rate trend, top misroutes.
• Monthly: refresh few-shot examples and category definitions as services change.
• PII: ticket text often contains personal data; apply your logging and data-handling rules (see the secrets/PII in logs post).

Tip

The biggest win is often the summary, not the category: a two-line summary at the top of long email-generated tickets saves every agent time, even when routing stays manual.

#Checklist

• Historical data exported and labels cleaned
• Schema, definitions and examples defined
• Held-out evaluation with per-group accuracy
• Shadow → suggest → auto-route stages
• Monitoring of accuracy and reassignment rate