Home / AI & GenAI

AI & GenAIHow-toIntermediate

Transformation: turning scattered ops runbooks into an on-call knowledge assistant

Runbooks live in wikis, Git repos, old tickets and people's heads. Collect and tag them, index them for retrieval with service and freshness metadata, answer with links to the exact runbook step, surface stale content, and keep the assistant read-only so it guides engineers instead of acting for them.

Deepak Bhardwaj3 min read

Applies toRunbooks in Confluence, SharePoint, Azure DevOps/GitHub wikis or Markdown repos; RAG on Azure AI Search, Bedrock Knowledge Bases, OpenSearch or pgvector; delivery in Teams or Slack

Why on-call is different from a FAQ bot

  • Questions arrive under pressure at 3 AM: "payments-api 502 after deploy, what do I check?"
  • A wrong or outdated step can make an incident worse.
  • The best answer is usually "this runbook, section 3", not a freshly written procedure.

So the assistant's job is to find and quote the right runbook quickly, with sources and freshness visible.

1. Collect and normalise

Source Action
Wiki pages Export or connect; keep page URL and last-modified
Markdown in Git Index from the default branch; keep path and commit date
Resolved incident tickets / postmortems Index resolution notes as a separate, lower-ranked source
Tribal knowledge Ask service owners to write the top 5 missing runbooks

Add front-matter or page properties to each runbook:

service: payments-api
component: [ingress, database]
severity_scope: [sev1, sev2]
owner: team-payments
last_reviewed: 2026-09-14

2. Index for ops questions

  • Chunk by heading so each step block stays intact (don't split numbered procedures mid-way).
  • Hybrid search: error codes, hostnames and alert names need keyword matching.
  • Filter or boost by service when the question or alert names one.
  • Index alert rule names → runbook links as a direct lookup table; the most common question is "what do I do with this alert".

3. Answer format

System prompt rules:

  • Answer only from retrieved runbooks; quote the steps, don't rewrite commands.
  • Always show source title, link and last_reviewed.
  • If no runbook matches, say so and suggest escalation (owner team from metadata).
  • Never claim an action was performed.

Example answer shape:

From "payments-api – 5xx after deployment" (reviewed 2026-09-14):
  Step 2: Check the rollout status: kubectl rollout status deploy/payments-api -n payments
  Step 3: If new pods are failing readiness, roll back: kubectl rollout undo ...
Source: https://wiki.example/ops/payments-api-5xx
Owner: team-payments (escalation: #payments-oncall)

4. Freshness and gaps

  • Flag runbooks with last_reviewed older than 12 months in answers ("may be outdated").
  • Weekly report to owners: stale runbooks, questions with no good match, most-used runbooks.
  • After each major incident, the postmortem action "update runbook" closes the loop.

5. Delivery

  • Teams/Slack bot in the on-call channel; also callable from the incident tool (alert → "suggested runbook" link in the page).
  • Authentication through your IdP; respect wiki permissions for restricted runbooks.

6. Keep it read-only (at first)

The assistant suggests commands; an engineer runs them. If you later add actions, follow the bounded-agent approach with approvals.

7. Measure

Metric Source
Runbook found for alert (hit rate) Alert → runbook lookups
Time to first useful step On-call survey, incident timeline
Thumbs-down reasons Bot feedback
Stale runbook count Weekly report

Warning

An assistant that confidently writes its own procedure when no runbook exists is dangerous during incidents. Test the "no match" path as carefully as the happy path.

Checklist

  • Runbooks collected with service, owner and review date
  • Heading-based chunks, hybrid search, alert-name lookup
  • Answers quote steps with link and review date
  • Stale and gap reports to owners
  • Read-only delivery in chat and incident tool

Related fixes