Before and after
| Single prompt | Bounded agent |
|---|---|
| Script collects everything up front, pastes it in | Model asks for what it needs through tools |
| Breaks when data exceeds the context window | Fetches only relevant slices |
| Same steps for every incident | Picks steps per incident |
| Predictable, cheap | Less predictable; needs limits and logs |
Don't make everything an agent. Keep the single prompt where the input is small and fixed.
1. Pick the first use case
Good first agents are read-only and human-reviewed: incident summary, "why is this job failing", change risk notes. Output goes to a person, not a system.
2. Expose tools through MCP
from mcp.server.fastmcp import FastMCP
mcp = FastMCP("itops-readonly")
@mcp.tool()
def get_alerts(service: str, minutes: int = 60) -> list[dict]:
"""Recent alerts for a service (max 120 minutes)."""
return monitoring.alerts(service=service, minutes=min(minutes, 120))
@mcp.tool()
def search_logs(service: str, query: str, minutes: int = 30) -> list[str]:
"""Up to 50 matching log lines for a service."""
return logs.search(service, query, minutes=min(minutes, 60))[:50]
@mcp.tool()
def recent_changes(service: str, hours: int = 24) -> list[dict]:
"""Deployments and config changes for a service."""
return cmdb.changes(service, hours=min(hours, 72))
if __name__ == "__main__":
mcp.run()
Tool design rules: narrow inputs, clamped ranges, truncated outputs, clear docstrings (the model reads them), service account with read-only rights.
3. The agent loop with limits
MAX_STEPS = 8 # tool calls per task
MAX_TOKENS = 60_000 # total per task
TIMEOUT = 120 s
loop:
model → tool calls? → check allow-list → run via MCP → append results
→ final answer? → stop
stop on limits with "partial result: reached step limit"
Whatever framework you use, make sure these limits exist and are enforced outside the model.
4. Guardrails
- Allow-list tools per agent; the incident agent can't call anything that writes.
- Treat tool output as untrusted: logs and tickets can contain injected instructions.
- Approval step before any future write tool (restart, scale, ticket update): agent proposes, human approves.
- Kill switch per tool and per agent (see the kill switch series).
5. Observe
Log a trace per task: prompt version, each tool call with arguments and result size, tokens, duration, outcome. Review 20 traces a week early on. You'll find loops, useless calls and missing tools quickly.
6. Compare with the old workflow
Run both on the same 30–50 past incidents:
| Measure | Single prompt | Agent |
|---|---|---|
| Summary correctness (reviewer score) | ||
| Root cause found / suggested | ||
| Tokens and cost per incident | ||
| Time to result |
Adopt the agent where it clearly helps; it usually costs more tokens per task.
7. Add write actions later, carefully
Only after the read-only agent is trusted: add one write tool at a time, approval required, scoped (one service, one action), with audit logs.
Caution
Never give the agent's MCP server broad credentials "for now". The tool's service account defines the worst case if the model is manipulated. Scope it as you would for a new junior engineer's first week.
Checklist
- Read-only, human-reviewed first use case
- MCP tools narrow, clamped, documented
- Step, token and time limits enforced in code
- Allow-list, untrusted-output handling, kill switch
- Traces reviewed; compared against old workflow