Checklist: production readiness for multi-agent systems - handoffs, shared state, budgets and failure isolation

#Why a separate checklist

A single agent with tools has one loop, one context and one budget. Several agents add a graph: handoffs, delegation, shared state and parallel branches. The common production failures are in the edges of that graph, not the prompts:

Use the sections below as a go-live gate. Every unchecked item needs a named owner and a date.

#1. Topology and responsibilities

• The graph is written down: which agents exist, who can call or hand off to whom, and which edges are forbidden.
• Each agent has a one-sentence responsibility and a disjoint tool set. Two agents that can both "fix the ticket" will argue about it.
• You checked that a single agent with more tools, or a fixed workflow, can't do the job. Every extra agent adds latency, tokens and failure modes.
• Parallel branches are used only where the sub-tasks are truly independent.

#2. Handoffs and loop control

• A run-level handoff depth limit (e.g. 8) and a per-pair limit (A to B at most twice).
• A handoff carries a structured payload (task, constraints, what's already been tried), not the whole transcript.
• Ping-pong detection: the same pair handing off with no new state ends the run with an explicit failure.
• A terminal state for every path: success, escalate to human, or fail with a reason. No path ends with "the model stopped talking".

A guard that sits in the orchestrator, outside any model:

from collections import Counter

class RunGuard:
    def __init__(self, max_depth=8, max_pair=2, max_tokens=200_000, max_tool_calls=60):
        self.depth, self.pairs = 0, Counter()
        self.tokens, self.tool_calls = 0, 0
        self.limits = dict(depth=max_depth, pair=max_pair, tokens=max_tokens, tools=max_tool_calls)
        self.last_state_hash = {}

    def on_handoff(self, src, dst, state_hash):
        self.depth += 1
        self.pairs[(src, dst)] += 1
        if self.depth > self.limits["depth"]:
            raise RuntimeError(f"handoff depth {self.depth} exceeded")
        if self.pairs[(src, dst)] > self.limits["pair"]:
            raise RuntimeError(f"ping-pong {src}->{dst}")
        if self.last_state_hash.get((dst, src)) == state_hash:
            raise RuntimeError(f"no progress between {src} and {dst}")
        self.last_state_hash[(src, dst)] = state_hash

    def on_llm_usage(self, usage):
        self.tokens += usage.input_tokens + usage.output_tokens
        if self.tokens > self.limits["tokens"]:
            raise RuntimeError(f"run token budget exceeded: {self.tokens}")

    def on_tool_call(self):
        self.tool_calls += 1
        if self.tool_calls > self.limits["tools"]:
            raise RuntimeError("run tool-call budget exceeded")

Usage field names differ by SDK (input_tokens/output_tokens vs prompt_tokens/completion_tokens); adapt the method to your client.

#3. Budgets

• Token, tool-call, wall-clock and cost budgets are per run (and per user per day), shared by all agents in the run.
• Sub-agents get a slice of the remaining budget, not a fresh one.
• The run stops cleanly when a budget runs out and returns partial results with a clear status.
• Per-run cost is logged so you can see the p99 run, not just the average.

#4. Shared state

• State has one schema and one writer per field, or optimistic concurrency (version number checked on write).
• Parallel workers write to their own keys; the supervisor merges.
• State is persisted at each step (checkpointing), so a crashed run resumes rather than starting again and repeating side effects.
• Tool outputs placed in shared state are size-capped and marked as untrusted data, not instructions.

Optimistic concurrency in PostgreSQL:

UPDATE run_state
SET data = jsonb_set(data, '{findings}', $1::jsonb), version = version + 1
WHERE run_id = $2 AND version = $3;
-- 0 rows updated = someone else wrote first; re-read and retry or fail the step

#5. Identity and permissions

• Each agent runs with its own scoped identity; tool permissions are per agent, not per system.
• Actions taken on behalf of a user carry the user's identity or an on-behalf-of token, so a sub-agent can't exceed what the user may do.
• Agents that read untrusted content (email, web, tickets) have no write tools. Writing is done by a different agent that only sees structured, validated output.
• Side-effecting tools are idempotent with a run-scoped idempotency key, because resumed runs and retries will call them again.

Warning

The "reader with no write tools" split is the main structural defence against indirect prompt injection spreading through a multi-agent system. If one agent both reads attacker-controllable text and can send email or change tickets, an injected instruction only needs to get through one hop.

#6. Failure isolation

• Every agent call and tool call has a timeout; the supervisor has its own, longer one.
• A failed or timed-out branch returns a typed error the supervisor can reason about, not an exception that kills the run.
• Retries happen in one layer only (SDK or orchestrator, not both), with backoff and a cap.
• Model fallback per agent: a cheaper or different-provider model for non-critical workers when the primary is throttled.
• A kill switch can disable one agent or one edge without stopping the rest.

#7. Observability

• One trace per run, with a span per agent turn, handoff and tool call, linked parent to child.
• Span attributes include agent name, model, prompt version, tokens and handoff reason.
• Dashboards show handoff depth distribution, runs ended by guard, and cost per run.

If you export traces to Application Insights, a KQL check for runs that hit the guard (adjust to your span names and attributes):

dependencies
| where timestamp > ago(1d)
| where name == "agent.run"
| extend outcome = tostring(customDimensions["run.outcome"]),
         depth = toint(customDimensions["run.handoff_depth"])
| summarize runs = count(), guard_stops = countif(outcome startswith "guard"),
            p95_depth = percentile(depth, 95) by bin(timestamp, 1h)
| order by timestamp asc

#8. Testing

• Trajectory tests per agent and for the whole graph, with recorded tool responses.
• Adversarial tests: injected instructions in tool output, a worker that always fails, a worker that always hands back.
• Load test at expected concurrency. Fan-out multiplies requests to the model endpoint, so check the TPM quota against the peak, not the run rate.
• A pass-rate gate in CI for prompt, model or graph changes.

#9. Operations

• Runbook for "runs are looping", "cost spike" and "agent X misbehaving", including which switch to pull.
• Owner per agent; prompt and tool changes go through review like code.
• Human escalation path tested end to end, including what the user sees while waiting.
• Retention for run state and traces agreed, with personal data handling defined.

#Prevent it next time

Keep the guard limits in config, not code, and review them monthly against real run distributions. Most teams start too loose, then find the p99 run costs 40x the median. Tighten limits until under 1% of legitimate runs hit them, and treat every guard stop as a signal to investigate.