AI agent tool calls executed twice: retries, timeouts and re-planning that send duplicate emails, tickets and payments

#Symptoms

• Duplicate tickets, emails, messages or records created by an agent, usually seconds apart.
• More rarely: a payment, refund or account change applied twice.
• Logs show one user request but two (or more) executions of the same tool.

Agents are built from several layers that each retry or repeat work. Without protection, a side-effecting tool runs once per repetition.

#Where duplicates come from

#Step 1: trace one duplicate

For a duplicate pair, collect:

• Tool call IDs from the model responses (tool_call_id / toolUseId).
• Timestamps of each model call and tool execution.
• Whether the two executions share a model response (same tool call ID) or come from two model turns.

Tracing (OpenTelemetry spans per model call and tool call) makes this a one-minute job instead of a log hunt.

#Step 2: make side-effecting tools idempotent

The tool, not the model, must guarantee "at most once". Give every side-effecting operation an idempotency key derived from the business intent, not from the attempt.

import hashlib, json

def idempotency_key(conversation_id: str, intent: dict) -> str:
    # Same user turn + same normalised intent = same key, whatever the retries
    payload = json.dumps({"conv": conversation_id, **intent}, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()

def create_ticket(conversation_id, title, body, requester):
    key = idempotency_key(conversation_id, {"op": "create_ticket", "title": title.strip().lower(), "requester": requester})
    existing = dedup_store.get(key)
    if existing:
        return {"status": "already_created", "ticket_id": existing}
    ticket_id = ticketing_api.create(title=title, body=body, requester=requester,
                                     headers={"Idempotency-Key": key})
    dedup_store.put(key, ticket_id, ttl_seconds=86400)
    return {"status": "created", "ticket_id": ticket_id}

• Use the downstream API's own idempotency support where it exists (many payment and messaging APIs accept an Idempotency-Key header).
• Store keys with a conditional write (INSERT ... ON CONFLICT DO NOTHING, DynamoDB attribute_not_exists) so two concurrent attempts can't both pass.
• Return a clear result for the duplicate case ("already created, ticket 4711"). This also stops the model from trying again.

#Step 3: give the model unambiguous tool results

Many re-plans happen because the tool returned something vague ("error", a timeout, an HTML page). Return structured, final outcomes:

{"status": "created", "ticket_id": "INC-4711", "note": "Do not create another ticket for this request."}

For timeouts, check the downstream state before reporting failure: "a ticket with this key exists" means success.

#Step 4: separate deciding from doing for risky actions

proposal_id = store_proposal(action="refund", order_id=..., amount=...)
# UI shows proposal; user approves
execute_refund(proposal_id)   # executes once; repeated calls return the existing result

#Step 5: framework and infrastructure settings

• Disable automatic retries on tool execution in the agent framework for non-idempotent tools, or make them idempotent first.
• Durable workflow engines replay steps by design; mark side-effecting steps as activities with idempotency keys.
• Queue consumers: deduplicate on message ID before running the agent step.
• LLM gateway retries: retrying the model call is fine as long as tools are idempotent; the risk is in what the new response triggers.

Caution

Never rely on prompt instructions ("never call create_ticket twice") as the protection. Models follow them most of the time, not all of the time. Enforce it in code.

#Monitoring

• Count tool executions per conversation turn by tool name; alert when side-effecting tools exceed 1.
• Log idempotency-key hits; a rising hit rate shows retries or re-planning increasing.
• Review duplicate incidents in a weekly agent quality review.

#Prevent it next time

• Classify every tool as read-only or side-effecting at design time.
• Require idempotency keys for all side-effecting tools, enforced in the tool server (or MCP server), not in the agent.
• Return explicit, final statuses from tools.
• Add human confirmation for high-impact actions.