Home / AI & GenAI

AI & GenAIHow-toIntermediate

Optimization: keeping agent costs under control - step limits, trimmed tool outputs and context management

Agents call the model many times per task and resend a growing context each time, so costs can grow quickly. Cap steps and spend per task, return compact tool results, summarise or drop old turns, cache stable instructions, use smaller models for sub-steps, and track cost per completed task.

Deepak Bhardwaj2 min read

Applies toLLM agents built with any framework (OpenAI Agents/Responses API, Anthropic tool use, Semantic Kernel, LangGraph, Bedrock Agents) and MCP tool servers

Where agent tokens go

Source Why it grows
Re-sent context Every step resends instructions, tools and history
Tool results Large JSON or documents pasted into context
Tool definitions Dozens of tools with long descriptions on every call
Extra steps Loops, retries, re-planning

Budgets

MAX_STEPS = 12
MAX_TOKENS_PER_TASK = 150_000

for step in range(MAX_STEPS):
    resp = call_model(context)
    spent += resp.usage.total_tokens
    if spent > MAX_TOKENS_PER_TASK:
        return stop("Budget reached", partial=resp)
    if resp.done:
        break

Return a clear message to the user when a budget is reached, and log it for review (see the endless loops troubleshooting post).

Compact tool outputs

Instead of Return
Full API response JSON Only fields the agent needs
500 log lines Top errors with counts and a reference to fetch more
Whole documents Relevant excerpts with IDs
Large tables Summary plus first N rows; offer paging tool

Design MCP tools with limit parameters and concise default outputs.

Context management

  • Summarise older turns into a short state note once history exceeds a threshold.
  • Drop tool results after they've been used, keeping a one-line summary.
  • Send only the tools relevant to the current task (tool selection or grouping).
  • Put stable instructions and tool definitions first for prompt caching (see the prompt caching post).

Model choice per step

Step Model
Planning, hard decisions Large
Simple tool argument filling, extraction, summarising tool output Small
Final answer writing Medium or large

Measure cost per outcome

Track tokens and cost per completed task, not per call, along with success rate. A cheaper configuration that fails more often may cost more per success.

Tip

Look at the five most expensive agent runs each week. They usually reveal one tool returning far too much data or a loop the agent can't escape.

Checklist

  • Step and token budgets per task
  • Tool outputs trimmed and paged
  • Old context summarised or dropped
  • Only relevant tools sent
  • Smaller models for simple steps
  • Cost per completed task tracked

Related fixes