Applies toHosted LLM APIs (Azure OpenAI, OpenAI, Anthropic, Bedrock, Gemini) and self-hosted models; reasoning models with effort/thinking budget controls
Why output matters
- Output tokens are typically priced higher than input tokens (check current pricing).
- Generation is sequential: each extra token adds latency.
- Reasoning models also produce hidden reasoning tokens, which are billed.
Measure first
Log usage.completion_tokens (or output_tokens) per request with the feature name, then look at the distribution:
-- Example on a log table of LLM calls
SELECT feature, model,
percentile_cont(0.5) WITHIN GROUP (ORDER BY output_tokens) AS p50_out,
percentile_cont(0.95) WITHIN GROUP (ORDER BY output_tokens) AS p95_out,
count(*) AS calls
FROM llm_calls
WHERE ts > now() - interval '7 days'
GROUP BY feature, model
ORDER BY p95_out DESC;
Levers
| Lever | Example |
|---|---|
max_tokens / max_completion_tokens (reasoning models on Chat Completions) / max_output_tokens (Responses API) per feature | Classification: 20; chat answer: 600 |
| Explicit length guidance | "Answer in at most 3 sentences" / "Return 5 bullet points" |
| Structured outputs | JSON schema with only needed fields, enums instead of free text |
| Stop sequences | Stop after the closing tag of the needed section |
| Reasoning effort / thinking budget | Low for simple tasks; higher only where evaluation shows benefit |
| No repetition of input | "Do not restate the question or the document" |
resp = client.chat.completions.create(
model=deployment,
messages=messages,
max_tokens=300,
response_format={"type": "json_schema", "json_schema": {
"name": "ticket_triage", "strict": True,
"schema": {"type": "object", "additionalProperties": False,
"required": ["category", "priority"],
"properties": {"category": {"type": "string", "enum": ["network","database","access","other"]},
"priority": {"type": "string", "enum": ["P1","P2","P3","P4"]}}}}},
)
Watch for truncation
When max_tokens is hit, the response finish reason is length (or equivalent). Track the rate:
- Rare and acceptable → fine.
- Frequent → raise the limit or fix the prompt; truncated JSON breaks parsers (see the invalid JSON Sev1 post).
Reasoning models
- Use the lowest effort that passes your evaluation.
- Reasoning tokens count toward output limits on some APIs; leave room so the visible answer isn't cut off.
Tip
For machine-consumed outputs, structured outputs with enums can reduce output tokens dramatically and remove parsing failures at the same time.
Checklist
- Output tokens logged per feature
- max_tokens set per feature
- Length and format instructions in prompts
- Structured outputs for machine consumers
- Reasoning effort tuned by evaluation
- Truncation rate monitored