Home / AI & GenAI

AI & GenAITroubleshootingAdvanced

LLM answers cut off mid-sentence: finish_reason length, max_tokens, reasoning tokens and context limits

Detect and fix truncated LLM output in production - read the stop reason for each provider, budget output tokens including reasoning tokens, handle context window overflow, continue safely, and alert on truncation rates.

Deepak Bhardwaj4 min read

Applies toOpenAI / Azure OpenAI Chat Completions and Responses APIs, Anthropic Messages API, Amazon Bedrock Converse, Google Gemini API, vLLM OpenAI-compatible server

Symptom

  • Answers stop mid-sentence, JSON is missing its closing braces, or generated code ends abruptly.
  • It happens more for long documents, long conversations or "reasoning" models.
  • Nobody notices until a parser fails downstream, because the API returned HTTP 200.

Step 1: read the stop reason on every response

Every major API tells you why generation stopped. If you don't log it, you can't see truncation.

API Field Truncated value
OpenAI / Azure OpenAI Chat Completions choices[0].finish_reason length
OpenAI Responses API status / incomplete_details.reason incomplete / max_output_tokens
Anthropic Messages stop_reason max_tokens
Bedrock Converse stopReason max_tokens
Gemini candidates[0].finishReason MAX_TOKENS
vLLM (OpenAI-compatible) finish_reason length

Other values matter too: content_filter (OpenAI/Azure) means the output was blocked, not truncated, and stop / end_turn means normal completion or a stop sequence was hit.

resp = client.chat.completions.create(model=deployment, messages=messages, max_tokens=800)
choice = resp.choices[0]
if choice.finish_reason == "length":
    log.warning("truncated", extra={"deployment": deployment,
                "prompt_tokens": resp.usage.prompt_tokens,
                "completion_tokens": resp.usage.completion_tokens})

Step 2: find which limit you hit

A. The output limit you set. max_tokens (or max_completion_tokens / max_output_tokens) was too small for the task. Common when a default of 256 or 512 was copied from a sample.

B. Reasoning tokens. For reasoning models, hidden reasoning tokens count against the output limit. A request with max_completion_tokens=1000 can spend 950 on reasoning and return almost nothing visible. Check usage.completion_tokens_details.reasoning_tokens (OpenAI/Azure) and raise the budget or lower the reasoning effort.

C. The context window. Input + output must fit in the model's context. With a 128k context and a 126k prompt, only 2k tokens are left for the answer, whatever max_tokens says. Some APIs reject the request instead (context_length_exceeded); others silently shrink the output.

D. Server-side caps. Self-hosted servers have their own limits: in vLLM, --max-model-len caps prompt + output. A gateway may also enforce a max output.

Check usage numbers against the model's limits:

u = resp.usage
print(u.prompt_tokens, u.completion_tokens, u.total_tokens)

If completion_tokens equals your max_tokens, it's A or B. If prompt_tokens + completion_tokens is near the context size, it's C.

Step 3: fix by cause

Cause Fix
Output limit too low Set it from the task: estimate output length and add margin; different limits per use case
Reasoning tokens consume the budget Larger output budget for reasoning models, or lower reasoning effort
Prompt too large Trim history, summarise old turns, retrieve fewer/better chunks; see context length exceeded
Very long outputs needed Generate in sections (outline first, then each section), not one huge call
Structured output truncated Smaller schema per call; validate and retry; see structured output failures

Step 4: continuing a truncated answer (carefully)

For free text, you can ask the model to continue: append the partial assistant message and a user message like "Continue exactly where you stopped." This works for prose and code, but:

  • It doubles cost and latency.
  • The join point can repeat or skip a few words.
  • For JSON, continuing is fragile. Prefer re-generating with a higher limit or a smaller task.
def complete_with_continuation(messages, max_tokens=1500, max_rounds=2):
    parts = []
    for _ in range(max_rounds + 1):
        r = client.chat.completions.create(model=deployment, messages=messages, max_tokens=max_tokens)
        c = r.choices[0]
        parts.append(c.message.content or "")
        if c.finish_reason != "length":
            return "".join(parts), False
        messages = messages + [{"role": "assistant", "content": c.message.content},
                               {"role": "user", "content": "Continue exactly where you stopped. Do not repeat."}]
    return "".join(parts), True   # still truncated after max_rounds

Step 5: streaming

With streaming, the stop reason arrives in the last chunk. Code that stops reading early or ignores the final chunk never sees length. Make sure your streaming handler records it, and that proxies don't cut the stream; see streaming responses buffered behind a proxy.

Step 6: monitor

Add the stop reason to your LLM call log and alert on the truncation rate per use case:

AppTraces
| where TimeGenerated > ago(1d) and Message == "llm_call"
| summarize calls = count(), truncated = countif(tostring(Properties.finish_reason) == "length") by tostring(Properties.use_case)
| extend truncation_pct = round(100.0 * truncated / calls, 2)
| order by truncation_pct desc

A sudden jump usually follows a prompt change (longer instructions), a model change (reasoning model) or longer source documents.

Prevent it next time

  • Treat a truncated response as an error in code paths that parse output.
  • Set output limits per use case, and review them when you change models.
  • Include stop reason, prompt tokens and completion tokens in every LLM call log.

Related fixes