Home / AI & GenAI

AI & GenAITroubleshootingBeginner

context_length_exceeded and max token errors: fixing long prompts and cut-off answers

Understand context windows vs output limits, count tokens before calling the model, and trim chat history and RAG context so requests fit and answers aren't truncated.

Deepak Bhardwaj2 min read

Applies toOpenAI / Azure OpenAI, Amazon Bedrock, OCI Generative AI, Anthropic, Gemini and self-hosted models

The errors

Message Meaning
context_length_exceeded / This model's maximum context length is N tokens. However, your messages resulted in M tokens Input + requested output is larger than the context window
Bedrock: ValidationException: Input is too long for requested model Same
Answer stops mid-sentence; finish_reason: "length" (OpenAI) or stop_reason: "max_tokens" The output limit (max_tokens / max_completion_tokens) was reached

Context window = input tokens + output tokens. Reserving 4,000 output tokens on a 16k-context model leaves about 12k for the prompt.

1. Count tokens before sending

import tiktoken                       # OpenAI-family tokenizers
enc = tiktoken.get_encoding("o200k_base")
n = sum(len(enc.encode(m["content"])) for m in messages)

Other providers have their own token-count endpoints or return usage in the response. Rough rule for English: 1 token ≈ 4 characters; other languages and code use more tokens.

2. Where the tokens come from

Log the token count per part of the prompt:

Part Typical cause of growth
Chat history Every turn is resent, so long conversations grow without limit
RAG context 20 chunks × 800 tokens = 16k tokens
Tool results A tool returns a full JSON API response or log file
System prompt Long policy text, many few-shot examples

3. Fixes

  • History: keep the last N turns, and replace older turns with a running summary.
  • RAG: rerank and pass the top 3–5 chunks; cut each chunk to the relevant part.
  • Tool output: return only needed fields; paginate; summarise large logs before passing them back.
  • System prompt: remove duplication; move rarely needed rules into retrieval.
  • Use a model with a larger context window only after trimming. Bigger prompts cost more, are slower, and quality can drop when the relevant text is buried in the middle.

4. Cut-off answers

  • Raise max_tokens / max_completion_tokens (within the model's output limit).
  • For reasoning models, hidden reasoning tokens count against the completion limit. A low limit can produce an empty visible answer.
  • Ask for a structured, concise format, or generate long documents section by section.
  • Always check the finish/stop reason in code and handle truncation (retry with a higher limit, or continue).

Tip

Put a token budget into the code: e.g. system 1k, history 3k, context 6k, output 2k. Enforce each budget before the call. It turns a random production error into a predictable trim.

Related fixes