Token counts don't match the bill: why tiktoken estimates differ from the usage your LLM API reports

#Symptom

One or more of these:

• Your code counts 1,200 tokens with tiktoken, but the response says prompt_tokens: 1,950.
• Cost dashboards built from your own counts come in well under the invoice.
• Azure OpenAI returns 429 even though your measured tokens per minute are under quota.
• Output token cost is far above what the visible answers could explain.

The tokens are real. Your counting just leaves some of them out. Use the API's usage as the source of truth, and find the gap with the checks below.

#Quick checks

Log the full usage object for one request, not just total_tokens:

from openai import OpenAI
client = OpenAI()

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    tools=tools,
)
print(resp.usage.model_dump_json(indent=2))

Fields to look at (Chat Completions):

The Responses API uses input_tokens, output_tokens, input_tokens_details.cached_tokens and output_tokens_details.reasoning_tokens. Bedrock Converse returns usage.inputTokens, usage.outputTokens and, when prompt caching is on, cacheReadInputTokens and cacheWriteInputTokens.

#Symptom → cause → fix

#Fix 1: use the right tokenizer

tiktoken only matches OpenAI-family models, and only when you pick the matching encoding. GPT-4o, GPT-4.1 and the o-series use o200k_base. GPT-4, GPT-3.5 Turbo and the text-embedding-3 models use cl100k_base.

import tiktoken

def encoding_for(model: str):
    try:
        return tiktoken.encoding_for_model(model)
    except KeyError:
        # Newer model names may not be in your tiktoken version yet
        return tiktoken.get_encoding("o200k_base")

Upgrade tiktoken when you adopt a new model family; older releases don't know new model names.

For non-OpenAI models, tiktoken is only a rough guide:

• Anthropic: call the token counting endpoint. It accepts the same messages, system and tools as a real call.

  import anthropic
  client = anthropic.Anthropic()
  r = client.messages.count_tokens(model="<model-id>", messages=messages, tools=tools)
  print(r.input_tokens)
  

• Amazon Bedrock: some models support a CountTokens API. Check your model in the Bedrock documentation; otherwise rely on usage in the Converse response.

• Self-hosted models: use the model's own tokenizer and chat template, because the template adds role markers and special tokens.

  from transformers import AutoTokenizer
  tok = AutoTokenizer.from_pretrained("<org>/<model>")
  ids = tok.apply_chat_template(messages, tokenize=True, add_generation_prompt=True)
  print(len(ids))
  

  vLLM's OpenAI-compatible server also has a /tokenize endpoint that applies the served model's tokenizer:

  curl -s http://localhost:8000/tokenize \
    -H "Content-Type: application/json" \
    -d '{"model": "<served-model-name>", "prompt": "How do I restart the agent?"}'
  

#Fix 2: account for message and tool overhead

Each chat message carries a few formatting tokens on top of its content. OpenAI's cookbook uses about 3 tokens per message plus 3 to prime the reply for current chat models. Treat this as an approximation that can change between models:

def estimate_chat_tokens(messages, model="gpt-4o-mini"):
    enc = encoding_for(model)
    n = 3  # reply priming
    for m in messages:
        n += 3 + len(enc.encode(m.get("content") or ""))
        if "name" in m:
            n += 1
    return n

Tool definitions are the bigger gap. The exact format used to inject them isn't published, so measure the cost of your tool set directly:

base = client.chat.completions.create(model=m, messages=msgs, max_tokens=1)
with_tools = client.chat.completions.create(model=m, messages=msgs, tools=tools, max_tokens=1)
print("tool overhead:", with_tools.usage.prompt_tokens - base.usage.prompt_tokens)

Store that number per tool set and add it to your estimates. If it's large, send only the tools each step needs and shorten long descriptions and enum lists.

#Fix 3: capture usage on streamed responses

Streamed Chat Completions don't include usage unless you ask. With the option on, the final chunk has an empty choices list and the usage object:

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    stream=True,
    stream_options={"include_usage": True},
)
usage = None
for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="")
    if chunk.usage:
        usage = chunk.usage

Warning

If the client disconnects or your code stops reading the stream early, you never receive the final usage chunk, but the provider may still bill for tokens it generated. Don't treat "no usage logged" as "zero cost". Reconcile against provider metrics.

#Fix 4: stop Azure OpenAI 429s caused by max_tokens

Azure OpenAI checks TPM limits against an estimate made when the request arrives. That estimate includes the prompt size and the max_tokens you request, not the tokens actually generated. A service that sends max_tokens: 4000 on every call but gets 200-token answers uses up the rate limit far faster than its real usage suggests.

Set max_tokens / max_completion_tokens per use case, from the 95th percentile of actual output plus a margin. For reasoning models, leave room for reasoning tokens, or the visible answer comes back empty.

#Reconcile with provider metrics

Compare your logged totals with what the platform recorded for the same window.

Azure OpenAI (metric names vary by API version and resource type; list them first):

RID=$(az cognitiveservices account show -n <aoai-name> -g <rg> --query id -o tsv)
az monitor metrics list-definitions --resource $RID --query "[].name.value" -o tsv
az monitor metrics list --resource $RID \
  --metric ProcessedPromptTokens GeneratedTokens \
  --interval PT1H --aggregation Total \
  --start-time 2026-10-30T00:00:00Z --end-time 2026-10-31T00:00:00Z -o table

Amazon Bedrock:

aws cloudwatch get-metric-statistics --namespace AWS/Bedrock \
  --metric-name InputTokenCount \
  --dimensions Name=ModelId,Value=<model-id> \
  --start-time 2026-10-30T00:00:00Z --end-time 2026-10-31T00:00:00Z \
  --period 3600 --statistics Sum

Run the same for OutputTokenCount. If the provider total is well above your logged total, look for calls that bypass your logging: retries inside the SDK, evaluation jobs, abandoned streams or another app using the same key.

Tip

SDK retries are billed calls. A request retried twice after timeouts can be billed three times while your app logs it once. Log usage per attempt, not per logical request.

#Prevent it next time

• Log the full usage object, model name and caller on every attempt, including streams.
• Use estimates only for pre-call budgeting and trimming. Use usage for cost reporting and chargeback.
• Keep a measured overhead number for each tool set and system prompt, and re-measure after model upgrades.
• Price input, cached input, output and reasoning tokens as separate lines. Check current prices with your provider.
• Set max_tokens per use case instead of one large default.
• Reconcile your logs with provider metrics weekly and alert when they drift apart by more than a few percent.