Why your chatbot forgets: LLM chat history basics, trimming and summaries in Python

#The goal

You've built a chatbot. A user says "My name is Asha", and the next reply is "I don't know your name." Or the bot works until a long conversation fails with a token limit error. Both come from the same fact: the model doesn't remember anything between API calls.

If you haven't called an LLM API yet, start with your first LLM call.

#Step 1: see the problem

pip install openai tiktoken
export OPENAI_API_KEY="<your key>"
export LLM_MODEL="<a model you have access to>"

import os
from openai import OpenAI

client = OpenAI()   # base_url can point at any OpenAI-compatible server
MODEL = os.environ["LLM_MODEL"]

def ask(text):
    r = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": text}],
    )
    return r.choices[0].message.content

print(ask("My name is Asha."))
print(ask("What is my name?"))

The second call has no knowledge of the first. Each one stands alone.

#Step 2: send the history yourself

"Memory" just means that your code keeps the messages and sends them all each time:

SYSTEM = {"role": "system", "content": "You are a concise IT support assistant."}
history = []

def chat(text):
    history.append({"role": "user", "content": text})

    r = client.chat.completions.create(
        model=MODEL,
        messages=[SYSTEM] + history,
    )
    reply = r.choices[0].message.content

    history.append({"role": "assistant", "content": reply})
    print("tokens in:", r.usage.prompt_tokens, "out:", r.usage.completion_tokens)
    return reply

print(chat("My name is Asha."))
print(chat("What is my name?"))

Now it answers correctly. Watch the tokens in number, though. It grows with every turn.

#Step 3: understand the two limits

Long histories also slow responses, and old off-topic content can make answers worse. If you've already got the error, see context length exceeded.

Check your model's current limits in its documentation, rather than hard-coding a number from memory.

#Step 4: count the tokens

Count before you send:

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def count_tokens(messages):
    total = 0
    for m in messages:
        total += len(enc.encode(m["content"])) + 4   # rough overhead per message
    return total + 2

This is an estimate, and the encoding differs by model. Treat the API's usage figures as the truth. See estimate vs billed usage. Leave a safety margin, plus room for the reply.

#Step 5: trim old messages

The simplest approach is to drop the oldest messages until it fits:

MAX_INPUT_TOKENS = 4000

def trim(messages, limit=MAX_INPUT_TOKENS):
    trimmed = list(messages)
    # Keep at least the latest user message
    while len(trimmed) > 1 and count_tokens([SYSTEM] + trimmed) > limit:
        trimmed.pop(0)
    return trimmed

def chat(text):
    global history
    history.append({"role": "user", "content": text})
    history = trim(history)

    r = client.chat.completions.create(model=MODEL, messages=[SYSTEM] + history)
    reply = r.choices[0].message.content
    history.append({"role": "assistant", "content": reply})
    return reply

Always keep the system message, because it defines how the bot behaves. Tool calls also need care: don't remove a tool result without its matching call, as many APIs will reject that.

The downside is simple. The bot forgets anything old, even if it's important, like the user's name.

#Step 6: summarize instead of forgetting

When history gets long, replace the older part with a short summary:

SUMMARY_TRIGGER = 3000
KEEP_RECENT = 6
summary = ""

def summarize(old_messages, previous):
    transcript = "\n".join(f'{m["role"]}: {m["content"]}' for m in old_messages)
    r = client.chat.completions.create(
        model=MODEL,
        temperature=0,
        messages=[
            {"role": "system", "content":
             "Update the summary of this support conversation. Keep names, IDs, "
             "decisions, errors and open questions. Only include facts that are stated. "
             "Use under 150 words."},
            {"role": "user", "content": f"Current summary:\n{previous}\n\nNew messages:\n{transcript}"},
        ],
    )
    return r.choices[0].message.content

def chat(text):
    global history, summary
    history.append({"role": "user", "content": text})

    if count_tokens(history) > SUMMARY_TRIGGER and len(history) > KEEP_RECENT:
        old, history = history[:-KEEP_RECENT], history[-KEEP_RECENT:]
        summary = summarize(old, summary)

    system = SYSTEM["content"]
    if summary:
        system += f"\n\nConversation so far:\n{summary}"

    r = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "system", "content": system}] + history,
    )
    reply = r.choices[0].message.content
    history.append({"role": "assistant", "content": reply})
    return reply

The trade-offs:

Tip

Keep the stable part of your prompt first, and unchanged. Some providers discount repeated prefixes. See prompt caching. A summary placed in the system message changes it, so you might put it after the fixed instructions.

#Step 7: keep users separate

A global history list is fine for a demo. On a server, it would mix up every user's conversation. That's a security incident, not merely a bug.

from collections import defaultdict

sessions = defaultdict(list)   # for a demo. Use Redis or a database in production.

def chat_for(session_id, text):
    history = sessions[session_id]
    history.append({"role": "user", "content": text})
    r = client.chat.completions.create(model=MODEL, messages=[SYSTEM] + history)
    reply = r.choices[0].message.content
    history.append({"role": "assistant", "content": reply})
    return reply

Multiple servers and restarts need shared storage. A Redis key with a TTL works well, such as chat:{user_id}:{session_id}. Get the session ID from your authenticated login, never from a value that a client can freely change.

Warning

Conversations often contain personal data and secrets. Stored chat history is sensitive data, so set a retention period, encrypt it and control who can read it. See redacting PII. Text that users put in the history can also try to steer the model later. See prompt injection defenses.

#Common problems

#Prevent it next time

• Log prompt tokens per turn, so that you can catch runaway conversations.
• Set a token limit and a maximum session length.
• Test a very long conversation before launch.
• Keep important user facts in your own database, instead of relying on the chat history.
• Next, learn about streaming through a proxy and estimating cost.