How to Stop an LLM App Forgetting Earlier Turns

An LLM app forgets earlier turns because the model has no memory. Each call sees only the text you send in that request. If a fact is not in the request, the model cannot use it. To stop the forgetting, you decide what goes into every request: recent turns word for word, a rolling summary of older turns, a short list of pinned facts, and, for long-lived sessions, old turns retrieved on demand.

Most “my bot forgot” bugs are not model bugs. They are bugs in the code that builds the request. The fix is to manage that code on purpose, not to buy a bigger window.

Why does an LLM forget earlier turns?

A chat API is stateless. Anthropic’s context window guide describes the context window as the model’s “working memory”: everything in the request counts, including the system prompt, every message, tool results, tool definitions and the reply being generated. Nothing carries over from the last call unless you send it again.

That means “memory” is something your app builds. Three common ways it breaks:

One correction to a common picture: the API does not quietly drop old notes off a whiteboard. If the input alone is too long, the API returns an error (a 400 “prompt is too long”). The silent forgetting comes from trimming code that you or your framework wrote, or from chat interfaces that roll the window forward. Either way, you did not choose what to lose, and that is the bug.

Why a bigger context window does not fix it

Window sizes are large now. Anthropic’s docs list a 1M-token window for many current Claude models, and other models at 200k. It is still the wrong first move, for two reasons.

Cost grows with every resend. Input is billed per token, and without any memory strategy you resend the whole history on every turn. Here is an illustrative conversation. Assume a 500-token system prompt and about 300 tokens added per turn (user message plus reply). These are round numbers for the arithmetic, not measurements.

By turn 50 you have sent about 407,500 input tokens to carry about 15,000 tokens of new content. The total grows with the square of the number of turns, because each turn resends everything before it. For how to price this for your own prompts, see what LLM prompts actually cost.

Attention degrades, even when everything fits. Anthropic’s docs say that as token count grows, accuracy and recall degrade, which they call context rot. The research behind this is the Lost in the Middle paper, which found that performance is often highest when the relevant information is at the beginning or end of the input, and drops when it sits in the middle. That held even for models built for long contexts. A bigger window gives you more room to lose things in.

Prompt caching helps with price. As of October 2026, Anthropic’s prompt caching docs list cache reads at a fraction of the normal input price (0.1x for most models, 0.05x on some newer ones), with a default cache lifetime of 5 minutes. But the docs also say cached tokens still occupy the context window. Caching changes what you pay, not what the model has to read through.

What are the main ways to manage LLM memory?

There are four, and you usually stack them in this order.

1. Keep the full history (the buffer)

Append every turn to every request. It costs nothing to build and recalls perfectly. It is fine for short sessions and prototypes. The table above shows where it stops being fine.

2. Keep recent turns plus a rolling summary

When the history crosses a token threshold, compress the older turns into a summary and keep only the last few turns word for word. Think meeting minutes plus the last five minutes of transcript.

A summary that only compresses by time will drop exactly the facts that are easy to lose. Make it carry four things forward every time it is rewritten:

• the user’s stated goal
• decisions already made
• open items still waiting on someone
• persistent facts about the user, such as name, plan or stated preferences

Skip one and the bot re-asks a question it already had the answer to.

The other trap is a summarizer that reads only the current window and overwrites its old output. On the second round it forgets what the first summary knew. Feed the previous summary back in as part of the input, and ask for an updated one.

SUMMARY_LIMIT = 3000   # tokens of history before we compress
KEEP_VERBATIM = 6      # most recent messages kept word for word

def build_request(summary, history, user_msg):
    history = history + [{"role": "user", "content": user_msg}]
    if count_tokens(history) > SUMMARY_LIMIT:
        old, recent = history[:-KEEP_VERBATIM], history[-KEEP_VERBATIM:]
        summary = summarize(previous_summary=summary, new_turns=old)
        history = recent
    system = BASE_PROMPT + "\n\nConversation so far:\n" + summary
    return system, history, summary
Copy

The key line is previous_summary=summary. The summarizer sees its own last output, so a fact survives many compressions.

3. Pin facts outside the summary

Some things should never pass through a summarizer: the user’s name, their plan, a hard constraint such as “answer in Spanish”. Store them as a short list of key facts and put them in the system prompt on every call. Summaries lose detail. A pinned list does not.

This is also the difference between two kinds of memory. Episodic memory is what happened in this conversation. Semantic memory is lasting facts about the user that survive across sessions. Assistants that feel sharp have both. Most chatbots have only the first.

4. Retrieve old turns on demand

When a session spans days, or a user comes back next week, store past turns and facts in a database and fetch only what is relevant to the current message. It is the same pattern as RAG, covered in Understanding RAG, applied to a conversation instead of a document set. You can check how well it works with the ideas in measuring whether a RAG answer is grounded.

The catch is the query. A follow-up like “show me another one” has almost no search signal. Build the query from the rolling summary plus the last two or three verbatim turns, not from the latest message alone.

What do the vendors give you now?

You do not have to build all of this by hand. As of October 2026, Anthropic documents three building blocks:

• Compaction. Server-side summarizing of earlier conversation so it can continue past the window limit. The context window guide calls it the primary strategy for long-running conversations, and lists it as a beta feature for Claude 4.6 and later models. See the context window guide.
• Context editing. Clears old tool results, and optionally old thinking blocks, in agent workflows. Useful when tool output, not chat text, is what fills the window.
• The memory tool. Claude stores and reads files under a /memories directory, but your application runs the file operations and owns the storage. That makes it a way to keep facts across sessions without keeping everything in the window. The memory tool docs suggest using it with compaction: compaction keeps the active context small, and memory keeps what must survive summarization.

These cover the same ground as strategies 2 to 4 above. Knowing the manual version first helps you debug the managed one.

Tips to get more out of your context

• Put what matters at the start or the end. Instructions and pinned facts go at the top. The user’s current question goes last. Do not bury key facts in the middle of a long block.
• Measure before you tune. Every API response reports how many tokens the request used, and there is a token counting endpoint to estimate a request before you send it. Set your summary threshold from real numbers, not a guess.
• Trim tool output. A single large tool result can outweigh twenty chat turns. Keep only the part the next step needs.
• Retrieve less, not more. Five relevant snippets beat twenty loosely related ones.
• Cache the stable part. The system prompt and pinned facts barely change between calls, so they are good candidates for caching.
• Test for forgetting. Write a script that plants a fact on turn 3, runs 40 more turns, and asks about it. If the answer is wrong, you have a failing test, not a vibe.

Which strategy should you pick?

Treat the switch from one row to the next as a number-driven decision. Track tokens per call, and add summarization when the cost of resending history passes the cost of one extra summary call. Do not add retrieval because it sounds more advanced. Add it when a session truly has to outlive one sitting.

Next step: build it and test it

Reading the strategies is different from watching a summarizer drop a fact on its second compression round and then fixing it. Session 4 of the AI Engineering interview-prep course covers context and memory management, and it builds on the retrieval work in Session 3 on embeddings and RAG.

If you would rather build a small project, the context window manager and persistent agent projects sit on the AI engineering fundamentals path. For a quick check of where you stand, try the AI/ML readiness quiz. Agents that use tools hit this problem hardest, so how AI agents call tools is a good companion read.