Context Window Manager

Every LLM has no memory of its own — it sees only what you send it on this call, and nothing else. The moment a conversation runs past a handful of turns, you have to decide how to manage that history yourself, or the assistant either forgets things it was just told or the cost of every call quietly balloons. This project is where you design and build that decision, not just recite it in an interview.

What you’re building

context-window-manager extends a retrieval-backed assistant (Project 3’s territory) with a memory layer that decides, on every turn, whether to keep appending to a simple in-context buffer or to compress older turns into a structured summary — one that explicitly tracks the user’s goal, key decisions, open items, and stated preferences, not just a narrative recap. The same system also draws the line between memory that’s scoped to this one conversation (episodic) and facts that should persist across sessions (semantic) — most assistants only ever build the first kind, which is exactly why the second kind is what makes an assistant feel like it actually knows the user.

How a turn is handled

The two decision points — “over the summarization threshold?” and “should this fact persist across sessions?” — are exactly the two places a naive chat loop either quietly forgets something or quietly gets expensive.

Core concepts, three levels deep

1. Context window and the in-context buffer

Definition: the context window is the fixed per-call token budget — everything you send (system prompt, history, retrieved data, current message) has to fit inside it, on every single request. The in-context buffer is the simplest possible memory strategy: append the full conversation history to every call, with zero extra setup.
In this project: the buffer is your starting point and it works fine for short sessions — the failure mode only shows up once history grows past what fits, and it fails silently: the oldest turns get dropped with no signal to you or the model.
Practical consequence: never let the buffer grow unbounded and unmonitored. Decide a threshold before you ship, not after a user notices the assistant “forgot” something.

2. Summarisation memory and “lost in the middle”

Definition: summarisation memory compresses old turns into a compact structured summary once a threshold is crossed, then replaces those turns with the summary. “Lost in the middle” is a separate, related fact about attention: models recall the beginning and end of a long context more reliably than content buried in the middle — even when everything technically fits.
In this project: these two facts together are why the summary can’t just be “a shorter version of the conversation.” If it compresses chronologically instead of preserving specific decisions, the important fact can still get buried and effectively lost, the same way it would in an un-summarized but overly long context.
Practical consequence: design your summary format explicitly — user’s goal, decisions made, open items, stated preferences — instead of asking the model for a generic recap. A summary that only captures “what was discussed” is not doing the job.

3. Context poisoning

Definition: a failure mode where a harmful or incorrect instruction that enters early in a conversation — through user input, not the system prompt — gets folded into a summary and then keeps propagating through every later turn, because nothing in the pipeline distinguishes it from legitimate conversation content.
In this project: your summarization call reads raw user-turn text and produces something that gets injected back into the system context on every future turn — that’s exactly the path poisoning travels down if you’re not careful about what’s allowed to influence it.
Practical consequence: keep system-level rules in the system message layer only, and never let summarized user content get treated as an instruction. This has to be a code-level guarantee, not a prompt-level request.

4. Episodic vs. semantic memory

Definition: episodic memory is what happened in the current conversation — it resets at the end of the session and is what your in-context buffer and summarization both implement. Semantic memory is persistent facts about the user (stated goal, experience level, preferences) that survive across sessions, stored separately and injected as system context at the start of a new one.
In this project: the summarization layer only ever gives you episodic memory. Semantic memory is a deliberate, separate write — extracting a durable fact and saving it somewhere that outlives this conversation object.
Practical consequence: most chatbots only ever implement episodic memory, which is why they reset to a blank slate every session. The ones that feel like they “know you” are the ones that deliberately built the semantic layer on top — it doesn’t happen automatically.

Decision rules

If…	Then…
Sessions are short (under ~20 turns) or stateless/one-off	Use a plain in-context buffer — zero setup, and you don’t need anything more sophisticated yet
History is approaching a token threshold (start around 4K tokens / a handful of turns)	Trigger summarisation — compress older turns into a structured summary, keep the most recent turns verbatim
A follow-up message is vague or reference-heavy (“show me something different”)	Build the retrieval query from the rolling summary plus the last 2-3 verbatim turns, not the bare message alone
A fact should be true of the user in every future session, not just this one	Write it to a separate semantic memory store and inject it as system context at session start — don’t rely on episodic summarization to carry it forward

Common mistakes

Letting the in-context buffer grow unmonitored until it silently truncates instead of deciding a threshold and summarizing before you hit it.
Summarizing chronologically instead of structurally — a summary that just compresses “what happened” can still bury the one decision that mattered, the same way an unsummarized long context can.
Letting raw user-turn text influence system-level behavior through an unguarded summarization pass — this is exactly how context poisoning propagates.
Treating “use a bigger context window” as a fix for either forgetting or cost — it doesn’t address lost-in-the-middle, and it makes every call more expensive, not less.

Key concepts at a glance

Concept	One-line definition	Why it matters for `context-window-manager`
In-context buffer	Append full history to every call, zero setup	The simplest strategy, and the one that silently fails once it overflows
Lost in the middle	Models attend well to the start/end of a long context, poorly to the middle	Why a summary has to preserve specific decisions, not just compress narratively
Context poisoning	An early bad instruction propagates through every later summary	Why system rules must stay isolated from summarized user content
Semantic memory	Persistent user facts injected at session start, stored separately from episodic history	The deliberate extra step that makes an assistant feel like it remembers you across sessions

Go deeper: Context & Memory Management — the full concept walkthrough (lost in the middle, summarisation design, episodic vs. semantic memory) →

Interviewers ask 'how do you manage memory in an LLM application when conversations get long?' by default for any AI Engineer role — this project is where you build a real, defensible answer instead of a rehearsed one.

Clone this project, run it locally, and compare your output against the description above.

Context Window Manager on GitHub ↗

Self-check only — no score is recorded anywhere. Answer each question yourself, then click to reveal.

1. A support assistant handles billing questions. On turn 14, a user says "like I told you earlier" — but the assistant asks them to repeat everything, even though they described the issue clearly on turn 2. What's the most likely root cause?
- a) The model has a random memory bug
- b) The in-context buffer overflowed and the oldest turns — including turn 2 — were silently dropped to make room
- c) The API connection reset between calls, wiping history
- d) Temperature is set too high, causing the model to ignore prior context
Reveal answer

B. An in-context buffer has no overflow warning — it just keeps appending turns until the context window is full, then the oldest turns fall off with no signal to the model or the user. This is exactly the failure mode summarisation and retrieval strategies exist to prevent.
2. Your assistant's context window comfortably fits the full 40-turn conversation, but it still seems to have forgotten a decision made on turn 18. Truncation isn't the cause — everything technically fits. What's the more likely explanation?
- a) The tokenizer failed to encode turn 18 correctly
- b) "Lost in the middle" — models attend more reliably to the start and end of a long context than to content buried in the middle, even when nothing was dropped
- c) The system prompt overwrote the conversation history
- d) Turn 18 exceeded the model's per-message token limit
Reveal answer

B. Fitting inside the context window is necessary but not sufficient — attention quality degrades toward the middle of a long context. This is why a good summary preserves key decisions explicitly instead of just compressing turns in chronological order: chronological compression can still bury the decision that mattered.
3. In 1-2 sentences: why is "just use a model with a bigger context window" a weak answer to "how do you handle long conversations," even though it's technically true that it would fit?

Reveal answer

Model answer: a bigger window doesn't fix the two real problems — cost (a 200K-token context costs roughly 40x more per call than a 4K one when full, and you pay that on every single request) and attention quality ("lost in the middle" gets worse, not better, as the context grows). Growing the container doesn't fix retrieval quality inside it.
4. Your assistant summarizes conversation history every 4 turns, but a support ticket mentioned on turn 3 later contradicts information the assistant confidently states on turn 40. What is the most likely gap in the summarization design?
- a) The summarization threshold of 4 turns is too aggressive and should be raised to 40
- b) The summary compressed the conversation narratively without explicitly preserving the specific decision or fact from turn 3, so it got lost in a later re-summarization pass
- c) Summaries should never be regenerated once created
- d) The model was run at too high a temperature during summarization
Reveal answer

B. A summary that only compresses "what was talked about" loses exactly the kind of concrete fact or decision the model needs to stay consistent later. A production-grade summary should explicitly track: the user's goal, key decisions made, outstanding open items, and any persistent user facts — not just a narrative recap.
5. What is context poisoning, and what's the one design rule that prevents it?

Reveal answer

Model answer: context poisoning is when a harmful or incorrect instruction that entered early in a conversation — deliberately or by accident — gets folded into a summary and then keeps propagating through every later turn, because the summarizer treats it as legitimate conversation content. The prevention rule is to keep system-level rules in the system message layer only, never let raw user-turn content get promoted into it, and validate that summarization doesn't quietly absorb instruction-like text from the user's side of the conversation.
6. A user says only "actually, show me something different" with no other context, and this needs to trigger a retrieval query against an external memory store. Using just that message as the query will likely fail. Why, and what should the query be built from instead?
- a) It will fail because retrieval queries can't contain the word "different" — rephrase the message first
- b) It will fail because the message is pronoun/reference-heavy with no retrieval signal on its own — build a composite query from the rolling summary plus the last 2-3 verbatim turns instead
- c) It won't fail — the latest message is always the best retrieval query
- d) It will fail because retrieval queries must be at least 20 tokens long
Reveal answer

B. Vague follow-ups carry conversational intent, not retrieval signal, on their own. A composite query — the rolling summary plus the last few verbatim turns — reconstructs enough context for retrieval to find the right thing, which the bare message "show me something different" cannot do by itself.

These three tasks build on Project 3's retrieval layer and turn it into something that actually holds a conversation without either repeating itself or quietly losing track of what it already told the user. Do Task 3 even if you're tempted to skip it — the token-count drop is the part that makes summarisation feel real instead of theoretical.

1. Wire up retrieval and confirm the baseline
20-30 min

Confirm you can search a small knowledge source (your Project 3 retrieval pipeline, or a simple in-memory list of 5-10 items if you want to isolate this project) and get back 2-3 relevant results for a query, then pass those results into an LLM call as context. This is the thing memory management sits on top of — get it working stand-alone before adding history.
2. Add summarisation memory with a real threshold
45-60 min

Build a `chat()` function that appends every turn to an in-context buffer, and once the buffer crosses a token/turn threshold (start at 4 turns), summarizes the older turns into a structured summary — not free prose — that explicitly captures: what the user is trying to do, what's already been shown or decided, what's still open, and any stated user preferences. Replace the summarized turns with that summary, keep the most recent 1-2 turns verbatim, and confirm a 6+ turn test conversation doesn't repeat something it already showed on an earlier turn.
3. Add token monitoring and watch the drop
20-30 min

Log input/output token counts on every call across the same 6+ turn test conversation. Confirm you can see the exact turn where summarisation triggers and the token count drops — this is the concrete evidence that the strategy is doing its job, not just a design you implemented on faith. Capture the log or a screenshot.

Deliverable

A working `chat()` function with summarisation memory, a token-count log showing the drop after the summarisation trigger fires, and one concrete example from your test conversation where the assistant correctly avoided repeating something it had already shown or decided.