context›window.py
window.py
Hints
Picture a group chat you've had open for months. Your phone doesn't keep every message loaded forever, that would be wasteful, so it only keeps the most recent stretch ready to scroll and quietly lets the older ones drop out of active memory.
Early on, someone pinned something important. Not a formal pin, just a message: "reminder, don't post the venue address in this thread, DM it individually." Everyone saw it. It made sense at the time.
Two hundred messages later, someone new asks where the venue is. And someone answers, right there in the thread, full address. Not because they're being reckless. They've genuinely never seen that reminder. It scrolled out of what got kept loaded, long before they joined.
That's what a language model's context window does, except it happens on every single request, not once. A model doesn't remember a conversation the way a person does. Each time it responds, something assembles everything it's allowed to see this turn, instructions, chat history, any documents pulled in, and hands over that whole bundle fresh. If the bundle is too big to fit, something gets left out.
context = build_context(history, max_tokens=650) # 12 turns in, len(context) looks fine. The system prompt just isn't in it anymore.
No error thrown. No warning logged. Just a slightly shorter bundle, missing the one part nobody thought to check for.
So far, that's the pipeline working as intended: everything gets assembled and handed to the model fresh, every single turn. This concept starts one step later, at the moment that assembly stops fitting.
A context window is a token budget, not a message count. Every model has a fixed limit on how much text it can be shown at once, measured in tokens, not characters and not turns. And every single thing you send it shares that one budget: the system prompt, the entire conversation history, any documents pulled in for this turn. There's no separate, protected lane for instructions. It's all just text competing for the same fixed number of seats.
Something has to be cut, and cutting is usually blind. The simplest truncation strategy is a sliding window: keep the most recent tokens, drop whatever's oldest, first in, first out. It doesn't read the content to decide what matters. It checks position in the list, nothing else.
The oldest thing in the buffer is usually the system prompt. It's written once, at the very start, before a single user message exists. In a sliding window, "oldest" and "first evicted" are the same list position. The instruction meant to govern the entire conversation is, structurally, the first thing in line to be forgotten.
The trap: a system prompt isn't a persistent setting, it just reads like one. Anyone who's written a strong system prompt tends to treat it like a config file: set it once, trust it for the life of the conversation. It never announces that it's stopped applying. The instruction is still sitting right there in your code, in full, exactly as written. It's just not in the box being handed to the model anymore.
Don't take our word for it. Grow a conversation and watch the window fill up.
Grow the conversation and watch what fits. Toggle the truncation strategy to see the difference between dropping oldest-first and reserving room for instructions.
Drag the conversation past the point where the system prompt gets evicted, then switch to "Reserve budget" and confirm it survives, to unlock the problem stage.