What agent memory actually is, and why the context window isn't it

Agent memory is whatever a system keeps between calls: held somewhere outside the model, and put back in front of it on the next turn. The context window is not that. It is the working area for one call — the space a model reads before it produces an answer — and it holds nothing once the answer is finished.

Confusing the two is expensive rather than pedantic. A team that treats the window as storage ships a product that forgets, discovers that it forgets, and then buys a bigger window, which does not help. Most of the useful questions about agent memory follow from getting this right first.

The model starts from nothing

A language model has no state. Each call begins with nothing carried over from the one before it, and the only reason a conversation feels continuous is that the application resends the earlier turns every time. The 2023 paper most of the field’s memory vocabulary comes from (Cognitive Architectures for Language Agents, §4.1) opens on exactly this: “Language models are stateless: they do not persist information across calls.”

LangChain, writing for practitioners rather than reviewers, says the same thing more bluntly: “LLMs themselves do NOT inherently remember things — so you need to intentionally add memory in.”

The illusion is cheap to produce, which is why it is so easy to mistake for a feature. Tell a chatbot your name, ask for it two turns later, and it answers correctly. Nothing inside the model changed between those turns. The application kept a transcript and pasted it back in front of a model that was, as far as it was concerned, meeting you for the first time.

Nothing in the model remembers. Everything that looks like memory is machinery outside the model, doing work.

What the window actually holds

The context window is a token budget for a single call, and by the time a user’s question arrives a large share of it is already committed. Two independent reverse-engineering analyses have measured what a production coding agent sends before the user’s prompt is processed at all. One puts Claude Code’s first-turn payload at roughly 32,800 tokens: about 6,500 for the system prompt, about 24,000 for tool schemas across 27 tools, and about 1,900 for injected scaffolding. The other, measuring a different boundary and almost certainly a different version, finds the system prompt — instructions, tool definitions and a project instruction file together — sitting at a consistent ~14,328 tokens on every call.

Tool schemas are the largest single item in the larger measurement, and that is worth sitting with: the biggest fixed cost in that window is a description of what the agent is permitted to do, not anything the user said.

One number from that comparison travels without its label. The same analysis measures OpenCode, a different coding agent doing the same task, at about 6,900 tokens — and that figure is sometimes quoted as a low-end estimate for Claude Code. It is not one. It belongs to a different product with 10 tools instead of 27.

The window is refilled every call, not retained between them.

Bigger is not the fix

A bigger window does not fix forgetting, and it does not reliably fix recall either. Anthropic’s engineering team, writing on 29 September 2025, defines the failure directly: “As the number of tokens in the context window increases, the model’s ability to accurately recall information from that context decreases.” The constraint underneath it, in the same piece, is an attention budget — “Every new token introduced depletes this budget by some amount.”

Chroma Research tested 18 models in July 2025 and found that “models do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows.” Chroma sells vector databases, and “long context is not a free lunch, you still need retrieval” is a convenient thing for a retrieval vendor to establish. The data is multi-model and the method is disclosed, so the finding stands — but the funder belongs next to it. I went back to the report to check one thing, because this is a number people invent: it names no degradation threshold at all. The round “300–400K token” figures circulating in secondary coverage are not in it.

A May 2026 measurement puts a number on the same shape with no vendor attached to the result. On a monitoring task — spotting a dangerous action inserted into a long agent transcript — one frontier model’s recall of that action fell from 99.7% when it sat inside 100K tokens to 69% when the identical action sat inside 800K.

A larger window buys room, not reliability.

So where does memory live

Anything that has to survive the end of a call lives outside the model and gets re-supplied on the next one. Anthropic describes the technique it recommends for long-horizon work without any ambiguity: “Structured note-taking, or agentic memory, is a technique where the agent regularly writes notes persisted to memory outside of the context window. These notes get pulled back into the context window at later times.”

Notes persisted outside the window are not part of any call until something fetches them and puts them there. A stored memory that nothing retrieves has no effect on any answer the system gives, which is a failure mode teams reliably discover some time after they have finished building the store.

LangChain organises the same mechanics as four moves — write, select, compress and isolate — where write means saving something outside the window and select means pulling it back in when it is needed. Two different vendors, two vocabularies, one arrangement: a place that is not the model, and a step that moves things between there and the window.

Memory is a store plus a fetch. Neither half does anything alone.

What has to be true for any of this to work

Three separate things have to happen before a system can be said to remember: something writes, something stores, and something retrieves at the right moment. They are three mechanisms, not three names for one, and each fails in its own way — a write that never fires, a store that fills with duplicates, a retrieval that returns the wrong turn from six weeks ago.

Two further pieces here take the first two of those. CM-007, How agent memory gets written: hot path versus background, is about the write and the two moments it can happen in. CM-006, The three types of agent memory: semantic, episodic, procedural, is about the vocabulary the field borrowed for sorting what gets stored.