Context Windows Are Not Memory: Engineering Recall for AI Agents

Diego Vallejo • July 20, 2026 • 5 min read EN

1. The Problem: The "Infinite Context" Fallacy

Every few months a model ships with a bigger context window, and every few months teams respond the same way: they stop designing memory and start stuffing prompts.

The reasoning seems sound — "if the window fits 1M tokens, just put everything in it." It is not sound. It confuses storage with recall, and it fails in three predictable ways:

Attention Dilution: Retrieval quality degrades as the window fills. The "lost in the middle" effect is not a bug to be patched; it is a structural property of attention over long sequences. Your critical instruction on line 40,000 competes with 39,999 lines of noise.
Cost Linearity: Tokens are billed per request, not per session. A 200K-token prompt replayed across a 30-turn conversation is not 200K tokens — it is 6M tokens. Prompt stuffing turns your context window into a recurring tax.
Non-Determinism Amplification: The more irrelevant context you include, the larger the space of plausible-but-wrong completions. Noise in, entropy out.

Axiom: The context window is a register file, not a hard drive. It is the working set of the current inference step — nothing more. Treating it as long-term storage is the same category error as writing your database to CPU registers.

2. The Solution: Tiered Memory with Deterministic Eviction

The fix is the same one hardware engineers made 60 years ago: a memory hierarchy. Fast, small, expensive storage at the top; slow, large, cheap storage at the bottom; and explicit, deterministic policies for what moves between tiers.

The Tiers:

Tier	Analog	Contents	Lifetime
L0 — System Frame	Registers	Role, constraints, output contract	Every request
L1 — Working Set	Cache	Last N turns, active task state	Current task
L2 — Episodic Store	RAM	Summarized past episodes, decisions	Session
L3 — Semantic Store	Disk	Embedded documents, facts, code	Permanent

Only L0 and L1 are ever injected verbatim. L2 and L3 enter the prompt exclusively through retrieval, and retrieval is governed by a token budget, not by vibes.

2.1 The Retrieval Budget

Every request gets a fixed context allocation, partitioned explicitly:

interface ContextBudget {
  total: number; // hard ceiling, e.g. 12_000 tokens
  system: number; // L0: fixed frame, ~800
  workingSet: number; // L1: recent turns, ~4_000
  episodic: number; // L2: summaries, ~2_000
  semantic: number; // L3: retrieved chunks, ~4_000
  reserve: number; // output headroom, ~1_200
}

function assemble(budget: ContextBudget, memory: MemoryTiers): Prompt {
  const frame = memory.l0; // always included, never truncated
  const turns = takeUntil(memory.l1.reverse(), budget.workingSet);
  const episodes = rankAndTake(memory.l2, budget.episodic);
  const chunks = rankAndTake(memory.l3.query(turns.last), budget.semantic);
  return compose(frame, episodes, chunks, turns);
}

The key property: assembly is a pure function of memory state and budget. Same inputs, same prompt. You can snapshot it, replay it, and diff it between runs. If the agent misbehaves, you inspect the assembled prompt — not a black box of accumulated conversation.

2.2 Eviction Is Summarization, Not Deletion

When the working set (L1) exceeds its budget, naive implementations truncate — silently deleting the oldest turns. This is how agents "forget" the user's name mid-conversation.

Correct eviction is a demotion with compression: evicted turns are summarized into an L2 episode before removal.

async function evict(l1: Turn[], l2: EpisodicStore, budget: number) {
  const overflow = splitAtBudget(l1, budget);
  if (overflow.length === 0) return l1;

  const episode = await summarize(overflow, {
    preserve: ['decisions', 'entities', 'unresolved_questions'],
    maxTokens: 300,
  });

  l2.append(episode); // demote, don't delete
  return l1.slice(overflow.length);
}

The preserve contract matters. A summary that loses decisions or open questions is corruption, not compression. Treat the summarizer as a lossy codec with a spec, and test it like one.

Field note: The single highest-ROI change I have made to production agents is replacing "truncate oldest" with "summarize-then-demote." It costs one cheap model call per eviction and eliminates the entire class of "the agent forgot what we agreed on" bugs.

3. Interactive: Watch the Hierarchy Work

The demo below simulates a conversation filling the working set. Watch what happens when L1 hits its budget: turns are demoted into a compressed L2 episode instead of vanishing. Toggle to "Naive Truncation" to see the same conversation lose information.

Open in new tab ↗

4. Retrieval Ranking: Recency Is Not Relevance

L3 retrieval is where most RAG systems quietly fail. Cosine similarity alone ranks what sounds like the query, not what the task needs. A production ranker should be an explicit, weighted function:

$scor e (c) = w_{s} \cdot s im (q, c) + w_{r} \cdot d ec a y (t_{c}) + w_{p} \cdot p in (c)$

$s im (q, c)$ — semantic similarity between query and chunk.
$d ec a y (t_{c})$ — exponential recency decay, because yesterday's stack trace outranks last quarter's.
$p in (c)$ — a manual override channel. Some facts (API contracts, user preferences, safety constraints) must be retrievable regardless of similarity. If your memory system has no pinning mechanism, it has no guarantees.

Weights are configuration, not folklore. Log every retrieval with its score breakdown, and you can answer "why did the agent bring that up?" with data instead of shrugs.

5. Closing: Memory Is an Architecture, Not a Parameter

Model vendors will keep selling bigger windows, and bigger windows are genuinely useful — as larger L1 caches. They do not eliminate the hierarchy any more than a bigger CPU cache eliminated RAM.

The agents that behave coherently across hundred-turn sessions are not the ones with the largest windows. They are the ones where a human sat down and answered, explicitly:

What is always in context? (L0)
What is in context now? (L1, budgeted)
What can be recalled? (L2/L3, ranked)
What happens at eviction? (summarize-then-demote, with a preservation contract)

Answer those four questions in code, and "the agent forgot" stops being a bug report — because recall is no longer an accident of window size. It is a system you designed.