Google's SLEEP framework vs agent memory files: weights drop the context tax and the address with it

Google Research put a post up on 27 October about a framework it calls sleep (I will write it SLEEP, to keep it apart from the verb), and the headline numbers traveled fast: a model that keeps learning after deployment, scores near-perfectly on a long-context benchmark out to 10 million tokens, and does it by moving what it saw into its own weights. The paper behind it is older than the news. It has been on arXiv since June, and its comment line says a version has been public on OpenReview since September 2025. What is new is the blog post. That is worth knowing before you treat it as this week’s breakthrough, and it is not a reason to dismiss it. The question it puts to anyone building memory for agents is a good one.

If a model can turn context into parameters, why keep notes at all? Earlier this month I argued that moving agent memory into Markdown files buys you a memory you can read, and that readable is not the same as current, consulted or agreed on. Weight memory goes the other way. It gives up readability entirely and, in exchange, fixes two problems files cannot. My read is that the trade is right for some of what an agent knows and wrong for the rest, and that the line between the two has nothing to do with size or speed. It is about whether you will ever need to point at one thing and remove it.

What Google’s SLEEP framework does to the weights

The mechanism is more specific than the neuroscience vocabulary suggests, and the authors say so themselves:

our framework operates entirely within a fixed model architecture and human-defined training rules.

While the model is awake, new information lives in its fast components: attention over the context, plus high-frequency MLP blocks that update quickly. Sleep is not triggered by anything the model notices. It runs on a schedule, every so many steps. When it runs, the model samples a dataset from itself as the teacher, adds a new low-rank expert to the next, slower block, and trains that expert by distillation until the student reproduces what the teacher would have said. The blog calls this knowledge seeding. Then the faster block’s own base weights are updated, the experts it had accumulated from earlier sleeps are reset, and the new expert one level up is switched on.

The second stage, dreaming, is where self-improvement comes in. Each router also fires a random expert to mix domains, the model generates synthetic practice samples, and the samples are screened by a gradient-based importance score, the top ones kept along with a few random ones for diversity. Each kept dream fine-tunes an isolated copy of the model, and the dream is rewarded only if that copy does better downstream. In the paper’s knowledge-incorporation ablation, the base model scores 31.9, the full method 48.9, the method without dreaming 35.7. In that experiment, at least, dreaming carries most of the margin.

The numbers are the authors’ own. Near-zero drop across stages on continual translation from Manchu and Kalamang into English, where in-context learning and plain fine-tuning forget badly. Near-perfect BABILong scores to 10M tokens, though the small models in that chart, SLEEP’s and Titans, were fine-tuned on the benchmark while GPT-4 was evaluated zero-shot. 48.9% and 46.2% on two factual-incorporation settings, about two points ahead of SEAL on the first. 79.2%, 69.0% and 46.1% on AIME 24, AIME 25 and HMMT 25 with Qwen3-8B. The base models run from 1B to 8B parameters. There is no code link, no limitations section on SLEEP itself, and no absolute cost for a sleep cycle. The efficiency appendix reports that plain fine-tuning is four times more efficient at equal step counts and needs 3.6 to 4.8 times the wall-clock to match SLEEP on the three reasoning benchmarks. The results may well hold. The sources just offer no independent replication yet, which is where most papers are in their first months.

What weight memory fixes that agent memory files cannot

Two things, and I would not wave either away.

The first is the context tax. A memory kept as text has to be read back into the window to be used, every session, and the bill is paid whether or not the session needed most of it. Claude Code loads only the first 200 lines or 25KB of its memory index at startup, and every harness that loads notes makes the same trade between what it carries and what it spends. It is the token tax on tools you never call, paid for facts instead of schemas. A fact in the weights costs no context tokens per call. The cost moves into parameters and training, but it stops scaling with how much the agent remembers.

The second is the file nobody opened. In a file memory the agent decides which document to read by judging a one-line description against the task, and when that judgment misses, the file stays closed and nothing records it. Weights take the agent’s decision out of that loop. They do not take out the failure entirely: a router that does not fire the right expert, or a model that holds a fact and fails to use it, is the closed file one level down. But no step depends on the agent having thought to look.

Agent memory needs an address, and weights do not have one

Everything an operator does to a memory is a verb with an object. Read this entry. Diff it against last week. Revert that change. Delete what this person asked you to forget. Refuse this note before it is stored, because it came from a web page and reads like an instruction. Each of those needs something to point at. A file memory has addresses everywhere, which is most of what I meant by calling it readable.

A consolidated fact has no address. In SLEEP it is not stored anywhere in the sense a file stores it. It is a direction in a low-rank update, learned by making the student imitate samples that happened to carry it, and mixed with whatever else those samples carried. You can ask the model about it and see what comes back. You cannot list it, and an answer that no longer contains it is no proof that it is gone. Staleness, the problem I spent most of the earlier essay on, gets worse here for the same reason: a file at least records when it was last written, and a direction in a weight matrix has no field for that unless you keep one beside it.

SLEEP’s rollback unit is a checkpoint, not a night

It is tempting to read the architecture as keeping one handle. New knowledge lands in a newly added low-rank expert, and the blog says it is stored there without disrupting older parameters. So surely last night’s consolidation is a block, and a bad night can be dropped the way you unload an adapter.

The paper’s own update order says otherwise. A sleep step does three things together: it updates the base weights of the faster block that is handing knowledge on, it resets the low-rank experts that block had accumulated from earlier sleeps, and it activates the new expert one level up. Dropping the new expert removes what was consolidated, and leaves the faster block with changed weights and none of the experts it had before. Dreaming then trains on top of all of it. There is no piece of the model you can take out to undo one night. The paper does not discuss rollback, so this is my reading of its update rule, and I think it holds.

What you can roll back to is a checkpoint, a saved copy of the whole state from before. And the hierarchy makes checkpoints the only boundary that survives. Each consolidation into a slower level merges what several earlier sleeps put into the faster one and then erases them as separate pieces. In the live model, the line between two earlier sleeps is gone once both have moved a level up. A checkpoint from Monday still has it. A file memory with history in Git keeps every old change identifiable and inspectable, even when reverting one means working through the edits that came after it.

And no unit is ever a fact. You can return to Monday. You cannot remove one sentence a user asked you to forget and keep everything since. Machine unlearning is a real research field, and model editing can change a specific association in place, but neither proves for a model like this that a particular fact is gone, and SLEEP offers neither.

Who decides what SLEEP consolidates into the weights

In my earlier essay the weak spot of a document memory was the write. Loading is enforced by the harness, but saving is an instruction to the agent, and a prompt is not an invariant.

SLEEP closes that gap by removing the decision. Sleep runs on a step count, the consolidation set is sampled from what the model now holds, and the only selection in the pipeline is in dreaming, where samples are kept for how much they help performance. Nothing in it asks whether a thing is true, whose it is, or where it came from. That is more honest than a memory tool the agent is asked to use with good judgment, since there is no pretense of judgment at all. It is also the weakest point of the design for anyone running it on real inputs.

Think about the note that should never have been stored. In a file memory, an injection that lands in a stored note is at least a line. It comes back weeks later with the standing of the agent’s own words, which is bad, but you can grep for it and delete it. In a weight memory, text that sat in the context during waking shapes the fast weights, and the consolidation set is sampled from a model those weights have shaped. Whether a given injection survives into the slower blocks is an empirical question the paper does not test. If it does, there is no line to grep. You can only ask the model questions and hope you ask the right one.

Agent memory design: weights as a cache compiled from a record

The answer I gave last time was a record that keeps its history with a readable view on top. Weights fit the same split, as a second view over the same record, and the split decides whether a weight memory is safe to run at all.

Keep the record first: whatever entered the context during waking, with an address and a date, kept after it has been consolidated. That record is also the only place the refused note can be refused. Filtering has to happen before text reaches a session that learns, because after that the model has already moved. Then treat the weights as a cache compiled from the record and a checkpoint. To delete a line, take it out of the record and replay waking, sleep and dreaming from a checkpoint saved before the line went in. The result is not the same model minus one fact, since the sampling and the random experts differ on every run. What it guarantees is narrower and more useful: that line never went in.

The cost of that replay is set by the line’s age, not by how often you checkpoint. Forgetting yesterday means replaying a day. Forgetting something from last spring means replaying everything since last spring. The older a memory, the more it costs to remove, which is the opposite of a file, where an old line deletes as cheaply as a new one. A consolidation whose inputs were thrown away once they were distilled has no replay at all, and its mistakes can be patched at best, never removed.

That cost curve is where the line falls. Put into weights what you will not need to remove piece by piece: a house style, fluency in a domain, the shape of a codebase that changes slowly. Keep as records whatever has an owner who can ask for it back, or a truth that can expire, such as facts about people, decisions and who made them, and anything that came from outside unchecked.

SLEEP is a serious attempt at a real problem, and if the numbers hold up outside the authors’ lab it will change what agent memory costs. What an operator needs from a memory stays the same. A model that learns while it works will learn something it should not, and the record is how you find out what.