Where Experience Goes

Learning the update rule

A learned update rule is a network whose output is a change to another network’s weights, or to its own. The idea is old. Schmidhuber’s fast weight programmers of the early 1990s were slow networks that wrote into the weights of fast ones. Three recent instances bear on continual learning.

Kazuki Irie, Róbert Csordás and Jürgen Schmidhuber meta-learned continual-learning algorithms directly, in a method they call Automated Continual Learning. Their learner is a self-referential weight matrix that rewrites itself with a delta rule as examples stream in. The rewriting rule is trained, across many sequences of tasks, so that after reading a sequence the matrix classifies examples from every task in it. The learned rule therefore has to solve the stability problem itself, since only an update that does not erase earlier tasks scores well. On split MNIST in the class-incremental setting, with five tasks and no replay, the learned rule reached 84.3% accuracy where elastic weight consolidation reached 19.8%. On harder image benchmarks it still trailed methods built on large pretrained models. The delta rule here is the same error-correcting update that runs inside the recurrent layers of current hybrid language models. Below, it reappears as the second rung of the ladder.

Hypernetworks make the learned update explicit. Text-to-LoRA, from Sakana AI, maps a natural-language description of a task to a low-rank adapter for a language model in one forward pass. Doc-to-LoRA maps a document to an adapter, so the model can answer questions about the document without the document in its context. Within the window, reading the document in context still does better: on SQuAD the adapter recovers about 82% of in-context performance. The adapter’s advantage appears past the window, where the base model cannot read the document at all, and in cost. Internalizing a document takes under a second, and the adapter needs megabytes where a long context needs gigabytes. Ilija Lichkovski made the point in his podcast interview that Doc-to-LoRA is a learned learning algorithm: a sample-efficient map from new content to a weight change. Nothing in its setup is shaped like continual learning, and yet it is exactly the consolidation step that Part 1 attributed to self-distillation, amortized into a single pass.

Lichkovski presented an experiment of his own in the same interview, as preliminary and unpublished, and its negative result is the instructive part. He trained a hypernetwork on split MNIST, the digit benchmark presented two classes at a time. Given the current weights and the identity of the next task, it was to output the weights that multitask training on all tasks so far would have produced. Multitask training was the oracle, and the hypernetwork was to reach it sequentially. It worked surprisingly well. Average accuracy rose with each new task toward the multitask level, while fine-tuning stayed at the one-fifth that total forgetting gives. Backward transfer was positive: earlier tasks improved after later ones were learned. Each task cost one forward pass at deployment. Then the ablations showed that the hypernetwork ignored the task identity. It had learned a general operator that moves any weight configuration toward the region of weight space where multitask solutions lie. The weights carried far more information than the small task embedding, so the operator attended to the weights.

That result fits a pattern The Given and the Found described for iterative reasoners (§6): supervision constrains a model only where training visits, and leaves the rest free. Here every training update moved toward multitask solutions. Nothing in the loss penalized ignoring the task identity, and the cheapest way to predict the updates was to learn that move, not to learn the tasks. The failure is also a finding. A learned “move toward multitask” operator is a learned stability mechanism, a consolidation step that does not need to know what it consolidates.

One more line connects these to composition. Lake and Baroni’s meta-learning for compositionality (MLC) trained a standard transformer on a stream of small compositional tasks, each built from invented words. On their pseudoword instruction task, where “dax” might mean a red circle and “dax fep” three of them, people produced the systematic answer about 81% of the time, and the network’s sampled answers about 82% of the time. Lichkovski’s proposed benchmark, discussed below, is a sequential version of the same task. The words arrive one at a time and must be composed later.

A ladder of stores

Here are the stores a deployed language-model agent can use, from fastest to slowest.

1. The context. Attention over the key–value cache is in-context learning. The forward pass writes it, and nothing in it is ever overwritten. It lasts for the session, or until entries are evicted or the history is compacted into a summary.
2. The recurrent state. Hybrid models such as Kimi K3 and Qwen3.8 interleave attention with gated delta-rule layers. Each such layer keeps a fixed-size matrix that every token updates by one error-correcting step. This is a fast-weight memory, and it is bounded.
3. Harness memory. Notes the agent writes, skill files, retrieved documents and optimized prompts persist indefinitely, as text.
4. Prediction heads. These are Part 2’s banks of general value functions: small learned predictors, updated online by temporal-difference learning, that answer questions about what would happen under various behaviours.
5. Adapters. Low-rank weight deltas, written by gradient descent or by one hypernetwork pass.
6. Core weights. Changed by fine-tuning, reinforcement learning or self-distillation.
7. The release cycle. A lab trains specialist policies and distils them into the next checkpoint.

Four questions separate the rungs: what writes the store, what persists in it and for how long, how much it holds before entries interfere, and what it costs to read.

Four claims follow from setting the stores out this way.

In-context learning is the best learner on the ladder, example for example. Part 1 cited Lampinen and colleagues: a model reasons from facts in its context, to reversals and syllogisms, far better than it reasons from the same facts trained into its weights. The question for every slower store is whether it can keep that quality once the content has left the context.

The recurrent state is parametric continual learning, already running inside frontier models. A delta-rule layer treats its state S as the weights of a linear map from keys to values. For each token it reads the current prediction S k_t, compares it with the token’s value v_t, and writes the correction:

S_t=S_{t-1}+\beta_t\,(v_t-S_{t-1}k_t)\,k_t^\top,

usually after a learned decay of S_{t-1}. This is one step of gradient descent on \|S k_t-v_t\|^2/2, the Widrow–Hoff rule of 1960. Schlag, Irie and Schmidhuber showed in 2021 that linear attention is a fast weight programmer, and introduced this write to keep its purely additive memory from saturating. What matters here is that the update is error-correcting, so it can revise. The temporal-difference update of How Can a Reward Train a Neural Network? has the same shape. With a linear value estimate \theta^\top\phi, TD(0) adds \alpha\,(r+\gamma\,\theta^\top\phi'-\theta^\top\phi)\,\phi. Set \gamma=0, and the target is the observed signal alone: the delta rule, with \phi as the key and r as the value. A delta-rule layer is a TD(0) learner with no future to bootstrap from. The GQ(λ) heads of Part 2 are the same learner given a future, a target policy and a trace. The second and fourth rungs of the ladder share an update rule, and they differ in their targets and in how long they keep what they learn.

Worked example G: overwriting a redefined word. Give a delta-rule memory two orthogonal keys, k_{\text{zup}}=(1,0) and k_{\text{dax}}=(0,1), and let the values be colours. Write zup → red, then dax → blue, then zup → green, each with \beta=1. The third write reads the current prediction for zup (red), and adds (\text{green}-\text{red})\,k_{\text{zup}}^\top. Now zup reads green and dax still reads blue: the redefinition replaced the old association in place. With \beta=0.5, zup reads half red and half green; each repetition of the new definition shrinks the remaining error by the factor 1-\beta\|k\|^2. Purely additive linear attention, which writes v\,k^\top without reading first, would make zup read red plus green. Now let the keys overlap: if k_{\text{dax}} had cosine 0.6 with k_{\text{zup}}, the same correction would move dax’s readout by 0.6 times (\text{green}-\text{red}). Interference is visible in one line. Fine-tuning a contradicting fact into core weights has no such targeted erase. The new fact is added by gradient steps that also move every other fact sharing the affected weights.

Simple online learners, such as stochastic gradient descent on linear models or recursive least squares, are therefore not free of interference. The Widrow–Hoff rule revises cleanly when keys are orthogonal and interferes in proportion to their overlap. A delta-rule layer is exactly such a learner, and short trigger strings are known to overwrite the hidden state of Mamba models irreversibly. The interference that makes continual learning hard in deep networks is not absent in simple learners. It is easier to see.

The capacity is bounded by the size of the matrix. Alex Wang’s account of associative memory makes the bound concrete for one criterion: recalling a clean value vector, with random keys. By that criterion a memory written by adding outer products, as purely additive linear attention does, recalls about d_k associations in d_k dimensions before interference swamps the signal. The least-squares memory that repeated delta-rule writes approach recalls about 2d_k. A looser criterion counts more. If the readout only has to pick the right answer from a fixed set, as next-token prediction does, a d_k\times d_v matrix holds on the order of d_k d_v associations, up to a logarithm (Nichani, Lee and Bietti). The logarithm is not small: with d_k=d_v=128, an outer-product memory picks out about 350 associations correctly and the best possible matrix about 1,160 (Barnfield and colleagues). An MLP’s nonlinearity suppresses interference, so the core weights hold far more than the residual width suggests, in superposition. Softmax attention over a context recalls vastly more, about a million associations at d_k=64 with the usual scaling, because the exponential suppresses every key that does not match. The gates set the rate of forgetting. Kimi’s KDA decays each channel at its own learned rate. The contents vanish at the end of the session unless the serving protocol keeps the state.

That a model has a fast-weight store does not show what it uses the store for. In Qwen3.5 hybrids, an interpretability study of associative recall swapped internal states between runs. Swapping the recurrent states moved the answer by only 0.2 to 3.6% of the full effect, while swapping the attention caches moved it by 96 to 100%. For that task the recall ran through the context. The prompts were not built to overflow the state, so the routing was learned, not forced by capacity.

A second study found what the recurrent layers carry instead. After a prompt was processed, it kept either the attention cache or the recurrent state and let the model generate. Exact retrieval survived only through the cache. The requested output language and persona survived mainly through the recurrent state; Kimi Linear, a smaller relative of Kimi K3, showed the same division. The cache is the store that grows with the context, and it holds the content. The state is the store that every token reads, and it holds a few settings that condition everything generated. On the ladder, the second rung is so far a store of dispositions for the session rather than of facts.

Slow-weight continual learning is best understood as consolidation from faster stores. Self-distillation moves what the context achieved into the core weights (Part 1). Doc-to-LoRA moves a document into an adapter in one pass. Learning, Fast and Slow interleaves prompt evolution, the fast learner, with reinforcement learning on the weights, the slow one. On a synthetic graph task where reinforcement learning alone earned nearly nothing for about 250 steps, the combination was earning reward by step 50: the evolved prompts lifted reward off zero, so the weight updates had a signal to work with. The authors’ broader finding is a division of labour. The prompts absorb task-specific detail, so the weights drift up to 70% less from the base model. Lichkovski described this as a remedy for “discovery-shaped” problems, where reinforcement learning otherwise waits for a lucky success. In each case a faster, better learner produces the target and a slower store absorbs it. The open problems are therefore less about stability than about two other things. One is coverage: what revisits the old skills, and the old alternatives, that the new training never touches. The other is composition across time, which the benchmarks below are only starting to measure.

The architecture these methods point toward has been proposed in outline. A position paper from the Dagstuhl seminar on continual learning proposes a core model, a working memory and a long-term memory. In that architecture an internal regime replays stored memories through the core model to distil them into its parameters, “sparsely after accumulating sufficient experience”. Context-conditioned self-distillation and the laboratories’ on-policy distillation are concrete versions of that consolidation pathway, now with evidence about when it retains and when it fails. Different contents should also change at different rates. A library’s changing syntax can be revised by one new example, while a regularity attested many times should resist a single contradiction. Multi-timescale teachers (a moving average, an episodic buffer, a frozen base anchor) and per-channel decay gates are working instances of that idea.

A Google paper, Language Models Need Sleep, builds part of the ladder inside the weights. Its feed-forward blocks are updated at different frequencies. The fastest hold short-term memories, and the slowest almost never change. Periodically, in a phase the authors call sleep, the contents of a fast block move into a slower one. The teacher is the model as it stands before the move. The student is the model after it: the fast block’s recently added experts are cleared, and the slower block gains a new low-rank expert. Only that expert is trained, by on-policy distillation, so what the slower block already held stays fixed. This is consolidation as Part 1 described it, written into capacity allocated for the purpose. A second phase, which the authors call dreaming, has the model generate its own training data, in the manner of SEAL (see the box below). Most of the experiments run on the authors’ own multi-frequency architecture, so the method’s gains are hard to separate from the architecture’s.

The cut between parametric and non-parametric memory is the wrong one. The recurrent state is parametric and is wiped at the end of every session. Harness memory is non-parametric and permanent. Prediction heads are parametric, are written by temporal-difference learning toward their own next predictions rather than toward data, and store a different kind of content: knowledge of what would happen, not of what to do. The distinctions that predict behaviour are the four columns of the table.

What the benchmarks can and cannot see

Two recent benchmarks measure learning from experience in language-model agents. Continual Learning Bench has six tasks, among them adapting to a codebase and exploiting a weak poker opponent. It scores each method by its gain over the same model without memory of earlier episodes. Every method on its leaderboard is in-context learning or harness memory, including Claude Code and Codex; the authors state that they do not evaluate parametric approaches. FutureSim asks models to forecast events from a corpus of news articles released in chronological order. It is not branded as a continual-learning benchmark, but it is one, since the world drifts while the model reads. The best agent answered a quarter of its questions correctly. One controlled experiment started every agent from the worst agent’s initial forecasts, and none of them recovered: in the authors’ words, “frontier agents fail to adapt away from bad initial anchors.”

Lichkovski’s point about such benchmarks is that they must be agnostic about method. In-context methods and weight-changing methods should receive the same inputs. His proposal makes the agent’s own traces the common substrate. In-context methods keep the traces in context; distillation and reinforcement learning train on them. His proposed benchmark then treats dependencies between pieces of knowledge as the object of study. Invented words and functions over them (from MLC’s task) arrive at different times: kiki means purple, tufa means brown, later “twice”, later “thrice”, and probes ask for compositions such as “thrice tufa surround”. These need a function learned at one time applied to words learned at another. Some words are redefined partway through (“zup” changes meaning at a later step), which is worked example G as a test. Some functions arrive before their arguments. The ablations vary how far apart in time the dependent pieces arrived and how recently either was seen.

That design addresses what the coarse measures miss. The usual matrix of performance on task j after training through task i measures forgetting and transfer between tasks. It does not measure whether knowledge acquired at different times can be combined, and it does not distinguish revision (zup now means green) from interference (dax drifted too). Nor does it distinguish erasure from lost access. Palit and colleagues found that much of the forgetting during fine-tuning shifts all the old representations along one shared direction, which hides old facts without erasing them. Removing that direction from each weight update restored old facts in a pretrained language model. Only a slower erosion of individual facts was permanent. Part of a drop in the matrix can therefore be reversed, and a probe for the shared shift belongs beside it.

Part 1 adds a requirement the benchmarks do not yet meet. Forgetting has a second dimension, the alternatives within a task. A continual-learning benchmark should report how much a model gains from many attempts, best-of-k or pass@k at large k, alongside its single-attempt score, before and after each phase. A method can look perfect on retention while its later attempts become copies of its first.

Our essay What Breadth Reaches (§2) set Nathan Lambert’s substitution argument against Dwarkesh Patel’s complaint that today’s assistants cannot be taught on the job. Lambert held that context, reasoning and scaffolding can substitute for continual learning across much of what people use assistants for. On the benchmarks we have, he is winning, and the reason is that the benchmarks are short. Lichkovski calibrated his runs so that by step 700 an in-context learner would have accumulated at least a dozen million tokens of experience. That is the regime where the cost argument should start to bite: the context outgrows any window, retrieval must choose, and consolidation must start paying for itself. This is a prediction, not a result. Lichkovski reported that the in-context runs were proving expensive and held them back.

The boundary

One method compresses continual learning into a single problem. Learning to Discover at Test Time trains a model’s weights at test time on one hard problem, optimizing for the best attempt rather than the average. Several rungs of the ladder appear in miniature. Attempts accumulate in the context, the weights consolidate what worked, and the objective rewards the best member of a set, as VPO’s does. The weight updates do work that search does not. On a GPU-kernel task, the same model reusing past attempts in the same way, but with frozen weights, stalled at 2,061 microseconds. With test-time training it reached 1,203, below the best human entry’s 1,371. What Breadth Reaches (§2) asked whether long work on one project leaves a residue that training breadth does not deliver. The kernel result is a first data point at the scale of a single problem. Whether it holds at the scale of a project, over months, is open.

Like VPO, and like every general value function in Part 2, the method needs a verifier. The verifier scores attempts. It does not choose the problem, which attempts to build on, or what the model trains on. An autonomous learner would have to construct these teaching conditions itself: in Part 1’s terms, find the context in which it knows more, and targets it can absorb.

Some of this is already learned. SEAL, in the box above, learns to write its own training data. Schmidhuber’s PowerPlay invents its own tasks, each the simplest that its current solver cannot yet solve. OPST, from Part 1, poses its own questions from a topic name and keeps those its student cannot yet answer consistently, so its curriculum moves with the student. Open-ended search goes further and drops the target. Lehman and Stanley’s novelty search, met in Part 2, rewards behaviour unlike any seen before. OMNI, which both co-authored, has a foundation model choose, among the tasks a learner can learn, those that people would find interesting. Each works under a condition that is given one level up. SEAL’s reward is accuracy on questions someone wrote, and PowerPlay’s test for a new task is fixed in advance. OPST’s topics are named by people, in its experiments the category labels of the benchmarks it is scored on, and its only test of an answer is the model’s agreement with itself. Novelty is measured in a space of behaviours someone chose, such as the robot’s final position, and OMNI’s sense of what is interesting was learned from human writing. Constructing teaching conditions moves the given condition up a level; it does not remove it. Predictive questions are the one kind a learner can pose for itself and have graded for free, because the world answers them. For any other correctness condition, the boundary drawn in The Given and the Found (§7) still stands, between amortizing search under a given condition and discovering the condition.

Sources. Schmidhuber, Learning to control fast-weight memories: An alternative to dynamic recurrent networks, Neural Computation 4 (1992). Schlag, Irie and Schmidhuber, Linear Transformers Are Secretly Fast Weight Programmers, ICML 2021, arXiv 2102.11174. Irie, Csordás and Schmidhuber, Metalearning Continual Learning Algorithms, TMLR (2025), arXiv 2312.00276. Charakorn et al., Text-to-LoRA: Instant Transformer Adaption, ICML 2025, arXiv 2506.06105. Doc-to-LoRA: Learning to Instantly Internalize Contexts, arXiv 2602.15902. Lake and Baroni, Human-like systematic generalization through a meta-learning neural network, Nature 623 (2023). Lampinen et al., arXiv 2505.00661. Berglund et al., The Reversal Curse: LLMs trained on “A is B” fail to learn “B is A”, ICLR 2024, arXiv 2309.12288. Yang, Band, Li, Candès and Hashimoto, Synthetic continued pretraining, arXiv 2409.07431. Zweiger et al., Self-Adapting Language Models, arXiv 2506.10943. Lee et al., How Linear Attention Remembers, arXiv 2609.33093. Wang, You could have invented transformers from memory (blog post, October 2026), alexwang.ai/posts/associative-memory. Nichani, Lee and Bietti, Understanding Factual Recall in Transformers via Associative Memories, arXiv 2412.06538. Barnfield, Kim, Nichani, Lee and Lu, Sharp Capacity Thresholds in Linear Associative Memory, arXiv 2605.05189. Afendulev, Dontsov, Tutubalina and Korznikov, What Attention Recalls and Recurrence Controls in Hybrid Language Models, arXiv 2609.04434. Zucchet and Linderman, Divergence controls entropy in distillation, arXiv 2610.03529. Palit et al., When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting, arXiv 2610.08718. HiSPA, arXiv 2601.01972. Tiwari et al., Learning, Fast and Slow: Towards LLMs That Adapt Continually, arXiv 2605.12484. Asawa et al., Continual Learning Bench, arXiv 2606.05661. Goel et al., FutureSim: Replaying World Events to Evaluate Adaptive Agents, arXiv 2605.15188. Dorovatas et al., Position: Modular Memory is the Key to Continual Learning Agents, arXiv 2603.01761. Behrouz, Hashemi, Javanmard and Mirrokni, Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories, arXiv 2606.03979. Technical reports and announcements for DeepSeek V4, GLM-5.2 and GLM-5.3, Kimi K3 and MiMo-V2.6-Pro, as collected in our architecture dossier. Tanzer et al., A Benchmark for Learning to Translate a New Language from One Grammar Book, ICLR 2024, arXiv 2309.16575. Gemini Team, Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv 2403.05530. Xu et al., A Theory of Usable Information Under Computational Constraints, arXiv 2002.10689. Ellis et al., DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning, PLDI 2021, arXiv 2006.08381. Yuksekgonul et al., Learning to Discover at Test Time, arXiv 2601.16175. Schmidhuber, PowerPlay: Training an Increasingly General Problem Solver by Continually Searching for the Simplest Still Unsolvable Problem, Frontiers in Psychology 4 (2013), arXiv 1112.5309. Sick et al., On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models, arXiv 2610.32770. Lehman and Stanley, Abandoning Objectives: Evolution Through the Search for Novelty Alone, Evolutionary Computation 19 (2011). Stanley and Lehman, Why Greatness Cannot Be Planned: The Myth of the Objective, Springer (2015). Zhang, Lehman, Stanley and Clune, OMNI: Open-endedness via Models of human Notions of Interestingness, arXiv 2306.01711. Lichkovski, “Defining Continual Learning” (2026), and his interview on Deep Learning with Yacine (2026).

This article was co-authored by Łukasz Stafiniak and Claude (Opus 5.5).