Where Experience Goes

Learning the update rule

A learned update rule is a network whose output is a change to another network’s weights, or to its own. The idea is old. Schmidhuber’s fast weight programmers of the early 1990s were slow networks that wrote into the weights of fast ones. Three recent instances bear on continual learning.

Kazuki Irie, Róbert Csordás and Jürgen Schmidhuber meta-learned continual-learning algorithms directly, in a method they call Automated Continual Learning. Their learner is a self-referential weight matrix that rewrites itself with a delta rule as examples stream in. The rewriting rule is trained, across many sequences of tasks, so that after reading a sequence the matrix classifies examples from every task in it. The learned rule therefore has to solve the stability problem itself, since only an update that does not erase earlier tasks scores well. On split MNIST in the class-incremental setting, with five tasks and no replay, the learned rule reached 84.3% accuracy where elastic weight consolidation reached 19.8%. On harder image benchmarks it still trailed methods built on large pretrained models. The delta rule here is the same error-correcting update that runs inside the recurrent layers of current hybrid language models. Below, it reappears as the second rung of the ladder.

Hypernetworks make the learned update explicit. Text-to-LoRA, from Sakana AI, maps a natural-language description of a task to a low-rank adapter for a language model in one forward pass. Doc-to-LoRA maps a document to an adapter, so the model can answer questions about the document without the document in its context. Within the context window the document in context still does better: on SQuAD the adapter recovers about 82% of in-context performance. Its advantage appears past the window, where the base model cannot read the document at all, and in cost. Internalizing a document takes under a second, and the adapter needs megabytes where a long context needs gigabytes. Ilija Lichkovski made the point in his podcast interview that Doc-to-LoRA is a learned learning algorithm: a sample-efficient map from new content to a weight change. Nothing in its setup is shaped like continual learning, and yet it is exactly the consolidation step that Part 1 attributed to self-distillation, amortized into a single pass.

Lichkovski presented an experiment of his own in the same interview, as preliminary and unpublished, and its negative result is the instructive part. He trained a hypernetwork on split MNIST, the digit benchmark presented two classes at a time. Given the current weights and the identity of the next task, it was to output the weights that multitask training on all tasks so far would have produced. Multitask training was the oracle, and the hypernetwork was to reach it sequentially. It worked surprisingly well. Average accuracy rose with each new task toward the multitask level, while fine-tuning stayed at the one-fifth that total forgetting gives. Backward transfer was positive: earlier tasks improved after later ones were learned. Each task cost one forward pass at deployment. Then the ablations showed that the hypernetwork ignored the task identity. It had learned a general operator that moves any weight configuration toward the region of weight space where multitask solutions lie. The weights carried far more information than the small task embedding, so the operator attended to the weights.

That result fits a pattern The Given and the Found described (§5): an amortized learner learns whatever the distribution of its training updates rewards. Here the training updates all moved toward multitask solutions, and the cheapest way to predict them was to learn that move, not to learn the tasks. The failure is also a finding. A learned “move toward multitask” operator is a learned stability mechanism, a consolidation step that does not need to know what it consolidates.

One more line connects these to composition. Lake and Baroni’s meta-learning for compositionality (MLC) trained a standard transformer on a stream of small compositional tasks, each built from invented words. On their pseudoword instruction task, where “dax” might mean a red circle and “dax fep” three of them, people produced the systematic answer about 81% of the time, and the network’s sampled answers about 82% of the time. Lichkovski’s proposed benchmark, discussed below, is a sequential version of the same task. The words arrive one at a time and must be composed later.

A ladder of stores

Here are the stores a deployed language-model agent can use, from fastest to slowest.

1. The context. Attention over the key–value cache is in-context learning. The forward pass writes it, and nothing in it is ever overwritten. It lasts for the session, or until entries are evicted or the history is compacted into a summary.
2. The recurrent state. Hybrid models such as Kimi K3 and Qwen3.8 interleave attention with gated delta-rule layers. Each such layer keeps a fixed-size matrix that every token updates by one error-correcting step. This is a fast-weight memory, and it is bounded.
3. Harness memory. Notes the agent writes, skill files, retrieved documents and optimized prompts persist indefinitely, as text.
4. Prediction heads. These are Part 2’s banks of general value functions: small learned predictors, updated online by temporal-difference learning, that answer questions about what would happen under various behaviours.
5. Adapters. Low-rank weight deltas, written by gradient descent or by one hypernetwork pass.
6. Core weights. Changed by fine-tuning, reinforcement learning or self-distillation.
7. The release cycle. A lab trains specialist policies and distils them into the next checkpoint.

Four questions separate the rungs: what writes the store, what persists in it and for how long, how much it holds before entries interfere, and what it costs to read.

Four claims follow from setting the stores out this way.

In-context learning is the best learner on the ladder, example for example. Part 1 cited Lampinen and colleagues: a model reasons from facts in its context, to reversals and syllogisms, far better than it reasons from the same facts trained into its weights. The question for every slower store is whether it can keep that quality once the content has left the context.

The recurrent state is parametric continual learning, already running inside frontier models. A delta-rule layer treats its state S as the weights of a linear map from keys to values. For each token it reads the current prediction S k_t, compares it with the token’s value v_t, and writes the correction:

S_t=S_{t-1}+\beta_t\,(v_t-S_{t-1}k_t)\,k_t^\top,

usually after a learned decay of S_{t-1}. This is one step of gradient descent on \tfrac12\|S k_t-v_t\|^2, the Widrow–Hoff rule of 1960. Our essay Solving the Loop told this history (§2). What matters here is that the update is error-correcting, so it can revise. The temporal-difference update of How Can a Reward Train a Neural Network? has the same shape. With a linear value estimate \theta^\top\phi, TD(0) adds \alpha\,(r+\gamma\,\theta^\top\phi'-\theta^\top\phi)\,\phi. Set \gamma=0, and the target is the observed signal alone: the delta rule, with \phi as the key and r as the value. A delta-rule layer is a TD(0) learner with no future to bootstrap from. The GQ(λ) heads of Part 2 are the same learner given a future, a target policy and a trace. The second and fourth rungs of the ladder share an update rule, and they differ in their targets and in how long they keep what they learn.

Worked example G: overwriting a redefined word. Give a delta-rule memory two orthogonal keys, k_{\text{zup}}=(1,0) and k_{\text{dax}}=(0,1), and let the values be colours. Write zup → red, then dax → blue, then zup → green, each with \beta=1. The third write reads the current prediction for zup (red), and adds (\text{green}-\text{red})\,k_{\text{zup}}^\top. Now zup reads green and dax still reads blue: the redefinition replaced the old association in place. With \beta=0.5, zup reads half red and half green; each repetition of the new definition shrinks the remaining error by the factor 1-\beta\|k\|^2. Purely additive linear attention, which writes v\,k^\top without reading first, would make zup read red plus green. Now let the keys overlap: if k_{\text{dax}} had cosine 0.6 with k_{\text{zup}}, the same correction would move dax’s readout by 0.6 times (\text{green}-\text{red}). Interference is visible in one line. Fine-tuning a contradicting fact into core weights has no such targeted erase. The new fact is added by gradient steps that also move every other fact sharing the affected weights.

The capacity is bounded by the size of the matrix, and the gates set the rate of forgetting; Kimi’s KDA decays each channel at its own learned rate. The contents vanish at the end of the session unless the serving protocol keeps the state. A caution belongs here as well. That a model has a fast-weight store does not show that it uses the store for memory. In Qwen3.5 hybrids, an interpretability study of associative recall swapped internal states between runs. Swapping the recurrent states moved the answer by only 0.2 to 3.6% of the full effect, while swapping the attention caches moved it by 96 to 100%. For that task the recall ran through the context, and the recurrent layers did something else.

Slow-weight continual learning is best understood as consolidation from faster stores. Self-distillation moves what the context achieved into the core weights (Part 1). Doc-to-LoRA moves a document into an adapter in one pass. Learning, Fast and Slow interleaves prompt evolution, the fast learner, with reinforcement learning on the weights, the slow one. On a synthetic graph task where reinforcement learning alone earned nearly nothing for about 250 steps, the combination was earning reward by step 50: the evolved prompts lifted reward off zero, so the weight updates had a signal to work with. The authors’ broader finding is a division of labour. The prompts absorb task-specific detail, so the weights drift up to 70% less from the base model. Lichkovski described this as a remedy for “discovery-shaped” problems, where reinforcement learning otherwise waits for a lucky success. In each case a faster, better learner produces the target and a slower store absorbs it. The open problems are therefore less about stability than about two other things. One is coverage: what revisits the old skills, and the old alternatives, that the new training never touches. The other is composition across time, which the benchmarks below are only starting to measure.

The cut between parametric and non-parametric memory is the wrong one. The recurrent state is parametric and is wiped at the end of every session. Harness memory is non-parametric and permanent. Prediction heads are parametric, are written by temporal-difference learning toward their own next predictions rather than toward data, and store a different kind of content: knowledge of what would happen, not of what to do. The distinctions that predict behaviour are the four columns of the table.

What the benchmarks can and cannot see

Two recent benchmarks measure learning from experience in language-model agents. Continual Learning Bench has six tasks, among them adapting to a codebase and exploiting a weak poker opponent. It scores each method by its gain over the same model without memory of earlier episodes. Every method on its leaderboard is in-context learning or harness memory, including Claude Code and Codex; the authors state that they do not evaluate parametric approaches. FutureSim asks models to forecast events from a corpus of news articles released in chronological order. It is not branded as a continual-learning benchmark, but it is one, since the world drifts while the model reads. The best agent answered a quarter of its questions correctly. One controlled experiment started every agent from the worst agent’s initial forecasts, and none of them recovered: in the authors’ words, “frontier agents fail to adapt away from bad initial anchors.”

Lichkovski’s point about such benchmarks is that they must be agnostic about method. In-context methods and weight-changing methods should receive the same inputs. His proposal makes the agent’s own traces the common substrate. In-context methods keep the traces in context; distillation and reinforcement learning train on them. His proposed benchmark then treats dependencies between pieces of knowledge as the object of study. Invented words and functions over them (from MLC’s task) arrive at different times: kiki means purple, tufa means brown, later “twice”, later “thrice”, and probes ask for compositions such as “thrice tufa surround”. These need a function learned at one time applied to words learned at another. Some words are redefined partway through (“zup” changes meaning at a later step), which is worked example G as a test. Some functions arrive before their arguments. The ablations vary how far apart in time the dependent pieces arrived and how recently either was seen.

That design addresses what the coarse measures miss. The usual matrix of performance on task j after training through task i measures forgetting and transfer between tasks. It does not measure whether knowledge acquired at different times can be combined, and it does not distinguish revision (zup now means green) from interference (dax drifted too).

Part 1 adds a requirement the benchmarks do not yet meet. Forgetting has a second dimension, the alternatives within a task. A continual-learning benchmark should report how much a model gains from many attempts, best-of-k or pass@k at large k, alongside its single-attempt score, before and after each phase. A method can look perfect on retention while its later attempts become copies of its first.

Our essay What Breadth Reaches (§2) set Nathan Lambert’s substitution argument against Dwarkesh Patel’s complaint that today’s assistants cannot be taught on the job. Lambert held that context, reasoning and scaffolding can substitute for continual learning across much of what people use assistants for. On the benchmarks we have, he is winning, and the reason is that the benchmarks are short. Lichkovski calibrated his runs so that by step 700 an in-context learner would have accumulated at least a dozen million tokens of experience. That is the regime where the cost argument should start to bite: the context outgrows any window, retrieval must choose, and consolidation must start paying for itself. This is a prediction, not a result. Lichkovski reported that the in-context runs were proving expensive and held them back.

Revisiting The Dynamics That Matter

Two claims in that essay need correcting, and the evidence for both has appeared across these three parts.

First: we claimed that composition requires parametric integration, which harness memory cannot provide. We wrote that harness memory “keeps the cognitive labor homogeneous”, that composition “requires the integration to have happened at the parametric level rather than the harness level”, and that the desiderata depend on a heterogeneity “which parametric CL can provide and harness-based CL cannot”. The controlled evidence runs the other way for single facts. Content held in context composes: models reason from it to reversals and syllogisms, and they learn to translate Kalamang, a language with fewer than two hundred speakers, from a grammar book in their context. The same content trained into the weights composes poorly, unless the training data spells out the implications. Retrieval does not prevent later computation from integrating what it retrieved; attention integrates by construction. What survives is the cost and automaticity argument of the previous section. Retrieval degrades with the size of the store, context has a ceiling, and retrieved competence is slower than automatic competence. Whether composition survives at scale, and across long gaps in time, is an empirical question, and Lichkovski’s benchmark is built to ask it. The necessity claim was ours, not his. His argument was about automaticity, and we turned it into a claim about what composition requires.

Second: we claimed that low sample efficiency signals deep integration. We wrote that parametric updates being less sample-efficient than in-context learning “is a signal that the integration is happening at a level that requires evidence to support”, and that “pushing sample efficiency too low would mean either superficial integration or fragile integration”. We retract this. Sample efficiency measures the update rule and the prior, not the depth of what is integrated. The clearest case is the contrast the earlier essay leaned on, between fine-tuning and in-context learning. Fine-tuning needs many augmented examples to learn the implications of a fact, and those examples are the inferences that in-context learning draws for free. Its hunger for samples shows that the update rule cannot reach what was latent in the data. That is the opposite of deep integration. The theory of usable information states the general point: a learner with bounded computation may be unable to use information that its data contains. Fast updates are not therefore shallow. Doc-to-LoRA writes a document into an adapter in one pass, at some cost in quality against the document in context. Lichkovski’s hypernetwork reached positive backward transfer at one pass per task, even if what it learned was a generic consolidation move. The measure we should have proposed is the one the benchmarks above are moving toward: integration shows in transfer, composition and the pattern of interference, read off the performance matrix and dependency probes, not in how many samples an update needed.

Other parts of The Dynamics That Matter stand, with one update. Its §4 introduced a position paper from the Dagstuhl seminar on continual learning, which proposes a core model, a working memory and a long-term memory. In that architecture an internal regime replays stored memories through the core model to distil them into its parameters, “sparsely after accumulating sufficient experience”. Context-conditioned self-distillation and the laboratories’ on-policy distillation are concrete versions of that consolidation pathway, now with evidence about when it retains and when it fails. The essay’s overwriting paragraph also survives, in a narrower form: different contents should change at different rates. Multi-timescale teachers (a moving average, an episodic buffer, a frozen base anchor) and per-channel decay gates are working instances of that idea.

A related remark in What Breadth Reaches §2 needs narrower conditions. It said that simple online learners, such as stochastic gradient descent on linear models or recursive least squares, update without destroying what they learned. Worked example G shows the condition. The Widrow–Hoff rule revises cleanly when keys are orthogonal and interferes in proportion to their overlap. A delta-rule layer is exactly such a simple online learner, and short trigger strings are known to overwrite the hidden state of Mamba models irreversibly. The interference that makes continual learning hard in deep networks is not absent in simple learners. It is easier to see.

The boundary

One method compresses continual learning into a single problem. Learning to Discover at Test Time trains a model’s weights at test time on one hard problem, optimizing for the best attempt rather than the average. Several rungs of the ladder appear in miniature. Attempts accumulate in the context, the weights consolidate what worked, and the objective rewards the best member of a set, as VPO’s does. Like VPO, and like every general value function in Part 2, it needs a verifier. Something outside the learner has to say which attempt was best.

That is where the three parts end up. Self-distillation consolidates what the context made available. VPO keeps the alternatives that a future objective might select. Off-policy prediction learns about behaviours the agent is not following. Consolidation makes the useful parts cheap. Each of these takes a correctness condition as given: a demonstration, a test suite, a reward component, the world’s answer to a predictive question. A continual learner built from these parts can keep learning for as long as such conditions keep arriving. What none of them supplies is the condition itself. In the terms of The Given and the Found (§6), they amortize search under given conditions, and the boundary between that and discovery still stands.

Sources. Schmidhuber, Learning to control fast-weight memories: An alternative to dynamic recurrent networks, Neural Computation 4 (1992). Irie, Csordás and Schmidhuber, Metalearning Continual Learning Algorithms, TMLR (2025), arXiv 2312.00276. Charakorn et al., Text-to-LoRA: Instant Transformer Adaption, ICML 2025, arXiv 2506.06105. Doc-to-LoRA: Learning to Instantly Internalize Contexts, arXiv 2602.15902. Lake and Baroni, Human-like systematic generalization through a meta-learning neural network, Nature 623 (2023). Lampinen et al., arXiv 2505.00661. Lee et al., How Linear Attention Remembers, arXiv 2609.33093. HiSPA, arXiv 2601.01972. Tiwari et al., Learning, Fast and Slow: Towards LLMs That Adapt Continually, arXiv 2605.12484. Asawa et al., Continual Learning Bench, arXiv 2606.05661. Goel et al., FutureSim: Replaying World Events to Evaluate Adaptive Agents, arXiv 2605.15188. Dorovatas et al., Position: Modular Memory is the Key to Continual Learning Agents, arXiv 2603.01761. Technical reports and announcements for DeepSeek V4, GLM-5.2 and GLM-5.3, Kimi K3 and MiMo-V2.6-Pro, as collected in our architecture dossier. Tanzer et al., A Benchmark for Learning to Translate a New Language from One Grammar Book, ICLR 2024, arXiv 2309.16575. Gemini Team, Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv 2403.05530. Xu et al., A Theory of Usable Information Under Computational Constraints, arXiv 2002.10689. Ellis et al., DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning, PLDI 2021, arXiv 2006.08381. Yuksekgonul et al., Learning to Discover at Test Time, arXiv 2601.16175. Lichkovski, “Defining Continual Learning” (2026), and his interview on Deep Learning with Yacine (2026).

This article was co-authored by Łukasz Stafiniak and Claude (Opus 5.5).