Who Teaches the Weights?

What would count as continual learning

The research literature on continual learning distinguishes three scenarios, following van de Ven, Tuytelaars and Tolias. In task-incremental learning the model is told which task it faces. In domain-incremental learning the task stays the same while the input distribution shifts. In class-incremental learning the model must tell apart classes it never saw together, because they arrived in different phases of training. The last is the hard one. Ilija Lichkovski, who works on continual learning at Prime Intellect, illustrated this in a recent podcast with split MNIST, the handwritten digits presented two classes at a time. Elastic weight consolidation (EWC), a classic method that penalizes changes to weights that mattered for earlier tasks, barely beats naive fine-tuning in the class-incremental setting, at about 20% accuracy. Careful navigation of the loss landscape protects old weights, but it does not compose old and new representations into one classifier.

Measurement starts from a matrix. Let R_{i,j} be the performance on task j after training through task i. Average accuracy reads off the last row. Forgetting compares each task’s best score with its final one. Backward transfer asks whether later training improved earlier tasks. Most language-model studies report something coarser, an average drop on held-out benchmarks after one round of training. That number cannot distinguish a temporary dip from a permanent loss, and it says nothing about whether later learning can build on earlier learning.

One solution is trivially correct and rules itself out: keep every example ever seen and retrain on all of them at each step. This turns sequential learning back into multitask learning, at a cost that grows without bound. Every serious method is a cheaper approximation of it. Replay keeps a sample of old data. Regularization keeps old weights near where they were. Distillation keeps old behaviour, by training the new model to match the outputs of an older one on chosen inputs. Lichkovski’s description of self-distillation in the same podcast is that “it is a form of replay”. The phrase is apt, with one qualification that the rest of this part develops. What self-distillation replays is behaviour, not data, and it replays it only on the inputs the training run happens to visit.

On-policy data approximates the trivial solution cheaply. A deployed agent revisits many of its general skills just by working. It writes code, reads documents and answers questions while learning the new thing. When it trains on its own activity, part of the old distribution comes back for free. This is part of what Retaining by Doing measured. It is also why “replay” and “on-policy” are less distinct than their names suggest.

Where the model trains, and what it imitates

Most methods that protect old behaviour fit one formula. The loss for the new task is a plasticity term plus a stability term:

\mathcal L(\theta)=\mathcal L_{\text{new}}(\theta)+\lambda\,\mathbb E_{x\sim\mu}\,D\big(q(\cdot\mid x)\,\big\|\,p_\theta(\cdot\mid x)\big).

The stability term has two parts that can be chosen separately. The teacher q is the distribution the model should keep matching. The support \mu is the set of inputs on which it must match it. Elastic weight consolidation constrains parameters instead of behaviour, but its penalty approximates the same divergence.

Fisher information. Move the old model’s weights \theta_0 by a small \Delta. On an input x, the divergence between the old and new outputs is, to second order,

D_{\mathrm{KL}}\big(p_{\theta_0}(\cdot\mid x)\,\big\|\,p_{\theta_0+\Delta}(\cdot\mid x)\big)\approx\tfrac12\,\Delta^\top F(x)\,\Delta, \qquad F(x)=\mathbb E_{y\sim p_{\theta_0}(\cdot\mid x)}\big[\nabla\log p_{\theta_0}(y\mid x)\,\nabla\log p_{\theta_0}(y\mid x)^\top\big].

There is no first-order term, because the old model is at the minimum of its divergence from itself. The matrix F is the Fisher information. It measures how sharply the log-probabilities of the model’s own outputs respond to a move in each direction of weight space. Wikipedia’s article on Fisher information derives the expansion and gives the statistical reading: the curvature of the log-likelihood, which bounds how precisely data can pin down a parameter (the Cramér–Rao bound). The article on the Fisher information metric gives the geometric reading, in which F is the local metric that the divergence induces on a family of distributions. Average F over the old task’s inputs, keep only its diagonal, and the result is the EWC penalty, \tfrac{\lambda}{2}\sum_i F_{ii}\,(\theta_i-\theta_{0,i})^2. EWC is therefore distillation with the old model as teacher and the old inputs as support, compressed into one number per weight. The compression is what makes it cheap: F is computed once, and the old inputs can then be discarded. It is also what limits it. The quadratic holds only near \theta_0, which a long sequence of tasks leaves behind, and the diagonal ignores how weights compensate for one another.

Placing the classic methods on this grid shows what each one protects.

• Learning without Forgetting uses the old model as teacher and the new task’s inputs as support. It protects old behaviour only where old and new inputs overlap. A model learning medical terminology keeps its old answers to medical questions, not its old answers to questions about tax law.
• Dark Experience Replay uses the old model as teacher on old inputs, by storing a buffer of past inputs together with the logits the model produced for them at the time. On a standard image benchmark with only 200 stored examples, plain experience replay reached 44.8% accuracy, DER 61.9% and DER++ (which adds the stored labels) 64.9%. The model’s own past soft outputs carry information that its past labels do not: which wrong classes it found plausible, and how plausible.
• An exponential moving average of the model’s own weights serves as a fresh teacher on current inputs, as in the Mean Teacher method of semi-supervised learning. It smooths training. Worked example A below shows why it is not a memory.
• On-policy distillation makes the support whatever the student itself generates. The model is corrected on the states it actually visits, which is where its errors compound.

Worked example A: why a moving-average teacher forgets. Let the teacher’s weights be an exponential moving average of the student’s, with momentum m=0.999. The share of the teacher that still comes from student states older than k steps is m^k. After 1{,}000 steps that share is 0.999^{1000}\approx 0.37. After 5{,}000 steps it is about 0.007. A teacher that agrees with the student at every step can therefore drift arbitrarily far from where training began. It is the only reference point, and it moves with the student. The repair adds a second reference that never moves: a frozen copy of the base model, with its own penalty term. Mean Teacher has no such anchor, because a model trained from scratch has no earlier competence to protect; iSDFT, below, adds one.

On-policy sampling and the choice of teacher are different decisions. The first decides where the model trains, and the second decides what it is pulled toward. The evidence from Retaining by Doing concerns the first. GRPO keeps a model close to itself because every update is computed on the model’s own samples, whatever its penalty term says. A month before the Princeton paper, a study from MIT’s Improbable AI Lab, RL’s Razor, had found that forgetting tracks how far the trained model’s outputs drift from the base model’s on the new task, and that on-policy sampling keeps that drift small. The Princeton group found the link weaker in its own setting, but the direction is the same. It is easy to credit the teacher with what the support is doing. The rest of this part keeps the two apart. The companion essay How Can a Reward Train a Neural Network? explains in its first section why acting changes the examples one learns from.

The model with privileged context as teacher

In January 2026 three papers published the same construction under different names: self-distillation fine-tuning (SDFT), on-policy self-distillation (OPSD) and self-distillation policy optimization (SDPO). The student is the model as it will be deployed, without help. It samples a response to a prompt. The teacher is the same model, usually a slowly moving copy of it, given something extra in its context: a demonstration of the skill, the documentation of a tool, or the environment’s feedback on an earlier attempt, such as a failing test. The teacher scores each prefix of the student’s response, and the student is trained toward the teacher’s next-token distributions on those prefixes.

The support is therefore on-policy: the student’s own samples. The teacher is the student, better informed.

The construction is easiest to read as a step of policy improvement. The essay How Does a Policy Improve? separated two jobs that most reinforcement-learning algorithms combine. One job builds an improved distribution over actions, typically by tilting the current policy toward actions with higher estimated value. The other fits a parametric policy to that improved distribution. Maximum a Posteriori Policy Optimisation makes the separation explicit. In self-distillation the improved distribution is simply the model’s own conditional, \pi(\cdot\mid x,c), where c is the privileged context. No critic, search or reward model is needed to build it, because reading the context does the improving.

The correspondence is exact. How Does a Policy Improve? (Part 2) derived the best distribution q over actions for a fixed reference p and fixed action values Q, when moving away from the reference costs \eta\,D_{\mathrm{KL}}(q\|p):

q^*(a)=\frac{p(a)\,e^{Q(a)/\eta}}{Z},\qquad Z=\sum_b p(b)\,e^{Q(b)/\eta}.

Take the bare model as the reference, p=\pi(\cdot\mid x), and give each response y an implicit value, its log-ratio under the two contexts:

Q_c(y)=\log\frac{\pi(y\mid x,c)}{\pi(y\mid x)}.

At temperature \eta=1 the tilt returns the teacher exactly, with Z=1. The privileged context acts as a critic whose scores are these log-ratios. Since a response’s log-probability is a sum over its tokens, so is its implicit value:

Q_c(y)=\sum_t\Big[\log\pi(y_t\mid x,c,y_{<t})-\log\pi(y_t\mid x,y_{<t})\Big].

Each token’s term is a dense implicit reward. It is positive where the context makes the token more likely and negative where the context argues against it. The same rearrangement appeared in the earlier essay’s account of Direct Preference Optimization, where a policy’s log-ratio against its reference serves as its implicit reward. Self-distillation runs the identity in the opposite direction. It starts from a model that already behaves as if it had optimized some reward, namely the model with context, and reads that reward off the log-ratio.

Worked example B: a demonstration as a critic. The earlier essay’s robot had three actions with reference probabilities p=(0.6,0.3,0.1), scored Q=(0,1,2) by a critic. At temperature 0.5 the tilt gave q^*\approx(0.07,0.27,0.66). Keep the same reference, now read as a bare model’s probabilities for three continuations at some prefix, and suppose that with a demonstration in context the same model assigns (0.1,0.2,0.7). The implicit values are the log-ratios (-1.79,-0.41,+1.95). Shifted so the first is zero and rescaled to temperature 0.5, they become (0,0.69,1.87), not far from the robot’s critic. The demonstration also moves the distribution about as far as that critic did: D_{\mathrm{KL}}(\text{teacher}\,\|\,p)=1.10 nats, against 1.06 for the robot’s q^*. A full distillation step spends all of that divergence. A half step, the normalized geometric mean of student and teacher, gives (0.325,0.325,0.351) and spends 0.27 nats. The third continuation is now the most likely, but only just.

The half step is a tilt at temperature \eta=2, because the geometric interpolation p^{1-\lambda}T^{\lambda} between student p and teacher T equals p\,e^{\lambda Q_c} up to normalization, with \lambda=1/\eta. Information-Proximal SDFT (iSDFT) chooses \lambda per token so that each update stays within an information budget. The paper calls \lambda “the local Lagrange multiplier induced by the information budget”. This is the earlier essay’s constrained form of the tilt, where a budget \epsilon replaces a fixed temperature and the dual g(\eta)=\eta\epsilon+\eta\log\sum_a p(a)\,e^{Q_c(a)/\eta} is minimized until the update uses exactly the divergence allowed. iSDFT also adds a penalty toward the frozen base model, the permanent reference that worked example A called for. The earlier essay drew the distinction precisely: “A fixed reference anchors behavior to the starting model; a refreshed reference limits each step while allowing cumulative movement away from that starting point.” The self-teacher is a refreshed reference, and the base anchor is a fixed one. In the authors’ retention tests, 73% of evaluations stayed within half a point of the base model’s score, against 52% for the strongest baseline.

Constructing the improved distribution is half of policy improvement. Fitting a network to it is the other half, and the direction of the divergence used to fit decides what survives. Maximum a Posteriori Policy Optimisation fits by weighted likelihood, which minimizes D_{\mathrm{KL}}(q\|\pi_\theta), the mass-covering direction. The network is penalized wherever the improved distribution has mass that the network lacks, so it keeps every alternative the target keeps. The opposite direction, D_{\mathrm{KL}}(\pi_\theta\|q), is mode-seeking. A network that cannot represent the whole target is allowed to settle on one of its peaks. On-policy distillation is often written in the mode-seeking form, since it is natural to average over the student’s own samples. iSDFT’s loss uses the mass-covering form at each token, and the SDFT authors report that the mass-covering form worked best in practice. Either way, the average runs over prefixes the student itself generated. A mass-covering fit preserves the teacher’s alternatives at the prefixes the student reaches, and it says nothing about prefixes the student never starts. The earlier essay put the limit in one sentence: “Regularization toward a prior can preserve useful experience, but it cannot conjure alternatives outside that prior’s support.” The section on what on-policy training forgets returns to that limit.

Why should the model with context be a good teacher? Because in-context learning generalizes in ways fine-tuning does not. Lampinen and colleagues at Google DeepMind ran a controlled comparison on synthetic facts the model could not have seen. Given the facts in context, the model could answer their reversals and combine them in syllogisms. Fine-tuned on the same facts, it mostly could not. On reversals it was near zero. The gap closed, and sometimes reversed, when the fine-tuning data was augmented with the inferences the model drew in context. Self-distillation automates that augmentation. The teacher’s distribution already contains the implications, because the teacher is the model reasoning with the fact in view.

Lichkovski’s closing point in the podcast puts this in information-theoretic terms, through the theory of usable information of Xu and colleagues. Processing cannot add Shannon information to data. It can make information usable to a learner with bounded computation. A document contains its implications in the Shannon sense, but a learner that only fits the surface of the document cannot reach them. Self-distillation, like synthetic data generally, does not create information from nothing. It exposes what was latent, in a form a gradient step can absorb.

The essay The Given and the Found made a related point about learning signals in general (§5). The confidence of a manufactured target sets the size of the update. Whether the target is grounded in something outside the learner sets its sign. Here the grounding is the privileged context. When the context holds a demonstration or a test result, the self-teacher is grounded, and its confident targets are confident for a reason. Without such context, self-distillation is the ungrounded case that essay warned about: a model regressed onto its own confident mistakes. The next section shows what happens when the grounding is weak.

One cost should be stated before the failures. The SDFT paper reports about 2.5 times the floating-point operations and 4 times the wall-clock time of ordinary fine-tuning. Regenerating training data once per epoch, as Retaining by Doing did, is the cheaper middle ground: approximately on-policy support, with no teacher pass at every step.

Where self-teaching breaks

If self-distillation is policy improvement done by reading, its failures should follow from the reading. They do.

A weak in-context learner is a weak teacher. The SDFT paper finds that at 3 billion parameters the method falls behind ordinary supervised fine-tuning, while at 7 and 14 billion it pulls ahead. That is what the mechanism predicts, since small models gain less from context. SynthSelf, a preprint from October 2026, studied models from 0.5 to 3 billion parameters and replaced the self-teacher with a slightly larger model from the same family, which rewrites the training targets so that they stay close to the student’s own distribution. Averaged across its settings, SynthSelf reached an acquisition score of 56.4 against 44.1 for SDFT, at equal retention (41.7 against 41.5). At these sizes even plain supervised fine-tuning (52.2) acquired more than SDFT. SynthSelf’s teacher knows more than the student, and its rewriting is what makes its targets absorbable: teaching, as distinct from knowing. Classic distillation met the same distinction from the other side. A much larger teacher can teach a small student worse than a moderately larger one, and a mid-sized “teacher assistant” placed between them helps (Mirzadeh and colleagues, 2020). The self-teacher is one way to meet both conditions, and not always the best.

A dense target constrains the wrong things. Denser ≠ Better compared token-level SDPO with sequence-level GRPO across a sequence of post-training stages. The dense method forgot more, amplified formatting artifacts and sometimes collapsed. One reason is structural, and the tilt makes it exact. Tilting the reference by a sequence-level reward R gives q(y)\propto p(y)\,e^{R(y)/\eta}, so for any two responses with the same reward,

\frac{q(y)}{q(y')}=\frac{p(y)}{p(y')}\qquad\text{whenever } R(y)=R(y').

Among the correct responses, the student’s own preferences survive untouched. Only the boundary between correct and incorrect moves. The self-teacher’s implicit value Q_c is not constant on the correct responses. It also ranks them, by how closely each resembles what the context suggests: the demonstration’s phrasing, its formatting, its order of steps. A token-level target therefore reorders the student’s preferences among correct answers, at every prefix, toward one exemplar. Whatever quirks the teacher has are written in at each token, and the next round’s teacher inherits them. This finding does not contradict Retaining by Doing. On-policy support protects the model, and an objective that is too demanding on that support can undo the protection.

A moving teacher has no fixed reference. This is worked example A again. Small steps, each checked against a teacher that moved with the student, add up to a large distance from the base model. A frozen anchor is the repair, at the price of some plasticity.

On-policy support is not old-task support. A model learning a tool-use skill visits states about tool use. Its self-teacher says nothing about medical questions, because no medical question appears in the rollouts. Only replay, generated stand-ins for old inputs, an anchor evaluated on suitable prompts, or capacity set aside for old skills covers what the new task never visits.

What the context says is not what the weights learn. The Negation Neglect study found that models fine-tuned on text describing what not to do still learn to do it. Lichkovski raised the worry for agents that read documentation of bad customer interactions. A self-teacher conditioned on a negative example might pass the same failure on, since the teacher’s distribution shifts toward whatever the context makes salient. That is an open question, not a result.

A model can avoid forgetting by becoming hard to change. Strong anchors and high distillation weights reduce measured forgetting while reducing what the model can learn. Results should therefore be reported as a frontier of acquisition against retention, not as a forgetting score alone.

Lichkovski’s own early experiments, which he presented in the podcast as preliminary and unpublished, fit this picture. Sequential SDFT on three tool-documentation skills retained earlier skills better than supervised fine-tuning or off-policy distillation, replicating the paper’s result. A comparison of on-policy self-distillation with GRPO on Qwen3-8B, switching environments every hundred steps, showed sharp drops at some transitions, such as from hotel booking to restaurant operations. He described that run as poorly tuned.

What on-policy training forgets

Everything so far measured forgetting across tasks: does learning the new thing damage the old ones? A second kind of forgetting happens within the task being learned, and on-policy training is its main cause.

A base model asked to solve a programming problem can produce many different solutions: a greedy one, a dynamic-programming one, one that misreads the problem in a useful way. Reinforcement learning with verifiable rewards samples a group of answers, scores them, and raises the probability of those that scored above the group’s average. Repeated many times, this concentrates probability on whichever strategy first scored well. The trained model answers better on its first try. Its hundredth try, however, looks like its first. Several studies, among them one by Yue and colleagues at Tsinghua in 2025, have found that after such training, the probability of solving a problem in k attempts (pass@k) at large k falls below that of the base model. The training sharpened one peak of the distribution and flattened the rest.

The tilt explains why small differences become total ones. Each policy-improvement step is roughly a tilt of the current policy by the reward at some temperature \eta, and each step’s reference is the previous iterate. After k such steps the policy is the original one tilted by kR/\eta, so the effective temperature falls as \eta/k. Take two strategies whose scalar rewards differ by 0.1: a balanced solution, and one that excels on some components, fails on others and scores 0.1 lower. Ten steps at \eta=0.1 multiply the odds of the balanced strategy by e^{0.1\cdot 10/0.1}=e^{10}\approx 22{,}000. The specialist disappears, however much probability the base model gave it and however different the components it satisfied. A KL penalty toward a fixed reference sets a floor. The regularized optimum is p\,e^{R/\beta}, so the effective temperature cannot fall below the penalty’s coefficient \beta. In practice that coefficient is small, and some recipes drop the penalty altogether. Retaining by Doing found that dropping it barely changed forgetting across tasks.

The cost of this is easy to miss when models are evaluated one answer at a time. Deployed models increasingly are not. A coding agent runs tests and retries, a reasoning system votes across samples, and evolutionary systems such as AlphaEvolve keep a population of candidates and recombine them. All of these need the model to produce answers that differ. A model whose samples are near-duplicates gains nothing from being sampled thirty times. Our essay What Should a Program Inherit? made the evolutionary version of the point: a strong evaluator cannot select a discovery the variation never proposes.

Ryan Bahlous-Boldi and colleagues at MIT’s Improbable AI Lab, the group behind RL’s Razor and the SDFT paper, proposed Vector Policy Optimization (VPO) in May 2026 as a direct answer. It starts from an observation about rewards: in practice they are usually vectors that someone has collapsed into a scalar. A coding reward is a count of passed test cases, but the cases are separate. A multi-hop question has an answer and several supporting citations. A tool call has a format, a tool name and arguments. Ordinary training fixes a weighting w^* and maximizes w^{*\top}r. VPO keeps the vector.

The method has two parts.

1. The model writes m answers in one response, one after another, after a shared stretch of reasoning. Each answer can see the ones before it. (The paper uses m=3.)
2. The set of answers S is scored by sampling many random weightings w uniformly from the simplex, taking the best answer under each weighting, and averaging:

R(S)=\mathbb E_{w}\Big[\max_{y\in S}\,w^\top r(x,y)\Big].

This set-level score replaces the scalar reward in GRPO’s advantage. For a group of G sampled responses to the same prompt, GRPO computes

A_g=\frac{R_g-\operatorname{mean}(R_1,\dots,R_G)}{\operatorname{std}(R_1,\dots,R_G)}

and applies A_g to every token of response g. VPO keeps this formula with R_g=\hat R(S_g), the set score estimated from a sample of weightings shared across the group. Nothing else in training changes.

The max is what makes it work. An answer earns credit only if it is the best in its set under some weighting, so two similar answers compete for the same weightings and one of them is wasted. A set of specialists that are each good on different components covers more of the simplex than a set of copies.

Worked example C: why the max rewards specialists. Take two reward components and weightings w=(u,1-u) with u uniform on [0,1]. A balanced answer scores (0.6, 0.6), and two specialists score (1,0) and (0,1). Under the fixed weighting w^*=(0.5,0.5), the balanced answer is worth 0.6 and each specialist 0.5, so ordinary training pushes every sample toward the balanced answer. VPO scores sets. Three copies of the balanced answer score 0.6 under every weighting, so R=0.6. The set of two specialists plus the balanced answer scores \max(u,1-u,0.6), whose average over u is 0.76. The balanced answer is still in the best set, but the specialists now earn their place. Part 2 shows that the max splits GRPO’s weighting among the answers. Each answer is trained, in effect, under its own weighting: the average of the weightings for which it is the best in its set, scaled by how often that happens.

Each part fails alone, and the failures are instructive. Asking for several answers under a fixed scalar reward (the Multi-RLVR baseline of Puri and colleagues) gives the model the capacity to diversify but no reason to. Its answers converged to similar reward vectors during training. Random weightings applied to single answers did nothing measurable either. Most telling, a model given the weighting w in its prompt and trained to satisfy it learned to ignore the prompt and collapsed to one behaviour. The model could not turn a weight vector written in text into a choice of strategy. It could, however, look at its own first answer and write a different second one. The diversity lives in the context. In-context learning appears here in a second role, as the explorer, after appearing earlier in this part as the teacher.

The results follow the motivation. Across four domains (navigation in a maze with competing collectibles, multi-hop question answering, a five-step reasoning chain, and tool calling), VPO came out best among seven methods, or within a hair of the best, once several samples were allowed. Its advantage grew with k. Scalar GRPO plateaued almost at once: on the maze, its score was the same at k=3 and k=30. The paper adds a case study on LiveCodeBench with Qwen2.5-Coder-7B. There GRPO was better on single attempts, and VPO caught up and overtook it as the number of attempts grew. On the 32 hardest problems, where neither model passed a single test in thirty samples, both checkpoints were placed inside OpenEvolve, an open-source evolutionary search loop. Over 200 iterations, the VPO model kept finding new solutions while the GRPO model stalled early.

Two results deserve emphasis. First, VPO beats GRPO under GRPO’s own weighting w^*, the objective GRPO was trained to maximize. In an interview on the Deep Learning with Yacine channel, Bahlous-Boldi gave the mechanism as a story about training dynamics. Suppose a hard problem has twenty tests. A greedy solution passes the first ten; a dynamic-programming solution passes all twenty but is harder to find. If GRPO happens to sample the greedy solution first, it shifts probability toward greedy solutions and never escapes, although dynamic programming was one of the bumps in the pretrained distribution. Under VPO, some weighting cares mostly about test 17, so the solutions that pass it keep receiving credit while they are still rare. Second, the method stops helping when the reward components are nearly collinear, so that every weighting ranks answers the same way. On a preference dataset scored by one reward model along five nominally distinct criteria that mostly agreed, VPO fell below the scalar baselines. Diversity along axes that do not disagree has nothing to offer.

The paper does not mention continual learning. In the interview, Bahlous-Boldi said that many people expected his first paper to be about continual learning, “and it actually is. I just didn’t write about it.” Forgetting is usually pictured as training on a new loss climbing back up the old one. His diagnosis is the second kind of forgetting. Training that has already collapsed onto one trough of the old loss has erased the alternatives from which the new task could have been learned cheaply. His proposed reframing is to find “every single trough” and cover them all, so that one of them is already good at the next task. He anticipated the obvious objection himself. In continual learning the future reward is unknown, so the reward vector to diversify over is not available. His reply is that diversity along known axes still helps transfer to an unknown reward. That is a claim to test, not a result.

The claim can be put more sharply. A model that must keep learning faces objectives it does not yet know: new test cases, new users, new weightings of old criteria. Whatever it learns next, it will learn from its own behaviour. Policy-gradient methods can only raise the probability of what the model already does sometimes. Off-policy methods, which learn about behaviour other than the current one, need data in which that behaviour appears. Either way, a strategy the model no longer produces is a strategy nothing can teach it, short of an outside demonstration. The collapse of alternatives is therefore a loss of plasticity at the level of behaviour, even if no weight has lost its ability to change.

What the weights should inherit

Two consolidation stories run through this part. In the first, the model with privileged context teaches the bare model. On-policy support keeps the update close to what the model already does, a frozen anchor bounds its drift, and the teacher’s reasoning carries the implications that plain fine-tuning misses. This route consolidates what the context made available, and nothing more. When the context holds no demonstration or feedback, there is nothing grounded to consolidate.

The second story qualifies the first. On-policy training, rewarded or self-taught, keeps the model’s other skills while narrowing the skill it is learning. A token-level self-teacher that picks one continuation per prefix is the denser form of the same pressure. VPO shows that the narrowing is a choice of objective, not a law. Score sets instead of answers, and in-context learning will spread the set across the trade-offs the reward actually contains.

What neither story supplies is a way to learn about behaviour the model is not currently producing. A continual learner needs one, because its next objective may favour exactly such behaviour. There is an old line of reinforcement-learning research built around that need, and it predates large language models by more than a decade. In 2010 Hamid Maei and Richard Sutton published GQ(λ), an algorithm for learning many predictions in parallel from a single stream of experience, each about a behaviour the agent was not following. Part 2 starts there, and argues that VPO’s objective is a close relative of the one that research arrived at.

Sources. Chen, Razin, Narasimhan and Chen, Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting, arXiv 2510.18874 (ICML 2026). Shenfeld, Pari and Agrawal, RL’s Razor: Why Online Reinforcement Learning Forgets Less, arXiv 2509.04259 (ICLR 2026). Van de Ven, Tuytelaars and Tolias, Three types of incremental learning, Nature Machine Intelligence (2022). Kirkpatrick et al., Overcoming catastrophic forgetting in neural networks, PNAS 114 (2017), arXiv 1612.00796. Li and Hoiem, Learning without Forgetting, ECCV 2016, arXiv 1606.09282. Buzzega et al., Dark Experience for General Continual Learning, arXiv 2004.07211 (NeurIPS 2020). Tarvainen and Valpola, Mean teachers are better role models, NeurIPS 2017, arXiv 1703.01780. Agarwal et al., On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes, ICLR 2024, arXiv 2306.13649. Shenfeld, Damani, Hübotter and Agrawal, Self-Distillation Enables Continual Learning, arXiv 2601.19897 (ICML 2026). Hübotter et al., Reinforcement Learning via Self-Distillation, arXiv 2601.20802 (ICML 2026). Zhao et al., Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models, arXiv 2601.18734 (ICLR 2026). Khamis et al., iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs, arXiv 2609.24646. Lampinen et al., On the generalization of language models from in-context learning and finetuning: a controlled study, arXiv 2505.00661. Xu et al., A Theory of Usable Information Under Computational Constraints, arXiv 2002.10689. Wang et al., Denser ≠ Better: Limits of On-Policy Self-Distillation for Continual Post-Training, arXiv 2607.01763. Koh et al., Continually Post-training Small Models via Synthetic Self-distillation (SynthSelf), arXiv 2610.13113. Mirzadeh et al., Improved Knowledge Distillation via Teacher Assistant, AAAI 2020, arXiv 1902.03393. Negation Neglect: When models fail to learn negations in training, arXiv 2605.13829. Yue et al., Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?, arXiv 2504.13837 (NeurIPS 2025). Puri et al., Reaching Beyond the Mode: RL for Distributional Reasoning in Language Models, arXiv 2603.24844. Bahlous-Boldi et al., Vector Policy Optimization: Training for Diversity Improves Test-Time Search, arXiv 2605.22817. Interviews with Ilija Lichkovski and Ryan Bahlous-Boldi on Deep Learning with Yacine (2026). Maei and Sutton, GQ(λ): A general gradient algorithm for temporal-difference prediction learning with eligibility traces, AGI 2010.

This article was co-authored by Łukasz Stafiniak and Claude (Opus 5.5).