Across the track: How a model keeps learning

• The self-teacher is the bare model tilted by the context’s log-ratio at \eta=1 (Part 1).
• A half step of distillation, and iSDFT’s per-token budget, are tilts at higher temperature (same section).
• A sequence-level reward preserves the ratios among equally rewarded answers, which a token-level target does not (Part 1).
• Repeated on-policy improvement lowers the effective temperature to \eta/k, which is why alternatives vanish (Part 1).

Teacher: the model with privileged context supplies the targets (Part 1). Explorer: the model reads its first answer and writes a different second one (Part 1). Library builder: writing answer i after reading the earlier ones performs the step “add the policy that most improves the set” (Part 2). E-step: the context is the fastest store and the best learner on the ladder, and slower stores absorb what it builds (Part 3).

• Reward essay: a robot that never opens the door has no example revealing the 6.
• Policy essay: regularization toward a prior cannot produce alternatives outside the prior’s support.
• Part 1: a strategy the model no longer produces is a strategy nothing can teach it, short of an outside demonstration.
• Part 2: where the behaviour never acts as the target would, the importance ratio is zero and nothing is learned.
• Part 3: a mass-covering fit cannot restore the alternatives a teacher lost.

The reward essay uses it for policy evaluation (V^\pi(J)=1.675), for policy iteration (a second greedy step finds the door) and for the policy gradient at the door (estimates 4.5 and 0, mean 1.125). Part 2 asks an off-policy question about it: the return of “approach and open” estimated from a behaviour that mostly does neither, by reweighting (\lambda=1) or by bootstrapping (\lambda=0), and not at all once the behaviour always takes the detour.

For self-distillation the candidates are the student’s samples, the evaluation is the same model with context, the improved distribution is the teacher’s conditional, and the improvement is kept in core weights. For GQ(λ) heads the candidates are the behaviour’s experience, reweighted toward many target policies, and what is kept is a prediction. No policy is fitted. They differ in the last question: one stores what to do, the other stores what would happen.

• Every Layer a Store: a selector is a second, cheaper store placed in front of the cache. Its choice is a hard top-k, so no gradient flows through it, and it brings a new failure, the miss.
• Filling the Stores: the selector gets a teacher, the attention it replaces. The KL sits behind stop-gradients so that the student does not teach the teacher. Selection has been added at every point from the first token to a released, frozen checkpoint.
• The memory essay: the teacher meets both conditions of Who Teaches the Weights? in a clean form. Dense attention has looked at every key, and the student only has to learn where the teacher would look. An indexer that scores every key makes finding the bottleneck, and Latent Sparse Attention replaces the scan with an index. A later weight update would stale that index along with the keys.

In Every Layer a Store, a longer context puts more keys in competition with the right one. Sharpening the softmax handles random competitors and cannot separate the near-duplicate keys of real text, so a model that accepts a million tokens may still fail to use them. Filling the Stores treats this as a training problem. The longest context arrives late and in a small share of the tokens, and it can leave as fast: self-distillation could erase long-context behaviour, which may be the same interference problem in the weights. The memory essay finds the problem at both ends of its line. In 2023 it was named the distraction issue and met by training on other documents as negatives. In 2026 full context trails both a trained reader and a distilled sparse attention.

On the ladder it is the second rung: a fixed-size matrix that every token updates by one error-correcting step, a TD(0) learner with no future to bootstrap from, and parametric continual learning already running inside frontier models. Every Layer a Store adds the learned decay before the write, per head in Gated DeltaNet and per channel in KDA, and that the rule buys overwriting, not capacity. In trained hybrids the state holds settings such as language and voice, the cache holds the content, and the state forgets quickly by design. The memory essay calls it a fast form of the third placement, into connections. But the store is bounded, it carries settings and not content, and it ends with the session.

In Part 2 the behaviour policy is an older checkpoint and the target a slightly updated one. A language model’s token ratios are exact, but their product over a long episode has enormous variance, so traces are cut. In Filling the Stores sampling and learning run at once, and the labs cap the staleness: they discard rollouts past a bound and mask tokens whose ratios leave a band. The memory essay is where the parallel breaks. A stale key has no ratio. It is a vector computed by a function that no longer exists, and reweighting does not change coordinates. The remedies are of another kind: re-encode, tolerate, translate, constrain, or keep keys the model did not compute.

Filling the Stores shows the release cycle’s consolidation from the inside. Specialists are distilled on-policy into one checkpoint, cheaply, with students falling below the base model outside the teachers’ domains. Self-distillation on a run’s own rollouts can copy a policy too exactly and can erase long-context behaviour. It also repairs a collapsed run, because it copies what the model does and leaves the weights behind. The memory essay splits the argument in two. Cheap reading raises the break-even of the amortization case, which is the right account of explicit memories. It leaves untouched what consolidation builds beyond them: representations no retrieved item contains. And a model that consolidates while it keeps a cache meets stale keys: a store written by old weights can undo what consolidation taught the new ones.