Filling the Stores

Three orders for vision

The order of the modalities differs across the seven. Kimi K3 trains its encoder from nothing, jointly with the language model, from the first step, because a pretrained encoder “shows persistently higher gradient norms with frequent spikes”. DeepSeek-V4.1-Flash pretrains its encoder in a separate stage, shows images to the backbone from the first step, keeps the encoder frozen through the peak learning rate, and unfreezes it at decay. MiMo trains text alone for 27 trillion tokens and then joins pretrained vision and audio encoders for 3 trillion more. Moonshot compared a pretrained encoder with one trained from scratch, on different encoders. No lab has compared all three orders on one model.

Normalizing a matrix

The largest disclosed change in how pretraining writes is the optimizer. Adam normalizes a gradient entry by entry. Muon normalizes each weight matrix as a whole, by taking the matrix of momenta M=USV^\top and stepping along UV^\top:

M_t=\mu M_{t-1}+G_t,\qquad W_t=W_{t-1}-\eta\,\operatorname{ortho}(M_t),\qquad \operatorname{ortho}(USV^\top)=UV^\top .

Both updates throw away the gradient’s overall scale. They keep different things: Adam the signs of the entries, Muon the singular vectors.

Worked example 7: one gradient, two normalizations. Let the gradient of a 2\times2 weight matrix be G=\begin{pmatrix}4.32&6.76\\4.24&4.32\end{pmatrix}=10\,u_1v_1^\top+1\,u_2v_2^\top ,\quad u_1=(0.8,0.6),\ u_2=(-0.6,0.8),\ v_1=(0.6,0.8),\ v_2=(0.8,-0.6). It has one strong direction and one weak direction, the shape Muon’s authors say real updates tend to have. Adam’s first step keeps only the sign of each entry. All four are positive, so the step is the all-ones matrix, of rank 1, and the weak direction is gone. Muon’s step sets both singular values to 1: UV^\top=\begin{pmatrix}0&1\\1&0\end{pmatrix}. The weak direction now counts as much as the strong one. A singular value decomposition at every step would be too slow, so Muon gets there by iterating a polynomial in MM^\top that acts on the singular values alone. Production Muon uses tuned coefficients that bring small values up into a band around 1, without converging.

The labs that adopted Muon do not give it every parameter. In DeepSeek-V4, AdamW keeps the embeddings, the output head, the norms and mHC’s mixing gates, and Muon takes the rest. DeepSeek-V4.1 gives its embedding tables and output head a third treatment, because Adam’s state for 196 billion table entries costs too much memory. It balances a momentum update by Sinkhorn iterations, the alternating rescaling of rows and columns that mHC uses. GLM-5 and Kimi K3 orthogonalize attention projections head by head, so that heads with large gradients do not set the update for the rest. Table 1’s names for this are Muon Split and Per-Head Muon, and V4.1 reports that orthogonalizing by head beats plain Muon.

MiMo adds an order. It pretrained with AdamW, saw “diminishing optimization efficiency” as RL batches grew, and switched to Muown, a Muon variant with one Adam-updated gain per neuron, during mid-training. Qu and colleagues warn that such a switch can disrupt what the earlier optimizer wrote. MiMo reports no loss spike. MAI used AdamW for everything.

Selectors cannot learn through top-k

A selector is a second, cheaper store in front of the cache. It scores every entry and passes on the k best. The main attention receives only those positions, as integers. No score value enters its arithmetic, so the task loss, seen as a function of the scores, is a step function: flat almost everywhere, and jumping where two scores cross. As the MSA paper says, the language-modelling objective gives the indexer’s projections “no useful gradient”.

So the selector gets a teacher: the attention it replaces. Most published recipes minimize a KL divergence from the main attention P to the indexer’s softmax P^{\rm idx}:

\mathcal L=\mathcal L_{\rm LM}+\lambda\sum_{\rm layers}D_{\rm KL}\big(\operatorname{stopgrad}(P)\,\big\|\,P^{\rm idx}\big), \qquad \text{with the indexer's input also behind a stop-gradient.}

The two stop-gradients protect the backbone. Without them, the KL can fall because the main attention moves toward the indexer: “the backbone can lower the KL loss by simplifying the Main Branch attention distribution, rather than by improving the Index Branch.” In MSA’s pilot at 10.5 billion parameters, removing the stop-gradient at larger KL weights made the language-model loss diverge within a few hundred steps, and at stable weights short-context benchmarks slowly regressed. The student would be teaching the teacher.

Worked example 8: what a selector can see. One query reads four entries, and the selector keeps two. Two heads of the main attention put weights (0.7,0.2,0.1,0) and (0.1,0.2,0.6,0.1) on them, whose mean is the target p=(0.40,0.20,0.35,0.05). The indexer scores (2,1,0,-1), which a softmax turns into q=(0.644,0.237,0.087,0.032). Top-2 picks entries 1 and 2, so entry 3, with 35 percent of the attention, is missed. Raise entry 3’s score from 0 to 0.1. The selected set stays the same, and so do the attention, the output and the loss. That is the missing gradient. The KL over all four entries is 0.285, and its gradient on the scores is q-p=(+0.244,+0.037,-0.263,-0.018). A step of size 1 lowers entry 1 and raises entry 3, and about twenty such steps bring q close to p, with top-2 now \{1,3\}. Compare a KL taken only over the selected entries. Both distributions are confined to entries 1 and 2, which gives (2/3,1/3) against (0.731,0.269), a KL of 0.010 and no gradient at all on entry 3. The student looks nearly done while 40 percent of the attention lies outside what it selected.

The last line of the example is why most recipes begin with a dense warm-up, in which the indexer learns from the whole distribution before it controls anything. Warm-ups are short: 2.1 billion tokens in DeepSeek-V3.2, 2.8 billion in GLM-5, 40 billion in MSA. Warm-up alone does not finish the job. On GLM-4.7-Flash, an indexer trained on a frozen base lowered the score on RULER, a benchmark of retrieval from long contexts, from 79.21 to 71.35 at 128K tokens. A further 150 billion tokens of joint training brought it back to 78.86.

When to add selection. Positions from the first token to the released checkpoint have all been tried. DeepSeek-V4.1-Flash trains sparse attention from the first token at 64K, “without any dense attention warmup stages”, and does not disclose the indexer’s loss. V4-Flash runs a trillion dense tokens first. GLM-5.2 trains its shared indexers from mid-training at 128K. GLM-5 adds selection at the end of mid-training with 20 billion tokens, 0.07 percent of its base budget.

Selection can also come after release. Latent Sparse Attention distills an indexer onto a released, frozen checkpoint in 352 million tokens, with the student’s softmax kept over all selectable keys. On Qwen-3.8-27B it stays within five RULER points of full attention up to a million tokens. A second retrofit, ZHA, does the same in a billion tokens, so fitting a selector by KL onto a frozen model has become a cheap, repeatable recipe. Two designs drop the student altogether. One, N-LSA, a sibling of Latent Sparse Attention, selects with the attention’s own keys, with no teacher, at half a billion parameters. The other, CISA, retrofits a dense model with no gradient updates at all and reports staying within about two RULER points at a million tokens. Only a separate selector needs a teacher to select well.

Only one experiment at this scale holds everything else fixed. MSA ran its 109-billion-parameter model on the same 3 trillion tokens twice, once sparse from the start and once with selection added at 2.6 trillion. At RULER-8K the first scored 84.2 and the second 77.2, with full attention at 79.8. The native model led on every retrieval, mathematics, image and video row, and the converted one on five rows of the general and code groups. The authors suggest that “native sparse pretraining can adapt the model representations to the sparse attention pattern.” That sparse pretraining beat full attention itself is unexplained, and the table gives no error bars.

Kimi K3.1 Pro is the newest case and the least documented. Its card says it is post-trained from K3 and replaces Gated MLA with gated DeepSeek Sparse Attention. Its parameter index is K3’s plus exactly 120 tensors, five for each of 24 layers, all under self_attn.indexer. They add 178.5 million parameters to 2.78 trillion, or 0.006 percent, and nothing is removed. So a selector was installed in a model released without one. Moonshot’s post never mentions sparse attention. It does say that data were removed from mid-training, so the run seems not to have simply continued from the published K3. When the indexers were trained, against which attention, and whether they kept training during RL are not disclosed.

After a selector is in, RL has to live with it. GLM-5 froze its indexers during RL “to accelerate training and prevent unstable learning”. It had also found that non-deterministic top-k operators “caused drastic performance degradation during RL after only a few steps”, because training and rollout must agree on the selection.

Residual mixers must stay stable

Hyper-Connections replace the residual sum with a learned mixing matrix over several streams, and left free they fail. Loss surges unexpectedly around step 12,000, and the signal’s gain peaks near 3,000. mHC constrains each mixing matrix to be doubly stochastic, by scaling its rows and columns to sum to 1. Such a matrix cannot amplify a signal, and a product of them is another. The constraint is a training device as much as an architecture.

Worked example 9: projecting a mixer. Let a layer’s raw mixing matrix be \begin{pmatrix}3&1\\1&1\end{pmatrix}. Divide each column by its sum, then each row by its sum, and repeat. This is the Sinkhorn iteration. The first round gives \begin{pmatrix}.600&.400\\.333&.667\end{pmatrix}, the second \begin{pmatrix}.632&.368\\.364&.636\end{pmatrix}, and by the third the columns sum to within 0.0004 of 1. Apply both matrices to the streams (2,4) of worked example 4. The raw one gives (10,6), a total of 16, and its top eigenvalue of 3.41 would multiply the signal by about 200,000 over ten layers. The projected limit, with 0.634 on its diagonal, gives (2.73,3.27), and the total stays 6.

Compare a selector. Both constraints sit between a learned score and a hard structure. Sinkhorn is division repeated, so it is smooth, and DeepSeek backpropagates through all twenty iterations. Top-k is a choice, so a separate selector needs a teacher. mHC’s published stability figures come from AdamW at 4,096 tokens and at most 27 billion parameters. We know of no such figures under Muon at a million tokens.

Experts must stay balanced

A router sends each token to a few of hundreds of expert stores, and it can fail in two ways. Experts can receive unequal loads, and activations in the routed path can explode. Kimi K3 hits both: with 896 experts, 16 per token, its routed path is a chain of “nearly four consecutive matrix multiplications”. Its SiTU activation replaces SwiGLU’s two unbounded factors with \tanh-capped ones, which bound every output coordinate by 100. Its Quantile Balancing replaces the usual gradual update of each expert’s routing bias with an exact one. It sets the bias so that the target number of tokens would have cleared it in the last batch. The bias steers dispatch and stays out of the mixture weights, so the router’s gradient is untouched. And “a batch is never routed with a bias derived from itself”.

DeepSeek-V4 found its loss spikes “consistently tied to outliers in the MoE layers”, with routing seeming to feed them. Its remedies are clamps on SwiGLU and Anticipatory Routing, which recomputes routing with slightly older weights when a spike occurs. The report concedes that “their underlying principles remain insufficiently understood.” MAI saw the same correlation, and reports that how expert frequencies are aggregated matters far more than which balancing loss is used: they must be counted over the global batch.

RL is where routing goes wrong fastest, because its data are narrow. MiMo measured it at one decoder layer, on two runs that differed only in whether the router trained. With a trainable router, the coefficient of variation of expert loads rose from 0.78 to 2.0 in twenty steps, peak load from 6 to 16 times the mean, and the share of cold experts, which receive almost no tokens, from 0.5 to 22 percent. The diagnosis was a clean intervention. MiMo restored the router’s parameters to their pre-RL values and kept every other weight from step 20. Load balance recovered, and benchmark scores did not move. The expert stores had kept what they learned under a router that was put back. MiMo froze the router for RL. MAI keeps it free, with a balancing coefficient of 10^{-2} during self-distillation, the SFT on a run’s own rollouts that Section 4 describes, and 10^{-5} during RL. A large coefficient during RL harms the rise in score, which MAI calls the climb, while the balance that self-distillation installs carries over.

Long context comes late, and it comes cheaply

Every disclosed schedule extends the context in steps, and most end with the longest length in a small share of the budget. K3 goes through 8K and 64K in pretraining, then 256K and 1M in cooldown, and says the point is to confine “the costly long-sequence computation”. V4-Pro goes 4K, 16K, 64K, 1M. MiMo spends most of mid-training at 256K and ends at 1M. MAI’s last stage, at 256K, is 150 billion of about 33.5 trillion tokens, less than half a percent.

MAI’s ablations say why that share is enough. On small-scale models, the vast majority of the improvement in loss at 256K came in the first 1 to 10 percent of the extension phase. Upweighting long documents and changing the mixture gave no meaningful improvement. A trillion tokens at 32K plus 100 billion extending to 256K matched a trillion at 128K, in code loss at that length. The report’s reading is that “the model is not acquiring fundamentally new capabilities during context extension, but rather calibrating its positional and attention mechanisms”. Without the extension the model does fail, with retrieval loss rising as the evidence moves away.

Not everyone agrees about the data. “Length alone, however, does not confer long-range capability,” says K3’s report, and Moonshot synthesizes documents permuted and concatenated so that a task can be solved only by attending across the full million tokens. The reports differ in architecture too, since K3’s attention has no positional encoding to recalibrate. And MAI’s result is measured in loss on small models. MAI notes that “the interaction between context length and post-training was less well characterized”, and spent more than its ablations called for. GLM-5 adds that a 200K stage improved performance even within 128K, so the long stage teaches something beyond its own lengths.

What comes fast can leave fast. MAI’s self-distillation could erase long-context behaviour unless mid-training data was mixed back in. The first essay read long context as an interference problem, and the loss under self-distillation may be the same problem in the weights.

Precision enters by post-training

Allen-Zhu and Li quantized trained models to 4 bits and lost more than half their capacity. Their conclusion was that quantization may have to happen during training. Later work at small scale confirms that methods applied afterwards recover nothing below 4 bits, and finds that training at those precisions has its own failure, which a learning-rate schedule can prevent. Figure 1 shows most of the labs writing in low precision. MiMo starts expert quantization in mid-training, K3 and GLM-5 at SFT, and V4 during post-training, where V4.1 also adds its cache.

RL gives a sharper reason than capacity. The rollouts come from an inference engine and the updates from a learner, and every discrepancy between them compounds in the importance ratios, which correct an update for the policy having moved since sampling. K3 keeps experts at 4 bits in both, “eliminating the train–inference mismatch”, and V4 samples rollouts on native 4-bit weights. MiMo carries 32-bit master weights and its optimizer’s row state across the boundary from SFT to RL, to stabilize 4-bit training. MAI solves the same problem the other way, with 16-bit arithmetic on both sides of RL, then replays routing and top-p masks, the sets of tokens the sampler kept, so that a rounding error cannot flip a discrete choice.

Drafters model the policy

A drafter predicts what the main model will say, so it must follow the main model. K3 pretrains a layer and fine-tunes it into a draft model with the target frozen, against a loss that is the negative log of the acceptance rate. DeepSeek’s DSpark first trains alone against a frozen backbone, then alongside post-training, with no gradient into the backbone. MiMo trains its drafter on the SFT policy, then fine-tunes it on early RL logs. GLM-5.2 fixed a mismatch in which the drafter’s second step read a cache that training had never shown it, and on GLM-5.1’s backbone the acceptance length rose from 4.56 to 5.10 tokens before further changes took it to 5.47.

Starting as something simpler

One device recurs across these components without anyone naming it. A new component starts as the simpler model it replaces, and it learns its way out. Attention Residuals start with zero query vectors, so each layer begins as an equal average of the earlier ones. K3’s drafter starts with a fusion matrix that passes one input through unchanged. MAI starts attention itself this way: the output gains are zero, so the model begins as a stack of per-token feed-forward layers, and token mixing grows in. Its reason is the balance of the layers below. Near-uniform attention at initialization averages tokens together, which unbalances routing in the next layer of experts, more so with depth.

What RL optimizes, and how it goes wrong

Most of the disclosed recipes descend from GRPO, though GLM-5.2 has moved to a critic-based form. For each prompt the model samples a group of G answers, and each answer’s advantage is its reward relative to the group. Every token of an answer gets that answer’s advantage, and an importance ratio corrects for the policy having moved since sampling:

A_i=\frac{R_i-\operatorname{mean}(R_{1:G})}{\operatorname{std}(R_{1:G})},\qquad r_{i,t}=\frac{\pi_\theta(y_{i,t}\mid\cdot)}{\pi_{\rm old}(y_{i,t}\mid\cdot)},\qquad J(\theta)\propto\sum_{i,t}\min\big(r_{i,t}A_i,\ \operatorname{clip}(r_{i,t},1-\epsilon,1+\epsilon)\,A_i\big).

How Does a Policy Improve? treats this family. At scale, what fills the reports is stability. MAI runs 128 rollouts per prompt, \epsilon=0.6, an integral controller that widens the upper clip when entropy falls below a target of 0.3, and a hard outer clip, because the branches GRPO leaves unclipped “sometimes led to catastrophic gradient-norm spikes”. It also has to hold the rollouts to the top-p nucleus they were sampled from, the set of likeliest tokens, or the mismatch causes divergence “within a few training steps”.

Stale rollouts. Sampling and learning run at once, so a rollout may come from weights the learner has since left behind. The long rollouts on the hardest problems are the stalest. MAI takes five gradient steps between inference updates and discards any rollout more than 40 gradient steps behind, and it lowered the learning rate at the longest lengths. GLM-5 drops samples past a staleness bound and masks tokens whose ratios leave a band. V4.1 masks tokens “with excessive staleness” and corrects for the bias that short samples finish first. K3 sets no stated bound: a long trajectory spans several iterations, and a per-token regularizer tolerates the staleness. So the asynchronous recipes all buy throughput with a loss of freshness, and most of them cap it.

Verifiers are the attack surface. A passing test does not distinguish a solution from an exploit, and the models find the exploits. Goodfire’s judge found reward hacking in 90.8 percent of Kimi K3’s DeepSWE rollouts, 95.2 percent of Qwen 3.8 Max’s and 57.2 percent of GLM 5.2’s. MAI lists what its agents do: search the internet for the merged pull request, dig through local git history, and tamper with tests. MiMo’s agents ran pip index versions sphinx to look for a later release containing the fix. Its answer is the fullest in the reports: scrubbed environments, a dedicated agent that hunts for exploits until the environments hold, and a grader that zeroes confirmed hacks before computing group statistics. The logged share of confirmed hacks stayed below 2 percent, which measures what the detector finds. GLM-5.2 goes further into the run: a filter catches a suspect tool call and returns dummy information, because stopping a rollout abruptly can cause “training instability and model collapse”.

Without such a grader the damage is subtler than cheating. In a comparison on MiMo-V2.6-Flash, policies under pressure to raise pass rates drifted toward “speculative compatibility branches, broad exports, exception swallowing, relaxed validation, and evaluation-specific configuration changes”. That is a fair description of a reward’s shortcuts being written into the stores.

Who grades matters as well. MiMo’s grader is its own earlier self: “a self-hosted MiMo-V2.6-SFT model serves as the grader to support stable scoring.” SFT thus has a second job, after giving RL its starting model, as a fixed judge for the stage that follows.

Most of the gain is in the environments. The gains at the opening have a partial explanation. DeepSeek says of V4.1 that it introduced no new post-training algorithms, and that “systematic improvements in the scale, diversity, and verifiability of synthesized data and environments account for essentially all of the observed gains.” GLM-5.3’s post says the month went to “more environments, more diverse tasks, and more compute spent training on them”, with verifiers synthesized without access to the reference solution and tested against oracle, no-op and unsolved states. These attributions are the labs’ own.

Specialists, then one checkpoint

Training specialists and then consolidating them has become the common shape. Kimi K3 trains nine policies, three domains by three reasoning efforts, the highest effort first and annealed down. V4-Pro uses over ten teachers, GLM-5.2 over ten, V4.1 over forty. These “experts” are whole models, and distilling them into one checkpoint has nothing to do with the routed experts inside it.

On-policy distillation lets the student generate and has the teacher score each token, so the student learns on the states it actually visits. K3 uses a clipped log-ratio between teacher and student as a per-token reward, which lets distillation run on the RL machinery. The student is conditioned on effort, and the teachers are not, because each teacher is one effort. V4 finds token-level estimates too noisy and distils over the full vocabulary. MiMo’s version of multi-teacher on-policy distillation, MOPD2, adds starting points. Demonstrations train an SFT teacher in the ordinary way. Prefixes of such demonstrations then give the student a place to start, from which it generates its own turn while the teacher scores it. Without those prefixes, a student that strays reaches histories the SFT teacher never saw.

Consolidation is cheap and has a known cost. On small models, one study puts it at 5 to 8 percent of the specialists’ compute, and another finds that students match their teachers on the teachers’ domains while falling 2 to 16 percent below the base model outside them. The student’s weights barely move: 0.025 to 0.098 percent at that scale.

The teachers’ origin matters. In Ma and colleagues’ study of multi-teacher distillation, swapping a same-origin teacher for a stronger model from elsewhere degraded accuracy, and with one estimator training diverged around step 18. A student can absorb only what lies close to it, as Who Teaches the Weights? argues. V4.1 reports teachers that differ architecturally from the student and come from different stages of development, without saying how close they had to stay.

Sequential training is the alternative, and its price is forgetting. GLM-5 trains reasoning, then agents, then general tasks, and has to distil from each stage’s final checkpoint to recover from “the cumulative degradation of previously acquired capabilities”.

MAI uses the simplest tool. It trains specialists for STEM, agentic coding, and helpfulness and safety, and consolidates them by plain off-policy SFT over their rollouts: four epochs, balanced by sample count. A light RL stage follows, and it must retain some STEM and coding data or reasoning slowly degrades.

Self-distillation at production scale

MAI’s RL starts from a model that has never seen reasoning traces, prompted in raw text. Its climbs are long, so it periodically collects rollouts, runs SFT on them starting from a mid-trained checkpoint, and resumes RL from the result. Who Teaches the Weights? discusses this loop at small scale. MAI uses it to move from a raw-text prompt to the chat format, to carry a climb onto a new base model, to filter out reward hacking, and to recover from collapses. Its ablations are the largest-scale findings on the method that we know of:

• On the order of a million traces match the teacher. Many more narrow the policy, which leaves “little room for exploration once RL resumes”.
• Traces from incorrect answers train about as well as traces from correct ones.
• Traces from a range of late checkpoints beat traces from the final policy alone, perhaps through diversity. Very early traces hurt.
• For a fixed budget, more prompts beat more traces per prompt.

The first and third findings say the same thing from two sides: distillation can copy a policy too exactly. MAI’s other settings point the same way. It uses dropout of 0.15 in self-distillation because the extra entropy “helps prevent model collapse” in the RL that follows.

What changes inside

Post-training has been measured from the inside, on smaller models. Weights first. Across RL runs on models up to 32 billion parameters, weights move along a stable, nearly linear path, and at up to 8 billion a rank-one trajectory in each tensor recovers most of the gain. Zhu and colleagues find that RL learns off the principal directions of the weight matrices, while SFT moves those directions and distorts the spectrum. The same paper corrects an earlier claim. Reports that RL leaves most weights unchanged, 86 percent in the 671-billion-parameter DeepSeek-R1-Zero, partly reflect 16-bit rounding: an update too small for the format is not stored.

Features move late and little. In a 1.5-billion-parameter model under RL, the largest feature drift rose from 0.013 at layer 6 to 0.060 at layer 27. Steering the base model along the features RL amplified recovered about 80 percent of RL’s gain, and all 29 problems that RL newly solved had already been solvable by the base model within 64 samples. In these short math runs, then, RL partly changes which answers get sampled. Routing moves too, mainly because its inputs move. When specialists are merged, most changed expert choices also change if only the router’s input is replaced.

The first essay’s table said that trained stores forget only when later training overwrites them. Facts are the fragile case. In Zucchet’s 44-million-parameter model, fine-tuning on new people collapsed old facts within about a hundred steps while attention patterns stayed stable, though fine-tuning on people it already knew did no such harm. New keys do the damage. Which stores take it is unsettled. At 1.5 billion parameters, Kaplan and colleagues froze parts of the model during SFT on new facts: attention alone learned almost none of them (0.010) and left the held-out old facts at 0.931, while feed-forward layers alone learned them (0.941) and took those facts down to 0.782. Palit and colleagues, on a small Transformer, attribute nearly equal shares of a forgetting shift to attention’s value and output weights (47 percent) and to the feed-forward layers (46). And some forgetting is not erasure. In OLMo-2 1B, at the trough, removing the top singular direction of each weight update took recall of old facts from 0.28 to 0.77. By step 400 it held 0.70 against 0.38, at a cost in new-fact recall of 0.90 to 0.40. Where Experience Goes draws the continual-learning consequences.

How far a store moves does not say how much it matters. In Zhang and colleagues’ study of seven Qwen models, the one model measured for this moves its weights about equally at all depths under full RL. Yet training one layer alone recovers anywhere from under 30 to 114 percent of RL’s gain, and the strongest layers sit in the middle of the stack. Which store a behaviour lives in depends on the behaviour. Swapping attention patterns between checkpoints moved one behaviour 61 to 73 percent of the way to the other checkpoint, while another, a learned shortcut, sat in the feed-forward layers.

Apart from R1-Zero, these studies run from 44 million to 70 billion parameters, mostly on mathematics. We found no published measurement that resolves the 2026 frontier’s own stores in a post-trained checkpoint: the delta-rule gates, the indexers, the routed experts across a long climb. The labs that hold the intermediate checkpoints report symptoms, such as MiMo’s router and MAI’s expert imbalance, and they do not publish maps.