Study notes: Every Layer a Store

Wk_j=v_j\,(k_j^\top k_j)+\sum_{i\neq j}v_i\,(k_i^\top k_j). With unit-length keys the first term is the stored value. The sum is interference: every other value, weighted by the overlap of its key with the query. Each overlap is about \pm d^{-1/2}, and the M-1 terms add up like a random walk, so the interference grows as (M/d)^{1/2} relative to the signal. It matches the signal at about M=d. The least-squares store recalls exactly up to d pairs and reaches the same ratio at about 2d.

A ReLU zeroes the negative overlaps, so about half the interfering terms vanish: a factor of two. Squaring makes small overlaps very small while the stored key’s overlap of 1 stays 1. At d=64 the capacity is 2\times64^2/3\approx2{,}700 pairs. Raising the overlap to the power p gives capacity of order d^p.

The exponential is a sum of all the powers, e^{\beta s}=\sum_p(\beta s)^p/p!. With \beta=8 the exponent is 2\times8\times(1-1/8)=14, and e^{14}\approx1.2 million pairs. Nothing is trained into attention. Its keys and values are computed from the context while the model reads, and its M is the number of positions it sees. The feed-forward layer is written by training and holds what the model knows; the attention cache is written by reading and holds what the model has been told.

As clean vectors: about 128 and about 256. With d^2=16{,}384 the thresholds are n\ln n=2{,}048 and n\ln n=8{,}192, and 350\ln350\approx2{,}050, 1{,}160\ln1{,}160\approx8{,}185. Language models do not need clean recall. They need the right answer to win among a vocabulary of candidates, and under that criterion a linear store holds many more pairs. The counts are asymptotic, and “order d^2” hides a logarithm that costs about a factor of ten.

Full history, a rolling window, selected entries, compressed entries, a recurrent state.

• 1. keeps 1,000,000 entries and reads all of them.
• 2. keeps 128 and reads 128 (MiMo’s window layers).
• 3. keeps all 1,000,000 and reads 16\times128=2{,}048 positions per group of heads (MiniMax-M3).
• 4. keeps about 250,000 and reads 1,024, plus the 128 most recent tokens uncompressed (DeepSeek-V4-Pro’s compressed sparse layers).
• 5. keeps about 7,800 and reads them all (the same model’s heavily compressed layers).
• 6. keeps a matrix per head whose size does not depend on the prompt’s length, and reads that (Kimi K3).

A selector is a second, cheaper store placed in front of the first. It scores keys, and the main attention reads only the winners. The choice is a hard top-k, so no gradient flows through it, and the indexers learn by imitating the main attention’s distribution. The new failure is the miss: an entry the indexer did not choose cannot be read, however relevant it was.

Gated DeltaNet: the row decays to (1,1/2), the prediction for the key is 1, the error is 2, and the write adds 1/2\times2=1 to the first coordinate, giving (2,1/2). KDA: the row decays to (1,1), the write again adds 1, and the result is (2,1). The same association was corrected, and the second channel’s content survived. The delta rule buys the ability to overwrite, not capacity. Repeated writes approach the least-squares store, whose advantage is about a factor of two.

The cache is the store that grows with the context, and it holds the content: values, names, the facts a later question will need. The recurrent state is the store that every token reads, and it holds a few settings that condition everything the model writes: which language, which voice. When one prompt’s cache was paired with another prompt’s state, the answer took its content from the first and its language from the second. The dictionaries in these tests held 15 to 30 entries, well within what a 128-dimensional state can hold, and recall through the state failed even in minimal versions of the tasks. The division is learned.

As a context grows, so does the number of keys that compete with the right one. Softmax attention suppresses random competitors exponentially, but its capacity is finite at a fixed sharpness. The remedy is to sharpen the softmax as the context grows, multiplying \beta by a factor that grows with the logarithm of the context length. Real contexts are not random: they repeat entities, templates and phrasings, and no sharpening separates near-duplicate keys. A model whose architecture accepts a million tokens may therefore still fail to use them, which is why the second essay treats long context as a training problem.

1. The mixed residual is (2.5,3.5), and its total is still 6. The block reads 3 and adds 3 to each stream: (5.5,6.5). The mixing conserved the old total, and the block added new content on top. (b) The softmax weights are 1/4 and 3/4, and the input is (1/4,3/2). AttnRes is the explicit store: a growing cache of layer outputs, read by attention. mHC is the implicit one: a fixed-size state, written by outer products, whose stream differences contract at every layer. A store over depth lives only for the duration of one token’s computation.

The stores written in training lose contents only when training writes over them. The window gives up the old. Compression gives up detail: distinctions within a pool. Selection gives up whatever its indexer misses. The recurrent state gives up what decays and what is overwritten. The full cache gives up nothing, and it pays with a reading cost that grows with the context.

• Recurrent layers with a delta-rule write, mixed with attention.
• Learned selection of what to read.
• Learned compression of what is kept.
• A split between reading a prompt and writing an answer.
• Replacements for the plain residual sum.

Most of them are answers to one question: what a memory should be allowed to forget.

A Transformer with a fixed number of layers can do only a bounded amount of sequential computation for each token it emits. A task that needs more has to be done in steps, and the steps appear in the text, where a monitor can read them. Two affine updates compose into one, (A_2,b_2)\circ(A_1,b_1)=(A_2A_1,\,A_2b_1+b_2), and composition is associative, so L updates combine in a balanced tree of about \log L rounds: \log_2 10^6\approx20. The three paths:

• Recurrence across positions: a numerical state carries information that the emitted text need not express. Ordinary Transformers have this path too, through their caches.
• Recurrence through depth: repeated passes through the same layers before a token is emitted. None of the seven does this.
• Generated text, which adds steps of computation but need not explain them faithfully.

1. The counts are per key dimension, not per parameter, and measured per parameter architectures differ less than the counts suggest. They also assume random keys. Real keys share structure, which raises interference beyond what any count predicts.

2. The models were far smaller than the seven, the answers short, and the results for the largest models came from a single run. Discarding one store also leaves the model in a state it never met in training.

3. Whether the delta-rule layers escape \mathrm{TC}^0 depends on their transition matrices, and the released code keeps the write step between 0 and 1, so names like “linear attention” or “delta rule” do not settle where these models sit. Every catch the probe paper shows was also visible in the text, so the probes added detection, not access to hidden reasoning. Depth bounds what a model can do between tokens, but it does not certify what the model did.