Every Layer a Store

The outer-product store

Store pairs of vectors, a key k_i and a value v_i, by adding their outer products into one matrix:

W=\sum_i v_i k_i^\top .

To recall the value stored under key k_j, multiply:

Wk_j=v_j\,(k_j^\top k_j)+\sum_{i\neq j}v_i\,(k_i^\top k_j).

With unit-length keys the first term is the stored value. The sum is interference: every other value, weighted by how much its key overlaps the query. If the keys are random vectors in d dimensions, each overlap is about \pm d^{-1/2}, and the M-1 terms add up like a random walk. The interference grows as (M/d)^{1/2} relative to the signal, and it matches the signal at about M=d. A d\times d matrix written this way recalls about d random associations as clean vectors.

The linear associative memory. The outer-product store is half a century old. Teuvo Kohonen and James Anderson published it independently in 1972. Kohonen showed that recall is perfect when the keys are orthogonal and named the “crosstalk” that non-orthogonal keys produce. Anderson, writing about neurons, computed a signal-to-noise ratio that falls with the number of stored traces. Neither gave a capacity number. The first one came a year later. Kohonen and Ruohonen replaced the sum of outer products with the pseudo-inverse of the key matrix. That map recalls every stored value exactly when the keys are linearly independent, so for arbitrary values it stores at most as many pairs as the keys have dimensions. Beyond that it returns the least-squares approximation. An online route to the same map is older still. In 1960, Bernard Widrow and Ted Hoff trained an adaptive linear unit by measuring its error on one pattern at a time and moving each weight against the gradient of that error. Section 3 meets their rule again, inside Kimi K3 and Qwen3.8.

The least-squares store does better than the outer-product store, though not by much. It cancels the interference of correlated keys, recalls exactly up to d pairs, and in Wang’s estimate reaches the same interference-to-signal ratio as the outer-product store at about 2d pairs.

Nonlinearity suppresses interference

A feed-forward network computes

f(x)=\sum_i v_i\,\sigma(k_i^\top x),

which is the outer-product store with a nonlinearity \sigma applied to each overlap before it weights a value. The rows of the first weight matrix are keys and the columns of the second are values. A ReLU zeroes the negative overlaps, so about half the interfering terms vanish. That buys a factor of two. Squaring after the ReLU does far more. Small overlaps become very small, the stored key’s overlap of 1 stays 1, and the capacity grows to about 2d^2/3: some 2,700 pairs at d=64. A gated unit with a tied gate behaves the same way. Raising the overlap to the power p gives capacity of order d^p.

That feed-forward layers store associations this way is more than an analogy. A version of Attention Is All You Need already replaced the feed-forward layers with attention over learned key–value pairs, at about the same translation quality. Later versions dropped the passage. Sukhbaatar and colleagues folded such learned pairs into self-attention itself and removed the feed-forward layers without loss. Geva and colleagues took a trained 16-layer language model and looked at what the keys of its feed-forward layers respond to. For 65 to 80 percent of the inputs that most activate a key, annotators found a shared pattern: shallow ones, such as a common last word, in the lower layers, and semantic ones in the upper layers.

Hopfield and the 0.14 N limit. In 1982 John Hopfield stored patterns in a network of binary neurons with outer-product weights and let the network settle from a corrupted cue. From simulations with up to 100 neurons he reported that “about 0.15 N states can be simultaneously remembered before error in recall is severe”: at that load about half the memories were lost. Amit, Gutfreund and Sompolinsky derived the limit in 1985 with the tools of spin-glass physics: retrieval states exist up to 0.138N patterns, and at that load 1.5 percent of their bits are wrong. Demand exact recall and the count falls. McEliece and colleagues showed that most memories are recovered exactly up to about N/(2\log N), and all of them only up to N/(4\log N). Change the learning rule and the count rises. Elizabeth Gardner computed the capacity of the best possible couplings, whatever rule finds them, and got 2N. Error-correcting rules of the perceptron type reach it, and Hebb’s rule does not. Between these results the number of stored patterns moves by a logarithmic factor and more, depending on what counts as remembering and how the weights were set.

Dense associative memory. In 2016 Dmitry Krotov and John Hopfield generalized the energy of the Hopfield network so that each overlap enters through a power n. With small recall errors allowed, the capacity grows as N^{n-1}, and n=2 recovers the classical 0.14 N. Wang’s power p is their n-1. Demircigil and colleagues proved that an exponential energy gives a capacity exponential in N: 2^{N/2} patterns can be fixed points. Ramsauer and colleagues carried the exponential version over to continuous patterns and showed that its update rule is Transformer attention.

Softmax attention uses every power

The exponential is a sum of all the powers: e^{\beta s}=\sum_p(\beta s)^p/p!. Let a store weight each value by e^{\beta s}, where s is the overlap of its key with the query and \beta sets the sharpness. It suppresses interference more than any single power does. Normalize the weights by their sum, and the store is softmax attention:

f(q)=\sum_i v_i\,\frac{e^{\beta k_i^\top q}}{\sum_j e^{\beta k_j^\top q}} .

Wang’s estimate for random keys gives a capacity of about e^{2\beta(1-\beta/d)}. With the usual sharpness, \beta=d^{1/2}, that is about 1.2 million pairs at d=64.

Attention is a different kind of store from the feed-forward layer. Nothing is trained into it. Its keys and values are computed from the context while the model reads, and its M is the number of positions it sees. The feed-forward layer is written by training and holds what the model knows; the attention cache is written by reading and holds what the model has been told.

Sparse distributed memory. Pentti Kanerva’s model of 1988, described in his later summary, places about a million “hard locations” at random addresses in a space of 1,000-bit words. Writing a word at an address adds it to every location within a fixed Hamming distance of that address. Reading at an address sums the locations within that distance and takes a majority vote. Seen as a network, it has two layers: a fixed random first layer that decides which locations an address activates, and a second layer of counters. Its capacity grows with the number of locations, not with the word size. Kanerva’s practical guide was to store 1 to 5 percent as many words as there are locations. Bricken and Pehlevan showed that attention approximates its read operation when keys and queries are normalized to unit length and the sharpness is suitable. So the thresholded store of a feed-forward layer and the exponential store of attention have a classical model between them. Sparse distributed memory also bears on a current debate in interpretability, about whether a network’s neurons are clean. With random addresses, one location holds unrelated items by design. When Bricken and colleagues let a sparse distributed memory learn its addresses under the right constraints, its units specialized to classes of inputs instead.

What counts as remembering

All these numbers count clean recall: the recalled vector must be closer to the stored value than the interference is large. Language models do not need that. They need the right answer to win, among a vocabulary of candidates. Under that criterion a linear store holds many more pairs. Nichani, Lee and Bietti proved that an outer-product matrix of size d\times d can pick out on the order of d^2 answers, up to logarithmic factors. Barnfield and colleagues found the sharp constants: the best possible linear map succeeds while d^2/(n\ln n) stays above 2, and the outer-product map while it stays above 8.

Worked example 1: one head, two criteria. Take a store with 128-dimensional keys and values, the size of one head in several of the models below. As clean vectors, it recalls about 128 random associations written as outer products, and about 256 written to the least-squares solution. Picking the right answer from the stored values, it recalls about 350 written as outer products, and about 1,160 written by the best possible linear map. The last two numbers come from solving n\ln n=d^2/8 and n\ln n=d^2/2 with d^2=16{,}384. They are asymptotic. In Barnfield and colleagues’ simulations the best map holds more at finite sizes than the asymptotic count, and their finite-size formula gives about 1,800 here. Either way, “order d^2” hides a logarithm that costs about a factor of ten.

Two further cautions hold for every number in this section. The counts are per key dimension, not per parameter. A feed-forward store that holds d^2 pairs needs about d^2 hidden units to do it. Measured per parameter, architectures differ less than these counts suggest. Training language models on synthetic biographies, Allen-Zhu and Li found that they store about 2 bits of knowledge per parameter when each fact has been seen about a thousand times, with gated feed-forward layers, plain ones, or none. When each fact has been seen a hundred times the figure falls to about 1 bit, and there gated feed-forward layers store 1.3 times less. And the counts assume random keys. Real keys share structure: the same entity appears in many facts, and templates repeat. Shared structure raises interference beyond what any count here predicts.

Experts and lookup tables

A mixture of experts splits one large feed-forward store into many smaller ones and sends each token to a few. In storage terms the router performs a coarse key match before recall: it removes most of the distractors before any expert sees the query. The cost in capacity is small. In Allen-Zhu and Li’s experiments a model with 32 experts stored 1.3 times less knowledge per parameter than a dense model, even when each token went to a single expert and used 8.8 percent of the parameters. For facts seen only a hundred times the gap was 1.5 times. How much a mixture of experts can store is better judged from its total parameters than from its active ones.

Not every layer needs to be split. Microsoft’s MAI-Thinking-1, whose weights are not released, alternates a feed-forward store split among 512 experts with a small store that every token reads. In Microsoft’s ablations this layout matched experts in every layer at equal computation and trained faster on the hardware, and adding a shared expert brought it little or nothing.

Two of the seven address some stores without a learned router. DeepSeek-V4-Pro routes tokens in its first three layers by a hash of the token’s identity, so the address is the token itself. DeepSeek-V4.1-Flash adds Engram: 196 billion parameters of tables in two layers, looked up by hashes of the last two to four tokens. Both are stores whose keys are fixed in advance and whose values are learned.

Feed-forward layers and fast weights. The distinction between stores written in training and stores written while reading has its own history. In 1992 Jürgen Schmidhuber proposed a slow network that writes, as it reads a sequence, the weights of a fast network, with each change an outer product of two of the slow network’s outputs. Katharopoulos and colleagues showed in 2020 that attention without the softmax can be computed as a recurrence over such a matrix. Schlag, Irie and Schmidhuber then identified that linear attention with fast weights. They also drew the consequence from the outer-product store: in d dimensions there are at most d orthogonal keys, so storing more than d associations must produce retrieval errors. Their remedy was a write that first reads what the memory predicts and stores only the correction. It is the subject of the next section’s recurrent layers.

Selection is learned retrieval

Four of the seven choose which entries to read. GLM-5.3 uses DeepSeek Sparse Attention: a small “lightning indexer” scores every earlier position for each query, and the main attention reads the 2,048 best. Its IndexShare lets one indexer serve a group of four consecutive layers. DeepSeek-V4-Pro’s compressed sparse attention applies the same kind of indexer to compressed entries, and DeepSeek-V4.1-Flash stacks two levels of it (Section 5). MiniMax-M3’s sparse attention (MSA) selects blocks of 128 positions, separately for each group of heads that share keys and values, and always includes the block that contains the query.

A selector is a second, cheaper store placed in front of the first. It scores keys, and the main attention reads only the winners. Because the choice is a hard top-k, no gradient flows through it, so the indexers learn by imitating the main attention’s distribution. The second essay describes how. What they learn can be inspected. In MiniMax’s analysis of its 109-billion-parameter research models, the indexer learns to select the first block of the sequence, which holds the “attention sink” that many heads use as a default. The new failure that selection brings is the miss: an entry the indexer did not choose cannot be read, however relevant it was.

Compression pools positions

DeepSeek-V4-Pro’s compression is learned. Each entry of its compressed sparse attention pools 8 consecutive tokens. In each of 512 channels separately, the layer computes softmax weights over the 8 tokens and takes the weighted average. Adjacent entries overlap by 4 tokens. One pooled vector serves as both key and value, shared by all 128 query heads. The heavily compressed layers pool 128 tokens per entry without overlap and attend to all of them, so they need no indexer. Each query also sees the 128 most recent tokens uncompressed, in the same softmax as the compressed entries.

In storage terms, pooling writes several positions into one entry. Within a pool the positions share a key, and attention cannot tell them apart. A fact spread across two tokens of the same pool can still be read. Two facts in one pool that a later query needs to separate cannot.

The recurrent state learns while it reads

Kimi K3 and Qwen3.8 put a recurrent layer in three of every four positions. Each keeps a matrix S per head, treated as the weights of a linear map from keys to values. For each token the layer reads the map’s current prediction for the token’s key, compares it with the token’s value, and writes the correction:

S_t=S_{t-1}+\beta_t\,(v_t-S_{t-1}k_t)\,k_t^\top .

This is one step of gradient descent on \lVert S k_t-v_t\rVert^2/2, and it is Widrow and Hoff’s rule of 1960. Schlag, Irie and Schmidhuber introduced it into linear attention so that a full memory could overwrite instead of piling up interference. The write step \beta_t, between 0 and 1, is computed from the token. Before the write, a learned decay shrinks the old state. In Qwen3.8’s Gated DeltaNet the decay is one number per head and token. In Kimi K3’s Kimi Delta Attention (KDA) it is one number per channel, so a head can keep some directions for a long time and let others fade quickly. Where Experience Goes works through an example of the rule overwriting a redefined word.

Worked example 3: per-channel forgetting. Let one row of a memory be S=(2,1), and write the value 3 under the key (1,0) with \beta=1/2. Gated DeltaNet first decays the whole row, say by 1/2, to (1,1/2). The prediction for the key is 1, the error is 2, and the write adds 1/2\times2=1 to the first coordinate: the row becomes (2,1/2). KDA can decay the two channels differently. With retention (1/2,1) the row first becomes (1,1), the write again adds 1 to the first coordinate, and the result is (2,1). The same association was corrected, and the second channel’s content survived.

What the delta rule buys over purely additive linear attention is the ability to overwrite, not capacity. Repeated writes approach the least-squares store, and Section 2 put that store’s advantage at about a factor of two. The decay sets how long contents last. In both models the gates are bounded: the write step stays between 0 and 1 and the decay between full retention and near-erasure. Section 6 returns to why those bounds matter.

What the trained hybrids do with their stores

That a model has a recurrent store does not show what it uses it for. Three recent measurements on trained models give the first answers.

Lee and colleagues placed key–value facts in a prompt to Qwen3.5-4B, a smaller relative of Qwen3.8, and asked for one of them. Then they swapped internal states between two runs that differed in the stored value. Swapping all the recurrent states moved the answer by 0.2 to 3.6 percent of the full effect. Swapping the attention caches moved it by 96 to 100 percent. Exact recall ran through the attention layers.

Afendulev and colleagues found what the recurrent states carry instead. After processing a prompt, they kept either the attention cache or the recurrent state, discarded the other, and let the model answer. With the cache alone, Qwen3.5-4B still retrieved a value from a dictionary in the prompt 89 percent of the time; with the state alone, never. Asked in the prompt to answer in another language, it complied 70 percent of the time with the state alone and 1 percent with the cache alone. A requested persona survived the same way, 49 percent against 14. Then they paired one prompt’s cache with another prompt’s state, and the answer took its content from the first and its language from the second. Kimi Linear, Moonshot’s smaller relative of K3, showed the same division. The models were far smaller than the seven, the answers short, and the results for the largest models came from a single run. Discarding one store also leaves the model in a state it never met in training, which the paired swap partly addresses.

The recurrent layers also forget quickly, by design. In Qwen3.8-27B the median recurrent head halves its contents in about 24 tokens, and the ranking of heads from fast to slow is almost the same for every document, fixed by the weights. In Kimi K3, an estimate from the weights finds that 98 percent of recurrent heads mix channels that fade to a thousandth within 64 tokens with channels that last longer.

Together these results divide the labour. The cache is the store that grows with the context, and it holds the content: values, names, the facts a later question will need. The recurrent state is the store that every token reads, and it holds a few settings that condition everything the model writes: which language, which voice. Capacity is compatible with this division but does not explain it. Afendulev and colleagues’ dictionaries held 15 to 30 entries, well within what a 128-dimensional state can hold, and they report that recall through the state fails even in minimal versions of their tasks. The division is learned.

Long context is an interference problem

As a context grows, so does the number of keys that compete with the right one. Softmax attention suppresses random competitors exponentially, but its capacity is finite at a fixed sharpness. One remedy sharpens the softmax as the context grows, multiplying \beta by a factor that grows with the logarithm of the context length. Qwen’s LogN scaling is an early instance. In Wang’s simulations with random competitors, this keeps recall nearly perfect at every length he tried, up to two million entries. Real long contexts are not random: they repeat entities, templates and phrasings, and no sharpening separates near-duplicate keys. That is one reason a model whose architecture accepts a million tokens may still fail to use them, and why the second essay treats long context as a training problem.

Several streams, mixed

DeepSeek’s Manifold-Constrained Hyper-Connections (mHC) keep four copies of the residual stream for each token, stacked as the rows of a matrix X_l. Around each block \mathcal F_l, attention or feed-forward,

X_{l+1}=B_lX_l+C_l\,\mathcal F_l(A_lX_l).

The row vector A_l mixes the four streams into the block’s input. The column vector C_l distributes the block’s output back over the streams. The matrix B_l mixes the streams among themselves. All three are computed from the current streams, so they differ from token to token. Twenty rounds of alternately normalizing its rows and columns push B_l toward a doubly stochastic matrix: one with non-negative entries whose rows and columns each sum to 1. A doubly stochastic mix conserves the total over the streams, which keeps the signal from growing or vanishing across many layers.

Worked example 4: mixing two streams. Take two scalar streams, x=(2,4), and the doubly stochastic mix with rows (3/4,1/4) and (1/4,3/4). The mixed residual is (2.5,3.5), and its total is still 6. If the block reads the average of the streams, 3, computes 6 from it, and sends half to each stream, the result is (5.5,6.5). The mixing conserved the old total; the block added new content on top. In the trained model the rows of the mixing matrix sum to 1 exactly and the columns only approximately, so the conservation is approximate.

A doubly stochastic mix has a further property. Its average channel is an ordinary residual update. The differences between the streams contract at every layer, because only a permutation preserves them all, as Li, Sha and Yang show. The extra streams are a memory that fades over depth. Zhao and colleagues measured how a trained model uses them. In DeepSeek-V4-Flash, a typical block reads about two of the four streams and writes about two. In the later half of the network the mixing is close to the identity: replacing it with the identity raised perplexity by 1.9 percent. In the earlier half the mixing matters: replacing it raised perplexity by 41.4 percent. Yet it hardly depends on the input: fixing it to its average over tokens cost 0.2 percent.

DeepSeek-V4.1-Flash changes one coefficient. Its Single-Pass mHC reads each block’s input with A_{l-1}, the mixing vector computed one step earlier, instead of A_l. With a fused kernel, the shift lets the hardware read each tile of the streams once instead of twice, and it halves the memory traffic of the original implementation. The report says the shift costs negligible quality. It was used in pretraining, so it changes the model and not only its kernel.

Attention over depth

Kimi K3’s Attention Residuals (AttnRes) replace the sum with a lookup. A layer’s input is a softmax-weighted combination of earlier representations:

h_l=\sum_i\alpha_{i\to l}\,v_i ,

where the weights come from a learned query for layer l compared with the normalized candidates. In K3 the candidates are the embedding, the completed sums of earlier groups of 12 layers, and the running sum within the current group: up to nine sources at the last layer. (The AttnRes paper calls these groups blocks.)

Worked example 5: retrieval over depth. Let two earlier representations be (1,0) and (0,2), and let the layer’s query score them at 0 and \ln 3. The softmax weights are 1/4 and 3/4, and the layer’s input is (1/4,3/2). The lookup runs over layers of the same token, not over earlier tokens.

González-Martínez and colleagues read both designs as memories over depth. AttnRes is an explicit store: a growing cache of layer outputs, read by attention. mHC is an implicit one: a fixed-size state, written by outer products. The vocabulary of Section 3 carries over unchanged, with depth in place of position. Meta’s Muse Spark Open, released in October, replaces the plain residual sum with a learned gate. The plain sum is becoming one choice among several.

Neither design feeds a later layer’s output back into an earlier layer within a pass. A store over depth lives only for the duration of one token’s computation. Whatever persists from one token to the next persists in the stores of Section 3.