Across the track: Grammar and search inside networks

• The Algebraic Mind Meets the Neural World reads the query as a role address: attention is approximate tensor-product unbinding, the key advertises a role and the value is the filler retrieved (there).
• Merge and Selection reads it as a valence specification: the query says what a position seeks, as a head’s valence list says what it must combine with (there). It adds that the query is never among the summed values, so the head governs without being deposited.
• A Parser Without a Grammar reads it as a functor seeking an argument, with CCG’s slash as the directional signature (there). It adds that application consumes its functor, so directionality and monotonicity have to be separate mechanisms.

• The attention graph. Smolensky’s criticism of first-generation systems is that the compositional structure is used within a layer and not encoded in the activations passed on (Algebraic Mind).
• The selector. Merge and Selection calls Smolensky’s complaint the same observation as the binding problem one layer up: a value carries no trace of the key it matched, and the query is consumed in forming the weights (there).
• The abstractions. In Word Grammar types and relations are nodes that carry activation. In the Transformer they live in the weights, an operator applied to the stream. A Parser Without a Grammar calls this the same fact Merge and Selection located one rung down (there).

Depth entrenches one basin and breadth keeps several alive (The Given and the Found). In GSC, cooling T to zero converges on the single maximum-Harmony structure, and holding T yields a Boltzmann distribution over structures (Merge and Selection). The third essay identifies depth with the optimization regime and breadth with the sampling regime, and reads the limit cycle as the sampling regime realized in time instead of across restarts (there). The empirical separation of the knobs comes from the iterative reasoners. The single dynamics whose schedules they are comes from GSC. The limit cycle is also what a gradient blend can do and Dynamic Syntax’s underspecification cannot: never resolve.

At matched inference compute and with far fewer parameters, a tied network of 2 blocks run for 21 iterations beats a 42-block feedforward network on scarce data, and an operator trained at 16 iterations extrapolates to over a thousand (The Given and the Found). In a stacked Transformer each layer has its own projections, so the depth at which a filler was merged acts as an implicit timestamp and partly preserves who attended to whom. Tying removes the timestamp, and the collapse of (a/b)/(c/d) into (a/d)/(c/b) becomes exact (Merge and Selection). Merge and Selection notes that these looped models are the case the other essay treats as architecturally interesting.

It cannot locate where the correctness condition comes from. Every formalism, and the Harmony function beneath them, presupposes a grammar that is given, and the coordinates describe how the Transformer parses under a grammar it has learned (there). Construction Grammar and Word Grammar say that grammar is learned from usage, which names the outcome and not the induction that selects what to posit. The Given and the Found draws the same line for iterative reasoners: they amortize search over a condition that is handed over, and none amortizes the discovery of the condition (there). The learning of the grammar falls on the far side of that line.