Study notes: A Parser Without a Grammar

The retired question is which grammatical formalism the Transformer secretly implements. Probing shows that structure is recoverable from hidden states, and is silent on what operation produced it. Stanojević and colleagues found that structure-building metrics from a CCG parser explain brain activity over and above the next-word predictability estimated by a Transformer, and that the two effects sit in different places. The lesson the essay takes is that structure-building and predictability are different functions, so a model can excel at the second while the first stays a separate question. In place of the retired question, the formalisms are read as a coordinate system. They disagree along identifiable axes, each disagreement is a design decision, and the architecture comes down somewhere on each. No single formalism shares all its coordinates, because each bundles choices the architecture pulls apart.

Bare Merge is monotone set-formation: it takes two objects and forms the set containing them, adding structure and removing none. It sits near the residual stream’s pole. The destructive end is anchored by Internal Merge, that is movement, together with deletion and trace: the operations that displace and erase. The architecture’s monotonicity is “a vote for Merge-without-Move”. Where a transformational grammar would move a constituent and leave a gap, the Transformer can only write another vector into the stream. Displacement is achieved by addition, not relocation.

Given an NP to its right the verb yields S\NP. Given an NP to its left that yields S. At each step the functor is consumed: (S\NP)/NP is gone, discharged into the result, which is what it means for the slash to cancel. On the attention side, the query is a function seeking an argument of a certain type. The key advertises the type of argument a position can serve as. Attention is the application of one to the other. The slash direction is the directional signature that attention has and a symmetric join lacks. The TP-Transformer’s authors already read an attending cell as a subject querying for an object. The sharpening through CCG, with application’s non-monotonicity, is the essay’s own mapping.

HPSG’s join is commutative, A\sqcup B=B\sqcup A. It merges descriptions without governance, so it is monotone and headless. CCG’s application is directional and not monotone: it wins directionality and loses accumulation. A discrete operation that builds a constituent takes its step at one point and cannot be two things there. Its single combine is either symmetric and monotone or directional and consuming. In the Transformer the two are different mechanisms at different sites: the query/key inner product is directional, and the residual write is monotone. Each formalism therefore matches the architecture on one axis and misses on the other. Word Grammar escapes by drawing relations instead of building constituents, and pays by having no constituents at all.

C-structure is a constituency tree encoding precedence and dominance. F-structure is an attribute-value matrix of grammatical functions, monotone and unification-based. A correspondence \phi says which tree nodes project which parts of the functional structure. LFG keeps them apart because position and function need not line up within one sentence: a fronted which book sits high in the tree and bears the object function of an embedded verb, and a dropped subject is a function with no tree position. The Transformer has one residual stream, so its coordinate is collapse. The essay reads the hybrid roles induced by the Tensor Product Generation Network, neither purely positional nor purely grammatical, as the two strata superimposed on one vector. Here the architecture bundles what a formalism keeps separate. It is not uniformly a splitter: “fusing where the formalisms split and splitting where they fuse”.

CCG remains a competence grammar with a separate notion of what is well-formed. Its incrementality is a property of how its derivations can be ordered. Dynamic Syntax has no competence/performance split to be incremental within. The grammar is the procedure for building an interpretation word by word, strictly left to right, driven by requirements that later words discharge, and the state of the parse is in general underspecified. A causal Transformer also works left to right and cannot attend to the future. Its residual state is a blend of partial commitments, sharpened as computation proceeds. It stores no declarative grammar apart from the parser: the weights are the procedure. The pair that comes apart is “what is well-formed” and “the system has a stored characterization of well-formedness”. The Transformer has the first without the second.

The first four axes describe what a layer does to a representation as it passes through. This one describes what is stored in the parameters. For Construction Grammar the basic unit is the construction, a learned pairing of form and meaning at every granularity, and there is no separate rule component, only a continuum from the idiosyncratic to the schematic. The first mapping: the MLP layers, read in the interpretability literature as a key-value memory, store pairs of a pattern and a vector written in response, and such a pair is functionally a construction. The second: feature superposition admits every degree of specificity without a sortal boundary, so the lexicon-grammar continuum is the feature continuum. The generative tradition fuses productive combination with a rule component distinct from the listed lexicon; the Transformer has the first without the second. Formalism and mechanism agree about epistemology: grammatical knowledge is generalization over experienced instances.

Both are activation networks over a weighted directed graph. An attention matrix is such a graph over positions, neither has phrase nodes, and both grow only by adding. Word Grammar holds that language is one network, processed by spreading activation.

What bears the activation. In Word Grammar the abstractions, such as the relation subject or the lexeme BOOK, are nodes that become active and fire into their neighbours. The store of knowledge and the medium of processing are one graph. In the Transformer activation rides only particulars, token positions. The abstractions live in the weights, an operator applied to the activations.

What the substrate is made of. Word Grammar’s nodes are individuated and addressable, and each link carries its type. The architecture’s are superposed directions, and an attention weight is a bare scalar with no label, something like the pre-quantized form of a typed dependency.

Subsumption becomes geometry. Park and colleagues define the subordinate relation by token-set inclusion and prove that containment forces orthogonality: the parent feature’s vector is orthogonal to the differences that distinguish its children, so inherited content lies in a shared parent direction. Word Grammar consults a hierarchy and copies a value down; the architecture arranges directions. Override becomes additive cancellation, which the essay reads into negative name-mover heads and suppression neurons. Hudson’s inheritance is monotonic: the more specific value is simply the one inherited, so the default is never laid down and never retracted. The architecture writes the default and a cancelling term and forwards the sum, with the suppression clustered in the last layers. A strictly additive stream cannot retract. The non-monotonicity that exceptions need is bought by writing a vector that points the other way.

Un-bundling is a factorization, not a blurring. Directionality lives in the query/key inner product and monotonicity in the residual write, two mechanisms at two sites, and that has nothing to do with continuity. A wholly discrete architecture running selection and accumulation as separate steps would un-bundle them too, and Word Grammar already does. The blend also has symbolic precedent: Dynamic Syntax holds a node’s value open, chart parsers carry packed forests, and disjunctive feature structures hold alternatives in one description. What the discrete side lacks is the conjunction in one homogeneous medium. The axes are factored apart, each is held at a graded value, and the alternatives are superposed, all by a single repeated operation. Gradient Symbolic Computation is the theory of that medium, not a seventh formalism.

Structures. The blend is a point in a representation space, and points can be convexly combined; no vector is half application and half unification. Two coordinates are superpositions in the structural sense. Word Grammar’s unlabelled gradient link is a binding held at low q, not yet quantized to a typed dependency. LFG’s collapsed strata are c-structure and f-structure superposed in one state. The axes coming apart is a different, non-metric fact about separate mechanisms: “What blends is parses under a fixed scheme; what un-bundles is the machinery.” Dynamic Syntax’s underspecification is a partial description over an enumerated set of completions, and the parse must end as one of them. The gradient blend answers to no discrete alternative and need never resolve.

Depth is iteration along a single trajectory, which entrenches whatever attractor has the state. It is the optimization regime: T is driven to zero and the dynamics converge on the single maximum-Harmony structure. Breadth is restarts from many initializations, which keep several attractors alive. It is the sampling regime: T is held and the result is a distribution over high-Harmony structures. The limit cycle, a trajectory that visits the valid readings in turn and commits to none, is the sampling regime realized in time instead of across restarts. So depth and breadth are one selection-plus-merge dynamics under two schedules, not two networks. The empirical separation of the knobs is the companion essay’s; the single dynamics is GSC’s.

• Monotonicity: HPSG, with transformational grammar at the far pole. The stream only adds.
• Directionality: CCG. The query/key asymmetry, run as a mechanism separate from the monotone merge.
• Strata: LFG. One stream carrying position and function together.
• Time: Dynamic Syntax. Grammar as process, with no stored grammar.
• Inventory: Construction Grammar. A continuum of stored units, with no rule component.
• Topology: Word Grammar. A network and not a tree, without discrete typed links or reified abstractions.

Each formalism stakes out a corner by bundling choices, and the architecture’s coordinates cut across every bundle, so it sits in the interior. A parser without a grammar is not a parser that lacks structure. It has no discrete, stored, declarative grammar object, only soft constraints learned into the weights. “The structure is real. What is missing is the commitment.”

• Recoverability is not computation. Every coordinate is a claim about what is represented in the stream, one step removed from what operation a layer performs. The hierarchy geometry is a theorem about the unembedding, and the inference to “the layers compute by inheritance” is one “we have not earned and do not claim”.
• Scope. That geometry is demonstrated for lexical-semantic taxonomy and only conjectured to extend to grammatical categories.
• Mappings of the essay’s own. CCG application as attention, Dynamic Syntax as causal decoding and suppression as override are offered as readings. A reader may reject any one and keep the thesis, which is the pattern of un-bundling.
• A debt. The coordinate system cannot locate where the correctness condition comes from. Every formalism, and GSC beneath them, presupposes a grammar that is given. Construction Grammar and Word Grammar name the outcome of learning and not the discovery. This is the boundary The Given and the Found drew between amortizing search over a given condition and amortizing the discovery of one.