A Parser Without a Grammar: Where the Transformer Sits Among the Formalisms

2.1 Monotonicity: HPSG and the rewrite grammar at the far pole

“Merge and Selection” established the positive pole of this first axis: because the residual stream is never overwritten, only added to, the core update is a monotone merge — the join in the information ordering of a constraint grammar — rather than the destructive rewrite of a phrase-structure rule, which consumes its left-hand side. That is what made Head-driven Phrase Structure Grammar the comparison, the lexicalized unification formalism whose derivation is a bundle of features that only grows. We take that result as given and ask what sits at the other end of the axis.

The negative pole is anchored, cleanly and somewhat unfairly, by Transformational-Generative Grammar. The transformational tradition is built on operations that are destructive in the way unification is not: movement leaves a trace and displaces material, deletion removes it, the derivation rewrites its own intermediate structures on the way to the surface. A grammar of transformations is a rewrite system par excellence, and it is the formalism the residual stream’s accumulate-don’t-overwrite character is most directly unlike. This is why we include it as an anchor rather than a contestant: its value to the coordinate system is to mark the far end of the axis.

But the anchoring has to be done with care, because the obvious version is wrong, and a reader who knows the Minimalist Program will catch the error. Bare Merge, the basic structure-building operation of minimalist syntax, is not destructive at all — it is monotone set-formation, taking two objects and forming the set containing them, adding structure without removing any, and so sits near the residual stream’s pole rather than opposite it. What anchors the destructive end is specifically Internal Merge — movement — together with the older transformational apparatus of deletion and trace: the operations that displace and erase. The transformational tradition occupies the negative pole because of movement, not because of Merge, and the architecture’s monotonicity is a vote for Merge-without-Move: combination that only ever adds, with no operation that relocates or deletes what an earlier step established. Where a transformational grammar would move a constituent and leave a gap, the Transformer can only write another vector into the stream — displacement achieved by addition, not relocation. This will matter again when we reach the one operation that looks like retraction.

2.2 Directionality: CCG and the head the join throws away

The fracture “Merge and Selection” arrived at was this: the residual merge is a symmetric operation. A join is commutative — A ⊔ B = B ⊔ A — and unification does not distinguish a head from a dependent; it merges descriptions without governance. So the pure merge reading, faithful as it is to the monotonicity of the stream, erases the head-driven character that names HPSG. The Transformer’s combination, taken as a join, is headless.

There is a formalism whose combinatory operation is built to be directional from the start, and it is the natural exemplar of a second axis. Combinatory Categorial Grammar — Steedman’s framework, with roots in the categorial grammars of Ajdukiewicz and Bar-Hillel and a semantics that runs in lockstep with the syntax in the Montague tradition — assigns each word a category that is either basic or a function seeking arguments. A transitive verb is (S\NP)/NP: something that, given an NP to its right, yields something that, given an NP to its left, yields S. Combination is function application, and application is inherently asymmetric: one category is the functor, the other the argument, and the slash even encodes the direction in which the argument is sought. This is the asymmetry HPSG’s symmetric join discards, recovered as the primitive of the combinatory operation.

And it maps onto the Transformer with surprising directness — onto the part of the architecture the monotonicity reading left out. Consider the query/key mechanism on its own. A position computes a query, a specification of what it is looking for, and attends to positions whose keys match that specification. A query is, functionally, a function seeking an argument of a certain type; the key advertises the type of argument a position can serve as; attention is the application of the one to the other. The functor/argument asymmetry of CCG is the query/key asymmetry of attention. This reading is not original to us — the authors of the TP-Transformer described an attending cell as a subject querying others for an object, which is selection by another name — but CCG sharpens it: the slash direction, the thing categorial grammar adds to bare valence, is the directional signature attention has and a symmetric join lacks.

So CCG appears to repair HPSG’s defect. It does — and in doing so it incurs the opposite defect, the first instance of this essay’s thesis. CCG’s application is not monotone. When a functor applies to its argument, the functor is consumed: (S\NP)/NP combined with an NP yields S\NP, and the original category is gone, discharged into the result. Application is as destructive of its functor as a rewrite rule is of its left-hand side — that is what it means for the slash to “cancel.” So CCG sits on the wrong side of the monotonicity axis: it wins directionality and loses accumulation, where HPSG wins accumulation and loses directionality. Neither formalism gives both. And the Transformer, manifestly, wants both: it has the directional query/key asymmetry of CCG application and the monotone never-overwrite accumulation of HPSG unification, because in the architecture these are two different mechanisms — the query/key inner product on one hand, the residual write on the other — rather than two aspects of one combinatory operation. A discrete operation that builds a constituent must fuse them: its single combine is either symmetric-and-monotone (HPSG) or directional-and-consuming (CCG), because that step happens at one point and cannot be two things there. (Only a formalism that draws relations rather than building constituents escapes this — Word Grammar, §2.6 — and pays for the escape in having no constituents at all.) The Transformer factors them apart and runs them at different sites. This is un-bundling, and it is why CCG and HPSG each match the architecture on one axis and miss on the other: they are each a bundling of a directionality choice with a monotonicity choice, and the architecture has unbundled the pair.

One thing CCG genuinely contributes beyond the directionality axis, and we should credit it because it complicates a later section: CCG is incremental. Its flexible constituency — type-raising and composition allowing many derivations of the same string — lets even a simple left-to-right parser combine words as they arrive, without waiting for a phrase to complete. The brain study above leans on this: CCG’s left-branching incremental derivations are what let it track word-by-word processing. We flag it here because incrementality will return as its own axis, and CCG’s claim to it is real; what distinguishes the temporal exemplar we reach in §2.4 is not incrementality as such but the dissolution of the grammar/parser distinction, which CCG does not perform.

2.3 Strata: LFG and the conflation of position with function

A third axis concerns how many parallel structures a grammar maintains. Most formalisms produce one structural object — a tree, a derivation. Lexical-Functional Grammar maintains two, related but autonomous. There is c-structure, a constituency tree encoding precedence and dominance — who comes before whom, what groups with what — and there is f-structure, an attribute-value matrix encoding grammatical functions — subject, object, the features that unify — which is monotone and unification-based in just the way §2.1 described. The two are linked by a correspondence, the φ mapping, that says which nodes of the tree project which parts of the functional structure. LFG’s bet is that positional facts and functional facts are different kinds of information that need separate representations and a mapping between them, because within a single sentence the two need not line up: a fronted which book sits high in the tree but bears the object function of an embedded verb; a raised she occupies one position while filling the subject slot of two predicates; a dropped pro-subject is present as a function with no tree position at all. The non-alignment is intra-sentential, not merely cross-linguistic — it is there in one tree — which is why one structure with a fixed labeling will not do.

The Transformer does not obviously have two structures. It has one residual stream. And this is where LFG earns its place as a coordinate, by reframing an otherwise puzzling empirical result. The Tensor Product Generation Network finding that “Merge and Selection” engaged — that a model made to induce its binding roles unsupervised discovers neither pure positional slots nor pure grammatical functions but a hybrid of the two — becomes, on the LFG reading, evidence. It is the signature you would expect if the architecture is collapsing c-structure and f-structure onto a single vector, superimposing the positional stratum LFG keeps separate from the functional one. LFG predicts that a system with only one representational layer, asked to carry both kinds of information, would carry them entangled; the TPGN roles are that entanglement, observed.

So the architecture’s coordinate on the stratal axis is collapse: it occupies the position of a grammar that has run LFG’s two strata together into one. This rhymes with a recurring, unresolved finding in the interpretability survey — that syntactic and semantic information are sometimes reported as living in separate linear subspaces and sometimes as conflated, with no settled verdict. The stratal axis suggests why: a single stream can encode two strata in approximately-orthogonal subspaces (looking separate) or in overlapping ones (looking conflated), and which you observe depends on the probe, the layer, and how hard training pressed the two apart. And the un-bundling here runs the other way from §2.2: the architecture bundles what LFG keeps separate. So it is not uniformly a splitter — it unbundles selection from merge and bundles position with function, fusing where the formalisms split and splitting where they fuse.

2.4 Time: Dynamic Syntax and the grammar that is a parser

Every formalism so far describes a static object — a well-formed structure, a license on strings — and leaves to a separate “parser” the business of building it left-to-right in time. The competence/performance distinction is the orthodoxy that licenses this division: the grammar specifies what is well-formed, the parser is a performance mechanism that recovers it. Dynamic Syntax — Kempson, Meyer-Viol, and Gabbay’s framework — refuses the division. In Dynamic Syntax there is no static structure that the grammar characterizes and the parser then recovers; the grammar is the procedure for building interpretation word by word. Parsing is the incremental, strictly left-to-right growth of a semantic tree, driven by requirements — open goals, of the form “a formula of this type is needed here” — that later words discharge. The state of the parse at any moment is in general underspecified: nodes whose content is not yet fixed, requirements not yet met, structure that is provisional and gets resolved as more input arrives.

This is the formalism for decoder-only, causal Transformers, and it is the natural home of a reading “The Given and the Found” developed at length. A causal model processes strictly left-to-right; it cannot attend to the future; its representation at each position is built from what has come before and revised by what comes next, in the manner of a requirement awaiting discharge. Dynamic Syntax’s central dynamic — underspecification resolved to specification as parsing proceeds — is close to the picture that essay drew of the layer stack (or the iteration count of a looped model) driving an initially high-entropy, many-interpretations-superposed state toward a committed one. Where Dynamic Syntax has a node whose value is not yet fixed and a requirement that some later word will satisfy, the Transformer has a residual state that is a blend of partial commitments, sharpened as computation proceeds. The competence/performance collapse is the deepest agreement: a Transformer has no separate “grammar” stored apart from the “parser” that applies it — the weights are the procedure, there is no declarative grammar object anywhere in the system, and this is the most Dynamic-Syntax-like commitment in the entire architecture. The empirical layerwise findings from the survey fit the picture: positional information processed in lower layers, a switch to more hierarchical encoding in higher layers, is what incremental specification looks like from the probe’s side.

Dynamic Syntax thus owns the temporal axis as CCG owns the directional one, and the distinction we promised in §2.2 can now be made precise. CCG is incremental — it can combine as it goes — but it remains a competence grammar with a separate notion of what is well-formed; its incrementality is a property of how its derivations can be ordered. Dynamic Syntax is incremental in a stronger sense: it has no competence/performance split to be incremental within, because the growth process is all there is. The architecture’s coordinate on this axis sits at the Dynamic Syntax end — grammar-as-process, not grammar-as-object-plus-parser — and this is the un-bundling of a fourth bundled pair: every other formalism fuses “what is well-formed” with “the system has a stored characterization of well-formedness,” and the Transformer has the first without the second.

2.5 Inventory: Construction Grammar and what lives in the weights

The axes so far concern the per-layer operation — how a step combines, in what direction, maintaining how many structures, growing in time. A different axis concerns what the grammar contains. The mainstream generative tradition draws a sharp line between the lexicon (idiosyncratic, listed) and the grammar (general, rule-governed), and treats the rules as the locus of productivity. Construction Grammar — Goldberg, Croft, and in its usage-based form Bybee and Hilpert — denies the line. For Construction Grammar the basic unit is the construction: a learned pairing of form and meaning, at every level of granularity, from morphemes through words and idioms up to abstract argument-structure schemata like the ditransitive. There is no separate rule component; there is a single continuum from the wholly idiosyncratic to the wholly schematic, all of it stored, all of it the same kind of object, and the grammar is the inventory of these constructions. The framework is resolutely usage-based: constructions are entrenched by frequency, learned from data, with abstract schemata generalized over stored exemplars.

This is a description of the Transformer’s weights, and it lands at a different level from the previous four axes, which is its value. The other formalisms describe what a layer does to a representation as it passes through; Construction Grammar describes what is stored in the parameters that do it. And the match is close enough to be more than analogy. The feed-forward (MLP) layers of a Transformer have been read, in the interpretability literature, as a key-value memory: each MLP neuron responds to a pattern in its input (a key) and writes an associated vector into the stream (a value), and the layer is a large store of such pattern-response pairs retrieved by the current context. A stored mapping from a form-pattern to a content-contribution, retrieved by frequency-shaped activation, is — at the level of functional description — a construction. And the feature superposition of the residual stream, the near-orthogonal directions carrying many features at once, is the representational image of Construction Grammar’s central commitment: a continuum of stored units from the concrete to the abstract, with no categorical break between “lexical” and “grammatical,” because superposed directions admit every degree of specificity without a sortal boundary. The lexicon-grammar continuum is the feature continuum.

Construction Grammar is silent on the combinatory dynamics — it says what is stored, not how stored units are composed in a derivation — and so it is complementary to Dynamic Syntax rather than competing with it: Construction Grammar is the inventory axis, Dynamic Syntax the combination axis, and the architecture has coordinates on both. The bundled pair the architecture unbundles here is the lexicon/rule distinction itself — the generative tradition fuses “productive combination” with “rule component distinct from the listed lexicon,” and the Transformer has productive combination with no such distinct component, everything graded along one continuum of stored, retrievable units. That this is the most usage-based, most learned-from-frequency framework on our list, matched to the most thoroughly trained-from-data architecture, is no coincidence: it is where the formalism and the mechanism agree about epistemology, both holding that grammatical knowledge is generalization over experienced instances rather than a given rule system. We will return to that agreement, because it is also where the architecture’s deepest limitation lives.

2.6 Topology: Word Grammar and the grammar that is an activation network

The sixth axis is topology — the shape of the structural object the grammar builds: at one pole a phrase-structure tree, with constituents, brackets, and part-whole containment; at the other a flat network of word-to-word links, with no node that is a phrase rather than a relation. The Transformer sits at the network pole, and here the mapping barely needs an argument. An attention matrix is a weighted directed graph over positions; the biaffine scorer of the neural dependency parsers, which assigns a score to every candidate head-modifier arc, is structurally that same matrix; and the interpretability survey keeps rediscovering the fact from the data side — heads that specialize in particular dependency relations, while constituency probes fare worse, one cited study conceding the models barely acquire constituency at all. What the architecture builds is a network, not a tree. Generic dependency grammar sits at the same pole but is theoretically thin, nearer an annotation scheme than a theory of the object; as a coordinate it says little beyond “head-modifier links, no phrases.” Richard Hudson’s Word Grammar occupies the pole with a commitment that turns the shared shape into a genuine point of contact.

The commitment is that language does not produce a network, it is one. Where the other formalisms yield a structural object standing apart from the process that builds it — a tree, a derivation, a pair of strata — Word Grammar denies there is any such separate object. Knowledge of language is a single network of atomic nodes — word-types, sub-lexemes, the tokens of use, and an open-ended taxonomy of relational concepts (subject, sense, realisation) — joined by labelled directed links, with no phrase nodes anywhere; processing is spreading activation over the network, and generalization is default inheritance up the isa-hierarchy. The machinery is the knowledge-representation tradition’s — semantic networks, frames, inheritance — and at its furthest edge, in Lamb’s relational-network linguistics, a grammar is offered as something that could be wiring. This is the commitment no other coordinate makes. Dynamic Syntax (§2.4) makes grammar a process, but a symbolic procedure unfolding in time, with nothing said about activation; Construction Grammar (§2.5) says what is stored, but not that the store is a graph that lights up and fires into its neighbours. Word Grammar is the one formalism whose primitive object is an activation network over a weighted directed graph — which is, by construction, the Transformer’s kind of object.

So the form is shared, and shared at a level the other formalisms do not reach: the residual stream as an activation pattern over the graph, attention as the per-layer scoring of the links, the flatness that follows from neither having phrase nodes, and a structure that grows only by adding links — Word Grammar never consumes a node, the residual stream never overwrites. The interesting facts are therefore not the agreements but the two places where the shared form is cashed out differently. They are independent of each other, and the section turns on keeping them apart.

The first is what bears the activation. In Word Grammar the activated nodes are reified concepts, and the network holds types and tokens together in one graph: the abstraction subject, the lexeme BOOK, the relational concept sense are themselves nodes that grow active and fire into their neighbours — Hudson’s evidence for the links is priming, one concept raising the activation of related ones, abstractions activating abstractions. The store of knowledge and the medium of processing are the same activated graph; program and data are not separated. The Transformer separates them. Activation rides only particulars — token positions, contextualized further up the stack but always “this token, here.” The abstractions live in the weights: a learned map engaged by every matrix multiply, but an operator applied to the activations, never a node that carries state forward or fires into a neighbour. (The weights are not inert — each matmul engages them — but they are not activation-carrying; they transform the stream rather than flow in it.) The operator/operand split that Word Grammar refuses is the architecture’s organizing fact, and it is the same fact “Merge and Selection” located one rung down: the abstract relational apparatus — there the head and its governance, here the type and the relation — is kept out of the activated representation and held in the machinery that acts on it.

The second difference is the older one, now properly parallel to the first rather than containing it: what the substrate is made of. Word Grammar’s nodes are individuated and addressable — a node that is the subject relation, a link that is this dependency — and each link carries its type on its face. The architecture’s “nodes” are superposed directions in a vector space, and the superposition is the survey’s most robust structural finding read from the inside: a single relation smeared across many heads, a single head carrying several, agreement riding subspaces of as few as three dimensions and tied to no unit. An attention weight is a bare scalar with no label, its grammatical force — if it has one — implicit in which subspace the routed value lands in. The unlabelled gradient link is something like the pre-quantized form of Word Grammar’s discrete typed dependency, and it is the seam at which §3’s substrate re-enters.

Both differences show most sharply where Word Grammar would seem strongest — in its generalization mechanism, default inheritance, which does two jobs and does both, in Word Grammar, by consulting discrete addressable nodes in an explicit hierarchy. Neither crosses into the architecture as that. Subsumption — a token inheriting its type’s properties, the type its supertype’s — becomes geometry: Park, Choe, Jiang, and Veitch define the subordinate relation by token-set inclusion (Word Grammar’s isa written extensionally) and prove that the containment forces orthogonality — the parent feature’s vector orthogonal to the difference vectors distinguishing its children, hierarchical concepts encoded as direct sums of polytopes, inherited content in the shared parent direction and distinguishing content in an orthogonal complement, recursively up the tree; validated on Gemma and LLaMA-3 and, against the worry that this is a noun artifact, replicated on the verb hierarchy. The inheritance hierarchy has a real analogue, but one implicit in the angles between directions: where Word Grammar consults a hierarchy and copies a value down, the architecture arranges directions so the parent’s content is a shared component of the child’s. Override becomes additive cancellation: the IOI circuit’s negative name-mover heads write against the token they attend to, and Gurnee and colleagues’ suppression neurons — late-layer, the mirror of prediction neurons — together with partition neurons that boost one group while suppressing another, are the mechanism, unified with the self-repair (Hydra) line. But where Hudson now stresses that Word Grammar’s inheritance is monotonic — the more specific value is simply the one inherited, resolved at the moment a concept is created, so the default is never laid down and never retracted — the architecture does the opposite: it writes the default and a cancelling term and forwards the algebraic sum, the suppression clustered in the very last layers, after the affirmative predictions have accumulated through the middle. Nothing is selected over a node; two pushes are added and one dominates. This also closes the loop to §2.1: a strictly additive stream cannot retract, so the non-monotonicity exception-handling needs is bought not by deletion but by writing a vector that points the other way.

So the formalism whose primitive object is closest in form to the architecture’s — an activation network over a weighted directed graph — diverges from it on two independent counts: what bears the activation, and what the substrate is made of. The abstractions Word Grammar reifies as activable nodes are, in the Transformer, the static operator; the discrete typed links it can point to are superposed gradient directions. Both divergences are the question §3 takes up — the medium in which structural commitments can stay un-quantized, and the abstractions can stay in the map.