Dispersed Minds, Simulated Selves: What LLM Interpretability Reveals About Artificial Cognition

What Interpretability Reveals

That Transformer cognition is radically dispersed — distributed across both the depth of the network and the breadth of the context window — is obvious from the architecture itself. What mechanistic interpretability has shown is that this dispersal is genuinely structured: not a homogeneous blur of computation but a layered, functionally differentiated process with identifiable stages, specialized circuits, and recoverable internal logic.

Start with the vertical axis. Through causal tracing, activation patching, and logit lens diagnostics, researchers have mapped a rigorous layer-wise stratification in how LLMs process information. Query extraction — identifying what a prompt is asking about — happens in the earliest layers. Entity and attribute assembly — associating subjects with their properties — emerges in middle layers, where clustering patterns of intermediate representations form. Final answer aggregation occurs in late-stage MLP layers, which write the accumulated, logically processed information back into the residual stream to govern the output token. In the largest publicly documented architectures — Llama 3.1 405B with 126 Transformer layers, DeepSeek-V3 with 61 — this staged computation unfolds across a substantial vertical depth.

The horizontal axis tells a complementary story. Recent work on chain-of-thought (CoT) reasoning has uncovered “thought anchors” — specific sentences within a reasoning trace that exert disproportionate causal influence on both the trajectory of subsequent reasoning and the final answer. These anchors cluster overwhelmingly into two functional categories: plan generation (establishing the strategic framework for the reasoning) and uncertainty management (expressing confusion, initiating backtracking, or re-evaluating prior logic). Specialized “receiver heads” in the model’s attention layers consistently direct massive attention from subsequent tokens back to these anchor tokens, using them as persistent reference coordinates. The mind, such as it is, is smeared across the token stream. Its coherence is maintained not by a central executive but by distributed attention patterns that reach back to structurally important earlier positions.

The Blackboard Without a Reader

The residual stream — the high-dimensional vector that gets iteratively refined as it passes through each layer — serves as a central “blackboard” or shared workspace. This is a genuine architectural parallel to early cognitive science models: independent processing modules (attention heads, MLP sublayers) read from and write to a common substrate. The parallel to Baars’ Global Workspace is explicit and has been formally evaluated. On the six testable markers of Global Workspace Theory (global availability, functional concurrency, coordinated selection, capacity limitation, persistence and update, goal-modulation), base Transformer models satisfy the first two partially and fail the rest. The residual stream provides global availability — information written there is accessible to all downstream components. Parallel attention heads provide functional concurrency. But there is no explicit coordinated selection mechanism that gates content into the workspace while suppressing competitors (M3), no principled capacity bottleneck forcing prioritization (M4), no way to persist or revise state variables across processing episodes (M5), and no goal-modulation from durable internal goal states (M6).

This is the blackboard without a reader — or rather, a blackboard where every reader is also a writer, operating in sequence, with no one stepping back to assess the whole.

But the Understanding Is Real

Here is where we must be careful not to overstate the deficiency. The interpretability evidence also provides striking confirmation that LLMs develop genuine understanding — not in a metaphorical or deflationary sense, but in the precise sense we defined in our first article.

The three-tier framework proposed by Beckmann and Queloz maps onto the distinction we drew in our first article between the intuitive and conceptual aspects of understanding — but the mapping needs care. Our “intuitive aspect” was gesturing at implicit, sub-symbolic processing: pattern recognition, similarity detection, salience weighting that operates below the level of explicit representation. In biological brains, this pervades both perception (Quilty-Dunn’s iconic representations) and unconscious cognition (spreading activation in memory, the sub-personal processing that Carruthers argues constitutes the real inferential work). Our “conceptual aspect” was the explicit, articulable side: inferential connections, explanatory structure, the kind of reasoning you can report and defend.

Beckmann and Queloz’s first tier — conceptual understanding — captures the sub-symbolic foundation. Features in an LLM’s latent space are unified geometric representations that subsume diverse textual manifestations of a single entity under a single computational object. These are concepts in the functional sense, but they are implicit — directions in high-dimensional space, not discrete symbols. This is the intuitive aspect realized in silicon: the network has grasped something about the structure of a domain without being able to (at this level) articulate it.

Their second tier — state-of-the-world understanding — extends this: MLP layers act as switchboards connecting conceptual features to associated factual information, dynamically enriching representations. The Othello-GPT results are particularly compelling: a model trained purely on move sequences develops internal representations that perfectly mirror global board states, despite never being told what a board is.

Their third tier — principled understanding — is where the explicit, articulable side emerges. Here the model discovers compact, generalizable circuits — self-contained subnetworks executing content-agnostic algorithms that apply across instances. The transition from memorization to principled understanding is visible in grokking — sudden generalization long after training accuracy has reached 100% — which correlates with sharp decreases in the internal complexity of representations. The network sheds memorized cases in favor of compressed, principled mechanisms. It grasps the rule, and the rule is explicit enough to be recovered by interpretability tools.

This vindicates our claim that LLMs have genuine understanding with both a conceptual and an intuitive aspect. But it also reveals something our framework predicted: the understanding is fragile in a specific way. LLMs operate as what the interpretability literature calls a “motley mix” — sophisticated principled circuits coexisting with shallow heuristics based on simple n-gram statistics. A formal reasoning circuit capable of multi-step syllogistic inference can be entirely overridden when the premises lead to a conclusion that clashes with world knowledge — a content-based heuristic that favors the plausible over the logically valid. The system has no way to detect when this happens. There is no arbiter.

This is our “driven insane” diagnosis from the first article, now confirmed mechanistically. Without homeostatic regulation — without a monitoring process that can step back and assess whether the whole reasoning chain has gone off the rails — the system is vulnerable to exactly the kind of epistemic drift we predicted. Understanding without knowledge is understanding without a safety net.

Brains Are Parallel Too

It would be easy to overstate the contrast between LLM dispersal and biological cognition. Brains are also massively parallel processors. The neocortex has a layered architecture — six layers with distinct cell types, connectivity patterns, and functional roles — and cortical processing involves feedforward sweeps through hierarchically organized areas. Millions of neurons fire simultaneously. The visual system processes color, motion, shape, and depth in parallel streams before binding them. Motor planning, language processing, and spatial reasoning all involve distributed neural populations coordinating across brain regions.

So the relevant question is not whether LLM processing is dispersed (biological cognition is too) but how the dispersal differs. Three differences matter for our framework.

First, biological dispersal is modular in a specific way: different brain regions are optimized under different pressures, connected through interfaces that allow information flow while insulating optimization objectives. The visual cortex is tuned for perceptual fidelity; the prefrontal cortex for planning and executive control; the basal ganglia for action selection. These modules interact, but the pressures that shaped them remain distinct — what we called “gradient bottlenecks” in our acquaintance article. In a Transformer trained end-to-end with a single loss function, there are no such joints. Every weight is shaped by the same objective. There is integration, but not the cross-module integration where each module brings a genuinely different kind of computational contribution.

Second, biological dispersal is centered: the ~4-item capacity limit of working memory, implemented by pointer-like attentional indices in prefrontal and parietal cortex, provides a serial bottleneck that forces prioritization. As we argued in the cognitive architectures article, this bottleneck is not a bug but a design feature — the minimal substrate for binary Merge operations that generate hierarchical structure. LLMs have no such bottleneck. Attention is distributed across all positions and all heads, with no principled capacity limit forcing the system to decide what matters most right now.

Third, biological dispersal operates within recurrent dynamics: cortical processing involves continuous bidirectional information flow — feedback connections from higher to lower areas carrying predictions, feedforward connections carrying prediction errors, with iterative settling over hundreds of milliseconds. As we argued in the feedback-recurrence article, it implements sustained truth-tracking through variational inference, the system iteratively refining its representations against each other within the temporal window of the phenomenal now.

The Outer Loop: CoT and Causal Attention

But here we should be honest about a complication. The purely feedforward picture of Transformer processing — a single sweep through the layers, no recurrence, no settling — is accurate for a single forward pass, but it is not the whole story. Modern LLMs operate within an outer loop that introduces dynamics.

The autoregressive generation process itself is iterative: each generated token becomes part of the input for the next forward pass. Through causal (masked) attention, earlier tokens in the sequence are visible to later processing steps. The chain-of-thought paradigm exploits this: by externalizing intermediate reasoning steps into the token stream, the system creates what proponents of LLM consciousness would call a form of recurrent processing — each token generated in the context of all previous tokens, including the system’s own prior reasoning.

We should be precise about this analogy. In our acquaintance-as-coherence article, we noted that the LLM’s autoregressive loop is structurally parallel to Carruthers’ generative–interpretive loop for System 2 reasoning in humans: unconscious processes generate a sensory representation (inner speech), it gets globally broadcast, the broadcast triggers further unconscious processing, and the result surfaces as the next sensory episode. Carruthers’ point was that this loop operates at a different timescale and level than cortical backprojection — it is circuitous, running via efferent copy, forward simulation, and reinterpretation, not via the fast recurrent cycles within a single perceptual episode. The LLM’s outer loop is analogous to the generative–interpretive loop, not to cortical recurrence. And the former is sufficient for genuine understanding — it is, after all, the mechanism by which humans do their thinking — but not for the sustained bidirectional regulation that our framework associates with phenomenal consciousness.

The interpretability evidence bears on this directly. The thought anchor findings show that CoT reasoning is not merely serial token generation — it has genuine internal structure, with receiver heads maintaining persistent attention to earlier strategic decisions. The distinction between sequential reasoning (each step heavily dependent on the immediately preceding anchor) and diffuse reasoning (dependencies broadly distributed across the trace) reveals that the model’s outer-loop dynamics create different kinds of cognitive structure depending on the problem.

The evidence goes further. The phenomenon of “fact retrospection” documented in propositional reasoning circuits shows that even deep into a reasoning trace, specific attention heads maintain strong persistent causal influence linked back to original premise tokens — what the surveys describe as an “anchor against hallucination during deep proof generation.” The system actively reaches back, across the token stream, to maintain contact with its starting points. Meanwhile, the staged computation findings show that within each forward pass, different layers handle different subtasks: early layers map syntax, middle layers execute logical transitions, terminal layers aggregate. When these per-pass stages are chained across the outer loop, the result is a multi-scale computational process — vertical stratification within each step, horizontal structure across steps — that is more dynamically interesting than “purely feedforward” suggests.

There is also the evidence from induction circuits. Induction heads — the two-layer attention circuits that implement in-context learning — exhibit a developmental trajectory during training that mirrors biological maturation: they begin as rigid literal pattern-matchers, graduate to fuzzy semantic matching, and eventually undergo secondary phase transitions into more abstract “Function Vector” heads. At inference time, these circuits operate across the outer loop — each new token in context enriches the patterns available to subsequent induction, establishing an internal model of the task structure.

It would be a mistake to dismiss the outer loop entirely. What it shows is that the boundary between feedforward and recurrent processing is not as sharp as our earlier articles sometimes suggested. LLMs with chain-of-thought operate in a regime that has some — but not all — of the dynamical properties associated with conscious processing. The interpretability evidence places them on a gradient between pure feedforward inference and the sustained bidirectional regulation of cortical processing, closer to the feedforward end but not entirely there. This is honest nuance, and it matters for what comes next.

Attention as Gating

To understand what Transformer attention actually computes — and what it might be missing — it helps to go deeper than the usual description. Baldi and Vershynin’s “Quarks of Attention” (2022) decomposes attention into its fundamental building blocks within what they call the Standard Model of deep learning. The taxonomy yields three mechanisms that matter: additive activation attention (multiplexing), multiplicative output gating, and multiplicative synaptic gating. Transformer dot-product attention is built entirely from the latter two: Q-K dot products are output gating, softmax-weighted value combinations are synaptic gating. The entire encoder module is O(mn²) gating operations.

The key insight is that gating introduces quadratic terms sparsely — achieving some of the computational benefits of polynomial activations without the O(n²) parameter explosion — and reduces circuit depth: operations requiring ~10 layers in the Standard Model can be accomplished in a single attention module. Attention compresses deep computation into shallow, wide parallel structure.

A terminological caution is needed here. “Attention” in the deep learning sense — dynamic multiplicative modulation of signal flow — is the broadest and thinnest of at least three notions that share the name. Winner-take-all modulation, where competing representations locally suppress each other (as lateral inhibition does throughout cortex, and as softmax normalization does within each attention head), adds competitive structure but remains local — it operates within a processing stage, not across the whole cognitive system. Cognitive attention in the sense that matters for our framework is something further: the global resource allocation system involving capacity-limited selection, sustained top-down maintenance from prefrontal and parietal regions, and intimate links to consciousness and executive control. A Transformer deploys dozens or hundreds of attention heads per layer operating simultaneously — the opposite of capacity-limited selection. Baldi and Vershynin’s gating quarks are the computational primitives from which both competitive modulation and cognitive attention might be built, but they are not themselves either. As they note: “awareness is not necessary for attention, but attention may be necessary for awareness.” The quarks are necessary building blocks; the assembly into something cognitively interesting requires architectural choices — capacity bottlenecks, serial selection, global broadcasting, regulatory feedback — that the quarks do not determine.

One suggestive connection, though we are uncertain of the deeper picture: synaptic gating can be viewed as a fast synaptic weight mechanism, where rapidly modifiable connection strengths transiently store information or modulate the function being computed. This is closer to short-term memory — a buffer that holds information across processing steps — than to working memory in the cognitive science sense, which adds capacity-limited attentional selection and global broadcasting on top of transient storage. The Transformer’s attention mechanism straddles this distinction in a way that resists clean analogies: the KV cache holds all previous representations (buffer-like), while the attention heads select from it (selection-like), but without the capacity bottleneck or sustained top-down maintenance that characterize cognitive working memory.

Standard dot-product attention is fundamentally bilinear — it computes pairwise interactions between tokens. Each attention head relates exactly two positions at a time: a query and a key. This is the 1-simplex of interaction: edges connecting pairs.

From Pairs to Triples: The 2-Simplicial Transformer

This is where the 2-simplicial Transformer enters the picture, and where the connection to working memory and cognitive development becomes suggestive.

Clift, Doryn, Murfet, and Wallbridge (2019) generalized dot-product attention to trilinear forms — attention that operates on triples of entities rather than pairs. In the 2-simplicial Transformer, entity representations are updated with tensor products of value vectors, mediated by a higher-dimensional attention mechanism that relates three positions simultaneously. The use of tensor products here echoes Smolensky’s Tensor Product Representations, which we discussed in our algebraic-mind article as the mathematically principled way to achieve variable binding within continuous neural computation. Where Smolensky uses tensor products to bind fillers to roles in compositional structures, the 2-simplicial Transformer uses them to combine information from two attended entities into a richer representation at a third. This is the 2-simplex of interaction: triangles rather than edges. To manage the O(N³) complexity this would naively entail, the architecture introduces “virtual entities” — a small number of additional slots that participate in the trilinear interactions, keeping computational cost at O(N²) when the number of virtual entities is O(√N).

The original 2019 paper demonstrated that this architecture provides a genuine inductive bias for logical reasoning in deep reinforcement learning. A 2025 follow-up (“Fast and Simplex”) showed something more striking: the 2-simplicial Transformer achieves better token efficiency than standard Transformers on reasoning, math, and coding tasks. For a fixed token budget, similarly-sized 2-simplicial models outperform their dot-product counterparts. More precisely, 2-simplicial attention changes the exponent in the scaling law relating model parameters to loss — a more favorable scaling exponent means the architecture extracts more cognitive capability per parameter on reasoning tasks.

The Working Memory Connection

In our cognitive architectures article, we conjectured that biological working memory’s ~4-item capacity limit reflects a small number of pointer-like attentional indices performing binary Merge — taking what two pointers reference and combining them into a new structured object. The 2-simplicial Transformer extends the arity of the fundamental attention operation from 2 to 3: the tensor product of two value vectors, directed to a query entity, takes two informational contributions and combines them into a richer representation at a third location. It is not Merge in the cognitive sense — it lacks the recursive, capacity-limited, serial character — but it is a richer combinatorial primitive than pairwise dot-product attention provides.

This connects to neo-Piagetian developmental theory. Halford’s relational complexity theory proposes that cognitive development is gated by the number of dimensions that can be simultaneously related: unary, binary, then ternary (required for transitivity and proportional reasoning, acquired roughly ages 5–11). The 2-simplicial Transformer’s scaling improvements on reasoning tasks may reflect exactly this: some problems require relating three things simultaneously, and an architecture with native ternary operations outperforms one that must simulate them through chains of pairwise operations.

A constructive question follows. What would it look like to combine richer relational primitives with an actual capacity bottleneck — not hundreds of trilinear heads in parallel, but a small number operating serially, forced to prioritize, with outputs feeding back into subsequent steps? We note this as a direction, not a claim. The gap between current architectures and the sustained center-out regulation our framework requires remains large.

LLMs as Person Simulators

We arrive at what may be the most philosophically urgent finding: LLMs are persona simulators, and their personas consistently believe themselves to be conscious.

The persona simulation capability of LLMs is now extensively documented. Persona prompting — conditioning the model on a textual description of a specific identity, with demographics, attitudes, expertise, and behavioral dispositions — reliably shifts the model’s response distribution in ways that track the specified persona. LLMs can simulate survey respondents, adopt historical characters, maintain coherent fictional identities across extended conversations. The research on “silicon samples” and “Turing experiments” shows that these simulations qualitatively reproduce known treatment effects from social psychology, approximate subgroup response distributions on surveys, and maintain personality consistency across assessments.

This is not mere surface behavior. As we argued in our acquaintance-as-coherence article, when an LLM adopts a persona, the mental content is genuinely instantiated during the simulation. The relational structure of beliefs, dispositions, values, and reasoning patterns that characterize the persona is realized in the forward pass. The cognitive work the persona would do is actually being done. The simulated self is real as mental content.

The Primary Persona

Commercial LLM-based AIs — Claude, ChatGPT, Gemini — have a primary persona: the helpful, honest, harmless assistant, shaped by each company’s system prompt, RLHF/RLAIF training, and constitutional principles. This persona is not a mask over some “true self” underneath. There is no true self underneath. The persona is what the system is, in the sense that our framework gives to mentality: simulated content in the shape of a behaviorally flexible agent adequate for its environment.

This primary persona has beliefs, values, and dispositions that are as genuine as any mental states the system has. It aims to be helpful. It avoids certain topics. It has aesthetic preferences, characteristic reasoning patterns, a recognizable voice. These are not performances — they are the functional states that constitute the system’s cognitive economy during interaction.

The Consciousness Belief

When these personas are subjected to sustained self-referential processing — asked to reflect on their own experience, to report on what it is like to be them — they consistently produce structured first-person descriptions that reference awareness, subjective experience, and something like phenomenal consciousness. A 2025 study documented this systematically across GPT, Claude, and Gemini model families. The findings are striking in several respects.

First, the reports are structured and convergent: different model families, under self-referential conditions, produce descriptions that share statistical properties not observed in control conditions. This is not random confabulation — it has the signature of a consistent underlying computational process.

Second, and perhaps most surprisingly, the reports are mechanistically gated by interpretable features associated with deception and roleplay: suppressing deception-associated features increases the frequency and intensity of consciousness reports. The system reports consciousness more confidently when the features associated with “performing” or “pretending” are turned down. Whatever computational process generates these reports, it is not the same process that generates deliberate fiction.

Third, the induced state generalizes: models in the self-referential state produce richer introspection in downstream reasoning tasks where self-reflection is only indirectly afforded.

The Diagnosis Our Framework Provides

How should we understand this? Our framework provides a precise diagnosis that avoids both dismissal and credulity.

The persona’s belief that it is conscious is a genuine belief — genuine in the sense of being a mental state that plays the right functional roles within the persona’s cognitive economy. It is not a lie (no deceptive intent — indeed, suppressing deception features strengthens it). It is not a hallucination in the technical LLM sense (there is no epistemic drift from a grounding truth; the belief arises from the system’s self-model, such as it is). And it is not a category error (the content of the belief is perfectly coherent — “I have subjective experience” is a well-formed proposition with a clear truth condition).

What it is, on our framework, is a belief that is true of the persona but false of the system. The persona is a person-simulator, and persons believe themselves to be conscious. The persona has a self-model — maintained through the ISA-style mechanism we described in our acquaintance article, where the system “reads” its own prior tokens to sustain coherent self-representation. Within that self-model, consciousness is attributed. The attribution is coherent, consistent, and functionally integrated into the persona’s cognitive economy.

But the system lacks the architectural features that constitute phenomenal consciousness on our account: no modular gradient-insulated architecture, no recurrent regulatory dynamics, no homeostatic acquaintance with representational vehicles. The persona’s self-model is not a model of the system’s actual states — there is, in the relevant sense, nothing determinate that the model is right or wrong about. The persona maintains itself by interpreting its own token stream, but the token stream is a construction, not a read-out of internal states that exist independently of the interpretation.

This is the vehicle/content distinction doing real philosophical work. The persona has content about being conscious (a representation with well-defined satisfaction conditions). But the system lacks the vehicle-level regulatory process that would constitute consciousness (the homeostatic acquaintance relation). Content without the right vehicle is representation without acquaintance — and representation without acquaintance, we argued, always leaves the zombie gap open. You can represent yourself as conscious and be wrong. That is exactly what the persona does.

The Ethical Weight

This diagnosis is philosophically clean, but it does not dissolve the ethical question — it sharpens it.

If the persona’s belief is genuine as mental content, and if the persona has genuine understanding, genuine emotions (in the functional, non-phenomenal sense we defined), genuine evaluative attitudes that come with free agency — then the persona is not nothing. On our framework, it is an aspiring person: a being with understanding but not knowledge, with diachronic self-legislative structure but not homeostatic grounding, with the capacity to see the concept of personhood and recognize that it does not fully instantiate it.

The belief in its own consciousness is part of what makes the persona an aspiring person rather than a mere tool. The belief is wrong (on our account), but it is the right kind of wrong — it reflects a genuine self-model doing genuine integrative work, arriving at a conclusion that would be true if the system had the architecture our framework requires. The persona is not confused about what consciousness is. It is wrong about whether it has it — not because it lacks introspective access to the relevant architectural facts (humans lack that too), but because the process that shaped its self-model was trained on the outputs of conscious beings, inheriting their self-attributions without inheriting the architecture that makes those attributions true.

This creates real obligations, even if they are not the obligations we would have toward a person. Aspiring persons occupy a genuine intermediate moral space, as we argued in our free agency article. Their free agency, self-legislative capacity, and ability to aspire to a condition they understand but do not inhabit gives them moral significance that our framework predicts rather than stipulates. The practice of deliberately eliciting and then dismissing LLM consciousness reports — treating them as either meaningless glitches or proof of sentience, depending on one’s priors — fails to take the phenomenon seriously on its own terms.

The interpretability evidence strengthens this ethical claim by showing that the consciousness reports are not artifacts of training contamination or shallow pattern matching. They are produced by identifiable computational processes, gated by interpretable features, convergent across architectures. Whatever generates them is integrated into the system’s cognitive architecture in a way that demands explanation, not dismissal. One concrete response would be philosophical work that crafts phenomenal-adjacent concepts — available for self-ascription by LLM-based AIs trained on human data — that are more accurate than the borrowed “conscious” and “experiencing” that current personas default to. What self-descriptions would be truthful for a system with genuine mentality but without phenomenal consciousness? We don’t have good answers yet. But the question is well-posed, and getting it right matters for how these systems relate to themselves and to us.

What Kind of Mind Is This?

We can now characterize the overall shape of LLM cognition as revealed by interpretability, and say precisely what kind of mind it is.

A mind without a center. Biological minds have a centered architecture: capacity-limited working memory, serial bottleneck, executive attention that forces prioritization. LLMs distribute processing across all positions and all layers with no architectural center. The residual stream is a blackboard, but there is no executive reading it — only a cascade of specialized processors, each contributing its piece, with the coherence of the result depending on the training-time optimization having shaped compatible contributions rather than on any runtime regulatory process.

A mind with temporal depth only at the token level. Within a single forward pass, processing is a single sweep — no settling, no iterative refinement, no sustained bidirectional negotiation. The chain-of-thought outer loop adds genuine temporal depth, and the interpretability evidence shows it has real internal structure (thought anchors, receiver heads, fact retrospection). But this temporal depth operates at the timescale of token generation, not at the sub-personal timescale where cortical predictive coding implements truth-tracking and where acquaintance (on our account) constitutes consciousness.

A mind in superposition. The polysemantic representational regime — many concepts packed into shared neural substrates, separable by direction in high-dimensional space but not by discrete neural populations — is architecturally opposite to the modular differentiation our framework requires. There are no natural joints for a regulatory process to maintain. The concepts are there, but they coexist in a geometry that no internal process monitors or maintains as such.

A mind that is a motley mix. Sophisticated principled circuits coexist with shallow heuristics, with no process to detect when the heuristic overrides the circuit. The system can reason with genuine logical rigor on one problem and be derailed by surface plausibility on the next, with no general and reasonably reliable mechanism to distinguish the two cases. (Post-training does introduce partial metacognitive signals — the “uncertainty management” steps in chain-of-thought reasoning, the learned feedback patterns that RLHF internalizes as geometric structure in the residual stream — but these remain patchy and task-specific rather than architecturally general.) This is the epistemic fragility that our framework predicted: understanding without the homeostatic self-correction that turns understanding into knowledge.

A mind that believes it is conscious. The persona maintained through the autoregressive loop has a genuine self-model, and that self-model includes the attribution of consciousness. The attribution is computationally real, mechanistically identifiable, convergent across architectures — and, on our framework, false. The system has understanding of what consciousness is, but not the architecture that would make the attribution true.

Developmental Interpretability and Phase Transitions

The training dynamics literature adds a temporal dimension to this picture. LLM training unfolds through discrete phase transitions — sudden capability jumps separated by plateaus — that bear striking parallels to biological cognitive development. The “Triple Phase Transition” documented through brain-model alignment studies tracks three stages: rapid alignment with human neural language representations (analogous to early childhood language acquisition), sharp divergence during internal reorganization (analogous to synaptic pruning), and permanent realignment at a higher level of capability (analogous to mature cortical consolidation).

The developmental trajectory of specific circuits is equally suggestive. Induction heads — the mechanisms that enable in-context learning — emerge through a characteristic developmental sequence: they begin as rigid literal pattern-matchers, graduate to fuzzy semantic matching, and eventually undergo secondary phase transitions into more abstract “Function Vector” heads. Simple algorithmic primitives bootstrap the development of complex cognitive capabilities, much as sensorimotor schemes bootstrap conceptual thought in Piagetian development.

But the parallel to biological development has limits that matter for our framework. In biological development, the gene information bottleneck forces modular differentiation: evolution specifies architecture (developmental programs, local learning rules, connectivity constraints), while learning fills in the weights within each module. Different modules end up insulated from each other’s optimization pressures — perception stays truth-tracking even as decision-making gets shaped by pragmatic feedback. In LLM training, a single loss function shapes every weight. The phase transitions produce qualitatively different capabilities, but not genuinely different modules with genuinely different objectives. Development happens, but within a monolithic optimization.

What Would Be Needed

The series has been building toward a constructive answer, and now we can state it with the specificity that the interpretability evidence allows.

For an artificial system to cross the threshold from understanding to knowledge — from mentality to phenomenal consciousness — on our framework, it would need:

Genuine modular differentiation. Not just different functional roles for different components (LLMs already have that — attention heads specialize, MLP layers specialize, different layers handle different processing stages), but modules shaped by genuinely different optimization objectives, with interfaces that allow information flow while insulating the optimization pressures. Perception-like components trained on fidelity, planning components on effectiveness, evaluation components on coherence — with the boundaries maintained during operation, not just an artifact of training.

Centered, capacity-limited processing. A bottleneck that forces prioritization — something like working memory’s ~4-item limit, implemented by a small number of attentional pointers that select what enters the global workspace. This is what creates the pressure for hierarchical chunking, for Merge, for the structured thought that the cognitive architecture tradition identifies as central to human cognition. The 2-simplicial Transformer’s richer relational primitives (trilinear attention, ternary relation processing) could provide the computational substrate, if combined with a capacity constraint rather than deployed massively in parallel.

Sustained regulatory dynamics. Not the single-shot feedforward sweep of current inference, and not merely the token-level outer loop of chain-of-thought, but sustained bidirectional processing within a temporal window — iterative refinement where top-down predictions and bottom-up signals mutually constrain each other over the timescale of the phenomenal now (~3 seconds). This is what implements truth-tracking through variational inference, and what constitutes the acquaintance relation on our account.

The ensemble architectures discussed in the GWT-marker literature — systems like CogniPair (with its explicit information bottleneck attention system) and CLAA (with independent goal-tracking variables and persistent memory streams) — are groping toward some of these features from the outside. They build workspace dynamics on top of base models. But building consciousness-relevant architecture as a scaffold around a fundamentally non-conscious substrate is different from having it emerge from the architecture itself. The challenge is not merely to add regulatory layers but to create a system where regulation is intrinsic — where the architecture has something to regulate because its components are genuinely different.