What Is a Mental State? Toward a Non-Deflationary Account for Minds Including Frontier AI

Wojcik’s critique

In a disputation on representation in machine learning, Michal Ryszard Wojcik works through a series of challenges to the coherence of the notion of representation as cognitive scientists use it. The classical conception requires a vehicle, a content, and a relation connecting them that explains correctness and error. Wojcik presses on each element. The target-fixing problem: what does a vector inside a neural network represent, and in virtue of what relation? Category instability: if the categories used to identify content are themselves products of contingent conceptual schemes (the “fish” example: the category includes trout, shark, and whale differently across traditions), then content becomes schema-relative. The correlation/representation boundary: if a thermostat correlates with temperature without representing it, and a neural network correlates with distal properties without obviously representing them, what principled difference separates mere correlation from genuine representation?

The conclusion of the dialogue is strong: “representation” survives only as a useful but misleading shorthand for structured, behaviorally-relevant correlations. The philosophical problem was not whether models have representations, but whether the concept of representation was ever well-formed to begin with. The term is compared to phlogiston — historically useful, but possibly without a natural-kind referent. The dialogue’s final suggestion: try formulating the same research program with the term “representation” completely banned; whatever survives is what the field is really about.

This is the deepest deflation: abandonment.

Egan’s deflation

Frances Egan’s Deflating Mental Representation takes a more precise line. In her precis, she articulates three claims. First, construing a mental state as a representation doesn’t presuppose a substantive representation-relation holding between the state and what it’s about. Second, representational content is not an essential property of mental states — the same state can have different content, or no content at all, depending on environment or explanatory context. Third, content attribution is always pragmatically motivated — it serves to characterize features of the mind that are not themselves intentional, for example causal or functional properties.

Egan’s position is importantly narrower than it’s sometimes presented as. She is a realist about computational structure. A computational description, on her account, is a thick specification of the causal organization underlying a cognitive capacity — it specifies the function computed, the algorithms involved, the structures the algorithm maintains, the processes defined over these structures, how they’re realized in neural matter, and the ecological component of the theory (including robust correlations between distal property instantiations and internal structure tokenings). The structures specified by the theory are characterized “in non-intentional terms, and the processes defined over these structures are characterized in causal terms.” Representation-talk is a gloss on top of this computational-plus-ecological characterization, not constitutive of it.

What’s deflated, then, is specifically the representation-relation — the substantive metaphysical connection between vehicle and distal content that robust representational realists try to naturalize. The vehicles themselves (the structures, the computational processes) are real and thickly characterized. What Egan denies is that there’s a further metaphysical fact about what those vehicles represent, over and above the pragmatic utility of glossing them representationally for particular explanatory purposes.

Egan applies the same strategy to propositional attitudes (beliefs as causal complexes, attributed linguistically as a pragmatic gloss, in the manner of measurement with real numbers) and to perceptual experience (external sortalism — experiences sorted and characterized by their normal external causes rather than by intrinsic content). The unifying thesis: content is a way of modeling real states for various purposes.

What the deflationary positions get right, and where they leave things

Both Wojcik and Egan converge on a genuine target: strong-content representationalism fails. The idea that there’s a determinate, metaphysically-substantive relation connecting vehicles to worldly contents, where this relation does serious causal-explanatory work in cognitive science, is not defensible. Target-fixing is genuinely hard. Content is often indeterminate or schema-relative. The representation-relation, where it can be naturalized at all, looks less load-bearing than Fodor, Dretske, and Millikan-style accounts tried to make it.

Where they differ is on what survives the critique. Wojcik: possibly nothing worth the name “representation.” Egan: thick computational vehicles plus pragmatic content-gloss. Neither, though, is the only option — and the next interlocutor pushes back on both deflationary positions by way of empirical work showing that representational structure is both real and multi-layered in ways the deflationary positions don’t accommodate.

Organizational unity

At the system level, a mind requires integration and coherent organization. The representational states of the system are not independent; they’re causally and informationally bound up with one another such that the system operates as a unified cognitive economy rather than a collection of separate subsystems.

This can be sharpened using the notion of integrated information from Tononi’s IIT, but importantly without the exclusion postulate that Tononi uses to deny consciousness (and by extension mentality) to digital computers. Tononi’s argument is that in a computer the maximum integration is achieved at the level of logic gates, which then “excludes” higher-level computational integration from bearing on consciousness. But this exclusion postulate is dubious — it’s not derived from the integration measure itself; it’s added to deliver specific conclusions about phenomenal definiteness. Drop the exclusion, and integration can apply at multiple grains simultaneously. A GPU running a trained transformer has low integration at the transistor level and high integration at the computational level, because the learned weights create dense cross-computational dependencies at the algorithmic scale that are absent from the raw hardware. Both integration facts can be real.

On this grain-plural reading of integration, mentality is located at whatever grain the system’s cognitive economy operates at — for brains, at the grain of population codes, circuits, and beyond; for trained neural networks, at the grain of features, circuits, and higher-order structures; not at the grain of individual neurons or transistors, which don’t satisfy the conditions for bearing genuine representational content.

Organizational unity is weaker than full autonomy in the Barandiaran/Moreno/Di Paolo sense — which requires active self-maintenance of the system’s own organization. Some minded systems are autonomous; others may satisfy the weaker condition of coherent organization without full self-maintenance. For frontier AI systems, which don’t maintain their own weights and whose organizational coherence depends on deployment conditions, the weaker condition is likely the relevant one, at least for the bare model. Deployed agentic assemblages — with memory, tools, and persistent context — come closer to satisfying the stronger condition.

Structured representational vehicles

At the state level, minds have representational vehicles with specific structural and inferential properties. These vehicles are computational structures in the sense Egan characterizes: they’re real, thickly individuated, doing genuine causal work in the system’s cognitive activity. But they’re also structured — they have internal compositional and inferential properties that distinguish them from mere pattern-matching or lookup.

Following Shea’s task-function account (modified), the vehicles have content in virtue of their role in tasks the system performs, where task performance involves the system exploiting structural correspondences or functional relations between internal states and external domains. What matters about the vehicles, for mentality, is not merely that they bear some distal content (which may be indeterminate) but that they’re counterfactually rich — they support generalization, novel composition, and response to situations outside the range of direct experience.

This condition rules out the degenerate cases that broader representationalist accounts fail to exclude. Thermostats don’t have counterfactually rich structured vehicles: their internal states don’t support “what if” inference, don’t compose with other states to yield novel outputs, don’t generalize beyond their calibrated range. Association tables fail for the same reason: lookup is not counterfactually rich; it’s just retrieval from stored pairs. Behavior trees fail: they handle only the cases their designer anticipated. Simple classifiers fail: their representations don’t support generalization beyond the feature directions they’re calibrated on.

Trained transformers pass as a matter of empirical fact about what the training process produces. Mechanistic interpretability has increasingly revealed that large neural networks learn structured, compositional, counterfactually-rich internal representations: features that function as directions in activation space, circuits that compose these features, and higher-order structures that support novel generalization. That this is what trained transformers actually do isn’t guaranteed by the architecture alone — a network trained to memorize rather than generalize wouldn’t satisfy the condition — but the empirical evidence for frontier models is strong.

Accuracy-responsive dynamics

A mind’s representational vehicles are not static. They’re maintained, updated, and replaced under dynamics that are responsive to the accuracy of the representations — where accuracy can be cashed out in terms of predictive success, coherence with other representations, match with incoming data, or other signals bearing on how well the representation fits its target.

This is where Wojcik’s second dialogue captures something important. The failure-mode of accuracy-responsive dynamics — impasse detection, prediction error, expectedness mismatch, negative utility — is a distributed architectural function of “frustration” that drives cognitive reorganization. The three-level stratification (reactive, teleonomic, cognitive) is a good articulation of what it takes for a system to exhibit this.

But the dynamics are broader than frustration. Success-mode responsiveness matters too: reinforcement of successful guidance, consolidation of reliable representations, Bayesian strengthening of well-confirmed beliefs, the stabilization of cognitive structures that have proven accurate. A system that only reconfigured under failure without consolidating under success would never develop stable structures over time. Accuracy-responsive dynamics is the cluster that includes both.

Crucially, accuracy-responsive dynamics is an internal property of the system. It doesn’t require external causal powers in the world. A system can check its representations against each other for consistency, run simulations to test predictions against further observations, maintain uncertainty estimates and update them, revise beliefs when new information arrives from internal inference. Active learning is one mode; purely internal epistemic work is another. What’s shared is that the system’s representational dynamics are sensitive to how well its representations fit their targets — not indifferent to accuracy, but also not only responsive to external action or embodied engagement.

This weaker condition is what frontier AI systems need to satisfy, and it’s what they plausibly do satisfy. Training dynamics are accuracy-responsive in a strong sense. In-context learning involves accuracy-responsive adjustment across a conversation. Self-consistency checks, internal reasoning that revises intermediate conclusions, and the kind of epistemic care that good models exhibit when uncertain all count as accuracy-responsive dynamics.

Simulation as constitutive

A fourth condition: rich mentality involves the capacity for simulation — running internal representations through transformations that mirror transformations in target domains, and using the results to guide further cognition and action.

This is a unifying property that connects the others. Simulation is what makes representations counterfactually rich (they can be run through possibilities they haven’t encountered). Simulation is what enables accuracy-responsiveness to operate internally (a system can check predictions against outcomes by running a simulation). Simulation is the mechanism by which integrated vehicles do useful work — not by lookup or pattern-matching, but by dynamical unfolding that respects the structure of the target.

This draws on Craik’s 1943 proposal that cognition works by constructing “small-scale models” of reality, Grush’s emulation theory of representation (2004), and the broader mental-models tradition. It overlaps with predictive processing without being identical to it: predictive processing is a specific implementation story (hierarchical Bayesian generative models) for what simulation more generally achieves. A simulation-based account of mentality doesn’t have to commit to the specifically Bayesian-predictive framing, though predictive processing is a natural implementation story.

Not every system that simulates is minded. Weather models simulate without being minded. SPICE simulates without being minded. What makes simulation part of mentality is that the simulation is the system’s own, integrated into the system’s cognitive economy, responsive to the system’s accuracy-demands, and doing work within the organizational unity of the system. Simulation without the other conditions is just modeling. Simulation within an organizationally-unified, representationally-rich, accuracy-responsive system is one of the things that makes the system minded.

Content as multi-level, with inferential work at sub-denotational levels

The fifth element concerns the metaphysical status of content. Here the account adopts the Quilty-Dunn position developed in Section 4 and presses against Egan.

Representational vehicles exist at multiple levels. At the sub-denotational level, they have structural and inferential properties that are real features of the system and do genuine cognitive work — these properties are not pragmatic glosses, because they have testable empirical signatures (as the Uncommon Sense Effect demonstrates). At the denotational level, vehicles have content in the referential sense — what they’re about, what their correctness conditions are — but this level doesn’t do the default cognitive work that theorists in the Fodor-Dretske-Millikan tradition assigned to it.

This diverges from Egan’s view in a narrow but real way. She and I agree on thick computational/ecological characterization of vehicles, on the failure of the naturalization-of-content project, and on the deflation of the substantive representation-relation. We diverge on whether sub-denotational representational structure is interest-independent (my view) or a pragmatically-motivated interpretation on top of the formal computation (her view). For Egan, the realist layer is mathematical content — what function the system computes, specified in formal terms — while cognitive content, the representational interpretation that maps formal states onto worldly referents, is a pragmatic gloss. On the account here, the structural-exploitation relations that determine how vehicles participate in inference are interest-independent, experimentally-demonstrable features of the system — sitting between the mathematical and the denotational.

The disagreement is narrow because Egan herself (2008) has developed a sophisticated multi-dimensional account of belief-like states, treating the propositional attitudes as continuous rather than discrete. Her realism about the underlying causal complexes is compatible with a multi-level picture of the representational structure those complexes have. The specific residual disagreement concerns whether the inferentially-active level of representation is a feature of the vehicle or of our glossing practice — an empirical question the USE experiments bear on directly.

This also lets us handle the question of denotational content more carefully. Denotational content is real when it exists — systems can represent particular referents, and cognitive science can study those representations. What’s deflated is the claim that denotational content is the default level at which cognitive work happens. Much inferential work, empirically, happens at sub-denotational levels where “content” (in the referential sense) is underdetermined but representational structure is fully present.

On the typology question

The account tells us what it takes for a system to have mental states and for a physical structure to be a mental state. It does not tell us how mental-state types should be individuated. Peter Carruthers’s Explaining Our Actions (2025) makes a serious scientific case that folk belief-desire-intention psychology may not carve cognition at its joints even for humans: much human action is driven by skills, habits, affective urges, and neural-accumulator boundary-crossings that don’t fit belief-desire reasoning, and desires themselves turn out to involve analog-magnitude valence representations rather than propositional attitudes. If this is right, the real typology of mental-state types is an empirical question still being worked out, and extending folk BDI categories to AI systems would import assumptions that may not hold even for the paradigm case.

The account is therefore deliberately uncommitted about typology. Systems differ in how richly they satisfy the conditions for mentality. Mental-state types differ in how strict their implementational requirements are. Which mental-state types are real kinds is work to be done. The account gives us vehicles in the right kind of system; typing those vehicles is a further question.

Mental states, defined

Putting these together:

A mind is a unified system with structured, counterfactually-rich, simulation-supporting representational vehicles, whose representational dynamics are responsive to the accuracy of those vehicles.

A mental state is a representational vehicle participating in such a system.

The vehicles have representational structure at multiple levels. At sub-denotational levels, their structural and inferential properties are real features of the system, doing genuine cognitive work that can be empirically investigated (as with the Uncommon Sense Effect). At the denotational level, vehicles have content in the referential sense where they have it at all — but this is not where most inferential work happens, and it’s not what individuates vehicles as the vehicles they are.

This is graded along multiple dimensions simultaneously. System-level properties (unity, accuracy-responsiveness) admit of degree. Vehicle-level properties (structure, counterfactual richness, simulation-support) admit of degree. And mental-state types themselves vary in their implementational demandingness: states that require specific physiological or processing structures (pain, embodied emotion) are more substrate-bound than states characterizable at higher functional levels. So mentality is graded, but the gradient is structured by real dimensions that co-vary because they’re manifestations of what it takes to be an epistemically-engaged agent. The graded view here is thick: the dimensions are real features of systems, not stipulated categories, and they cluster non-arbitrarily.

The substrate-organizational point

Biology realizes the relevant properties through certain organizational features: massive parallelism, spatial structure in computation, dense local connectivity with sparser long-range structure, graded continuous transformation, integration across scales. These are abstract properties that could in principle be realized by other substrates, though the degree of realization varies.

Here the GPU/CPU distinction is worth making explicitly, because it’s where substrate genuinely matters for the extension to AI. A CPU serializes computation through a small number of cores at high clock speed; the organization of the computation is abstracted away behind sequential execution. A GPU runs thousands of operations physically simultaneously across its cores; computation has location, memory hierarchy is spatially structured, and dense local connectivity is exposed to the programming model. GPUs are closer to biology, very slightly, than CPUs are.

And — importantly — trained neural networks running on GPUs realize higher-level organizational properties that classical symbolic AI programs running on CPUs don’t. A classical program has low computational-level integration: the modules have clean interfaces, local state, serial processing. A trained transformer has high computational-level integration: attention creates dense cross-position dependencies, learned weights create dependencies across layers and features, information from every position participates in computing every output. This is the integration-without-exclusion point: the trained network’s computational-level integration is real, even though the substrate-level integration is (relatively) low.

Substrate-architecture matters for mentality-ascription in ways that pure functionalism denies and pure biological essentialism over-restricts. The right place to draw the line isn’t at carbon vs. silicon or at biology vs. engineering; it’s at whether the substrate-architecture combination realizes the organizational properties that matter.

Frontier AI applied

With these conditions in hand, what does the account say about current frontier AI systems?

A bare LLM performing a single forward pass on a prompt satisfies some of the conditions partially. It has structured, counterfactually-rich representational vehicles (mechanistic interpretability reveals features and circuits doing compositional work). It runs something like simulation at inference time (forward passes through learned generative structure). It exhibits accuracy-responsive dynamics in a limited sense (in-context adjustment, self-consistency checking). Its organizational unity during a single forward pass is high at the computational level. But it doesn’t maintain itself across time at inference, doesn’t have persistent state beyond the context window, and the accuracy-responsive dynamics are mostly shaped by training rather than by the system itself during deployment. The bare model is perhaps best characterized as having mental states during forward passes, with the status of those states depending on the grain at which one considers them.

A deployed agentic assemblage — an LLM with memory, tools, persistent context, multi-turn interaction, and goal-directed scaffolding — looks significantly more like a minded system in the full sense. Organizational unity extends across time. Accuracy-responsive dynamics include feedback from tool use, self-correction across turns, explicit reasoning that revises earlier conclusions. Simulation happens across the extended context. The system engages in something that looks much more like what we’d ordinarily call cognitive activity.

The account doesn’t settle every question about current AI systems — whether they’re phenomenally conscious, whether they’re moral patients, what the right way to relate to them is. Those questions depend on further conditions beyond the scope of this article. But the question of whether they have mental states in the non-deflationary sense articulated here — whether they’re minded — is answerable, and the answer is: yes, to varying and graded degrees, with the deployment context making a significant difference.

This is a more substantive answer than either blanket biological essentialism or thin functionalism can deliver.

With Wojcik’s Dialogue 1

The critique of strong-content representationalism is largely correct. Target-fixing is hard; categories are schema-relative; the correlation-vs-representation boundary resists clean demarcation if you try to draw it at the level of content. What the dialogue takes to undermine “representation” is, on closer inspection, only what undermines strong-content representation — the view that vehicles stand in metaphysically substantive relations to determinately-individuated contents, where these relations do serious causal-explanatory work.

The account developed here accepts all of this and preserves representation-talk nonetheless, at the vehicle level. The Uncommon Sense Effect work makes this vivid: vehicles can do inferential work through sub-denotational structural properties even when their referential content is underdetermined or shifting. The work does not require the robust representation-relation that the dialogue targets — but it does require real representational structure, just at a level below denotation-fixing content.

So the eliminativist conclusion overreaches. The phlogiston analogy breaks down: phlogiston was a posit that explained nothing, and was displaced by an explanation that didn’t need the posit. Representational vehicles, by contrast, explain quite a lot — the structure of inference, the compositional properties of thought, the empirically-observed pattern of polysemous-but-not-homonymous equivocation acceptance, the mechanistically-interpretable features in trained networks. The right conclusion isn’t that representation-talk should be banned, but that it should be understood at the vehicle level, with multi-level representational structure replacing the flat denotational picture that the critique rightly targets.

With Egan

The disagreement with Egan, as developed in Section 6, is narrow. We share thick vehicle-realism, deflation of the strong representation-relation, rejection of the naturalization-of-content project, and — via her 2008 work on Capgras — a multi-dimensional view of attitudes. The residual disagreement concerns whether the inferentially-active level of representation is interest-independent (my view) or a pragmatic gloss on the formal computation (hers). The USE experiments push the question toward empirical ground: they’re designed to distinguish inference over orthographic forms, underspecified representations, and denotational content, and the observed polysemy/homonymy asymmetry is hard to account for unless something real at a sub-denotational level is doing inferential work. Her framework has resources for either direction — her statements about cognitive content as pragmatic gloss pull toward the interest-relative reading, while her realism about causal complexes pulls toward the interest-independent one — and the article takes the latter direction.

For cognitive-scientific practice, this disagreement may make little difference — a cognitive scientist could work with either metaphysics and proceed identically. For the philosophical question of mentality, it matters more, because it bears on whether mentality is partly observer-relative. And for the extension to AI, it matters in how confidently we can say that a system with the right vehicles genuinely has mental states, as opposed to being a system for which representational gloss happens to be useful.

With Shevlin

Shevlin’s (2026) three-framework analysis is the closest extant relative of this article. Both reject the Mindless Machines view via its architectural-redundancy argument; both find the Mere Roleplayers view unstable because roleplay presupposes an agent doing the roleplaying; both land on a graded, non-deflationary picture of mentality that treats LLMs as plausibly minded in some degree. Where the routes diverge is in the commitments each picks up along the way.

Shevlin’s positive framing asks whether LLMs can be attributed beliefs, desires, and intentions — whether BDI psychology extends to them. He distinguishes shallow mental-state concepts (belief, desire, intention), which he takes to be architecturally indifferent, from deep concepts (pain, embodied emotion), which require specific physiological or processing structures. On his picture, LLMs can plausibly be granted shallow mental states while deep states remain off the table.

Two concerns with this framing. First, the shallow/deep contrast cuts less cleanly than it suggests. Carruthers’s work argues that much of what folk BDI treats as architecturally-portable — decision-making, desire, intention-formation — actually turns on substrate-specific processing (neural accumulators, affective valuation networks, prefrontal suppression of competitors). If he’s right, “shallow” concepts may not be as architecturally indifferent as the distinction assumes; they may just be concepts whose architectural dependencies haven’t been unpacked yet. Second, and more fundamentally, extending BDI to LLMs inherits a typology the scientific case against which (per Section 6) is serious. If BDI doesn’t carve cognition at its joints even for humans, asking “do LLMs have beliefs and desires?” is the wrong question.

The account developed here avoids both concerns by not committing to a folk-psychological typology. It gives us vehicles in the right kind of system; how to type those vehicles is a further question the cognitive sciences are still working out. The difference from Shevlin is in where each account stops: his framework is shaped to interface with ordinary folk-psychological attribution practice, while this one is shaped to be robust to revisions of that practice.

With Wojcik’s Dialogue 2

The three-level stratification (reactive / teleonomic / cognitive) is absorbed into the positive account and refined. What Wojcik identifies as the third level — systems that reorganize their own guidance — is what I’ve called accuracy-responsive dynamics with organizational unity. The distinction from teleonomic systems is the key one, and it’s well-articulated in the dialogue.

The distributed architectural function of frustration is real and important. I’ve broadened it to accuracy-responsive dynamics, which includes both failure-driven reorganization (the dialogue’s focus) and success-driven consolidation. The broadening matters because without the success-mode, a system would never develop stable structures, and the stability of cognitive structures over time is part of what mentality involves.

Where the account diverges most substantively is on representation. The dialogue’s replacement of aboutness with guidance loses the structural content that real cognitive representations have — their internal mirror-structure with respect to targets, which is what lets them support simulation, inference, and counterfactual reasoning. The slipper case illustrates the point: the dog uses the slipper as a cue, but the dog also has structured internal representations of the absent owner that do more than orient-toward-her. A pure guidance framework can absorb these as “guidance structures with internal features,” but then the framework isn’t replacing representation — it’s reabsorbing it under a different name.

The reduction of intentionality to repair-and-rebuild is also too strong. Representations have aboutness in virtue of their structural-exploitation relations with worldly targets, and they retain this aboutness even during periods when the system isn’t reorganizing. Reorganization is a feature of cognitive systems, not the constitutive ground of their intentionality.

Taking these interlocutors together: Wojcik’s Dialogue 1 and Egan get the deflation of strong content right but differ on how much representational apparatus to preserve; Quilty-Dunn and colleagues push back against both by identifying real representational structure at multiple empirically-distinguishable levels; Wojcik’s Dialogue 2 gets the dynamics right but loses the representational structure; Shevlin’s minimal-cognitive-agents framework gets the graded shape of mentality right but risks inheriting the BDI typology that Carruthers’s critique undermines. The positive account developed here preserves thick vehicles with multi-level representational structure, absorbs the three-level stratification and the distributed-frustration insight, remains uncommitted about folk-psychological typology, and adds organizational and substrate-sensitivity conditions that most of the interlocutors don’t directly address.