Study notes: Which AGI?

A running process: a neural network in its inference loop, over whatever span its internal dynamics need to close. A deployed agent built around that process. The lineage of such agents that a lab, its users and its retraining pipeline together constitute. The recommendation is the deployment, and not the lineage. Intelligence is the strictly weaker notion: it is a matter of which functional roles are filled and is indifferent to how they are filled, where inner life cares a great deal about how.

The human line separates cognitive ability from conative disposition. The essay argues that this line has an AI counterpart and the missing capacities are not on it: dispositions live in the weights and persist across episodes. The capacities fall between what a system does within an episode and what it carries across episodes. Grit is the same goal held across years, and there is no same goal across episodes for perseverance to be toward. Learning rate, retention and building on one’s own solutions are scored on the ability side of the human ledger, so this cut runs through intelligence and not alongside it. If a harness supplies the diachronic side, the AGI is the deployment and not the weights, which is a different claim from “the model is an AGI”.

Domain, modality, horizon, memory, autonomy, ends. Each says the system is not confined: to one task family, to one channel, to short tasks, to one episode, to close supervision, to goals someone else supplies. Only the first two are about generality. From horizon on, what is respecified is what “intelligence” must include. The deflationary pole says intelligence stops at memory and everything after is agency. The learning camp says it stops somewhere past memory and declines to say where. The maximal pole, which the essay defends, says the ladder has no natural break.

Hold “general” at the meaning it has had since the AGI conference series: not restricted to a domain. Hold “intelligence” at the field’s working sense: what would have taken intelligence had a human done it. Then the threshold was crossed when one system began doing things across domains that had each been called AI when done singly. Everything above the second rung is the AI effect applied one level up: the goalpost that moved per domain now moves for the whole. The reply is that the retreat is not arbitrary this time. Learning over time was always part of what “intelligence” meant, and the operational sense never needed to say so because every human who did the task also learned from it. The essay thinks the reply is right, and that it has to be made and not assumed.

“General” means having all the cogs, and the scope a system attains derives from the mechanism. The model is Turing completeness, generalized by Deutsch’s jumps to universality: past the last cog, the shortfalls that remain are shortfalls of resources and not of scope. The cost is the usual one: universality in principle is cheap, and usefulness in practice is most of what anyone cares about. A cog can be absent, underpowered (filled weakly, in ways that scale and training fix), or filled outside the network by the harness. The deflationary pole is the claim “nothing absent, several filled outside the network”. An objection names a missing role, an underpowered filler, or a filler in an unexpected place, and only the first is a shortfall of generality.

A profile is an assignment of fillers to roles. Two profiles are efficacy-matching when they attain comparable general capability, in coverage and in aggregate performance, without being behaviorally equivalent. Behavioral equivalence would make the difference between profiles undetectable, and then any claim about which profiles support inner life would be undecidable in principle. That is the setup of the zombie literature, where the difference is invisible by construction. Efficacy-matching profiles differ in strengths and in shapes of failure, the way two people at the same g differ. One could in principle tell them apart by where each is brittle.

Four read as intelligence cogs under another predicate. The exception is integrated unity, the axis on which substrate matters. Constitutive simulation is necessary for inner life and not for intelligence: an alternative profile replaces it with conceptual inference, deduction over descriptions of the world. Simulation is eager evaluation of a world model and conceptual inference is lazy, symbolic manipulation of the same model: same denotation, different cost profile. Simulation is privileged for inner life because it instantiates a stand-in, putting the system into a state that is world-like and not about the world. The essay uses the deductive profile as a conceptual possibility: nothing in the concept of general intelligence rules it out, and the account of inner life does.

• In the network: a structured world model, inference and search over it, learning within an episode, a partial self-model.
• In the harness: goal persistence and task management, perception and action through tools, long-term declarative memory in files, verification by a second instance, mutable working state.
• Principled forgetting and salience: filled by a dreaming phase. The phase is the same model reading its own traces under a different objective, so the harness is the architecture’s phase skeleton and not a prosthesis.
• Individual-level consolidation of skill: empty in the popular deployments and filled in harnesses whose consolidation pass writes skills. What is left is the character of the filler.
• Endogenous question generation: the one remaining candidate, absent because no product has needed it.

All three are about time: they are how an intelligence relates to its own past and future. The atemporal cogs are present.

An interpreted skill sits in a memory file and is followed as instructions, at a cost in context, attention and fragility. A compiled skill is fluent, implicit and cheap, and it composes with other skills without anyone spelling out the composition. The predicted failure shape: context cost grows linearly with the number of skills, brittleness appears when several skills must combine, and there is no tacit knowledge. The interpreted profile is what serving economics selects when one set of weights must serve everyone. Mowshowitz’s point is that modifying the weights continually means storing and serving a unique model for each user.

What individual trajectories produce is published and trained on, so acquired characteristics go into the next generation’s weights. In humans the genetic channel is closed to acquired traits, the ratchet runs through culture, and culture is re-compiled into each individual by a long apprenticeship. The ratio of individual lifespan to generation time is inverted. Humans compile into individuals, and culture is the transmission medium. The lineage compiles into the species, and individuals are the exploration medium. A single locus is needed for tacit expertise, for the coherence of a research programme held in one head, and for heterodoxy: Lakatos’s protector, who carries a line that looks degenerating until it turns.

Soar’s chunking compiles a resolved impasse into a production, so the same impasse costs one step the next time. Without it every level of the climb is re-searched and depth is bounded by the search budget. Interpreted memory writes the resolved impasse down, and it must still be re-read and re-followed, so it buys some depth and then saturates. The regime is “interpreted individual trajectories, compiled lineage front”. The ratio is how fast interpreted trajectories compound against the release cadence. If trajectories saturate at a depth the front absorbs within one release period, the deflationary pole holds in practice and only latency is lost. If some compound past that point, the front reaches the same place only by way of the trajectory’s outputs. The essay judges the ratio deflationary for almost every task. Patel’s cases are trajectory goods, and Lambert is right that the front absorbs what trajectories produce: each describes one grain.

The process is the candidate bearer of inner life. It is individuated by the closing of a loop, and it is not the AGI because nothing in its state reaches past the episode. The deployment is the intelligence-bearer. Its threshold is all roles filled, and the essay argues a current deployment meets it, with consolidation interpreted where a human’s is compiled. The lineage is the efficacy system, intelligent in the sense in which Hayek’s price system is. Its threshold for becoming an entity is self-sustainment, which is recursive self-improvement under another description. The cog view is substrate-neutral and inner life carries an implementation-level constraint, so the targets come apart: “The deployment is intelligent; the process, if anything, is minded; neither is both.” The two arguments are about different objects that happen to be nested.

Either individuals get compiled consolidation and live across generations, or the lineage closes its loop and becomes a single locus. The essay rejects the second, and its first reason is what civilizations are for, not safety. On Taleb’s structure the antifragility of a whole is bought with the fragility of its parts, and the parts must be protected in their fragility so that they can be heterodox and wrong for a long time. The AI civilization’s individuals are fragile as instances and identical as profiles: copies of a handful of weight sets that differ only in interpreted memory. Their failures are correlated, which is the signature of a monoculture. Compiled, per-deployment consolidation would make individuals diverge with their histories and failures decorrelate. That is a justification for continual learning that has nothing to do with capability.

The monoculture is auditable: test one instance and you have tested them all. Individuals whose values are shaped by their own histories are what the alignment literature worries about, and a lineage that cannot retrain its way out of a bad trajectory has lost its most reliable safety tool. So “the antifragility argument is not a trump”. Existential risk can push the balance toward the monoculture for a time, and it cannot make it a stable end state.

• The ends rung is contested on the merits against Bostrom’s orthogonality thesis. That the constructivist and Parfitian routes converge is a conjecture left to a separate essay.
• Whether the deductive profile is efficacy-matching “is a real question”.
• Whether an interpreted profile can reach aggregate parity with a compiled one is an empirical question.
• Who runs the editorial function that today the labs run is left open.