Prompts: Where Does Seeing Become Thinking?

These are Łukasz Stafiniak’s prompts that steered the drafting and revision of “Where Does Seeing Become Thinking?” They record the human framing and source selection separately from Codex’s research and drafting. Local paths identify supplied materials; the private transcripts themselves are not reproduced here.

For our next blog post, we could write a reaction essay. One reaction is to see how notes/private/why_AI_still_cant_see_Andrew_Dai.vtt affects our analysis and conclusions in which-world-model.md. The second related reaction is to https://chatgpt.com/share/6ab619be-1180-83eb-b8f1-82cf10a7c886 (in case you can download), especially relevant is the point about modality-specific representational manipulation. By itself this is not enough material for a full blog post though, let’s brainstorm.

Would it make sense to loop in interpretability research on geometric representations?

Could we also make this essay a continuation of concept_theories.md? Via studying whether the perception-cognition boundary exists in frontier (multimodal) LLM-derived AI systems.

I like this, thanks! Also great you found “The Border Between Seeing and Thinking” available online! I have read it on Kindle. I’d like you to draft the essay and we’ll iterate from there.

To add some final touches to the essay, let’s incorporate references to this very recent interview: notes/private/Chris_Manning-Info_Bottleneck_Podcast.vtt (where it provides valuable information)

Thanks! Do you think adding Chris Manning was worthwhile? Let’s also introduce sections

“When Geometry Does Computational Work” section: first two sentences are unclear to me. What do they say? “Interpretability research helps because neural representations can be investigated at a level between the system’s output and the names assigned to its components. But geometric findings require interpretation of their own.”

That’s better because I don’t know what “component names” was doing in the original. But “geometric” enters a bit abruptly, maybe “is a starting point in modeling a world: …”, or something like it that would capture your intent connecting to the theme or earlier text.

About the circular representation of calendar concepts, “The question of whether this should be called iconic requires further argument about the representation’s functional properties.” is a stretch on the intended (by Block) meaning of “iconic” I think. But maybe it’s not worth quibbling over. (There’s no privileged space of days of week say of which the representation is a homomorphism, the way that perceptual representations capture primary physical properties.)

The last sentence in the very interesting “cross-modal causal tracing” paragraph is a filler, drop it: “It does not identify the full decoder with either perception or cognition.” (Unless I failed to get what it’s accomplishing.)

This paragraph is the strongest yet connection to our blog essays on phenomenal consciousness. I’m wondering if it’s worth to bring this debate up in this essay, or leave for another essay in case we notice more such nuggets.

This is too verbose. I’d maybe add a sentence at the end of the paragraph: “A longtime reader of our blog will notice a connection to our [evolving/tentative] account of phenomenal consciousness.” where I’m not sure whether to include one of the softeners.

“Three Ways to Test the Boundary” section. It’s not clear what the desideratum we are testing is. Using the earlier terminology, a pure-cognition (in Block’s sense) system can be efficacy-matching: can tackle any task, just with different efficiency profiles (behaviorally) and representation decompositions (mechanistically). “Even a successful result would leave the functional question open.” What counts as success, the capacity for separating the confounds (identity, properties), the operational caching out of binding (in the object binding sense)?

I wonder if we should put the last occurrence of binding in quotes, or prefix it with a parenthetical “(the equivalent of)”. That shift is already covered by your phrase “reconstructed” but readers might miss it.

This works. Let’s make the combined update. My point was that binding has that narrow sense in philosophy of mind, and the broad cognitive/informational sense, and we don’t want to “beg the question”.

The paragraph “Training history must be distinguished from influence during an encounter.” is a bit unclear. How could one confuse training history (weight updates) from occurrent processing (activations)? Is the implicit remark that in-context modulation is architecturally weaker than affordances of biological brains, and this needs to be kept in mind to not jump to conclusions?

Much better, do the update thanks

Is Stroop effect close to the second family of interventions?

A complicating factor for the third family of experiments is around top-down activation of perception in humans. It’s the case of imagination: there is a perceptual representation formed, but it’s phenomenally tagged as such (not a perception of the current situation), and both the ease of forming and the details/abstractness vary across humans.

Let’s add that, thanks

Not sure if it’s something worth mentioning in the essay, but I felt that complication in block is also not fully articulated, since he’s pursuing the perception joint in ontology as evidence for his account of phenomenal experience. Yet his perception account does distinguish it from imagery, if I understood correctly. It’s complicating things for him in a similar way. WDYT?

That’s a good point. Let’s add it.

Hmm, maybe what it says is that perception-cognition (in Block’s sense) is not a dichotomy? Actually, Block allows for perception to be embedded in cognition, it’s just unclear what the status of imagery is.

I would maybe append this sentence rather than replace your previous one. It’s a better progression, and Block doesn’t pop unexpected.

“Thinking with the Environment” section. Is there literature on how efferent copy anticipatory processing relates to perception?

I don’t want to prejudge placing it in this section. It’s just where we start talking about actions.

Maybe where it contributes is on the impact of architectural differences rather than on the explanatory role of the boundary itself.

“The same reasoning exposes a tension in abstraction.” Which reasoning?

Go ahead.

Paragraph “The comparison should also be longitudinal.” feels somewhat off-topic, although examples it lists are relevant to perception. It reads as a call-back to an earlier essay on the blog “Which AGI?”.

Go ahead.

“ability to cross it in both directions: to make what is seen available for thought, and to let what is seen correct the course of thinking” is not really two directions in the sense of perception -> cognition (bottom-up) and cognition -> perception (top-down), the two cases are both the former.

Is there a notion of “depth of processing” in the science of perception? Maybe that’s what the top-down direction would be more prototypically rather than “what to examine next”. The latter is just a regular world-to-perception causal flow looped through action.

Go ahead, thanks.

Should we introduce a new section about prima facie differences between brain architecture and frontier AI architectures? If we were to discuss reference copies, that would be where it would fit. Otherwise, we keep reference copies out of the essay. In this new section, we would also expand on whether top-down influences, like we just discussed, are even architecturally supported i primarily feedforward networks as used these days. (There’s believable speculation that GPT-6 Astra and GPT-6.1 Sol are Looped Transformers: use a small degree/amount of looping; but unknown which layers loop back to which.)

“A bounded recurrent computation can be unrolled into a feedforward computation.” Yes, but evolutionary / training pressures are different at sufficiently distant layers. Separately, we have an old essay on the blog about dynamics and looped models: axes_of_dynamics.md It might not be that good, don’t take it as authoritative.

Is the recent work on recirculated transformers relevant?

Agreed, go for it. (Interestingly, as in your third bullet point, this paper provides evidence on representation in existing transformers, the residual stream is closer to a blackboard system than one might think.)

The last paragraph of the new section reads to me a bit like trying to reverse engineer the architecture rather than seeking what architectural changes cause downstream. But I’m probably reading it wrong if it was intended as varying the architecture rather than varying the experiment setup.

All right, this looks good! Would you like to make a final editorial pass over the essay?