Prompts for How LLMs Could Place Explicit Memories

The idea

For a future essay, we will trace the lineage from Memorizing Transformers https://arxiv.org/html/2203.08913v1 to this: https://prefrontal.ai/news/lsa/ , https://arxiv.org/html/2610.42147v1 . The essay will land after the LLMs’ two-part deep dive into architecture and training (which is in progress still), so it shouldn’t be redundant. It will also point back to multiple other essays, especially the continual learning series. I attended a weekly reading group organized by Christian Szegedy, so the authors of the paper are familiar to me. WDYT?

The outline

All I have is notes/private/emergent_scratchpad-to_Marcus_Rabe.txt because I no longer have my meeting notes. I attended the reading group online. Draft the outline, thanks!

Q1: I think the paper was discussed, I was referring to the paper in the letter/email to Marcus. Q2: use the email as source of truth; the exact idea was mine, but the group was working on retrieval-augmented contrastive training, where the training batch would be used constrastively IIRC. Q3: indexer-as-student, keep it here. Q4: a full section. https://osf.io/preprints/psyarxiv/j75zw_v1 argues that in PFC “the abstract structure of the world is embedded locally in connections between prefrontal neurons encoding conjunctions of action and context”; would that be relevant? Q5: keep one sentence. Q6: don’t know, sth like “How LLMs Could Place Explicit Memories”? Q7: sure.

Two views of consolidation

“If consolidation is amortization (Where Experience Goes)” – this is a deflationary view. There’s also a substantive view that consolidation does representation-building work not reachable by retrieval even if intelligent (What Breadth Reaches).

Before drafting

Any further questions or ideas you’d raise before writing the essay draft?

Q1: we can also wait till the training draft exists, WDYT? Q2: sounds good. Q3: I think you can correct the typos (not wording or grammar), that’s jarring without benefit. Q4: Sounds good. Staleness: good point, it’s a bit like the off-policy problem.

Memorilla

This is probably great to pull into the essay, wdyt? https://arxiv.org/html/2610.21375v3

Drafting

Alright, the first draft of the training essay is here: notes/filling-the-stores.md (It will probably still evolve, but unlikely to drop info.)