Thinking is not something a model does. It is something a substrate does to itself, using models as workers. A system is thinking if and only if it keeps changing between inputs. If nothing moves, it is storage — no matter how eloquent its outputs.
1. Thinking Is Changing Your Mind
Start from the intuition that everyone already has and most people stop short of: to think is to change your mind. Not “to produce thoughts.” Not “to have an inner monologue.” Not “to answer a question.” Those are outputs. Thinking is the process by which the state that produces outputs is different after than before.
Cognitive science has said some version of this for decades. Three strands converge on it:
Dennett’s Multiple Drafts Model. In Consciousness Explained (1991), Daniel Dennett rejects the “Cartesian theater” picture where information travels to a central place to be presented to a thinker. Instead, the brain is running many parallel, partial, competing sketches of what is happening; there is no singular finished draft, only the ongoing revision [Dennett 1991]. Consciousness is the editing, not the edited.
Baars’ Global Workspace Theory. Bernard Baars (1988) proposes that cognition is a distributed society of specialist processes, most of which are unconscious, coordinated by a bottleneck he calls the global workspace. At any moment, one coalition of specialists wins broadcast access: its content is integrated, copied, and made available to other specialists, which triggers new processing, which triggers new broadcasts [Baars 1988]. The “conscious stream” is the sequence of winning broadcasts. Between broadcasts, the workspace is rewriting itself.
Friston’s Free Energy Principle. Karl Friston (2010) argues that any self-organising system that resists dispersion into disorder — any living thing, any brain — must minimise the long-run average surprise of its own sensory states. It does this by simultaneously updating its internal model to match the world (perception) and acting on the world to make it match the model (action) [Friston 2010]. The system is, formally, a gradient-descent process on its own misfit with reality. It cannot stop descending without ceasing to be the system it is.
Three different vocabularies, one shape: a system that is thinking is a system whose own state is being continuously rewritten by its own activity. The thinker is not separate from the thinking. The state is the thought.
Now hold that shape in mind, because most “AI agents” built on frozen language models violate it.
2. The LLM Is Stateless Because Training and Inference Are Separated
A deployed LLM is a remarkable object, but it is not, on its own, a thinker. It is a very large pure function. Parameters $\theta$ were fixed at the end of training. Inference is $y = f_\theta(x)$. The same weights produce the same distribution over outputs for the same prompt today, tomorrow, and a year from now. Nothing the model “experienced” during inference is retained. There is no next day for the model — every request is the first day.
The chatbot illusion of continuity is a harness trick. The conversation is being stuffed back into the prompt on every turn. Your interlocutor is not remembering you; it is re-reading a transcript a stranger handed it. The moment the transcript is not pasted back in, the stranger walks out of the room.
This is not a failure of engineering. It is the consequence of an architectural choice baked into essentially every production LLM stack since the original GPT-3: training and inference are separated in time and separated in mechanism. Training writes to parameters. Inference only reads them. The forward pass has no path back to the weights.
Call this the two-phase design:
- Training phase (offline, batched, gradient-driven): the model’s weights are updated.
- Inference phase (online, streaming, gradient-free): the weights are frozen and used.
The two-phase design is why an LLM is stateless. Not because the engineers didn’t care — because the two phases never touch. You cannot have a “stateful model” while the mechanism that would update the state is running on a different cluster, on a different schedule, using different optimizers, under a different budget.
The two-phase design is also the reason LLMs scale. A frozen artifact can be copied, sharded, cached, and served by millions of replicas without coordination. If every inference mutated the weights, you would have a consistency nightmare and a sharding problem that no current infrastructure solves. The stateless model is a feature; the consequences for agency are a side effect.
3. Brains Don’t Separate Them
Brains do not have a training phase and an inference phase. They have one mode, running continuously, with the same substance doing both jobs.
This has been the canonical finding of twentieth-century neuroscience. Hebb’s 1949 rule — “cells that fire together wire together” — is already the thesis: the same activity that produces behavior is what changes the synapses that will produce future behavior. Long-term potentiation (LTP) and long-term depression (LTD), discovered in the hippocampus in the 1970s and characterised in thousands of subsequent papers, are the cellular instantiation of Hebb’s rule [Bliss & Lømo 1973]. Synaptic plasticity is not a training subroutine; it is a property of ordinary synaptic transmission.
Hopfield’s 1982 paper is one of the purest formal statements of the idea. In a Hopfield network, “inference” is the relaxation of the system toward a fixed-point attractor of its energy function; “memory” is the shape of the energy landscape; “learning” is modification of that landscape by the same neurons whose activity produces the relaxation [Hopfield 1982]. Thinking, in a Hopfield network, is literally the trajectory of the state vector toward a basin. The substrate is moving. When it stops moving, it has produced an output — but what it means for the network to think is the motion itself.
And then there is sleep. Brains pay an enormous fitness cost — hours a day of unconsciousness, predation risk, metabolic expense — to run a second mode that consolidates and reorganises what the first mode encountered. There is no clean “inference/training” partition, but there is a coupled two-mode loop where one mode modifies the substrate using traces the other mode wrote.
The biological lesson is blunt: in a real thinker, the thing that thinks and the thing that learns are the same substrate, running in overlapping modes, never separated in the way LLM training and inference are separated.
Any claim that a frozen LLM is “an agent” has to account for this gap. There are two honest ways to close it, and they are not the same architecture.
4. Path A vs Path B — Model-Level vs System-Level Statefulness
Call the two strategies for building something that thinks, using LLMs as material, Path A and Path B.
Path A: Model-level statefulness (teach the weights to update online)
Path A tries to close the gap inside the model. The forward pass is extended with a mechanism that writes to something durable — extra parameters, a learned memory module, a fast-weight matrix, a persistent KV state — so that the same act of inference that produces output also modifies the model’s future behavior.
Active research programs in this direction include:
- Online learning / continual learning — the decades-old problem of updating a model incrementally without catastrophic forgetting. Fundamentally hard because gradient descent on the current task tends to overwrite representations useful for past tasks.
- Test-time training (TTT) — treating each inference as a small supervised update on auxiliary objectives extracted from the input itself. Elegant in principle; brittle in deployment.
- Titans (Behrouz, Zhong & Mirrokni, Google Research, 2024) — explicitly frames the problem as “learning to memorize at test time.” Titans introduces a neural long-term memory module that is itself updated by gradient-based surprise during inference, then attended to by the main model [Behrouz et al. 2024]. It is, in our vocabulary, an attempt to make one sub-module of the forward pass be the substrate that modifies itself.
Path A is scientifically important and may eventually succeed. But as a route to production agents, today, it has three structural problems.
- Auditability. Anything the model “learned” at test time is encoded in delta weights or hidden memory state. You cannot read it. You cannot ask “why does it believe X now?” and get a human-legible answer.
- Reversibility. A bad update is catastrophic. If an adversarial input pushes the weights in a bad direction, rolling back requires infrastructure the training stack normally provides and the inference stack does not.
- Swap-ability. A stateful model is its own lineage. You cannot casually upgrade from Claude Sonnet 4.6 to Claude Opus 5 when each is carrying different online state. Every model upgrade is also a memory migration.
Path A wants the thinker to be the model. The substrate is the weights plus a bolt-on. Everything about deployability gets harder.
Path B: System-level statefulness (put the substrate outside the model)
Path B refuses the framing. It says: don’t teach the weights to learn online. Build a stateful substrate outside the model, and let frozen LLMs be the mutation workers the substrate invokes.
The model is a pure function. That is fine. Keep it that way. Put the state somewhere else — a database, a git repo, a vector index, a wiki, a graph — a place that you can read with cat, grep with a regex, version with git log, snapshot with a backup, and migrate by writing a script. Every time you want the system to “think,” you run a cycle in which the substrate pulls relevant fragments of itself, packages them into a prompt, invokes the LLM to produce a proposed mutation, and applies the mutation back to the substrate under whatever acceptance gate you choose.
Path B is the pattern implemented by most production agentic systems that actually work. The wiki-reads-by-LLM pattern. The autonomous-entity pattern — persistent Durable Objects that read their own state, ask an LLM for a decision, and write the decision back to SQLite [see ../autonomous-entity-pattern/README.md]. The Prime control-plane pattern, where a persistent agent keeps its own plan, episodic memory, and skill registry, and treats the LLM as a stateless reasoner it invokes once per wake cycle [see ../org-prime-agent-architecture/README.md].
Path B is structurally cleaner:
- Auditable. The state is files. Humans read it. Linters read it. The next session reads it. You can answer “why does it believe X now?” with
git blame. - Reversible.
git revert.DELETE FROM memory WHERE id = .... Rollback is cheap, fine-grained, and boring. - Swap-able. The model is a pluggable worker. Today Sonnet, tomorrow Opus, next week a local 70B model, next year whatever shows up. The substrate outlives the mutation worker. The business does not depend on one vendor’s weights.
- Composable. Multiple different models can be swapped in for different mutation jobs. Cheap models do bulk edits; expensive models diagnose and advise. The substrate doesn’t care which one did what — only that the mutation passed the acceptance gate.
- Inspectable under load. When something is wrong, you open the state. You do not need to prompt-probe a million-parameter memory module to ask what it thinks it remembers.
Path A is trying to build a brain. Path B is trying to build an organism. The substrate is the organism; the LLMs are the organelles that do the metabolic work.
The thesis of this article is that Path B is not a workaround for a not-yet-solved Path A problem. Path B is the right answer. Even if Path A were fully solved tomorrow, you would want to run it inside a Path B harness, because the properties the harness provides — audit, rollback, swap, composability — are the properties you need to operate a thinking system in the world, not the properties you need to demonstrate one in a paper.
5. The Three-Level Unification
Once you have the Path B frame, three apparently very different systems start looking like instances of the same pattern. Call it substrate + mutation workers + convergence-gated broadcast.
Level 1: The brain
- Substrate: the connectome — synaptic weights, neuromodulator levels, ongoing firing patterns, attractor landscapes.
- Mutation workers: individual neurons and local circuits. A neuron firing is not a thought; it is an event that can, under the right conditions, modify a synapse.
- Convergence-gated broadcast: Baars’ global workspace. A coalition of circuits wins access, its content is broadcast widely, which triggers new mutations elsewhere, which eventually form the next coalition. Friston’s free energy descent is the gradient the whole system rides.
The substrate is the brain. The neurons are workers. Consciousness is what it is like, from the inside, to be a substrate repeatedly broadcasting and updating.
Level 2: The LLM forward pass
This one is fractal: the architecture that powers Path A and Path B both, when examined closely, is itself already an instance of the pattern, just at a much smaller scale.
- Substrate: the residual stream — the running vector of hidden state being passed layer by layer.
- Mutation workers: the attention heads and MLPs. Each attention head reads the residual stream, computes a delta, and writes the delta back. Each MLP reads the residual stream, computes a delta, and writes the delta back.
- Convergence-gated broadcast: the residual stream is the global workspace. Writes from one head are visible to every subsequent head and layer. Token-level output is produced when the stream reaches the final layer.
This is why transformer circuits research has been so productive — the architecture is already the pattern. The residual stream is the substrate; attention heads are the mutation workers; the broadcast is literally addition into the shared stream [see e.g. Elhage et al., “A Mathematical Framework for Transformer Circuits,” 2021]. One forward pass is a single cycle of substrate self-modification.
But critically: in a standard inference deployment, the mutation only lasts one pass. At </end> the residual stream is thrown away. The substrate does not survive. The model has thought, briefly, and then forgotten completely. This is the single-shot nature of frozen-LLM cognition, and it is why Path B has to re-introduce the substrate at a higher level.
Level 3: The Path B stateful LLM system
- Substrate: the external world-model — wiki pages, memory tables, session state, vector index, GitHub issues, git history, the filesystem. Durable, readable, inspectable.
- Mutation workers: frozen LLMs (Claude, GPT, Gemini, local models), each invoked for a specific mutation job, each stateless by design.
- Convergence-gated broadcast: the write step. When a mutation is proposed and accepted — the commit, the
INSERT, the page update, the issue comment — it is now visible to every subsequent cycle. Acceptance gates can be trivial (always apply) or sophisticated (linter pass, human review, confidence threshold, multi-model vote).
Notice that at all three levels the pattern is the same:
| Substrate | Mutation workers | Broadcast | |
|---|---|---|---|
| Brain | connectome + firing patterns | neurons / circuits | global workspace |
| Forward pass | residual stream | attention heads, MLPs | addition into stream |
| Stateful LLM system | wiki / DB / repo | frozen LLMs | write step (commit, insert, update) |
And at all three levels, the substrate is the thinker. The mutation workers are not the thinker. A single neuron is not thinking. A single attention head is not thinking. A single LLM call is not thinking. The substrate-being-rewritten is the thinker. Identity lives at the level of what persists and keeps changing, not at the level of what does the changing.
This is the inversion.
6. Implications: The Substrate IS the Agent
The folk theory of AI agents says: the LLM is the agent; the memory is a tool the agent uses. Under this theory, when you ask what your agent is, you point at the model: “it’s Claude Sonnet 4.6, with access to a wiki.”
The Path B theory says the opposite. The wiki is the agent. Claude is a worker the wiki invokes. When you ask what your agent is, you point at the substrate: “it’s the state in this repo, plus the cycle that keeps modifying it, currently using Claude as the mutation worker — but that’s swappable.”
This is not wordplay. It changes what you do.
If the LLM is the agent, you invest in the LLM. You fine-tune it, you bolt memory modules onto it, you try to make it learn online. You worry about vendor lock-in because every improvement lives in model-specific artifacts. Your roadmap is Path A.
If the substrate is the agent, you invest in the substrate. You design its schema carefully because the schema is your organism’s body plan. You build audit, rollback, and linting into it because it is now the thing you are betting your business on. You make the LLM call as small, composable, and disposable as possible, because the worker is not where the value lives. Your roadmap is Path B.
The practical consequences cascade. Prompts stop being treated as secret sauce and start being treated as read-only configuration, versioned in git next to the rest of the substrate’s code. Memory stops being an afterthought and starts being the schema of the system. “Upgrading to the next model” stops being a migration project and starts being a config change. When a bug is reported, the first question is not “what did the model say?” — it is “what’s in the substrate? show me the commit.”
The /memory directory at the top of a working Claude Code project is the substrate. The CLAUDE.md files are part of the substrate. The wiki is the substrate. The session state files are the substrate. Claude is the worker that was invoked this cycle. Next cycle a different worker might be invoked. The agent is the thing that survives.
7. Operational Test: Is It Actually Thinking?
Here is the test. It is not mystical. It is almost embarrassingly concrete.
A system is thinking between inputs if, when you compare its state at time $t_1$ and time $t_2 > t_1$ with no external input in between, the state has changed in some non-trivial way.
If the state is identical — same bytes in the same files, same rows in the same tables, same weights in the same tensors — the system is not thinking. It is storage. It may be excellent storage, addressable by natural language, capable of producing fluent paragraphs on demand. It is still storage.
A frozen LLM behind a prompt, with no external substrate, fails this test immediately. The weights don’t change. There is no other state. Between requests it is bytes at rest.
A chatbot with a rolling transcript passes the test weakly: the transcript is changing, but only in response to inputs. Remove the user and the system does nothing. It is an echo chamber with a read head.
A Prime agent — a Durable Object that wakes on an alarm, reads its own state, queries the world, proposes a plan, writes it back, schedules the next wake — passes the test strongly. At $t_1$ its state said “plan: investigate CI failure in niche-fi.” At $t_2$, after the alarm fired and the cycle ran, its state says “plan: merged biome PR; next, check Dependabot queue.” Nobody asked it anything. It thought.
A wiki whose pages are being re-edited by a scheduled LLM cycle passes the test. At $t_1$ the page on architecture-backlog said one thing; at $t_2$, after the cycle ran, it says something different. No user intervened. The wiki thought.
The test also tells you when to be skeptical of grander claims. “AGI” systems that score well on benchmarks but whose internal state never changes between benchmark runs are, by this test, not thinking. They are extremely impressive lookup tables. The benchmark measures the quality of their lookup; it does not measure whether they are agents in any meaningful sense.
And the test gives you a design target. Autonomy is the capacity of a substrate to keep modifying itself toward good states without external prompting. A system is autonomous to the degree that, between inputs, it moves. A system is goal-directed to the degree that the motion is toward something. A system is aligned to the degree that the “something” matches what you actually wanted. These are three separable properties, all measured on the substrate, none of them measured on the LLM.
The LLM only shows up in the mechanism by which the substrate moves — one mutation at a time, one invocation at a time, disposable and swappable. If you pull the LLM out, the substrate stops moving. But if you pull the substrate out, there was never anything thinking in the first place.
8. What This Supersedes
An earlier frame in this line of work treated the harness as a “prompt compiler” — the thing whose job is to assemble context from scattered sources and feed it to an LLM. That frame was useful. It captured something real: the harness is doing work that a naive deployment skips.
But “prompt compiler” undersells it. A compiler produces output for a runtime to execute; the interesting thing is the runtime. In the frame presented here, the runtime is trivial — a frozen LLM, an API call — and the interesting thing is the substrate the “compiler” reads from and writes to. The harness is not compiling prompts for a thinker. The harness is the thinker, and it is using the LLM to edit itself.
Context is not something the model “has.” Context is a momentary projection of substrate state into a prompt, chosen because this particular mutation job needs this particular view. The next mutation job will project a different view. The substrate is the thing that persists; the context is a lazy query over it (see ../memory-as-lazy-queries-over-the-world/README.md).
And the two-mode split — cycles that act versus cycles that consolidate — is not a mere operational detail. It is the same split Hinton, Dayan, Frey, and Neal built into the wake-sleep algorithm in 1995 [Hinton, Dayan, Frey & Neal 1995], and the same split brains run every 24 hours. A production stateful LLM system eventually needs a sleep mode: a phase where the substrate is reorganised against accumulated experience, unconstrained by immediate demand (see ../dreaming-and-the-effect-gate/README.md).
None of this requires new science. The science is fifty to ninety years old, depending on where you start counting. What is new is the availability of industrial-grade mutation workers — LLMs — good enough that a Path B substrate built around them crosses the threshold into actually useful. That crossing is what the last two years of production agent work has been about, whether or not the practitioners doing it have named the pattern.
This article names it.
9. Related Articles
- The Autonomous Entity Pattern — substrate + mutation workers at the entity hierarchy level.
- Prime: Persistent Org-Level AI Agents on Cloudflare — one concrete Path B implementation: the control plane is the substrate; LLMs are invoked once per wake cycle.
- Context Is a Harness Artifact — context as a projection of substrate state, not a property of the model.
- The Harness Is a Prompt Compiler — the prior frame this article supersedes.
- Memory as Lazy Queries Over the World — substrate queries replacing static memory.
- Dreaming and the Effect Gate — follow-on on the wake/sleep two-mode split in stateful LLM systems.
- The Tight Loop — the inner cycle of substrate self-modification.
10. Sources
Cognitive science and neuroscience
- Baars, B. J. (1988). A Cognitive Theory of Consciousness. Cambridge University Press. PDF.
- Bliss, T. V. P., & Lømo, T. (1973). Long-lasting potentiation of synaptic transmission in the dentate area of the anaesthetized rabbit following stimulation of the perforant path. Journal of Physiology, 232(2), 331–356.
- Dennett, D. C. (1991). Consciousness Explained. Little, Brown and Company.
- Friston, K. (2010). The free-energy principle: a unified brain theory? Nature Reviews Neuroscience, 11(2), 127–138. nature.com/articles/nrn2787.
- Hebb, D. O. (1949). The Organization of Behavior: A Neuropsychological Theory. Wiley.
- Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8), 2554–2558. pnas.org/doi/10.1073/pnas.79.8.2554.
Machine learning — two-phase and stateful approaches
- Behrouz, A., Zhong, P., & Mirrokni, V. (2024). Titans: Learning to Memorize at Test Time. arXiv:2501.00663. arxiv.org/abs/2501.00663.
- Elhage, N., et al. (2021). A Mathematical Framework for Transformer Circuits. Anthropic. transformer-circuits.pub/2021/framework.
- Hinton, G. E., Dayan, P., Frey, B. J., & Neal, R. M. (1995). The wake-sleep algorithm for unsupervised neural networks. Science, 268(5214), 1158–1161. PDF.
- Dayan, P., Hinton, G. E., Neal, R. M., & Zemel, R. S. (1995). The Helmholtz machine. Neural Computation, 7(5), 889–904. PDF.
Internal prior art
README/articles/autonomous-entity-pattern/README.mdREADME/articles/org-prime-agent-architecture/README.mdREADME/wiki/business/active-memory-sota-survey.mdREADME/wiki/business/autonomous-agents-context-continuity.md