AI can think fast but not slow

It has System 1, not System 2 — or System 3

Image generated by me using Midjourney… don’t know why this guy looks like ‘80s Freddie Mercury that wasn’t in the prompt.

In Thinking Fast and Slow, System 1 is the fast automatic part of our thinking while System 2 is slow and deliberate. In the creeping backlash against this highly influential bestseller, one criticism is the simplistic personification of these systems which don’t correspond to any brain physiology. But Mrs Kahneman raised no fools — the author of the book, Israeli psychologist Daniel Kahneman, knew what he was doing. He offers them as “fictitious characters,” two “agents within the mind,” because he knows this will be more captivating for the minds of his human readers than talking about mental modules or cognitive mechanisms. And in a book explaining how our minds work in terms of biases and mental shortcuts, it would be treacherous for Kahneman to do anything else.

Systems 1 and 2 are still useful metaphors. They correspond pretty well to two different classes of brain functions, if not brain regions. In fact, I’ve been trying to extend the metaphor to include a System 3, like so:

System 1. Fast thinking, mainly unconscious. It’s old and utilises the exquisite learning mechanisms found in the nervous systems of animals from sea slugs to chimps. Judgements are made in well under a second. It is in sync with the rhythms of life: darting eye movements to scan the scene, reflexes, motor control, metabolic adjustments.

System 2. Slower, conscious thinking that unfolds over timescales from 300 milliseconds to several seconds — the span of working memory. It is for tasks that need sustained thinking: planning, arguing, recollecting, demonstrating. These have a sequential, narrative, elongated aspect to their event timings and mainly concern social interactions which happen at a macroscopic level in space and take up more time too. Physicists might say their characteristic time is longer than unconscious processing, which is more attuned to the momentaneous pop and fizz of biochemical events.

System 3. Even slower thinking that is supra-conscious. It involves multiple human brains, and nonhuman artefacts, unfolding over minutes to centuries in networks of knowledge. It’s a cultural overlay on top of primate cognition, allowing tools to be improved, customs to propagate, kinship networks to be established, institutional knowledge to form. This thinking can elapse over the timespan of generations. 

What kind of thinking can AI do?

So far, large language models (LLMs like ChatGPT and Claude, which I’ll use here to represent the most “advanced” AI) do something akin to System 1 thinking. Neural nets are roughly analogous to the networks of neurons that allow us to find a signal amid the noise of data entering our senses. Something as basic as recognising an object is a purely System 1 exercise (although we can deliberately pay attention to something in a visual scene, elevating certain signals to System 2). LLMs can do this and they do it quickly, even faster than our unconscious thought.

What about System 2? Although LLMs output plausible sentences in natural language, they are not at all aware or conscious, they have no working memory. True, they have an impressively large context window and attention. But these are nothing like the temporal sense involved in human working memory, i.e. the feeling of a “thick moment” or “specious present” that is smeared over a second or two. An LLM can carry on an argument and even make rudimentary plans. You can prompt it to go through its thinking step-by-step. But few would suggest it’s doing so in the deliberative daydreaming way that a human brain does. (I’ll come back to this point.)

Do LLMs have a System 3? Does it have access to some superstructure or AI culture? LLMs are connected to the Internet and continue to interact with humans, becoming ever more exposed to human culture. This on top of the norms and conventions of culture recorded in the text, images, and videos that made it into their training data. But I don’t think they have System 3 in anything like the human sense. And this is good because I think the key to human or general intelligence is System 3, not System 2 — at least not directly. 

Our System 3 is new

Early humans had brains identical to ours. But whereas we, every day, play with new gadgets, hear new stories, and learn about changes to our environment, our ancestors lived in a stable world. A Palaeolithic technology, like an stone blade, might update every few thousand years. Then the neolithic revolution happened. This brought agriculture, living in permanent settlements, faster development of tools — the rate of change of everything increased. 

There are debates over what triggered the sudden shift to cultural and technological innovation. Best I’ve heard is that population density reached a crucial threshold.¹ Once there were enough groups of people living within trading distance of each other, once the Earth started to fill with humans, horizontal transmission of ideas, tools, and practices became possible. Previously, if you had a good idea you could pass it on to your kids and hope they remember it too. But if your whole tribe is wiped out the idea dies. But with a network of trade, it’s like RAID data storage: there’s redundancy so even if one tribe is wiped out the idea lives on among others. 

The network was much smarter than any individual or any tribe because it could use specialisation. Nowadays we think of this as the knowledge embedded in the economy. Like I saw some guy on YouTube once who made a chicken sandwich from scratch and it took six months. You get what I mean. 

People plugged in to System 3 can do so many things no isolated person can. Even annoying bureaucracies enable a thousand tasks I could never achieve myself. And when I don’t know something I just dip into a repository like YouTube or the library to find it out. I have the exact same brain as my ancestors and I’m more personally useless, but I can think much more, thanks to all the cognition I outsource to System 3.

And now we’re trading and recombining knowledge in still more ways. This includes the use of the Internet, obviously, but also AI which now increases my cognitive output by writing emails and bullshit grant applications. I mean, it’s not the most noble calling, but in theory it frees up my capacity for other work. So LLMs are themselves part of our System 3. Equivalently, System 3 — the network of institutions, protocols, agreements, deals, conventions, techniques, laws, rituals, practices, and memes — is the first artificial intelligence. We taught artefacts and protocols how to think.²

We think of System 2 as our glory: higher cognition, the seat of reason, the soul, the edge we have over other primates. But it’s evolution’s beta version of System 2, less than a million years in development and still kind of buggy. Whereas our System 1 is an exquisitely robust model, used by other mammals successfully for tens of millions of years. And our System 3 is only a few thousand years old, much of it only a few hundred: just some specs jotted on a whiteboard. 

System 1 acts fast but learns slow

AI is astonishing when you think about it for like five seconds. Although there’s hype and overblown claims, it’s just wild that some bits of the Earth’s crust (silicon), arranged in lots of little gates, can be used as a personal tutor, that can, for example, teach me about how computers work. 

Getting to this stage of AI wasn’t easy. Take a simpler algorithm: an image classifier learning to recognise Margot Robbie. You show it a picture of a ficus plant and it guesses it’s Margot. You say no. Now it knows it has to learn something, because of the negative feedback you gave it, so it adjusts its weights. Next you show it a picture of Margot and it guesses correctly. You give it positive feedback. But it won’t know what about the image makes it Margot Robbie and it won’t even adjust anything because it hasn’t received negative feedback. It doesn’t know what it is about the image that deserve credit for the accurate guess. 

In reinforcement learning, this is called the credit assignment problem. Basically, you want feedback to be as targeted as possible so that a learner not only learns when they do a good or bad job but can zero-in on precisely what was good or bad about it. This is especially true of complex tasks. Imagine you want to learn how to make a new recipe, say spaghetti all’assassina, but will only receive a thumbs up or thumbs down at each step, no other information. You would just have to trial-and-error your way through the enormous space of possible actions until you alight, after millions of moves, on the dozen or so steps in the recipe, in the right order.

Interestingly, evolution by natural selection works according to this maddeningly slow and uninformative process. If an organism dies before it reproduces, that’s negative feedback, a thumbs down, given by the environment. There’s not even an equivalent of thumbs up. The organism is never positively selected; if it reproduces, its genes are merely not negatively selected for now. It’s a mindless, aimless process. And there’s no credit assignment. The organism might have died because of a recent mutation that was maladaptive; or because of an old mutation that no longer suited the environment; or it could have been bad luck. As much as the collective genes in the population of organisms in a species can be said to “learn”, all they can do with the negative feedback is “try” some different mutations. Trial and error.

It would be so much more efficient if evolution sent positive signals. If it could somehow learn what worked and why — rather than only what doesn’t work and with no explanation — it could mutate strategically rather than randomly.

Why hasn’t evolution cracked this? It’s had a few billion years and unlimited free energy from the sun. Well, it has cracked it, a little bit. First, it enabled organisms that can learn within their lifetime by evolving brains. Second, circuits of gene expression are like a kind of demi-brain, at least for a limited range of options. Given environmental cues, different genes will switch on or off, allowing for a range of “mutations” within an organism’s life. Third, we are products of evolution and we have an even more powerful way to learn, a method for actually solving credit assignment. 

(Everything from here down is new thoughts and somewhat speculative.)

Teaching and learning

If you’re trying to sharpen an axe by yourself, or cook spaghetti all’assassina, you don’t know where you’re going wrong and you might not even have a clear idea of what the right way would look like anyhow. You’re left with blind trial and error. But if you have a teacher, they can collapse the space of possibilities and say, “This way,” or, “Like this”. You still have to try and possibly fail, but the scope of your failures is narrowed down and you can be told which direction to head for.³ 

The critic George Steiner felt the most important feature of language was the subjunctive, the counterfactual, the what if? We can conjure unreal futures and thereby help create them. In contrast, I nominate the demonstrative this way or like this as the germ of civilisation. They are deictic expressions which, coupled with actions, form a technique that enabled humans to advance beyond the blind learning of natural selection and the unconscious learning of our nervous systems.

The jump in sophistication is chiefly one of improving error-correction to make it much more targeted. The impact is well dramatised by our hunter-gatherer ancestors on the Serengeti. Underbellies exposed, slower than lions, weaker than water buffalo, senses duller than the leopard’s: yet the ability to learn how to throw a spear, start a fire, and use a bladder to carry water, elevated them to the top of the global food chain.

It might seem like this aspect of System 2 is relatively easy. We can already do it with AIs. We can communicate instructions to them directly — “like this” — rather than relying on them to learn everything via reinforcement learning, struggling with credit assignment.

But primate evolution had to work with crooked timber. And you can’t enter instructions into another primate’s brain using a command line interface. Teaching could only happen via the channels of speech and physical demonstration, together. 

Speech is an information bottleneck: you can only articulate one syllable at once, so all the ideas you want to convey get extruded out into a one-dimensional string of words. We’re limited by the physiology of our tongues and how quickly we can jabber. Physical demonstration, meanwhile, is approximately as slow as the task it’s reenacting or modelling. For the learner to track these channels, they needed a special way of thinking that took the flashes of attention given to salient objects and knitted them into overlapping threads of attention sustained over seconds: working memory, the sine qua non of System 2.⁴

LLMs can be told this way. I can instruct, directly, ChatGPT in the precise way I want it to complete certain tasks. E.g. “I’m going to paste in some text, please put it into a table with three columns labelled…” and so on.⁵ But obviously I can’t yet demonstrate an action for an AI to imitate. Maybe one day robots will advanced to the point that one could look over my shoulder while I show it the proper way to cut and peel an avocado.⁶

I note that some other animals have a rudimentary System 3. Eusocial insects have protocols or rituals they follow and these help the colony achieve feats of cognition no individual ant ever could. (Hat tip to my collaborator Mathew McGann for this point.) But no other animal has anything like our System 3, which trades knowledge horizontally and liberally. And yet many nonhuman animals do have the requisite population density. This suggests that System 2, seemingly unique to humans, might be necessary for our style of System 3. In other words, even though System 3 — culture, technology, and institutions — appears to be the thing that grants humans “general” intelligence and certainly transformed us into a planet-changing species, it all stems from some crucial ability in System 2, the slow, conscious thinker.

Perhaps our System 3 is so much more advanced than the ant’s or the beaver’s or the chimpanzee’s because we can update it so quickly, thanks to the “like this” teaching of System 2. So even though institutions last for longer than a human lifespan, they can change within a lifespan, within a moment. The longevity of genes and the malleability of brains. Impressive system.

AI that teaches AI

To explore the space of possibilities, to learn, you need variation and selection. Natural selection is a blind selector. But selectors can be made to see so their selections are targeted. System 2 is like an enzyme that speeds up the discovery process of System 3. Recent upgrades to System 3, what we’d roughly call the scientific method, have been particularly powerful enzymes. Mathematics, code, diagrams, peer review — these are not knowledge in themselves. (The encyclopaedia can’t make anything happen.) They are highly protocolised ways of correcting errors and assigning credit, the ability we first unlocked with System 2. They help us perform many more actions than our ancestors.

AIs are currently leapfrogging System 2.⁷ Their intelligence can easily surpass our own in many ways without them ever having to think slow. I believe this means that it is entirely possible to have automata that are, in any normal sense of the term, much more intelligent than us without being conscious.

If AIs begin to act in the real world, to cooperate with each other, to make improvements to themselves that aid their survival — that’s when we’re looking at AGI and something to be frightened of.  As yet, they are best thought of as the latest extension of our own System 3.

1. This is probably less complicated than what triggered the massive changes of the last few hundred years. Like, did the scientific revolution cause the industrial revolution? But if so, what caused the scientific revolution? There are many excellent books about the scientific revolution, including convincing ones that argue it’s a misnomer or a process that started in the middle ages or even earlier.

2. I like Venkatesh Rao’s recent writing about protocols as the bones of time. Also, this general idea about our thinking being done beyond our skulls is not new. This accords with theories of knowledge by disparate thinkers including Karl Popper, Andy Clark, Alva Noë, and Ludwig Wittgenstein among others.

3. A kind of gradient descent.

4. I’m playfully suggesting that teaching is the thing that drove the evolution of consciousness. But it might be a happy byproduct of the more traditional candidates for selection pressures that produced uniquely human cognition: caring for toddlers and planning a hunt. Both these activities require a situational awareness and sustained attention that includes shared attention of at least one other person. See the whole lifetime of work by two amazing anthropologists: Sarah Blaffer Hrdy (mothering) and Michael Tomasello (shared attention). The question underneath this is, why are we conscious at all? Maybe it’s simply impossible to do complex things without having a System 2. But we already know, deep down, that this isn’t true. Robots can already do quite a lot of flexible behaviour. And so can sleepwalkers. People with NREM sleepwalking can be utterly unconscious and perform very complex behaviours, including eating, having sex, and even driving a car.

5. Don’t know about you, but I’m always pathetically courteous to LLMs. Not so with Siri. I changed Siri’s voice to male on my phone so I felt less uncomfortable yelling at it. Feminism.

6. Meta is already training robots on hundereds of hours of egocentric camera footage of people doing domestic tasks: so you wear a camera while you go about your day peeling avocados and such. Then machine learning techniques are used to recognise all the objects and actions in the videos. Then they get the robots to practise in 3D simulations (VR) before getting them out in the real world peeling avocados and whatever else we humans do. Things are about to get even weirder.

7. Although I’ve just watched, after I’d written this, a presentation by Yoshua Bengio on how LLMs can develop a System 2 to become conscious.