AI doom is near: for and against
Forecasts are junk but wishful thinking is rife
Last week I had a great Substack Live discussion with author and activist Sarah Wilson about AI. In an earlier post, I’d collected the best recent writing about AI causing doomsday, or at least why we should be very worried about AI ruining existing systems and leading to war or other global catastrophes. I also linked to the best recent writing I’d found that was sceptical of artificial general intelligence (AGI) or artificial superintelligence (ASI) going rogue and taking over the world.
I myself have criticised doomers for offering simplistic sci-fi narratives and overly precise numbers (“AI doom is 13.5% likely to happen”). I’ve also criticised sceptics for wishful thinking and for mainly attacking the doomers, rather than actually showing why AI doom won’t happen… and I’ve fallen prey to that myself.
I’m another researcher with an opinion. But I’m an independent voice. I’m not a tech bro, or a rationalist. I don’t believe in the technological singularity and I don’t own shares in any tech companies. My view of the world was formed from different sources to the doomers’. But the Stark Way’s ethos is to avoid wishful thinking (or “cope” or “hopium”). So when I hear people accusing others of “fear-mongering” or “alarmism”, I’m vigilant. We don’t want mindless alarmism; but what if the alarm actually needs to be sounded? It was not the strength of doomer arguments that made me take notice of the threat of prospective AGI or ASI. It was the weakness of the arguments against them that redlined my wishful thinking detectors.1
So I’d better explain my feelings about near future AI as an existential threat. In this post, I summarise the strongest points against, and then for, AI doom.2
And then next post, I’ll detail the warning signs. These are the next technological developments in AI, robotics, or nanotechnology that I’m personally looking out for, the ones that should cause us to riot in the streets or buy coffins.
Personal note. Researching this is a process of inhabiting radical uncertainty. I don’t just mean the invisible terrain of tomorrow or the far future. Day-to-day, I swing between whole arguments and whole worldviews as I interrogate sources that, on the one hand, convincingly predict we’re about to die and there’s nothing we can do about it, and on the other, those which soberly deflate the threat of sci-fi AI, to refocus attention on more homely problems like imminent global conflict, new financial crashes, or catastrophic climate change. Fun!
Philosophical note. The Stark Way is meant to be an ultra-pragmatic, hard-nosed account of all the difficult and disputed concepts tangled up with predictions about AI, such as intelligence, knowledge, agency, evolution, prediction, consciousness, values, and goals. Heavy stuff. And none of these concepts has anything like a definitive theory.3 My intellectual project, such as it is, assumes all these concepts have been undermined by the science and technology of the 20th century.4 Given we’re in the 21st and things are only getting weirder, a new way to navigate this alien terrain is overdue. That project forms the background to the points below.
Now, before the for and against, I have to make some comments about this fool’s errand of trying to see into the future.
Against forecasts
Condensing a lifetime’s worth of scholarship, I offer a couple of nice lines from the investor Morgan Housel:
The correct lesson to learn from surprises is that the world is surprising […] we should use past surprises as an admission that we have no idea what might happen next.
And:
History is often the study of surprises ironically used as a guide to the future.5
It feels like we can predict the future because we look back on the past and see patterns or narratives that explain the present. We overlook that people in the past were continually surprised. So we extrapolate what has already happened and imagine the trend line continuing, even though that didn’t work for people in the past.
Extrapolation works for some extremely regular things, like the Earth’s orbit or chemical reactions performed in a lab. Most of us want more. We want to know about the irregular and interesting things, like whether we can find love on a dating app, our career prospects, the outcome of the next election, or the future values of our assets. These are unpredictable — though there are many more people who make money offering forecasts in these areas. These facts are related.
For AI doom, we want a forecast that somehow captures the highly irregular and interactive behaviour of millions of individuals, thousands of corporations, and dozens of governments, all in response to each other and to technologies that haven’t been invented yet. No one can do it. Even a superintelligence cannot predict the behaviour of a complex system that may one day include, for example, superintelligence.6
Having said all that, let me now offer non-forecasty points against AI doom, points that are instead about what is/isn’t possible.
Points against doomers
This is an idiosyncratic list of what I find to be the most compelling problems with the fear of an intelligence explosion, or other scenario leading to an AI takeover. Each of these deserves an essay of its own, yada yada yada.
Intelligence doesn’t equal prowess. Algorithms excel in their curated environments: like playing chess and solving logistics problems. So far, they’re not general purpose problem solvers. They perform tasks that are data-heavy. They can’t dig ditches or self-repair, while humans are pretty good at this. One day, humanoid robots will be quite good at digging ditches, much better than humans. And they might even get better at self-maintenance tasks. However, this conceals a deeper problem: simply having more information, data, and the ability to process it is not the same as success and survival out there in the real world — what I’m calling prowess.7 Rocks last longer than individual organisms. Sea sponges have been around longer than brainy apes. The smartest humans don’t leave the most descendants. So it’s not clear what the relationship is between intelligence (or information processing or cognition or computation) and action. Knowing about a problem is not the same as solving it. We academics are patron saints of this problem; I know a lot but can do little.
Humans and AI working together can’t take over the world. Imagine you’re a wannabe supervillain seeking world domination. Starting today, you summon all your wiles and use the latest tools at your disposal to try to become a global dictator. This includes using AI in the most dastardly ways imaginable. No offence, but I doubt you’d would get anywhere near this goal. In short, the job is too complicated — even with the help of the latest ChatGPT — and other people would stop you.
(Note: this is an analogy for AI taking over the world, not AI killing all humans — subtle difference. Killing all humans is easier. Maybe you could even do it yourself if you were a combo of Ted Kaczynski, Osama Bin Laden, and Guy Edward Bartkus.8 You might be able to get some novel pathogen assembled by a lab, or assassinate a world leader in a way that leads to a nuclear exchange… I don’t know, I’m not gonna do your homework for you. Point is, this would lead to humans, including you, dying, not you taking over the world. If you are able to take over the world, then you presumably have kill-all-humans as an optional add-on later. Likewise, relatively simple AI might somehow hack command and control systems and launch all the nukes, kamikazeing itself along with us. Dumb move, but strictly easier than doing the leg work of first taking control of the world and then exterminating us.)
So, if you couldn’t do it with the help of AI, how is current or near-future AI gonna do it by themselves? The suggestion is that we must not be thinking about AI and intelligence in the right way. Intelligence is not prowess, etc.
Taking control is messy. Regardless of your intelligence or prowess, the world of agents is too chaotic and ever-changing for any one agent to beat the game. Here’s a big generalisation I don’t have room to unpack: the kinds of problems AI and conventional computers are good at involve using existing data to generalise the next instance of something. E.g. the next good move in chess, whether or not the next Instagram image contains nudity, what the next most likely word is in a sentence. These problems are more solvable the more data you have and the more feedback, through reinforcement learning, for example. (You might notice this is related to the point above about prediction only working for regular things.) An important assumption is that the world still behaves the way it did in your training data. More to the point, the world doesn’t change in response to you learning about it. You could train an AI to recognise different rock types: great, because rocks aren’t going change their behaviour in response to being learned about and the AI will remain useful. Not so if you want an AI to beat the stock market. Other agents — humans or algorithms — who are also playing the stock market will alter their behaviour based on your AI’s behaviour. This means you’re not in the world of data-crunching, pattern-recognition, and reinforcement learning.
You’re in the world of game theory: the study of competitive interaction between strategic players (agents). It is a constantly shifting landscape where yesterday’s winning strategy is now a loser’s strategy. There’s some advantage to having historical data, but if you’re dealing with other intelligent agents they’re likely to be coming up with new moves that haven’t been seen before. Rather than raw processing power, you need to be resilient to inevitable shocks, adaptive to new threats, and flexible enough to do whatever is necessary to survive. If this sounds like running a business or winning a war, that’s right. Those are domains where game theory is used as the imperfect but best-of-a-bad-lot way to analyse the situation. Taking over the world is the game of all games, far more complicated and with even higher stakes than losing a war or going bankrupt. To achieve global suzerainty, you need to win all the wars and control all the economies. This means beating, forever, all the humans, groups of humans, other AIs, and human-AI teams out there.
Basically, if an AI tried to take over the world people would try to stop it. Imagine the Chinese or US militaries grappling with rogue AIs within their midst. This doesn’t sound great, in any case, but it’s possible the AIs lose those battles.
Intelligence is infrastructure, not brains. Our brains are identical to what they were in the year 200,000BC. Yet we have satellites and vaccines and smartphones. Our brains run the same System 1 (fast, unconscious judgements) and System 2 (slower deliberate reasoning) — all that has changed is our System 3: the institutions, networks, and traditions we draw on, as well as the cognitive tools like writing, math, and computers. Future AI will not be more capable just by having a larger electronic brain (more data, more GPUs). Intelligence is actually about being plugged-in to the infrastructure where real knowledge is embedded, as opposed to being a single smart operator.
Taking over the world requires delayed gratification. I like playing the game Secret Hitler. For those who don’t know, it’s a board-game set in 1930s Germany, where people are dealt secret identities, either liberals or fascists. One of the fascists is Hitler himself. If you’re dealt that card, the way to win is to pretend to be liberal, to do everything a dyed-in-the-wool liberal would do, even to actively disadvantage your fascist colleagues in the short-term, so that you’re eventually trusted beyond doubt and elected chancellor late in the game, thus winning the game for the fascists — whereupon you reveal your true identity to the shock and mirth of the other players. Such a strategy requires a kind of sacrifice. You have to do something you don’t want to do now, in return for a payoff in the future.
A would-be AI Hitler probably needs to be able to delay gratification. AI sucks at this. In fact, everything sucks at this. Organisms evolved by natural selection, which can’t “look ahead” or plan. Whatever survives now, is what reproduces. Over time, generally resilient and provident organisms come to dominate because they weather the larger, less common shocks that wipeout organisms totally optimised to whatever is immediately in front of them (squirrels that cache nuts for the winter are an adorable example). Nonetheless, there is a strong bias towards the near term. You can only learn via feedback and you can’t receive feedback from the future before it happens. Humans are highly unusual in that we can plan for the future, somewhat, and sacrifice current performance for later reward. Even still, most of that is with the aid of System 3, the institutions that allow for thinking over a time span longer than even a human lifetime. AI lacks anything like our System 3. The best algorithms gobble up data and use their reward signal to speedily get the job done. While they can make a plan, they struggle to stick to it.9 Being too focused on the long-term also fails.10 Ironically, it is the unpredictability of the future that means organisms playing the long con are hit by unforeseen events before their plots can hatch. Again: forecasting sucks.
Points for doomers
Don’t worry, though, there are plenty of reasons to be shit-scared too.11 Some of these are counterpoints to the reasons against doom that I just offered.
We’ve created the System 3/infrastructure/technosphere. Most obviously, we’ve built the servers and the data centres, and we pay for the electricity and water. Beyond that, control over the world may well be aided by having a modern world that is highly networked.12 All important public infrastructure is hooked-up to the internet: energy grids, payment systems, food distribution, and so on. We’re also filling the world with humanoid robots, autonomous vehicles, and even dark factories. So if we built some ASI, it would find it much easier to exert global influence if the globe is “smaller”, in the sense of everything being connected and centrally controlled.
Incentives are in place to make this happen. If you think some horrible AI scenario is possible, then we certainly seem to be setting up a global effort to at least try to make it happen. Many of the smartest people born since the 1980s, in China, Eastern Europe, the US, etc. have been funnelled into AI or Big Tech more broadly. They are financially rewarded for accelerating the uptake by businesses, governments, and ordinary users of these systems. This means the people most informed about the risks and capabilities of AI are those who financially benefit most from its widespread use. Even more perversely, those who are most terrified about AI are also the most evangelical about its adoption, because they typically believe in the utopian upside outweighing the risk of the dystopian downside. This is why there are strange schisms in the doomer community and why they don’t all simply say, “Shut it down.” Many think the chance of achieving immortality through uploading their consciousness, or some other egomaniacal fantasy, is worth the bet of other people’s lives.13
Those closest to the technology are literally building bunkers. This doesn’t apply to all of them,14 but when one observes the tech oligarchs simultaneously putting all their chips on an AI arms race and building bunkers or pursuing other prepper activities, one starts to worry about one’s own “stockpile” of food, consisting of three tins of kidney beans and a jar of weird local honey. It’s like watching the pilots of your plane strapping on their parachutes.
There’s no physics reason why intelligence (or prowess for that matter) cannot grow exponentially. There are various limits, perhaps, on biological intelligence or the capabilities of current AI paradigms like deep learning. Beyond that, I know of no physical law prohibiting the growth of intelligence in some suitably intricate system, nor anything prohibiting the solving of physical problems given enough resources like energy and skill, intelligence and prowess. This one deserves way more unpacking. All I’ll say here is if you think what we loosely call knowledge is a non-supernatural thing, then it’s hard to see why there couldn’t be something in the future with orders of magnitude more knowledge than us. And if knowledge is ultimately about making things happen or solving problems, then it seems like it’s physically possible to create an AI that is unfathomably beyond us. This doesn’t mean it will happen. It might be extremely hard. It might take centuries. It might not happen. But the naysayers who claim that ASI can’t happen in principle are, I think, mistaken.
ASI might be better at planning than us. Even an ASI cannot forecast the future, but it might have better techniques for executing long-term plans. It might be better at acting in a seemingly selfless way now in order to be selfish at a later critical moment. If it can sacrifice now for later it can beat us at Secret Hitler.15
ASI might also be better at strategy. Situations that include the flux of other agents making decisions based on one another’s decisions are impossible to predict in advance. Still, as game theory shows, there are better or worse strategies to adopt and better meta-strategies: ways of choosing which strategies to adopt and when to switch strategies. ASI could be better and certainly faster at doing this.
Anything more powerful than us would be a threat by definition. If we continue with the assumption that an ASI would be able to do way more shit than us, then it would absolutely be an existential threat. Look past the speculation about what ethics or values we would somehow instil in an AI. Think instead of the problem of a benevolent dictator. Once you put them in charge, you live on borrowed time. Even if they really are benevolent when you do it, you no longer have any way to control them if they cease being benevolent. You also have to pray that their successor is benevolent, and their successor in turn, and so on forever. Maybe the first ASI would be “aligned” with human values to begin with. But if it changed its mind for any reason, or upgraded, or built a new ASI more powerful than itself,16 we would be living purely at its pleasure. This is why benevolent dictators are infinitely worse than clunky democracies, and why we shouldn’t voluntarily create something that can squish us: in the long run, “can” bends towards “will”.
What’s gonna happen?
One million years ago, some particularly sapient primates might have stood around saying that the so-called "cognitive niche” — the environmental niche occupied by hominins with our abilities in language, social cognition, and tool use — was a bold new way to take over the planet. But anyone relying on the past to that point as a guide, would have told them:
No way! If intelligence was such a winning move, the dinosaurs would have tried it. Besides, these new, chattering apes mainly just use it to gossip, go on stag hunts, tell stories, and make campfires. Language and social intelligence seem to be useful for narrow tasks, but let’s not over-inflate them, they’re not going to change the world!
Cut to: today. The face of the Earth is marked by the roads, lights, farms, dams, and wastelands that are the product of whatever ability we had over and above the other apes.
Now, in hindsight, we have various guesses at why and how human intelligence was so world-changing. Notably, we don’t have a settled theory of that, and it already happened. Hindsight is blurry. Foresight is blind. How then do we prepare for the unforeseeable?
I don’t know. Cue my recurrent theme: beware those who provide too specific a future — they don’t know either!17
But one thing we might do — I’m just spitballing here — is stop deliberately creating something that could kill us. This sounds luddite, or alarmist… It also sounds pretty fucking unimpeachable when you say it back to yourself.
It’s also unremarkable advice that we habitually ignore. We have an interlocking system of nuclear threats, with weapons on hair-trigger alerts: a good-to-go omnicide option that we spent decades and billions of dollars setting up. We also continue the planet-wide geo-engineering experiment to increase the greenhouse effect. We do this intentionally. It’s not easy. It takes massive effort, and resources, everyone has to pitch in. We do this — pay money to dig our own graves — because, if we fail to act collectively, we get short-term rewards for individual actions that lead to collective doom.18 Saving the world also requires delayed gratification.
It’s possible. We can both halt crises and avert them. We haven’t had a world war for a while. We didn’t develop human cloning. We avoided Y2K. We (largely) stopped enslaving and torturing people. This is part of what makes forecasting impossible. Saying something is inevitable, even when it seems to be following some clear process like failure of collective action, is a claim to infallible foresight. That’s right, even fatalism is a kind of wishful thinking, because it abnegates the work of trying to change the future. The Stark Way is rough.
You might think I’m being flippant or ironic about all this. It’s interesting. Because I’m such a nuclear weapons nut, I’ve been gazing into the abyss so long that the abyss doesn’t stare back any more, it just goes about its day and doesn’t even care if I’m watching while it’s getting changed. Even the worst AI futures are, to me, more remote and less urgent than the ready-made doomsday option of good ole, down-home nuclear war.
Regardless of the specifics of future AI, lots of people will have to cooperate and sacrifice something in order to wrest control from Big Tech.19 Whether that is because they’ve developed dangerous AI or simply continued their march towards surveillance capitalism, the cause is the same. They will continue to offer us happiness in return for our freedom and privacy. It will always seem worth it and it never will be.
I’m familiar with all the doomer arguments and load-bearing concepts and I know the stock arguments against alarmism. I have different fundamental assumptions to the doomers, so I find some of their arguments to be misguided for obscure philosophical reasons. But I find the naysayers’ arguments to be utterly invalid and driven almost entirely by wishful thinking. These people are all smarter than me, though, so I’m doubtless missing important points from both sides. I welcome corrections.
Or at least really bad, world-ruining or world-enshittifying outcomes, if not outright doom.
Evolution does but experts disagree over its implications, especially concerning teleology. Most biologists and philosophers are wishful thinkers on this point.
I notice that concepts around language, meaning, and understanding are seldom cited as barriers to AGI. They’ve been manifestly undermined by LLMs. I’d bet on agency, intelligence, and goals as next to fall. The holdouts will be consciousness and values: harder to implement in machines; equivalently: they’re not at all what we think they are.
See The Psychology of Money, ch. 12; second is this tweet.
This epigram is my own, but it’s an obvious enough point that many have made it. Fans of David Deutsch will recognise it in the dictum that to predict the growth of future knowledge would be to have that knowledge. Fans of Eliezer Yudkowsky might recognise Vingean uncertainty.
This is to distinguish it from shakier concepts like agency, will, and sentience. Prowess is simply: ability to do things.
Bartkus is an antinatalist psycho. Read this fascinating post by Katherine Dee.
Although this Twitter article by Benjamin Todd looks at how AI is increasing the length of task it can do at an exponential rate.
In the long-long-run, the universe will reward the anticipators that have the best portfolio of short-term and long-term goals.
In addition to sources I cited in my previous post, it’s worth watching an accessible but long podcast interview Yudkowsky just did.
As a corollary, consider Ben Goertzel’s point that we should develop AGI ASAP precisely so there’s less infrastructure in place for it to use. If we make it later when we’ve also got crazy nanotech, full-blown internet of things, etc., we’re sitting ducks.
This accelerationist, and essentially fascistic, impulse gets some treatment in Jesse Armstrong’s new film, Mountainhead.
Others prep for AI to be a worthy successor to humanity. A minority think there’s no risk.
Some evidence of that already (Twitter thread).
If you think an ASI would have to be mad to build something more powerful than itself, then you’ve grasped the core of this issue.
Everyone should read Nassim Nicholas Taleb’s The Black Swan at some point.
In game theory, this trap is coordination failure. AI doomers personify it in the figure of Moloch from the Hebrew bible.
Gen Z and alpha are always pitied for occupying the charnel house of the “meaning crisis”, “polycrisis”, “metacrisis”, etc. Yet they inherit the most gravid mission of all.