Should engineers quit AI companies?

Yes, obviously. But I probably wouldn’t.

Image generate by me with Midjourney.

Welcome new subscribers, you’re all champions. A few people have even pledged money(!) and I’ll soon be generating extra content for paid subscribers and turning payments on. My main weekly post will always remain free.

---

The idea that concerned engineers working for AI companies should quit their jobs in ethical protest sounds shrill and alarmist. But it’s worth delving into why we might judge it that way.

Some engineers, by their own estimation, think there is a 5%, 10%, 50% chance that the current AI takeoff, spurred by companies they work for, will end humanity in the next couple of decades.¹

Please indulge me for a moment by putting to one side whatever you personally think the odds are. Even if you think there’s no chance of it happening, it’s fascinating to see how people behave given their own perception of catastrophic risk.

Imagine you worked for an organisation in a well paid job that was stimulating and high status, but the product or service you make might be dangerous. There’s a 5% chance that in the next decade or two, it will kill one of your family members. Horrible, I know. But think about it for one second. It won’t kill anyone else, just one person very close to you. Would you quit that job? Would you blow the whistle on this company? Would you raise public awareness of the threat and lobby for this thing to be shut down? I hope so. 

Returning to employees of AI companies, if they are halfway accurate in their assessment of the threat, then they are working for a company that has a 5% chance (or by many estimates, greater) of killing their own children.

I know it’s tasteless to phrase it like this. But it is also accurate because their children are a subset of humanity and they think there is a 5% chance of their product killing humanity.² 

And yet, so far as I know, none of them have quit.

What does this tell us?

I personally don’t think there is a straight line between current technology, such as that of ChatGPT, and some AI apocalypse scenario. (Although I could be wrong.) But, observing the behaviour of those who do, I see more gloomy indications that people don’t deal with risk in a way that is sane given our modern world. And this worries me about how we’ll act when the threat is more imminent.

In a word, people don’t panic enough. I know this is uncool. We’re meant to scoff at people who say the sky is falling, laugh at conspiracy theorists, roll our eyes at millenarians. But people panic way more about stuff like trans kids and trans fats than about nuclear weapons or climate change. Even the experts whose professional lives attest to the severity of those threats often lack the sort of urgency and agitation one might expect from people who fear for their grandchildren’s lives. 

I suspect at least some of the engineers who say there is a 10% chance that what they’re doing will lead to an AI apocalypse, are sincere and well calibrated.³ These engineers are turning up to work each day and enjoying their shift at the kill-my-own-family factory, while hoping it is only 10% effective. 

I don’t mean to be rude, but anyone who genuinely believes there’s a 10% chance their job could help kill humanity and continues to work one minute more on it, is, morally speaking and according to their own worldview, a bag of shit. Right? How can they not be?⁴ At OpenAI, for instance, their own CEO says that what they’re trying to build poses an existential risk to humanity. 

This is like religious people who profess a literal belief in hell but who do nothing to save their loved ones from this ultimate and eternal torture, beyond some light scolding about sin. Either they don’t actually believe in hell (it’s mainly this), or they’re just not focused on it because it’s a remote as yet un-actualised threat (little bit of this). 

I don’t expect even the sincere engineers to actually quit their jobs. This makes them both insane and typical. 

The only prominent example of a quit-in-sort-of-protest we have so far is from Google. Even then, the high profile researcher, a self-professed “socialist”, worked there for years, did very well out of it financially, still owns stock in other AI startups, and was of retirement age anyway. Geoffrey Minton hasn’t actually criticised Google or any other company, but he has said the existential risk is real and he wants to be unfettered to talk about it.

I’m poking fun at him because I don’t expect a brilliant researcher who pushed neural networks when others baulked will necessarily be an ethical genius or have a coherent political ideology. But I like his candour. Minton said part of the reason he kept working on it was simply because it’s so devilishly interesting, even if dangerous.⁵

That’s how hard it is to de-align people’s personal interests from those warped incentive structures of the system — what people in certain circles increasingly call Moloch.⁶

The main takeaway, I think, is that we don’t act appropriately towards unprecedented risks. Even when we earnestly and rationally believe something might happen, if it has never happened before, we systematically underestimate it. Not that we numerically underestimate it. Rather, our actions don’t reflect that estimate.⁷ 

Quitting is (just) possible

During the Manhattan Project, it started to leak out, internally, that Germany had abandoned its effort to make an atomic bomb. This made it clear that the US still wanted the bomb for other purposes, like annihilating two Japanese cities to end the war in the Pacific. Los Alamos was full of scientific heroes like John von Neumann, Robert Oppenheimer, and Richard Feynman. How many of these legends, many of whom would go on to denounce the bomb after it was used to immolate thousands of Japanese children, resigned in protest or even quietly requested reassignment? Just one. Joseph Rotblat, who asked if he might be allowed to leave the project on grounds of conscience. With fairly standard odiousness, a dossier full of ginned up charges of communist sympathies was prepared and he was threatened.⁸ 

But, let’s not worry about historical analogies too much. As I argued a few weeks ago, history provides examples of just about everything but mainly of people being surprised. Only in hindsight do we see a pattern or narrative. Hence, AGI, if it is invented, will be unique in important ways and groping for historical analogies is forlorn. But we can also hope for new behaviours from people working in AI. 

At least they openly discuss risk

I’m encouraged by the diversity of opinion within the AI vanguard: OpenAI, Anthropic, Meta, Google, Microsoft, Hugging Face, Nvidia. Seems like people within these organisations are allowed to raise concerns about AI and these companies are at least humouring them and often spending money on safety teams, etc. 

Of course, my own number 1 rule is: no wishful thinking. It doesn’t strike me as a febrile conspiracy theory to think in the future, the Big Tech giants will increase their enmeshment with the NSA or other agencies. And given that, I wouldn’t expect them to behave like boy scouts towards whistle-blowers. 

The other factor is that many people working in AI see it the way Homer Simpson sees alcohol: the cause of, and solution to, all of life’s problems. While there’s a 10% chance it might go bad, that means it’s 90% likely to go well — and going well with AGI apparently means inaugurating a utopia. So depending on one’s weightings, the expected return from this 9:1 situation is very great indeed and worth the risk that their children will die — at the hands of something they worked hard every day to build.

At least these folks are honest. In my experience, people working in unethical industries usually have a series of lame rationalisations for their choice, loath to admit their industry is ruining the world. Perhaps AI’s relative openness betokens a greater opportunity for moral action.

It takes a lot of courage and maybe some amount of foolishness to quit a job over any principle. Something as abstract as a maybe-future-AGI-scenario is harder to sell to your boss, your peers, yourself, and certainly to any dependents you might have (the same ones, ironically, who will be destroyed by the very job that brings home their bacon). But it is definitely easier for affluent people. 

Would I quit in their position? And risk present and future income, looking stupid, being wrong, missing out on an intellectual adventure like no other? Hell yeah. I’m a guy writing a post impugning other people’s virtue, now that’s moral bravery.

Seriously, I’m not sure that I would! That’s because I don’t think the existential risk of current technology is imminent.

But if I did find myself somehow working on such a project I do hope I would be a whistle-blower, a Daniel Ellsberg. The trick is to be genuinely comfortable with going to prison, or becoming destitute. For such huge stakes, working on nuclear weapons or, in the future, on some near-AGI that might pose a truly catastrophic risk, a person should be willing to lose their job and their future. Sacrificing one’s freedom or one’s life to mitigate the risk of apocalypse, would be truly effective altruism.

1. A nice breakdown of some recent collections of predictions here.

2. Naturally, though somewhat perversely, killing one child sounds worse than killing “humanity” which is too abstract. Recall, “The death of one man is a tragedy the death of a million is a statistic.” (This was Kurt Tucholsky’s line, not Stalin’s, apparently.) We’re more likely to donate to a cause with a picture of one starving kid than a picture of 100 starving kids. (Although this is one of those findings that has “failure to replicate” written all over it.)

3. As in, they have actually thought about the problem and 10% is not some meaningless figure, but represents what they genuinely think the odds are, just like they’d understand betting on a 10-sided die. People can do calibration training. I did, it was fun!

4. This is weird because I don’t think they need to quit. I’m some kind of consequentialist, so I don’t think there’s much of a problem with someone not living up to their own ethics if there’s no actual downside for the world. But, as I say in the main text, this is still dispiriting because it signals that even well-informed people don’t act to obviate existential risks (albeit this one is a false alarm, from my POV).

5. Annoyingly I can’t find the exact quote. I read it in several articles that appeared after he announced he was leaving Google but can find no trace now. I’d love if someone could point me to it.

6. Moloch = a personification of the impersonal forces that incentivise individuals to pursue behaviours that make sense for them in isolation but which add up to a terrible shared outcome for everyone. Basically, coordination failures. This usage originates with Scott Alexander’s (aka Slate Star Codoxxed) influential post on the topic.

7. I could vomit up an evolutionary rationale for this. Our nervous systems respond to feedback, not hypotheticals. So we’re good at being once bitten twice shy. But even when we know, intellectually, that something is dangerous, we seem not to accept it viscerally. Nassim Nicholas Taleb would simply point out that we are crap at dealing with high impact, low likelihood events and indeed unprecedented events: black swans.

8. I note, with disappointment, that Christopher Nolan’s upcoming Oppenheimer appears not to have anyone credited in the role of Rotblat. Shame they couldn’t fit his story in. But overall, I’m hopeful the movie will generate some renewed interest in nuclear disarmament.