AI’s biggest labs are suddenly calling for a slower pace of development. Is it because AI risks are catching up with the technology, or because the economics of the race are getting harder to justify?
For years, the defining characteristic of the artificial intelligence race has been speed. Rinse and repeat, with litte regard for the unknown unknowns.
The average “p/doom” (probability of AI eventually going catastrophically wrong) among AI researchers was estimated to be between 15% and 20% in 2024.
A year later, Anthropic CEO Dario Amodei upped the stakes, saying he believed there was a 25% chance “that things go really, really badly.” Even as far back as 2014, xAI chief Elon Musk warned : And OpenAI CEO Sam Altman acknowledged in 2015 that AI would “probably, most likely, sort of lead to the end of the world,” but that, in the meantime, there would be “great companies created.” With better odds of cheating death playing Russian Roulette, anyone with even a fleeting interest in the topic has had an uncomfortable feeling in the pit of their stomach for a while now.
Why are the companies driving the AI race suddenly asking to slam on the brakes? That’s what happened over the weekend, when Amodei published an essay calling for frontier AI development to be “paced,” warning that AI capabilities are advancing faster than the industry’s ability to understand and control them, and that the internet could get taken over by AI swarms within six to 12 months.
Related: Nvidia buys Hugging Face for $12.9B in push into AI software Altman broadly agreed, saying the world deserves the “confidence” that the companies developing ever-more capable AI will act “responsibly,” and Musk backed Amodei’s proposal, simply commenting: The concern is not confined to the companies building the technology either.