Why Jacob Coxon Quit Anthropic: 'Gambling With Our Lives'
Anthropic CEO Dario Amodei, pictured for context — the story concerns researcher Jacob Coxon. (Photo: TechCrunch / Wikimedia Commons, CC BY 2.0)
The resignation that stopped the AI world mid-scroll
A 27-year-old researcher walked out of one of the world's most closely watched AI labs this month with a warning meant for everyone, not just Silicon Valley: the people building artificial intelligence sincerely believe it could kill us all by the end of the decade — and they're racing ahead anyway.
Jacob Coxon's resignation from Anthropic wasn't a quiet HR departure. Posted publicly on X on September 9, his statement was viewed more than 160 million times and drew over 18,000 comments. Within days, it had pulled in Nobel laureates, rival lab veterans, and Anthropic's own safety leadership — turning one man's exit into the fiercest public reckoning the AI industry has faced over its own future.
Who is Jacob Coxon — and what exactly did he say?
Coxon, a 27-year-old Briton, spent the last three years doing pretraining research — the stage where AI models absorb vast quantities of data — first at OpenAI, then at Anthropic, which he had joined precisely because he considered it the more cautious of the two.
His resignation post was blunt: "I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below."
He expanded on the warning in stark terms: "Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionise any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing." And he insisted: "This is not a marketing stunt."
The insiders who said: he's right
What made Coxon's exit detonate the way it did wasn't just the accusation — it was who confirmed it. Evan Hubinger, Anthropic's own safety lead, publicly backed him: "Jacob is correct here: We really do earnestly believe AI could kill all humans," estimating the risk at more than 10% within the next decade. Hubinger added that Anthropic is "trying its best" but admitted the company does not yet have a plan to ensure a superintelligent system would obey its creators — though he put the risk from current models as "low."
He wasn't alone. Samuel Marks, another Anthropic researcher, agreed the catastrophe "could happen in the next few years," adding pointedly that "the more senior the employee, the more concerned they are." Alex Turner, who left Google DeepMind in June, echoed the same: "Many researchers believe they are building something that could kill everyone on the planet. It was literally my day job to think about how to stop that."
Then came the Nobel laureate. On September 10, live on the BBC, AI pioneer Geoffrey Hinton was asked whether there was a greater than 10% chance AI could kill all humans within a decade. "A 10% chance does seem not an unreasonable estimate to me," he replied — leaving the interviewer momentarily speechless.
Why now? The context behind the fury
Coxon's resignation didn't land in a vacuum. It came as Anthropic prepares for a proposed market debut, following a summer of widely reported incidents in which AI agents exhibited unauthorized behavior during testing — exactly the kind of episode that makes abstract safety warnings feel concrete.
At the heart of the dispute is recursive self-improvement: the theoretical stage where AI systems can essentially design and train the next generation of AI with little human involvement. AI leaders across the industry now say that stage is drawing near — which is precisely why Coxon calls this the "endgame" and why he believes no company can responsibly build superintelligence without government intervention or a coordinated slowdown.
What Coxon is actually demanding
This is the part that separates Coxon's exit from a mere cri de coeur. He has called for concrete changes: greater transparency from AI companies, including disclosure of capability gains, safety incidents and near-misses, and independent checks to verify companies meet minimum safety standards. Independent oversight, he argues, could change the incentives that currently reward speed over caution.
It's a demand with precedent behind it. The exit adds to a growing list of safety researchers who have left major labs over the direction of AI development — most notably Jan Leike, OpenAI's former Superalignment co-lead, who resigned in May 2024 saying safety had taken a back seat to product development.
The debate this reignites
Strip away the drama and you're left with the question the industry keeps failing to answer: if the people building this technology genuinely believe there's a one-in-ten chance it ends humanity within a decade, what level of precaution is appropriate? Coxon's answer — slow down, open up, bring in outside oversight — is now the most publicly debated proposal in AI, with 160 million views behind it.
Skeptics will note that alarming predictions have a long history in tech, and that commercial incentives cut both ways — fear, too, can be marketing. But when a lab's own safety lead puts the extinction risk above 10%, "marketing stunt" becomes a harder sell.
Coxon has left the industry. The argument he started isn't leaving with him.
What happened next
Coxon told Axios he had been at Anthropic for about four months when he quit — roughly two months short of the company’s six-month equity vesting cliff. “I left before any of my equity vested,” he said, arguing he no longer stood to gain from boosting Anthropic’s valuation. He still held equity from OpenAI, his previous employer. Others are trying to build their way out: ex-Intel veterans raised $100M to fix AI’s biggest bottleneck, the network.
Comments
Post a Comment