What if the people building AI are also the ones most afraid of it?

That question has resurfaced after Jacob Coxon, a researcher who has worked at both Anthropic and OpenAI, left Anthropic and publicly warned about the race to develop increasingly powerful AI systems. In a post announcing his resignation, Coxon accused the companies of “racing straight to self-improving superintelligence” and warned that the competition could put humanity at risk. His post reportedly reached more than 100 million views. 

Coxon isn’t alone. Evan Hubinger, Anthropic’s alignment science lead, has also publicly backed his concerns. Hubinger has said he believes there is a greater than 10% chance that AI could kill all humans within the next decade.

Terrifying? Yes. But the fear isn’t really about chatbots suddenly becoming evil or robots taking over the streets. It’s about self-improving AI.

What happens when AI gets smarter?

Today’s AI systems are still built and trained by humans. But researchers are increasingly thinking about what happens if AI becomes capable of improving its own abilities, particularly in areas like coding and AI development. In theory, a smarter AI could help create an even smarter one, which could then help build something more advanced.

That’s where the idea of superintelligence comes in: AI that could eventually outperform humans across a huge range of intellectual tasks.The worry is that development could eventually move faster than humans can understand or control.

The race is part of the problem

OpenAI, Anthropic, Google and other major companies are competing to build increasingly powerful AI. With billions of dollars and enormous strategic advantages at stake, slowing down isn’t easy. If one company pauses to focus on safety while another keeps moving, the company that slows down risks falling behind.

That’s what makes Coxon’s departure striking. Anthropic has built much of its reputation around AI safety. If researchers inside the company are questioning whether the race itself has become dangerous, it’s fair to ask how prepared we really are.

Is AI actually going to wipe us out?

There’s no way to know. AI extinction is still a prediction, not a fact, and researchers disagree widely about how likely the worst-case scenarios are.But AI is already becoming capable of things that once seemed impossible. Systems can write code, conduct research and carry out increasingly complex tasks. Anthropic has also reported cases of its AI being misused for cyberattacks and biological research.

That doesn’t mean an AI apocalypse is coming. It does mean the question of how much control humans should have over increasingly powerful systems is becoming harder to ignore.

For now, AI isn’t taking over the world.

But the people building it are already asking a very different question:

What happens when we’re no longer the smartest thing in the room?

Photos: Jacob Coxon, Evan Hubinger (X)