The most unsettling warnings about artificial intelligence may not come from critics outside the industry, but from the people who have spent years building the systems themselves.
On September 9, 2026, AI researcher Jacob Coxon announced his resignation from Anthropic after three years working in pretraining research at both Anthropic and OpenAI. He said he was leaving because he believed the companies were pursuing increasingly powerful, potentially self-improving AI systems without adequate safeguards. His warning quickly became a major subject of discussion across the technology industry. (Axios)
Coxon's argument is particularly striking because he does not describe AI development as a distant theoretical problem. He describes an industry already approaching a point where systems could become capable of operating with increasing autonomy, exploiting vulnerabilities, acquiring resources and potentially improving their own capabilities. He argues that the competitive race between laboratories creates a dangerous incentive: even if researchers recognise the risks, each company may feel compelled to continue because it fears that someone else will move ahead first.
That creates a troubling paradox. The more seriously companies take the possibility of extremely powerful AI, the stronger the argument becomes for caution. Yet the same competitive pressures that make caution necessary can make it difficult for individual companies to slow down.
Coxon's resignation therefore raises a question much larger than whether one researcher's predictions are correct. What happens when the commercial and geopolitical incentives surrounding a technology begin moving faster than our ability to establish rules for controlling it?
There is legitimate disagreement over how probable the most catastrophic scenarios are. Some researchers and commentators regard extinction-level predictions as highly speculative, while others inside the AI industry argue that the risks deserve extraordinary precaution. What is harder to dismiss is the governance problem itself. AI capabilities are advancing rapidly, while governments, international institutions and even the companies developing them are still debating what meaningful oversight should look like. (AP News)
Perhaps the most important lesson from Coxon's decision is therefore not that humanity should fear AI, nor that it should abandon its development. It is that building something powerful does not automatically mean we understand how to control it.
The question now is whether humanity will develop the institutions, agreements and safeguards necessary to govern increasingly powerful AI before competitive pressures make those decisions for us.