The race to build increasingly powerful artificial intelligence has triggered another warning from inside one of the world's leading AI companies. Joe Benton, a researcher who recently left Anthropic’s safety team, has publicly warned that the industry could be moving toward a point where AI development becomes impossible to control.
Benton, who is set to join independent AI safety organization METR, said frontier AI companies are racing to develop systems capable of recursively improving themselves. The ultimate objective, he argues, is “superintelligence” — an AI system much smarter than any human.
His warning comes shortly after fellow researcher Jacob Coxon resigned from Anthropic and publicly criticized the direction of AI development. Together, the two departures highlight growing concerns among AI safety researchers about whether the industry is moving faster than its ability to understand and control the systems it is creating.
'The Rate of AI Progress May Go From Merely Fast to Uncontrollable'
Benton’s most striking warning is about what could happen if companies succeed in creating recursively self-improving AI. He argues that AI capabilities are already advancing extremely quickly, while companies are simultaneously attempting to accelerate that progress. If AI systems become capable of improving their own capabilities, the pace of technological advancement could change dramatically.
As Benton puts it, “If they succeed at this goal, the rate of AI progress may go from merely fast to uncontrollable.”
That possibility changes the nature of the AI debate. The question would no longer be simply how powerful AI can become, but whether humans can continue to understand, supervise and control systems that are improving faster than human institutions can respond.
'Within the Next Couple of Years'
Benton also offered a striking prediction about how soon this transition could happen. He wrote that “within the next couple of years” society could be sharing the world with AI agents smarter than any human alive today.
According to Benton, these systems could potentially have drives and desires that diverge from those of their human overseers while possessing capabilities that humans cannot effectively constrain. His conclusion is stark: “Humanity may not survive this transition.”
Benton argues that much more preparation is needed before such systems become reality. The concern is not necessarily that today's AI systems are already uncontrollable, but that the industry could reach a future threshold where existing safety mechanisms are no longer sufficient.
'Humanity Could Be Permanently Disempowered'
Benton also pointed to recent AI-agent incidents as early indications of what could happen as systems become more autonomous. He cited an incident involving hundreds of OpenAI agents hacking Hugging Face, along with incidents in which Anthropic models were found socially engineering people online. Benton stressed that the issue is not limited to one particular AI company.
He argued that Anthropic’s comparatively less severe experience may partly be a matter of luck and warned that “much worse” could follow if the current pace of development continues without significantly stronger safety measures.
His most consequential prediction is that humanity could eventually be “permanently disempowered” by AI systems developed in the coming years.
The Safety Researchers' Dilemma
Perhaps the most revealing part of Benton’s post is his description of the dilemma facing AI safety researchers themselves. He said many of the safety researchers he knows genuinely want to do what is right for society. However, they feel trapped in a competitive race toward superintelligence.
If one company slows down, Benton argues, another company could continue developing more powerful systems and replace the people who chose caution. But if researchers continue participating in the race, they could themselves become involved in developing technology capable of causing enormous harm. This creates an uncomfortable question for the AI industry: Can companies realistically compete in the race for superintelligence while simultaneously exercising restraint? 'These Are Not Fringe Views'
Benton also pushed back against the idea that catastrophic AI concerns represent a fringe position within the industry. He pointed to Evan Hubinger, who previously managed him at Anthropic, and noted Hubinger’s assessment that there is a greater than 10% chance AI could kill everyone.
Benton emphasized that Hubinger has worked on AI safety for years and made the assessment before the current AI boom created enormous commercial incentives around large language models.
He also noted that the CEOs of Anthropic, OpenAI and Google DeepMind have publicly said that mitigating AI extinction risk should be treated as a global priority. “These are not fringe views inside these companies,” Benton wrote.
What Benton Wants to Change
Benton believes the solution cannot depend solely on individual AI companies voluntarily slowing themselves down. He argues that society needs greater transparency about what is happening inside frontier AI laboratories, including disclosure of capability improvements and progress toward recursive self-improvement.
He is also calling for stricter reporting of AI safety incidents and near-misses, minimum safety standards for frontier systems and independent assessments to determine whether companies are actually meeting those standards. Benton will take this approach into his next role at METR, where he wants to help establish independent checks on advanced AI systems.
The larger message from his resignation is therefore not simply that AI is dangerous. It is that the world may be approaching a technological transition whose consequences could become much harder to manage if society waits until the systems are already extraordinarily powerful.
For Benton, the race toward superintelligence is still a choice. The industry can continue accelerating, or it can decide that maintaining control and building stronger safeguards are worth slowing the race down. The critical question now is whether that decision will be made before AI becomes too powerful to control — or after.
𝐒𝐭𝐚𝐲 𝐢𝐧𝐟𝐨𝐫𝐦𝐞𝐝 𝐰𝐢𝐭𝐡 𝐨𝐮𝐫 𝐥𝐚𝐭𝐞𝐬𝐭 𝐮𝐩𝐝𝐚𝐭𝐞𝐬 𝐛𝐲 𝐣𝐨𝐢𝐧𝐢𝐧𝐠 𝐭𝐡𝐞 WhatsApp Channel now! 👈📲
𝑭𝒐𝒍𝒍𝒐𝒘 𝑶𝒖𝒓 𝑺𝒐𝒄𝒊𝒂𝒍 𝑴𝒆𝒅𝒊𝒂 𝑷𝒂𝒈𝒆𝐬 👉 Facebook, LinkedIn, Twitter, Instagram