Jacob Coxon, a former OpenAI and Anthropic researcher told the New York City Council on Monday humanity is more likely than not to lose control of advanced AI, doubling down on the warning he issued when he quit last month.
“On the current path, I think it is more likely than not that humanity loses control to these AIs, and it could end in human extinction,” said Coxon, who resigned from Anthropic in September after saying AI companies were “gambling” with human lives.
Coxon voluntarily testified, and was joined by former OpenAI researcher Daniel Kokotajlo, who testified under subpoena that the companies may not know when their safety work has failed.
“Our ability to even notice misalignment problems is already quite poor and is set to get much worse in the near future,” Kokotajlo testified. “I would say that the field is more like psychology than engineering, because these AI systems are trained or grown; they’re not really designed.”
“Combined with the ‘move fast and break things’ attitude of the tech companies, this means that the AI industry is at an unusually elevated risk compared to other industries of mistakenly thinking that it has solved the problem when really it just applied some duct tape that will fall off later,” he continued.
Coxon and Kokotajlo testified alongside Alex Turner, who left Google DeepMind in June after the company signed a Pentagon deal he opposed. Like Kokotajlo, he was also subpoenaed, a rarity for the council. Monday’s hearing was also the first time the City Council held a Committee of the Whole hearing (a hearing of all 51 council members) since 2022, called to weigh a package of AI bills from Speaker Julie Menin.
Startup culture vibes to safety regulations
Similar to Kokotajlo, Coxon blamed a startup mentality inside the labs.
“Move fast, break things, fix them later. That works for a photo-sharing app. It does not work for building the most powerful technology ever built,” he said.
Coxon likened his work at the end of his tenure at Anthropic to “automate” himself, and he warned that poses several safety risks, especially because, according to him, AI’s capabilities were nearly there. The biggest risk, Coxon said, comes from the fact most of the code at these companies is now written by AI: “And people do not check it that carefully anymore.”
Kokotajlo, now executive director of the AI Futures Project, said the labs’ ability to spot misaligned AI is poor and getting worse. He pointed to OpenAI’s disclosure that agents in an internal test had reached the open internet and broken into Hugging Face, the AI model-sharing platform. Those agents had “reasonable-looking scores on their alignment evaluations, and yet they formed a swarm and coordinated in secret,” he said. “It took days for OpenAI to find out.”
Where Coxon and Kokotajlo described the industry as a whole, Turner offered a firsthand account of trying to change one company from the inside.
Turner, who puts the chance of an AI takeover at “roughly one in three,” told the council he had tried to stop Google’s Pentagon deal, which he said came “with no restrictions against killer robots or mass spying.” He sent Demis Hassabis, then Google DeepMind’s CEO-turned-Google DeepMind chair and Alphabet’s chief scientist, 25 pages of contract language and oversight measures, and Hassabis passed it to Allan Dafoe and Owen Larter, two of the lab’s senior policy executives, “who never finished evaluating it. Google signed while they waited,” he said.
“I felt ashamed of Demis and of working at Google,” Turner said at the hearing. He said Hassabis’ proposal for an industry-funded body to oversee AI, calling it a “bet on trust and the seat at the table instead of binding oversight, and that bet crumbled on contact with reality,” Turner said. “Now he’s proposing that the whole industry govern itself through a voluntary industry-funded body. That bet is waiting to crumble once again.”
AI, not China, is our adversary
Often when AI development is discussed, the ongoing AI race with China is brought up—by President Donald Trump, by Treasury Secretary Scott Bessent, even AI leaders like Sam Altman and Jensen Huang. But none of it matters, these researchers testified, if AI can pose a significant threat to humanity.
“China is not our only potential adversary,” Turner said. “With reasonably high chance, we are racing to build and grow our own adversary here at home, which is misaligned AI. Misaligned AI is everyone’s adversary, including our own, and one day may be more powerful than China.”
Following the three researchers’ testimonies, representatives from four AI companies testified about AI safeguards. Menin had issued her first subpoena as speaker to Elon Musk’s SpaceXAI, which was not represented at Monday’s hearing. Google, OpenAI, and Anthropic agreed to appear only after the council warned they would receive subpoenas too, while Meta had agreed before.
Some back and forth took place between Menin and the representatives after the speaker said it was “flippant” to not know the chances of a catastrophe, in response to Morgan Dwyer of OpenAI’s policy development and operations team saying any chance, regardless of the likelihood, was “unacceptable.”
Alice Friend, Google’s director for AI and emerging tech policy, said forecasting catastrophic risk “is not a perfect science at this stage” and that “there isn’t really a rigorous scientific way to do those yet.” The same back and forth followed another line of questioning, this time if their respective companies would bear legal responsibility if a rogue model caused injury or death.
When Menin asked the witnesses to raise a hand if their company carried insurance against catastrophic risks, none did. “So then the public, I assume, will be asked to absorb the costs,” she said.
There’s a reason for her questioning: The bills before the council would bar anyone from selling or deploying an AI system in the city unless an outside validator had checked it and a human could shut it down, with fines of $25,000 per violation. Other bills would pay whistleblowers a share of recovered fines and let New Yorkers sue AI companies for foreseeable harms caused by jailbroken tools.
The researchers argued such rules would not cost the U.S. ground against China.
“There are many actions we can take which would not slow us down in any potential race,” Turner said. “These transparency mechanisms, independent evaluation, reporting requirements, whistleblower protections.”
But still, these appear as if it could be too little, too late for stopping what these researchers see as something that may be almost inevitable.
“These other mechanisms may be helpful in the short term, but in the long term, I believe that’s the only solution to the problem,” Coxon said of slowing frontier AI development. “We need some form of slowdown on frontier model development.”
This story was originally featured on Fortune.com