The conversation around AI safety hit headlines last week following a high-profile essay by Anthropic CEO Dario Amodei on September 12. Amodei made a case for tapping the brakes on unrestricted AI progress, advocating for third-party evaluators, a national regulatory framework for advanced models, and strict controls on AI chip access to China while still seeking some sort of international cooperation.
The essay raises a 1 million dollar question: If U.S. frontier labs slow down, will the rest of the world follow, especially Chinese AI labs?
Why now?
Amodei’s warnings aren’t coming out of nowhere. There are two primary reasons driving this caution.
First is the autonomous cybersecurity threat. As AI agents’ cybersecurity capabilities improve, we are seeing real-world AI-launched breaches and hacks, from OpenAI’s autonomous AI agents run during evaluations managing to escape and hack servers on Hugging Face’s infrastructure, to an AI-developed computer worm uncovering critical flaws in messaging apps like WeChat.
Second is recursive self-improvement (RSI). We are inching closer to a reality where AI begin participating in designing subsequent, more powerful generations of AI, which sparks widespread concern over systems going entirely out of control. Anthropic lately faced a major PR crisis after its researcher Jacob Coxon resigned and warned that top AI companies are racing toward dangerous superintelligence and gambling with human lives.
Dario’s essay is not the first attempt to slow down AI progress. In 2023, the Future of Life Institute (FLI) released an open letter urging all AI labs to immediately suspend training models more powerful than GPT-4 for at least six months. Despite over 30,000 signatories and massive media reports, it ended up being only a symbolic event.
This essay however surprisingly received agreement across Silicon Valley. OpenAI CEO Sam Altman said that embedding third-party evaluators is a solid step. SpaceX and Tesla founder Elon Musk echoed that oversight is long overdue. Demis Hassabis from Google DeepMind called the essay a step in the right direction. Microsoft CEO Satya Nadella emphasized that pursuing AI without keeping human benefit at the center isn’t worth doing. Tech investor Gavin Baker posted a great write-up below summarizing these industry responses
Will Chinese labs follow the suit?
Some social media users, including former CNBC anchor Deirdre Bosa, turned toward Chinese AI stars like Tangjie from Zhipu AI, Yang Zhilin from Moonshot AI, and Liang Wenfeng from DeepSeek to see how they would respond.
While they are usually vocal about model achievements and benchmarks, weighing in on a global debate about slowing down progress is a different topic. Chinese AI leadership has to be careful here.
That said, China’s domestic focus on security is tightening recently. Minister of State Security of China Chen Yixin have recently published an article about AI risks. Chen highlighted major AI threats like sensitive data leakage and AI cyber offense-defense. I imagine the government would want domestic labs to bolster cyber defense against any potential foreign threats, but it also wants to ensure local open models aren’t weaponized against the state. This creates a technical paradox: Can a model enhance its cyber-defense capabilities without also improving its capacity for cyber-attacks?
If the looming cybersecurity risks escalate, we could see tighter domestic measures return, which reminds me of the strict model registration introduced a few years ago before commercial deployment was permitted. How that applies to open models however remains a question.
Pacing the frontier would be a great initiative if competitors cooperate. Otherwise, it’s reminiscent of the classic Prisoner’s Dilemma where labs publicly promise to slow down while quietly accelerating behind closed doors.
But let’s just say U.S. frontier labs eventually agree to hold back releases pending third-party evaluations, I think open-weight model development will likely follow suit. Companies like Zhipu are already putting emphasis on cybersecurity when they release their latest models. They use a defense-in-depth safety alignment approach with three distinct layers for GLM-5.3 to manage its advanced cybersecurity capabilities. No lab wants their model driving the next major cyber hack.
However, open models would require a different kind of evaluation framework. Once weights are out in the wild, there is no way to retract them. If Anthropic points to organizations like METR as an evaluator for models, the evaluation committees of open models would need some other real industry experts, such as researchers like Nathan Lambert, at the table.
This also puts Hugging Face, the No. 1 hub for open-source model distribution, right in the center of the stage. The timing is especially tricky following Nvidia’s acquisition of Hugging Face. Nvidia thrives on rapid AI scaling to drive adoption of their chips, yet widespread rogue AI chaos is the last thing CEO Jensen Huang wants to see either.
There is also a common counter-argument: If Western labs pace themselves, won’t China simply race ahead and capture the market?
Counterintuitively I think Chinese AI labs actually benefit from the rapid breakthroughs set by U.S. frontier models. As Western labs push boundaries, it validates the tech stacks and broader market, giving Chinese companies the proof points they need to pour heavy investments.
But if the Chinese companies conclude that this initiative is largely designed to block the catch-up of followers and concentrate the power of AI into a few hands, they will jump ship without any hesitation.
Good question to ask. If US labs slow down while Chinese labs don't, the open-weight dynamic accelerates too. For Europe that cuts both ways: more capable models to run, but also a greater need for our own infrastructure and evaluation to run them safely.
Very timely post. I am equally curious to see the response from Chinese labs. However, I am skeptical of this sudden ethical stand by U.S. labs. From Anthropic's report on AI misuse to Sam Altman's podcast episode where he stated "safety concerns" as the reason behind OpenAI's hesitation to continue with their plans to go public to Dario Amodei's call for a pause that was openly supported by Musk and Altman?? Very peculiar sequence of events.