Anthropic calls for global pause in frontier AI development

Anthropic Calls for Global Freeze in AI Development

Anthropic, now the world’s most valuable AI startup, has called for a worldwide pause in frontier AI development, warning that humanity risks losing control of the technology as systems approach the ability to improve themselves autonomously.

The call, made in an essay published Thursday by Anthropic’s head of research Marina Favaro and president Jack Clark, comes just days after the company filed preliminary paperwork for a public listing — underscoring a tension at the heart of the company’s identity between safety advocacy and commercial ambition.

A Call to Slow Down

“Should it be feasible to effectively decelerate the evolution of this technology, thereby granting us additional time to address its profound implications, we believe that would likely be beneficial,” the executives wrote, according to The Telegraph. They likened the development of powerful AI systems to an “arms control dilemma” and warned that “recursive self-improvement” — where AI systems autonomously enhance themselves — could lead to a surge in capabilities beyond human oversight.telegraph

“We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to progress accordingly,” the company stated.yahoo

Yet Anthropic acknowledged the difficulty of coordinating such a freeze, citing the intense rivalry between the United States and China. “A significant slowdown or pause would necessitate multiple well-resourced laboratories at or near the cutting edge, across various nations, agreeing to halt under uniform conditions,” the essay noted. “Training exercises are considerably easier to disguise than missile silos.”telegraph

Safety and Commerce in Tension

The timing of the announcement raises questions about Anthropic’s strategic positioning. On June 1, the company confidentially filed a draft S-1 registration with the Securities and Exchange Commission, stepping ahead of rival OpenAI in the race to access public markets. According to Reuters, the IPO could come as early as this fall, with the company reportedly seeking a valuation near $900 billion.reuters

The company recently surpassed OpenAI as the world’s most valuable AI firm, reaching a valuation of $965 billion. Anthropic has also withheld its most advanced model, Mythos, from public release over concerns it could be exploited for catastrophic cyber warfare.youtube

A Pattern of Evolving Safety Commitments

The freeze call marks a shift from Anthropic’s earlier moves. In February, the company dropped its landmark 2023 pledge to halt development if safety measures could not keep pace with capabilities — a change that drew criticism from the AI safety community. CEO Dario Amodei has separately estimated a 25 percent chance that “things could go very, very wrong” with AI development.time

Some analysts have suggested Anthropic may be overstating AI risks to advocate for regulations that could disadvantage less safety-focused competitors. The company’s essay noted that its team is now generating eight times more code than it was between 2021 and 2025, with AI increasingly able to propose novel ideas and plan research — developments it views as steps toward self-building AI.telegraph