San Francisco AI heavyweight Anthropic is raising its hand and asking the world to slow down, warning that advanced systems may soon be able to design and improve themselves in ways that edge humans out of the driver’s seat. The call, laid out in a detailed research post and echoed in media interviews, has quickly reignited local debates over safety, regulation and what a global slowdown might mean for Bay Area jobs, investment and downtown office space.
Anthropic's report: 'When AI builds itself'
In a report titled "When AI builds itself," The Anthropic Institute describes technical trends suggesting AI is already automating parts of its own development, and warns that full recursive self-improvement also might increase the risks of humans losing control over AI systems. The piece, co-authored by Marina Favaro and Jack Clark, uses internal metrics and public benchmarks to argue that the timeline for such a future could be closer than many people assume. The report calls for tools and processes that would let developers slow or pause frontier work so researchers and policymakers have a chance to keep up.
Co-founder: Build a 'brake pedal'
Anthropic co-founder Jack Clark has been taking that message on the road, saying the industry has a gas pedal but no brake. As reported by ABC7, Clark told Anderson Cooper that rapidly scaling research teams and automated development pipelines are making it tougher to keep human oversight firmly in the loop. His comments reinforce Anthropic's argument that the industry should explore verifiable ways to slow frontier development if clear warning signs start to appear.
Inside Anthropic's numbers
According to The Anthropic Institute, more than 80% of the code merged into Anthropic's production codebase was authored by its Claude models as of May 2026, while output per engineer has surged in recent quarters. The report lays out metrics, including model success rates on long tasks, to show how AI agents are taking on larger engineering and research roles. Those trends, the company warns, create a feedback loop that could compress timelines for any future system that helps build its own successors.
Why a pause would be hard
Anthropic also concedes that any meaningful pause would require verifiable, multinational cooperation, a high bar given how easy it is to conceal training runs and how intense the commercial and geopolitical pressure is to keep pushing forward. As reported by AFP via TheJournal, critics argue that calls to slow development can look like competitive maneuvering that risks handing an edge to rivals. Anthropic says it plans to bring together policymakers, scientists, civil-society groups and other labs in the coming months to study what a credible "brake" would look like and how it could realistically be enforced.
Local stakes: IPO, jobs and office leases
The warning lands just as Anthropic edges toward public markets; ABC7 reports that the company has filed confidentially for an IPO that could raise tens of billions of dollars for data centers and compute. At the same time, Anthropic's rapid downtown expansion has been traced through its growing real-estate footprint in San Francisco, including large commitments at Foundry Square. Put together, those local moves turn the abstract safety debate into a concrete neighborhood question about how fast growth should proceed and on whose terms.
Whether the industry can build a verifiable "brake pedal" remains very much unresolved, but Anthropic's push has already yanked the conversation out of niche technical forums and into weekend television segments and city planning talks. San Francisco, home to the company and to much of the AI talent and infrastructure it relies on, will be watching closely as the fight over how hard to hit the brakes plays out.
