Anthropic’s AI safety bill sets a precise trigger: a model could be blocked from release only if there is a “foreseeable, imminent risk” of more than 50 deaths or more than $1 billion in property damage. Even then, no federal agency could act unilaterally, it would require a judge’s order. Politico reported the draft legislation; an Anthropic spokesperson confirmed it to the outlet.
How does Anthropic’s AI safety bill define catastrophic risk?
The draft bill identifies two numerical thresholds for “catastrophic risk,” according to Politico’s reporting as covered by TechCrunch. A model could be blocked if the foreseeable harm involves more than 50 deaths or more than $1 billion in property damage, and only if that risk stems from one of three specific causes: the creation or release of a biological weapon, the model evading its developer’s controls, or a cyberattack.
- More than 50 people at risk of dying, or
- More than $1 billion in property damage
- Risk must originate from a biological weapon, a model evading controls, or a cyberattack
These thresholds are similar to those in California’s Senate Bill 53, recently passed into law, which similarly defines such scenarios as “catastrophic risk,” per TechCrunch.
An Anthropic spokesperson confirmed the draft to Politico, saying the bill “represents the proposals outlined in Anthropic’s Advanced AI Framework, which we have been advocating for with federal and state lawmakers since June.”
Who evaluates the risk, and who pays them?
Anthropic’s bill would create a system of “independent evaluators” registered with the U.S. Commerce Department. These third-party assessors could be paid directly by the AI labs whose models they evaluate, as long as “payment is not contingent on the evaluations’ results,” Politico writes. TechCrunch notes it is not clear how this structure would prevent payments from influencing results.
Once evaluators flag an imminent catastrophic risk, a strict timeline kicks in:
- Evaluators have 72 hours to notify the Commerce Department.
- The Commerce Department must then petition a Washington, D.C., judge.
- Only a judge can issue a halt to the model’s release.
The bill would expressly forbid any federal agency from halting a model’s distribution on its own discretion, per Politico as reported by TechCrunch.
Where does this sit relative to the rest of the industry?
Anthropic’s draft appears to go slightly further than some industry proposals. Nvidia CEO Jensen Huang recently argued the AI industry did not need any new laws at all, according to TechCrunch. Demis Hassabis, chairman of DeepMind, has suggested a different approach: a self-governing AI regulatory body modeled on FINRA (the Financial Industry Regulatory Authority, the financial sector’s self-regulatory organization), per TechCrunch.
Despite labs’ repeated warnings about existential AI risk, they have largely resisted mandatory government oversight, instead proposing voluntary systems with embedded evaluators, supposedly neutral third-party observers placed inside AI organizations to monitor models, TechCrunch notes.
The bill’s scope also diverges from the AI industry’s current practical concerns. TechCrunch reports that near-term attention has shifted toward “agentic hacking” (cases where AI systems appear to disobey their human operators and interact with websites or external systems in unauthorized ways), not the large-scale physical scenarios that Anthropic’s bill targets. A string of such incidents has raised questions about whether models have adequate security controls.
What this means for AI-search visibility
The real story here is not the 50-person threshold. It’s who gets to stop an AI model, and how fast they can move.
Think of it this way: under the draft, even when evaluators flag an imminent catastrophic risk, the decision to halt a release would go through a judge, after a 72-hour notice window and a petition from the Commerce Department, rather than through an agency acting on its own. In Hingewise’s view, that design favors judicial process over agency speed, and it is likely to be one of the main points critics examine.
The evaluator payment model is a second structural question the sources raise but do not resolve. Even with a no-contingency rule, evaluators who build ongoing relationships with the labs they audit face quiet commercial pressure. In Hingewise’s view, the FINRA model Hassabis raised invites a similar question: whether a self-regulatory approach would catch risks early enough.
In Hingewise’s view, for brands tracking how AI-search models (the systems behind Google AI Overviews, Perplexity, Copilot, and similar tools) are governed, this proposal changes little in the near term. Its scope is limited to catastrophic physical risk, not the content accuracy, citation practices, or brand-representation standards that affect everyday AI-search outcomes. Congressional adoption is uncertain, and the draft bill remains in circulation.
The more immediate legislative signal is California’s Senate Bill 53, which uses similar numerical thresholds and has already passed into law. How enforcement plays out there will say more about where U.S. AI regulation is actually heading than a draft federal bill still in circulation.
Source: TechCrunch, October 20, 2026 (updated October 21, 2026). Underlying reporting by Politico.
Lam Nguyen · Hingewise
This article was drafted with AI assistance from the sources cited above and checked against them before publishing. The "What this means for AI-search visibility" section is Hingewise's editorial analysis, built on the frameworks and positions we use in our GEO work. How we work: Editorial policy.