This paid post was unlocked thanks to Granola, the AI notepad for back-to-back meetings. Upgrade to an annual paid Pirate Wires subscription, and get four months free of a paid Granola plan ($56 value).

Anthropic CEO Dario Amodei’s blog post earlier this month, “We Must Pace the Frontier,” warned of out-of-control AI agents taking over the internet and argued for regulation to slow the industry’s progress.

Amodei also committed to allowing “embedded evaluators” into Anthropic, where they’ll have “employee-like access to verify safety practices and report incidents.” (Later, he announced that Accenture will do the “embedded” evaluation.)

Not to be outdone, Politico recently reported that OpenAI now supports provisions in Congress’s FRONTIER Act (a pending bill) that would require AI companies to allow designated outsiders — “independent verification organizations,” or IVOs — to monitor and minimize risk.

IVOs are the essential tool in a new model of AI oversight, one that could keep us safe as our understanding of AI risk evolves or kneecap the industry with an endless list of rules.

Long before disaffected androids poison the atmosphere, we’re all going to hear so much more about IVOs than we might wish. In a moment when normie sentiment against data centers is rising and some insiders tease the prospect of rogue AI ending humanity, IVOs are the industry’s newest pitch for winning the public’s trust.

Get 14 day free trial

Independent Verification, in theory

The concept of an independent organization to verify AI safety originated in a pair of papers co-authored, in 2019 and 2023, by Anthropic cofounder Jack Clark and legal and economic scholar Gillian Hadfield.

Then, in 2025, the nonprofit AI policy organization Fathom outlined its vision for IVOs. AVERI, another AI risk NGO, followed up with yet another paper on the concept earlier this year.

Together, these documents describe a future in which governments set up licensing regimes to certify IVOs, each of which checks that AI products aren’t dangerous in a certain way. An IVO wouldn’t have to assess all potential risks; instead, they’re likely to specialize. One might only certify against cybercrime, for example, checking that a bot quits helping a user if it suspects they’re planning something illegal.

Government certifiers would determine whether an IVO serves the public’s interest, furthers the state’s goals, and relies on adequate methods. If approved, the IVO would be licensed to offer state-sanctioned risk evaluations.

But here’s the thing: No one has to hire them — at least not under Fathom’s approach, which was just signed into California law via SB 813.

Lauren Wilson, from Fathom’s policy team, told me that, in their model, “it’s all carrot.”

So, while hiring an IVO will be completely optional in California, there are two reasons why a company might want to. First, safety certifications might help them close sales, Wilson told me. (She knows of an EdTech company that closed more contracts after getting a safety certification.) Eventually, consumers could start expecting safety certification.

Second, a state could make IVO certifications admissible in civil lawsuits.

For example, imagine an IVO that offers “self-harm prevention verification,” checking that a chatbot won’t play along if a user wants to discuss, for example, suicide. In a state where IVO certifications are admissible in court, a company might seek out this verification so they’ll have a better defense in the event that they’re sued by a family whose loved one took their life after chatting with their bot.

Independent Verification, in fact

There’s nothing stopping lawmakers in other states, or in Congress, from mandating AI safety verification — and turning it into a massive industry of its own.

In fact, the FRONTIER Act would require AI giants to retain an IVO to assess for catastrophic risks (defined as, basically, lots of deaths or lots of property damage). Other states are at work on this, too. A new Massachusetts law would require IVOs; legislation introduced in Ohio would incentivize IVOs (but not require them). Virginia, the Mecca of data centers, is officially studying the concept, and Connecticut approved running a pilot.

The obvious question: Why should the public have faith in these assessments if they’re paid for by the companies being assessed?

That’s not to mention potential preexisting conflicts of interest. Some of these “independent” verification organizations have ties to the companies they’re verifying. Faculty, for example, which was acquired by Accenture and will “embed” inside Anthropic, was backed by Jaan Tallinn, who led Anthropic’s Series A. And METR, the external evaluator that “independently” assessed OpenAI’s recent Hugging Face incident, was founded by Beth Barnes, who previously worked at OpenAI. In fact, the whole AI safety realm is thick with alums from frontier labs.

Wilson argues we should trust these IVOs because the government can always rescind their certification if bureaucrats believe they’re too amenable to clients.

It remains to be seen: Will the threat of losing government certification will be enough to keep IVOs honest?

Subscribe now

Mark Dalton, from the policy team of the R Street Institute, a think tank promoting free markets, is highly skeptical of these regulators-for-hire. He believes the IVO model — even if voluntary — will privilege incumbents who have the money to retain these regulators, and disadvantage startups.

Beyond that, he foresees a more pernicious outcome: IVOs will start lobbying for more authority. As they make more money, they’ll offer additional services at the state’s behest, turning what was meant be a threat mitigation program into a permanent sinecure for the doomer class, one that could quite possibly grow large enough to undermine US competitiveness much more effectively than it lowers “p(DOOM).”

Dalton warned, “There really isn’t an incentive for these things sunsetting themselves. They don’t go away.”

— Brady Dale