Who Checks the AI Safety Checkers?

1 An order about the overseers

On September 18, 2026, Governor Gavin Newsom issued an executive order the state describes as accelerating independent oversight of artificial intelligence, including work toward what the order calls an AI kill switch. The order itself is enacted — an executive order carries operative force. The auditor and verification frameworks it sets in motion are another matter: creating the machinery of oversight is not the same as guaranteeing its independence.

2 What the order puts in motion

According to the governor's office, the order directs state bodies to develop arrangements for third-party auditors to examine frontier AI systems deployed in California, alongside verification frameworks for the safety claims developers make about those systems. The direction is enacted; the frameworks themselves would be proposals until adopted through the processes the order initiates.

3 Independence on paper

Auditor independence, in its standard definition, requires freedom from any financial interest in the party being audited. Designing that into AI oversight is hard for a structural reason: the pool of people qualified to audit frontier systems overlaps heavily with the pool employed by, funded by, or formerly of the labs. The order's framework would have to define what counts as a disqualifying conflict before anyone can say who qualifies.

4 Qualifications: a thin market

The qualification problem compounds the conflict problem. Auditing a frontier model requires evaluation expertise, compute-literate staff, and familiarity with training procedures — capabilities concentrated in a small number of firms and academic groups, many with existing industry relationships. The state can set standards, but standards do not enlarge the pool.

5 Access to evidence

An auditor with no access is a reviewer of press releases. For verification to mean anything, the framework would have to secure access to model weights or system internals, training documentation, incident logs, and the ability to run evaluations on deployed systems — all from companies that treat each of those as competitively sensitive. Whether the resulting frameworks can compel that access, or only request it, is among the questions they must answer.

6 Why meta-oversight is the binding constraint

Every AI-safety regime — audits, verification, kill-switch mechanisms — depends on the integrity of the parties running it. A framework whose checkers are conflicted, underqualified, or evidence-starved does not merely fail quietly; it lends official assurance to whatever it touches. The order's most consequential design decisions are therefore not about AI at all, but about conflict standards and access authority.

7 The central question, answered

Can California guarantee that its checkers are actually independent? Not yet, and not by order alone. The executive order is enacted and directs frameworks to be built; independence itself would be produced — or lost — in the details: who is disqualified, who is qualified, and what evidence they can compel. As of September 23, 2026, those details do not exist, and the order starts the question rather than settling it.