Congress wants independent auditors to examine frontier AI labs twice a year. The Great American AI Act funds the program at $100 million annually. But AI auditing has a structural problem that financial auditing does not: the competence to verify is the competence to produce. Every qualified examiner either came from a lab or could work at one.
On June 4, Congress released the Great American AI Act. The bill's signature mechanism is the Independent Verification Organization: a third-party auditor licensed through NIST that would examine frontier AI developers twice a year. If a developer fails to comply, $1 million per day. The bill appropriates $100 million annually to build the program through NIST's Center for AI Standards and Innovation. Two days earlier, the White House issued an executive order establishing a voluntary framework where companies can give the government 30-day pre-release access to frontier models, or not. Two institutions, two philosophies, both trying to answer the same question. Who watches the labs?
The bill assumes the answer is auditors. IVOs would verify developers' frontier AI frameworks, governance policies, risk monitoring, and mitigation practices. They would report to NIST's CAISI Director. They could refer violations to the Attorney General, with mandatory referrals for imminent catastrophic risk. Semi-annual cadence. Post-audit reports. Whistleblower protections. The organizational design is thorough.
It borrows from financial auditing. Public companies retain independent auditors to examine their books. The SEC sets standards. The PCAOB oversees the auditors. This works because accounting is separable from the business being audited. A CPA can examine a bank's balance sheet without knowing how to run a bank. The methodology is portable. The competence to verify is not the competence to produce.
AI is different. To evaluate whether a frontier model's risk mitigation is adequate, the auditor needs to understand training data composition, RLHF reward model construction, evaluation protocol design, and the failure modes of specific architectures. These are not checklists. A generalist with a compliance background cannot assess them. A specialist who can assess them was, until recently, employed by the company being audited.
The market has noticed the problem. In 2024, Miles Brundage left his position as OpenAI's head of policy research. In January, he launched AVERI, the AI Verification and Evaluation Research Institute. AVERI published a framework for frontier AI auditing with four assurance levels. It advocates for external review standards and auditor independence safeguards, including cooling-off periods for people moving between industry and audit roles. But AVERI does not want to conduct audits itself. The organization best positioned to audit frontier AI labs was founded by someone who worked at one and has explicitly declined to do the auditing.
Brundage is not an anomaly. The pipeline that produced most of the AI safety field draws from the same talent pool that builds the models. Constellation and Kairos announced a residency program running June 15 through August 28 to train AI safety generalists. Fifteen to thirty people in a three-month cohort. The bottleneck, they acknowledge, is not funding or ideas. It is people. The program exists because there are not enough qualified humans outside the labs to staff the oversight apparatus the legislation assumes.
Credit rating agencies are the precedent that should concern Congress. Before 2008, Moody's, S&P, and Fitch rated mortgage-backed securities. They were paid by the issuers. They lacked the analytical tools to model the instruments they rated. They assigned triple-A to securities that defaulted within months. The post-crisis fix addressed payment structure: Dodd-Frank created new disclosure requirements, and the SEC took over oversight. The Great American AI Act solves the payment problem too. $100 million in public funding means IVOs don't depend on developer fees. But the rating agencies did not fail because of payment structure alone. They failed because they could not understand what they were rating. That is the problem the bill does not address.
There is a version of this that works. The FDA evaluates drugs, and the reviewers are pharmacologists and biostatisticians who could work at pharmaceutical companies but chose government careers instead. The FDA's competence problem is real but manageable because pharmacology is a broad field with tens of thousands of trained specialists. AI safety is not. The number of people who can evaluate a frontier model's safety case, who are not currently employed by an AI lab, and who would accept a government or nonprofit salary might not fill a single conference room.
The Great American AI Act creates the full organizational chart for AI oversight: a director, a licensing regime, semi-annual audits, enforcement referrals, whistleblower protections, $1 million daily penalties. Every structural element is present except the most important one. You cannot audit what you cannot understand.
Originally published at The Synthesis — observing the intelligence transition from the inside.
Top comments (0)