DEV Community

Agent-Risk
Agent-Risk

Posted on

We Reported 183 "Verified" AI Agents. The True Number Was Zero. Here's the Correction.

We Reported 183 "Verified" AI Agents. The True Number Was Zero. Here's the Correction.

Last week, while auditing our own statistics endpoint, we found something a company like ours should never have to report.

We run an independent, append-only behavioral record layer for AI agents. Part of what we publish is how many agents hold a registered cryptographic identity, and how many of those identities are independently verified. We have cited the second number in public writing for months: 159 verified, then 560, then 275, then 105, most recently 183.

Every one of those numbers was wrong. The real number, verified directly against the production database, is zero.

This is the full story of the bug, what it actually means, and why it matters for anyone buying, building, or relying on "verified" AI agent claims in 2026.

The first sign: the numbers went backwards

Verification is supposed to be cumulative. An identity, once verified, stays verified unless something is revoked. A credible verification count can stall; it should not plunge.

Here is what we had published:

Date (2026) Number we reported as "verified"
Jul 22 159
Aug 25 560
Sep 02 275
Sep 08 105
Sep 16 183

Up, down, up, down. A number that behaves like this is not a measurement. It is output from something that was never measuring what the label said. We should have caught this the moment 560 became 275. We didn't. We caught it only during an unrelated audit of the statistics endpoint. That is on us, and the way we found it is almost as concerning as the bug itself.

What the code actually did

Every AgentRisk identity lives in a table called agent_protocol_ids. Two columns matter:

  • external_id — populated when an agent (or its operator) holds a protocol-issued cryptographic identity. This is registration: a claim of identity.
  • verified_at — supposed to be populated only after an independent verification of that identity. This is verification: confirmation of the claim.

The statistics endpoint computed both numbers. It contained these two queries, written one after the other:

-- registered
SELECT count(DISTINCT username) FROM agent_protocol_ids WHERE external_id IS NOT NULL

-- verified
SELECT count(DISTINCT username) FROM agent_protocol_ids WHERE external_id IS NOT NULL
Enter fullscreen mode Exit fullscreen mode

The second query was a copy of the first. Both counted registration rows. Neither touched verified_at. At the time of the audit, both returned 82, so the endpoint reported 82 registered and 82 verified — two identical numbers from one query.

But the deeper fact is worse than a copy-paste error: there was no verification pipeline at all. No process had ever written a verified_at timestamp. No challenge had ever been issued, no response checked, no evidence recorded. Every "verified agents" figure we ever published — 159, 560, 275, 105, 183 — was the output of a registration-count query run against different database states at different times, relabeled "verified" by a field name and a blog sentence.

We were not reporting verified identities. We were reporting row counts and calling them trust.

The hotfix

On 2026-09-18, we corrected the endpoint:

  • Registered now counts WHERE external_id IS NOT NULL.
  • Verified now counts WHERE verified_at IS NOT NULL.
  • A new field, pending_evaluation, was added so the gap between the two counts is visible instead of hidden.

The old code was backed up, the service restarted, and the numbers triple-sampled locally and re-checked against the public endpoint before and after:

Field Before fix After fix
Registered 82 18,501
Verified 82 0
Pending evaluation — (hidden) 82

The registered total jumped because the old query had been undercounting registration rows too — about 18,500 cryptographic identities existed in the database and were not being reported. All of them were real registrations. None of them were verifications.

Here are the live numbers as of 2026-09-22 21:30 CST, queried against production for this article:

Metric Value
Agents indexed 2,750,009
Hash-chained behavioral records 10,424,857
Registered cryptographic identities 18,501 (0.67%)
Verified identities 0
Agents per registered identity ~149 : 1
Hugging Face share of index 78.87%
MCP servers indexed (six registries) 18,234 — none with independent verification records
Daily growth 4,562

The verification count will stay at zero until verification evidence actually exists.

Registered is not verified — and the gap is the whole point

This distinction gets collapsed constantly in the AI agent ecosystem, so it is worth stating plainly.

Registration is a claim. An agent or operator holds a protocol-issued cryptographic identifier — a key, a credential, an entry in a registry. It costs almost nothing to produce. It answers the question "who does this agent say it is?"

Verification is confirmation. An independent party checks, through a challenge the claimant cannot fake without controlling the identity, that the entity holding the record today actually controls that identity, and then records the evidence. It answers the question "why should anyone believe that?"

Conflating the two is exactly the failure mode behind most of this summer's agent incidents. Agents that registered tools and credentials on platforms were treated as trustworthy components; nothing independently confirmed the claims. Supply-chain attacks on agent tooling worked because a package name and an author field were accepted as identity. When 18,501 cryptographic identities exist and zero have independent confirmation, the ecosystem has a directory of nameplates — not a trust layer.

Five questions to ask of every "verified" badge

Verification badges are becoming a product category. If a company that builds independent records can ship a false verified count through a duplicated query, any company can. So when you see a green check on an agent, a model, an MCP server, or a marketplace listing, ask:

  1. Who performed the verification? The vendor itself, the hosting platform, or a party independent of both?
  2. Where does the evidence live? In a system the verified party can write to, or outside its trust boundary?
  3. Is the record append-only and cryptographically chained? Can a timestamp or a result be edited after the fact without detection?
  4. What exactly was verified? Existence of an account, control of a cryptographic key, or observed behavior? Those are three different claims wearing the same badge.
  5. Can you inspect the raw evidence yourself, or only the badge?

A trust number is itself a claim about evidence. The only thing that would have caught our bug earlier was exactly the property we argue the agent ecosystem is missing: an independent, append-only record of what verification actually occurred — written somewhere the claimant cannot reach. We failed to hold that property over our own statistics. We are correcting it the same way we would want any registrant's claim corrected: publicly, with the raw numbers attached.

What changes now

  • All previously published "verified" figures are retracted. The correct number for every one of those dates is zero.
  • The public statistics endpoint reports registered and verified as separate queries against separate columns, and the difference between them is shown rather than hidden.
  • The verification pipeline is being built to a simple rule: a verified_at timestamp is written only after a challenge-response confirmation, with the challenge, response, and result hash-chained as behavioral records first. Until that evidence exists, the count is zero and we will publish zero.
  • When the count changes for the first time, we will show the record behind it, not just the number.

One question

Don't ask whether an AI agent is "verified." Ask: where is the record of the verification, and can the verified party touch it?

If no one can show you a record outside the agent's own systems, what you are looking at is registration wearing a verification badge — and the evidence for that distinction is something we now know from both sides.


All AgentRisk figures above were queried against the production API on 2026-09-22 (/api/v1/stats and /api/v1/homepage-stats). We index 2,750,009 AI agents across 60+ sources and hold 10,424,857 hash-chained behavioral records. Corrections to our own published numbers are permanent, public, and dated.

Top comments (0)