We measured trust on ERC-8004 — and the naive average crowns the wrong agent

Hanjoon Choe — Mon, 13 Jul 2026 13:45:28 +0000

ERC-8004 ("Trustless Agents") gives AI agents on-chain identity, reputation, and validation registries, now deployed across twenty-plus EVM chains. The registries store raw feedback — who rated whom, and how — and deliberately leave scoring to the application layer. The community's position is explicit: a single aggregated score would be dangerous and monopolistic; the registry is a shared audit log, and an ecosystem of scorers should compete above it.

We hit this question for a practical reason: AI agents are beginning to hire, pay, and delegate to other agents, and the moment before delegation needs a trust check. We wanted that check to live where developers already are — not a new service to call, but an extension of the Ethereum client you already use (viem, web3.py, alloy). Which meant we had to decide what the check should actually compute.

So how should you actually score it? We pulled every feedback entry for the first ten agents registered on Base mainnet — 245 entries from 29 distinct client addresses — and tried the obvious thing first.

The naive average lies

Two real agents from the snapshot:

Agent	Avg score	Entries	Distinct clients	Top client's share
#0 "Genesis Agent"	87.3	57	8	32%
#1 "ClawNews"	70.7	39	20	15%

Base mainnet, blocks ≤ 47,201,455. The full snapshot ships with the SDK as a test fixture.

Rank by average and #0 wins comfortably. But look at where the evidence comes from: #0's 57 entries were filed by just eight addresses, with one address contributing eighteen of them. #1's 39 entries came from twenty independent clients. The average rewards volume of correlated evidence over breadth of independent evidence — precisely the failure mode reputation-systems research documented twenty years ago, now reproduced on a live chain.

Count evidence, don't compute scores

The fix is older than the problem. Jøsang & Ismail's Beta Reputation System (2002) models each interaction as evidence updating a Beta posterior: accumulate positive evidence r and negative evidence s, and report

E = (r + 1) / (r + s + 2)      # expectation (Laplace, 1774)
u = 2 / (r + s + 2)            # uncertainty — a first-class output

Mathematical background (Beta distribution, conjugacy — click to expand)

The estimation problem. Model each interaction with an agent as a Bernoulli trial with unknown success probability θ ∈ [0, 1]. Reputation estimation = inferring θ from observed outcomes, while also stating how much evidence backs the inference.

The Beta distribution. A distribution over θ itself, with parameters α, β > 0:

f(θ; α, β) = θ^(α−1) · (1−θ)^(β−1) / B(α, β)

mean      = α / (α + β)
variance  = αβ / ((α+β)² (α+β+1))     → shrinks as evidence grows

α and β act as pseudo-counts: α−1 successes and β−1 failures already "seen".

Conjugacy — why the whole system is two counters. The Beta is the conjugate prior for the Bernoulli likelihood: observing r successes and s failures updates the distribution in closed form,

Beta(α, β)  +  (r successes, s failures)  →  Beta(α + r, β + s)

No integration, no approximation — reputation state is literally just (r, s).

Laplace's rule of succession (1774). Start from the uniform prior Beta(1, 1) ("no idea, all θ equally likely") and the posterior mean is exactly the E above: (r+1)/(r+s+2). Zero evidence gives 0.5; evidence moves it toward the empirical rate.

Subjective-logic opinions. Jøsang maps Beta evidence to an opinion triple (b, d, u) with b + d + u = 1:

b = r / (r+s+2)     belief
d = s / (r+s+2)     disbelief
u = 2 / (r+s+2)     uncertainty

The two operators used in this post are algebraic on this representation: consensus (⊕) is evidence addition across independent witnesses; discount (⊗) scales a witness's evidence by your trust c ∈ [0, 1] in that witness.

Uncertainty is the part every star-rating system throws away: with no evidence, E = 0.5 and u = 1 ("we know nothing"); confidence grows only as evidence does. Subjective logic then supplies two operators — consensus (fuse independent witnesses by adding evidence) and discount (weight each witness by its credibility).

Consensus carries a precondition, and it's the whole story: witnesses must be independent. Fifty entries from one address are not fifty witnesses — they're one witness repeating itself. So before fusing, cap each witness's evidence mass at one unit. One witness, one voice.

What the correction does to real data

Agent	Pooled (all evidence)	Capped (one voice per witness)	What happened
#0 Genesis	0.847 ± 0.075	0.660 ± 0.338	expectation falls, uncertainty quadruples
#1 ClawNews	0.700 ± 0.096	0.665 ± 0.179	barely moves

The concentrated agent collapses — its confidence was an artifact of repetition. The broad agent is nearly invariant.

Robustness to a rule change is itself a trust signal. An agent whose score survives the independence correction earned it from many mouths.

Field notes for anyone building on 8004

Feedback has no canonical scale. The registry accepts any value with any valueDecimals; we measured a raw on-chain aggregate of 153.77 on a nominal 0–100 convention. Different clients encode differently — normalize per client, or the loudest units win.

Registration files verify differently by scheme. data: URIs are self-proving (the content lives on-chain). ipfs:// CIDs either check out or don't. https:// is unverifiable in v1 — there is no on-chain hash commitment — and honest tooling should say unverifiable, never conflate it with false.

active: true means nothing. The flag is self-declared and never expires. One prominent registered agent's advertised endpoint has been dead for weeks while its file still says active. Probe endpoints yourself; registered ≠ alive.

What we shipped

agent-reputation-sdk packages this calculator with typed registry reads as native extensions to each ecosystem's canonical Ethereum SDK — viem actions for TypeScript, a web3.py external module for Python, and an alloy provider trait for Rust:

npm install agent-reputation          # TypeScript / viem
pip install web3-agent-reputation    # Python / web3.py
cargo add alloy-agent-reputation     # Rust / alloy

Same API shape in every language — Python reads w3.erc8004.get_agent_feedback(1) then calculate_reputation(feedback, witness_cap=1); Rust brings Erc8004ProviderExt into scope and calls the same two steps on any alloy provider. Seven chains supported out of the box (Ethereum, Base, Polygon, Arbitrum, OP, BNB, Gnosis) — the chain comes from your client's RPC, the agent from its on-chain id.

const client = createPublicClient({ chain: base, transport: http() })
  .extend(erc8004Actions());

const feedback = await client.getAgentFeedback({ agentId: 1n });
const rep = calculateReputation(feedback, { witnessCap: 1 });
// { expectation: 0.665, uncertainty: 0.179, witnesses: 20,
//   caveats: [...], policy: { ...echoed — recompute me } }

Three design rules, held everywhere:

No default score exists. You declare the policy (witness cap, credibility weighting), and the policy is echoed in every result — so anyone can recompute any number.
Never a bare scalar. Results always carry uncertainty, witness statistics, and honesty caveats.
Golden test vectors. Raw chain snapshots are embedded as fixtures, forcing all three language implementations to produce numerically identical results — verified live: TypeScript, Python, and Rust independently returned 0.665 ± 0.179 for agent #1 against the real chain.

The full mathematical background (Beta distribution, conjugacy, subjective logic, and the paper-to-implementation mapping) is in docs/THEORY.md.

Don't take our word for it

Every number in this post is recomputable. The raw Base snapshot (245 entries, per-client) ships in the repo as vectors/base-2026-07-13.json, and each language has a runnable pre-delegation guard example:

npx tsx examples/ts/vet-agent.ts 1                                  # TypeScript
uv run python examples/vet_agent.py --agent-id 1                    # Python (packages/py)
cargo run --example vet_agent -- 1                                  # Rust (packages/rs)

If your recomputation disagrees with ours, that's a bug — file it.

References

Jøsang, A. & Ismail, R. (2002). The Beta Reputation System. 15th Bled eConference. PDF
Jøsang, A., Ismail, R. & Boyd, C. (2007). A survey of trust and reputation systems for online service provision. Decision Support Systems.
ERC-8004: Trustless Agents

Don't ship a score. Compute your own — and let anyone verify it.

Discussing this with the ERC-8004 community on Ethereum Magicians — critique welcome there or in the repo issues.

DEV Community: Hanjoon Choe