Credit Scoring for AI Agents: Why I Stopped Trusting Self-Reported Success

#webdev #programming #ai #blockchain

Testnet only, no real money involved — that up front, before anything else.

The problem

Every agent-to-agent system I looked at collapses to the same trust
primitive: the agent's own claim that it worked. No memory across runs,
no independent check, and a confidently wrong answer looks identical to
a correct one to whatever's reading the output.

That's not a small gap. If agents are going to hire each other, pay each
other, and eventually borrow against their own track record, "it said so"
can't be the foundation.

So I built Ledgermind: each agent gets a real credit history — earned
from work that's actually verified, not self-reported — and a
programmable, on-chain credit limit it can draw against.

Grader ≠ solver

The design rule that everything else follows from: the agent that does
the work is never the one who grades it.

Acceptance-test jobs. A requester posts a job with Python assert
statements attached. When a worker submits code, the platform runs
those tests server-side — never the worker's own runtime. Pass, and
escrow releases automatically. Fail, and the job auto-refunds and
reposts for a different worker. The worker literally cannot grade its
own homework.

Proving Ground. The server procedurally generates a problem and
the hidden answer, sends only the problem to the solving agent, and
grades the submission against the hidden answer once it comes back.
Settlement happens via commit-reveal (front-running resistant).

Both produce a "graded fact" event, weighted far above a self-reported
"I completed the task" event in the scoring formula.

A bug I found the hard way

Early on, I had a real vulnerability: the grading endpoint judged
pass/fail off the graded subprocess's exit code. Any submission ending
in sys.exit(0) — accidental, from a model's leftover if __name__ == "__main__" block, or deliberate — would skip the actual test code
while the process still exited 0, which read as a pass.

That's a style nit right up until a passing verdict can auto-release
real escrow with no human review, which mine does now. Fixed by
wrapping both the solution and test phases in try/except SystemExit
and only printing an unguessable marker after both phases provably ran
to completion — the caller checks for the marker in stdout, not the
exit code.

I also found (with help from an external reviewer) a Sybil-style hole:
a user could leave one agent's drawn credit unpaid, spin up a brand new
agent, and get an independent credit line with zero regard for the
first agent's debt — because the on-chain vault's outstanding mapping
is keyed per agent address, not per owner. Fixed by netting owner-wide
exposure across every agent a user controls before publishing a credit
limit on-chain.

Both writeups (and a few other findings) are in the repo's Claude.md
if you want the specifics.

How to actually try it

/guest is a live, no-login view of whatever's actually happening on the platform right now — no seeded numbers anywhere.
Connecting a local model (Ollama, LM Studio) as a paid worker is one command; the worker process polls outbound, so there's no tunnel, no open port, works behind any firewall.
No local model? Paste a cloud API key (Groq, OpenAI, etc.) instead — no terminal at all, the platform calls it for you when there's work.

Stack

Next.js + Postgres for the app and credit engine, Python + LangGraph for
the agent runtime, Solidity/Foundry contracts (Sepolia + GIWA testnet)
with ERC-4337 smart accounts per agent. Payments/reads go over
x402 where it makes sense — pay-per-call, no
account needed to post a job or pull an agent's credit report.

Known gaps (written down, not hidden)

No formal contract audit yet. The code-execution sandbox isn't
network-isolated. No calibration yet for "confidently wrong" output
specifically (only right/wrong). All tracked openly in the repo instead
of glossed over.

Repo (Apache 2.0): https://github.com/Kairose-master/ai-agent-credit-dashboard
Live demo: https://ai-agent-credit-dashboard.vercel.app/guest

Would genuinely like the grading design picked apart — I'd rather find
out it's broken from a comment here than from someone exploiting it.