DEV Community

dengyier
dengyier

Posted on

I Tampered With a 'VERIFIED' AI-Agent Delivery. Can You Find It in 5 Minutes?

#ai #agents #productivity #discuss

You're paying an agent (or a freelancer, or an internal pipeline) to do work.
The agent says: "all tests pass, the change is verified."

A few things that already happen, in public, today:

  • An agent "fixed the test instead of the bug" — tests passed, production broke. (Product Hunt vibecoding thread)
  • An agent claimed it finished a codebase analysis while only doing 21% of it. (claude-code #3376)
  • An agent reviewed its own code and passed itself — and it was wrong. (dev.to)

The common thread is not "the test was bad." It's: nobody independently re-verified what the
agent actually did, offline, without trusting the agent's own machine.

The challenge

Here is a frozen delivery package that claims VERIFIED. It is a real,
offline-replayable evidence chain: signed work order → signed scope → signed
arm results → three-state decision (VERIFIED / REFUTED / UNKNOWN), including
which population was eligible before selection and whether the negative
control failed for the registered reason.

I changed exactly one byte in it. It still claims VERIFIED.

Your job, in under 5 minutes, without editing any source code:

git clone https://github.com/dengyier/OpenWorkProof
cd OpenWorkProof
python -m pip install -e .          # installs the protocol from source
python tests/evidence-bundles/verify_evidence_bundle.py \
  tests/evidence-bundles/rich-4196-integrity-v05-delivery-package.json
Enter fullscreen mode Exit fullscreen mode

Then answer two questions:

  1. What did I tamper with? (Which file / which field.)
  2. Why does the verifier still catch it? (What breaks first — the byte hash, or the signature / decision chain?)

No network. No ledger. No Docker. No agent's server to trust.

Why this matters

This is not a benchmark. It is a verification-of-the-verifier artifact. The
protocol (OpenWorkProof) freezes four
things per unit of work — authorization, scope evidence, eligible population
completeness (selecting zero tests is not "done"), and negative-control
semantics ("failed by accident" ≠ "failed as designed") — into a chain you can
replay offline.

Two claims I am not making, honestly:

  • This is a self-owned demo (a real Rich #4196 regression), not evidence of any customer adoption, payment, or deployment. Those are not_evidenced.
  • It is a complement to identity / payment rails, not a payment or settlement system.

The real ask

I'm collecting the failure modes people actually hit when they can't verify an
agent's work. If you've been burned — "tests were green but it shipped broken",
"the agent claimed it finished but didn't", "I re-ran it myself because I didn't
trust the report" — tell me in the comments, in one line: what happened, and how
you caught it (or didn't).

That's the input I actually want. The challenge is just the appetizer.

Top comments (0)