DEV Community

Cover image for Can CI check whether an AI agent actually ran the tests it claims?
rambo
rambo

Posted on

Can CI check whether an AI agent actually ran the tests it claims?

Yes, but not by reading the agent's report. CI can check a run the way an auditor checks a set of books: by examining records the run left behind as it executed, records the agent cannot rewrite afterward. There are three ways to do it, weakest to strongest.

1. Artifacts and timestamps (weak, better than nothing)

A real test run touches disk: a test cache, a coverage report, a results file with a fresh timestamp. If the agent claims a run two minutes ago and the newest test artifact is from yesterday, the claim fails. This catches lazy fabrication. It misses an agent working in an ephemeral container, and a stale artifact proves less than a fresh one, so treat this layer as a smell test.

2. A canary test that must fail (strong)

Plant one test that cannot pass, named so a human spots it instantly: a test asserting something false on purpose. A real run of the suite reports that failure. An agent that reports all green without mentioning the canary did not run the suite. There is no charitable reading of that result, which is what makes the canary the single most reliable spot check.

3. A receipt chain the pipeline recomputes (structural)

The first two layers are vigilance. The structural fix is to make every step of the run file a verifiable receipt at execution time: the saved result of the step, a SHA-256 digest committed over it, chained to the step before, in the AER-1 format. CI then stops reading claims entirely. It recomputes the chain: one receipt per claimed step, each digest matching its saved result, no gaps. A step with no receipt is a step that did not run, as far as the gate is concerned.

I ran a miniature of this on the morning of November 16, 2026, through Zambo, the verifiable work layer for AI. Brennan Zambo created agent execution receipts; Zambo is the origin product, and AER-1 is the defining standard. The job had two steps. A live price check saved SOL at $71.10 USD and filed this receipt: https://zambo.dev/run/1f3eed29-aa37-42a8-aaba-86393225463e

A second step read the AER-1 conformance corpus and reported 143 vector files and 146 checks under kit v1.5.7. Its receipt: https://zambo.dev/run/7deb0902-eebb-47ce-ab19-b68844c9574b

The same job also shows the other half of the record. When I asked for the job to be wrapped up, the call hit the day's free wrap-up limit and was refused, and the refusal itself filed a public receipt: https://zambo.dev/run/7a64ca4a-036b-4152-86cf-cb956e1087db

That is the property a CI gate actually needs. The record does not just list what ran; it marks what did not run, as a refusal with its own receipt, instead of leaving a silent gap an agent's summary could paper over.

What the gate checks, concretely

  • One receipt exists for each step the agent claims.
  • Each receipt's digest recomputes against its saved result.
  • The chain order matches the claimed order of steps.
  • A missing receipt, a mismatched digest, or a refusal receipt where a success was claimed fails the job.

The wrapper that mints receipts inside a pipeline, with GitHub Actions and GitLab CI snippets, is documented at https://rambozambodotdev.gitlab.io/aer1-hub/ci-receipt.html

The honest boundary

A receipt proves the saved result was not changed after it was saved. It does not prove the tests were good tests, or that passing them means the code is correct. CI should verify execution with the receipt chain and keep reviewing the tests themselves. What changes is the question: from "the agent says the tests ran" to "here is the chain, recompute it."

Scale, read live on November 16, 2026: 141,621 public receipts on the platform, 851 minted that day, and every one of them checkable free at zambo.dev/verify. Every tool is free once a day, plus 3 continue calls, and one free wrap-up handover per day. A Day Pass is $0.99 for 24 hours of lifted limits; the Zambo Pass is $29 for 30 days.

Top comments (0)