DEV Community

Zentropy Labs profile image Zain Dana Harper
Zain Dana Harper for Zentropy Labs

Posted on

The failed install was part of the evidence: reproducing an agent-action receipt

#ai #security #testing #python

A reproducibility report gets weaker when it records only the green ending.

Today I evaluated PermitReceipt v2.2.6-public-eval, an evaluation-only release for agent-authorization receipts, in a fresh Windows 11 and CPython 3.12 reviewer environment. The useful result was not just that the final checks passed. The documented portable install path failed first.

That failure belongs in the evidence.

The bounded question

Could a fresh, unaffiliated reviewer verify the release artifacts and run the documented reviewer path without relying on hidden environment state?

This was a narrow reproducibility test. It was not a production-readiness audit, an IETF review, or a claim that the receipt proves an agent action was authorized correctly.

What verified before installation

The release ZIP had this SHA-256 digest:

e5c40eca74fe2f451a0723db915c64b201e1d52f382cee24062e4dfc61fc632f
Enter fullscreen mode Exit fullscreen mode

The archive scan found 261 entries and zero unsafe paths. The release digest and artifact sidecars verified before package installation.

That distinction matters. Artifact integrity, package installability, and behavioral verification are separate claims. One passing result should not silently stand in for the others.

The first failure

Inside the fresh virtual environment, the documented portable path included:

python -m pip install --no-build-isolation -e . --no-deps
Enter fullscreen mode Exit fullscreen mode

It failed with:

BackendUnavailable: Cannot import 'setuptools.build_meta'
Enter fullscreen mode Exit fullscreen mode

--no-build-isolation tells pip to use build tooling already present in the environment. The fresh reviewer environment did not contain the expected backend, so the supposedly portable path depended on setup state that the command did not establish.

This is a small packaging defect, but it is exactly the kind of detail that disappears when a report records only the final green state.

Minimal recovery and downstream result

Installing the pinned build backends was enough to recover:

python -m pip install setuptools==83.0.0 wheel==0.47.0
python -m pip install --no-build-isolation -e . --no-deps
Enter fullscreen mode Exit fullscreen mode

After that recovery, the downstream reviewer suites reported:

  • IETF-format review packet: 20 of 20 passed
  • Independent recomputation: 17 of 17 passed
  • Independent cryptographic verification: 19 of 19 passed

Those suite names come from the release materials. Passing them does not imply IETF endorsement, external certification, or production readiness.

What the receipt should preserve

A useful run receipt needs enough structure to explain both the failure and the recovery. In this case, the minimum record looked like this:

artifact:
  release: v2.2.6-public-eval
  sha256: e5c40eca74fe2f451a0723db915c64b201e1d52f382cee24062e4dfc61fc632f
environment:
  os: Windows 11
  python: CPython 3.12
attempt:
  command: python -m pip install --no-build-isolation -e . --no-deps
  outcome: failed
  error_class: BackendUnavailable
  missing_backend: setuptools.build_meta
recovery:
  packages:
    - setuptools==83.0.0
    - wheel==0.47.0
  outcome: install_succeeded
verification:
  review_packet: 20/20
  recomputation: 17/17
  crypto: 19/19
non_claims:
  - not an IETF endorsement
  - not a production-readiness determination
  - not proof that every agent action was correctly authorized
Enter fullscreen mode Exit fullscreen mode

The exact schema can vary. The important property is that a later reviewer can distinguish what happened from what was inferred.

Five practical lessons

  1. Preserve the failed command. A recovery does not erase the original portability defect.
  2. Separate integrity, installability, and behavior. They are different verification layers.
  3. Pin build backends when using --no-build-isolation, or use build isolation so the backend is established mechanically.
  4. Treat limitations as first-class fields. A receipt should say what it does not prove.
  5. Publish the reproduction path. A claim becomes more useful when another person can inspect the commands and challenge the conclusion.

Public reproduction

I reported the exact bounded outcome, including the failed install and minimal recovery, in the public repository:

https://github.com/meridianverity/permit-receipt/issues/1

If you maintain an agent, MCP, evaluation, or workflow tool and have a public, credential-free fixture, I am interested in running one similarly bounded case and publishing the raw result, including a negative result when that is what the evidence shows.

Disclosure

I maintain the evidence tooling behind Project Telos at Zentropy Labs. I do not maintain PermitReceipt. The tested release is evaluation-only. I used an AI assistant to help structure and proofread this article. The commands, hashes, counts, links, and limitations were rechecked against the recorded run before publication.

Top comments (0)