DEV Community

Ventse
Ventse

Posted on • Originally published at traceseal.io

How to verify what an AI agent actually did

#ai #security #agents #devops

Your agent says it finished the job. Its log file agrees. Neither of those is evidence: the log was written by the same process it describes, and it can be rewritten afterwards by anyone with disk access. If you need to show a customer, an auditor or a court what an agent did, you need a record that fails loudly when altered and that a stranger can check without trusting you. This is a practical walkthrough of doing exactly that with an open verifier and a signed execution receipt.

Step 1: install the verifier

The verifier is a small open-source Python package with a published receipt specification:

$ pip install traceseal-verify
$ traceseal-verify receipt.json
[OK] receipt.json — operator signature verified
Enter fullscreen mode Exit fullscreen mode

The check runs offline. It needs no account, no API call and no access to the operator's infrastructure — which is the point. Verification that requires the operator's cooperation is not independent verification.

Step 2: read the receipt — the anatomy

A receipt is a single JSON document with three blocks. Here is a real one, abridged:

{
  "attestation": {
    "attested_at": "2026-04-15T05:29:04Z",
    "operator_fingerprint": "ed25519:d8d13a6f...",
    "operator_public_key": "3ba02728...",
    "signature": "4678a52c..."
  },
  "execution": {
    "skill_name": "yoast-seo-audit",
    "skill_version": "0.1.0",
    "inputs_hash":  "sha256:6ac78ba8...",
    "outputs_hash": "sha256:66858bb9...",
    "sandbox_profile_hash": "sha256:e7a3e6b8...",
    "skill_manifest_hash":  "sha256:79c14974...",
    "exit_code": 0,
    "ok": "true",
    "wall_time_ms": 47
  },
  "provenance": {
    "manifest_hash": "sha256:79c14974...",
    "publisher_fingerprint": "ed25519:d8d13a6f...",
    "transparency_log_seq": 5
  },
  "receipt_version": "1.0"
}
Enter fullscreen mode Exit fullscreen mode

Each block answers a different question:

  • provenance — which code ran. The publisher signs a content-addressed manifest: every artefact (the skill definition, its capability declaration, the code itself) is listed by its SHA-256 hash (FIPS 180-4). Change one byte of the code and the manifest hash no longer matches.
  • execution — what happened. Inputs, outputs and the sandbox policy are recorded as hashes of their canonical JSON form. That proves integrity without exposing the data — you can demonstrate the output is unchanged without publishing the output.
  • attestation — who vouches for it. The operator signs the whole record with an Ed25519 key (RFC 8032). The signature covers a canonical JSON encoding — sorted keys, fixed separators — so there is exactly one valid byte sequence for a given receipt. Any edit anywhere breaks the seal.

Step 3: understand what the check proves

When traceseal-verify prints [OK], four things have been established:

  1. The receipt is byte-for-byte what the operator signed — nothing was edited afterwards, by the agent, the operator's tooling, or anyone who handled the file since.
  2. The code that ran is exactly the code the publisher signed, down to the hash of each file.
  3. The recorded inputs, outputs and sandbox policy are the ones present at execution time — anything you are later shown can be checked against the hashes.
  4. The signing key is identified by fingerprint, so repeated receipts from the same operator are linkable, and the receipt can be anchored in a public transparency log (the transparency_log_seq field), which prevents the operator quietly maintaining two versions of history.

What a receipt does not prove

Honest tools state their limits, and this matters if you ever rely on a receipt in a dispute:

  • It does not prove the work was good. A receipt proves what ran and what it produced — not that the output was correct. Checking outcomes against ground truth is a separate, complementary verification step.
  • It does not prove events it never covered. A receipt seals one execution. Actions taken outside sealed executions are simply absent — which is why instrumentation has to start before the incident, not after.
  • It binds a key, not a person. The signature proves the holder of the operator key vouched for the record. Tying that key to a legal identity is a key-management question, the same as with any signing scheme.

The trust inversion: with logs, the burden is on the reader to trust the writer. With signed receipts, the burden is on the record to survive verification. That is the difference between "our system says it behaved" and "check it yourself".

Why this beats screenshots and log exports

The evidence teams typically produce today — screenshots, log exports, dashboard PDFs — shares one flaw: it is all produced by, or under the control of, the party whose behaviour is in question. Computer-security guidance on log management has warned for years that logs require protection precisely because they are alterable by whoever controls the system (NIST SP 800-92). A detached signature over a canonical record removes the alterability, and an open verifier removes the need to take anyone's word for it.

Everything shown here is open: the receipt spec, the verifier on PyPI, and the transparency log. You can implement the format yourself; the spec even includes a reference canonical-JSON encoder.

Originally published on traceseal.io.

Top comments (0)