A Screenshot Is Not an Agent Failure Artifact

A screenshot of an execution tree looks useful in a pull request.

It is also missing almost everything a skeptical reviewer should ask:

• Which trace produced it?
• Was the trace changed after capture?
• Which redaction policy was applied?
• Did the crop remove a warning?
• Can I inspect the failing step without asking the author to reproduce it?

Raw JSONL answers some of those questions and creates another problem: it may contain identifiers, attributes, or bounded previews that should not leave the developer's machine.

The practical handoff unit is neither screenshot nor raw trace. It is a derived, reviewable evidence artifact.

A picture is a view, not the evidence chain

A screenshot preserves pixels. It does not preserve provenance.

Reviewer question	Screenshot	Evidence package
What path ran?	Sometimes	Yes
Which source produced it?	No	Source run IDs and hashes
What was redacted?	No	Redaction report and policy
Did a file change later?	No	Packaged-file hashes
Can I inspect offline?	Only the crop	HTML, JSON, summary, trace copy

This is the model I want instead:

private source trace
        |
        | derive + redact
        v
reviewable package
  ├─ evidence.html
  ├─ summary.md
  ├─ redacted trace
  ├─ check results
  ├─ redaction report
  └─ evidence.json ----> hashes + provenance
                              |
                              v
                       offline verification

The source remains private. The reviewer gets a bounded object with enough context to question the result.

Build a bundle from one run or a session

AgentInspect's Evidence v2 bundle is one implementation of this pattern. These commands were verified against agent-inspect@6.17.6.

Create a package for one local run:

npx agent-inspect bundle <run-id> \
  --dir .agent-inspect \
  --profile share \
  --out ./agent-evidence

Or package a multi-run session with stricter redaction:

npx agent-inspect bundle \
  --session sess-support-042 \
  --dir .agent-inspect \
  --profile strict \
  --out ./agent-evidence

The source traces are read-only. The output contains derived copies and review surfaces such as:

agent-evidence/
├─ evidence.html
├─ evidence.json
├─ trace.html
├─ trace.jsonl
├─ summary.md
├─ metadata.json
├─ check-results.json
├─ redaction-report.json
├─ eval-results.json
├─ performance-summary.json
└─ assets/runs/...

evidence.html is the convenient entry point. evidence.json is what makes the folder inspectable as a package.

What the manifest records

The Evidence v2 manifest separates four facts:

{
  "evidenceFormatVersion": "1.0",
  "generator": {
    "name": "agent-inspect",
    "version": "6.17.6"
  },
  "source": {
    "runIds": ["run_example"],
    "traceSchemaVersions": ["1.0"],
    "sourceHashes": [
      {
        "runId": "run_example",
        "algorithm": "sha256",
        "hash": "..."
      }
    ]
  },
  "policy": {
    "redactionProfile": "share",
    "verificationPolicy": "share"
  },
  "files": [
    {
      "path": "trace.jsonl",
      "sha256": "...",
      "role": "redacted-trace"
    }
  ]
}

The important separation is between sourceHashes and files[].sha256:

• source hashes identify the pre-redaction bytes used to derive the artifact;
• file hashes protect the exact bytes inside the finalized package.

The source hash does not reveal the redacted content. It also does not preserve the original for you; retention remains a separate policy decision.

Verify before review

Run the offline verifier:

npx agent-inspect bundle verify ./agent-evidence

It checks:

• the Evidence manifest shape;
• required listed files;
• SHA-256 hashes for packaged files;
• unexpected files, which fail by default;
• the recorded assessment; and
• generator and source provenance.

If someone edits summary.md, replaces the redacted trace, or drops an unlisted file into the finalized directory, verification should stop passing.

That makes a precise claim: the finalized package is internally consistent with its manifest.

Imagine a reviewer receives two folders named agent-evidence. One contains the original bundle. In the other, someone corrected a sentence in summary.md after generation because the explanation was unclear. The edit may be innocent, but the artifact now describes bytes different from the ones recorded in the manifest.

The correct workflow is to fix the source or approved summary input, regenerate the bundle, and verify again. Hash failure does not tell you whether a change was malicious. It tells you that the finalized review object is no longer the same object.

Integrity is not authenticity

This boundary is easy to overstate, so it deserves blunt language.

bundle verify is an integrity check. Evidence v2 is not cryptographically signed by a trusted identity. Anyone who can replace the entire folder could create a different valid bundle.

It is also not:

• a live hash-chained write-ahead log;
• proof that every event was durable before a side effect;
• proof of replayability or exactly-once execution;
• nonrepudiation; or
• a compliance or audit certification.

If authorship or trusted delivery matters, add signing, access control, and trusted storage in your approved pipeline. Do not rename a local hash check into a stronger security property.

“Share-checked” still means “review it”

The default share profile is intended for pull requests and internal support. strict is a better starting point for public or external sharing.

The bundle command assesses the redacted artifact and refuses UNSAFE or UNKNOWN output unless --allow-unsafe is explicitly supplied. That is a useful fail-closed guard. It is not a guarantee that arbitrary free text contains no sensitive information.

Before attaching a package:

1. open evidence.html and summary.md;
2. review redaction-report.json and check-results.json;
3. run bundle verify;
4. confirm the destination is approved; and
5. apply your organization's data-retention and sharing policy.

Capture less in the first place. Redaction should not be the only protection between a raw customer payload and a public issue.

For pull requests, I would also record the verified bundle path or CI artifact name in a small behavior-evidence section. That gives reviewers one canonical object instead of a screenshot in Slack, a trace in local storage, and a summary pasted into the PR with no durable relationship among them.

Better evidence changes the review conversation

A good artifact does not decide whether an agent change is acceptable. It lets everyone inspect the same bounded object:

• the first explicit failure;
• the observed outcome;
• the contract finding;
• the sanitized execution path;
• the applied policy; and
• the package's integrity status.

That moves the discussion from “trust my screenshot” to “here is the evidence, here is what was removed, and here is what the verifier can—and cannot—prove.”

The pinned bundle documentation and Evidence format document the current layout and trust boundaries.

What would your reviewer need before they could safely inspect—and challenge—an agent failure artifact?