DEV Community

Cover image for A Screenshot Is Not an Agent Failure Artifact
Raju Dandigam
Raju Dandigam

Posted on

A Screenshot Is Not an Agent Failure Artifact

A screenshot of an execution tree looks useful in a pull request.

It is also missing almost everything a skeptical reviewer should ask:

  • Which trace produced it?
  • Was the trace changed after capture?
  • Which redaction policy was applied?
  • Did the crop remove a warning?
  • Can I inspect the failing step without asking the author to reproduce it?

Raw JSONL answers some of those questions and creates another problem: it may contain identifiers, attributes, or bounded previews that should not leave the developer's machine.

The practical handoff unit is neither screenshot nor raw trace. It is a derived, reviewable evidence artifact.

A picture is a view, not the evidence chain

A screenshot preserves pixels. It does not preserve provenance.

Reviewer question Screenshot Evidence package
What path ran? Sometimes Yes
Which source produced it? No Source run IDs and hashes
What was redacted? No Redaction report and policy
Did a file change later? No Packaged-file hashes
Can I inspect offline? Only the crop HTML, JSON, summary, trace copy

This is the model I want instead:

private source trace
        |
        | derive + redact
        v
reviewable package
  ├─ evidence.html
  ├─ summary.md
  ├─ redacted trace
  ├─ check results
  ├─ redaction report
  └─ evidence.json ----> hashes + provenance
                              |
                              v
                       offline verification
Enter fullscreen mode Exit fullscreen mode

The source remains private. The reviewer gets a bounded object with enough context to question the result.

Build a bundle from one run or a session

AgentInspect's Evidence v2 bundle is one implementation of this pattern. These commands were verified against agent-inspect@6.17.6.

Create a package for one local run:

npx agent-inspect bundle <run-id> \
  --dir .agent-inspect \
  --profile share \
  --out ./agent-evidence
Enter fullscreen mode Exit fullscreen mode

Or package a multi-run session with stricter redaction:

npx agent-inspect bundle \
  --session sess-support-042 \
  --dir .agent-inspect \
  --profile strict \
  --out ./agent-evidence
Enter fullscreen mode Exit fullscreen mode

The source traces are read-only. The output contains derived copies and review surfaces such as:

agent-evidence/
├─ evidence.html
├─ evidence.json
├─ trace.html
├─ trace.jsonl
├─ summary.md
├─ metadata.json
├─ check-results.json
├─ redaction-report.json
├─ eval-results.json
├─ performance-summary.json
└─ assets/runs/...
Enter fullscreen mode Exit fullscreen mode

evidence.html is the convenient entry point. evidence.json is what makes the folder inspectable as a package.

What the manifest records

The Evidence v2 manifest separates four facts:

{
  "evidenceFormatVersion": "1.0",
  "generator": {
    "name": "agent-inspect",
    "version": "6.17.6"
  },
  "source": {
    "runIds": ["run_example"],
    "traceSchemaVersions": ["1.0"],
    "sourceHashes": [
      {
        "runId": "run_example",
        "algorithm": "sha256",
        "hash": "..."
      }
    ]
  },
  "policy": {
    "redactionProfile": "share",
    "verificationPolicy": "share"
  },
  "files": [
    {
      "path": "trace.jsonl",
      "sha256": "...",
      "role": "redacted-trace"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

The important separation is between sourceHashes and files[].sha256:

  • source hashes identify the pre-redaction bytes used to derive the artifact;
  • file hashes protect the exact bytes inside the finalized package.

The source hash does not reveal the redacted content. It also does not preserve the original for you; retention remains a separate policy decision.

Verify before review

Run the offline verifier:

npx agent-inspect bundle verify ./agent-evidence
Enter fullscreen mode Exit fullscreen mode

It checks:

  • the Evidence manifest shape;
  • required listed files;
  • SHA-256 hashes for packaged files;
  • unexpected files, which fail by default;
  • the recorded assessment; and
  • generator and source provenance.

If someone edits summary.md, replaces the redacted trace, or drops an unlisted file into the finalized directory, verification should stop passing.

That makes a precise claim: the finalized package is internally consistent with its manifest.

Imagine a reviewer receives two folders named agent-evidence. One contains the original bundle. In the other, someone corrected a sentence in summary.md after generation because the explanation was unclear. The edit may be innocent, but the artifact now describes bytes different from the ones recorded in the manifest.

The correct workflow is to fix the source or approved summary input, regenerate the bundle, and verify again. Hash failure does not tell you whether a change was malicious. It tells you that the finalized review object is no longer the same object.

Integrity is not authenticity

This boundary is easy to overstate, so it deserves blunt language.

bundle verify is an integrity check. Evidence v2 is not cryptographically signed by a trusted identity. Anyone who can replace the entire folder could create a different valid bundle.

It is also not:

  • a live hash-chained write-ahead log;
  • proof that every event was durable before a side effect;
  • proof of replayability or exactly-once execution;
  • nonrepudiation; or
  • a compliance or audit certification.

If authorship or trusted delivery matters, add signing, access control, and trusted storage in your approved pipeline. Do not rename a local hash check into a stronger security property.

“Share-checked” still means “review it”

The default share profile is intended for pull requests and internal support. strict is a better starting point for public or external sharing.

The bundle command assesses the redacted artifact and refuses UNSAFE or UNKNOWN output unless --allow-unsafe is explicitly supplied. That is a useful fail-closed guard. It is not a guarantee that arbitrary free text contains no sensitive information.

Before attaching a package:

  1. open evidence.html and summary.md;
  2. review redaction-report.json and check-results.json;
  3. run bundle verify;
  4. confirm the destination is approved; and
  5. apply your organization's data-retention and sharing policy.

Capture less in the first place. Redaction should not be the only protection between a raw customer payload and a public issue.

For pull requests, I would also record the verified bundle path or CI artifact name in a small behavior-evidence section. That gives reviewers one canonical object instead of a screenshot in Slack, a trace in local storage, and a summary pasted into the PR with no durable relationship among them.

Better evidence changes the review conversation

A good artifact does not decide whether an agent change is acceptable. It lets everyone inspect the same bounded object:

  • the first explicit failure;
  • the observed outcome;
  • the contract finding;
  • the sanitized execution path;
  • the applied policy; and
  • the package's integrity status.

That moves the discussion from “trust my screenshot” to “here is the evidence, here is what was removed, and here is what the verifier can—and cannot—prove.”

The pinned bundle documentation and Evidence format document the current layout and trust boundaries.

What would your reviewer need before they could safely inspect—and challenge—an agent failure artifact?

Top comments (9)

Collapse
 
bickov profile image
Alex @Bickov •

Hashes prove the bundle was not altered after creation, not that the trace itself was genuine to begin with. If the harness can forge or quietly drop frames before packaging, the manifest still checks out and the evidence is still wrong. Worth stating that distinction plainly, since reviewers tend to assume a verified bundle means more than a hash check actually gives them.

Separate issue with any screenshot or trace bundle used for UI review: the viewer usually downsizes the image before anyone looks at it again, so detail removed by redaction and detail lost to resizing end up looking identical, just missing pixels. A note in the manifest about what was intentionally redacted versus what never survived capture resolution would close that gap.

Collapse
 
raju_dandigam profile image
Raju Dandigam •

@bickov You're right: bundle verify attests that the finalized files match the manifest, not that capture was complete or honest. A useful next manifest boundary would record harness version, dropped-event/frame counts, original image dimensions, and any downsampling transform separately from the redaction report. That still would not prove the declarations true, but it would let a reviewer distinguish intentional redaction from detail that never survived capture. Would you require a minimum-resolution gate too?

Collapse
 
bickov profile image
Alex @Bickov •

Yes, but I'd gate on what the reviewer will actually see, not on the capture size. A tall capture can be huge and still useless: 2540x18410 lands at 355x2576 on current Claude, so 16px body text ends up around 2px tall.

So the check could be: take the smallest text height in the capture, apply the viewer's resize, and fail or split into tiles if it drops under about 8px. That keeps the gate tied to the model or viewer, since the limits differ, 1568 on older Claude and 2576 on 4.7 and later.

The numbers behind it: slimsnap.ai/blog/screenshot-downsc...

Thread Thread
 
raju_dandigam profile image
Raju Dandigam •

@bickov That’s a better gate: evaluate the artifact after the viewer resizes it, not just at capture resolution. I’d record original and output dimensions, the scale factor, and minimum post-resize text height, then fail or tile the image if text drops below the threshold. Your tall-capture example makes the failure mode concrete. Have you found a reliable way to measure the smallest readable text in code and dense tables?

Thread Thread
 
bickov profile image
Alex @Bickov •

Two ways, depending on where the capture comes from. If it's a browser page, skip measuring pixels and read it from the DOM: getComputedStyle(el).fontSize times devicePixelRatio gives the real text height for every element, code blocks and table cells included.

For plain screenshots, OCR boxes work. Tesseract's TSV output or Apple Vision's VNRecognizeTextRequest give a bounding box per line, and the smallest box height is your number. One catch: OCR misses the tiniest text first, so the minimum it finds is a bit optimistic. I'd take the 5th percentile of box heights, not the min.

Thread Thread
 
raju_dandigam profile image
Raju Dandigam •

@bickov The DOM/OCR split is useful. For DOM captures I’d combine computed font size, devicePixelRatio, and the viewer’s final scale; canvas or transformed text still needs a pixel check. Your OCR caveat matters: if the smallest labels disappear before recognition, even a percentile can look healthy. I’d tile or mark the check inconclusive when OCR coverage is low. Have you tried comparing both paths on the same page?

Thread Thread
 
bickov profile image
Alex @Bickov •

Not side by side yet, but the DOM number makes a good ground truth for testing OCR. Run both on the same page and the gap is the OCR miss rate.

One thing to check on the Apple side: Vision's minimumTextHeight defaults to 1/32 of the image height, per Apple's docs. On a 1080px tall capture that's about 34px, so small labels can be skipped before recognition even starts. Setting it lower is the first fix before any percentile logic.

Thread Thread
 
raju_dandigam profile image
Raju Dandigam •

@bickov That minimumTextHeight default is a useful catch: a 34px gate means OCR can never report the small labels we’re trying to protect. I’d lower it on the scaled artifact, then compare OCR boxes with a DOM-labeled fixture and flag missing labels separately from detected-size percentiles. Thanks for the concrete Vision setting.

Some comments may only be visible to logged-in visitors. Sign in to view all comments.