The concrete problem
A security pull request can be green for the wrong reason. Unit tests may pass, the vulnerable endpoint may return a different status code, and a scanner may stop reporting the original finding. None of those results necessarily shows that the attacker lost the capability that mattered. The same identity might reach the sensitive action through another route, inherit a broader token, or trigger an equivalent workflow with slightly different input.
This becomes especially uncomfortable when an automated tool proposes or reviews the fix. A plausible patch explanation is not behavioral evidence. The reviewer still needs to know which identity was used, which preconditions were established, which requests ran, where privilege was gained before the fix, and at which exact step the patched build denied it. Without that trace, “fixed” is partly an assertion about code rather than an observation of the attack path.
The current signal
On August 17, Wiz described a GitHub Actions script-injection flaw in a Snowflake repository. The vulnerable workflow change reached production on June 18 and Wiz reported exploiting it on June 23. The final squash commit credited Copilot Autofix as a co-author, while AI-assisted review did not flag the injection. Wiz later clarified that it could not determine whether the code change itself was AI-generated. That distinction matters: the lesson is about assurance around AI-assisted workflows, not proof that a model wrote the bug.
The Hacker News discussion was active when RayTally captured it at 2026-08-18 00:33 UTC: 306 points, 123 comments, and rank 5. Those are historical attention numbers, not market validation. The useful engineering signal is narrower. Teams now have a concrete incident in which an apparently protective condition and an escaping routine still produced a reachable credential-exfiltration path.
Bright STAR and StackHawk show that dynamic testing in CI is already real. Bright documents building and starting applications, reproducing findings, applying fixes, and rescanning. StackHawk supports authenticated DAST, multi-role authorization testing, custom scripts, and CI integration. The remaining product question is not whether scanners can run against a live application. It is whether a reviewer can preserve one known attack as an acceptance test and compare its behavior across both sides of a patch.
A product direction: make the attack trace the review artifact
Treat the reproduction as a versioned object attached to the security PR. It should declare test identities, prerequisite state, ordered requests, expected sensitive effects, and the evidence needed to prove each step. The system deploys the baseline and patched commits into separate isolated environments, runs the same trace against both, and captures authorization decisions, audit events, and selected application spans.
The result should read like a small proof: the baseline gained access at step six; the patched build was denied at step four; no alternative privileged call appeared in the remainder of the trace. If the replay cannot complete because a short-lived token expired or a third-party callback is unavailable, the check should say “unresolved,” not quietly convert uncertainty into a pass.
The full product brief and source evidence on RayTally includes the historical observation boundary, competitor comparison, implementation outline, and original sources.
Minimal entry point
Start as a GitHub App for web applications using GitHub Actions. Let teams describe roles, seed requests, reset hooks, and success assertions in a repository file. Check out the baseline and patched commits separately, start each with Docker Compose, then replay HTTP and browser steps with a bounded runner. Pull server-side evidence from structured audit logs or OpenTelemetry traces and report redacted differences through the Checks API.
The first version should cover authentication, object-level authorization, and ticketing-style integrations. It should not attempt to invent arbitrary exploits, scan every repository, or replace penetration testing. A team brings a known reproduction; the product makes that reproduction safe, repeatable, and reviewable.
The strongest case against
Dual-environment replay is expensive to adopt. Many applications cannot start cleanly in CI, and realistic authorization tests require resettable data plus least-privilege accounts. SSO, expiring tokens, asynchronous jobs, and third-party callbacks introduce nondeterminism. Attack scripts are hostile by design, so the runner also needs strict egress controls, managed credentials, resource limits, and automatic cleanup.
The evidence can still lie. Comparing status codes is too shallow, but adding traces and audit logs requires application-specific instrumentation. A patch may block the recorded route while leaving an untested equivalent open. As role matrices grow, execution time and compute cost can turn a useful check into a merge bottleneck. Worst of all, one false declaration that an exploitable issue is fixed can destroy trust in the gate.
Existing DAST products may also absorb this workflow faster than a new vendor can earn security approval. The product is defensible only if its dual-version trace is substantially easier to author, inspect, and preserve than a custom regression test.
Question for readers
When you approve a security fix, what evidence would make you trust it most: a scanner rescan, a replay of the original attack, an authorization trace, or a deliberately independent second exploit attempt?
Top comments (0)