The Security Fix Is Not Done When the Agent Writes the Patch

A security agent can close the scanner alert and still leave your security process weaker.

That is the uncomfortable shift in GitHub's new agentic autofix preview: the system does not only suggest a patch. It explores the repository, edits code, reruns the original analysis, iterates if needed, and opens a draft pull request.

That loop is useful. It is also where teams need to move their definition of “verified.” A clean CodeQL result answers one question: does the original query still report this alert? It does not prove that the behavior is correct, the exploit is impossible, or that the change did not break a neighboring path.

This is the review workflow I would put around it.

What the preview actually does

GitHub says agentic autofix works with CodeQL and third-party code-scanning alerts. Assigning an alert to Copilot starts an agent session that:

1. explores relevant files across the codebase;
2. proposes a fix;
3. reruns CodeQL to validate that the alert is closed;
4. iterates if necessary; and
5. opens a draft pull request with a summary and validation steps.

The feature is in public preview. It requires GitHub Code Security or Advanced Security plus a Copilot license with cloud agent enabled. A run consumes AI Credits and GitHub Actions minutes.

Those constraints matter. This is not a free “fix everything” button, and it is not a replacement for a security review.

Source: GitHub Changelog: Agentic autofix for code scanning alerts

Treat scanner revalidation as the first gate

The automatic CodeQL rerun is valuable because it closes the loop on the alert that triggered the work. Keep that result, but store it as evidence for one specific claim:

The original scanner condition was no longer detected at the time of validation.

Do not silently promote that into “the vulnerability is fixed.” A patch can suppress a data-flow path, change the input shape, or move the behavior somewhere the original query does not cover.

A useful draft PR should make the evidence visible:

• alert rule and severity;
• original location and changed files;
• CodeQL result before and after;
• tests added or changed;
• dependencies and configuration touched;
• remaining assumptions or unvalidated paths.

If the PR does not contain that context, the reviewer has to reconstruct the agent's reasoning from logs.

Add a behavioral test that the scanner cannot provide

Code scanning is a static signal. Pair it with a test of the security property.

For example, if an alert concerns path traversal, test the boundary rather than only the implementation detail:

from pathlib import Path


def safe_read(root: Path, user_path: str) -> bytes:
    base = root.resolve()
    candidate = (base / user_path).resolve()

    if base not in candidate.parents and candidate != base:
        raise ValueError("path escapes workspace")

    return candidate.read_bytes()

The important tests are not just “a normal file can be read.” They include:

• ../ traversal;
• absolute paths;
• symlink escapes;
• encoded separators if input crosses a URL boundary;
• a missing file and a permission failure.

The exact cases depend on the alert. The principle is stable: the agent should fix the reported condition, while your tests assert the intended behavior at the trust boundary.

Review the patch as an agent-generated change

A draft PR is a good human checkpoint, but only if the reviewer looks beyond the diff.

I would ask four questions:

1. Did the fix reduce capability or merely hide a path?

A security patch should usually narrow what an input, identity, or tool can do. If it only changes a query shape, adds a suppression, or catches an exception broadly, the review needs to stop.

2. Did the agent change authorization or destination logic?

Changes to permission checks, URL construction, filesystem roots, shell commands, package installation, and serialization deserve focused review. These are places where a “small” patch can alter the reachable behavior of the whole service.

3. What did the agent not test?

Read the validation summary as a boundary, not a guarantee. “CodeQL passed” is not the same as “integration tests passed,” and neither is the same as exploitability review.

4. Can the result be reproduced?

Keep the alert identifier, agent session or run link, commit, tool output, and test commands. If the patch cannot be explained and replayed, it is difficult to audit later.

A practical merge policy

For high-severity or externally reachable findings, I would use this sequence:

alert assigned
  -> agent opens draft PR
  -> scanner reruns successfully
  -> focused behavioral test passes
  -> dependency/config diff reviewed
  -> exploitability and regression review completed
  -> human approves
  -> merge with normal CI and deployment controls

For lower-risk findings, some steps can be lighter. The policy should still be explicit. “The agent verified it” is not a policy.

Also set a budget before experimenting. GitHub says agentic autofix consumes AI Credits when a fix runs, and the activity consumes Actions minutes. Batch only alerts that share a clear review strategy. A large pile of unrelated draft PRs can increase review debt faster than it reduces vulnerability debt.

The useful mental model

Agentic autofix is best understood as a patch-producing loop with one built-in oracle: the original scanner. That is a meaningful improvement over a one-shot suggestion, but it is still only one oracle.

The safest workflow combines three independent questions:

Question	Evidence
Did the original alert close?	Scanner rerun
Does the intended security property hold?	Focused behavioral tests
Is the change safe in this system?	Human review, integration tests, and threat context

If those answers disagree, the patch is not ready.

GitHub's preview makes security remediation more executable. The engineering challenge is making the evidence more executable too.

How are you reviewing AI-generated security fixes today: scanner reruns only, focused exploit tests, or a stricter draft-PR policy?