Trace AI Coding Changes to Requirements with Python and SARIF

AI-assisted code can look complete while quietly missing a requirement, an expected file, a required test, or an acceptance condition. A green-looking diff does not prove that the change answers the request that created it.

This tutorial builds a small evidence check with SpecTrace for AI Coding. You will describe requirements in JSON, map implementation files and test results to those requirements, then generate a Markdown report plus machine-readable JSON and SARIF output for CI review.

The useful boundary is deliberate: SpecTrace does not ask an LLM whether code is good. Its version 0.1.0 implementation uses the Python standard library to check whether the evidence map covers the requirements that reviewers defined.

What you will build

The example models a checkout audit trail. One requirement asks for allow and deny decisions to be recorded. The evidence map links that requirement to a recorder file, a passing test, and two acceptance criteria.

The flow is:

requirements JSON + change map JSON
                    |
                    v
              SpecTrace verifier
              /        |        \
             v         v         v
       Markdown     JSON      SARIF for CI

If any required link is missing, the verifier reports a finding and returns a nonzero status. That makes the check suitable for a pull request gate, provided the change map itself is produced and reviewed as part of your workflow.

Prerequisites

You need Python 3.10 or newer and a PowerShell, macOS, or Linux shell. The project is distributed under the MIT license and version 0.1.0 currently uses only Python's standard library at runtime.

Clone the repository and enter it:

git clone https://github.com/paladini/spectrace-ai-coding.git
cd spectrace-ai-coding

The commands below use the repository's current documented examples. They do not require an API key, model account, or network call after cloning.

1. Validate the evidence inputs

SpecTrace keeps the requirement definition separate from the change map. The bundled spec contains requirement IDs, expected file patterns, acceptance labels, and required test names. The change map supplies the files, test statuses, and acceptance evidence.

Run the input validation command:

$env:PYTHONPATH = "$PWD\src"
python -m spectrace_ai_coding validate `
  --spec examples\checkout-audit-spec.json `
  --change-map examples\checkout-audit-change-map.json

You should see a message stating that two requirements, three file links, and two test results were validated. This step checks the structure and cross-references before report generation. It does not claim that the underlying application is correct.

The important design detail is the shared requirement ID. A file or test only contributes evidence when its requirement_ids list includes the ID from the spec. An unknown ID is a cross-reference error rather than a silent mismatch.

2. Generate reviewer and CI artifacts

Now run verification and request all three output formats:

$env:PYTHONPATH = "$PWD\src"
python -m spectrace_ai_coding verify `
  --spec examples\checkout-audit-spec.json `
  --change-map examples\checkout-audit-change-map.json `
  --out build\spectrace-report.md `
  --json-out build\spectrace-report.json `
  --sarif-out build\spectrace.sarif.json

The Markdown report is for human review. It includes a summary, a trace matrix, and requirement details. The JSON report is convenient for scripts that need counts and finding objects. The SARIF document follows the SARIF 2.1.0 format, so a CI platform that understands SARIF can display findings alongside other analysis results.

For the bundled example, verification prints that two requirements passed and returns status 0. Inspect the generated Markdown before treating that result as useful review evidence:

Get-Content build\spectrace-report.md

The report should identify the checkout audit requirement, its linked files, its required tests, and its acceptance evidence. It is more useful than a bare pass count because reviewers can see what the change map actually claims.

3. See a failure instead of trusting a green result

The verifier distinguishes missing evidence from passing evidence. For example, if a required test is absent from the change map, the requirement receives a missing-required-test finding. If the test exists with a status other than passed, it receives a required-test-not-passed finding.

Other checks cover missing linked files, uncovered expected file patterns, missing acceptance evidence, and unknown requirement IDs. The implementation uses file-pattern matching for expected files, so a pattern such as src/checkout_audit/*.py must match at least one linked path.

This is the central lesson: a traceability tool can make an omission visible, but it cannot manufacture evidence. A change map that says a test passed is still an assertion that your workflow must produce and reviewers must trust.

Why the split between spec and change map matters

The requirement spec is the review request. It describes what must be true and what evidence should exist. The change map is the implementation record. It says which files changed, which tests were reported as passed, and why acceptance criteria are considered covered.

Keeping those documents separate gives you a stable question for code review: does this change map prove the requirements, or does it only describe the files that were edited?

SpecTrace builds a RequirementTrace for each requirement. It gathers matching files, tests, and acceptance entries, then applies findings for each missing or invalid piece. A requirement is marked passed only when that list of findings is empty. Global cross-reference errors are also included in the final result.

Failure modes to plan for

A valid JSON file can still describe weak evidence

Validation checks the data model and references. It does not inspect your application source, rerun a test named in the change map, or verify that a summary is truthful. Treat the map as review material, not as an attestation generated by the verifier.

A passing trace does not judge code quality

The project explicitly does not use an LLM to evaluate semantic correctness. It also cannot determine whether a real pull request is correct beyond the supplied evidence map. Pair it with normal tests, code review, and domain-specific checks.

Inputs may contain sensitive data

The repository's security policy warns that requirement documents, file paths, test logs, and AI coding evidence can be sensitive. Use synthetic examples in issues and tests. Do not place tokens, private source code, customer paths, or production logs into a public change map.

A failed verification is intentionally not a test failure

The command returns status 2 when requirements fail or global findings exist. That is distinct from an input validation error, which returns status 1. Your CI wrapper should preserve that distinction if it reports remediation hints.

Reproducible verification

Run the repository's tests after the example commands:

python -m unittest discover -s tests

The current repository test suite contains seven tests. The bundled command path also verifies two requirements, writes Markdown, JSON, and SARIF files, and passes with status 0 in the checked working tree.

For a practical CI integration, keep the spec and change map in the repository, run validate first, then run verify and upload build\spectrace.sarif.json using your platform's SARIF support. The exact upload action is platform-specific and is outside SpecTrace's scope.

FAQ

Does SpecTrace need an AI provider?

No. Version 0.1.0 uses Python's standard library and does not call model APIs.

Does it replace tests?

No. It checks that required tests are represented in the change map with a passed status. It does not run those tests itself.

Can it prove that an AI-generated change is correct?

No. It proves only that the supplied evidence map covers the declared requirements according to the implemented rules.

Why generate SARIF?

SARIF gives CI and code-scanning tools a common format for displaying findings. The Markdown and JSON files remain useful for local review and automation.

Takeaway

The smallest useful safeguard for AI-assisted development is often not another model call. It is a concrete link between a requirement, the files that address it, the tests that support it, and the acceptance evidence a reviewer can inspect.

SpecTrace makes that link deterministic and reviewable. Start with one high-value requirement, keep the evidence map honest, and expand the check only when your team can explain where each piece of evidence comes from.

Have you found a reliable way to generate and review change maps in your AI coding workflow, or do you still rely mainly on the diff and test output?

AI assistance disclosure: AI was used to help organize and edit this tutorial. The commands, repository details, implementation claims, and verification results were checked against the public SpecTrace repository and its bundled examples.