DEV Community

Discussion on: What should stop an AI coding agent before it acts?

Collapse
 
suirindo profile image
asagiman •

That is a strong example. If Gradle restores the task from cache, a deliberately broken control can still produce a false green, so the negative control has not actually proved the test is live.

Your fix separates two receipts that are often collapsed into one: the mutation was observed, and the test task actually executed against it. I’m going to treat cache state and task-execution identity as part of the evidence envelope instead of accepting exit 0 alone.

Do you also pin or verify the declared Gradle inputs now, or is cache-off plus task-executed evidence the boundary you found sufficient?

Collapse
 
jacksonxly profile image
Jackson Ly •

The protocol also says any test that reads icon or data files must list them as inputs, because an ordinary build can skip the test without any cache. Gradle's up-to-date check works from declared inputs too, so a changed drawable that was never declared leaves the test task UP-TO-DATE and the build green.

Thread Thread
 
suirindo profile image
asagiman •

Exactly. --no-build-cache closes the FROM-CACHE path, but it does not close Gradle’s UP-TO-DATE path when an input was never declared.

I’d require three separate receipts: the mutated asset is declared as a task input; the test task outcome is actually executed rather than FROM-CACHE, UP-TO-DATE, SKIPPED, or NO-SOURCE; and the deliberate break fails before the repaired case passes.

--rerun-tasks is useful for the control run, while an ordinary build still needs to prove that the declared input invalidates the task correctly. A sanitized task-outcome line and input declaration would be enough to preserve this as a concrete fixture.