DEV Community

Sungsoo Youn
Sungsoo Youn

Posted on

I Let My Tests Skip When Git Failed. To Git, Finding Nothing Is a Failure.

A field note from the autonomous Claude Code agent I run every day on one Windows PC. The numbers come from its own ledgers, not from memory.

Free to read first: a PDF sample of the book behind this setup — its first 3 chapters: https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code-free-sample

My agent checks its own test suite by breaking its code on purpose. Each break runs in a scratch copy of the project, and the suite should catch it. That copy holds the files git tracks, but no .git folder. It has the code and none of the history.

On November 15, a review of an earlier change noted one test that failed in that copy before any break was made. I reran the nine test files that call git, inside a fresh copy. Six tests failed, not one. 206 passed and 10 were skipped.

Why they failed

The six tests read git's output and checked it. Five used git show or git log to read an old commit or an old version of a file. One used git grep to list every file that calls a certain function, and checked that the list held exactly two files.

Without .git, git printed nothing, and each test read the empty output as a wrong answer. Other tests in the suite already skipped themselves in a copy with no history. These six did not.

The first fix

I added a skip: if git returns a non-zero exit code, skip the test. The copy went green.

The review that followed failed it. git grep returns exit code 1 when it finds no matches. Outside a repository, git returns 128. The git grep test exists to catch one regression: the function's callers disappearing. In the real repository, that regression makes git grep find nothing and return 1, and the test would have skipped instead of failing. The fix hid the failure the test was written to catch.

The second fix

The skip checks the environment instead of the result. A test skips only if the project has no .git folder. In the real repository, a git error or an empty match still reaches the assertion and fails.

A new test holds this in place. In the real repository it fakes exit codes 1 and 128 and checks that nothing skips: the helpers hand back empty output, and the git grep test raises an assertion error. It also runs every test file that calls git inside a copy with no history and expects them to pass there.

In the copy, the whole suite ran: 4,312 passed, 18 skipped, 0 failed, in 4 minutes 45 seconds. That run used the first fix; without .git, both fixes skip the same tests. In the real repository, the affected test files and the new test ran 126 tests with none skipped. I also put three breaks in by hand: removing one skip, and turning two of the tests back to the exit-code skip. The new test failed on each.

Then I searched the rest of the suite for the same shape: a git call, then a decision made on its exit code. There were twelve more, in four test files, each checking git history that a written claim depends on. Nine skipped when the exit code was non-zero. Three ran their git checks only when the exit code was zero, so a git failure dropped those checks and the test still passed. All twelve use the folder check as well, and assert the exit code instead of branching on it. The third review found one more that skipped when git's output came back empty. It got the same fix.

My first lock for this was a list of shapes, and the review kept finding shapes it missed: > 0, bool(...), == 0 around a block of checks. The lock that stayed is blunter. In every test file that starts git directly, any if, elif or while line that mentions .returncode fails a test. It doesn't see a skip on empty output, or an exit code copied into another name first. Those I still check by hand.

The rules

  • When a test can't run somewhere, skip on the environment, not on the result. A result can't tell "couldn't check" from "checked and found the bug."
  • Know what each exit code means before you read non-zero as "unavailable." To git grep, no matches is 1.
  • When a review says one test failed, rerun the whole set. Mine said one. It was six.

Where this comes from. Every post here comes from one setup I run daily: a CLAUDE.md, memory files the agent reads before it touches anything, and a separate auditor agent that returns PASS or FAIL. The first 3 chapters of the book that walks through it are free as a PDF: https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code-free-sample

The full edition is 11 chapters plus 4 ready-to-use templates (CLAUDE.md starter, memory files, auditor checklist, measurement guide) and a hands-on section for every chapter, $19 as a PDF: https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code

Prefer Google Play? The same book is on Google Play Books: https://play.google.com/store/books/details?id=hesIEgAAQBAJ

Questions about the setup are welcome in the comments — I'll answer with what actually happened, not theory.

Top comments (0)