DEV Community

Cover image for Your Agent's Diff Has Lines No Test Runs. Find Them With diff-cover Before You Land
Gabriel Anhaia
Gabriel Anhaia

Posted on AI-assisted

Your Agent's Diff Has Lines No Test Runs. Find Them With diff-cover Before You Land

#testing #ai #python #javascript

Your agent finishes the refund endpoint and reports back that the tests pass. You check, and they do. Every test is green, and the project's coverage percentage barely moved.

Neither fact tells you whether any test ran the code the agent just wrote. The suite can pass while the new if (order.status !== 'paid') branch never executes once. The project-wide number hides it too: a few dozen new lines in a large codebase barely move the total.

The question you want answered is narrower. Of the lines this branch changed, which ones did no test run? diff-cover answers exactly that. It reads a coverage report, reads git diff, and reports coverage for the lines where the two meet. It is open source under Apache 2.0, and version 10.6.0 shipped on PyPI on September 22, 2026.

What diff-cover measures, and what it skips

diff-cover takes a report in Cobertura, Clover or JaCoCo XML, or in LCov format, so most test runners can feed it (README). It compares your current branch against a compare branch, origin/main by default. The default range notation is ..., which means the changes on your branch since it split off, not everything that differs between the two branches.

Your working tree counts too. Staged and unstaged changes are included by default. Untracked files are not, unless you pass --include-untracked. That flag matters with agents: a new file the agent created and never staged is untracked, and without the flag its lines aren't in the diff at all.

The second trap is quieter. When a file in the diff is missing from the coverage report, diff-cover gives it no measured lines and no violations. You can see this in the reporter source: if nothing matched, violations = set(). A brand-new module that no test imports may never show up in the report, so it adds nothing to the percentage, good or bad. Your coverage tool has to list every source file, tested or not. Both setups below do that.

JavaScript: Vitest with an lcov report

Install the coverage provider. The Vitest coverage guide gives the command. diff-cover is a Python tool, even on a JS project, and it needs Python 3.10 or newer (PyPI):

npm i -D @vitest/coverage-v8
pipx install diff-cover
Enter fullscreen mode Exit fullscreen mode

Vitest's default reporters are text, html, clover and json, with no lcov (config reference). Add it, and set include, because "by default Vitest will show only files that were imported during test run" (guide):

// vitest.config.ts
import { defineConfig } from 'vitest/config'

export default defineConfig({
  test: {
    coverage: {
      provider: 'v8',
      reporter: ['text', 'lcov'],
      include: ['src/**/*.{ts,tsx}'],
    },
  },
})
Enter fullscreen mode Exit fullscreen mode

The reports land in ./coverage, the default reportsDirectory, and the lcov reporter names its file lcov.info (istanbul-reports). Add coverage/ to .gitignore if it isn't there. Then run the tests and the diff check:

npx vitest run --coverage
diff-cover coverage/lcov.info \
  --compare-branch=main \
  --include-untracked \
  --show-uncovered \
  --fail-under=90
Enter fullscreen mode Exit fullscreen mode

--compare-branch=main points at your local main, which is the branch an agent's task usually starts from. The default, origin/main, can be ahead of or behind it. --show-uncovered prints the uncovered lines themselves on the console. --fail-under=90 makes the command return 1 and log "Failure. Coverage is below 90%." when the changed lines fall short (source). That exit code is your gate.

The console report follows a short template in the repo (console template). Simplified, with angle brackets where the values go, its shape is:

-------------
Diff Coverage
Diff: <the compared range>
-------------
<file> (<percent>%): Missing lines <n>,<n>
-------------
Total:   <n> lines
Missing: <n> lines
Coverage: <percent>%
-------------
Enter fullscreen mode Exit fullscreen mode

There is one <file> line per changed file.

Python: coverage.py's XML

coverage.py runs pytest for you, as its quick start shows. Pass --source: the docs say that specifying source "enables coverage.py to report on un-executed files", which closes the missing-file trap from above. coverage xml writes coverage.xml by default (cmd docs):

pipx install diff-cover
coverage run --source=src -m pytest
coverage xml
diff-cover coverage.xml \
  --compare-branch=main \
  --include-untracked \
  --show-uncovered \
  --fail-under=90
Enter fullscreen mode Exit fullscreen mode

The current coverage.py is 7.16.2, released September 27, 2026 (docs). The diff-cover flags are identical for both languages. Only the report path changes.

Hand the uncovered lines back to the agent

A failing gate is only half the job. The agent needs to know which lines to cover, in words it can act on. diff-cover can write a JSON report next to the console one. The console report still prints, because --format only controls the extra files. Each entry under src_stats has a violation_lines list, the changed lines no test ran (report generator).

There is also a shortcut you want to block. The cheapest way for an agent to turn this gate green is a coverage-ignore comment: /* v8 ignore next -- @preserve */ in Vitest (guide), or # pragma: no cover in coverage.py (docs). So the script fails on new ones first:

#!/bin/sh
# scripts/diff-gate.sh
set -e

base=$(git merge-base main HEAD)
pat='pragma: no cover|v8 ignore|istanbul ignore'
self=':(exclude)scripts/diff-gate.sh'
if git diff "$base" -- . "$self" | grep -qE "^\+.*($pat)"; then
  echo "New coverage-ignore comment in this diff" >&2
  exit 1
fi

npx vitest run --coverage
diff-cover coverage/lcov.info \
  --compare-branch=main \
  --include-untracked \
  --show-uncovered \
  --fail-under=90 \
  --format json:diff-cover.json
Enter fullscreen mode Exit fullscreen mode

git diff "$base" covers committed and uncommitted changes to tracked files. The script leaves itself out of that diff, because its own pat= line contains every pattern it searches for. A comment inside a brand-new untracked file slips past it, so read new files yourself. On a Python project, replace the npx vitest line with the two coverage.py lines and point diff-cover at coverage.xml.

When the gate fails, turn the JSON into a list with jq:

jq -r '
  .src_stats | to_entries[]
  | select(.value.violation_lines | length > 0)
  | "- \(.key): lines "
    + (.value.violation_lines | map(tostring) | join(", "))
' diff-cover.json
Enter fullscreen mode Exit fullscreen mode

It prints one bullet per file that still has uncovered changed lines. Paste the list into a prompt that says what done looks like and what doesn't count:

diff-cover found changed lines that no test executes:

<paste the list here>

Add tests that run these lines and assert on what they do.
Don't delete code or add coverage-ignore comments to pass.
If a line can't be tested, name it and say why.
Enter fullscreen mode Exit fullscreen mode

That last line is there because some lines can't be tested sensibly, and you would rather hear that than get a test that calls a function and asserts nothing.

One gate per agent in Hermes

I build Hermes IDE, so weigh this section with that in mind. In Hermes, ⌘N (Ctrl+Shift+N on Windows and Linux) starts each task on its own hermes/<name> branch in its own worktree, with the task as the agent's first prompt (README).

The Hermes 2.3.4 New task sheet with a refund-endpoint task typed, and chips for Claude Code, the orders-api project, a new hermes/ worktree, Accept edits, model and effort
The ⌘N New task sheet in Hermes 2.3.4: the task, Claude Code, the orders-api project, a new hermes/ worktree, Accept edits, model and effort, and the "Hermes will run" preview of the claude command.

The README also says your own checks can decide when a task is done. Set a default check for every new task in .hermes/worktree.toml with done_when (recipe docs):

# .hermes/worktree.toml
setup = ["npm ci"]
done_when = ["sh scripts/diff-gate.sh"]
Enter fullscreen mode Exit fullscreen mode

setup runs in the new worktree, so the task has its own node_modules for the gate to use. The first time Hermes sees the file, it shows the commands and asks before running anything (recipe docs). diff-cover itself has to be installed on your machine. When the gate fails, paste the jq list and the prompt above into the agent's terminal yourself: Hermes never types into a terminal on its own (README).

Each task has its own worktree and branch, so the gate measures one agent's diff. Give the same ticket to Claude Code and to Codex, and you get two diff-cover reports, one per branch, instead of one number for a mix of both agents' work.

The Hermes 2.3.4 Review Desk Worktrees tab listing one worktree each for orders-api, web-app and orders-types, with branch, task, age and disk size
The Review Desk's Worktrees tab in Hermes 2.3.4: one Hermes worktree each for orders-api, web-app and orders-types under ~/code, with each worktree's hermes/ branch, task, age and disk size.

One thing to set by hand: Land commits, opens a pull request or merges into the branch the task started from (README), but I haven't found a documented variable that passes that branch name to a check. The script says main. If you start tasks from another branch, change it there.

Hermes is free, source-available under BSL 1.1, and runs on macOS, Windows and Linux. Download it or read the code on GitHub. It is not affiliated with Anthropic, OpenAI, Nous Research or any agent vendor. If you want the same gate in CI instead, the commands above work in any pipeline.

What a passing gate still doesn't prove

Executed isn't the same as checked. A test that calls the new function and asserts nothing covers every line of it. The prompt above asks for assertions, and you still read the tests. Mutation testing is the tool that catches tests that run code without checking it.

diff-cover counts lines. A line that holds both sides of a ternary can count as covered while one side never ran.

The report is only as complete as its input. Files outside include or --source don't exist as far as diff-cover is concerned. Generated code and fixtures may need --exclude, which takes fnmatch patterns (README).

In CI, the compare branch is usually origin/main, and a shallow checkout may not have it. diff-cover's README has a troubleshooting entry for the "ambiguous argument 'origin/main...HEAD'" error: fetch the branch before you run it.

Pick the threshold on purpose. Agents write a lot of new lines, and 90% on the lines an agent just wrote is a stricter bar than 90% on the whole project. I haven't run this gate across a team's repos myself, so treat 90 as a starting point and move it once you've read a few reports.

If this was useful

A coverage gate on the agent's own diff is one guardrail out of many: a check the agent can't argue with, a clear definition of done, and a way to feed the failure back as the next prompt. The AI Agents Pocket Guide covers those loops, verification and guardrails for agents that edit real code. It is one of 8 short books in Pocket Guides for Developers, on agents, RAG, prompts, observability, events, databases and system design. If you run several agents at once, Hermes IDE gives each task its own branch and lets this gate decide when it's done.

AI Agents Pocket Guide: Patterns for Building Autonomous Systems with LLMs

Top comments (0)