Generating unit tests is easy. Proving they help is not.

Vinay Rathod — Tue, 03 Nov 2026 18:01:38 +0000

What I learned building a measurable workflow for AI-written tests in Go, Java and Python.

Every coding agent can now produce files full of test cases. That was the hard part two years ago. It isn't anymore.

The hard part is everything around it: finding the right project, not clobbering the tests that already exist, proving the new ones actually run, and knowing whether the coverage number went up for a real reason, or because a test now mirrors the implementation line by line.

I built a small CLI called testgen-unit around that gap. It works with Go, Java and Python, has no npm runtime dependencies, and uses your project's existing toolchain for validation and coverage.

It does not try to replace your coding agent. It gives the agent a structured workflow, project context, and deterministic checks. Direct model generation prints suggestions. It does not edit files or run the loop. Your agent, or you, applies the changes. The CLI verifies what actually happened.

This is the first post on this publication, so let me start with the problem that made me build it.

The test that proves nothing

A test that restates the production formula will raise coverage and protect very little.

def discount(price, pct):
    return price * (1 - pct / 100)

def test_discount():
    assert discount(100, 10) == 100 * (1 - 10 / 100)

The expected value is computed using the same formula as the implementation. The test repeats the implementation's reasoning instead of stating an independent business expectation.

If the underlying assumption is wrong, this test may still pass for its chosen input. For example, it says nothing about how fractional results should be rounded.

A better test states the expectation independently:

def test_discount_applies_ten_percent():
    assert discount(100, 10) == 90.0

That one encodes a requirement, not a formula. A useful suite would also cover rounding boundaries and invalid inputs.

So the workflow has to do more than generate. It has to make the result inspectable: what changed, what ran, what the runner actually said, and whether the new test is distinguishable from the ones already there.

The loop

Measure → Generate → Validate → Measure again → Review.

Five commands. Four are deliberately boring and deterministic; only generate-unit-tests introduces the uncertainty of model output.

testgen-unit unit-status        --language go --check
testgen-unit unit-coverage      --language go --json
testgen-unit generate-unit-tests --language go --module ./internal/foo
testgen-unit validate-unit-tests --language go --forbid-source-changes
testgen-unit review-unit-tests   --language go --fail-on-issues

That split is the whole point.

validate-unit-tests runs the project's own toolchain: go test, Maven or Gradle, pytest or unittest. It doesn't parse the model's output and trust it. If the suite fails, the command fails.

unit-coverage reads a fresh report: a Go coverage profile, coverage.py JSON, or JaCoCo XML produced by the build it just ran. A stale artifact from an earlier run is rejected rather than counted.

review-unit-tests is static and heuristic. It flags sleeps, duplicate Go test names in a package, identical Python test bodies, files changed outside the test scope, and test files with no recognizable assertion. It is explicit that it cannot certify semantic correctness.

And --forbid-source-changes turns "only edit tests" from a prompt suggestion into a checked constraint.

The model is the least reliable component

That framing matters, because it changes what you optimise.

I ran qwen2.5-coder:14b-32k, about 9 GB, through Ollama on macOS ARM64, against three small demo repositories. Two of the three needed a corrected batch.

The first Go response deleted the existing error-path test. It was rejected, and a follow-up that explicitly asked to preserve existing tests produced the accepted batch.

The first Python response repeated a zero-discount test under a new name. That exposed a real gap in the review step, which now detects identical test bodies that differ only in docstrings.

A later Python "fix" invented a requirement: that fractional percentages raise ValueError. They don't. Type annotations don't enforce that. Running the suite through the CLI rejected the assertion immediately.

Java was the counterexample: its first response passed unchanged after I fixed a fixture problem that had committed build artifacts into the demo.

That is the argument for this design. A model output that never gets corrected in public may simply be output that was never executed.

With the applicable corrections and setup fixes, each demo moved from a deliberately partial baseline to full coverage:

Language	Metric	Before	After	Validation	Review
Go	Statements	2/3	3/3	Pass	Pass
Java	Lines	2/3	3/3	Pass	Pass
Python	Statements	3/4	4/4	Pass	Pass

Those demos contain three or four measured statements each. They verify that the integration works. They are not a benchmark of production repositories, and I'm not going to pretend otherwise.

Don't average across languages

Go and Python report statements. Java reports lines through JaCoCo. Different denominators.

So the tool never collapses them into one number. Each language is aggregated separately, and a project that failed to measure makes that language's total unavailable rather than quietly weighting it to zero.

A single "82% overall" across three languages is a number you cannot defend.

Using a local model

The appeal is obvious: no API key, no source code leaving the machine. The cost is also obvious once you try it.

Two practical notes from running a 9 GB model on a laptop:

Raise the timeout. Prompt prefill, not token generation, dominates. A 60,000-character prompt is roughly 15–20k tokens before the model writes anything. The CLI defaults to a two-minute request timeout; use something like --timeout 600000.

Keep the default output cap. It's 4096 tokens. Lowering it makes runs finish faster and produces truncated batches, which is worse than slow. Shrink the scope instead, with --module or --package.

For example:

testgen-unit generate-unit-tests --language python --file price.py \
  --provider ollama --model YOUR_INSTALLED_MODEL --timeout 600000 \
  -- "Cover validation and error paths"

One detail worth knowing: --file scopes prompt context and static review, not validation or coverage. Validation runs the project's suite, which is what you want, but it means an unrelated broken test in the repo will fail your run. Fix the baseline first, or you'll blame the model for a pre-existing failure.

Limits I'd rather state up front

Review is heuristic. It catches recognizable mistakes, not missing regression value.

Coverage is a denominator, not a judgement. Read the assertions.

Java coverage requires an existing JaCoCo setup that emits XML. The tool does not inject build plugins into your project.

Findings in files you didn't touch are labelled [baseline], so you can separate "the repo was already like this" from "the model just did this."

Validation executes project code. Only run it on repositories you trust.

Try it

You need Node.js 22+, npm and Git, plus the toolchain for the language you're testing.

npm install -g testgen-unit
cd /path/to/your/repository
testgen-unit doctor --language go
testgen-unit install-skill

Then ask your agent to measure, generate, validate and review, or drive the commands yourself.

The point was never to generate more tests. It was to make each generated test survive contact with a real toolchain, and to leave behind evidence that it did.

If you try it on a real repository, I'd genuinely like to know where it breaks. That's the part that makes the tool better.

testgen-unit is MIT-licensed.

Repository: github.com/iamvinay92/testgen-unit

NPM package: npmjs.com/package/testgen-unit

If this was useful, subscribe. The next post will be a full teardown of one real repository run, including the tests the model got wrong.

DEV Community: Vinay Rathod