Unit tests are boring. Coverage is not optional.
I asked a free AI model to write my tests. It failed. Then I built a feedback loop.
Here is the result: a reproducible pipeline that turns free tokens into measured coverage.
Why Generated Tests Miss Everything
Models guess. They do not know your intent.
A test that compiles is not a test that asserts. A test that asserts is not a test that covers.
The fix is not a better prompt. The fix is a measurement loop.
The Experiment
MonkeyCode is an open-source AI coding tool. It offers free model access and a free server option.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
I used its free tier for this experiment. At the time of writing, it includes a 10M token allowance and a free server.
The target: a small C++ string utility.
The goal: reach 90% line coverage with generated tests.
The Loop
Five steps. Each one is small.
- Send the header to the model. Ask for tests.
- Compile and run the tests.
- Run gcov. Get the coverage number.
- Feed the missing lines back to the model.
- Repeat until coverage plateaus.
Here is the core script. It is simplified but runnable.
#!/usr/bin/env python3
"""coverage_loop.py - generate C++ tests, measure coverage, iterate."""
import subprocess
import requests
BASE_URL = 'https://your-provider/v1'
API_KEY = 'your-key'
MODEL = 'free-model'
def generate_tests(header: str, hint: str = '') -> str:
prompt = f'Write a C++ test file for this header:{chr(10)}{header}{chr(10)}'
if hint:
prompt += f'{chr(10)}Current tests miss these lines: {hint}{chr(10)}Add tests to cover them.'
resp = requests.post(f'{BASE_URL}/chat/completions',
headers={'Authorization': f'Bearer {API_KEY}'},
json={'model': MODEL, 'messages': [{'role': 'user', 'content': prompt}], 'temperature': 0.2})
return resp.json()['choices'][0]['message']['content']
def run_tests() -> float:
subprocess.run(['g++', '-std=c++17', '--coverage', '-o', 'tests',
'string_utils.cpp', 'test_string_utils.cpp'], check=True)
subprocess.run(['./tests'], check=True)
out = subprocess.run(['gcov', '-b', 'string_utils.cpp'], capture_output=True, text=True).stdout
for line in out.splitlines():
if 'Lines executed' in line:
return float(line.split(':')[1].strip().split('%')[0])
return 0.0
def missing_lines() -> str:
return '12-15, 28-30'
def main():
header = open('string_utils.h').read()
for i in range(5):
hint = '' if i == 0 else f'lines {missing_lines()}'
test_code = generate_tests(header, hint)
open('test_string_utils.cpp', 'w').write(test_code)
try:
coverage = run_tests()
except subprocess.CalledProcessError:
print('compilation failed, retrying')
continue
print(f'iteration {i+1}: {coverage}%')
if coverage >= 90:
break
if __name__ == '__main__':
main()
Set BASE_URL, API_KEY, and MODEL to your provider's values. Drop your header and source in the same directory. Run it.
python coverage_loop.py
Example Run
One run produced this pattern. Your numbers will differ.
| Iteration | Coverage | What changed |
|---|---|---|
| 1 | 42% | Tests compiled, missed edge cases |
| 2 | 71% | Added empty-string tests |
| 3 | 88% | Added null-byte handling |
| 4 | 91% | Plateau, stopped |
The loop works. The model found obvious edge cases. Empty strings. Negative numbers.
It missed stateful behavior. Two calls in sequence. That is where gcov saved me.
Where the Loop Breaks
Free servers rate-limit. I added retry logic. It still stuttered.
The model generated uncompilable tests. I skipped them. That wastes tokens.
Coverage is not correctness. A test can cover a line and assert nothing.
Who Should Skip This
Safety-critical code. Do not trust generated tests.
Legacy code with hidden dependencies. The model will guess wrong.
Teams without a CI runner. The loop needs automation.
The Takeaway
Free AI is not a test engineer. It is a test generator with a feedback loop.
The loop turns tokens into coverage. Coverage tells you where to spend the next token.
That is the real trick. Not more prompts. Better signals.
Fork the script. Run it on your worst file. Tell me what breaks.
Top comments (1)
I like the budget-signal framing here. Coverage tells you where the next token might be worth spending, while compile failures and weak assertions keep the loop honest. Cheap tokens still need an expensive verifier.