Three months ago, our legacy service had a test suite that failed at random. Every push triggered a long debate: rerun or debug? Nobody owned the flaky tests. A recent DEV thread reminded me that AI keeps changing our habits, and I realized we were still running a 2019-style test workflow against 2026 code.
I decided to try a different approach. Instead of fixing every bad assertion, I let a free model generate new contract-level tests from changelogs, then ran them on a free server every night. MonkeyCode's project provides both free models and a free server, which made this experiment cost nothing to start.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
This is the story of how I rebuilt our regression confidence without buying any infrastructure.
The Flaky-Suite Trap
Our old suite had three problems. First, tests asserted exact strings that changed for cosmetic reasons. Second, integration tests depended on system time and random ports. Third, nobody could regenerate missing tests when a feature changed, because writing them by hand took too long.
We tried the usual fixes: retries, pinning env vars, and a quarantine folder. That worked for a while, but new features kept landing without matching test updates. The gap between PRs and tests grew until a dangerous bug shipped silent.
Why a Free-Tier Contract Harness
A generated test is only useful if it checks a contract, not a snapshot. I wanted a system that watches our OpenAPI definition and service changelog, then generates behavior-focused tests that fail for real reasons.
MonkeyCode's open-source stack appealed because it gives you a model API without upfront payment and a small server to run scheduled jobs. For a small internal tool, those two things are enough. I did not need a cluster. I just needed a text prompt, a few Python files, and a cron slot.
How the Harness Works
Here is the architecture I built. It is intentionally simple so you can copy it into your own weekend project.
- Watch the changelog. A script tracks the diff between the latest release and the previous release, extracting changed function names and modified endpoints.
- Ask for contract tests. For every changed function, we send its signature and docstring to a free model endpoint and ask for four pytest cases: happy path, empty input, bad type, and timeout behavior.
- Run the tests on a free server. The generated tests land in a temporary directory. A nightly cron job runs pytest on the server and publishes a JUnit XML report.
- Diff errors against existing tests. The harness flags tests that fail for the same reason as an old test, so we know when a regression is real or just a stale expectation.
That loop turns a model's fast text generation into a boring, repeatable check.
Code: The Changelog Watcher
The first script extracts changed public functions from a Python package using ast and the local git history. Here is a simplified version that produced useful results for our service.
import ast
import subprocess
from pathlib import Path
def changed_functions(repo_path, old_tag, new_tag):
diff = subprocess.run(
["git", "diff", old_tag, new_tag, "--", "*.py"],
capture_output=True, text=True, cwd=repo_path
).stdout
funcs = set()
for line in diff.splitlines():
if line.startswith("+") and "def " in line and not line.startswith("+++"):
name = line.split("def ")[1].split("(")[0].strip()
funcs.add(name)
return sorted(funcs)
if __name__ == "__main__":
funcs = changed_functions(".", "v1.0", "v1.1")
print("\n".join(funcs))
This script is deliberately naive. It misses class methods and async defs. For a small service we used a more robust AST walker, but this snippet shows the idea.
Code: Prompt and Test Runner
Next, I built a generator that sends each changed function signature to the free model endpoint and stores the returned pytest code in a temp file.
import os
import requests
import tempfile
def generate_contract_tests(func_sig, docstring):
prompt = (
"Write four pytest tests for this function. "
"Cover happy path, empty input, wrong type, and timeout. "
"Use assert and no mocks unless needed.\n\n"
f"Signature:\n{func_sig}\n\nDocstring:\n{docstring}"
)
resp = requests.post(
os.environ["MONKEYCODE_MODEL_URL"],
json={"prompt": prompt},
headers={"Authorization": f"Bearer {os.environ['MONKEYCODE_KEY']}"},
timeout=60
)
return resp.json()["text"]
def save_and_run(tests, test_path):
with open(test_path, "w") as f:
f.write(tests)
result = subprocess.run(["pytest", test_path, "-q"], capture_output=True, text=True)
return result.stdout + result.stderr
The key is that we request four specific behavior categories. That prevents the model from generating a single happy-path test and calling it done.
The Nightly Execution Flow
The cron job on the free server does five steps.
- Clone the repository at the new tag.
- Read the previous tag from a state file.
- Run the watcher to find changed functions.
- Call the free model for each function and generate a temporary pytest file.
- Run pytest, zip the report, and post the summary to a private Slack channel.
I deployed this as a shell script plus a small Python orchestrator. The whole setup used less than 200 lines, including comments.
Decision Table: Trust the Generated Test?
Not every generated test deserves your trust. Here is the matrix I used to decide where to rely on AI-generated contract tests.
| Area | Trust AI tests? | Reason |
|---|---|---|
| Pure utility functions | Yes | Clear input/output logic |
| API handlers with documented contracts | Mostly | Good coverage of status codes |
| Functions with side effects | Partial | AI misses external state |
| Functions using global time | No | Needs manual fixed clock |
| Crypto or auth logic | No | Human security review required |
| Legacy code with hidden wrappers | Partial | Run in quarantine first |
Use this table as a starting point. Your own failure history will refine it.
Results After Three Weeks
The nightly harness caught two real regressions in the first week. Both were cases where a developer changed a function's return type but missed a caller in another module. The generated tests flagged the mismatch because the contract test explicitly asserted the documented return type.
Our flaky failure count dropped from nine per week to one or two. The remaining flakies were all in legacy modules that we now block from AI generation because side effects made the checks unreliable.
Most importantly, the team stopped dreading morning build reports. The report either said "new contract issue" or "nothing changed," and that clarity made PR reviews faster.
Limitations and Who Should Not Use This
This free-tier harness is not a replacement for a proper QA process. The free server on MonkeyCode is sized for small loads, so a huge repo with thousands of changed functions will exhaust CPU quickly. The free model access is meant for experimentation, not for high-frequency production workloads; you should cache results and throttle requests.
Do not use this pattern if your system handles money, health data, or critical infrastructure. AI-generated tests are drafts, not guarantees. They can encode the same wrong assumption that caused the original bug.
Also, this harness only checks what the prompt asks for. If your contract documentation is missing, the generated test will reflect that missing knowledge. Keep humans responsible for reading the diff and rejecting nonsense.
Try the Loop Yourself
If you have a small service and a recurring pain with regression coverage, clone your repo, write the watcher, and give the free model endpoint your ten most-changed functions. Run it on a free server for one week and compare failure counts before and after.
MonkeyCode's repo has the API surface and server setup documented, so you can inspect everything before committing your weekend. I ended up with a measurable improvement in confidence and a much quieter morning channel. That was worth the experiment.
Top comments (0)