From Bug Log to Free Server Regression Gate: 142 Bugs, 51 Tests

Here is the short version: we turned 142 historical bug reports into 51 passing pytest tests, generated by a free model endpoint and run on a free server—zero paid infrastructure. This post walks through the exact workflow, including the decision table, generation script, review loop, and limitations. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

From Bug Log to Structured Data

Our bug log was a markdown file with two years of entries. The log was useful for postmortems but useless as a regression net. We converted the useful entries into a Python list, keeping only fields that matter for a test: title, repro steps, expected behavior, and module name.

bug_history = [
    {
        'id': 'BUG-0142',
        'title': 'Date parser rejects leap seconds',
        'description': 'Parsing the leap second string raises ValueError.',
        'repro_steps': 'Call parse_timestamp with the leap second string.',
        'expected': 'Return a datetime object with second equal to 60.',
        'module': 'timestamp_utils',
        'severity': 'high',
        'auto_test_eligible': True
    },
    {
        'id': 'BUG-0207',
        'title': 'Config loader fails on empty yaml file',
        'description': 'An empty config file crashes instead of using defaults.',
        'repro_steps': 'Call load_config with an empty yaml file.',
        'expected': 'Return the default config dictionary.',
        'module': 'config_loader',
        'severity': 'medium',
        'auto_test_eligible': True
    },
    {
        'id': 'BUG-0301',
        'title': 'Race condition in cache update',
        'description': 'Concurrent cache writes sometimes lose a value.',
        'repro_steps': 'Run 50 concurrent writes for the same key.',
        'expected': 'Final value is one of the written values, not corrupted.',
        'module': 'cache_store',
        'severity': 'critical',
        'auto_test_eligible': False
    }
]

The last bug was marked auto_test_eligible: False because the free server could not run a reliable concurrency check. The first two had clear inputs and simple expected values, making them good candidates.

Decision table for eligibility

We used a small decision table to stay honest. Before sending any bug to the model, we asked three questions: Are the repro steps clear? Is the expected output a simple value? Does the test require external services or flaky timing? The table we used looked like this:

Bug property	Good for auto test?	Reason
Clear repro steps	Yes	The model can map steps to a test.
Expected output is a simple value	Yes	Easy to assert.
Requires external API or database	No	Free server may lack access.
Intermittent or race condition	No	Test would be flaky.
Security-sensitive payload	No	Keep it out of third-party requests.
Well-named module	Yes	The generated import has a target.

We applied this filter to all 142 bugs. Only 61 passed. That was enough—we were not trying to cover every bug, just a fast, useful subset.

Generating Draft Tests with a Free Model

We wrote a small Python script that read the eligible bug list, called the free model endpoint, and saved the returned test code to a file. The exact route and API key changed over time, so we kept those values in environment variables and checked the current docs before each run. We used the requests library for the HTTP call.

import json
import os
import requests
from pathlib import Path

BUG_LIST = Path('bug_list.json')
MODEL_URL = os.environ.get('MC_MODEL_URL')
API_KEY = os.environ.get('MC_API_KEY')

def load_eligible_bugs(path):
    data = json.loads(path.read_text())
    return [b for b in data if b.get('auto_test_eligible')]

def build_prompt(bug):
    return (
        'Write one pytest function for this bug. '
        'Use only the module and function shown. '
        'Do not invent a fixture. '
        + 'Module: ' + bug['module'] + '. '
        + 'Title: ' + bug['title'] + '. '
        + 'Repro: ' + bug['repro_steps'] + '. '
        + 'Expected: ' + bug['expected'] + '. '
        + 'Name the test test_regression_' + bug['id'].lower().replace('-', '_') + '.'
    )

def generate_test(bug):
    payload = {
        'messages': [
            {'role': 'system', 'content': 'You write pytest tests. Use only the given imports and functions.'},
            {'role': 'user', 'content': build_prompt(bug)}
        ],
        'temperature': 0.2
    }
    headers = {'Authorization': 'Bearer ' + API_KEY}
    resp = requests.post(MODEL_URL, json=payload, headers=headers, timeout=60)
    resp.raise_for_status()
    return resp.json()['choices'][0]['message']['content']

def write_test_file(test_dir, bug, test_code):
    filename = 'test_' + bug['id'].lower().replace('-', '_') + '.py'
    test_file = test_dir / filename
    test_file.write_text(test_code)
    return test_file

The prompt deliberately asked for a boring, readable test. We set temperature to 0.2 to keep the output stable. We did not ask for clever code or edge-case handling beyond the bug itself. Each generated file was a draft, nothing more.

Running and Reviewing on a Free Server

We did not run all 61 tests in one batch. We processed 10 bugs at a time to keep memory and time low. The free server handled that easily. We scheduled a weekly run that pulled the bug list from a git repo, generated fresh tests, ran pytest, and wrote a small markdown report to our team channel.

import subprocess

def run_tests(test_dir):
    result = subprocess.run(
        ['pytest', '-q', str(test_dir)],
        capture_output=True,
        text=True
    )
    return result.returncode, result.stdout

def main():
    bugs = load_eligible_bugs(BUG_LIST)
    test_dir = Path('generated_tests')
    test_dir.mkdir(exist_ok=True)
    report = []
    for bug in bugs:
        try:
            test_code = generate_test(bug)
            write_test_file(test_dir, bug, test_code)
            code, output = run_tests(test_dir)
            report.append(
                {
                    'bug': bug['id'],
                    'exit_code': code,
                    'output': output[:200]
                }
            )
        except Exception as exc:
            report.append(
                {
                    'bug': bug['id'],
                    'error': str(exc)
                }
            )
    print(json.dumps(report, indent=2))

if __name__ == '__main__':
    main()

The first run produced 61 generated files and completed in under five minutes. Some tests failed because of bad imports or invented function names. We expected that and did not delete the failures; we logged them and moved on to review.

Human review before the shared suite

The model generated decent tests for simple bugs, but about one in five needed changes. We reviewed each test and checked three things: the function name matched the real module, the assertion matched the expected value, and no unnecessary sleep or monkeypatch calls had slipped in. We kept only tests that passed after a small fix. The final suite held 51 tests and ran in under two minutes on the free server.

This review step took one afternoon for the first batch. Later batches went faster because we learned which bug properties predict clean generation. We treated every generated test as a draft that required human approval before entering the shared regression folder.

Limitations We Accepted

This workflow is not a security guarantee, and it is not a replacement for code review. The free model can hallucinate function names, and the free server may throttle requests. We did not run the gate on every commit; we ran it once a day, then once a week. The free tier's limits forced us to be selective, which turned out to be a feature: it made us focus on high-value bugs.

Teams with sensitive data should not send bug descriptions to a model endpoint. Teams that need zero flakiness should not use generated tests as the only gate. Those teams need a paid, isolated environment. Teams with mostly concurrency or networking bugs will get little value; the model can write a test sketch for those bugs, but the free server cannot reliably run them. Those teams should invest in dedicated test infrastructure.

Action Plan: Start with Ten Bugs

We stopped sleeping through 4:00 a.m. breakages. We had a small, boring regression net that cost nothing but time. The bug log became a living test plan. Anyone with a bug list and a free model endpoint can try the same flow:

1. Export ten old bugs into a structured list with repro steps and expected values.
2. Filter each bug through the decision table above.
3. Generate one pytest draft per eligible bug using the free model.
4. Run the drafts on a free server, 10 at a time.
5. Review each test, fix small issues, and keep only the ones that pass.
6. Repeat weekly and move approved tests into your shared regression folder.

Start with ten old bugs, not all 142. Generate tests. Review them. Run them on a free server. Keep the five that help. That is enough.