DEV Community

Emery Yang
Emery Yang

Posted on

Screening AI-Generated Patches with a Free-Server Test Gate

#python #ai #testing #automation

Casey inherited a Python billing utility that normalized invoice rows. The script passed its old tests. A downstream parser started rejecting 2026-02-30 as a real date. The team had no budget for another model subscription. Casey wanted a small gate that would judge every AI-generated patch before anyone paid for tokens.

Casey had access to MonkeyCode's free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free model access meant the model calls could run without an initial token bill. The free server option meant the runner could live somewhere persistent for the duration of the test. Neither detail removed the need for a real evaluation. It only made the first pass cheaper.

The gate was not about style. It was about regression. A patch had to pass every existing test plus one new failing test for the invalid date. The case below uses Python and the standard library. The same loop works with any model endpoint that can return a complete file.

The failing test

The original module looked like this:

import re

def parse_date(value):
    if not isinstance(value, str):
        raise ValueError('date must be a string')
    match = re.fullmatch(r'[0-9]{4}-[0-9]{2}-[0-9]{2}', value)
    if not match:
        raise ValueError('date must use YYYY-MM-DD')
    year, month, day = map(int, match.groups())
    if not 1 <= month <= 12:
        raise ValueError('month out of range')
    return year, month, day
Enter fullscreen mode Exit fullscreen mode

The bug is narrow. The day value is never checked. That lets 2026-02-30 pass through, and a later system explodes.

import unittest
from invoice_normalizer import parse_date

class DateValidationTest(unittest.TestCase):
    def test_valid_date(self):
        self.assertEqual(parse_date('2026-08-13'), (2026, 8, 13))

    def test_impossible_date_raises(self):
        with self.assertRaises(ValueError):
            parse_date('2026-02-30')
Enter fullscreen mode Exit fullscreen mode

The gate

The harness copies the repository into a temporary sandbox, overwrites the candidate file, and runs the test suite. A candidate is accepted only when the new failing test passes and the old test still passes.

import json
import os
import shutil
import subprocess
import tempfile
import time
import urllib.request
from pathlib import Path

ROOT = Path(__file__).resolve().parent

def run_tests(sandbox):
    result = subprocess.run(
        ['python', '-m', 'unittest', 'test_invoice_date'],
        cwd=sandbox,
        capture_output=True,
        text=True,
    )
    return result.returncode == 0, result.stdout + result.stderr

def make_sandbox(candidate_file):
    target = Path(tempfile.mkdtemp(prefix='patch_gate_'))
    shutil.copytree(ROOT, target, ignore=shutil.ignore_patterns('.git', '__pycache__', '*.pyc'))
    shutil.copyfile(candidate_file, target / 'invoice_normalizer.py')
    return target

def generate_candidate(endpoint, api_key, model, original_code, failing_test):
    prompt = (
        'Return only the complete invoice_normalizer.py file. '
        'The test fails because parse_date accepts an impossible date.'
        + chr(10) + 'Original file:' + chr(10) + original_code
        + chr(10) + 'Failing test:' + chr(10) + failing_test
    )
    payload = {
        'model': model,
        'messages': [
            {'role': 'system', 'content': 'Return only Python code. No markdown fences.'},
            {'role': 'user', 'content': prompt},
        ],
    }
    request = urllib.request.Request(
        endpoint,
        data=json.dumps(payload).encode('utf-8'),
        headers={
            'Authorization': 'Bearer ' + api_key,
            'Content-Type': 'application/json',
        },
    )
    with urllib.request.urlopen(request) as response:
        body = response.read().decode('utf-8')
        response_data = json.loads(body)
        content = response_data['choices'][0]['message']['content']
        lines = content.splitlines()
        if lines and lines[0].startswith('```

'):
            lines = lines[1:]
        if lines and lines[-1].strip() == '

```':
            lines = lines[:-1]
        return chr(10).join(lines)

def mock_generator(original_code, failing_test):
    return '''import re

def parse_date(value):
    if not isinstance(value, str):
        raise ValueError('date must be a string')
    match = re.fullmatch(r'[0-9]{4}-[0-9]{2}-[0-9]{2}', value)
    if not match:
        raise ValueError('date must use YYYY-MM-DD')
    year, month, day = map(int, match.groups())
    if not 1 <= month <= 12:
        raise ValueError('month out of range')
    return year, month, day
'''

def main():
    use_mock = os.environ.get('GATE_MOCK', '1') != '0'
    original_code = (ROOT / 'invoice_normalizer.py').read_text(encoding='utf-8')
    failing_test = (ROOT / 'test_invoice_date.py').read_text(encoding='utf-8')
    if use_mock:
        candidate = mock_generator(original_code, failing_test)
    else:
        endpoint = os.environ.get('MODEL_ENDPOINT')
        api_key = os.environ.get('MODEL_API_KEY', '')
        model = os.environ.get('MODEL_NAME', 'default')
        if not endpoint:
            raise SystemExit('MODEL_ENDPOINT is required when GATE_MOCK is 0')
        candidate = generate_candidate(endpoint, api_key, model, original_code, failing_test)
    candidate_file = Path(tempfile.mkdtemp(prefix='candidate_')) / 'invoice_normalizer.py'
    candidate_file.write_text(candidate, encoding='utf-8')
    sandbox = make_sandbox(candidate_file)
    passed, output = run_tests(sandbox)
    result = {
        'timestamp': time.strftime('%Y-%m-%dT%H:%M:%S'),
        'passed': passed,
        'output': output,
    }
    print(json.dumps(result, indent=2))
    return 0 if passed else 1

if __name__ == '__main__':
    raise SystemExit(main())
Enter fullscreen mode Exit fullscreen mode

The generate_candidate function assumes an OpenAI-compatible response shape. If the free model access exposes a different SDK, replace that one function and keep the rest of the gate.

Host the runner on the free server option

The next file turns the gate into a tiny HTTP service. A teammate can POST an evaluation request without installing the repository locally.

import json
import os
import subprocess
from http.server import BaseHTTPRequestHandler, HTTPServer
from pathlib import Path

ROOT = Path(__file__).resolve().parent

def run_gate():
    result = subprocess.run(
        ['python', 'gate.py'],
        cwd=ROOT,
        capture_output=True,
        text=True,
    )
    return result.returncode == 0, result.stdout + result.stderr

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        raw = self.rfile.read(int(self.headers.get('Content-Length', '0')))
        payload = json.loads(raw or b'{}')
        ok, output = run_gate()
        self.send_response(200)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        body = json.dumps({'passed': ok, 'output': output})
        self.wfile.write(body.encode('utf-8'))

    def log_message(self, format, *args):
        pass

if __name__ == '__main__':
    port = int(os.environ.get('PORT', '8000'))
    server = HTTPServer(('0.0.0.0', port), Handler)
    print(f'gate server listening on {port}')
    server.serve_forever()
Enter fullscreen mode Exit fullscreen mode

Keep the service small. The free server option is a starting point, not a permanent production host.

Run it

First run the offline mock to prove the gate rejects the known-bad candidate.

GATE_MOCK=1 python gate.py
Enter fullscreen mode Exit fullscreen mode

The expected result is passed: false because the mock still accepts the impossible date.

Then point the same loop at the free model access.

GATE_MOCK=0 MODEL_ENDPOINT=your_endpoint_here MODEL_API_KEY=your_key_here MODEL_NAME=your_model_name python gate.py
Enter fullscreen mode Exit fullscreen mode

The exact endpoint, key, and model name come from the access details. No model name, quota, or hardware assumption is hard-coded here.

To expose the runner, deploy gate_server.py to the free server option and send a POST.

PORT=8000 python gate_server.py
Enter fullscreen mode Exit fullscreen mode 
curl -X POST http://localhost:8000/evaluate
Enter fullscreen mode Exit fullscreen mode

What the results mean

Result Meaning Next action
pass Candidate handles the new date test Keep the patch for human review
fail Candidate still accepts impossible dates Change the prompt or discard the candidate
error Candidate does not import or breaks a test Treat as fail and log the traceback

A single pass is not a proof. Add more edge cases after the first successful run.

Limitations and who should not use this

A one-test gate is narrow. It will not catch every regression. Free-tier quotas and server availability can change. Do not send proprietary or regulated code to an external model endpoint without a separate review. Do not run production patches automatically from this harness. Teams that need statistical model comparisons need repeated runs, controlled prompts, and larger test suites. Teams with strict data boundaries should run an equivalent gate on infrastructure they already trust.

Casey's gate costs one failing test and a few standard-library files. If you already have MonkeyCode free model access and a free server option, copy the gate into a scratch repository and point it at one real bug before you buy any token plan.

Top comments (0)