Build a Tiny Gate for AI-Generated Code Before You Merge It

The pull request looked safe.
A model had written a small parsing helper.
The diff was short.
The tests passed locally.

Then CI failed on a hidden edge case.

The problem was not intelligence.
It was trust without a gate.

The failure mode

Most AI code demos show a polished answer.
They rarely show the second run that breaks.

A code review needs a check for generated code.
It does not need a big agent framework.
It needs three things:

1. A fixed prompt.
2. A set of public tests.
3. A set of hidden tests the model never sees.

Hidden tests matter.
They catch memorized answers that only pass the examples in front of the model.

A minimal harness

Here is a small harness for one function.
It writes the generated code to a temp directory, runs it in a subprocess, and reports failures.

import json
import subprocess
import tempfile
from pathlib import Path

def call_model(prompt: str) -> str:
    # Wire this to your model endpoint.
    raise NotImplementedError('replace with your provider adapter')

def evaluate(model_name: str, task: dict) -> dict:
    code = call_model(task['prompt'])

    with tempfile.TemporaryDirectory() as tmp:
        code_path = Path(tmp) / 'solution.py'
        code_path.write_text(code)

        tests = task['tests'] + task.get('hidden_tests', [])
        test_code = f'''
import json
from solution import {task['function_name']} as fn

cases = {tests!r}
failed = []

for inp, expected in cases:
    try:
        got = fn(inp)
    except Exception as e:
        failed.append({{'input': inp, 'error': str(e)}})
        continue
    if got != expected:
        failed.append({{'input': inp, 'expected': expected, 'got': got}})

print(json.dumps({{'total': len(cases), 'failed': failed}}))
'''
        test_path = Path(tmp) / 'run_tests.py'
        test_path.write_text(test_code)

        try:
            out = subprocess.run(
                ['python', str(test_path)],
                capture_output=True,
                text=True,
                timeout=5,
                cwd=tmp,
            )
            result = json.loads(out.stdout)
        except Exception as e:
            return {
                'model': model_name,
                'task': task['name'],
                'status': 'harness_error',
                'error': str(e),
            }

    total = result['total']
    passed = total - len(result['failed'])
    return {
        'model': model_name,
        'task': task['name'],
        'status': 'ok',
        'total': total,
        'passed': passed,
        'failed': result['failed'],
    }

This is not a security boundary.
Generated code runs in a subprocess, but it still has filesystem and network access on your machine.
For untrusted code, add a container or a VM.

A task worth testing

Most toy prompts are too easy.
A good first test has a small spec with several edge cases.

The task: parse an ISO 8601 duration and return total seconds.

TASK = {
    'name': 'iso_duration_parser',
    'function_name': 'parse_iso_duration',
    'prompt': (
        'Write a Python function parse_iso_duration(s) that accepts '
        'ISO 8601 duration tokens in the form PnDTnHnMnS where n is an integer. '
        'Return total seconds as an int. Use only the standard library. '
        'Assume the input is valid.'
    ),
    'tests': [
        ('PT1H30M', 5400),
        ('PT45M', 2700),
        ('P1D', 86400),
        ('PT0S', 0),
    ],
    'hidden_tests': [
        ('PT1H', 3600),
        ('PT90M', 5400),
        ('P1DT2H', 93600),
        ('PT1M30S', 90),
    ],
}

Run it with any model adapter.
Then read three numbers:

• passed: the minimum bar.
• failed: the interesting part.
• harness_error: the model produced code that did not even run.

A model that looks great on public tests can still fail hidden ones.
That is the point.

What to do with the results

Result	What it usually means
Public tests fail	The model ignored part of the spec.
Public tests pass, hidden tests fail	The code pattern matches, but it missed edge cases.
Harness error	The output was not valid runnable code for this task.
All tests pass	The model cleared this tiny gate. It still does not prove general reliability.

Do not turn this into a leaderboard.
It is a smoke test, not a benchmark.

Where a free model tier and free server help

The harness is cheap to run.
The first cost barrier is usually model access and somewhere to execute the prompt.

MonkeyCode offers free model access and a free server option.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.

That is useful for this workflow if you want to start without paying for API credits or setting up a hosted runner.
I will not quote model names, quotas, or latency numbers here because those details change.

Limitations

This approach has real limits.

• It tests one small function, not a codebase.
• It runs generated code outside a strict isolation boundary.
• It does not check security, style, or whether the code fits your team's patterns.
• It cannot prove a model is safe for confidential code.
• Free access may change or include resource limits.

Use it before a code review.
Do not use it instead of one.

Who should not use this

Skip the harness if:

• You are evaluating proprietary code and cannot redact it.
• You need guaranteed latency, support, or compliance.
• You expect a free tier to behave like a paid production service.
• You need to test multi-file changes or non-deterministic output.

Start smaller than you think

Pick one function you already own.
Write four public tests and three hidden tests.
Run the harness.

The gate is not about trusting the model.
It is about making trust cheap to verify.

If you want a zero-cost place to run that first test, MonkeyCode's free model access and free server option are a low-friction starting point.