A generated patch can look correct and still fail in three ways that a prompt screenshot cannot show: it changes the contract, it imports a module the rest of the codebase does not allow, or it only works on the single example in the prompt. If you copy that patch into a real repository, you become the integration test, usually in a pull request late at night.
This article walks through a short validation loop that treats model output as untrusted input. The loop runs three checks before a candidate file can touch your working tree: contract, execution, and regression. The script is plain Python with no model-specific SDK, so the same gate works for output from any endpoint or manual file.
The three failure classes
Contract drift is the most common silent break. A model may rename a function, change its arguments, or return a different type while leaving the prompt's happy path intact. A static AST check can catch missing symbols before the code is ever imported.
Forbidden reach is the second issue. A coding model may decide to read the filesystem, shell out to git, or call a network when the task was supposed to be pure. If you allow generated code to run, its imports matter as much as its output.
Happy-path-only logic is the third issue. A candidate can print the expected string for one input and fail on a normal case, an empty string, or a Unicode slug. A regression gate runs a small test file against the candidate in isolation.
The gate script
Save the following as gate.py. It parses the candidate, checks the public functions and imports, then runs two optional code paths in a temporary directory.
import ast
import json
import pathlib
import subprocess
import sys
import tempfile
from dataclasses import dataclass
@dataclass(frozen=True)
class GateSpec:
candidate_path: str
expected_functions: frozenset[str]
forbidden_modules: frozenset[str]
probe_script: str
regression_test: str
timeout_seconds: int = 10
def parse_functions(source):
tree = ast.parse(source)
return frozenset(
node.name
for node in ast.walk(tree)
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))
)
def parse_modules(source):
tree = ast.parse(source)
modules = set()
for node in ast.walk(tree):
if isinstance(node, ast.Import):
modules.update(alias.name.split('.')[0] for alias in node.names)
elif isinstance(node, ast.ImportFrom) and node.module:
modules.add(node.module.split('.')[0])
return frozenset(modules)
def run_in_temp(candidate, code, timeout):
with tempfile.TemporaryDirectory() as tmp:
root = pathlib.Path(tmp)
(root / 'candidate.py').write_text(candidate, encoding='utf-8')
(root / 'probe.py').write_text(code, encoding='utf-8')
return subprocess.run(
[sys.executable, 'probe.py'],
capture_output=True,
text=True,
timeout=timeout,
cwd=root,
)
def evaluate(spec):
candidate = pathlib.Path(spec.candidate_path).read_text(encoding='utf-8')
failures = []
functions = parse_functions(candidate)
modules = parse_modules(candidate)
missing = sorted(spec.expected_functions - functions)
if missing:
failures.append({'gate': 'contract', 'reason': 'missing functions', 'details': missing})
forbidden = sorted(modules & spec.forbidden_modules)
if forbidden:
failures.append({'gate': 'contract', 'reason': 'forbidden import', 'details': forbidden})
if spec.probe_script:
try:
result = run_in_temp(candidate, spec.probe_script, spec.timeout_seconds)
if result.returncode != 0:
failures.append({'gate': 'execution', 'reason': 'probe failed', 'details': result.stderr.strip()[-500:]})
except subprocess.TimeoutExpired:
failures.append({'gate': 'execution', 'reason': 'probe timed out', 'details': ''})
if spec.regression_test:
test_code = '''import unittest
import candidate
''' + spec.regression_test + '''
if __name__ == '__main__':
unittest.main()
'''
try:
result = run_in_temp(candidate, test_code, spec.timeout_seconds)
if result.returncode != 0:
failures.append({'gate': 'regression', 'reason': 'test failed', 'details': result.stderr.strip()[-500:]})
except subprocess.TimeoutExpired:
failures.append({'gate': 'regression', 'reason': 'test timed out', 'details': ''})
return {'accepted': not failures, 'failures': failures, 'functions': sorted(functions), 'modules': sorted(modules)}
if __name__ == '__main__':
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--candidate', required=True)
parser.add_argument('--spec', required=True)
args = parser.parse_args()
spec_data = json.loads(pathlib.Path(args.spec).read_text())
spec = GateSpec(
candidate_path=args.candidate,
expected_functions=frozenset(spec_data.get('expected_functions', [])),
forbidden_modules=frozenset(spec_data.get('forbidden_modules', [])),
probe_script=spec_data.get('probe_script', ''),
regression_test=spec_data.get('regression_test', ''),
timeout_seconds=spec_data.get('timeout_seconds', 10),
)
print(json.dumps(evaluate(spec), indent=2))
The spec file is a small JSON object with these keys: expected_functions, forbidden_modules, probe_script, regression_test, and timeout_seconds. The probe script imports from candidate and should exit non-zero when the output is wrong. The regression test is the body of a unittest.TestCase class.
Run it like this:
python gate.py --candidate model_output.py --spec spec.json
The output is a JSON summary. If accepted is false, each failure names the gate, the reason, and the last part of the relevant output.
Where a free endpoint fits
To generate the candidate file in this example, you can use any endpoint you already have. If you want a no-cost path, MonkeyCode's free model access and free server option let you generate and run these checks without provisioning local inference. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The gate itself does not depend on MonkeyCode, and you can replace the candidate source with output from any model.
The useful property is that the gate runs separately from the model. You can send one prompt to several endpoints, write each response to model_output.py, and let the same spec decide whether any of them deserves a closer review.
Limits to keep in mind
The execution gate is not a sandbox. It runs generated code inside a temporary directory, but a malicious or careless candidate can still consume CPU, write outside its directory if it has the permissions, or try to access the network. Always run this on a disposable machine or container, and set a short timeout. Do not feed it secrets, deploy keys, or a mounted working tree.
The AST gate only sees what is present at parse time. It will not catch dynamic imports, obfuscated calls, or behavior hidden behind an eval. It is a fast noise filter, not a security guarantee.
The regression gate is only as strong as the cases you write. If you encode the same happy path the model saw, the loop can reassure you while still letting a bad patch through.
Who should skip this
Skip this loop if you already have a CI pipeline with isolated containers, dependency scanning, and human review. Skip it if the candidate needs a GPU, a long-running service, or credentials to complete the task. The harness is most useful for small, self-contained functions where the failure cost is high and the model output changes often.
The metric that matters is not how many patches pass; it is how often the gates reject a patch that looked clean but would have broken a hidden case. Try the loop with three candidate outputs for the same prompt and compare the rejection count. If everything passes, it is time to write better regression tests, not to trust the model more.
Top comments (1)
I like that this treats the model output as a file crossing a boundary. The import check is the one I would keep even for small scripts, because it catches the model reaching for filesystem or shell access before the test suite gets a vote. Do you ever snapshot the rejected candidates to tune the regression set later?