Compiling Is Not Enough: A Three-Stage Verification Workflow for Generated Code

The compile step is the weakest gate in the modern development pipeline. For human code, a clean build means the developer understood the language and its constraints. For agent-generated code, a clean build means nothing beyond syntax: the model produced text that the parser accepted. It says nothing about whether the function handles an empty list, whether the API call actually exists in the installed version, or whether the error path returns a useful message instead of a silent null.

This gap matters more as free model access becomes the default starting point for experiments. When you generate code through a service like MonkeyCode — which currently offers free models and a free server option for running agents — the cost of generation drops to zero, but the cost of verification does not. Someone still has to decide whether the output is worth merging. This article gives you a three-stage verification workflow that catches what CI misses, plus a script you can run against any generated patch.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Stage 1: Static Contract Checks

Before running anything, check what the code claims about the world. Generated code often references functions, fields, and environment variables that do not exist in your project. The compiler catches some of these, but not all: dynamic imports, string-based configuration keys, and optional chaining can hide missing contracts until runtime.

Run these checks first:

• List every import and confirm it resolves in the current dependency tree.
• Grep for environment variable names and verify each one is documented or set in your environment.
• Check every function call against the actual signature in the installed version, not the version the model trained on.
• Look for hardcoded paths, URLs, and ports that will break on a different machine.

A quick script can automate the first two:

#!/usr/bin/env bash
# contract-check.sh — verify generated code references real symbols
# Usage: ./contract-check.sh <file-or-directory>

TARGET="${1:-.}"

echo "== Imports =="
grep -rhoE '^(import|from) [a-zA-Z0-9_\.]+' "$TARGET" --include='*.py' | sort -u

echo "== Environment variables =="
grep -rhoE 'os\.(getenv|environ)\([^)]*\)' "$TARGET" --include='*.py' | sort -u

echo "== Hardcoded localhost references =="
grep -rnE '(localhost|127\.0\.0\.1|0\.0\.0\.0)' "$TARGET" --include='*.py' || true

echo "== TODO markers =="
grep -rnE '(TODO|FIXME|XXX|HACK)' "$TARGET" --include='*.py' || true

The output is a checklist, not a verdict. Each line is a contract the code assumes. Verify the assumption before moving on.

Stage 2: Behavioral Verification

Static checks confirm the code references real things. Behavioral verification confirms the code does the right thing. The fastest way is a differential test: run the old implementation and the new implementation against the same inputs, and compare outputs.

The key insight is that you do not need to understand the generated code to verify it. You need a reference behavior. If the function replaced an existing one, the old version is the reference. If it is new, write a small harness that asserts the behavior described in the prompt.

# verify_behavior.py — differential test for generated functions
import sys

# Import the old and new implementations
# old_impl and new_impl should expose the same callable
from old_module import process as old_process
from new_module import process as new_process

cases = [
    [],                        # empty input
    [1, 2, 3],                 # normal input
    [0, 0, 0],                 # repeated values
    [-1, 5, -3],               # negatives
    [10**6],                   # large value
    None,                      # null input
]

failures = 0
for case in cases:
    try:
        old_result = old_process(case)
    except Exception as e:
        old_result = f"EXCEPTION: {type(e).__name__}"

    try:
        new_result = new_process(case)
    except Exception as e:
        new_result = f"EXCEPTION: {type(e).__name__}"

    if old_result != new_result:
        failures += 1
        print(f"MISMATCH for input {case!r}:")
        print(f"  old: {old_result}")
        print(f"  new: {new_result}")

if failures:
    print(f"\n{failures} behavioral differences found.")
    sys.exit(1)
print("All cases matched.")

Run this before reading the generated code line by line. The mismatches tell you where to focus. If every case matches, the generated code is behaviorally equivalent to the old version, and the diff is safe to review for style and side effects.

Stage 3: Edge-Case Probing

Differential testing covers the inputs you thought of. Edge-case probing covers the inputs the model never considered. Generated code tends to handle the happy path well and collapse on the boundaries.

Probe these categories:

• Empty and null inputs. Empty string, empty list, None, missing keys.
• Type boundaries. Integers at the limits of the type, floats that lose precision, Unicode strings with emoji and combining characters.
• Concurrency. Two calls in parallel, reentrant calls, shared state across calls.
• Resource limits. Large inputs that exhaust memory, deep recursion that hits the stack limit, timeouts on slow operations.
• Failure injection. The underlying API returns an error, the network times out, the file is missing.

Each probe is a small test. The goal is not to prove correctness. The goal is to map the failure surface before the code reaches production.

Decision Table

Probe result	Action	Rationale
All differential cases match	Review for side effects	Behavior is preserved; check logging, state, and performance
One mismatch on an edge case	Fix the edge case	The model missed a boundary; patch it or reject
Mismatch on a normal input	Reject	The core behavior is wrong; do not patch around it
Static contract check fails	Reject	The code references something that does not exist
Concurrency probe fails	Reject or isolate	Race conditions are expensive to debug later
Failure injection handled gracefully	Accept with confidence	The model considered error paths
Failure injection crashes	Reject	The error path is untested or missing

Limitations

This workflow assumes you have an old implementation to compare against. For brand-new features, there is no reference behavior, and the differential test becomes a hand-written assertion suite. That is slower and requires more judgment.

The static contract check catches missing symbols but not wrong semantics. A function can exist, accept the right types, and still compute the wrong result. Behavioral testing covers that, but only for the inputs you chose.

The workflow also assumes the generated code is isolated. If the patch touches shared state, global configuration, or the database schema, the probes above are insufficient. You need integration tests and a staging environment.

Who should not use this: teams generating code for infrastructure changes, security-sensitive components, or financial calculations without an independent review layer. The workflow reduces risk. It does not eliminate it.

A Practical Starting Point

The cheapest way to learn this workflow is to generate a small patch, run the three stages, and observe where the model output breaks. Free access to models and a free server option make that experiment cost nothing but time. MonkeyCode currently provides both, which is why it is a reasonable place to start. Generate a utility function, run the contract check, write the differential test, and probe the edges. The failures you find will teach you more about verification than any tutorial.