A Deploy Smoke Test for Model-Generated Code

A model-generated Flask app can look clean in a diff: syntax valid, routes matching, local run returning 200. Then it fails on the server because an environment variable is missing, a dependency is pinned to a version that does not exist on the target, or the health check path is different. The model did not lie; it simply had never been asked to survive a real runtime.

The gap is not just prompt quality. Unit tests and static review usually happen before code meets the deployment environment, so they can miss the exact failures that appear after git push. This article shows a small deploy smoke test that closes that gap. It works with any base URL, including a free server, and is deliberately boring: no benchmark claims, no leaderboard, just a repeatable check.

Two availability claims matter for this workflow: MonkeyCode provides free model access and a free server option. Those are operator-supplied facts, so this article does not assume specific model names, quotas, uptime, or hardware. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The rest of the article remains useful if you plug a different provider into the same script.

What the artifact does

The script reads a small smoke.json manifest from a project directory, runs optional build commands, then probes routes on a deployed URL. A check passes when the HTTP status matches the expected value and, optionally, when the response body contains a substring.

This is not a unit test or a substitute for integration tests. It is a pre-merge deploy check. It catches the class of errors that appear only after code reaches a running environment.

Because the script accepts a base URL, the target can be local or remote. If you have a free server option, point --url at that server after deploying the generated change. The script does not need to know how the deployment happened; it only needs an HTTP endpoint.

The runner

Create smoke_deploy.py:

#!/usr/bin/env python3
"""Deploy smoke test for model-generated changes.

Usage:
  python smoke_deploy.py --dir ./model_output/app --url https://your-free-server.example
"""

import argparse
import json
import subprocess
import sys
import urllib.error
import urllib.request
from pathlib import Path


def load_manifest(path):
    with open(path, 'r', encoding='utf-8') as f:
        return json.load(f)


def run_build(step, cwd):
    result = subprocess.run(
        step,
        cwd=cwd,
        shell=True,
        capture_output=True,
        text=True,
    )
    if result.returncode != 0:
        print(f'BUILD FAILED: {step}')
        print('STDOUT:')
        print(result.stdout)
        print('STDERR:')
        print(result.stderr)
        sys.exit(1)
    print(f'build ok: {step}')


def probe(url, expected_status, contains=None, timeout=5):
    req = urllib.request.Request(
        url,
        headers={'User-Agent': 'smoke-deploy/0.1'},
    )
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            status = resp.status
            body = resp.read(4096).decode('utf-8', errors='replace')
    except urllib.error.HTTPError as e:
        status = e.code
        body = e.read(4096).decode('utf-8', errors='replace')

    print(f'{url} -> {status} (expected {expected_status})')
    if status != expected_status:
        return False, body
    if contains and contains not in body:
        return False, body
    return True, body


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--dir', required=True)
    parser.add_argument('--url', required=True)
    args = parser.parse_args()

    project = Path(args.dir)
    manifest_path = project / 'smoke.json'
    if not manifest_path.exists():
        print(f'missing {manifest_path}')
        sys.exit(2)

    manifest = load_manifest(manifest_path)

    for step in manifest.get('build', []):
        run_build(step, project)

    failures = []
    for check in manifest.get('checks', []):
        route = check.get('route', '/')
        expected = check.get('status', 200)
        contains = check.get('contains')
        ok, body = probe(args.url.rstrip('/') + route, expected, contains)
        if not ok:
            failures.append({
                'route': route,
                'expected': expected,
                'contains': contains,
                'body_snippet': body[:200],
            })

    if failures:
        print('SMOKE FAILURES:')
        for failure in failures:
            print(json.dumps(failure, indent=2))
        sys.exit(1)

    print('all smoke checks passed')


if __name__ == '__main__':
    main()

Add a manifest next to the generated app:

{
  "build": ["python -m py_compile app.py"],
  "checks": [
    {"route": "/health", "status": 200, "contains": "ok"},
    {"route": "/", "status": 200, "contains": "Hello"}
  ]
}

A practical loop

1. Put the model-generated change into a clean directory such as ./model_output/app.
2. Add smoke.json with one build step and two or three routes that your service must expose.
3. Deploy that directory to the free server using whatever deploy mechanism your provider exposes.
4. Run:

python smoke_deploy.py --dir ./model_output/app --url https://your-free-server.example

1. When a check fails, read the route, expected status, and body snippet. Feed that failure back to the model as a concrete bug report, or fix it manually. A failed check is more useful than a vague "the code didn't work" prompt.

The value of this loop is separation of concerns: the model generates, the server runs, and the smoke test checks runtime behavior. The same manifest can be reused across model candidates, so you are comparing deploy-time viability, not vibes.

Limitations

The script checks only HTTP status and body substrings. It does not validate authorization, database migrations, concurrency, race conditions, SQL injection, or cost amplification. A 200 from /health does not mean the service is correct under real traffic.

Free server options may have cold starts, request limits, or ephemeral deployments. Build retries and warm-up delays into your own workflow; do not assume production-like stability. The script also assumes the service is reachable over HTTP, so it is not suitable for non-HTTP workers, long-running jobs, or binaries that only exit with a status code.

Do not use this as the only check for production, regulated workloads, code that touches secrets, or endpoints that mutate data. It is a triage tool, not a security review.

Who should not use it

Skip this if you are shipping a production-critical service, handling sensitive data, or evaluating a model for tasks where a deploy check cannot reveal correctness. Also skip it if your service has no HTTP surface; the script will produce false confidence.

If you are comparing free coding models on small stateless apps, however, the deploy smoke test adds a missing signal. Start with one non-critical branch and a three-route manifest.