Test the Process, Not the Module: A Black-Box Characterization Gate for a Repo You Can't Import

A messy repo often cannot be imported safely. Import-time side effects, circular packages, and hard-coded paths block in-process tests. The fix is a black-box gate: run the CLI, pin what it emits, then change one thing.

This walkthrough builds a probe runner in about 80 lines of Python. It pins exit codes, stdout, and output-file digests. Cheap. Slow-ish. Honest.

Conclusion first

In legacy code, process-level characterization is the cheapest first test. It needs zero refactoring to exist. It also survives the refactor, which unit tests usually do not.

Three cases, pinned well, are worth more than fifty unit tests over a module you cannot import.

Why import-level tests fail here

Circular imports. The package imports fine at runtime, but only in one entry order. A test file that imports a submodule directly deadlocks or explodes.

Import-time work. Config reads, log setup, and network calls fire on import. Your test now needs the same environment as production.

Missing seams. Functions take no arguments and read module globals. To test one, you monkeypatch three modules.

All three are reasons to test at the boundary, not reasons to refactor first. Refactoring before you can observe behavior is how features break silently.

Step 1: Inventory entry points

List what invokes the code from outside. Be boring about it.

1. Console scripts from pyproject.toml or setup.py.
2. python -m package module entry points.
3. Cron jobs and shell wrappers in deploy/ or scripts/.
4. HTTP handlers if a framework routes to them.

Write the list down before touching code. Each line is a future probe case. Each line you miss is a blind spot.

Pick one entry point for the first pass. The one with the most production traffic, not the most fun.

Step 2: Build the argument matrix

You need inputs that exercise branches: empty input, one row, malformed row, missing file, bad flag. Reading argparse code to find them is slow and error-prone.

This is where a free model helps. Paste the --help output and the parser block. Ask for a table of argv vectors and expected fixture files. It proposes cases; you reject the impossible ones.

Drafting that matrix is the one place I used MonkeyCode's free model access in this workflow. No repo code left my machine beyond one parser file.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A rule that saved me time: reject any proposed case that needs a real credential, a live network call, or a date in the past. Those are environmental, not behavioral.

Step 3: Pin normalized observations

The runner executes the CLI in a fresh temp directory. It captures return code, scrubbed stdout, first stderr lines, and a digest of every file written.

# probe.py — sketch, not executed against your repo
import hashlib, json, os, re, subprocess, sys, tempfile
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
MANIFEST = ROOT / "tests" / "probe" / "manifest.json"
ENTRY = [sys.executable, "-m", "legacy_cli"]
NOISE = [(re.compile(r"0x[0-9a-f]+"), "<ADDR>"),
         (re.compile(r"\d+\.\d+s"), "<DUR>")]

def scrub(text, workdir):
    out = text.replace(str(workdir), "<WORK>")
    for pattern, token in NOISE:
        out = pattern.sub(token, out)
    return out.rstrip() + "\n"

def digests(workdir, inputs):
    rows = {}
    for path in sorted(workdir.rglob("*")):
        if not path.is_file():
            continue
        rel = str(path.relative_to(workdir))
        if rel in inputs:
            continue
        rows[rel] = hashlib.sha256(path.read_bytes()).hexdigest()[:16]
    return rows

def run_case(case):
    with tempfile.TemporaryDirectory() as tmp:
        workdir = Path(tmp)
        for rel, text in case.get("files", {}).items():
            target = workdir / rel
            target.parent.mkdir(parents=True, exist_ok=True)
            target.write_text(text)
        env = {**os.environ, "TZ": "UTC", "LC_ALL": "C",
               "PYTHONHASHSEED": "0", "NO_COLOR": "1"}
        proc = subprocess.run(ENTRY + case["argv"], cwd=workdir, env=env,
                              capture_output=True, text=True,
                              timeout=case.get("timeout", 20))
        return {
            "returncode": proc.returncode,
            "stdout": scrub(proc.stdout, workdir),
            "stderr_head": scrub(proc.stderr, workdir).splitlines()[:3],
            "outputs": digests(workdir, set(case.get("files", {}))),
        }

The manifest stores case inputs plus the pinned returncode, stdout, and outputs. It does not store file bytes. Digests keep diffs readable and repos small.

Scrub aggressively: absolute paths, object addresses, durations, thread ids, memory sizes. Every unscrubbed variable is a future false failure.

Run each case twice before pinning. If the two runs differ, the case is noisy. Exclude it or fix the source of drift first.

Step 4: Gate on the manifest

Add --check mode. It reruns every case and compares against the manifest. Any difference is a nonzero exit.

Wire it into pre-commit and CI. Now you have a behavioral contract that predates your refactor.

Manifest shape, deliberately flat:

{
  "entry": "legacy_cli",
  "cases": [
    {"id": "empty-input", "argv": ["report", "in/empty.csv"],
     "files": {"in/empty.csv": ""},
     "expect": {"returncode": 0, "stdout": "rows: 0\n",
                "outputs": {"out/report.json": "9f2c41ab77d0e315"}}}
  ]
}

Keep the manifest in the repo. Treat edits to it as behavior changes, never as cleanup.

If a failing probe is genuinely a bug fix, update the manifest in its own commit. That commit message is your audit trail.

Step 5: Make the smallest safe change

With the gate green, the refactor rule is narrow.

1. Change one behavior axis per commit: naming, or structure, or I/O. Not two.
2. Keep the manifest untouched in structural commits. If it must change, stop and re-plan.
3. Revert on any probe diff, even one that looks like an improvement.

Probes are coarse. They tell you the process changed, not which function did it. So bisect with commits, not with debugging sessions.

Long probe suites are good candidates for a cheap always-on box. Scheduled reruns on MonkeyCode's free server option kept my laptop free during a night of bisecting; check current terms and limits before you rely on any free tier for anything load-bearing.

When to use which test

Repo shape	Probe first?	Why
CLI tool, unimportable package	Yes	Boundary is stable, internals are not
Package with clean public API	No	In-process tests localize failures better
Cron or batch job with file output	Yes	Output digests are a cheap contract
Library used by other services	Partly	Probe the wrappers, unit-test the pure core
Interactive TUI	No	Needs a pty harness; probes flake badly

Limitations and who should skip this

Probes are slow: one process per case. Ten cases is fine; four hundred is a build problem.

Localization is weak. A digest mismatch names a file, not a line.

Coverage is exactly what you enumerate. Untested flags stay untested, and silence looks like safety.

Do not use this approach if your code needs interactive input, if stdout carries secrets, or if inputs include licensed data you cannot copy into a scratch directory. Do not send private source to a free model without checking its terms.

And skip all of this if your module imports cleanly and has real unit tests. You already have the better tool.

The code above is a sketch. Adapt ENTRY and the scrub rules to your repo, then start with five cases. Five honest probes beat a refactor you cannot verify.