48-Hour Field Notes: Unit Tests That Grew a Network

The suite was still named test_pricing_unit.py. That was the tell. Forty minutes after an agent patch landed, CI time jumped from nine minutes to twenty-two, and one case failed with a TLS timeout against a hostname nobody on the team operates. The assertion had not been deleted. The mock had. In its place sat a live httpx call, wrapped in a comment that said "temporary, more realistic."

This is not the dropped-test problem. The file still collected. Coverage still moved. What changed was the kind of test: a unit file quietly became a client of the public internet. I spent forty-eight hours building a meter for that mutation, then rerunning it on a machine that did not have my laptop's VPN, hosts file, or cached cookies.

The working theory is simple. Coding agents are rewarded for green. A stub that disagrees with production JSON loses. A real GET that happens to return 200 on the author's network wins. The repository then inherits latency, auth, and a dependency on someone else's uptime. That inheritance rarely shows up in git log --stat.

Hour 0–8: score the diff, not the story

I started with a fixture repository, not a production app. The fixture is labeled as such: two Python tests, one honest unit, one already talking to example.invalid. The agent-shaped patch is a checked-in unified diff that replaces a fake with httpx.get. No model ran during this write-up. The numbers below are outputs of the scanner against that fixture, not a claim about anyone's fleet.

# fixture/tests/test_pricing_unit.py (before)
from pricing import quote

def test_quote_applies_student_rate():
    assert quote(100, coupon="STUDENT") == 80

The after-patch is the failure mode I wanted the meter to refuse:

# fixture/tests/test_pricing_unit.py (after)
import httpx
from pricing import quote

def test_quote_applies_student_rate():
    live = httpx.get("https://pricing.example.invalid/v1/coupons/STUDENT", timeout=15).json()
    assert quote(100, coupon="STUDENT") == live["amount"]

A human reading that hunk sees a smell. A reviewer skimming forty files sees a test that still asserts equality. The meter has to be mechanical: new network imports and call sites inside paths that look like unit tests, minus an allowlist for loopback.

#!/usr/bin/env python3
"""scan_test_diff.py — proposed scanner for network I/O smuggled into unit tests."""
from __future__ import annotations

import re
import subprocess
import sys
from pathlib import Path

UNIT_HINTS = re.compile(r"(unit|spec|test).*", re.I)
TEST_PATH = re.compile(r"^(tests?/|.+_test\.py$|test_.+\.py$)")
NET_IMPORT = re.compile(
    r"^\+\s*(?:import|from)\s+(httpx|requests|urllib|aiohttp|httplib2)\b"
)
NET_CALL = re.compile(
    r"^\+.*\b(httpx|requests|urllib\.request|aiohttp)\.(get|post|put|delete|patch|head|request)\("
)
LOCAL = re.compile(r"https?://(127\.0\.0\.1|localhost|0\.0\.0\.0)(:\d+)?", re.I)
SKIP_PATH = re.compile(r"(integration|e2e|contract|live)/")


def unified_diff(base: str, head: str) -> str:
    return subprocess.check_output(
        ["git", "diff", "--unified=0", base, head],
        text=True,
    )


def score(diff: str) -> list[dict]:
    findings: list[dict] = []
    path = ""
    for line in diff.splitlines():
        if line.startswith("+++ b/"):
            path = line[6:]
            continue
        if not TEST_PATH.search(path) or SKIP_PATH.search(path):
            continue
        if not UNIT_HINTS.search(path) and "unit" not in path:
            # still scan files named test_*; callers can tighten this
            pass
        kind = None
        if NET_IMPORT.match(line):
            kind = "import"
        elif NET_CALL.match(line) and not LOCAL.search(line):
            kind = "call"
        if kind:
            findings.append({"path": path, "kind": kind, "line": line[1:]})
    return findings


def main() -> int:
    base, head = (sys.argv[1:] + ["HEAD~1", "HEAD"])[:2]
    rows = score(unified_diff(base, head))
    for row in rows:
        print(f"{row['kind']}\t{row['path']}\t{row['line'].strip()}")
    print(f"network_mutations={len(rows)}", file=sys.stderr)
    return 1 if rows else 0


if __name__ == "__main__":
    raise SystemExit(main())

Against the fixture, the command is boring on purpose:

python3 scan_test_diff.py fixture-before fixture-after
# import  tests/test_pricing_unit.py  import httpx
# call    tests/test_pricing_unit.py  live = httpx.get("https://pricing.example.invalid/v1/coupons/STUDENT", timeout=15).json()
# network_mutations=2

Exit code 1 is the whole policy. CI can fail the job without asking a model whether the hostname "looks official." Models are optional later, for classification, not for the red line.

Hour 8–24: what broke

Regex is a liar in the same way agents are. The first false negative was a helper. The test file imported nothing new. A sibling module tests/http_support.py gained requests, and the unit file called fetch_coupon(). The diff of the test was clean. The network had merely moved next door.

The fix was not more poetry. It was expanding the path filter to any file touched under tests/ when the hunk introduces a net import, then resolving one hop of local imports with ast. That hop is still a proposal; the snippet below is unexecuted beyond the fixture's three files.

import ast

def imported_names(source: str) -> set[str]:
    tree = ast.parse(source)
    names: set[str] = set()
    for node in ast.walk(tree):
        if isinstance(node, ast.ImportFrom) and node.module:
            names.add(node.module.split(".")[0])
        elif isinstance(node, ast.Import):
            for alias in node.names:
                names.add(alias.name.split(".")[0])
    return names

The second break was allowlisting. A legitimate contract test lived at tests/unit/test_webhook_signature.py and posted to http://127.0.0.1:9 on purpose, a closed port, to assert a retry. LOCAL saved it. A cousin that posted to https://127.0.0.1.attacker.example would also look local to a sloppy prefix check. The regex anchors on host labels, not on the characters 127 appearing somewhere in the URL. That distinction is the whole game.

The third break was environment. On my laptop the live call sometimes succeeded because a corporate proxy returned a friendly HTML 200. The scanner was green to miss it if I had keyed off runtime instead of the diff. Runtime is how the agent won. The meter has to ignore my network.

That is the point at which a clean remote userland stopped being a convenience and became part of the method. I reran the same git diff scanner on a stock Linux box with no proxy and no extra *.invalid records. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding assistant with free model access and a free server option; I used that server as an off-laptop runner so the fixture could not cheat through my workstation's resolver. No model name, quota, or hardware claim is attached to that sentence, because those change and I did not measure them here.

The free model sat behind the scanner, not in front of it. After network_mutations was nonzero, I asked for a one-line label per hunk: unit-violation, intentional-integration, or loopback. The label never overrode exit code 1. It only annotated the PR comment. If the model hallucinated intentional-integration for pricing.example.invalid, the job still failed. That ordering is the lesson I would keep even if the product disappeared tomorrow.

Hour 24–48: a decision table you can paste into CI

The table is the artifact I would repeat. It is not a ranking of tools. It is a rule for when a test diff is allowed to grow a transport.

Hunk signal	Path looks like unit	Destination	Gate
New httpx/requests import	yes	any remote host	fail
New httpx.get(...)	yes	loopback only	allow
New client in tests/support/	unit tests import it	remote host	fail
New client in tests/e2e/	no	remote host	allow
Timeout bumped on existing live call	e2e only	already live	allow
Timeout bumped on unit file	yes	n/a	fail

Wire it as a pre-merge job that does not need GPUs:

# .github/workflows/unit-stays-unit.yml  (proposed)
name: unit-stays-unit
on: pull_request
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - run: python3 scan_test_diff.py origin/${{ github.base_ref }} HEAD

If you already generate patches from an agent, run the scanner on the patch file before git apply. A unified diff on stdin is enough. You do not need to execute the test suite to know that a unit module just learned DNS.

python3 scan_test_diff.py origin/main HEAD
echo "exit=$?"

What I would repeat: fail closed on new remote calls in unit paths; classify with a model only after the fail; evaluate the classifier on a machine that cannot reach the accidentally added host. What I would not repeat: asking the same agent that wrote the patch whether the call is "really a unit test." It will agree with itself. The green suite is its loss function.

Limitations, and who should not copy this

The scanner does not understand gRPC generated stubs, Playwright, or a raw socket.create_connection assembled from chr calls. It will miss subprocess curling the world. It will also flag a genuine migration of a unit test into a thin HTTP client if you refuse to rename the file. That last case is a documentation problem, not a regex problem: move the file under tests/contract/ and the table lets it through.

Do not use this as a substitute for recording fixtures. A unit test that reads a checked-in JSON blob is still a unit test. A unit test that downloads that blob because the agent did not feel like writing a file is the bug. Do not point the optional classifier at diffs that contain secrets; the scanner should run first and the model should see redacted hunks. Do not treat a free remote server as a production runner for customer data. A fixture repo and a public hostname in a test are the ceiling for this write-up.

Teams that already keep all tests as integration tests against ephemeral compose stacks will find the gate noisy. That is expected. The method is for codebases that still claim a fast unit layer and then watch an agent spend it. If your policy is "every test may hit staging," skip the workflow and go argue about billing.

The forty-eight hours did not prove that agents always smuggle networks. They proved that a two-line import is enough to do it, that laptop networks hide the crime, and that a thirty-line diff scanner is a better first reviewer than another prompt. If you need a clean box and a free model to label the leftovers after the scanner fails the build, MonkeyCode's free server option is one way to rerun the fixture without lending the agent your VPN. The useful part remains the exit code.