Freeze the Export Surface Before a Messy Refactor: A Manifest Gate in Python

The smallest safe change is a measurable claim. Most refactors fail because nobody defined "small."

A rename here. A default flipped there. Six weeks later an import breaks in a repo you forgot existed. The fix is not more discipline. It is a machine-checked baseline of what your package exposes, taken before the first edit.

This post builds that baseline with 60 lines of stdlib Python. Then it wraps a characterization-test step and a CI diff around it, so "unchanged surface" becomes a pass/fail signal.

The failure mode this prevents

Messy repos hide their public API inside their internals. Nothing marks a function as load-bearing. So a plausible cleanup deletes a name that four other packages import.

Unit tests rarely catch this. They test behavior through the same edited code. The import graph is what breaks first, and it breaks downstream.

An export surface manifest records the contract instead of the behavior. It answers one narrow question. Did the names, shapes, and arities your package publishes change?

Step 1 — Record the surface on a clean branch

Start from a branch with no local edits. Parse every module, prefer __all__, and fall back to non-underscore top-level names.

#!/usr/bin/env python3
"""surface.py - freeze a package's public export surface as JSON."""
import ast
import hashlib
import json
import sys
from pathlib import Path


def exported_names(tree):
    """__all__ wins; otherwise every non-underscore top-level name."""
    for node in tree.body:
        if isinstance(node, ast.Assign):
            for target in node.targets:
                if isinstance(target, ast.Name) and target.id == "__all__":
                    try:
                        return sorted(ast.literal_eval(node.value))
                    except ValueError:
                        return []
    names = []
    for node in tree.body:
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
            if not node.name.startswith("_"):
                names.append(node.name)
        elif isinstance(node, ast.Assign):
            for target in node.targets:
                if isinstance(target, ast.Name) and not target.id.startswith("_"):
                    names.append(target.id)
    return sorted(names)


def describe(node):
    if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
        return {
            "kind": "fn",
            "pos": [a.arg for a in node.args.args],
            "kwonly": [a.arg for a in node.args.kwonlyargs],
            "defaults": len(node.args.defaults),
            "async": isinstance(node, ast.AsyncFunctionDef),
        }
    if isinstance(node, ast.ClassDef):
        return {"kind": "class", "bases": [ast.unparse(b) for b in node.bases]}
    return {"kind": "value"}


def walk(root: Path):
    manifest = {}
    for path in sorted(root.rglob("*.py")):
        if "test" in path.parts or path.name == "surface.py":
            continue
        try:
            tree = ast.parse(path.read_text(encoding="utf-8"))
        except SyntaxError:
            continue
        public = set(exported_names(tree))
        entries = {}
        for node in tree.body:
            name = getattr(node, "name", None)
            if name in public:
                entries[name] = describe(node)
        if entries:
            manifest[".".join(path.with_suffix("").parts)] = entries
    return manifest


if __name__ == "__main__":
    manifest = walk(Path(sys.argv[1]))
    blob = json.dumps(manifest, sort_keys=True).encode()
    out = {"digest": hashlib.sha256(blob).hexdigest()[:16], "modules": manifest}
    Path("surface.json").write_text(json.dumps(out, indent=2, sort_keys=True))
    print(f"digest={out['digest']} modules={len(manifest)}")

Run it and commit the artifact. This file is the refactor's baseline, not a build product.

python surface.py src/
# digest=4f9c1ab2e07d5531 modules=23
git add surface.json
git commit -m "chore: freeze export surface"

One rule matters here. Never regenerate surface.json on the refactor branch before you diff it. Overwrite the baseline and the gate is decoration.

Step 2 — Write three characterization tests, not thirty

Tests come before the move. But only for the seam you are about to touch.

Pick the function you intend to extract. Capture three calls that already work: the happy path, one edge input, one error path. Assert on outputs, not on internals.

def test_render_report_matches_golden():
    out = render_report(ROWS, width=72)
    assert out.splitlines()[0] == "| id | amount |"
    assert out.count("\n") == len(ROWS) + 2

def test_render_report_empty_rows():
    assert render_report([], width=72) == "| id | amount |\n"

def test_render_report_rejects_negative_width():
    with pytest.raises(ValueError):
        render_report(ROWS, width=-1)

Three tests pin behavior at the boundary. The manifest pins the name. Together they cover the two ways a refactor hurts callers.

If a test cannot fail, delete it. Assert something you could plausibly break.

Step 3 — Make the smallest change that still helps

Small means one seam per commit. Move the function into a new module. Keep the old import path alive with a re-export.

# src/reports/render.py
from .formatting import render_report  # re-export keeps the old path valid

__all__ = ["render_report"]

Do not rename the public function in the same commit. Do not change a default value. Do not touch the test file. One intent, one diff.

Step 4 — Diff the manifest in CI

Now compare the old artifact against the current tree. Any line here is a contract change that needs its own justification.

# surface_diff.py
import json
import sys


def flat(path):
    data = json.load(open(path))["modules"]
    return {f"{m}.{n}": json.dumps(v, sort_keys=True)
            for m, ents in data.items() for n, v in ents.items()}


old, new = flat(sys.argv[1]), flat(sys.argv[2])
drift = [k for k in sorted(set(old) | set(new)) if old.get(k) != new.get(k)]
for key in drift:
    print(f"DRIFT {key}\n  before: {old.get(key)}\n  after:  {new.get(key)}")
sys.exit(1 if drift else 0)

Wire it into a pre-commit hook so the check runs before review, not after.

# .pre-commit-config.yaml (local hook)
- repo: local
  hooks:
    - id: surface-gate
      name: export surface gate
      entry: bash -c 'git show HEAD:surface.json > /tmp/old.json && python surface.py src/ >/dev/null && python surface_diff.py /tmp/old.json surface.json'
      language: system
      pass_filenames: false

A pure internal move produces zero drift lines. That is the whole payoff. You now have evidence, not a vibe.

Where a model and a free server fit

Two steps above are mechanical but slow. Drafting the three characterization tests and reading a drift report both benefit from a model.

I use MonkeyCode, an open-source coding-assistant project, for exactly those two jobs. The project currently offers free model access and a free server option, per its own terms. Its advertised free allowance at the time of writing is 10 million tokens; confirm the current figure on the project page before you plan around it, because quotas and terms change.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use it for the boring parts only:

1. Paste one function plus its callers. Ask for three characterization tests that assert on outputs.
2. Run those tests locally. Discard any test that passes against a deliberately broken build.
3. Paste the drift report. Ask which single edit explains each line.
4. Re-run the suite on a clean machine before merging.

The free server option matters for step 4. A clean environment catches sys.path leftovers your laptop hides. Keep the manifest gate in your own CI regardless. Do not outsource the pass/fail decision to a chat window.

Decision table: what drift is allowed

Change you intended	Expected drift	Action
Move function to another module, re-export it	None	Merge after tests pass
Rename a private helper	None	Merge after tests pass
Split a module into a package	None, if __all__ re-exports	Merge after tests pass
Rename a public function	One renamed key	Add a deprecated alias in the same commit
Add an optional keyword argument	One changed signature	Bump minor version, note it
Change a default value	Possibly none	Caught by tests, not by the manifest
Remove an export	One deleted key	Major version, changelog entry, grep all importers

The last two rows are the honest limits. See below.

Limitations, and who should skip this

• ast cannot see getattr(mod, name), plugin registries, or star-import re-exports. Those surfaces stay invisible.
• Default values and decorator behavior are not hashed. Arity and names are all you get. That gap is why step 2 exists.
• C extensions, generated modules, and dynamically built classes will not parse into anything useful.
• Type annotation changes are ignored by this script. Add ast.unparse(node.returns) if you care.

Skip this workflow if your repo is a set of scripts with no importers. Skip it if you have no CI at all, because a hook nobody runs is worse than no hook. Skip it if you are deliberately breaking the API this sprint.

The takeaway

Freeze the surface. Pin three behaviors. Move one seam. Diff the manifest. That sequence turns "this refactor looks safe" into a diff you can review in ten seconds.

If you want to try the loop without paying for model calls, start on the project's free tier and check today's terms first. Then keep the gate in your own CI, where it belongs.