A shipping-cost module has twelve tests, all passing, and a refactor that looks mechanical. Someone extracts the rate calculation into a helper, the suite stays green, and the change ships. Three days later a batch job that passes an empty line-item list starts returning None instead of 0.0, because no test ever called that function with an empty collection. The interesting part is not the bug itself; it is that the suite had no way to tell anyone which input shapes it had never seen.
Coverage tools answer a different question than the one you are asking. A line-coverage report tells you which statements executed, but it says nothing about the diversity of values that reached them. A branch that has only ever received positive floats looks fully covered while being completely uncharacterized. This article describes a small, reproducible step you can run before touching a messy module: mine the call sites, list the argument shapes, and refuse to refactor shapes your fixtures never exercise.
Why coverage numbers lie about input shape
Characterization tests are only as good as their corpus, and most corpora are written from memory rather than from evidence. Engineers write the happy path, one failure path, and maybe a case from the bug tracker. The result is a suite that documents intent but not reality, and a refactor that preserves intent while quietly discarding reality.
The artifact below closes that gap in three steps, each of which is small enough to finish in an afternoon. First, extract every call site of the target function and bucket its arguments by shape. Second, compare those buckets against the shapes your fixtures already cover. Third, convert every unseen shape into a characterization row before you allow a single line to move.
Step 1 — Mine call sites into a shape manifest
The following script is plain Python with no dependencies beyond the standard library, and it works on any repository that parses as valid Python. It walks every .py file, finds calls whose name matches your target, and classifies each positional argument into a coarse shape bucket. Coarse is deliberate: StrLiteral and Number are enough to catch the empty-collection and None gaps that real refactors break on.
# tools/call_shapes.py
import ast, json, sys
from pathlib import Path
from collections import Counter
TARGET = "compute_quote"
def shape(node):
if node is None:
return "omitted"
if isinstance(node, ast.Constant):
if node.value is None:
return "none"
if node.value == "":
return "empty_str"
if isinstance(node.value, bool):
return "bool"
if isinstance(node.value, (int, float)):
return "number"
return "str_literal"
if isinstance(node, (ast.List, ast.Tuple, ast.Set)):
return "empty_collection" if not node.elts else "collection"
if isinstance(node, ast.Dict):
return "empty_collection" if not node.keys else "collection"
if isinstance(node, ast.Name):
return "name"
if isinstance(node, ast.Call):
return "call_result"
if isinstance(node, ast.Attribute):
return "attribute"
return "other"
rows = []
for path in Path(".").rglob("*.py"):
if ".venv" in path.parts or "tests" in path.parts:
continue
tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
for node in ast.walk(tree):
if not isinstance(node, ast.Call):
continue
fn = node.func
name = fn.id if isinstance(fn, ast.Name) else getattr(fn, "attr", None)
if name != TARGET:
continue
rows.append({
"file": str(path),
"line": node.lineno,
"args": [shape(a) for a in node.args],
"kwargs": sorted(k.arg for k in node.keywords if k.arg),
})
json.dump(rows, open("shape_manifest.json", "w"), indent=2)
print(f"{len(rows)} call sites of {TARGET}", file=sys.stderr)
Run it from the repository root with python tools/call_shapes.py. The manifest is evidence: every entry is a real call site with a file and line you can open. Filtering tests/ out is intentional, because test calls describe intent while production calls describe load.
Reading the manifest honestly
The output below is illustrative rather than measured, but the pattern it shows is the one you should expect: a long tail of literals plus a cluster of name arguments you cannot resolve statically.
[
{"file": "jobs/nightly.py", "line": 41, "args": ["collection", "number"], "kwargs": []},
{"file": "api/checkout.py", "line": 88, "args": ["empty_collection", "name"], "kwargs": []},
{"file": "etl/backfill.py", "line": 12, "args": ["name", "number"], "kwargs": ["currency"]}
]
Two buckets deserve attention immediately. empty_collection is the shape that most pure helpers silently mishandle, and name is the shape that static analysis cannot resolve. For name arguments, grep the enclosing function for the variable's assignment rather than guessing.
Step 2 — Turn unseen shapes into characterization rows
Now compare the manifest against your existing fixtures and write down the difference. A short comparison script is enough: collect the shapes your parametrized tests declare, subtract them from the manifest's shapes, and print the remainder. The remainder is your refactor backlog, ordered by how many production call sites depend on each shape.
Each remainder entry becomes one characterization row. Do not assert what the function should return; record what it does return today, even if today's behavior looks wrong.
# tests/test_quote_characterization.py
import json, pathlib, pytest
from pricing.quote import compute_quote
GOLDEN = pathlib.Path("tests/golden/quote.json")
CASES = json.loads(GOLDEN.read_text())
@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES])
def test_characterization(case):
assert compute_quote(*case["args"]) == case["expected"]
The golden file is generated once by a small recorder script and then reviewed by a human before it is committed. Reviewing the recorded values is the moment you learn whether the empty-list case really returns 0.0 or something uglier. If it returns something ugly, that is a finding, not a problem to fix during the refactor.
Step 3 — Make the smallest safe change
With the corpus in place, the refactor itself becomes almost boring, which is the point. Change one thing per commit, and let the characterization suite prove that every recorded shape still produces its recorded value.
- Add the guard clause for the empty collection, guarded by the new row that pins today's behavior.
- Run the full characterization suite and confirm the row still passes unchanged.
- Move the rate calculation into a helper, touching nothing else in the same commit.
- Re-run the suite; any diff here is a behavior change you did not intend.
- Only after the suite is green for the whole corpus do you correct the ugly behavior, in a separate commit with its own test.
A useful decision table keeps the team honest about when a refactor is allowed to proceed.
| Shape in production | Fixture covers it | Action |
|---|---|---|
| Yes | Yes, with recorded value | Safe to move code |
| Yes | No | Add a characterization row first |
| No | Yes | Keep the row, do not delete it |
| No | No | Leave it out; do not invent a case |
Where free model access and a free server actually fit
Shape manifests generate tedious work: writing fixture rows, resolving name arguments, and writing the one-line commit messages that explain each rung. This is a reasonable place to use MonkeyCode, an open-source project that offers free model access and a free server option; per the project's own description, the free tier includes a large token allowance, and the server option lets you run the tooling without provisioning a box first.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
A workflow that respects the evidence looks like this: run call_shapes.py locally, paste only the shape summary into the model, and ask it to propose fixture rows for the unseen buckets. Never paste proprietary source if your organization's policy forbids it, and treat every proposal as a draft to be verified against the actual function. The free server is useful for running the characterization suite in a clean environment where cached .pyc files and stale virtualenvs cannot hide a regression.
Limitations and who should skip this
This approach is static and shallow, so it will miss shapes that arrive through dynamic dispatch, *args unpacking, or data loaded at runtime. The shape buckets are intentionally coarse, and a number bucket hides every difference between -1, 0, and 10**9. Repositories with heavy metaprogramming will get a mostly-name manifest that provides little signal.
Skip this entirely if the module under refactor has no production call sites yet, or if a single well-understood caller exists that you can read in full. Skip it too if your test suite already runs against recorded production traffic, because that corpus is strictly better evidence than anything static analysis can produce.
A short checklist
- Mine call sites before you open the file you intend to change.
- Bucket arguments coarsely and count production sites per bucket.
- Subtract fixture shapes from production shapes and print the remainder.
- Record today's outputs verbatim; never assert intent during characterization.
- Change one thing per commit and keep the corpus fixed between commits.
The corpus is the deliverable; the refactor is just the thing you were finally allowed to do. If you try the shape-manifest step on a module of your own, the interesting part is usually the third bucket you never expected to find.
Top comments (0)