Public developer threads this week keep asking whether models already write production code better than working developers. The more local question for a messy totals module is narrower, and it is testable. Agents rewrite nested loops because the result looks cleaner in a side-by-side diff. Downstream jobs then break on sort order, cent rounding, and silent defaults that never appeared in the pull-request story.
A payments operations team inherited a 900-line report builder that folded mixed CSV rows into weekly merchant totals. Rounding rules, locale-aware sorting, and missing-field defaults lived inside one nested loop without tests. An agent patch removed duplication and renamed helpers, which looked like real progress during code review. Finance jobs then failed because group order shifted, and half-up cents quietly became banker's rounding.
That failure pattern now shows up regularly during agent-assisted refactors of legacy Python report services. The model optimizes for local cleanliness rather than for downstream consumers of order and money. Characterization goldens that freeze those observables let a team extract one helper without accepting a taste-driven rewrite.
What the first tests should actually pin
The first tests are not a proof that the loop is correct under a finance spec. They are a replayable record of the answers the loop already emits for a frozen corpus. If those answers move, the extract is not small, even when the new names look nicer.
Pin these observables before anyone touches helpers:
- Group order, including stable ties across period labels and merchant identifiers
- Money rounding, especially half-up versus banker's rounding at the cent
- Missing fields, such as a default tax rate, a skipped row, or a zero-filled amount
- String identity, including currency codes, trimmed names, and period labels
A one-helper extract is reviewable when those four stay byte-identical. Anything that also “fixes” sort keys is a second change and needs its own product note.
Worked example: a tangled weekly totals loop
The module below is a compact stand-in, labeled as an unexecuted example rather than a harvested production file. It still mixes parsing, tax math, rounding context, and a locale switch that only changes sort direction. That mix is enough to demonstrate the trap.
# weekly_totals.py — compact stand-in for a tangled report module
from collections import defaultdict
from decimal import Decimal, ROUND_HALF_UP, localcontext
def build_weekly_totals(rows, locale="en_US"):
buckets = defaultdict(lambda: {"amount": Decimal("0.00"), "tax": Decimal("0.00")})
for raw in rows:
if not raw.get("merchant_id"):
continue
period = (raw.get("period") or "unknown").strip()
key = (str(raw["merchant_id"]), period)
amount = Decimal(str(raw.get("amount") or "0"))
raw_rate = raw.get("tax_rate")
rate = Decimal(str(raw_rate if raw_rate not in (None, "") else "0.0875"))
buckets[key]["amount"] += amount
buckets[key]["tax"] += amount * rate
items = []
for (merchant_id, period), acc in buckets.items():
with localcontext() as ctx:
ctx.rounding = ROUND_HALF_UP
amount = acc["amount"].quantize(Decimal("0.01"))
tax = acc["tax"].quantize(Decimal("0.01"))
items.append({
"merchant_id": merchant_id,
"period": period,
"amount": format(amount, "f"),
"tax": format(tax, "f"),
"currency": "USD",
})
# Locale is ignored except as a hidden sort switch — the kind of trap agents "fix".
reverse = locale.startswith("de")
items.sort(key=lambda row: (row["period"], row["merchant_id"]), reverse=reverse)
return items
A cleanup that sorts by merchant first, or that swaps Decimal for round(float(...), 2), will still read as a refactor. Canonical goldens make that cleanup visible as a data change instead of a naming change.
Step 1: freeze a fixture corpus before the edit
Check the inputs into git before the loop is edited, and keep each case small enough to read in review. The useful cases are not happy-path totals; they are ties, missing rates, rounding edges, and the locale flag.
fixtures/weekly/
empty.json
single_row.json
tie_same_period.json
missing_tax_rate.json
half_up_edge.json
german_locale_flag.json
Example half_up_edge.json:
{
"locale": "en_US",
"rows": [
{"merchant_id": "m-9", "period": "2026-W37", "amount": "1.005", "tax_rate": "0.10"},
{"merchant_id": "m-9", "period": "2026-W37", "amount": "2.015", "tax_rate": "0.10"}
]
}
Store locale inside the fixture document rather than in the process environment. Hidden runner locale is a common way for goldens to rot after a host change.
Step 2: emit canonical golden JSON
Canonical JSON avoids false diffs from key order, spacing, and Unicode escapes. Readable goldens are the review surface; a later hash of those bytes is optional insurance. Emit once from the untouched module, then treat later diffs as either a real behavior change or a bad extract.
# tools/emit_goldens.py
import json
from pathlib import Path
from weekly_totals import build_weekly_totals
ROOT = Path("fixtures/weekly")
def canonical(obj):
return json.dumps(obj, sort_keys=True, indent=2, ensure_ascii=True) + "\n"
def main():
for src in sorted(ROOT.glob("*.json")):
if src.name.endswith(".golden.json"):
continue
payload = json.loads(src.read_text())
result = build_weekly_totals(
payload["rows"], locale=payload.get("locale", "en_US")
)
out = src.with_suffix(".golden.json")
out.write_text(canonical(result))
print(f"wrote {out}")
if __name__ == "__main__":
main()
python tools/emit_goldens.py
git add fixtures/weekly/*.golden.json
git diff --stat
Commit the goldens beside the fixtures so the lock travels with the module. Reviewers should see merchant, period, and cent fields, not a boolean from a loosely named unit test.
Step 3: assert today against yesterday, not against an ideal spec
The characterization test does not decide whether a default tax rate of 0.0875 is fair. It only decides whether the current loop still matches the committed answers. Empty corpora should fail loudly, because a missing fixture directory is not a green suite.
# tests/test_weekly_totals_characterization.py
import json
from pathlib import Path
from weekly_totals import build_weekly_totals
ROOT = Path("fixtures/weekly")
def canonical(obj):
return json.dumps(obj, sort_keys=True, indent=2, ensure_ascii=True) + "\n"
def test_weekly_totals_match_goldens():
sources = [
path for path in sorted(ROOT.glob("*.json"))
if not path.name.endswith(".golden.json")
]
assert sources, "fixture corpus is empty"
for src in sources:
payload = json.loads(src.read_text())
result = build_weekly_totals(
payload["rows"], locale=payload.get("locale", "en_US")
)
golden = src.with_suffix(".golden.json").read_text()
assert canonical(result) == golden, src.name
python -m pytest tests/test_weekly_totals_characterization.py -q
Run that command on the untouched module until it is green, and only then open an editor. If it later fails, print a unified diff of the two JSON strings so the changed cent is obvious.
Step 4: extract one pure helper, then stop
Do not split the loop, rename public keys, or introduce a money library in the same patch. Extract one helper that the goldens already constrain, and leave sort keys and skip rules untouched. The smallest honest change in this stand-in is the repeated quantize block.
def quantize_cents(value):
with localcontext() as ctx:
ctx.rounding = ROUND_HALF_UP
return value.quantize(Decimal("0.01"))
Replace the two inlined quantize calls with quantize_cents, and keep the Decimal values in their original accumulation order. Re-run the characterization test before reading the rest of the file for more ideas. A green result means order and money held; a red result means the helper quantized too early or lost rounding context.
Review checklist after the extract
- The public function signature is unchanged, including the
localeargument. - The golden JSON bytes are unchanged across every fixture file.
- The diff is the helper plus the two call sites, and nothing else.
- No locale library swap, float conversion, or default-rate move landed in the same patch.
If an agent also reordered items.sort, reject that hunk even when informal “business logic” checks still pass. Sort order is an observable for any consumer that writes CSV and then diffs weeks.
Use an agent only after the goldens exist
Coding agents are useful once the goldens exist, because the test file is the contract the model cannot narrate away. Without that contract, the agent will treat sort stability and rounding mode as style. The safe sequence is emit, test, allow a narrow edit, test again, then read the diff as data.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode is an open-source coding assistant with free model access and a free server option, which is enough to run this emit-and-test loop on a throwaway machine. That availability is an operator-supplied product fact, not a benchmark of extract quality. A green agent session still needs a human reading the sort key and the ROUND_HALF_UP context.
python tools/emit_goldens.py
python -m pytest tests/test_weekly_totals_characterization.py -q
# permit edits only in weekly_totals.py
python -m pytest tests/test_weekly_totals_characterization.py -q
git diff -- weekly_totals.py
Keep fixture files out of the writable set. If the agent rewrites goldens so the new loop passes, the lock has been replaced with the change under test, and the review is no longer a characterization review.
Decision table for the incoming diff
| Signal in the diff | Keep? | Why |
|---|---|---|
New quantize_cents helper that still uses ROUND_HALF_UP
|
Yes | Pure extract; goldens should hold |
Sort key changed from (period, merchant_id)
|
No | Order is an observable for finance exports |
Decimal replaced with round(float(...), 2)
|
No | Half-up edges drift on values like 1.005
|
| Default tax rate moved into a config file | Not in this patch | That is a second behavior change |
| Fixture or golden files rewritten to match new output | No | The corpus is the lock, not the implementation |
ensure_ascii dropped during JSON emission |
No | Golden bytes must stay comparable across hosts |
Limitations
This method records current behavior, including bugs that finance already absorbed. If the loop double-taxes a row, the golden will protect that bug until a product owner adds a new fixture and an explicit decision. Characterization is a stability tool, not a correctness oracle.
The function must be deterministic given the fixture document. Wall-clock timestamps, process locale, network calls, and unordered iteration on very old Python runtimes will make goldens flicker. Freeze time and isolate I/O before treating a red test as a real extract failure.
The corpus will not catch a change that appears only for a merchant identifier nobody recorded. Grow fixtures from production samples when that is legal, and scrub account numbers before commit. A golden that contains live identifiers is not a test asset; it is an exposure.
Who should skip this approach
- Teams that cannot build a replayable input corpus from logs, dumps, or synthetic rows
- Modules whose public contract is a GUI session rather than emitted data
- Patches that must change rounding or sort order on purpose, which need new goldens and a written product note
- Anyone treating a green characterization suite as proof that the tax math matches a statute
Pin the answers the downstream job already depends on, then extract one helper. Stop there so the next diff remains small enough to review against JSON, not against the agent's explanation of cleanliness.
Top comments (0)