Intentional Drift: Updating Characterization Tests When the Refactor Fixes a Bug

Characterization tests freeze current behavior. They freeze the bugs too. When a refactor fixes a bug on purpose, the contract breaks.

Most teams react in one of two ways. They delete the test and lose the safety net. Or they keep it and block the fix. Both responses are wrong. Deleting hides the change. Keeping it hides the reason.

The fix is a three-step drift protocol. Prove the change is intended. Review the exact diff. Update the oracle with a written reason. This article walks through the protocol with a real rounding bug.

The Setup: A Pinned Bug

Here is the legacy function. It lives in the messy middle of a payment module.

# legacy.py — the messy middle
def apply_discount(price, rate):
    # TODO: rounding is wrong for some prices
    return round(price * (1 - rate), 2)

Python's round uses banker's rounding. Float representation makes 2.675 slightly smaller than its decimal name. So round(2.675, 2) returns 2.67, not 2.68.

A previous refactor added a characterization test. It pinned the current output.

def test_apply_discount_pins_legacy_output():
    assert apply_discount(2.675, 0.0) == 2.67

The test passes. It also pins a bug. The new refactor fixes the bug with Decimal. Now the test fails.

Step 1: Prove the Drift Is Intended

Write a failing test for the new behavior first. This is test-driven refactoring, not feature development. You are documenting intent.

def test_apply_discount_uses_decimal():
    assert apply_discount(2.675, 0.0) == Decimal("2.675")

Run the full suite. The new test fails. The characterization test fails too. Compare both failures. They must describe the same behavior change. If they do not, stop and investigate.

Step 2: Review the Diff, Then Accept

Never regenerate the golden file blindly. Regeneration hides drift. Review every changed line first.

The script below enforces that review. It compares current output with the frozen oracle. It refuses to update without a reason.

#!/usr/bin/env bash
# drift.sh — update an oracle only with a reviewed reason
set -euo pipefail

GOLDEN="tests/golden/apply_discount.txt"
ACTUAL="tests/actual/apply_discount.txt"
REASON="${DRIFT_REASON:-}"

python -c "from legacy import apply_discount; print(apply_discount(2.675, 0.0))" > "$ACTUAL"

if diff -u "$GOLDEN" "$ACTUAL" > /tmp/drift.patch; then
  echo "PASS: behavior matches the oracle"
  exit 0
fi

if [ -z "$REASON" ]; then
  echo "FAIL: behavior drifted without a reason"
  echo "Review /tmp/drift.patch before accepting"
  exit 1
fi

cp "$ACTUAL" "$GOLDEN"
echo "Oracle updated. Reason: $REASON"

Run it with the accepted reason.

DRIFT_REASON="fix banker's rounding; use Decimal" ./drift.sh

Without a reason, the script exits non-zero. That is the whole point. The reason string becomes the audit trail.

Step 3: Record the Decision in the Test

The golden file is machine-readable. The test file is human-readable. Put the decision in both.

def test_apply_discount_pins_legacy_output():
    # 2026-08-28: intentional drift.
    # Old float behavior: 2.67. Refactor 42: Decimal("2.675").
    assert apply_discount(2.675, 0.0) == Decimal("2.675")

Git history keeps the old assertion. git diff proves the change covers one behavior. Reviewers see exactly what moved and why.

The comment is not decoration. It answers the question every reviewer will ask. Why did this oracle change? Six months later, the answer is one line away.

The Decision Table

Use this table when the drift check fails.

Diff result	Action
Only the intended fix	Accept with DRIFT_REASON
Unintended extra change	Fix the code, keep the oracle
Harness noise (ordering, locale)	Fix the harness, keep the oracle
No diff	Pass, no action

The table forces a classification. Classification prevents lazy regeneration.

Where a Free Model Fits

The loop has two expensive steps. Drafting the new oracle. Reviewing the drift patch. Both are pattern-matching tasks. A free model handles them well.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access can draft the Decimal oracle from the failing test. The free server option can run the verification loop in CI. That combination fits small legacy repos with no local test runner. The script above needs only a shell and two text files.

The model does not decide. The reviewer decides. The model only proposes the new oracle and flags suspicious lines in the patch.

Who Should Not Use This

Skip this protocol if your team skips code review. The reason string is only as honest as the reviewer. Skip it if your characterization suite has dead probes. Dead probes pass without asserting anything. Fix those first.

The protocol scales to one intentional drift at a time. A refactor that changes forty behaviors needs forty reviewed diffs. That is slow on purpose. Speed returns when the suite stays trustworthy.

This protocol assumes the repo is under version control. No git, no audit trail. The reason string is worthless without history.

The Takeaway

Characterization tests are not permanent walls. They are contracts that need explicit amendments. Fix the bug, then amend the contract. Never delete the contract silently.

If you try this on a legacy repo, start with one function. One bug, one oracle, one reason string. MonkeyCode's free model access and free server option are enough for that scale.