Dakota Huang

Posted on Aug 31

Differential Characterization: The Only Refactor Safety Net for Messy JSON Normalizers

#refactor #testing #python #webdev

You inherited a JSON normalizer. No tests. Six nested ifs. One silent default. Eight callers depend on it.

Characterization tests protect refactors. They capture current behavior. But characterization can lie.

Sample inputs miss branches. Exact-output assertions ignore edge cases. You finish a refactor. Tests are green. Production breaks.

Differential characterization closes the gap. It runs old and new versions side by side. It feeds both the same chaotic inputs. It reports every difference.

That is the safety net this article builds.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access can generate candidate normalizers. Its free server option can host two test endpoints. The method works without both. The artifact is what matters.

The Setup

You need two small endpoints. One runs the legacy function. One runs your candidate refactor. Both accept the same JSON body. Both return the normalized result.

The harness randomizes input. It mutates keys, types, nesting, arrays. It feeds each pair through both endpoints. It compares the JSON responses. Every difference becomes a formatted diff.

The Harness

Save this as diff_characterize.py.

import json
import random
import requests

LEGACY_URL = "http://localhost:8001/normalize"
CANDIDATE_URL = "http://localhost:8002/normalize"


def random_payload(rng):
    return {
        "user": {
            "id": rng.randint(0, 1000),
            "name": rng.choice(["ann", "", None, 42]),
        },
        "items": [rng.randint(0, 10) for _ in range(rng.randint(0, 5))],
        "flag": rng.choice([True, False, None, "true"]),
        "unknown" + str(rng.randint(0, 5)): rng.choice([{}, [], 0, "x"]),
    }


for seed in range(500):
    rng = random.Random(seed)
    payload = random_payload(rng)

    legacy_resp = requests.post(LEGACY_URL, json=payload)
    candidate_resp = requests.post(CANDIDATE_URL, json=payload)

    if legacy_resp.status_code != candidate_resp.status_code:
        print("STATUS DIFF", seed, legacy_resp.status_code, candidate_resp.status_code)
        continue

    legacy_body = legacy_resp.json()
    candidate_body = candidate_resp.json()

    if legacy_body != candidate_body:
        print("BODY DIFF", seed)
        print("  legacy   ", json.dumps(legacy_body, sort_keys=True))
        print("  candidate", json.dumps(candidate_body, sort_keys=True))

Run it with pip install requests. Point the two URLs at your endpoints. Increase the seed count to 5,000 later.

What a Diff Looks Like

At seed 417, the legacy returns {"id": 42}. Your candidate returns {"id": "42"}. The harness flags it.

That diff is gold. It is not proof of a bug. It is proof that behavior changed. Your job is to decide which behavior matters.

Triage the Diffs

Sort every diff into three buckets: intentional improvements, candidate bugs, and hidden callers.

Use this table.

Diff observed	Meaning	Action
Key order differs	Python dicts preserve insertion order	Check if API clients rely on order
Missing key	Candidate dropped an optional field	Check every caller for the key
Extra key	Candidate added a field	Compare against the documented contract
Type change	`"123"` became `123`	High risk; verify all consumers
Error vs success	Legacy crashed; candidate succeeded	Inspect the legacy error and the candidate path
Value changes	Both return valid but different values	Find the branch that changed

The Refactor Step

Only after triage do you touch the code.

Freeze the semantic contract. Write it as a comment.
Pick one behavioral difference.
Apply the smallest safe change: one branch, one function, one commit.
Re-run the differential harness.
Commit the characterization test alongside the refactor.

This workflow catches what characterization tests miss. Characterization tests sample a few cases. Differential testing explores many. It turns "should be fine" into "prove it".

Limitations

Differential characterization is not a correctness oracle. It tells you what changed. It does not tell you which version is right.

If the legacy normalizer drops all emails, your refactor will match that. You still need behavior tests for the intended contract.

The harness fails with non-determinism. Timestamps, UUIDs, random IDs make every run diff.

It also fails with side effects. A legacy function that writes files cannot run twice safely.

It fails when the old behavior is harmful. Do not preserve a security bug for consistency.

Who Should Not Use This

Teams building greenfield code should skip it. Projects with no legacy behavior gain nothing.

Developers who already understand every branch do not need this. The cost of building two endpoints is too high there.

Use it when the function is a black box. Use it when test coverage is zero. Use it when one normalizer feeds multiple services.

Final Thought

Free models can propose candidates. A free server can host both versions during triage. But the harness does the proof. Models propose. Your test data decides.

Try this harness on one messy function this week. Then keep the characterization test. Your future self will thank you.

DEV Community