Migration Diary: Run the Cutover in Shadow Mode Before You Cancel the Old Plan

Cancel the old plan only after a shadow window has replayed your real traffic through the new path and the diff report came back boring. A fixture suite proves that the new stack can answer questions you already knew how to ask; shadow traffic proves it survives the ones your users actually sent last week. In this diary entry I want to show you the rehearsal harness I keep reaching for, because it is the cheapest artifact that has ever stopped me from a bad cutover.

The setup is deliberately small: a deterministic sampler, a dual-send runner with a per-call deadline, and a normalizer that compares shape before it compares text. You can run all three in an afternoon, and you can throw the whole thing away once the migration ends, which is exactly the property I want from migration tooling.

1. Why the Shadow Window Beats a Bigger Fixture Suite

Fixtures are curated, and curation is a bias you cannot see from inside your own test suite. The prompts that break a provider switch are usually the long, messy, half-contextual ones that nobody bothered to freeze as a golden case. Shadow traffic samples whatever your logs already contain, so the ugly requests arrive with their real distribution attached.

The second reason is boring but decisive: a shadow run gives you per-request latency and error records alongside the output diff. That means one rehearsal produces both a correctness signal and a capacity signal, and you do not need two separate migration projects. Keep the fixture suite for regressions you must never break, and let shadow mode cover the long tail you never wrote down.

2. Step 1: Sample Deterministically From Your Existing Logs

Do not sample randomly at request time, because you will get a different slice every run and you will never be able to reproduce a failure. Sample by hashing the request ID instead, so the same request always lands in the same bucket across runs and across machines.

# sample_requests.py — stdlib only
import hashlib, json, sys


def bucket(request_id: str, mod: int = 100) -> int:
    digest = hashlib.sha256(request_id.encode()).hexdigest()[:8]
    return int(digest, 16) % mod


def main(src: str, dst: str, pct: int = 5) -> None:
    kept = 0
    with open(src) as fin, open(dst, "w") as fout:
        for line in fin:
            rec = json.loads(line)
            if bucket(rec["request_id"]) < pct:
                fout.write(json.dumps(rec) + "\n")
                kept += 1
    print(f"sampled {kept} requests into {dst}")


if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2], int(sys.argv[3]) if len(sys.argv) > 3 else 5)

Run it as python sample_requests.py logs/requests.jsonl shadow/sample.jsonl 5 and keep the output file in version control if it is small enough. Scrub anything sensitive before that file leaves your machine, and treat the sample as production data for retention purposes.

3. Step 2: Dual-Send With a Hard Deadline

The runner below sends the same normalized payload to your current path and to the candidate path, records both wall-clock times, and never lets a slow call hang the rehearsal. Note that the candidate endpoint here can be any OpenAI-compatible URL, including MonkeyCode's free model access, which is what makes a throwaway rehearsal affordable in the first place.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

# shadow_run.py — stdlib only
import json, os, time, urllib.request

DEADLINE_S = float(os.environ.get("SHADOW_DEADLINE_S", "30"))


def call(endpoint: str, key: str, payload: dict) -> dict:
    body = json.dumps(payload).encode()
    req = urllib.request.Request(endpoint, data=body, headers={
        "Content-Type": "application/json",
        "Authorization": f"Bearer {key}",
    })
    started = time.monotonic()
    try:
        with urllib.request.urlopen(req, timeout=DEADLINE_S) as resp:
            data = json.loads(resp.read().decode())
        return {"ok": True, "ms": int((time.monotonic() - started) * 1000), "body": data}
    except Exception as exc:
        return {"ok": False, "ms": int((time.monotonic() - started) * 1000), "error": repr(exc)}


def run(sample_path: str, old: tuple, new: tuple, out_path: str) -> None:
    with open(sample_path) as fin, open(out_path, "w") as fout:
        for line in fin:
            rec = json.loads(line)
            payload = rec["payload"]
            row = {
                "request_id": rec["request_id"],
                "old": call(old[0], old[1], payload),
                "new": call(new[0], new[1], payload),
            }
            fout.write(json.dumps(row) + "\n")

Keep both responses raw in the output file, because normalization logic always changes and you will want to re-score old runs without paying for new calls. Set SHADOW_DEADLINE_S to something stricter than your user-facing timeout, so the rehearsal fails the way production would.

4. Step 3: Normalize Shape Before You Compare Text

Most false alarms in a shadow report come from comparing prose literally, so compare structure first and text second. Walk both responses, replace every leaf value with its type name, and diff the resulting skeletons; a shape mismatch is a real integration break, while a text mismatch may be nothing more than a different phrasing.

# score_shadow.py — stdlib only
import difflib, json, sys


def shape(obj, path="$"):
    if isinstance(obj, dict):
        out = []
        for k in sorted(obj):
            out.extend(shape(obj[k], f"{path}.{k}"))
        return out
    if isinstance(obj, list):
        return [f"{path}[]:list"] + shape(obj[0], f"{path}[]") if obj else [f"{path}[]:empty"]
    return [f"{path}:{type(obj).__name__}"]


def text_of(body: dict) -> str:
    return body.get("text") or body.get("output_text") or json.dumps(body, sort_keys=True)


rows = [json.loads(l) for l in open(sys.argv[1])]
shape_ok = sum(shape(r["old"]["body"]) == shape(r["new"]["body"]) for r in rows if r["old"]["ok"] and r["new"]["ok"])
sim = [difflib.SequenceMatcher(None, text_of(r["old"]["body"]), text_of(r["new"]["body"])).ratio()
       for r in rows if r["old"]["ok"] and r["new"]["ok"]]
errors = sum(1 for r in rows if not r["new"]["ok"])
print(f"rows={len(rows)} shape_ok={shape_ok} new_errors={errors} median_sim={sorted(sim)[len(sim)//2]:.2f}" if sim else "no comparable rows")

Treat the similarity number as a triage tool rather than a grade, because two providers can be equally correct with very different wording. What you are hunting for is the cluster of near-zero scores, which usually points at a prompt that one side silently truncated.

5. Step 4: Gate the Cutover With a Written Table

Agree on the thresholds before you look at the results, or you will rationalize whatever the report says. Here is the table I now paste into the migration ticket and fill in at the end of every shadow window.

Gate	Pass condition	What a failure usually means	Action
Transport	≥ 99% of candidate calls return without exception	Endpoint, auth, or payload-shape error	Fix client, re-run before judging quality
Shape parity	100% of successful pairs share a skeleton	Response envelope changed	Patch the parser, not the prompt
Similarity floor	≥ 95% of pairs above 0.60 ratio	Truncation, refusal, or wrong field read	Inspect the worst ten pairs by hand
Tail latency	p95 under your user timeout	Cold starts or queueing	Rehearse again at your real concurrency
Error budget	Candidate error rate within 0.5pt of current	Rate limiting or capacity	Negotiate limits before cutover

Write the numbers into the table, link the raw JSONL, and store both next to the cutover calendar. When someone asks six weeks later why the switch happened, the table answers instead of your memory.

6. Step 5: Host the Rehearsal on a Node You Can Delete

Run the whole thing somewhere disposable, because a rehearsal host accumulates tokens, keys, and half-scrubbed logs within days. MonkeyCode's free server option is a reasonable home for the runner, and its free model access covers the candidate side of the dual send; check the limits currently shown in your own account before you size the sample around them.

# on the spare node
python3 -m venv .venv && . .venv/bin/activate
pip install --upgrade pip
nohup python shadow_run.py shadow/sample.jsonl \
  > shadow/raw.jsonl 2> shadow/run.err &
python score_shadow.py shadow/raw.jsonl | tee shadow/report.txt

Keep the sample small enough to finish inside one sitting, and delete the node together with its keys once the report is signed off. If the free tier throttles you mid-run, record the throttle as a data point about rate limits rather than retrying in a loop overnight.

7. What a Shadow Window Cannot Tell You

Replayed traffic is stale traffic, so a shadow run will miss whatever your product shipped after the log window closed. It also tells you almost nothing about multi-turn conversations, tool calls that mutate state, or anything where replaying the same request twice is itself a side effect.

Latency measured against a free tier under light load is not a capacity forecast, and you should not quote it as one. Treat the rehearsal as evidence about compatibility and gross regressions, then load-test separately if throughput is the actual risk.

8. Who Should Skip This

Skip shadow replay if your prompts contain data you cannot lawfully copy into a rehearsal store, or if your traffic is small enough that ten hand-picked conversations would teach you more. Skip it too if you need load realism, because this harness measures compatibility at toy concurrency and nothing else.

If none of those apply, the harness above is roughly a hundred lines and one spare node away. Run it once, read the table honestly, and then decide whether the old plan earns another month.