Every monolith extraction reaches the same question: how do you know the new service returns the same answers as the old code, on real production data, before anything depends on it? A test suite written against the new service only proves the new service is self-consistent. It proves nothing about the behaviour you inherited, which is undocumented, occasionally wrong, and already relied upon by callers who built around the wrongness.
We do this work regularly — Python 2 to 3 migrations, monolith-to-microservices rewrites of systems we did not write — and the constraint is always the same. There is no freeze. The monolith keeps shipping while the extraction runs, because the client's roadmap does not pause for our architecture diagram. That single constraint rules out the comfortable approach of branching, rewriting, and cutting over on a quiet Sunday. It forces a method where the old and new implementations run side by side in production for weeks, and the old one stays authoritative until the evidence says otherwise.
The spec is the old code's output
The first thing we stop doing is writing the new service from the ticket. The ticket describes what the endpoint is supposed to do. Production describes what it actually does: the rounding that happens one step earlier than the docs suggest, the field that is omitted rather than null when a relation is missing, the ordering that is technically unspecified but that a mobile client parses positionally anyway.
So we build the candidate service, deploy it, and give it no traffic. Instead, the monolith calls it out of band on real requests and compares the two answers. The candidate cannot affect the response, cannot add latency to it, and cannot fail the request. Those three properties are what make it safe to run this on a payment-adjacent read path in production rather than on a replayed sample that has already lost the interesting cases.
The dispatch is small on purpose.
import functools
import logging
import random
from concurrent.futures import ThreadPoolExecutor
log = logging.getLogger('shadow')
_pool = ThreadPoolExecutor(max_workers=8, thread_name_prefix='shadow')
def shadow_read(name, sample_rate, classifier):
def decorator(legacy_fn):
state = {'candidate': None}
@functools.wraps(legacy_fn)
def wrapper(*args, **kwargs):
authoritative = legacy_fn(*args, **kwargs)
candidate = state['candidate']
if candidate is not None and random.random() < sample_rate:
_pool.submit(
_compare, name, candidate, classifier,
authoritative, args, kwargs,
)
return authoritative
def register(candidate_fn):
state['candidate'] = candidate_fn
return candidate_fn
wrapper.candidate = register
return wrapper
return decorator
def _compare(name, candidate, classifier, expected, args, kwargs):
try:
actual = candidate(*args, **kwargs)
except Exception:
log.warning('shadow.error', extra={'endpoint': name})
return
verdict = classifier(expected, actual)
if verdict != 'match':
log.warning('shadow.diff', extra={'endpoint': name, 'verdict': verdict})
Two details in there are not stylistic. The pool is bounded and the submit is fire-and-forget, so a slow candidate degrades into dropped comparisons instead of queued work piling up in the request path. And the comparison swallows every exception from the candidate, because a traceback escaping into the monolith's response is exactly the incident this whole exercise exists to avoid.
Comparing is where the actual work is
A naive equality check reports a diff on every single request and teaches the team to ignore the log within a day. Real payloads differ in ways that do not matter and in ways that matter enormously, and the value of the method comes entirely from separating the two automatically.
We normalise first, then classify what survives normalisation.
from decimal import Decimal
def normalize(payload):
if isinstance(payload, dict):
return {
key: normalize(value)
for key, value in sorted(payload.items())
if value is not None
}
if isinstance(payload, list):
return [normalize(item) for item in payload]
if isinstance(payload, Decimal):
return payload.quantize(Decimal('0.01'))
if isinstance(payload, float):
return Decimal(repr(payload)).quantize(Decimal('0.01'))
if hasattr(payload, 'isoformat'):
return payload.replace(microsecond=0).isoformat()
return payload
def _keyed(items):
return {item.get('id'): item for item in items if isinstance(item, dict)}
def classify(expected, actual):
if expected == actual:
return 'match'
left, right = normalize(expected), normalize(actual)
if left == right:
return 'cosmetic'
if isinstance(left, list) and isinstance(right, list):
if _keyed(left) == _keyed(right):
return 'ordering'
if set(_flatten(left)) - set(_flatten(right)):
return 'missing_field'
return 'behavioural'
The classes are not decoration; they are a work queue. Cosmetic diffs mean the candidate is right and the serialiser needs alignment. Ordering diffs mean the old query had an implicit sort that a client is probably depending on, so we make it explicit in both. Missing fields are usually a null-versus-omitted disagreement, which is a one-line fix and a genuine breaking change if shipped unnoticed. Only the behavioural bucket needs someone to read the old code and decide whether the difference is a bug we are inheriting on purpose or a bug we just wrote.
We inherit deliberate bugs more often than people expect. If a report has undercounted a category for years and the client's own spreadsheets are reconciled against it, matching the old behaviour and raising it as a separate ticket is the correct engineering decision. Fixing it silently during an extraction turns an infrastructure change into a data incident that nobody will connect back to us for a week.
Cutover is a routing change, not a deploy
When a route's behavioural diff count has been zero for long enough to cover the slow paths — month-end jobs, retried webhooks, the tenant with the unusual configuration — reads move. We do this at the edge so that reverting takes seconds and touches no application code.
upstream monolith { server monolith:8000; keepalive 32; }
upstream invoices { server invoices:8000; keepalive 32; }
map $http_x_cutover_group $invoices_backend {
default monolith;
canary invoices;
everyone invoices;
}
location /api/v1/invoices {
proxy_pass http://$invoices_backend;
proxy_set_header X-Request-Id $request_id;
proxy_next_upstream off;
}
The header is set by an upstream layer that knows the tenant, so the group can widen gradually and a rollback is one map edit and a reload. We deliberately turn off retry to the next upstream here: silently replaying a request against the other implementation during a cutover is how you get a duplicate side effect and lose the ability to explain what happened.
Writes come last, and they do not get a shadow. Writing twice to two implementations means two sources of truth and a reconciliation problem we would have to build and then throw away. Instead the extracted service takes ownership of its tables in one step, and the monolith's write path becomes a thin client calling the service. That is the moment of real risk in the project, and the reason it lands late: by then, every read path has already proven the new service's model of the data matches the old one on production traffic.
What it costs to run
Honest accounting matters, because this method is not free and it is not always justified. Shadow reads add load to the database — the candidate usually runs its own queries against the same instance, so a heavy report endpoint gets sampled at a few percent rather than fully. The comparison log is verbose and needs a retention policy, and the classifier itself needs unit tests, because a classifier that quietly returns match on everything looks exactly like success.
The largest cost is not technical. While shadowing runs, every change the monolith team ships to a shadowed path has to be mirrored in the candidate, or the diff count rises for reasons unrelated to the extraction. That double-maintenance tax is bearable for weeks and corrosive over quarters, so we scope extractions to what a route group can clear in that window rather than announcing a rewrite of the whole system. A diff dashboard that has been flat and non-zero for a month is a signal to either finish the route or delete the candidate.
What this buys is a migration where nobody has to answer the question of whether the new service is correct with an opinion. The answer is a count, taken from production, on the traffic that actually exists — and the cutover is the boring part, which is where we want the risk in someone's live system to sit.
Originally published on shipmindlabs.com — where we write about payment systems, infrastructure and marketplace backends.
Top comments (4)
Strong distinction between shadowing reads and refusing to dual-write real payment effects. A useful middle step is to shadow the write plan, not the side effect: keep the legacy path authoritative, but have the candidate emit a non-executable effect manifest containing the idempotency key, ledger postings, provider request digest, and outbox/webhook events. Compare and classify those manifests, then discard the candidate plan. That catches write-semantic drift without double-charging or creating two sources of truth, and gives the cutover a much stronger evidence trail.
This is the right middle step, and we should have named it in the post — we've used a version of it, and the reason it didn't make the article is that the hard part isn't emitting the manifest, it's controlling what state the candidate plans against. The legacy write has already mutated the row by the time the candidate builds its plan, so a naive implementation reports semantic drift that is actually just a read-after-write race. We ended up capturing the input state alongside the request and planning against that snapshot, which works but quietly turns "shadow the plan" into "build a deterministic planner" — a bigger ask than it sounds when the legacy path reads six tables through an ORM.
Two things we'd add to the manifest comparison from experience: posting order can be semantics in a ledger (not cosmetic, unlike response-field order), and provider request digests need normalising before hashing — timestamps and nonces in the request body otherwise make every manifest unique. Curious how you handle the snapshot problem — do you pin the candidate to a consistent read (REPEATABLE READ against the same snapshot), capture inputs explicitly, or accept a tolerable rate of race-induced diffs and classify them out?
My default is explicit capture at the last pre-effect boundary. I would make the planner input a compact immutable envelope: IDs plus row versions for every object consulted, balances/reservations, pricing and provider-config versions, and normalized clock/nonce inputs. Both planners consume that same envelope.
REPEATABLE READ is attractive when both implementations share a database, but exporting or holding a snapshot across an out-of-process shadow couples the migration to the database and lets observation extend transaction lifetime. I use race classification only for a stale_snapshot outcome when recorded versions no longer match, not to excuse monetary or ledger diffs.
For the six-table ORM case, instrumenting repository reads during sampled legacy requests is a useful bridge: record the normalized read set once, replay it through a candidate adapter, then gradually turn that envelope into an explicit planner contract. And agreed on ordering: ledger posting order stays semantic; timestamps and nonces should be separated or canonicalized before digesting.
The envelope-as-contract framing settles it cleanly — and pinning race
classification to a single stale_snapshot outcome, never letting it become an
excuse bucket for monetary diffs, is the discipline most shadow setups lose
first. We're taking that.
One effect of recording the read set that's worth naming: the envelope doesn't
just neutralise the race between legacy and candidate — it makes the legacy's
own races visible. On sampled captures we've seen envelopes that were
internally inconsistent (row versions split across a concurrent writer), which
means the legacy path planned a real money movement against a state no
consistent reader could ever have observed. That diff indicts the legacy, not
the candidate — and catching it before cutover turned out to be half the value
of the whole exercise.
Where we still don't have a clean answer: exit criteria. Sampled
instrumentation biases coverage toward hot paths, so "N quiet days" mostly
proves your frequent flows agree while the rare branches stay silent. Do you
gate cutover on envelope coverage per transaction class, or force-capture the
long tail behind flags?