Every monolith extraction reaches the same question: how do you know the new service returns the same answers as the old code, on real production data, before anything depends on it? A test suite written against the new service only proves the new service is self-consistent. It proves nothing about the behaviour you inherited, which is undocumented, occasionally wrong, and already relied upon by callers who built around the wrongness.
We do this work regularly — Python 2 to 3 migrations, monolith-to-microservices rewrites of systems we did not write — and the constraint is always the same. There is no freeze. The monolith keeps shipping while the extraction runs, because the client's roadmap does not pause for our architecture diagram. That single constraint rules out the comfortable approach of branching, rewriting, and cutting over on a quiet Sunday. It forces a method where the old and new implementations run side by side in production for weeks, and the old one stays authoritative until the evidence says otherwise.
The spec is the old code's output
The first thing we stop doing is writing the new service from the ticket. The ticket describes what the endpoint is supposed to do. Production describes what it actually does: the rounding that happens one step earlier than the docs suggest, the field that is omitted rather than null when a relation is missing, the ordering that is technically unspecified but that a mobile client parses positionally anyway.
So we build the candidate service, deploy it, and give it no traffic. Instead, the monolith calls it out of band on real requests and compares the two answers. The candidate cannot affect the response, cannot add latency to it, and cannot fail the request. Those three properties are what make it safe to run this on a payment-adjacent read path in production rather than on a replayed sample that has already lost the interesting cases.
The dispatch is small on purpose.
import functools
import logging
import random
from concurrent.futures import ThreadPoolExecutor
log = logging.getLogger('shadow')
_pool = ThreadPoolExecutor(max_workers=8, thread_name_prefix='shadow')
def shadow_read(name, sample_rate, classifier):
def decorator(legacy_fn):
state = {'candidate': None}
@functools.wraps(legacy_fn)
def wrapper(*args, **kwargs):
authoritative = legacy_fn(*args, **kwargs)
candidate = state['candidate']
if candidate is not None and random.random() < sample_rate:
_pool.submit(
_compare, name, candidate, classifier,
authoritative, args, kwargs,
)
return authoritative
def register(candidate_fn):
state['candidate'] = candidate_fn
return candidate_fn
wrapper.candidate = register
return wrapper
return decorator
def _compare(name, candidate, classifier, expected, args, kwargs):
try:
actual = candidate(*args, **kwargs)
except Exception:
log.warning('shadow.error', extra={'endpoint': name})
return
verdict = classifier(expected, actual)
if verdict != 'match':
log.warning('shadow.diff', extra={'endpoint': name, 'verdict': verdict})
Two details in there are not stylistic. The pool is bounded and the submit is fire-and-forget, so a slow candidate degrades into dropped comparisons instead of queued work piling up in the request path. And the comparison swallows every exception from the candidate, because a traceback escaping into the monolith's response is exactly the incident this whole exercise exists to avoid.
Comparing is where the actual work is
A naive equality check reports a diff on every single request and teaches the team to ignore the log within a day. Real payloads differ in ways that do not matter and in ways that matter enormously, and the value of the method comes entirely from separating the two automatically.
We normalise first, then classify what survives normalisation.
from decimal import Decimal
def normalize(payload):
if isinstance(payload, dict):
return {
key: normalize(value)
for key, value in sorted(payload.items())
if value is not None
}
if isinstance(payload, list):
return [normalize(item) for item in payload]
if isinstance(payload, Decimal):
return payload.quantize(Decimal('0.01'))
if isinstance(payload, float):
return Decimal(repr(payload)).quantize(Decimal('0.01'))
if hasattr(payload, 'isoformat'):
return payload.replace(microsecond=0).isoformat()
return payload
def _keyed(items):
return {item.get('id'): item for item in items if isinstance(item, dict)}
def classify(expected, actual):
if expected == actual:
return 'match'
left, right = normalize(expected), normalize(actual)
if left == right:
return 'cosmetic'
if isinstance(left, list) and isinstance(right, list):
if _keyed(left) == _keyed(right):
return 'ordering'
if set(_flatten(left)) - set(_flatten(right)):
return 'missing_field'
return 'behavioural'
The classes are not decoration; they are a work queue. Cosmetic diffs mean the candidate is right and the serialiser needs alignment. Ordering diffs mean the old query had an implicit sort that a client is probably depending on, so we make it explicit in both. Missing fields are usually a null-versus-omitted disagreement, which is a one-line fix and a genuine breaking change if shipped unnoticed. Only the behavioural bucket needs someone to read the old code and decide whether the difference is a bug we are inheriting on purpose or a bug we just wrote.
We inherit deliberate bugs more often than people expect. If a report has undercounted a category for years and the client's own spreadsheets are reconciled against it, matching the old behaviour and raising it as a separate ticket is the correct engineering decision. Fixing it silently during an extraction turns an infrastructure change into a data incident that nobody will connect back to us for a week.
Cutover is a routing change, not a deploy
When a route's behavioural diff count has been zero for long enough to cover the slow paths — month-end jobs, retried webhooks, the tenant with the unusual configuration — reads move. We do this at the edge so that reverting takes seconds and touches no application code.
upstream monolith { server monolith:8000; keepalive 32; }
upstream invoices { server invoices:8000; keepalive 32; }
map $http_x_cutover_group $invoices_backend {
default monolith;
canary invoices;
everyone invoices;
}
location /api/v1/invoices {
proxy_pass http://$invoices_backend;
proxy_set_header X-Request-Id $request_id;
proxy_next_upstream off;
}
The header is set by an upstream layer that knows the tenant, so the group can widen gradually and a rollback is one map edit and a reload. We deliberately turn off retry to the next upstream here: silently replaying a request against the other implementation during a cutover is how you get a duplicate side effect and lose the ability to explain what happened.
Writes come last, and they do not get a shadow. Writing twice to two implementations means two sources of truth and a reconciliation problem we would have to build and then throw away. Instead the extracted service takes ownership of its tables in one step, and the monolith's write path becomes a thin client calling the service. That is the moment of real risk in the project, and the reason it lands late: by then, every read path has already proven the new service's model of the data matches the old one on production traffic.
What it costs to run
Honest accounting matters, because this method is not free and it is not always justified. Shadow reads add load to the database — the candidate usually runs its own queries against the same instance, so a heavy report endpoint gets sampled at a few percent rather than fully. The comparison log is verbose and needs a retention policy, and the classifier itself needs unit tests, because a classifier that quietly returns match on everything looks exactly like success.
The largest cost is not technical. While shadowing runs, every change the monolith team ships to a shadowed path has to be mirrored in the candidate, or the diff count rises for reasons unrelated to the extraction. That double-maintenance tax is bearable for weeks and corrosive over quarters, so we scope extractions to what a route group can clear in that window rather than announcing a rewrite of the whole system. A diff dashboard that has been flat and non-zero for a month is a signal to either finish the route or delete the candidate.
What this buys is a migration where nobody has to answer the question of whether the new service is correct with an opinion. The answer is a count, taken from production, on the traffic that actually exists — and the cutover is the boring part, which is where we want the risk in someone's live system to sit.
Originally published on shipmindlabs.com — where we write about payment systems, infrastructure and marketplace backends.
Top comments (0)