How you retire a live write path without a flag day: run both paths at once, make the switch a runtime decision, prove the two systems agree on what actually persisted - before and after you flip - and only then let the switch stop being reversible, because there's nothing left to switch back to.
👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go. For a while now I've been on the kind of project that teaches you the most: carefully breaking a large PHP monolith into Go microservices while it's still very much alive and serving a real business. The first three parts of this series lived on one seam - a 200 OK that saved nothing, the design fork of who owns identity, and moving the slow cascade off PHP. This part is the payoff: how that seam actually gets cut over in production, with nobody noticing. The running notes live on my GitHub: github.com/brilliant-almazov. No hype, just the real work.
Here's the thing nobody tells you about a monolith breakup: writing the new service is the easy half. The dangerous half is the day you make it authoritative - the moment the new code, not the old, decides what lands in the database. Get that wrong and you don't get a stack trace. You get a 200 OK that saved nothing - the exact production incident that opened this series, and the exact reason everything below exists.
The migration, in stages
A big-bang rewrite is a bet you make once and lose slowly. The alternative - the one that actually works on a live business - is a strangler fig: move the seam one named stage at a time, and never let two migrations run at once. Here is the seam this whole series lives on - the rule-set write path - laid out as stages, with a marker on where this article stands:
Stage 0 Monolith only.
PHP computes the cascade, persists the data, owns identity implicitly.
Stage 1 Identity extracted.
A small Go service becomes the immutable, content-addressed master of
identity (id + hash). PHP still computes and persists the DATA.
Stage 2 Write logic extracted.
The cascade is re-implemented in a stateless Go service that writes
directly into the monolith's DB. Two engines run in parallel.
Stage 3 Prove they agree. <-- THIS ARTICLE
Drive both paths and diff PERSISTED truth, case by case, on the deploy
that's actually running.
Stage 4 Flip, then delete. <-- THIS ARTICLE
Promote the new authority behind a live, reversible master-switch, then
retire the old path - and only then move the DATA out.
Everything upstream (the identity fix, the stateless cascade, the parallel run) exists to earn Stages 3 and 4. A cutover isn't a moment; it's the disciplined walk from "two systems, one is master" to "one system, no switch left." This article is that walk.
Context: what a cutover actually is
Strip away the specifics and the domain is simple. The system stores classification rules attached to a three-level hierarchy that cascades top-down - CLIENT ⊃ PROJECT ⊃ CONFIG - so writing a rule set at a parent node fans out and recomputes every descendant's effective set. That fan-out is the expensive write path this series has been extracting from PHP, piece by piece. (The full primer is in Part 1; here I only need the shape of it.)
Now the cutover. Strip away the domain and a cutover is a single question asked on every write: which system is allowed to be right?
Before the migration, there's one answer, hard-wired: the monolith. After the migration, there's one answer, hard-wired: the new path. A cutover is the messy middle where both answers are wired in at once and something has to choose between them per request - ideally something you can change while the service is running, without a rebuild and without a deploy.
That "something" is a master-switch flag. Not a feature flag for a UI toggle - a flag that decides which of two write paths is the master, the authority whose result is the truth. Two properties make it worth the name:
- Live. You flip it on the running service. No rebuild, no image push, no rolling restart.
- Reversible. If the new path misbehaves under real traffic, you flip it back in seconds - not "revert the PR and wait 20 minutes for CI and a deploy."
Reversible-in-seconds is the whole game. It's the difference between a cutover and a gamble.
The anti-pattern that quietly kills reversibility
I'll name the mistake first, because it's the one I see most and it's invisible until the worst possible moment.
You build the two write paths. You wire the switch. Then, somewhere in the name of "clean production config," someone prunes the unused path out of the object graph at build time - a compile flag, a DI profile, an if BUILD_ENV == "prod" that only ever constructs the new path. The reasoning sounds responsible: don't ship dead code, don't instantiate what you won't use.
The result is a switch that can't switch. A runtime environment variable cannot flip a seam that was compiled away. The legacy path isn't "off" - it doesn't exist in the running process. So the day the new path misbehaves, your reversible cutover reveals itself as a one-way door, and your rollback plan is a redeploy of the old binary under incident pressure. That's a flag day wearing a feature-flag costume.
The rule that avoids it: always register both implementations; let a factory decide at runtime. Dead code that a switch can reach is not dead - it's your rollback. In Go, that's a factory holding both, choosing per request:
// WritePath is the seam the cutover flips. Exactly two implementations exist —
// the legacy local path and the store path. BOTH are always constructed and
// registered. The switch chooses one per request; it never chooses at build time.
type WritePath interface {
Apply(ctx context.Context, desired RuleSet) (Persisted, error)
}
// MasterSwitch reads a runtime flag — env var, config row, control-plane value.
// The point is that StoreIsMaster can return a different answer on the very next
// request, on the same running process, with no rebuild.
type MasterSwitch interface {
StoreIsMaster(ctx context.Context) bool
}
type WritePathFactory struct {
legacy WritePath // always here — this IS the rollback
store WritePath // always here
master MasterSwitch
}
// One method: Resolve. It returns which path is authoritative right now.
// No instanceof ladder, no build tag — polymorphism decides, at runtime.
func (f *WritePathFactory) Resolve(ctx context.Context) WritePath {
if f.master.StoreIsMaster(ctx) {
return f.store
}
return f.legacy
}
Note the naming discipline, because it's load-bearing across this whole series: the factory has one method, Resolve - a resolver resolves, it doesn't grow a resolveOrBuild cousin. Both paths satisfy the same interface, so the caller routes by polymorphism, never by a type switch in shared code. And it's WritePath, MasterSwitch, RuleSet - plain PascalCase, initialisms as words, ID the one exception - not WRITEPATH or HTTPClient. Small rules, but they're what let a new implementation slot in behind the same contract without anyone downstream changing a line.
Where the switch really lives - and where it scales to
The factory above chooses between two code paths inside the monolith: the legacy local write and the store-backed write from Part 1. That's the smallest instance of the cutover. But the same discipline scales up one level, to the topology the parallel run runs on - and this is the picture that has to be exactly right, because it's easy to draw wrong:
LEVEL 1 two separate front doors — NOT connected to each other:
PHP monolith (PHP-FPM, legacy) Go API gateway (new service)
LEVEL 2 services:
domain-rule-map svc rule-set-markup service
identity (id+hash), its OWN DB, new cascade engine, Go,
a microservice FOR PERFORMANCE, STATELESS: routes + mirrors,
never touches the monolith DB no DB of its own — writes the
monolith DB directly
LEVEL 3 storage:
monolith PostgreSQL DB (both engines' only write sink) + DRM's own DB
Both engines resolve identity from domain-rule-map over gRPC. Both write the
monolith DB. The identity service never touches it.
Three things about this picture matter for a cutover:
- The new engine writes the monolith's DB directly. It has no store of its own yet - only the compute has moved, not the data. That's a deliberate, temporary Stage-2 coupling with an exit at Stage 4, not the end state. It's safe only because the engine is stateless: it keeps nothing, so it can't drift from the DB it borrows.
-
domain-rule-mapis a separate microservice for one reason: performance - one fast, immutable identity lookup that both engines share over gRPC. It owns identity in its own DB and never touches the monolith DB. Identity resolution is off both engines' write path to the monolith. - The two front doors are independent. The gateway fronts the new service; the PHP-FPM monolith is its own legacy front door. They aren't wired together - which is exactly what lets a consistency agent drive both engines and diff persisted truth, case by case. The companion to Part 1 tells that story in full.
Whether the switch is choosing two code paths or two whole services, the cutover discipline is identical: register both, flip live, prove agreement against the DB, promote, delete.
Proving the two paths agree - before you dare to flip
A live, reversible switch tells you that you can flip. It says nothing about whether you should. For that you need evidence that both paths produce the same truth in the database for the same input - because the failure mode from Part 1 was precisely two systems that agreed on the HTTP response and disagreed on what persisted.
So I built a harness whose only job is to answer "do these two agree?" - and to leave proof on disk. It runs one loop:
-
Snapshot. Capture the current persisted state of a production polygon (a safe, disposable slice of real data) -
before.json. You cannot verify a change you didn't measure before. - Act. Run the write. Dry-run by default - it computes and diffs but persists nothing. Real writes are gated behind an explicit flag and are reversible, so "act for real" is a deliberate, recoverable choice, never the default.
-
Verify truth in the DB. This is the step Part 1 was built on: don't trust the response. Re-fetch the persisted state and diff response-versus-persisted. A
200withwas_updatedmeans nothing until the row says so. - Restore the polygon. Leave the data exactly as found. The next run must start from the same known state, or your matrix drifts into noise.
- Persist the full transcript to disk. Input, response, persisted state, diff, verdict - all of it, written out. This path has no production logs. If the harness doesn't record what happened, nothing did. The transcript is the evidence.
// The harness is one interface with one honest method. It does not "test"; it
// records a verdict against truth and leaves a transcript behind.
type AgreementCheck interface {
Run(ctx context.Context, c Case) (Verdict, error)
}
func (h *Harness) Run(ctx context.Context, c Case) (Verdict, error) {
before, err := h.polygon.Snapshot(ctx, c.Target)
if err != nil {
return Verdict{}, fmt.Errorf("snapshot: %w", err)
}
defer h.polygon.Restore(ctx, before) // restore always — even on failure
resp, err := h.act.Apply(ctx, c.Input) // dry-run unless real writes are gated on
if err != nil && !errors.Is(err, ErrDryRun) {
return Verdict{}, fmt.Errorf("act: %w", err)
}
persisted, err := h.polygon.Fetch(ctx, c.Target) // TRUTH, not the response body
if err != nil {
return Verdict{}, fmt.Errorf("verify: %w", err)
}
v := Verdict{Case: c.ID, Deploy: h.deploy, Agree: resp.Matches(persisted)}
return v, h.transcript.Persist(ctx, c, resp, persisted, v) // no logs — this is the record
}
errors.Is(err, ErrDryRun) rather than err == ErrDryRun, because the moment a decorator wraps that error the direct comparison goes silently false and the harness starts "acting for real" when it thinks it's dry - the exact class of quiet lie this series is about.
A verdict is a coordinate, not a vibe
One run of the harness produces one fact: case X agreed (or didn't) on deploy version Y. That pairing is the unit that matters. A verdict isn't "it works now" - it's a coordinate (case, deploy-version).
The cases are stable and numbered, shared verbatim with a reviewer. When they and I say "case 16," we mean the same scenario forever - the project → own markup move that stayed red in Part 1 until the identity fix landed. That stability is what turns a conversation about correctness into a conversation about a grid:
| Case | v4.5.8 |
v4.5.9 |
|---|---|---|
| 4 - plain create | ✅ | ✅ |
| 15 - restamp own→project | ❌ | ✅ |
| 16 - move project→own | ❌ | ✅ |
| 18 - shrinking REMOVE | ❌ | ✅ |
| 22 - cascade to N configs | ✅ | ✅ |
The ❌→✅ flips between two deploy tags are the safety argument for the cutover. I don't flip the master because the code looks right; I flip it because every case that used to disagree now agrees, on the deploy that's actually running, with a transcript on disk to prove it. And the reason this rigor exists at all is coordinate (any case, the day we shipped a silent no-op) - a 200 OK that persisted nothing, in production, from Part 1. You earn the right to flip a master by making disagreement a thing you can see.
The end state: when the switch stops being reversible
Here's the part that surprised me. A reversible switch is not the goal - it's scaffolding. The goal is to make it un-reversible on purpose.
The cutover progresses in stages, each guarded by a green agreement matrix on the running deploy:
- Both paths live, legacy is master. The new path runs in shadow; the harness proves it agrees. Flip risk: zero, because you haven't flipped.
- New path becomes master, legacy stays registered. You flip the switch live. If anything drifts, you flip back in seconds - the legacy path is still in the object graph, still your rollback. This is the reversible window, and you live in it deliberately until the matrix is boringly green across many deploys.
- Identity moves fully into the store. The monolith's local copy stops being a source of truth and becomes a projection - a read-model derived from the store, not an authority that can disagree with it. There's now only one place that can be right.
-
The switch stops being reversible - because there's nothing to switch back to. Once the legacy path owns no truth, keeping it registered guards nothing; it just preserves a ghost. You remove it. The
MasterSwitchcollapses to a constant, then disappears. The seam is cut - and only then does the data migration (Stage 4 proper) begin.
That last step is how you retire an old write path without a flag day. There's no heart-stopping midnight deploy where the old world ends and the new one begins in one transaction. There's a slow, evidenced walk: register both, flip live, watch the matrix, promote the store to sole authority, and only then delete the path you no longer need. The big-bang deploy everyone fears is avoided not by being brave on the night, but by never having a night.
AI as a multiplier, and what it can't do for a cutover
I lean hard on AI coding assistants for work like this, and my honest, strongly-held take is that AI amplifies good engineers and exposes weak ones. It's a multiplier, not a crutch.
AI made the mechanics of this cutover cheap - generating the factory, the harness scaffold, the transcript serializer, the matrix formatter. Fast, tireless, genuinely useful. What it did not do - could not do - is any of the judgment the cutover actually turns on. It won't tell you that pruning the legacy path at build time silently voids your rollback. It won't insist you diff persisted state instead of the response, because the response looks authoritative and the model has no scar from a 200 that lied. It won't decide when the matrix is green enough to promote the store to sole authority - that's a risk call a human owns.
Point a multiplier at a disciplined cutover - both paths registered, a runtime switch, agreement proven against truth, verdicts as coordinates, an unhurried walk to irreversibility - and it collapses "fast or safe" into fast and safe. Point it at "the new service passes its unit tests, ship it as master" and it'll help you build a flag day faster than you can schedule the incident review.
This is Part 4 of a series
Four parts, one seam, told honestly - from the bug that exposed it to the cutover that closes it:
-
Part 1 - A
200 OKthat saved nothing: the silent success, and why identity must live in exactly one place. - Part 1.5 - Two rule engines, one truth: the parallel run behind this cutover - old PHP engine and new stateless Go engine live at once, proven equal through the gateway.
- Part 2 - Who owns a hash function: where to cut, and why ownership of identity is an architecture decision, not an implementation detail.
- Part 3 - A stateless cascade in Go: why the slow cascade left PHP for a stateless Go service, and how to measure the win honestly.
- Part 4 - this one: flipping the master live, proving agreement against truth, and retiring the old path without a big-bang deploy.
That's one complete seam. The pattern generalizes to every remaining seam in the monolith: extract a service, prove it agrees with the old one against persisted truth, flip a live and reversible switch, watch the matrix, promote the new authority, delete the old path. No six-month freeze. No flag-day gamble. Just good engineering, amplified - and a human deciding when it's actually safe to flip.
If you build serious backends - Symfony, Go, or the messy space between a monolith and its microservices - follow along. And if you're mid-cutover right now: can you flip your master back in ten seconds? If the honest answer is "we'd have to redeploy," you don't have a cutover yet - you have a flag day with extra steps. I'd genuinely like to compare notes.
Top comments (0)