How a single spurious was_updated exposed a hidden rule about ownership - when a system is split, exactly one place is allowed to own identity. This is the expanded cut of Part 1: the debugging and the fix, plus the piece the first version only gestured at - the staged migration this bug lives inside, and where the seam goes right after it.
👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go. For a while now I've been on the kind of project that teaches you the most: carefully breaking a large PHP monolith into Go microservices while it's still very much alive and serving a real business. This is an expanded re-cut of Part 1 - same bug, same fix, but with the migration map drawn in properly, because half the lesson is which stage you're standing in when the ground moves. The running notes live on my GitHub: github.com/brilliant-almazov. No hype, just the real work.
A teammate pinged me with a vague, ordinary-sounding report: domain markup for a region was coming back empty in a snapshot. Not an error, not a stack trace - just wrong output a human noticed because the business process it feeds looked off. I reproduced the write against production and the API told me, cheerfully, that everything was fine - 200 OK, action: was_updated, a response body carrying the correct, freshly computed rule set. And yet nothing had changed in the database. updated_at frozen. The downstream materialized count flat. A silent no-op wearing a success badge.
That one bug turned out to be the perfect lens on the whole migration I'm living in - but you can't see why until you can see the stages. So before the code, let me draw the map.
The migration, in stages
A big-bang rewrite is a bet you make once and lose slowly. The alternative - the one that actually works on a live business - is a strangler fig: move the seam one named stage at a time, and never let two migrations run at once. Here is the seam this whole series lives on - the rule-set write path - laid out as stages, with a marker on where this bug happened:
Stage 0 Monolith only.
PHP computes the cascade, persists the rule data, owns identity implicitly.
Stage 1 Identity extracted. <-- THIS ARTICLE
A small Go service becomes the immutable, content-addressed master of
identity (id + hash). PHP still computes and persists the rule DATA.
The bug: the monolith and the new identity master quietly DISAGREE.
Stage 2 Write logic extracted. (Part 1.5 — next)
The cascade is re-implemented in a stateless Go service that writes
directly into the monolith's DB. Two engines run in parallel.
Stage 3 Prove they agree.
An agent drives both engines and diffs persisted truth, case by case.
Stage 4 Flip, then delete.
Promote the Go engine, retire PHP, and only then move the DATA out.
This article is Stage 1 - the first moment a second system has an opinion about identity. Everything downstream (the parallel run, the cutover) is only safe if this stage is honest. It wasn't yet.
Context: what this is and how it works
Strip away the specifics and the domain is simple. The system stores classification rules - patterns that tag web domains with a meaning ("this domain is ours," "this one is a competitor," "this one is irrelevant"). A rule is roughly {pattern, type, match-mode, priority}. A rule set is an ordered collection of those rules attached to one node in a hierarchy.
The hierarchy has three levels, and it cascades top-down:
CLIENT rules here apply to everything beneath
└── PROJECT rules here apply to every config of the project
└── CONFIG a specific target (search-engine × device × locale)
Writing a rule set at a parent level doesn't just save that node - it fans out: the engine recomputes every descendant's effective rule set by merging what it inherits from above with what it owns locally, with a clear precedence (a narrower level overrides a broader one). The per-node result of that merge is the materialized state: the fully-resolved set of rules a given config actually sees at request time. That snapshot is what the business runs on.
Two concepts matter for the bug:
-
origin- a label on each rule in a materialized set saying which level it came from (client / project / own). Descriptive metadata about the cascade. - identity - the answer to "is this the same rule set or a different one?" Historically an internal detail of the monolith. It stops being an internal detail the moment a second system needs to agree on it. Hold that thought - it's the whole article.
Now the Stage-1 overlay, and this is the part that took me a couple of iterations and a correction from a colleague to state correctly. The write logic - the cascade, the validation, the transactional integrity - lives in the PHP monolith. A newer Go service, the domain-rule-map service (I'll also call it the rule-set store when its identity role is the point), has been introduced as the authority for rule-set identity. It is deliberately, aggressively dumb: immutable and content-addressed. You hand it a rule set, it hashes the content, and it either returns the existing master for that hash or creates a fresh one. It creates exactly what you give it - nothing more. The data and the cascade logic stay in the monolith; the store owns only the id/hash, in its own database.
There's a subtlety I initially got wrong, and a teammate set me straight in one sentence: creating a rule set is not the same operation as recalculating. Creation is send the computed set to the store + save our mirror, transactionally, in one shot. Recalculation is a separate, post-commit process that fans out over everything that changed. Conflating the two - treating the store's echo of a create as the source of truth for our content - is a category error, and it's exactly the kind of thing that reads fine in a diagram and detonates in production.
So at Stage 1 the monolith computes and writes, the store owns identity, and both back the same reads. Every actor now has to agree on two things: the content of a rule set, and whether a given write actually changed the persisted state. This bug is what happens when they quietly disagree about the second one.
How I got it wrong first - the iteration that matters
I want to be honest about the shape of the investigation, because the wrong turns are the lesson.
Iteration 0 - "there's no bug." My first conclusion, embarrassingly, was that nothing was broken. I had a green functional test exercising the write, and a production dry_run that returned the correct cascade. Both said fine. So I told the team it was probably stale data from deploy lag - re-submit and it'll re-materialize.
That was wrong for a precise, instructive reason. The test ran the legacy local write path. The dry_run only computes the cascade - it persists nothing. But production runs behind a runtime master-switch flag that routes writes through the store path instead. A different branch of code entirely. Neither of my two "proofs" had touched the path that was actually failing in prod. A green test on the wrong code path is worse than no test - it manufactures false confidence.
Iteration 1 - reproduce on the real path. The only way forward was an authorized, real (non-dry_run) write against a production polygon, on the store path, watching persisted state rather than the response. And there it was, undeniable:
RESPONSE 200 OK action: was_updated
config:15 -> master A (16 rules) "looks great"
DATABASE config:15 -> master B (8 rules) the scope row points at B
master A -> backs NOTHING A was never linked
the 200 described a world that does not exist in the DB
The monolith's computed identity and the store's persisted identity had diverged - and the scope row that ties a config to its rule set pointed at neither of the things the response bragged about.
Iteration 2 - the wrong hypothesis, corrected by a human. With the divergence in hand I built a tidy theory: the store must be origin-blind - it's collapsing two different sets into one master. I even took it to the team as an architecture question, floating the idea of changing the store. A colleague shut it down in one line: the store is immutable; it builds exactly what you pass it; if a rule didn't survive, the monolith is what dropped it. He was right, and it reframed the entire hunt. The fault had to be entirely on the monolith's side. That correction saved me from "fixing" the one component that was behaving correctly.
That's the part I most want to land: the tests were green, the AI-assisted code generation was fast, the diagram was clean - and the thing that turned the investigation was a person who knew the invariant and a person on the business side who noticed the output was wrong. Tooling found none of that. Judgment did.
The root cause, at the level of the code
Under the store path, three pieces of monolith code conspired to produce the silent no-op. None is exotic; all survive review because each looks locally reasonable.
1. The creator was insert-only. The store master id is stable per (scope, entity). When a rule got re-stamped - say origin: own → origin: project, identical pattern/type/match/priority - the content changed but the resolved id stayed the same. The creator looked the id up, found an existing master, and returned the stale local mirror as-is. Create had quietly become "create-if-absent, ignore otherwise."
2. Correlation was by content hash. After computing the cascade, a sync builder had to match each computed item to the persisted master it produced. It did that by re-hashing the computed content and looking it up:
// The fragile correlation: re-hash the (possibly mutated) content and hope it matches.
$persisted = $createdByHash[$newRuleSet->getHash()] ?? null;
if ($persisted === null) {
continue; // ← silent drop. The scope row never re-links. Data lost, no error.
}
The monolith's hash included origin; the store's, at that point, did not. So the instant a re-stamp changed the local hash, the lookup missed, the item fell into a silent continue, and the scope row was never re-linked. Cascade descendants and shrinking removes vanished exactly this way - no exception, no log, no trace.
3. The action verdict was decided by a local hash, never by what persisted.
// The spurious verdict — decided entirely by the client's local, origin-inclusive hash.
$action = $old->hash() !== $new->hash()
? BatchUpsertAction::WasUpdated // origin changed → local hash changed → "updated!"
: BatchUpsertAction::WasUnchanged;
old.hash !== new.hash was true - origin had changed - so the endpoint stamped was_updated, while under it nothing had landed. Two hash functions is one hash function too many. Asymmetric identity between a client and its master is a silent-corruption generator: no status code will ever reveal it, because the code that writes the status never asks the database what happened.
And note again where the fault is not. The store did its one job correctly. This is the whole reason it was worth extracting: a small, immutable, content-addressed authority is easy to reason about. The mess was in the seam - the monolith's assumptions about a partner that had only just come into existence.
The fix, and the constraint it enforces
The fix had to do three things, and - the part I'm proud of - it had to do them without mutating a single existing interface or DI contract, because those contracts are shared with the paths that were already working. In Symfony, that's what the container is for.
Pattern 1 - one resolver, one method
A resolver resolves. It has exactly one method - resolve - and the fact that now matters (the store-assigned master that actually landed) enters as a nullable argument, not as a second method or a parallel executed/executedFor interface zoo:
declare(strict_types=1);
// One method. The persisted master is just an argument — nullable, because
// "nothing landed" is a real, first-class outcome that must be expressible.
interface ActionResolverInterface
{
public function resolve(
StateItemInterface $item,
?RuleSetInterface $persisted,
): BatchUpsertAction;
}
Every implementation is-an ActionResolverInterface everywhere it matters, so the runner routes by polymorphism, never by an instanceof ladder in shared code. Adding an implementation never touches the orchestrator.
Pattern 2 - carry the truth on the aggregate, captured at creation
The verdict needs one fact that didn't exist before: for a given computed item, what master did the store actually assign? The wrong way to deliver it is to thread a handle through five signatures. The right way is to capture it at creation time, keyed by the item's original input hash, and hang it on the aggregate the pipeline already passes around - behind a readable micro-interface:
// The deterministic calculated → persisted link, captured by submission
// alignment at creation — never re-derived from (divergent) persisted content.
interface PersistedMasterReadableInterface
{
/** @return array<string, RuleSetInterface> keyed by input hash */
public function createdByInputHash(): array;
}
The key move is when and by what key the link is captured: at the moment of creation, under the original input hash, before any re-stamping can make the content diverge. That single decision makes correlation deterministic instead of hopeful.
Pattern 3 - correlate by store master, and make the action honest
// Correlate calculated → persisted on the store-assigned master, captured
// under the ORIGINAL input hash — never a re-hash of the mutated content.
$persisted = $context->createdByInputHash()[$item->inputHash()] ?? null;
if ($persisted === null) {
// Reported on this item — never a silent `continue` that drops a cascade child.
$result->reportUnchanged($item);
continue;
}
$result->link($item, $persisted); // the scope row re-links to real persisted identity
And the verdict now comes from persisted identity, never a local hash:
final readonly class PersistedActionResolver implements ActionResolverInterface
{
public function resolve(
StateItemInterface $item,
?RuleSetInterface $persisted,
): BatchUpsertAction {
if ($persisted === null) {
return BatchUpsertAction::WasUnchanged; // nothing landed → never claim updated
}
return $persisted->isNewMaster()
? BatchUpsertAction::WasUpdated
: BatchUpsertAction::WasUnchanged;
}
}
The principle underneath all three: identity is computed by the store, and only the store. The client never re-hashes, never dedups, never decides created/updated/unchanged from a local hash. Same rules submitted from two places resolve to one store id; the verdict is desired-versus-actually-persisted identity.
The tail: symmetry, on both sides of the wire
Fixing the monolith closed the silent no-op, but left a quieter asymmetry exposed. The store hashed content without origin; the monolith hashed with it. After the honest-verdict fix, a pure re-stamp own → project correctly reported was_unchanged... but a genuine project → own markup move could still collapse into a single master, because the two sides didn't agree on whether origin was part of identity.
The resolution was symmetry, not cleverness - one field, appended on the side that was missing it:
// origin IS part of the canonical form — the client includes it, so the store
// must too. Two sets identical in patterns but differing only in origin (own vs
// project) must hash differently, else FindOrCreate collapses them and prod
// loses markup on brand→region moves.
serialized := field(r.Type, r.Match, r.Pattern, r.Priority, r.Origin)
The lesson generalizes past this bug: if a field is part of a rule set's identity for one participant, it must be part of it for every participant - or you get silent divergence no status code will ever reveal.
Testing: prove it against the real persist path, with real people in the loop
The uncomfortable truth that started everything is that the first conclusion - "there's no bug" - came from testing the wrong path. So the regression test drives the real store-persist path: RED before the fix, GREEN after. A dry_run or compute-only test would never catch it, by construction.
public function testRestampCascadesToEveryConfig(): void
{
$this->forceStoreWriteSource();
$this->store->willResolveDivergent(); // reproduce the prod shape in-process
($this->batchApply)($this->request); // the real batch apply, real persist path
// Assert on PERSISTED state, never the response body.
// Before the fix this is RED: the response claims was_updated, the DB has nothing.
$this->assertScopeLinkedToPersistedMaster('config:15');
$this->assertMaterializedRuleCount('config:15', 16);
}
And - the part the automated suite can't do alone - real people validate the business process. A 21-case production write-matrix drove the hunt; two cases stayed red until the identity fix, then flipped green. Those cases live in a shared harness with stable, numbered case ids, so a verdict is a coordinate (case, deploy-version) rather than a vibe:
| Case | before the fix | after the fix |
|---|---|---|
| 4 - plain create | ✅ | ✅ |
| 15 - restamp own→project | ❌ | ✅ |
| 16 - move project→own | ❌ | ✅ |
| 18 - shrinking REMOVE | ❌ | ✅ |
| 22 - cascade to N configs | ✅ | ✅ |
Unit tests prove the mechanism; a person running the real process proves it matters. During a migration you need the second one most, because the failures that hurt are the ones that return 200.
Three things make the automated side fast and honest: one PostgreSQL testcontainer for the whole run (not one per test - isolate at the data layer with a transaction that rolls back), a stub that reproduces the divergence deterministically, and assertions on what landed in the DB, not on the 200.
Where this goes next - the parallel run
This is a snapshot of a seam mid-migration, and the interesting question isn't "is it fixed" - it is - but "what does Stage 1 make possible." The answer is Stage 2: the cascade compute itself leaves PHP for a stateless Go service, and for a while two engines run in parallel over the same data. Here's the topology that Stage 1's identity fix unlocks:
LEVEL 1 two separate front doors — NOT connected to each other:
PHP monolith (PHP-FPM, legacy) Go API gateway (new service)
LEVEL 2 services:
domain-rule-map svc rule-set-markup service
identity (id+hash), its OWN DB, new cascade engine, Go,
a microservice FOR PERFORMANCE, STATELESS: routes + mirrors,
never touches the monolith DB no DB of its own — writes the
monolith DB directly
LEVEL 3 storage:
monolith PostgreSQL DB (both engines' only write sink) + DRM's own DB
Both engines resolve identity from domain-rule-map over gRPC. Both write the
monolith DB. The identity service never touches it.
Three things about this picture are only safe because of the Stage-1 fix:
- The new engine writes the monolith's DB directly. It has no store of its own yet - only the compute is moving, not the data. That's a deliberate, temporary coupling with an exit at Stage 4, not the end state. It's safe only because the service is stateless: it keeps nothing, so it can't drift from the DB it borrows.
-
domain-rule-mapis a separate microservice for one reason: performance - one fast, immutable identity lookup that both engines share over gRPC. It owns identity in its own DB and never touches the monolith DB. If Stage 1 hadn't made identity a single honest authority, running two write engines over one database would multiply exactly the divergence this article is about. - The two front doors are independent. The gateway fronts the new service; the PHP-FPM monolith is its own legacy front door. They aren't wired together - which is what lets an automated consistency agent drive both engines and diff persisted truth, case by case, until the new engine is provably backward-compatible. (That's Part 1.5.)
Every one of those is guarded by a constraint pulled straight out of this incident: one identity model (no second hasher on any client), honest action (verdict from persisted state, never a computed hash), cascade and shrinking-remove must actually persist, and symmetric identity across the wire. A fixed bug protects one line of code. A constraint plus a regression test protects the design - so when Stage 2 runs two engines over one database, it physically cannot resurrect a ghost we already exorcised.
AI as a multiplier, and what that means for a migration
I lean hard on AI coding assistants for work like this, and my honest, strongly-held take is that AI amplifies good engineers and exposes weak ones. It's a multiplier, not a crutch.
Look back at what actually caught and fixed this bug. A teammate noticed the business output was wrong. A colleague corrected a false hypothesis with one sentence about an invariant. A real production write, watched at the persistence layer, exposed the divergence. AI made generating the code - the harness, the interfaces, the fixtures - cheap and fast. It did nothing to tell me a was_updated could be a lie; in fact it happily helped me build a plausible, tidy wrong hypothesis that a human who knew the system dismantled in a line.
Point a multiplier at a disciplined process - real design, small interfaces, deterministic tests, verification against truth, and humans validating the actual business flow - and it collapses the old "fast or correct" trade-off into fast and correct. Point it at no methodology and it produces plausible-looking wrongness faster than you can review it. A silent no-op wearing a 200 is exactly that kind of wrongness.
This is Part 1 of a series
I started here because identity split across a migration boundary is where the breakup can hurt users most quietly. The rest of the series follows the other fronts:
Part 1: A 200 OK That Saved Nothing: the parallel run - two engines over one database, and how an agent proves they agree on persisted truth.
Part 1.5 - Two rule engines, one truth: the parallel run - two engines over one database, and how an agent proves they agree on persisted truth.
Part 2 - Design trade-offs: what "identity" means when a migration splits it, and why the ownership of a hash function is an architecture decision.
Part 3 - Performance: why the cascade is leaving PHP for a stateless Go service - the timeout budget and the one-pass in-memory merge.
Part 4 - Cutover strategies: running both write paths live behind a runtime master-switch, and flipping the master safely.
If you build serious backends - Symfony, Go, or the messy space between a monolith and the services growing out of it - follow along. And if you're already running two systems that must agree on identity: how are you proving they agree - on the response, or on what actually persisted? I'd genuinely like to compare notes.
Top comments (0)