DEV Community

Cover image for Nobody Owned the Value
Anton Brilliantov
Anton Brilliantov

Posted on

Nobody Owned the Value

#architecture #microservices #testing #debugging

Cover reading Nobody Owned the Value, with the word Nobody on a vermilion slab, the figure 7 moves in 48 days, and seven pins on a line with the fifth one, the frame with no writer, glowing red

A value with no named owner does not have a bug. It has an unfinished analysis.

👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. This series is about one write path that left the monolith and what it took to accept it as working. This part is about the six runs I spent going in a circle, and what was actually missing. Running notes live on my GitHub: github.com/brilliant-almazov.

This is how I do it right now, with the price attached - maybe you already do it better, maybe you'd do it another way.

The path in one paragraph

A user saves a set of matching rules; every domain under that account gets marked by them. Four participants: the Go API gateway as the only door, the rule-set-markup service which takes the write and folds the cascade, the domain-rule-map service which owns rule-set identity and recalculates the map, and PostgreSQL behind them - the monolith's database for the rule data, the identity service's own for identity.

One user action, four processes, two databases. And one value that decides everything downstream: which rule set is currently active for this object.

The series system map with the UI, the Go API gateway, the rule-set-markup service and the domain-rule-map service lit, the PHP monolith and the acceptance repository dimmed, and the edge from the rule-set-markup service to the domain-rule-map service drawn as a dashed red line with a bolt and the note step 4: no owner

Six runs, one symptom, no code to blame

The symptom was flat and unhelpful. A rule is saved. The response is 200. The read shows the new set. The engine keeps marking by the old one.

Nothing failed. No error, no timeout, no bad payload. The verification I could do at that point was: repeat it. Eight rounds, thirty seconds apart - the read returned one identifier, the engine kept answering with another, for four minutes straight.

Two halves: on the dark side what the interface said, 200, rule saved and the set visible in reads; on the red side what the engine did, still the old set through 8 rounds of 30 seconds with no error; a stamp reads six runs, no bug

That defect lived through six iterations of the run - the longest of the whole period - and every one of those iterations went looking for a mistake in code. There wasn't one.

What the analysis found, once it was done

Sitting down and mapping the value took about an afternoon. Here is the whole finding.

The value lives in four carriers across two databases:

  the monolith's database                       the engine's database
  ├── [1] scope row        ← written by the      ├── [3] engine pointer  ← written by the
  │        read by the door   write service      │        read on every     engine, five
  └── [2] config mirror    ← written by the      │        match             separate paths
           read by listings    write service     └── [3a] derived fallback — recovered from
                                                          its own map when [3] is empty

  and between the two databases: no foreign key, and there cannot be one
Enter fullscreen mode Exit fullscreen mode

Two database panels: the monolith's holds [1] the scope row and [2] the config mirror, both written by the write service; the engine's holds [3] the engine pointer, written by the engine on 5 paths, and [3a] the derived fallback, not written at all; between them a dashed red line with a bolt and the words no FK

The migration that created the engine's pointer says it in a column comment, in as many words: this is a logical reference, there is no foreign key. Which is correct - the two databases belong to different services. It also means the only thing keeping four copies of one value in agreement is the order of calls and one comparison in code.

Written out as calls, the write looks like this - five steps, four of which had an owner named somewhere in a specification:

  1. door → write service          validate + gather              touches nothing yet
  2. write service → engine        create identity                 identity row
  3. write service, one tx         insert scope row + mirror        [1] and [2]
  4. write service → engine        recalculate for this object      [3]  ← or not
  5. engine worker                 rebuild the map rows             map rows
Enter fullscreen mode Exit fullscreen mode

A dark terminal card listing the five calls of one write, from the door to the write service through the engine worker rebuilding the map, with call 4, write service to engine, recalculate [3], highlighted and a red note: step 4, no owner

Step 4 is the whole defect. Not its implementation - its ownership.

Second half of the finding: the right to write that value had moved seven times in 48 days.

Animated timeline of seven states of who may write the pointer, E1 to E7: the monolith owns all, identity moves, a second writer, the engine's own pointer, no writer, handed back, one named writer; E5, no writer, glows red and a stamp lands reading 2 days, no owner

And in one of those states - two days long - it belonged to no one.

Two correct changes, one defect

That state was produced by two commits, on the same day, in two repositories, by two different people.

  • In the write service: stop naming the set on rebuild - the engine can derive which one is active.
  • In the engine: the object's own active set wins over the one passed in the request.

Read either one on its own and it is an improvement. Read them together and they form a circle:

  the active set  ──is derived from──►  the map
        ▲                                 │
        │                          is written only by
        │                                 ▼
        └──────────is run against──────  a rebuild
Enter fullscreen mode Exit fullscreen mode

Three tiles in a row joined by arrows: the active set is derived from the map, the map is written only by a rebuild, a rebuild runs against the active set; a dashed red arrow returns from the last to the first, with a note: a fresh set has no map rows, so it is never active

A freshly written set has no map rows yet. So it can never become active - by any path at all. The rule was saved, the read showed it, and the engine had no way to reach it.

Two tiles, the write service's change, stop naming the set on rebuild, and the engine's, the object's own set wins over the one passed in, each stamped correct, both pointing down into one red tile: a set with no map rows can never become active, six runs, closed by both services on one day

No code review inside either repository could have caught this. Each diff was right. The contradiction lived on the seam, and the seam had no owner.

Why this is an analysis failure, not a testing failure

Here is the part that stung. The class of problem was already written down - seventeen days before the first detailed run, as a marker in a verdict matrix: write ownership for this value is not settled. It was noted. It was never turned into anything that could fail.

A note that nobody has to pass is indistinguishable from no analysis at all.

A large amber tile reading 17 days before the first run, next to the quoted marker from the verdict matrix, write ownership for this value is not settled, stamped noted, never checked

What the analysis owed, and didn't deliver:

  • a list of carriers of the value - all four of them, with the database each lives in;
  • one named writer per carrier, and what happens when the right moves;
  • a check that fails when two carriers disagree - which, being two reads compared, needs no infrastructure whatsoever;
  • the same check frozen before the code that changes the ownership.

That last one is the whole point of this part. Requirement-level checks are written and frozen before the implementation moves, so that a change of ownership has something to break against.

Animated bars shrinking from 26 numbered defects to 18 after the spec names each writer, 13 after fields come from the contract, and 2 that need a real run, with a note that 24 of 26 are caught before the code and a stamp reading on paper

Across the whole period this is not a small effect: of 26 numbered defects, 24 would have been caught by acceptance written before the code, and 21 of 39 defect-iterations die on a single question in the specification - who writes this value. That question is answered by reading a document, not by running anything.

What traces did, and what they didn't

Two honest halves.

Traces are what made the circle visible. Repetition proved a disagreement existed; the spans showed that the write path never called the step that moves the pointer, and that the engine resolved its own value from its own map. Without that, the argument stays "your service is wrong" / "no, yours is" - and I have lost days to that argument before.

But traces did not find the cause. They showed where the value came from at runtime. Who is allowed to put it there is not in a trace, not in a log, and not in either repository. It is a decision, and it either exists in writing or it doesn't.

What the fix actually looked like

Not a patch. A shape:

Four green tiles of the fix: the right split in two, transfer and first registration; the caller names the set; the pointer moves at write time, no rebuild needed; closed by both services on the same day; a stamp reads cured by one write

  • the right to write the pointer is split into two named operations: transferring it and registering it the first time;
  • the caller names the set again - whoever wrote it says which one it is;
  • the pointer moves at write time, so nothing has to be rebuilt for it to take effect;
  • and it was closed by both services on the same day, because the seam was fixed as a seam rather than as two separate bugs.

After that, the disagreement that had survived six iterations was cured by a single write.

A four-panel strip: what we did, 2 commits, each correct on its own; what broke, 6 runs, saved, 200, marked by the old set; why, step 4, nobody owned the pointer; what now, 2 reads, one equality check over all 392 objects

The check that catches this, and why it is trivial to automate

There is no schema constraint that can span two databases. So the guard is not a constraint - it is a query, run from outside, twice:

  read A   the door's listing for the object      →  rule set = X
  read B   the engine's answer for the same one   →  rule set = Y

  X != Y   →  the object is marked by a set it does not point at
  repeat the pair 8 × 30 s: a difference that survives repetition is not a lag
Enter fullscreen mode Exit fullscreen mode

A dark terminal card: read A, the door's listing, rule set X; read B, the engine's answer, rule set Y; the highlighted line X != Y, marked by a set it does not point at, with a note: 8 times 30 seconds, not a lag

Every disagreement found in the whole period came out of that shape: two answers compared, on all 392 objects rather than on a sample. It needs no new service, no CDC, no consistency framework - two HTTP reads and an equality check.

Four number tiles from one pass: 392 objects swept, 11 disagreements, 8 times 30 seconds per object, 0 new services

Which is exactly the kind of work worth handing to an automated executor: mechanical, wide, repeated per build, and boring enough that a person skips it.

Where automation carried it, and where it stopped

Being precise about the split, because "AI helps" on its own is not a claim worth printing.

What the executor did, and did well:

  • swept all 392 objects instead of the three-object playground, per build;
  • ran the two-read comparison per object and folded the disagreements into one table;
  • repeated the pair eight times at thirty-second intervals to separate a real disagreement from replication lag;
  • attached the trace contents - spans, durations, statuses - to every failure line, so the report survived the retention window;
  • did all of it again the next morning on a new build, without being re-briefed.

What it did not do, and could not:

  • decide that a value having four carriers is a problem worth naming - it had no reason to ask how many carriers exist;
  • work out who is entitled to write the pointer, because that is nowhere in the running system;
  • notice that two commits in two repositories, each individually sensible, formed a circle - each repository looked clean;
  • turn the note in the matrix into a check that had to pass.

Every one of those is a decision, and the analysis that produces them is a person sitting down for an afternoon with the migrations and the specs. What the automation changes is the price of the consequence: once the owner is named, the check that enforces it costs two reads and runs forever.

That is the honest shape of the multiplier. Fast iteration amplifies whatever analysis exists; where analysis is missing, it amplifies the circling instead - six iterations of confidently reproducing a symptom whose cause was never in the code.

A torn paper receipt for a value with no owner: runs looking in code, 6 runs, no bug; the same pair repeated, 8 times 30 seconds, no error; analysis on paper, an afternoon, no owner; price 6 runs, with a stamp reading no bug

Anatomy of a service split — Part 3. Next: the other half of the same lesson - what happens when the system you compare against is being fixed at the same time, and the response code says one thing while the database holds another.

If you do this better, tell me how you record who owns a value across services. If you have been through this, what did your seam turn out to hide? If you see it differently, say where writing the check before the code costs more than it returns. How is it solved on your side, and what broke there?

Top comments (0)