Why the rule-set write path is leaving PHP for a single stateless Go service - the timeout budget that forced it, the one-pass in-memory cascade that replaces thousands of database round-trips, and how to measure the win without fabricating a hero number.
👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go. For a while now I've been on the kind of project that teaches you the most: carefully breaking a large PHP monolith into Go microservices while it's still very much alive and serving a real business. The running notes live on my GitHub: github.com/brilliant-almazov. No hype, just the real work. Part 1 was about a silent 200 OK that saved nothing - the bug that taught me two systems, each with its own database, must agree on identity. Part 2 went deeper on that identity question. This part is about the reason the whole write path moved at all: it was too slow, and on the biggest writes it timed out in production.
Where this stands today: the extraction is built and deployed. The new Go engine now runs in parallel with the old PHP one - the same cascade computed by two engines at once - while an automated agent proves they agree before either is trusted. That parallel run is its own story: a companion to Part 1 covers it in full. This part is the why underneath it - the timeout that forced the extraction, and the shape of the service that replaced the slow path.
The report that started it wasn't a bug. It was a dashboard. Our error tracker kept lighting up with the same signature: the Go API gateway returning 524 on rule-set writes, clustered on the largest parent nodes. 524 is the gateway giving up - the origin took longer than its ceiling (~100s) to answer, so the connection got cut out from under it. The client retried. The retry was just as slow. Same node, over and over, a steady drip of red.
Nothing was wrong, exactly. The writes that finished were correct. They just didn't always finish.
The migration, in stages
A big-bang rewrite is a bet you make once and lose slowly. The alternative - the one that actually works on a live business - is a strangler fig: move the seam one named stage at a time, and never let two migrations run at once. Here is the seam this whole series lives on - the rule-set write path - laid out as stages, with a marker on where this article stands:
Stage 0 Monolith only.
PHP computes the cascade, persists the rule data, owns identity implicitly.
Stage 1 Identity extracted. (Part 1)
A small Go service becomes the immutable, content-addressed master of
identity (id + hash). PHP still computes and persists the rule DATA.
Stage 2 Write logic extracted. <-- THIS ARTICLE
The cascade compute leaves PHP for a stateless Go service that writes
directly into the monolith's DB. It moved because the PHP path TIMED OUT.
Stage 3 Prove they agree. (Part 1.5)
An agent drives both engines and diffs persisted truth, case by case.
Stage 4 Flip, then delete.
Promote the Go engine, retire PHP, and only then move the DATA out.
A few years ago this was one PHP/Symfony monolith on PostgreSQL. We've been carving it into focused Go services one seam at a time - the old system paying the bills while each new service earns its place next to it. Part 1's seam was identity: a Go service, the rule-set store, became the immutable, content-addressed authority for "is this the same rule set or a different one?" This part is the next seam over - the write and cascade path itself - and unlike Part 1 it's a performance story, not a correctness one. The trigger was blunt: on the biggest tenants the PHP write path stopped finishing inside the gateway's budget.
Primer: what the cascade actually does
Strip away the domain and it's simple. The system stores classification rules attached to nodes in a three-level hierarchy that cascades top-down:
CLIENT rules here apply to everything beneath
└── PROJECT rules here apply to every config of the project
└── CONFIG a specific target (search-engine × device × locale)
Writing a rule set at a parent doesn't just save that node - it fans out. Every descendant's effective rule set is recomputed by merging what it inherits from above with what it owns locally, with a clear precedence: a narrower level overrides a broader one. The rule is replace-own-keep-inherited - a node's own rules replace its previous own rules, but everything inherited from ancestors is carried down untouched. The fully-resolved result for each node is its materialized state - what a given CONFIG actually sees at request time.
The cost hides in the fan-out. A CONFIG-level write touches one node. A CLIENT-level write can touch every CONFIG under that client - realistically a few thousand of them for a big tenant. That's the write that returns 524.
Why PHP timed out: O(descendants) round-trips inside one request
The PHP cascade was written the way you'd naturally write it against a database: node by node. For each descendant it did a small read-modify-write cluster - load the node's set, merge the parent's rules in, resolve identity against the rule-set store, write the mirror. Correct, readable, and completely fine for one node. But it ran serially, inside one synchronous request, once per descendant.
For a CLIENT with a few thousand CONFIGs that's a few thousand round-trip clusters, chained - each a millisecond or three of network and query time that's invisible at N=1 and lethal at N=thousands. Wall-clock climbed linearly with subtree size until the biggest parents drifted past the gateway's ~100s ceiling. The database wasn't the bottleneck. The chattiness was - thousands of sequential trips across a wire, each waiting for the last.
You don't fix that by making PHP faster. You fix it by stopping the round-trips.
The shape of the new service
The extracted service - I'll call it the rule-set-markup service - is deliberately small. It is:
- One synchronous gRPC process. One binary. No worker. No queue. No async anything. Desired state comes in on a gRPC call, the fully-persisted result goes back on the same call.
- Stateless. It owns zero persistent state of its own. It machines over two things it doesn't own: the monolith's database and the rule-set store. Restart it, run N replicas, kill it between phases - all equivalent, because there's no in-process state to lose.
One consequence deserves its own sentence, because it looks like a sin until you see the stage it's in: the service has no database of its own - it writes straight into the monolith's PostgreSQL. That's deliberate. Only the compute (the cascade) is moving out right now; the rule data still belongs to the monolith, which is still the source of truth. Moving the data is a separate, later migration. Letting the new engine write the monolith's own tables means its output lands exactly where the old engine's did - which is precisely what makes "do the two engines agree?" a question you answer by reading one database. The coupling is temporary, documented, and safe only because the service is stateless: it keeps nothing, so it can't drift from the database it borrows.
The whole thing is one method behind one interface:
// The entire service surface: desired state in, persisted result out.
// One method — no per-level variants, no per-node-kind branches.
type Markup interface {
Resolve(ctx context.Context, desired DesiredState) (Result, error)
}
Inside Resolve, four phases run in strict order: merge → cascade in memory → persist in one transaction → post-commit notify. No phase reaches for a resource a later phase hasn't earned yet.
Phase 1-2: the whole cascade, in memory, in one pass
The core move is to stop treating each node as a database trip. The service loads the entire affected subtree with one query, then does the whole cascade in memory - a single top-down walk, parent to child, carrying the inherited set down as it goes.
The cascade itself is one small interface with one method - the direct descendant of Part 1's resolve:
// A resolver resolves. One method. Inherited set + a node's own set → its
// effective set. Pure function of its inputs — no I/O, no clock, no DB.
type Resolver interface {
Resolve(inherited, own RuleSet) RuleSet
}
// replace-own-keep-inherited: own rules win at their own level; everything
// from ancestors is carried down as-is.
type CascadeResolver struct{}
func (CascadeResolver) Resolve(inherited, own RuleSet) RuleSet {
effective := inherited.Clone()
effective.ReplaceOwn(own)
return effective
}
And the one-pass walk that applies it across the loaded subtree:
// One top-down pass. Each node's resolved set becomes the inherited input for
// its children. No recursion into the database — the subtree is already in RAM.
func (m *markup) cascade(root *Node, desired DesiredState) []Resolved {
out := make([]Resolved, 0, root.DescendantCount())
var walk func(node *Node, inherited RuleSet)
walk = func(node *Node, inherited RuleSet) {
own := desired.OwnFor(node.ID) // the write; empty for untouched nodes
effective := m.resolver.Resolve(inherited, own)
out = append(out, Resolved{NodeID: node.ID, Set: effective})
for _, child := range node.Children {
walk(child, effective)
}
}
walk(root, desired.InheritedInto(root.ID))
return out
}
Round-trips for the whole cascade: one read for the subtree, regardless of how many thousand descendants it has. The identity resolution against the rule-set store is batched too - the computed sets go over in one call, not one-per-node. O(descendants) collapsed to O(1) in trips.
Phase 3: persist in one transaction - and tx is not an argument
Everything the cascade produced lands in one transaction. Either the whole fan-out commits or none of it does; there's no half-cascaded subtree to reconcile after a crash.
The part I care about stylistically: the transaction is a property of the store, never a parameter threaded through method signatures. A func Persist(ctx, tx, rows) is procedural and leaks the transaction into every caller. Instead the executor (pool or tx) is a field, and a scoped instance is minted with WithTx:
// The executor — pool or tx — is a field, never a method argument.
type Store struct {
exec Executor
}
// WithTx returns a tx-scoped Store. Same methods, now bound to the tx.
func (s *Store) WithTx(tx pgx.Tx) *Store {
return &Store{exec: tx}
}
// The tx lifecycle is encapsulated here. Callers never see BEGIN/COMMIT/ROLLBACK
// and never hold a tx handle — they get a scoped Store and use it.
func (s *Store) RunInTx(ctx context.Context, fn func(context.Context, *Store) error) error {
tx, err := s.exec.Begin(ctx)
if err != nil {
return fmt.Errorf("begin: %w", err)
}
if err := fn(ctx, s.WithTx(tx)); err != nil {
if rbErr := tx.Rollback(ctx); rbErr != nil && !errors.Is(rbErr, pgx.ErrTxClosed) {
return errors.Join(err, rbErr)
}
return err
}
return tx.Commit(ctx)
}
So the persist phase reads as one atomic block, with no transaction plumbing visible at the call site:
err := m.store.RunInTx(ctx, func(ctx context.Context, tx *Store) error {
return tx.PersistCascade(ctx, resolved) // one batched write, tx-scoped
})
if err != nil {
return Result{}, fmt.Errorf("persist cascade: %w", err)
}
Error identity, everywhere, goes through errors.Is / errors.As - never err == x, never a type switch - so a wrapped error five layers down still matches. (The ErrTxClosed check above is exactly why: something upstream may already have closed the tx, and a bare == would miss it the moment anyone adds a %w.)
Phase 4: post-commit notify - after the wire is quiet
Recalculation of anything downstream is not part of this transaction. Once the commit succeeds, the service tells the domain-rule-map service which CONFIGs actually changed, and that system owns the recompute. Notify-after-commit, never inside it: the write path stays short, and a slow or unavailable consumer can never hold the write transaction open. If notification fails, the committed state is still correct - the recalc is re-derivable from what landed, not from a message we have to guarantee delivery of.
Why Go here, and not Symfony
I write plenty of the write domain in Symfony and I'd defend it. But this service isn't rich domain logic - it's fan-out and serialization: load a subtree, merge sets in a tight loop, serialize thousands of rows, write them once. That workload rewards tight tail latency and lean, predictable memory, and punishes per-request warmup and per-object overhead - exactly Go's shape. The decision wasn't "Go beats PHP." It was that this specific shape (hot, wide, serialization-bound, no deep domain rules) is what Go is for, and the domain-heavy write logic that stays in the monolith is what Symfony is for.
Statelessness is what makes the cutover safe
Here's the property that matters more than the speed. The service holds no state of its own - every input is loaded fresh from the monolith DB and the rule-set store on each call, every output written straight back. There is nothing in the process to be inconsistent with the world.
That's what makes the parallel run safe - and it's not upcoming, it's live. The old PHP path and the new Go path run side by side today, traffic flips per-request, the Go service restarts mid-migration, scales to N replicas, or gets killed between phase 2 and phase 3 - none of it can corrupt anything, because a killed call simply didn't commit and the next one recomputes the same cascade from the same source of truth. Statelessness isn't a résumé buzzword here; it's the precondition that lets me change the write path under live traffic without a flag-day gamble. (Proving the two engines actually agree, run-for-run, is its own article.)
Measuring the win honestly
I'm not going to hand you a hero number bolted to a real tenant - that's how anonymized case studies quietly leak client identity, and the shape of the win is the honest part anyway.
The unit that matters is round-trips per write, because that's what scaled with subtree size and blew the budget:
| PHP client-side cascade | Stateless Go service | |
|---|---|---|
| DB round-trips |
O(descendants) - one cluster per node |
O(1) - one subtree read |
| Store identity calls | one per node | one batched call |
| Cascade | node-by-node, serial | one in-memory pass |
| Persist | interleaved per node | one transaction |
| Worst-case parent | past the gateway's ~100s ceiling → 524
|
sub-second for the same subtree |
The budget frame kept me honest: the gateway gives a write ~100 seconds before it returns 524. The PHP path spent that budget on network chatter - thousands of sequential trips, each mostly waiting. The Go path spends almost none of it there: one read, one in-memory pass, one write. I trust that framing because it's a structural claim - trips went from linear to constant - not a benchmark I can accidentally cherry-pick.
AI as a multiplier - the same throughline
I lean hard on AI assistants for work like this, and my honest take hasn't changed across the series: AI amplifies a good engineer and exposes a weak one. Generating this service - the interfaces, the walk, the store scaffolding, the tests - was fast. What AI did not do was tell me the problem was round-trip count rather than "PHP is slow," or that the fix was to load the subtree once rather than to parallelize the existing per-node loop (which would have kept the chattiness and just raced it). It didn't decide that the transaction belongs to the store as a field, or that notify has to live after the commit. Point the multiplier at a diagnosed problem and a designed shape and it collapses the old "fast or correct" into fast and correct. Point it at "make the cascade faster" with no diagnosis and it will cheerfully help you parallelize the wrong thing at high speed.
This is Part 3 of a series
The seam is diagnosed, designed, and built stateless on purpose. What's left is the scary part - swapping it in under live traffic.
-
Part 1 - A 200 OK that saved nothing. The silent
200 OK, and why identity must live in exactly one place. - Part 1.5 - Two rule engines, one truth. The service in this part is now live in parallel with the PHP one - how an agent proves the two engines agree, through the gateway, on persisted truth.
- Part 2 - Who owns a hash function. What "identity" means when two systems each own a database, and why owning a hash function is an architecture decision.
- Part 4 - Flipping the master live. Running the old PHP cascade and the new stateless Go service side by side behind a runtime master-switch, flipping traffic safely, and retiring the old path without a big-bang deploy. Statelessness (this part) is what makes that flip reversible - Part 4 is how you actually pull it.
If you build serious backends - Symfony, Go, or the messy space between a monolith and the services growing out of it - follow along. Concrete, code-first, honest about the mistakes.
And if you've moved a write path off a monolith: how did you measure the win without a number that fingerprints your client? I'd genuinely like to compare notes.
Top comments (0)