DEV Community

Cover image for The Client Decides How Much Work You Do
Anton Brilliantov
Anton Brilliantov

Posted on

The Client Decides How Much Work You Do

#architecture #performance #scalability #microservices

Cover reading The Client Decides How Much Work You Do, with the word Client on a vermilion slab, the figure 366 children behind one call, and four bars for 3, 100, 253 and 366 children with the last one red

A synchronous answer over work the caller sizes is not a contract. It is a promise the caller gets to break for you.

👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. This series is about one write path that left the monolith and what it took to accept it as working. This part is about the thing we never wrote down before shipping it: how much work one caller is allowed to ask for. Running notes live on my GitHub: github.com/brilliant-almazov.

This is how I do it right now, with the price attached - maybe you already do it better, maybe you'd do it another way.

The path in one paragraph

A user saves a set of matching rules; every domain under that account gets marked by them. Four participants: the Go API gateway as the only door, the rule-set-markup service which takes the write and folds the cascade, the domain-rule-map service which owns rule-set identity and recalculates the map, and PostgreSQL behind them - the monolith's database for the rule data, the identity service's own for identity.

The shape that matters here is not the topology, it is one sentence about the work: one call from the user, and then a pass over every object below the one they addressed. Save rules on a leaf and it is a small write. Save the same rules at the top of an account and it is the same call with a different amount of work behind it - decided entirely by how big that account happens to be.

The series system map with the UI, the Go API gateway, the rule-set-markup service and the domain-rule-map service lit, and a red note on the call from the gateway into the rule-set-markup service: 366 children, one call

Three objects was the whole test bench

For the first weeks the checks ran against a playground of three objects. Everything passed, and the passes were honest: real requests, real assertions, real data written and read back.

What three objects cannot show you is a slope. Every run cost about the same, so cost looked like a constant - a property of the endpoint. It is not a property of the endpoint. It is a property of the caller.

The slope only appeared when a run went across the whole environment: 76 top-level objects, 392 configurations, roughly 2 300 map rows - and, at the account level, one entity with 366 children.

Four number tiles for the whole environment: 76 top-level objects, 392 configurations, about 2 300 map rows, 366 children in one account

Three against 366. The sample was 1/122 of the largest legitimate case in the same environment, and nothing in the sample said so.

Two halves: the test bench, 3 objects with every check passed, against the largest account, 366 children below one call, stamped 1/122

What one write actually costs

Two writes from the same set of traced runs, drawn to the same scale:

  one object addressed directly
    121 spans · 4 services · 53 database round trips (11 of them repeats)

  one account addressed at the top, 366 children below it
    10 822 spans
Enter fullscreen mode Exit fullscreen mode

Two columns of traced spans: one object at 121, one account at 10 822, with a note: about 30 spans per child, derived as 10 822 divided by 366

10 822 ÷ 366 ≈ 30 spans per child. That number is derived, not measured directly - I divided one measured total by one measured count - and it is enough to answer the only question that mattered: is the cost of this call flat, or does it track the caller's data?

It tracks the caller's data. Which means the operation has a slope, and the response has a ceiling, and nobody had ever written either of them down.

Round trips are the unit that scales

The other measured number on that single-object write is the one I now put in the capacity budget instead of milliseconds: 53 database round trips, 11 of them repeats of a query already made in the same call.

Milliseconds are a property of the day - of the machine, the cache, the load next door. Round trips are a property of the code, and they are the thing that gets multiplied when the caller is bigger. Fifty-three round trips for one object is a number you can reason about before anything is deployed; "about a second" is not.

Three number tiles: 53 round trips for one object, 11 of them repeats, and a 20 s budget that never grows

  per one object            53 round trips · 11 of them repeats
  per account of 366        the same shape, multiplied
  what a repeat costs       nothing on the first call, everything on the thousandth
Enter fullscreen mode Exit fullscreen mode

So the two useful sentences about this endpoint, both available without a stopwatch:

  • its cost per unit of caller data is a small number of round trips and about 30 spans;
  • its budget is a fixed 20 seconds that does not grow with anything.

One of those scales with the customer. The other does not. That is the entire finding, and every measurement after it is decoration.

A constant against a line

The deadline on that call is a constant: 20 seconds, the same 20 seconds for a leaf with two children and for an account with 366. The work behind it is a line through the origin.

One observed run at the top of that account: the caller was cut off at 20.26 s, and the work carried on for another 8.7 s after the answer had already gone out - about 28.96 s of work behind a 20-second promise.

A torn paper receipt for one account-level write: 366 children below, 10 822 spans, 28.96 s of work against a 20 s budget, over the deadline 8.7 s, stamped still written

Divide again: 28.96 s over 366 children is roughly 79 ms per child. Same caveat - derived, one measured duration over one measured count, and it is a per-account average rather than a per-child guarantee. It still puts a number on the crossing point:

   children     derived work         budget
        3   ▏            0.24 s      1.2 % of it
      100   ████▏         7.9 s      40 %
      253   ██████████▏  20.0 s      the last one that fits
      366   ██████████████▏ 28.96 s  145 % - over by 8.7 s
                         ▲ 20 s deadline
Enter fullscreen mode Exit fullscreen mode

Animated bars of derived work against a dashed 20-second deadline: 3 children 0.24 s, 100 children 7.9 s, 253 children 20.0 s at the line, 366 children 28.96 s crossing it, with a note: over the 20 s deadline by 8.7 s

Around 253 children is where this endpoint stops working, and the number of accounts above that line is a product fact, not an engineering one. Nobody had asked for it. Part 1 of this series recorded what that looks like in aggregate: deadline failures on 13 of 76 objects, and then, after a change meant to help, on 27 of 76.

The failure is not "the service is slow". A service that is uniformly 3x faster moves the crossing point from 253 to 759 children and fails on the next account that has 800. Optimisation moves the line. It does not change the fact that there is one.

The answer that arrived before the work stopped

The second half of that run is the part I would want to know before designing anything similar.

At 20.26 s the caller got its failure. At 28.96 s the work finished - and it finished successfully, in the database, exactly as if nothing had gone wrong. Which is what happened: nothing did go wrong. A deadline is a decision by the caller to stop waiting. It is not a rollback.

  0 s                                  20.26 s               28.96 s
  ├───────────────────────────────────────┤───────────────────┤
  │  work in progress                     │  8.7 s more work  │
                                          │
                          caller is told: the call failed
                                                              │
                                          database holds: the write landed
Enter fullscreen mode Exit fullscreen mode

Animated trace of one account-level write: the caller bar ends with a warning, failed at 20.26 s; the work bar runs on to 28.96 s; the database bar lands after it, 8.7 s later: written; a stamp reads not a rollback

So the response code and the state of the database disagree, legitimately, and any check that treats the code as the verdict will be wrong in a way that repeats. Everything downstream inherits this: a retry on timeout is a second write, not a repair; an alert on 5xx counts calls that did their job; a parity comparison against a reference system reads a failure on one side and a row on the other and calls it a defect.

A dark log card of one account-level write: at 20.26 s an error, call failed on the deadline; at 28.96 s, highlighted, the write committed; a red note reads assert both, separately

Three notes on reading any of these figures, because I would rather hand over the caveats than the confidence:

  • The measurements come from three different observation windows with different instrumentation. Comparing across them compares methods of observation, not states of the system.
  • Round-trip time to the environment sits at a steady 220-260 ms regardless of endpoint. It is not part of the work above, and it is not ours.
  • No share is quoted without its absolute value. 28.96 s over a 20 s budget, never "145 %" on its own - a percentage of a moving denominator is the easiest honest-looking way to say the wrong thing.

The rule I now write into the acceptance notes before the endpoint exists:

A deadline bounds the wait, not the work. Response code and stored state are two separate measurements and must be asserted separately.

A dark quote card reading a deadline bounds the wait, not the work, stamped code is not state

The questions that belong in the design, not the postmortem

None of the above needed a trace to predict. It needed five answers, and none of them is an engineering answer:

Question What we had before the sweep What the sweep gave
How many children does a caller have? "a few" - three, in the test bench 366 at the largest account, 76 top-level objects, 392 configurations
What does one child cost? unknown, never asked ~ 30 spans, ~ 79 ms - derived
Where does the cost cross the deadline? no deadline written down ~ 253 children, at a 20 s ceiling
What happens when the data grows? not considered the crossing point moves into the middle of the population
What do we do for callers past the line? nothing defined still open - the reason for the next section

Five capacity questions with the answer before and after the sweep: children per caller, three then 366; cost per child, never asked then about 30 spans and 79 ms; deadline crossing, not written then about 253 children; as data grows, not considered then the line moves in; past the line, nothing then still open

This is what "know your traffic" means concretely, and it is cheap: four of those five rows are one query against your own data plus one division. We paid for them in iterations instead, which is the expensive currency.

What changed in how this gets accepted

The practices below cost nothing to adopt and are the reason the same class of surprise does not repeat:

  • Every run includes the largest caller in the environment, not a convenient one. A sample that is 1/122 of the largest case tells you about the sample.
  • Response code and stored state are asserted separately. Two checks, never one; the timeout case exists precisely because they can disagree.
  • Round trips per unit of caller data are recorded per build, next to the durations. A shape regression shows up there first and shows up nowhere else - the response stays 200 the whole time.
  • The population is part of the specification. How many callers sit above the crossing point is a line in the requirements, alongside the deadline, before anyone writes a handler.

None of that is tooling. It is four sentences written down before the code, which is the cheapest artefact in this entire story and the one we produced last.

A four-panel strip: what we did, 3 objects on the bench, all green; what broke, 20.26 s, the caller cut off while the write ran; why, 366 children, cost tracks the caller; what now, the largest caller in every run

Where this goes: change the shape, not the step

The exit is not a faster cascade. It is a different shape of operation.

  compute on request                     compute ahead, select on request
  ──────────────────────                 ────────────────────────────────
  caller sets the size                   size is known before the request
  cost grows with the caller             cost is paid where it can be scheduled
  deadline is a constant                 the request is a selection over ready rows
  failure mode: cut off mid-write        failure mode: stale, and visibly so
Enter fullscreen mode Exit fullscreen mode

Two tiles: compute on request, marked with a cross - the caller sets the size, the deadline is a constant, cut off mid-write; compute ahead and select, marked with a check - the size is known ahead, cost paid when scheduled, stale and visibly so

If the result is computed ahead of the request, a change of rules stops being a write that fans out and becomes a different selection over data that already exists. The question "will we fit inside the deadline" is then answered by the form of the operation rather than by optimising a step inside it. The cost does not vanish - it moves to a place where it can be scheduled, measured and bounded, instead of a place where a user's account size decides it.

I am not going to pretend that is free, or that it is done, or put a date on it. What I will say is that we arrived at it from the numbers above rather than from taste: an operation whose size is set by the caller and whose answer is synchronous has a population of callers it cannot serve, and you find out which ones by measuring your own data, not by tuning.

The multiplier line

The full sweep that produced these numbers was cheap because it was automated: run every object, pull the trace, fold it into counts, run it again on the next build and print the difference. What automation could not do is say how large a legitimate caller is allowed to be - that is a product answer, and it has to exist before the endpoint is designed, not after a run finds the ceiling. Fast measurement amplifies whoever wrote down the limits; where no limit was written, it produces a very precise description of a design that was never bounded.

Anatomy of a service split — Part 5. Next: the contract itself - a value that exists, is stored and is correct, and still cannot be distinguished from "not set" once it is on the wire; and the ordering rule nobody ever specified, which held as a norm for 17 months because it was a property of an implementation.

If you do this better, tell me where you write down the largest caller your synchronous endpoints are allowed to serve. If you have been through this, what did your timeout turn out to have written anyway? If you see it differently, say where precomputing costs more than it returns. How is it solved on your side, and what broke there?

Top comments (0)