DEV Community

Cover image for One Row, Twelve Stops
Anton Brilliantov
Anton Brilliantov

Posted on

One Row, Twelve Stops

#architecture #softwareengineering #testing

Why this is integration testing and not a unit test with a nice name: one acceptance row walks the whole chain and comes back.

👋 Hi, I'm Anton - a software engineer. These days I work on a platform: contracts, generated code, data paths and the tooling around them. Running notes live on my GitHub: github.com/brilliant-almazov.

The path below is the one the acceptance agent recorded for one real row, measured on two local test stands; the domain that row belongs to is not all green yet, and the four open rows in it are named as open.

This is part 2 of the series on one week of integration acceptance across a whole chain of services, after months of doing it the slow way. Maybe you already do it better, maybe you'd do it another way - this is how it went, and where it stops.

In real life

The whole system is a zoo. The services are the residents, the bus is the paths between the enclosures, and acceptance is the vet - Pickle, a guinea pig with a notebook and a card for each of eighteen enclosures. Nora is the keeper. Biscuit, the courier dog, carries things around.

Nora gave Biscuit a parcel at the ticket desk: "Take it to the far enclosure." Biscuit asked the one question a courier always asks: "And back?" And back. On the way there is a gate at every enclosure, and at every gate someone stamps the parcel. At gate seven Biscuit sighed, "Stamp number seven...", and Pip, the guide, said: "Five more. Keep going."

At the far enclosure Miso the cat opened the parcel, wrote her answer inside and handed it back. Biscuit looked at the path behind him: "Now I walk all the way back?" He did. At the desk Pickle counted twelve stamps and put one tick on one line of one card. Biscuit, tired and proud, asked: "Tomorrow on the other stand?" Yes. Tomorrow on the other stand.

A four-panel comic: Nora says take it to the far enclosure, and Biscuit asks and back; Biscuit says stamp number seven, and Pip says five more, keep going, next to a turnstile; Miso says here is my answer, and Biscuit, worried, asks now I walk all the way back; Pickle, proud, says twelve stamps, one tick, and Biscuit asks tomorrow on the other stand

The parcel is one acceptance row. The gates are the services, the bus and the stores it passes. The far enclosure is an emulator that plays the outside world. The stamps are spans of one trace. And one tick means: this whole walk worked, there and back, on this build.

What a row is, and what it is not

A unit test checks one piece with fakes around it. It is fast, cheap and good at what it does. It cannot tell you whether the queue service and the collector agree on what a "placed order" means, whether a webhook lands on the right exchange, or whether the result comes back to the place that asked for it.

An acceptance row in this plan is a behaviour of the whole chain. Take one row from the domain "gathering on the emulator". To pass, it has to start where real work starts - a task created upstream - and end where the answer is expected. Nothing in between is faked except the outside world, which is replaced by an emulator that answers like it.

The twelve stops

The acceptance agent recorded the path of that one row like this. I number the stops so the figure and the text agree:

# Stop How the row gets there
1 a task in the upstream system where work is created
2 the task service takes the task and slices it
3 the queue service gRPC
4 the collector server from the queue
5 the dispatcher inside the collector
6 a provider agent inside the collector
7 the emulator HTTP, as the outside world
8 the webhook the emulator answers back
9 the result bus RabbitMQ
10 result processing reads the bus
11 the queue service and the task service the result comes back
12 back upstream where the answer was expected

Twelve tiles with faces in three rows of four, joined by arrows: 1 a task, 2 task service, 3 queue service, 4 collector server, 5 dispatcher, 6 provider agent, 7 the emulator, 8 webhook, 9 result bus, 10 result processing, 11 queue and task, 12 back upstream; pills read tested on two local stands and gRPC, HTTP, RabbitMQ

And that is only the main road. Along the way the row also touches side paths: request bodies leave through an outbox into object storage, via three separate storage binaries. Several services keep their state in PostgreSQL. The bus runs on RabbitMQ virtual hosts. Traces are collected in Tempo, metrics in VictoriaMetrics.

Three tiles with faces in a chain: the outbox, 3 storage binaries (proud), object storage; a pill under the second arrow reads S3; pills read tested 74 of 74 rows and a binary knows only its own keys

That side path has its own domain in the plan, and its own lesson: a storage binary once carried another binary's configuration keys. After the cleanup, a check that went 0 of 13 became 13 of 13 - a binary knows only its own keys. The storage domain passes 74 of 74 rows.

Every gate keeps its own card

Each stop on that road is also covered by its own domain of the plan, so one walking row is not the only proof that a stop works. Here are the domains the path crosses, as the plan counted them on the last day:

Bars of the share of rows passing in the domains on this path: final processing 40 of 40; task, queue, collector 31 of 31; the webhook of provider B 30 of 30; gathering on the emulator 107 of 111, 96.4 percent, in amber; storage, three binaries 74 of 74; Biscuit says every gate has its own card too; pills read measured and one domain still open: 4 rows

The row I walked belongs to the one domain that is not at a hundred. Gathering on the emulator passes 107 of 111 rows, 96.4 %. The four open rows are open for named reasons, and the next two parts of this series go through them - which enclosure, and whose bug.

A donut: 96.4 percent, 107 of 111; legend: passing 107, still open 4; Pickle, worried, says not all green, yet; pills read measured and 4 rows open

What a verdict carries

A row does not pass because a status line says "ok". It passes with proof. Every verdict carries the trace id of the request that walked the path, and every verdict is checked on two local stands. The form is short:

# a verdict - a form, values elided
row        <domain>-<n>
verdict    passing
trace      <trace id: the spans of all twelve stops>
stands     stand one: passing, stand two: passing
build      <the live build the row ran on>
Enter fullscreen mode Exit fullscreen mode

The trace is what makes the verdict checkable by someone who was not there. Open it, and the twelve stops are the spans: the gRPC call into the queue service, the HTTP call into the emulator, the publish on the bus, the consume in result processing. If the row fails, the trace shows the last stop it reached, which is usually the first question anyone asks.

An animation: a parcel walks five tiles - task, queue, collector, emulator, results - and back to the task; the pills change from a task is created upstream, to the queue places the order over gRPC, to the emulator answers over HTTP, to results ride the bus over RabbitMQ, to back upstream, one verdict, one trace

Where the unit tests come in

None of this replaces unit tests, and the rules repository - the shared skills and hooks every agent session loads - now pulls the two closer together. Its lesson of the same week reads: acceptance found defects only on the stand. The conclusion is a shift left. A service change carries a "ready for acceptance" checklist of sixteen items, and every acceptance row that passes on the emulator becomes a CI test named after the row.

# shift left, as the rules repository states it (paraphrased)
a service change        -> carries a "ready for acceptance" checklist, 16 items
a row passes on the
emulator                -> becomes a CI test named after the row
the gain                -> a defect is found in the change in minutes,
                           not after a deploy and an acceptance pass
Enter fullscreen mode Exit fullscreen mode

So the walk of twelve stops happens on the stand, and its shape is then kept in CI, where it runs on every change. The stand finds; CI remembers.

Before and after: a unit test, 1 package, fakes around it, 0 stores, nothing real is touched; one acceptance row, 12 stops, services, a bus, an emulator, 2 stands, a trace on every verdict; a pill reads tested

The sign on the door

Nora, proud, waving next to a big sign reading a verdict carries a trace on two stands; Nora says a stamp from every gate, or it did not go; on the right the mechanism: a trace id on every verdict, two local stands, a passing row becomes a CI test; pills read built and tested

What went wrong

For months, the earlier acceptance did not walk rows like this. It checked cases per version, and what it could not do was say which stop on a long road broke. In this week the expensive lesson was a different one: defects in the chain showed up only on the stand, after a deploy, because nothing earlier walked the chain. That is what the shift left is for, and the rows already turned into CI tests are the first payment on it.

What I do not claim

  • Not that a unit test is useless. It is cheaper and faster; it answers a different question.
  • Not that every row walks all twelve stops. This is one row of one domain; other rows are shorter.
  • Not that the emulator is the outside world. It answers like it; the real one was not part of this week.
  • Not that this ran anywhere but local stands.

What of this is method

  • A row is a behaviour of the chain. It starts where work starts and ends where the answer is expected.
  • A verdict carries a trace. The stops are the spans; a failure names its last stop.
  • The stand finds, CI remembers. A row that passes on the emulator becomes a test named after it.

Eighteen domains in a week - Part 2. Next: Eighteen Enclosures, Before And After.

If your integration tests stop at the service boundary, who checks that the answer comes back to the one who asked? If you already trace every acceptance run, does the verdict carry the trace id, or does someone search for it? If you keep emulators of the outside world, how close are they to the real thing? How many stamps does your parcel collect?

Top comments (0)