DEV Community

Cover image for Whose Bug Was It
Anton Brilliantov
Anton Brilliantov

Posted on

Whose Bug Was It

#debugging #softwareengineering #testing

Twenty-three defects in the last twenty-two runs: eleven in the collector, four in config, three in other services, and five of our own.

👋 Hi, I'm Anton - a software engineer. These days I work on a platform: contracts, generated code, data paths and the tooling around them. Running notes live on my GitHub: github.com/brilliant-almazov.

The count covers only the last twenty-two acceptance runs of the week, where every fix is itemised; one collector fix is still in review and the three defects in other services are open and planned, and both are marked that way below.

This is part 4 of the series on one week of integration acceptance across a whole chain of services, after months of doing it the slow way. Maybe you already do it better, maybe you'd do it another way - this is how it went, and where it stops.

In real life

The whole system is a zoo. The services are the residents, the bus is the paths, and acceptance is the vet - Pickle, a guinea pig with a card for each of eighteen enclosures. Nora is the keeper.

Pickle's week of rounds found plenty. Biscuit, the courier dog, had a sore paw, and when Pickle confirmed it he said "Told you so!" with great satisfaction. Miso the cat was wearing a name sign that said BISCUIT, and observed, without opening her eyes, "That sign is not mine." Nora went to fix the sign. At one enclosure the cure was next door - and Pip, the guide, pointed out that next door was closed.

And then, five times in one week, the problem was Pickle. Mixed-up cards, an old page in the notebook, a medicine bottle already empty when she tried it again. Pickle admitted it: "I mixed up the cards." Ember the dragon, who misses nothing, added: "Five times. I counted."

A four-panel comic: Pickle says paw please, hm, sore, and Biscuit, angry, says told you so; Miso, bored, next to a sign reading Biscuit, says that sign is not mine, and Nora says fixing the sign; Pickle, bored, next to a fence, says the cure is next door, and Pip, worried, says next door is closed; Pickle, worried, says I mixed up the cards, and Ember says five times, I counted

The sore paw is a defect in the collector's code. The wrong name sign is configuration on the stand. The closed door next door is a defect in another service, which the acceptance agent can find and cannot fix. And the vet's own mistakes are the acceptance side's false reds. The honest answer to "whose bug was it" has four parts, and one of them points back at the tester.

The count

The acceptance agent counted defects only over the last twenty-two runs of the week, because those are the runs where every fix is written down item by item. Earlier runs found things too; they are not in this number.

Whose How many Status
the collector's code 11 10 fixed, 1 fix in review
configuration and the stand 4 fixed
other services 3 open, planned there
our own - the acceptance side 5 each listed with its fix
total 23

A waffle of 23 cells in two rows: 10 teal cells with checks, the collector, fixed; 1 teal cell without a check, the collector, a fix in review; 4 indigo cells with checks, config and the stand, fixed; 3 amber cells, other services, open; 5 plum cells with crosses, our own, the acceptance side; Pickle, worried, says five of them were mine; pills read measured and 3 open, planned in other services

Eleven in the collector

The collector's eleven are the ordinary findings of driving a chain end to end: a status probe that could time out, a watchdog whose threshold did not arrive where it was needed and whose probe read the order's scenario, a queue for rebuilding a buffer, the body of a task post, a request-log row that was not closed when the answer never came, a delivery that had to happen exactly once, a priority that had to be clamped, configuration defaults, per-call balance answers in the emulator, and a number type in one provider's override.

Two of them are worth telling. Priorities outside 0..100 were refused with an error. The owner's norm is different: a number on a scale is clamped to its bound, no error. After the fix, six rows went from red to green. And configuration defaults moved to one form for every leaf, so that a binary starts with sane values when a key is not set:

# every configuration leaf, the form (paraphrased)
WATCHDOG_STATUS_TIMEOUT=${WATCHDOG_STATUS_TIMEOUT:-10s}
<KEY>=${<KEY>:-<default>}
Enter fullscreen mode Exit fullscreen mode

The one still in review is the most instructive. The emulator lets a test override its answer per call, and the balance override worked for one provider - five calls of five - and not for the other, zero of five, because a JSON number fell back to the fixture. The lesson the acceptance agent wrote down: measure each provider separately; a pass on one does not prove the other. The fix is written and waits for its rerun.

# an emulator scenario override, answer by call number (form, values elided)
{"calls": {"N": {...}}}
one provider     5 of 5 calls took the override
the other        0 of 5 calls took the override   | fix in review, rerun pending
Enter fullscreen mode Exit fullscreen mode

Bars: a binary with its neighbour's keys, before, 0 of 13; after the cleanup, 13 of 13; balance override, one provider, 5 of 5; the other provider, fix in review, 0 of 5; Biscuit, worried, says one provider is not proof for the other; pills read built and the second provider in review

Four in configuration and the stand

These were not code. The account list for the balance poll needed a fix. An alert called "emulation in production" never fired, because its job label did not match - fixed on the stand. A content service was raised switched off; the switch was named and turned on. And a binary carried another binary's configuration keys: thirteen binaries were checked, none was clean, and after the cleanup all thirteen were. The rule that came out of it is short: a binary knows only its own keys.

Three in other services - open

Three defects sit in services the acceptance agent does not own, and the rows they break stay red until those services change:

  • results are delivered to a queue that has no consumer, so they stay unread;
  • a status call with tenant 0 is refused with "tenant_id must be > 0", so the empty-result mark is lost;
  • two norms for the collector contradict each other, and the choice between them is the owner's.

All three are planned in their services. They are the reason the plan stands at 749 of 753 and not at a hundred, and the previous part of this series explains why they stay red rather than leave the scope.

An animation: a red row, drawn as a letter, visits four tiles - the collector, the stand, next door, acceptance - and each tile frowns in turn; the pills change from the collector 11 times, to config and the stand 4 times, to another service 3 times, open, to our own probe 5 times

Five of our own

This is the part I would most like other teams to count. Five defects were the acceptance side's own:

A dark card with our own five as a form: a retry on a spent scenario, a false red, a fresh scenario per try; verdicts on a stale build, a false minus 4 rows, a per-build ledger; a norm from an old document; 3 wrong citations, found when rows became CI tests; 49 wrong ENV key names in plan rows; the highlighted line reads counted in the last 22 runs, where every fix is itemised; a pill reads listed with their fixes

The stale build was the costliest. Verdicts were measured on a build that did not contain the fix under test, and one run showed the plan falling from 732 rows to 728 - four rows lost that had never been looked at on the right code. The fix was a per-build ledger on the acceptance side and a rule: a red mark counts only on the build that carries the fix.

The spent scenario was the most embarrassing. A retry reused an emulator scenario whose call counter was already used up, so the second attempt failed for a reason that had nothing to do with the collector. Now every attempt gets a fresh copy of the scenario, and a test keeps it that way.

Before and after: a retry on a spent scenario, red, the call counter was used up, false, the collector was fine; a fresh scenario per try, green, the same row, honestly, 1 test that keeps it so; pills read built and tested

The three wrong citations were found from the other direction: when the collector turned plan rows into CI tests named after them, three rows turned out to quote the wrong source. They were fixed in the plan.

Eleven fixed, four added

In the same window the chain did not only get repaired; it also got new behaviour, each piece with its own plan rows:

  • the acceptance control calls moved onto a bus;
  • four new calls - a balance poll, page outcomes, balance changes and a utility poll - arrived with new plan rows;
  • alert rules are now generated into config;
  • the scenario id and the emulation flag are written to the request log.

Columns: fixed 11, added 4; Nora, proud, says eleven cured, four new tricks; pills read built and 1 of 11 fixes in review

The sign on the door

Nora, proud, waving next to a big sign reading first ask whose card it is; Nora says check your own cards before the paw; on the right the mechanism: a fresh scenario per try, a per-build ledger, out-of-scope-needs-owner-quote; pills read built and 3 open defects planned

What went wrong

Five of twenty-three is about one in five. The false reds among them - the spent scenario and the stale build - made the collector look worse than it was, and the wrong citations and key names made the plan itself less trustworthy until they were found. The most expensive was the simplest: a verdict without the name of the build it was measured on.

What I do not claim

  • Not that twenty-three is all the defects of the week. It is the count over the last twenty-two runs, where every fix is itemised.
  • Not that the in-review fix works. It waits for its rerun.
  • Not when the three open defects close. They are planned in services the acceptance agent does not own.
  • Not that the acceptance side is now free of its own mistakes. It has fewer ways to make the same five.

What of this is method

  • Count your own. A tester's false red is a defect, and it goes into the same table.
  • One provider is not proof for another. Measure each one separately.
  • Open is a status, not a hiding place. A defect in someone else's service stays red and named.

Eighteen domains in a week - Part 4. This closes the series.

If your test reports count defects by service, is there a row for the tests themselves? If you test against more than one provider, do you measure each separately? If a fix belongs to another team, does the failing row stay red, or does it leave the plan? Whose bug was your last one?

Top comments (0)