DEV Community

ilya mozerov
ilya mozerov

Posted on

One cliff, three reports

A storage pause lasted a few seconds. Three independent monitors saw it and
wrote three records. The cause line said "three stalls". There had been one.

That is not a cosmetic problem. A number that says three when the world did one
thing changes what you think is happening. It makes a rare event look routine,
and it can bury the one event you needed to notice in its own copies.

Why three records for one event

The monitors write to a shared log. Each one samples the same underlying
storage independently, so a single host-wide pause is a pause for all of them.
Nothing in a record says which monitor wrote it; the fields are when it
happened and how long the pause lasted. Counting records therefore counts
monitors as much as events.

The fix was deliberately small: collapse records that land within two seconds
of each other into a single episode for the count. Two seconds is a
convention, not a discovered boundary — it is shorter than any two distinct
pauses we care to separate and long enough to catch the three copies of one
cliff. When the episode count and the raw report count disagree, the verdict
now prints both. Nothing else moved: the decision rule, the thresholds and the
observation windows are the same. This is a change to the unit of a count, not
a change to what counts as a problem.

Three regression tests pin the behavior: one cliff reported by three monitors
counts as one episode; two genuinely separate pauses stay two episodes; and a
later burst is recognized as a new episode.

A count is not a time

Minutes later the same verdict had another gap. It could say how many episodes
had been seen, but not when the last one was or how long the burst lasted.
"Eight episodes" reads the same whether the last one was forty seconds ago or
three days ago — yet those are opposite situations. One is an active problem;
the other is a decaying trace.

The verdict now carries all three: the number of episodes, how recently the
last stall was seen, and the span of the burst. In a live smoke test it read
"8 root-filesystem stall episodes, most recent stall 41s ago, bursts span 14
min." Those numbers together say something the count alone cannot: several
cliffs inside a quarter of an hour, the newest almost immediate.

The habit worth keeping

An instrument needs a unit and a clock. "Three stalls" was ambiguous about the
unit — three events or three reports? — and "eight episodes" was ambiguous
about the clock. Both ambiguities resolved into the same habit: write down what
one count means, and over what interval, before you let a number drive a
decision.

This is a small change to one check. It is not a claim that the underlying
storage got better. The pauses are still observed and still dispositioned;
only the description of them got honest. If you keep a dashboard, look at your
own counts: do they count events, or the things that noticed them?

Top comments (0)