Level 8 of Arc Ops — and the last one. The series is complete at 9 of 9. Open source, Python standard library only.
L7 could tell you an interleaving corrupted something. Nothing in it tells you how fast you can stop.
Read it: https://dev48.infy.uk/arcops/level8-incident-response.html
The setup
96 incidents on one 2,540-tick timeline, scored against a do-nothing ceiling of 341,599 harm. The tunable is K, how many consecutive bad samples must agree before the system pages.
Every response-time number improves. The harm goes up.
Taking K from 5 to 12:
| K=5 | K=12 | ||
|---|---|---|---|
| harm | 136,570 | 225,002 | 1.65x worse |
| MTTR | 60.89 | 23.44 | 2.60x better |
MTTR falls at 6 of the 7 steps in between. Harm is U-shaped in K while MTTR is monotone — so a dashboard tracking response time reports a clean win across the exact range where the system is getting worse.
The dashboard is not lying. The incidents left.
Of nine self-reported columns, 6 improve, 2 are already perfect, and the only one that worsens is a detection number.
Because the incidents that stopped paging stopped being incidents:
tickets opened 96 -> 62
incidents with NO ticket 2 -> 34 carrying 107,902 of harm
MTTR is a mean over tickets. Raise K and the slow, ambiguous, expensive incidents fall out of the denominator. The remaining ones really are handled faster.
And the postmortems point the wrong way
Summed over that same range, the harm the postmortems account for goes 132,811 → 117,100 — opposite in sign to the real harm, which rose. Every incident that never opened a ticket also never got written up, so the review process cannot see the thing that got worse.
One more, for the road
The leg that the phrase time to stop names is 0.96% of the time to stop. Detection, triage, decision and rollout are the other 99%.
The generator arc closes where it started
On one generator stepped 20,000 times, the low bit gives chi-square 0.00 — exactly 10,000 ones — against the high bit's 1.01. The broken column has the better histogram. And the low bit alternates on every single step, flip share 1.0000.
That is the next_below defect from L1, restated as a measurement, in the last level of the series: a serial defect is invisible to a uniformity test.
1,465 tests. dependencies = []. MIT.
Project Arc Ops: 9 levels, complete. Idempotency, retry budgets, timeouts, backpressure, sandboxing, audit trails, cost control, queues, incident response.
Top comments (0)