Hao

Posted on Sep 16 Originally published at benchtruth.com AI-assisted

A run history cannot show you a run it never created

#devops #monitoring #opensource #architecture

There is a failure mode in every event-driven system that its own logs are structurally incapable of reporting. Not "hard to find". Incapable.

The system accepts a unit of work, answers success, and then the work does not happen. Nothing errors. No run is marked failed. No alert fires. The run history is not lying to you, it is telling the truth about a different layer: the message arrived and the server answered. Whether the operation actually occurred is a separate question that nothing in the transport path is in a position to answer.

I spent the last few months measuring how often this happens on no-code automation platforms, and the thing I got asked repeatedly was not about the results. It was "how are you counting that". So I wrote the method down as a specification: what the metric is, what has to be excluded from it, the data contract that makes it computable, and the conditions a published number has to meet before it means anything. It is free, CC BY 4.0, and there is no code to install.

This post is the engineering half of it. The parts worth reading even if you never touch the spec.

Why the run history cannot help

Two structural reasons, and neither is a defect anyone can patch.

A log cannot record a run that was never created. If a trigger silently stops firing, if an event is deduplicated away, if a quota edge swallows it before a run exists, then there is no row. You cannot find the absence of a row by reading rows. Every "check your execution history" answer to this problem is answering a different question.

Step success is evaluated on transport status, by design. A destination that returns 200 OK with {"ok": false, "error": "..."} in the body has answered successfully at the only layer the platform can evaluate. The platform cannot know what failure looks like inside an arbitrary API's response schema without being told. So it records a green checkmark, and it is correct to.

Those two together mean the measurement has to happen outside the system being measured.

The data model

Three records and a reconciliation. This is the whole thing.

The event, written by a controller you operate, before it sends anything:

{
  "run_id":   "unique per event",
  "wf":       "which workload",
  "platform": "which system this copy went to",
  "seq":      123,
  "fired_at": "2026-09-16T04:40:15.062Z"
}

run_id must be unique per event and preserved unchanged to the destination. That is what makes per-event reconciliation possible, and per-event reconciliation is what separates this from counting totals. Totals can match while the wrong records are missing. If you take one thing from this post, take that.

The receipt, posted by every action in the workflow to a destination you control, echoing the event and adding which step produced it:

{ "run_id": "...", "wf": "...", "platform": "...", "step": "action1", "fired_at": "..." }

The destination stamps received_at on arrival. Latency is received_at - fired_at, which includes platform processing and both network legs. Say so when you report it; it is not the platform's internal execution time and comparing the two is meaningless.

The send outcome, recorded by the controller for every event: did the platform accept it, and if not, why not.

This field looks like bookkeeping and is actually load-bearing. Without it, an event with no receipt is ambiguous between "the platform refused it, loudly, and we could have retried" and "the platform accepted it and lost it". Those are different findings. A measurement that cannot separate them is not a measurement.

Record the reason, not just the fact. I learned this the expensive way: a soak run had four sends fail, my harness recorded send_ok = 0 and threw the error away, and connection reset, timeout and non-2xx are now permanently indistinguishable for those four events. The fix took ten minutes. The data is unrecoverable.

Reconciliation: seven outcomes, not two

For each fired event, count receipts carrying its run_id and compare with what the workload expected:

Observed	Classification
receipts == expected	success
0 receipts, event was accepted	missed
0 < receipts < expected	partial
receipts > expected	duplicate
0 receipts, event was refused at send	rejected at send
0 receipts, workload expected 0	filtered (correct)
receipts > 0, workload expected 0	filter leak

The rate uses two of these. Publish all seven anyway, because the other five are how a reader checks that the two mean what you say they mean. A filtered count that is suspiciously high, or a filter leak that is nonzero, tells you the workload is not doing what you think it is, and your rate is measuring something else.

The exclusion that makes the number honest

        missed + partial
SFR = ──────────────────────────────────────
      runs accepted and expected to produce output

Events refused at send are excluded from the numerator and from the denominator, and reported separately.

This is counter-intuitive the first time. A refused send is a failure; why not count it?

Because your sender already knows. A non-2xx, a timeout, a dropped connection: the caller sees it at the moment of sending, can log it, alert on it, retry it. No run was created. Nothing is hidden. That is the exact opposite of the property being measured.

Mixing them gives you a number that cannot distinguish "the platform told you and you can retry" from "the platform didn't tell you and the data is gone".

A worked example from my own ledger. A five-hour run of 2,880 events: 4 refused at send, 0 accepted-then-lost. The conformant report is 0 silent failures in 2,876, with 4 rejections reported alongside. Writing "4 failures in 2,880" is wrong in both directions at once: it inflates the rate with events the sender saw immediately, and it hides that nothing accepted was lost.

It is also the rule that stops the metric being gamed. Without it, any inconvenient failure can be argued into a different bucket.

The statistics, briefly, because this is where it usually goes wrong

Silent failure rates live near zero, which is exactly where the tools people reach for fail quietly.

Use the Wilson score interval, not the normal approximation. The normal interval is p ± z·sqrt(p(1-p)/n), and at k = 0 that collapses to zero width. It will report 0% ± 0% from 40 runs and present it as conservative. It is not conservative, it is wrong, and zero observed failures is the case this metric spends most of its life in.

Wilson, for k failures in n runs at z = 1.96:

          p + z²/2n                    z·sqrt( p(1-p)/n + z²/4n² )
centre = ───────────      half-width = ───────────────────────────
          1 + z²/n                              1 + z²/n

p = k/n, interval = [centre - half, centre + half] clamped to [0, 1]

Zero failures in 40 runs gives an upper bound near 7%. Zero in 4,000 gives well under 0.1%. Both are "0%". They are not the same claim, and publishing both as "0%" misleads by omission.

Two consequences that surprise people:

A wider interval is not a worse system. If two things both show zero and one bound is wider, the difference is how much evidence exists, not how often each failed. If your samples are unequal, say so explicitly, because a reader compressing your table will otherwise supply the comparison themselves.
Zero is never proven, only bounded. More runs tighten the bound. Nothing ever establishes that the true rate is zero. A report describing its own zero as "confirmed" has said something false.

Reporting: the qualifiers travel with the number

Every published rate carries its denominator, its interval, its as-of date and its workload in the same sentence as the number. Not in a footnote, not on a linked methodology page.

This sounds pedantic until you follow a figure through extraction: table into summary, summary into quotation, quotation into someone's slide. Each step drops whatever was not adjacent to the number. The only qualifier that survives is the one inside the sentence.

One more that nobody does: say whether the figure is alive. Either it is refreshed, and how often, or it is a one-off snapshot that will not be updated. A one-off measurement is completely legitimate. A one-off in the present tense two years later is not, and the difference is a single sentence the publisher could have written.

Conformance, and what this deliberately is not

Three self-declared levels. L1 needs no harness, no paid plans and no experiment: compute the rate from your own operational data and report it with the rules above. L2 adds a controlled measurement with both endpoints under your control. L3 adds three edge probes: destination outage, success-wrapped failure, sustained load.

L1's bar is reporting discipline rather than effort, which is deliberate. It is a low bar in work and a surprisingly high one in practice: most published failure rates fail it on the reporting rules alone, because they arrive without a denominator.

What it is not:

Not a tool. Nothing to install. There is a reference implementation planned and unpublished, and I am not going to describe it as released while it isn't.
Not audited. No certifying body, no registry, nobody reviews claims including me. It only works because every requirement is something a reader can look for in your report and fail to find.
Not finished. One section is explicitly reserved, with the reason written in place: the requirement that belongs there is drawn from a single observation that did not recur, and a rule from n=1 is a rule from first principles with an anecdote attached.

DEV Community