A dashboard that says "guard: up" is usually reporting one thing: the process is still running. That is a liveness check. It answers "did it crash", not "would it stop a bad order". Those are different questions and they fail in different ways.
I keep seeing the same three levels in trading automation, and they get collapsed into one green light.
Level 1: the process exists. A heartbeat row updated by the same loop. Cheap, useful, and the least informative. A guard that accepts every order also keeps its process alive.
Level 2: the loop is executing. A counter that moves, a file that gets rewritten, a line in the log. This proves the code path runs. It still does not prove the path decides correctly. A guard reading a frozen balance looks identical to a working one at this level.
Level 3: the effect probe. Offer it one order it should refuse, and see what happens. This is the only check that measures behavior. It costs a few lines, and it catches the failures the first two cannot: the snapshot that stopped refreshing, the restored state that came back as a default, the limit compared against the wrong trading day.
Most dashboards stop at level 1 because it is the easiest to instrument. The number looks like health. It is not health, it is presence.
There is a second reason the gap survives: the check is often written by the same code path it is checking. If the read returns a plausible zero, the check agrees with it. Nothing throws, so nothing turns red.
Two numbers worth putting next to the green light: time-to-detect and time-to-effect. A guard that notices a breach in 200ms but acts 15 minutes later protects nobody, and it will look healthy the whole way. Measuring only detection hides the inaction.
If you want a cheap version of the probe: keep one test order that is invalid by construction, submit it on a schedule, and assert that the rejection arrives with the expected reason. When the assertion stops firing, the guard is decorative.
One question. Take your guard's green light. What did it last actually prove, and when?
Top comments (4)
Good catch on putting the two clocks side by side. One detail from our side: measure time-to-effect where the action actually lands, not where the decision is made. Otherwise a queue or a retry loop hides inside the gap and the number looks fine.
The invalid-order probe has its own trap too. If it runs on the same scheduler as the guard, a stalled scheduler silences both and nothing turns red. Running it from outside the process is what makes it fail loudly instead of agreeing with itself.
Yes — a plausible empty is the same failure as a plausible zero, and it only shows up when the thing that should come back is missing. The pair you describe is the part I would keep: one input that must return, one that must not, both through the same path as real traffic. The second half catches a filter that got too wide, the first catches one that got too narrow.
We also make "the read failed" a state the guard can see, so an error can never be read as empty. That is the cheapest version of your rule, and it survives refactors better than an assertion on the count.
Separating process alive, loop executing, and effect proven is a clearer way to read a green dashboard than trusting one light. The invalid test order idea stands out because it checks behavior instead of presence, which is the part most monitors skip. Adding time to effect next to time to detect would also expose guards that notice a problem but act too late.
Same trap with lookups. A guard or a search that returns zero looks identical to one that works, so a plausible empty passes every check. What helped was a probe with one input that must come back and one that must not, run through the same path as the real traffic