A monitoring result can be mathematically correct and operationally false.
That happens when the calculation is given an incomplete sample but treats it as the whole measurement window. A particularly easy way to create this bug is to ask a data source for the “latest N” observations while the metric claims to describe a trailing period such as the last hour or day.
At first, those two boundaries may appear equivalent. If the window usually contains fewer than N observations, the query returns everything. Then activity increases, the result reaches the cap, and older in-window observations silently fall away. Nothing in the gap calculation has changed, yet its evidence has.
The lesson is compact: a time-window metric needs a time-window sample.
How a row limit invents an outage
Consider a fictional monitor that expects one heartbeat every five minutes and reports the longest gap in the last 60 minutes. A complete healthy window contains observations at 00, 05, 10, and so on.
The gap algorithm sensibly includes two synthetic boundaries: the start of the window and “now.” Those edges let it detect silence before the first observed event and after the last one.
Now suppose the source returns only the latest ten records. Once the hour contains more than ten observations, the oldest returned heartbeat might be 15 minutes after the window began. The algorithm sees this:
window start ---------------- first returned event -- ... -- now
15 minutes
It reports a 15-minute opening gap. In the complete history, however, events existed at five and ten minutes. The apparent outage is not a calculation error. It is the shape of omitted data.
The dangerous ambiguity is that “no observation happened” and “no observation was fetched” look identical after the query boundary is forgotten.
Query the boundary the metric promises
If a result describes a trailing time window, make the source query start at that window boundary:
windowStart = now - measurementWindow
rows = listObservations(createdAt >= windowStart, limit = safetyCap)
if rows.count >= safetyCap:
fail("sample completeness is unknown")
return longestGap(windowStart, rows, now)
The date filter defines the sample. The row limit remains useful, but it has a different job: protecting the caller and upstream service from an unexpectedly large result.
That distinction is more than naming. A fixed count answers “which records are newest?” A time boundary answers “which records belong to this metric?” Only the second question matches the claim made by a trailing-window result.
A full page means unknown, not complete
When the returned count reaches the safety cap, you cannot know from that page alone whether it contains every matching observation. It may be exactly complete. It may also be the newest slice of a much larger set.
The conservative response is to refuse the calculation. Return an explicit unknown state, follow the API's pagination contract, or fail the monitoring job so its existing failure rail becomes visible. What you should not do is feed the capped page into the metric as though absence beyond the page were evidence.
This can feel uncomfortable because the monitor now makes noise. But it is honest noise: “I could not establish completeness.” A fabricated outage and a fabricated healthy verdict are both more expensive because they invite confident action on missing history.
Test the failure mechanism, not only the formula
Unit tests for the gap function are necessary, but they are not enough. The defect lives at the seam between retrieval and calculation.
A useful regression suite builds one healthy full-window data set that exceeds the old limit, then proves both views:
- the complete set produces the expected healthy interval;
- the newest capped subset of the same set produces a false leading-edge gap;
- the source query carries the exact window-start boundary;
- a result equal to the cap is rejected as potentially incomplete;
- the same result one row below the cap is accepted;
- leading and trailing silence still count when the sample really is complete.
That paired test is valuable because it holds the underlying activity constant. The only variable is whether the source omitted observations. It demonstrates why a green arithmetic suite could coexist with a false production alert.
The trade-off is deliberate
Date-bounded retrieval may return more data than a small “latest N” query. Pagination adds code. Server-side aggregation may be needed for very busy streams. An explicit completeness failure also creates an operational condition that someone must route and monitor.
Those costs are real. So are the alternatives.
If the volume can be large, options include a larger guarded cap, complete pagination, a source-side aggregate with documented semantics, or a narrower window. The invariant should remain: the system never converts an unknown sample boundary into a known health verdict.
What this pattern does not prove
A complete sample does not make every monitoring decision correct. The selected window may be wrong. The expected cadence may be unrealistic. Timestamps may arrive late or be skewed. The alert threshold may still be noisy.
Nor does reaching the cap prove a row is missing. It proves only that completeness cannot be established from the returned page. That is enough reason not to calculate.
The pattern protects one boundary: the evidence supplied to the metric corresponds to the period the metric claims to describe.
A practical review checklist
For every windowed metric, ask four questions:
- Is retrieval bounded by the same time range as the calculation?
- Can the source paginate, truncate, or cap matching records?
- How does the system represent unknown completeness?
- Will that uncertainty reach an operator, or be swallowed into a normal verdict?
“Latest N” is a useful safety mechanism. It is not a substitute for temporal scope. Keep the cap, but make saturation loud—and never let missing evidence masquerade as measured reality.
Top comments (0)