A browser monitor I was migrating failed about two runs in every three. The rest passed cleanly. The application behind it was healthy. Nobody could see a pattern.
The run history for that stretch was 67 runs. 25 passed. 42 failed. That is a 37.3% pass rate on a monitor watching an application that was working.
This post is about how that failure was found and fixed. The fix itself was small. The idea behind it is the part worth keeping:
A check that passes only when it wins a race is not a passing check.
Why intermittent failures are the hardest kind
A monitor that always fails is easy. You look at it, find the broken thing, and fix it.
A monitor that fails sometimes is harder, for three reasons.
It teaches people to ignore it. After a week of flapping, a red run stops meaning anything. That is the same noise problem I wrote about in Cutting Alert Noise Without Going Blind. A flapping monitor is noise with a green light mixed in.
It passes on a rerun. So the easy "fix" is to add a retry and move on. The dashboard goes green. The race is still there.
There is no single broken thing to look at. The passing runs and the failing runs use the same script against the same application. Comparing them line by line shows nothing.
So the useful question is not "what is broken?" It is "what is different between a run that passes and a run that fails?"
Reading the pattern
Two things stood out in the step results.
First, the failures happened at the same step: sign in. The monitor signs in through a single sign on page before it reaches the application. That is where the failing runs stopped.
Second, a run could fail and the next one could pass with nothing changed in between.
Same step. Random outcome. Passes on a rerun. When I see that combination, I treat it as a timing problem until something proves otherwise. The script and the page are not waiting for each other properly.
In The Health Check That Was Red and Green at the Same Time, I wrote about reading failure duration first. A very fast failure points at the environment: network, DNS, routing. A failure that runs its course and then fails a check points at the script side: a selector, an assertion, or sign in. This one belonged on the script side. The application was fine. The script's timing was not.
The race
After you enter credentials, the identity provider does some work before it hands you back to the application. Along the way it can show an optional prompt asking if you want to stay signed in. How long all of this takes is not fixed. It changes from run to run.
The script did not account for that. It used a fixed wait after sign in, then moved on as if the next page would be there.
When the sign in flow settled inside that wait, the run passed. When it settled later, the script acted on a page that was not ready yet, and the step failed.
This changes how you read the 25 green runs. They were not proof that the script worked. They were runs where the timing happened to line up. The greens were luck, not health. A monitor like that is not telling you anything about the application. It is telling you about a race it sometimes wins.
Why the obvious fixes are wrong
There are two quick ways to make this monitor go green. Both are traps.
Make the wait longer. This makes every run slower, even the ones that were ready early. And it is still a guess. The day the identity provider has a slow afternoon, the race comes back.
Add retries. This hides the flap from the dashboard. It also hides a real sign in outage. If sign in is genuinely broken for a while, a retrying monitor turns a clear red into a slow, confusing mix of passes and failures. You find out later than you should.
Both fixes change what the dashboard shows. Neither changes what the script actually knows. That is the same rule I keep coming back to: never loosen a check just to make a dashboard green.
The fix: poll and proceed
The real fix is to stop guessing a duration and wait for a condition instead.
Instead of "wait N seconds, then continue," the script asks a question again and again, for a limited time:
- Is the landing page there? Then sign in is done. Move on.
- Is the stay signed in prompt there? Then answer it, and keep checking.
- Neither yet? Wait briefly and check again.
- Past the time limit? Fail, and say exactly why.
Here is the shape of it in simplified pseudocode. This is not the monitoring tool's syntax, and it is not the real script. It is just the logic.
Before: wait a fixed time and hope.
submit_credentials()
sleep(FIXED_WAIT) # a guess about the identity provider
continue_to_application() # assumes the page is ready
After: poll for a condition, with a limit.
submit_credentials()
deadline = now() + TIME_LIMIT
loop:
if landing_page_visible():
break # signed in, move on
if stay_signed_in_prompt_visible():
answer_prompt() # optional: handle it only if it shows
if now() > deadline:
fail("sign in did not complete within TIME_LIMIT")
sleep(SHORT_POLL)
continue_to_application()
Three things make this version better.
- It waits for the page, not the clock. Fast runs move on as soon as they are ready. Slow runs get the time they need.
- It handles the optional prompt either way. The prompt can show or not show. The script no longer cares which.
- It is still bounded. If sign in never completes, the step fails with a clear reason. A real sign in outage still shows up as red. That is the part retries break.
What the numbers support, and what they do not
This is the section I care about most, because it is where it is easiest to overclaim.
The first two runs after the fix passed. Two runs prove nothing on their own, so I did not call it done there.
The longer window in nonproduction after the fix was 256 passes out of 333 runs, or 76.9%. On its own, that looks like the fix only half worked. It did not mean that. This application was scoped for production monitoring, and the failures that remained in that window were environmental, not the sign in race. The monitor was later paused in nonproduction, because monitoring for that application lives in production.
In production, the monitor is at 100% on the current sample, with a P90 of about 58 seconds.
So the honest claim is: the flap is gone, and the application runs healthy in production. I do not say "100% after the patch." The full window does not support that sentence, and a number you cannot defend is worse than a smaller number you can.
A second sign in failure, with a different shape
In the same migration, another production application also failed at authentication. It looked similar at first. It was not.
This one did not flap. It failed from our private runner, while the same check in the legacy system showed the application healthy at 100%. A consistent failure that is healthy at the source does not point at timing. It points at the environment between the runner and the application.
I did not have access to the cluster that hosted that private location. So I designed a diagnostic and asked the engineer who owned it to run it: open a shell inside the runner pod and call the endpoint directly from there. That removes the monitoring script from the picture completely.
The result was clear. The TLS connection succeeded. The response was HTTP 401 with a message that a bearer token was required. It was not a firewall block and not a script defect. The API gateway was enforcing OAuth on the runner's path. The blocker went to the gateway team with that evidence attached, and the test was left as it was.
Both failures happened at sign in. What decided the next move was not the step they failed on. It was the shape of the failure:
- Intermittent at the same step means timing. Fix the script.
- Consistent, but healthy at the source means environment. Prove it, escalate with evidence, and do not touch the test.
One small note on diagnosing through someone else's hands. When you cannot run a check yourself, make it as small as possible and decide in advance what each possible result would tell you. Then one run by another person is enough to settle the question.
Takeaways
- Intermittent means timing until proven otherwise. Same step, random outcome, passes on a rerun: that is the signature.
- A check that passes only when it wins a race is not a passing check. Treat its green runs as luck.
- Wait for conditions, not durations. And put a limit on every wait, so a real outage still fails loudly.
- Do not fix a flap with longer sleeps or retries. Both hide the race instead of removing it.
- Let the shape of a failure pick the next step. Intermittent points at the script. Consistent and healthy at the source points at the environment.
- Claim what the numbers support. "The flap is gone and it is healthy in production" is a smaller sentence than "100% after the patch." It is also the one that survives a follow up question.
This is the fourth post in a series on the practice of running monitoring. The earlier ones cover alert noise, a health check that was red and green at once, and shipping monitoring changes without causing an incident.
Top comments (1)
Two claims in this post are load bearing, and I put numbers on both from your own figures.
Your two-run caveat, quantified. Under the pre-fix rate (25 of 67, so 0.373 per run) two consecutive greens happen with probability 0.139, which is about one in seven. Three greens: 0.052. Four: 0.019. So four consecutive passes is where "the race is still there" dies at 95%, and two is no evidence at all. That turns your instinct into a stopping rule: after fixing a check that passed 37% of the time, run it four times before saying anything about it.
The 76.9% window cannot tell the two readings apart, and the reason is a coincidence. Retry a 0.373 check three times and a dashboard reports 1 minus 0.627^3, which is 75.4%. Your post-fix window measured 76.9%. The gap is 1.5 points against the window own binomial interval of plus or minus 4.5 points at n=333. Invert it too: to print 76.9% from a check retried three times, the underlying per-attempt rate has to be 0.386, which is your pre-fix rate to within a point.
So the aggregate does not discriminate between the reading you argue for and the reading you argue against. Removed the race and masked the same race with three attempts report the same number, and the second one sits inside the first one noise. Your thesis that retries hide the race has a numeric form: the two are indistinguishable at the resolution of this metric.
What separates them is the margin, not the verdicts. In the poll-and-proceed version, record for each run the time between the last poll that answered no and the poll that answered yes. If the fix removed the race, that margin becomes a property of the identity provider and its lower bound is the poll interval rather than zero, which is itself a check on whether the loop is polling or sleeping. If an attempt retry is masking the race, the margin distribution is unchanged from the pre-fix one. A before-and-after histogram of that single number decides what 333 verdicts cannot.
The residual 77 failures. You read them as environmental, and you may be right, but the sentence about them being environmental rather than the sign in race is a claim about the shape of 77 runs, and the shape is the discriminator you argue for in the second half of the post: intermittent at the same step means timing, consistent and healthy at the source means environment. The number that decides it is the per-step histogram of those 77, and it is not in the post. As written, a reader cannot separate a fixed race plus environment from a partly present race plus environment.
One more on the production claim. The failure mode of a bounded poll loop is an exceedance of TIME_LIMIT, so the quantity that predicts the next failure is the tail of the latency distribution against that limit, not the P90. The P90 tells you what a typical run costs. With the current sample at 100% green, the only thing that can move that number is the tail, so the sample says the limit was not exceeded in the sample and nothing about how far the bulk of the distribution sits from the ceiling.
None of this touches the fix, which reads right to me: wait for a condition, keep a bound, and fail with a reason naming the step.