A team I compared notes with recently runs continuous evals on production traffic: an LLM judge scores a sample of traces, the scores land on a dashboard, and a monitor pages when the hourly pass rate drops below 88 percent. Their baseline is around 92. The monitor paged on four separate days last week. Four investigations, four shrugs, four "it recovered on its own."
Nobody chose a false-alarm rate for that monitor. But it has one, and it is not small. With roughly 150 judge scores an hour, a fresh window each check, and a true pass rate of 0.92, the chance that at least one hourly check dips below 0.88 at some point in a day is 53 percent. That makes the expected number of alert days in a week 3.7, and the single most likely weekly count exactly four. Their monitor was not detecting regressions. It was sampling noise on a schedule, and the on-call rotation was the readout.
An alert threshold on an eval score is a hypothesis test. Every tool that ships eval monitoring runs that test for you. What none of the configuration surfaces I read this month asks you for is the one number the test depends on: how many scores are in the window.
The test you are actually running
The setup is ordinary binomial arithmetic, which is what makes it checkable. Each check looks at a fresh window of n judge verdicts. The true pass rate is p. The monitor fires when the observed rate falls below a threshold t. Then:
import math
from scipy.stats import binom
def alpha_per_check(n, p0=0.92, t=0.88):
"""P(false alarm): windowed rate < t although nothing changed."""
k = math.ceil(t * n) - 1 # largest count that fires
return binom.cdf(k, n, p0)
def p_fire_today(n, checks=24, **kw):
a = alpha_per_check(n, **kw)
return 1 - (1 - a) ** checks # tumbling windows, independent checks
def power_per_check(n, p1=0.85, t=0.88):
k = math.ceil(t * n) - 1
return binom.cdf(k, n, p1) # P(one check catches a real drop)
Run that for a 0.88 threshold against a 0.92 baseline, across realistic window sizes, and add the probability that a single check catches a genuine regression to 0.85:
| Scores per window | Fires per check on noise | Fires some time today | Expected alert days per week | Catches a real drop to 0.85, per check |
|---|---|---|---|---|
| 25 | 13.5% | 96.9% | 6.8 | 52.9% |
| 50 | 10.2% | 92.4% | 6.5 | 63.9% |
| 100 | 5.6% | 74.8% | 5.2 | 75.3% |
| 150 | 3.1% | 53.3% | 3.7 | 81.9% |
| 200 | 1.8% | 35.0% | 2.5 | 86.3% |
| 400 | 0.2% | 4.9% | 0.3 | 94.9% |
Read the 25-per-window row twice. That monitor pages nearly every day on pure noise, and a single check still misses a real seven-point regression about half the time. It manages to be trigger-happy and insensitive simultaneously, which is what happens when the threshold is chosen as a round number four points under baseline instead of as a quantile of anything.
The window size is not an abstract parameter. It is your traffic times your sampling rate. Sample 10 percent of 500 traces an hour and n is 50. The sampling knob every tool gives you for cost control is also, silently, the monitor's sensitivity knob: cutting sampling from 100 percent to 10 percent widens the score's confidence interval by a factor of about 3.2, the square root of ten. None of this is exotic. It is the standard behaviour of binomial proportions, catalogued in detail in Brown, Cai and DasGupta's "Interval Estimation for a Binomial Proportion" (Statistical Science, 2001).
So the question I brought to the current tooling was not "who has alerting." Most do. It was three narrower questions.
Question one: which traces does the score come from?
This decides n, so it decides everything above.
Langfuse runs judge evaluators at ingest time on observations that match your filters, with a sampling percentage you set; matching observations enter an evaluation queue and are scored asynchronously. Filters cover observation type, trace name, tags, user, session and metadata, so the scored population is well defined, and the docs are explicit that sampling exists to manage evaluation cost.
LangSmith does the equivalent through automation rules: a filter selects runs, a sampling rate decides what fraction of the filtered runs the evaluator sees, and the scores attach back to the traces.
Opik's online evaluation rules score live production traces with LLM-as-judge metrics and write results back as feedback scores on each trace.
Future AGI's continuous eval tasks take the same two controls; a forward-only cursor means history is never backfilled.
Phoenix evaluates production traces when you run evals over them by SDK or in the app; for continuously scheduled scoring of live traffic its own docs point you to Arize AX, the commercial platform. DeepEval sits at the same boundary from the other side: the framework is offline-first, and production scoring is the companion Confident AI platform's job.
Question two: what turns a score into a page?
Langfuse has the most developed vocabulary here of the set I read. Monitors watch numeric, categorical or boolean scores; you set an operator, an alert threshold and an optional warning threshold over a lookback window, and route through Slack, webhooks or GitHub Actions. It is also the only config I found that makes you decide what an empty window means: treat missing data as zero, hold the previous severity, record NO_DATA silently, or page after sustained NO_DATA. How many monitors you can create depends on plan tier; the features themselves do not.
LangSmith alerts on five metric types, run count, cost, errors, latency and feedback score, that last one being where online eval results live, and it routes to Slack, PagerDuty, Dynatrace or any webhook.
Future AGI's monitors take a static or percentage-change threshold, or learn one from the historical mean, with separate warning and critical levels and a check frequency in minutes.
Opik writes rule scores onto traces and tracks them on dashboards; its online-rules documentation describes score computation and display, and no notification configuration appears on that page.
Phoenix routes threshold-based triggers on production traffic to Arize AX, per its own docs. I did not audit AX's alert configuration, nor Confident AI's, so the two commercial companions stay out of the comparison below.
Question three: where does sample size enter?
I read three alerting configuration surfaces end to end this month: Langfuse's monitor settings, LangSmith's alert settings and Future AGI's monitor model. In all three, the vocabulary is drawn from the same short list: a value, a direction, a window, sometimes a check frequency. None of the three has a field for the number of scores the window is expected to contain, and none derives the threshold from one. The window is specified in time, five minutes, an hour, a day, and how many scores fall inside it is whatever traffic and sampling happen to produce.
Two of the threshold types deserve a specific caution. A percentage-change threshold compares two noisy window estimates, and the difference of two independent windowed rates carries roughly twice the variance of either one, so at small n it is noisier than the static threshold it replaces. And an auto or anomaly threshold that learns the historical mean solves a real problem, baseline drift, but the alert it fires is still a point-estimate comparison; learning where the baseline sits is not the same as knowing how far a healthy window wanders from it.
To be precise about scope: this is what the configuration surfaces expose as of early August 2026, from each vendor's public docs and, where the code is public, its source tree. Any of them could ship a sample-size-aware policy tomorrow, and the gap is easy to work around today, which is the last section.
Setting the threshold from n instead
You do not need any vendor to fix this. Invert the binomial: decide the false-alarm rate you can staff, then compute the threshold your window size implies.
from scipy.stats import binom
def threshold_from_n(n, alpha, p0=0.92):
"""Largest rate threshold whose per-check false alarm stays <= alpha."""
k = int(binom.ppf(alpha, n, p0))
if binom.cdf(k, n, p0) > alpha:
k -= 1
return k / n
| Scores per window | Page below (1% per check) | Warn below (10% per check) |
|---|---|---|
| 25 | 0.720 | 0.800 |
| 50 | 0.800 | 0.840 |
| 100 | 0.840 | 0.870 |
| 150 | 0.860 | 0.887 |
| 200 | 0.870 | 0.890 |
| 400 | 0.885 | 0.900 |
Two things fall out. First, at 25 scores a window, an honest 1-percent page threshold is 0.72, a full sixteen points below the 0.88 the team "felt" was right; the small-n rows are exactly where intuition overshoots most. Second, the pairing maps cleanly onto the warning-plus-critical structure that Langfuse and Future AGI both expose and that LangSmith approximates with two rules: put the warning at the 10 percent quantile and the page at the 1 percent quantile, and both levels inherit a false-alarm ceiling you actually chose.
If your platform only takes time-based windows, fix the count instead: check every N scores rather than every hour. A count-based window makes n a constant, which makes the table above exact instead of approximate, and it stops quiet hours from paging you simply because n collapsed overnight.
FAQ
Is 0.92 a magic baseline? No. Every number above recomputes for your baseline and your window in the four lines of code shown; the shape of the conclusion, thresholds must move with n, survives any realistic parameter choice.
My monitor uses a rolling window checked every five minutes. Does the daily 53 percent still apply? Not directly, and the direction matters: the five-minute checks include the hourly windows among them, so the daily probability is at least the 53 percent, not less. Plugging 288 checks into the independence formula overstates it badly, though; overlapping windows are correlated, the per-check false alarm is unchanged, and correlated checks cluster their false alarms into the same bad hour.
Why not just require two consecutive breaches? That is a legitimate sample-size-aware policy, and on tumbling windows it squares the per-check false-alarm rate. You pay in detection delay, one extra check period, and in power against short-lived regressions. It is the cheapest fix on this page; the quantile threshold is the principled one.
Open question
Auto thresholds that learn the baseline mean already read the score history. The same history contains everything needed to learn the window-to-window variance and set the alert line at a chosen quantile of it, which would make the monitor's false-alarm rate a configured property instead of an accident of traffic. None of the docs I read says whether any of the auto modes does this today. If someone from one of these teams can point me at a sample-size-aware or variance-aware alert policy in their product, mine is exactly the kind of dashboard it would quiet, and I will happily run the comparison again.
Sources for the tool claims, all read in the first week of August 2026: Langfuse docs, "Monitors and Alerts" and the LLM-as-a-judge evaluator pages; LangSmith docs, "Online evaluations" and "Alerts"; Opik repository README and its online evaluation rules doc (comet-ml/opik); Phoenix repository docs, llm-evals page (Arize-ai/phoenix); DeepEval repository README (confident-ai/deepeval); Future AGI repository, tracer models eval_task.py and monitor.py (future-agi/future-agi). The binomial arithmetic is standard; see Brown, Cai and DasGupta, "Interval Estimation for a Binomial Proportion," Statistical Science 16(2), 2001.
Top comments (0)