Short answer: poll a narrow, authenticated summary endpoint on a fixed schedule, but page only when an unresolved error group is both persistent and damaging enough to threaten the cohort experiment's SLO. Keep delivery separate from detection, use stable group keys, and persist the poll cursor. Email and chat webhooks are notification transports; neither should decide whether an incident exists.
The page that fires should tell the on-call which media tenant cohort is affected, which experiment is running, how many groups remain unresolved, when the oldest relevant event occurred, and which SLO window is at risk. A message that says only 12 errors detected creates work at the worst possible time. The responder still has to discover whether those errors are twelve copies of one failure, twelve unrelated failures, or background noise from tenants outside the experiment.
This is the central trade-off: grouping removes repetition, but aggressive grouping can hide distinct causes. Thresholds suppress noise, but a high threshold delays action. Treat those choices as capacity planning for human attention. The scarce resource is not webhook throughput. It is the number of pages the on-call can investigate without missing the one cohort whose experiment result is becoming untrustworthy.
Start there.
What should have fired before the page?
Work backward from the page. A customer-visible failure is late evidence. Earlier signals are an increase in newly unresolved groups for an experiment cohort, growth in affected tenants, sustained event volume inside those groups, and age without resolution. Those four signals answer different questions, so collapsing them into one raw count throws away information.
For a media experiment, compare the treatment and control cohorts before deciding severity. A failure present at similar rates in both cohorts may be a platform issue; one concentrated in treatment may invalidate the experiment. Raw event totals alone cannot establish that distinction because a single retry loop can generate many copies of the same logical error. Event grouping exists to combine similar events into issues, and grouping systems may use stack traces, exception details, or an explicit fingerprint. That makes the group key part of the measurement model, not an incidental database field.
The poller should retrieve only the fields required for that decision: group identity, cohort identity, status, first and last observation times, event count, and affected-tenant count. It should also record source freshness. If the source is stale, the correct state is unknown, not healthy.
No signal is free.
| Signal | What it protects | Likely noise source | Page only when |
|---|---|---|---|
| New unresolved groups | Detection of novel failures | Deploy-time churn | The group persists and touches the experiment |
| Affected tenants | Cohort breadth | One tenant with unusual traffic | Breadth crosses the defined impact threshold |
| Events per group | Failure intensity | Retries and loops | Intensity is sustained across polls |
| Oldest unresolved age | Recovery progress | Intentionally deferred work | Age consumes the allowed SLO window |
Instrument the decision, not merely the exception
OpenTelemetry's logs data model connects logs with traces through trace and span identifiers and defines fields for severity, body, resource, and attributes. That is useful here because the error record can carry experiment.id, cohort.id, and tenant.id as attributes while retaining trace context for investigation. Attribute names should be bounded and reviewed: a tenant identifier is useful; a full URL containing arbitrary query parameters invites cardinality trouble.
Emit the cohort assignment at the point where the request actually receives it. Inferring treatment later from a deployment timestamp is weaker because rollouts, retries, and long-running requests blur the boundary. Also emit a stable error fingerprint when the application understands the failure better than a generic stack-based grouper can. Keep the original exception details for diagnosis, but do not use message text itself as the group key when it contains IDs or other changing values.
The following Go type is deliberately small. It models the summary consumed by alert evaluation rather than a particular event payload.
package alerting
import "time"
type GroupSummary struct {
Fingerprint string `json:"fingerprint"`
ExperimentID string `json:"experiment_id"`
CohortID string `json:"cohort_id"`
Status string `json:"status"`
FirstSeen time.Time `json:"first_seen"`
LastSeen time.Time `json:"last_seen"`
Events int64 `json:"events"`
AffectedTenants int64 `json:"affected_tenants"`
}
type Snapshot struct {
GeneratedAt time.Time `json:"generated_at"`
Groups []GroupSummary `json:"groups"`
}
A stable fingerprint deserves a version. If grouping logic changes, prefix the new key with something like v2: and observe both schemes during rollout; otherwise a grouping change can look like a burst of novel failures. Do not silently rewrite history.
How should an error alerting API poll unresolved error groups?
Cron is acceptable as a scheduler when its overlap and failure behavior are explicit. The process still needs a request timeout, authentication, bounded response handling, durable state, and idempotent delivery. A schedule that launches a second poll while the first is running can duplicate notifications, so use a lease or a single-flight mechanism shared by all replicas.
Overlap is failure, too.
The evaluator below is intentionally independent of HTTP and notification code. It pages only after two consecutive qualifying observations, which filters a single transient poll without pretending that one duration is universally correct. The actual interval and thresholds must come from the experiment's error budget and response target.
package alerting
import "time"
type Policy struct {
MinTenants int64
MinEvents int64
MaxAge time.Duration
}
type AlertState struct {
Consecutive map[string]int
NotifiedAt map[string]time.Time
}
func Evaluate(now time.Time, snap Snapshot, p Policy, state *AlertState) []GroupSummary {
var firing []GroupSummary
seen := make(map[string]bool)
for _, g := range snap.Groups {
if g.Status != "unresolved" {
continue
}
key := g.ExperimentID + "/" + g.CohortID + "/" + g.Fingerprint
seen[key] = true
qualifies := g.AffectedTenants >= p.MinTenants &&
g.Events >= p.MinEvents && now.Sub(g.FirstSeen) >= p.MaxAge
if !qualifies {
state.Consecutive[key] = 0
continue
}
state.Consecutive[key]++
if state.Consecutive[key] >= 2 && state.NotifiedAt[key].Before(g.LastSeen) {
firing = append(firing, g)
state.NotifiedAt[key] = now
}
}
for key := range state.Consecutive {
if !seen[key] {
delete(state.Consecutive, key)
}
}
return firing
}
In production, AlertState belongs in transactional storage, not process memory. The poll cursor and notification record should be committed so a restart cannot turn every old group into a fresh page. Give each attempted notification an idempotency key derived from experiment, cohort, fingerprint, and the source update time. Record success only after the receiver accepts the request, and retry with a bounded policy while preserving that key.
Test the evaluator with table-driven cases for a new group, a resolved group, a threshold boundary, stale source data, two consecutive observations, and a changed LastSeen. Then test the adapter against malformed JSON, an oversized body, authentication failure, timeout, and a partial page of results. A happy-path cron example is easy; recovery semantics are the actual system.
Consider the awkward case in detail. The treatment cohort produces one fingerprint across six tenants, the control cohort produces none, and the poller sees the group once just before a deployment finishes. The first observation records state but does not page. On the next poll, the same fingerprint has a later LastSeen, still affects the treatment cohort, and now satisfies the age policy, so the evaluator creates one incident envelope. If notification delivery times out, the durable idempotency key permits a retry without creating a second logical incident. If the source summary is stale instead, the evaluator reports unknown source health and withholds a false recovery. None of those branches requires the notification adapter to understand experiments, and none permits a transport failure to erase the detection decision. The numbers here are fixture inputs for testing the state machine, not a proposed production threshold.
How should email and chat delivery differ?
Detection should produce one neutral incident envelope. Delivery adapters can render it for email or a chat webhook without changing severity. Put the stable incident key, affected cohort, threshold evidence, source freshness, and investigation link in both forms. Do not make chat the database of record: edits, retention, and channel membership are delivery concerns rather than alert state.
Email tolerates a denser summary and is useful for durable distribution. Chat is faster to scan, so its first line should contain impact and cohort, followed by the evidence that crossed policy. Both adapters need timeouts, response-status checks, redaction, and a dead-letter path that someone monitors. A successful poll followed by failed delivery is not a successful alert run.
Delivery is evidence.
Keep recovery messages paired with the original incident key. If the group resolves and immediately recurs with a later source update, that is a new state transition; it should not be suppressed forever by the first notification timestamp. Conversely, sending the same unchanged page every cron interval trains responders to ignore the channel.
Buy, build, or split the system?
The choice is less about feature count than ownership boundaries. A platform team should estimate ingestion capacity, retained event volume, query concurrency during incidents, delivery retries, upgrade labor, and on-call load before comparing architectures. Price sheets cannot represent those operational costs reliably, and a low entry price says little about behavior at the next retention or tenant boundary.
| Approach | Team owns | Main advantage | Main risk |
|---|---|---|---|
| Managed end to end | Policy and response | Lower operational surface | Lock-in around grouping, retention, and query semantics |
| Self-hosted end to end | Ingestion through delivery | Maximum control over data and policy | Capacity, upgrades, and availability become on-call work |
| Split architecture | Integration and state boundary | Replaceable delivery and analysis components | More contracts to test during failure |
I would require an exportable event schema and stable group identifiers in every option. I would also load-test the read path used by the poller, because an alert query that fails during an ingestion spike has exactly the wrong failure mode. The capacity model should include a burst multiplier, retention, index overhead, and replay traffic; the SLO should cover freshness of the alert decision, not merely availability of the ingestion endpoint.
Build the small policy layer when cohort-aware decisions are specific to the experiment and can remain narrow. Adopting an existing event pipeline can be sensible when the team does not want to own grouping, indexing, and retention. Splitting those responsibilities is reasonable only if the boundary is observable and failure-tested.
The threshold is an error-budget decision
A threshold is correct when its false positives and detection delay fit the service's response objective. Start with the user-facing SLO, derive how much bad experiment traffic can pass before the error budget is materially threatened, and choose the poll interval plus persistence rule inside that window. Then replay representative historical data if it exists, or use labeled synthetic fixtures if it does not. Uncertainty should stay visible until production evidence resolves it.
Track alert precision, pages per on-call shift, time from first qualifying event to notification, stale-source intervals, and delivery failures. These are control signals for the alerting system itself. A policy that catches every error but pages on retry noise is not reliable; neither is a quiet policy that notices damage after the experiment decision has already been contaminated.
The false-positive cost is concrete: interrupted investigation time, slower response to later pages, and pressure to mute a channel. The false-negative cost can be worse because a cohort comparison may remain numerically tidy while measuring a broken treatment path. Favor explicit, reviewable rules over clever scores until the team has enough labeled outcomes to justify complexity. Four well-defined signals, durable state, and a tested delivery boundary are enough to make that first version defensible.
Top comments (0)