TL;DR: For failure alerts on a logistics experiment, I would choose request-path counters over repeated log searches when every request already has a stable tenant cohort and outcome. Counters make alert cost attributable to a bounded set of cohorts and make the alert independent of log volume. Keep searchable structured logs for diagnosis. Choose log search as the primary signal only when the failure cannot be represented at the request boundary, or while the cohort model is still changing too quickly to encode safely.
I have been paged by missed scheduled jobs and duplicate deliveries. That experience imposes a plain rule on alert design: the signal has to describe the work the customer expected, not the amount of diagnostic text the service happened to emit. For a logistics API comparing an experiment across tenant cohorts, an alert that says "errors increased" is incomplete. The responder needs the affected cohort, the operation, and whether the request was accepted, rejected, or failed after acceptance.
This is the choice, then: emit a small, bounded counter at the decision point and use logs to explain it. Do not poll the entire log stream merely because the application already writes logs.
Count outcomes.
Should a Node.js API use log-based failure alerts?
The costly part of a vague page is the reconstruction. A responder searches for an error phrase, discovers that one code path phrases the same outcome differently, and then has to recover tenant and experiment context from nearby records. If retries also produce records, a line count can describe attempts rather than failed requests. The alert may be numerically precise and operationally wrong.
For the cohort experiment, I want one event at the point where the API knows the final request outcome. Its dimensions are deliberately boring: operation, cohort, outcome. Tenant identity belongs in the diagnostic log, but not necessarily in the alert counter; one series per tenant makes the monitoring bill and the responder's search space follow tenant growth. A cohort is useful only while it stays bounded and has an owner.
I first assumed log presence was evidence that a scheduled action had happened. Duplicate delivery pages corrected that assumption. I later learned to separate code-path evidence from completion evidence: a log line can prove that code ran, but it does not by itself prove that the intended unit of work completed exactly once. That distinction matters in a Node.js Express API because middleware, a retrying handler, and a downstream completion path can each describe a different unit. The invariant I kept is narrower and more useful: define the unit being counted before choosing the alert transport, then assign one code path responsibility for recording its terminal status.
Two signals, one ownership model
The comparison is not "logs bad, metrics good." It is about which signal carries the stable contract and which one carries investigative detail.
| Decision | Repeated log search | Request-path counter |
|---|---|---|
| Primary cost driver | Records scanned and retained | Number of bounded dimension combinations |
| Cohort attribution | Parsed from each matching record | Attached when the outcome is recorded |
| Query drift | Message or parser changes can alter matches | Counter names and dimensions require an explicit code change |
| Retry handling | Requires a request or job key during analysis | Can count only the terminal outcome |
| Best role | Diagnosis and failures discovered after the request | Paging on known request outcomes |
Cost attribution is clearer with counters in this scenario because the team can point to each admitted dimension. If cohort has three allowed values, the experiment adds three branches per operation and outcome combination; adding raw tenant IDs would make the cost boundary depend on the tenant population. Three is an example design constraint here, not a universal threshold. The important part is that the allowed values are reviewed in code.
Log search still earns its keep. It can find exceptions that never reach the normal outcome recorder, inspect a request key, and reveal the sequence around a duplicate delivery. I would retain enough structured context to answer those questions, but I would not make the paging loop reread that context every minute.
That is the trade-off.
Put the preventative path beside the outcome
The following Go sketch shows the contract I want even if the API itself uses another language. The metric sink is generic. Cohort normalization prevents an unbounded value from quietly becoming an alert dimension, while the diagnostic logger retains the tenant key for investigation.
package alerting
import (
"context"
"log/slog"
)
type Counter interface {
Add(ctx context.Context, name string, value int64, labels map[string]string)
}
type Outcome struct {
TenantID string
Cohort string
Operation string
Result string
}
func recordOutcome(ctx context.Context, counters Counter, logger *slog.Logger, o Outcome) {
cohort := normalizeCohort(o.Cohort)
labels := map[string]string{
"operation": o.Operation,
"cohort": cohort,
"outcome": o.Result,
}
counters.Add(ctx, "logistics_requests_total", 1, labels)
logger.InfoContext(ctx, "request completed",
"tenant_id", o.TenantID,
"cohort", cohort,
"operation", o.Operation,
"outcome", o.Result,
)
}
func normalizeCohort(value string) string {
switch value {
case "control", "candidate":
return value
default:
return "unassigned"
}
}
This code does not solve deduplication. It makes the counting point visible so reviewers can ask the harder question: does recordOutcome run once per attempt, once per accepted request, or once per completed shipment action? Pick one and name the counter accordingly. For paging, I normally care about terminal customer-visible outcomes; an attempt counter can remain a separate diagnostic signal.
The alert then compares failures with total terminal outcomes for each cohort over the same window. A minimum traffic condition matters because one failure out of one request should not be interpreted like one hundred failures out of ten thousand. Those numbers are illustrative, not recommended thresholds. Set the actual window, minimum, and ratio from the service's error budget and the cohort's traffic pattern, then test them against recorded traffic before paging anyone.
Deploy the counter before activating the page. During the overlap, compare the counter with a log query built from the same outcome definition. A mismatch is a contract bug to investigate, not a reason to average the two signals. Put the counter name, allowed dimensions, terminal-outcome definition, and page owner in the runbook. Also test unknown cohorts and retry paths. They are where quiet undercounting tends to hide.
When log search should remain primary
The counter-first choice has a real limitation. It is unsuitable when the failure is discovered only in free-form downstream evidence, when operations cannot yet agree on a stable outcome taxonomy, or when a short investigation is explicitly cheaper than maintaining a new signal. A scheduled reconciliation job may learn hours later that a carrier update never arrived; the request handler cannot count knowledge it did not have. In those conditions, a log-derived signal is the better alternative because it can evaluate evidence that arrived after the API status was returned.
This approach has limits.
Even there, turn the search result into a bounded derived signal before attaching a pager. Record the query definition, deduplication key, scan window, and late-arrival policy. Without those four items, repeated polling can alert twice for the same record or skip a record that arrived after its nominal window.
Do not force experiment labels into every signal forever. Feature cohorts have a lifecycle: creation, exposure, comparison, and retirement. Martin Fowler's feature-toggle guidance distinguishes categories of toggles and emphasizes managing them as inventory rather than leaving them indefinitely. Apply that discipline to observability too. When the experiment ends, remove the cohort dimension or convert it into the stable business classification that actually owns the outcome.
The decision rule I would put in the runbook
Choose request-path counters when the service knows the terminal outcome, cohort values are bounded, and the page must separate tenant cohorts without making cost scale with tenant count. Use structured logs to carry tenant IDs, request keys, retry history, and exception detail. This split gives the pager a stable contract and gives the responder evidence.
Choose log-derived alerts for failures learned outside that path, but define a deduplication key and convert query results into a bounded time series before paging. In either case, charge the signal to the owning operation and cohort, review it with the experiment rollout, and retire experiment-only dimensions with the experiment.
A cheap alert that cannot identify its unit of work is expensive at 03:00. Start with the invariant, then decide where to count it.
Top comments (0)