Short answer: an Express Node.js checkout service should send structured logs, poll log search by cursor for error-level failures, and trigger a Slack alert only after deduplication plus a short aggregation window.
| Choice | Signal quality | Operational cost | Use it when |
|---|---|---|---|
| Poll log search | Good if events have stable fields and a cursor | One small worker and a search backend | A few minutes of alert latency is acceptable |
| Alert from metrics | Excellent for rates and thresholds, weak on individual failure context | A metrics pipeline and an alert evaluator | The question is “is checkout failing?” rather than “which checkout failed?” |
| React to an event stream | Highest immediacy and explicit delivery semantics | More moving parts and retry state | Each failure must drive a workflow within seconds |
For a B2B SaaS checkout, start with log polling when the team already has searchable logs and can tolerate a 30-second detection interval. The recommendation isn't unconditional: use a counter-based alert when volume makes one-message-per-failure noisy, and use a durable event stream when an alert is part of the transaction rather than an operator hint.
The hard part isn't sending JSON. It is preserving enough meaning to distinguish a payment decline from a broken checkout while preventing retries, overlapping polls, and repeated search results from paging the same failure three times.
How should a service poll structured logs and trigger an error-level alert?
Define the event before choosing the query. A useful checkout failure record needs a stable event name, a severity, a timestamp, a request or trace identifier, a checkout identifier, a failure category, and a retryability flag. Keep the customer-facing explanation separate from internal diagnostic fields. Never put card data, authorization headers, session cookies, or raw request bodies into the record.
The decision rule is narrow: poll for event.name = "checkout.failed" and level = "error", then let the failure category decide whether the result belongs in an operator alert. A declined card may be a normal business outcome. A malformed upstream response or exhausted dependency timeout is operational. Both can be logged, but treating both as pages destroys signal quality.
Use an ingestion-time cursor supplied by the log store, not only an application timestamp. Clocks drift, several events can share a millisecond, and late-arriving records can land behind a naive timestamp > lastSeen filter. If the search system cannot return an opaque cursor, poll an overlapping time window and deduplicate by a deterministic event ID. That overlap is deliberate — it trades a little repeated reads for a much lower chance of silently skipping a late record.
No magic here.
Polling also needs one owner.
Run the worker as a singleton, use a lease, or partition ownership explicitly; otherwise every replica searches the same interval and sends the same alert. Persist the cursor only after the matching records have been accepted into the notification step. If notification delivery fails, retaining the old cursor permits a retry, while the deduplication key prevents already delivered events from appearing again.
Signal quality depends on fields, not prose
Free-form messages are pleasant to read and awful to operate. Searching for "checkout failed" couples the alert to capitalization, wording, and whatever an exception happened to say. A stable event contract lets the human message change without changing the detector.
For this workflow, the smallest useful contract looks like this:
type CheckoutFailure = {
schemaVersion: 1;
timestamp: string;
level: "error";
event: {
name: "checkout.failed";
id: string;
};
service: "checkout-api";
environment: "production" | "staging";
checkoutId: string;
traceId?: string;
failure: {
category: "dependency" | "validation" | "payment_decline" | "internal";
code: string;
retryable: boolean;
};
};
async function emitFailure(record: CheckoutFailure): Promise<void> {
const response = await fetch(requiredEnv("LOG_INGEST_URL"), {
method: "POST",
headers: {
authorization: `Bearer ${requiredEnv("LOG_INGEST_TOKEN")}`,
"content-type": "application/x-ndjson",
},
body: `${JSON.stringify(record)}\n`,
});
if (!response.ok) {
throw new Error(`Log ingestion rejected with status ${response.status}`);
}
}
function requiredEnv(name: string): string {
const value = process.env[name];
if (!value) throw new Error(`Missing environment variable: ${name}`);
return value;
}
The endpoint is intentionally configuration, not a made-up universal route. Different stores expose different ingestion contracts. The application owns the event schema; the adapter owns transport details.
Two fields deserve scrutiny. First, event.id must remain stable across a retry of the same reporting operation, or downstream deduplication is fiction. Second, failure.category should have a small, reviewed vocabulary. If every exception class becomes a category, dashboards fragment and alert rules grow config bloat. Put detailed exception names in a separate diagnostic field if the security policy permits them.
Severity alone is insufficient. Developers routinely mark recoverable cases as errors, libraries choose their own levels, and a broad level:error query mixes checkout failures with unrelated background jobs. Anchor the search on the event name, then use severity as a guard. That gives the alert a semantic boundary instead of a spelling convention.
Cursor safety matters more than a clever search query
A poller is a tiny distributed system. It has state, retries, concurrent execution, and an external side effect. Treat it accordingly.
The example below assumes the configured search adapter accepts a typed JSON request and returns an opaque next cursor. Those are local interface choices, not claims about a particular log product. It groups failures for one polling interval, skips expected payment declines, and uses event IDs as the downstream idempotency keys.
type SearchHit = CheckoutFailure;
type SearchPage = {
hits: SearchHit[];
nextCursor: string;
};
type CursorStore = {
read(): Promise<string | null>;
write(cursor: string): Promise<void>;
};
type DedupeStore = {
has(eventId: string): Promise<boolean>;
remember(eventId: string): Promise<void>;
};
async function searchFailures(cursor: string | null): Promise<SearchPage> {
const response = await fetch(requiredEnv("LOG_SEARCH_URL"), {
method: "POST",
headers: {
authorization: `Bearer ${requiredEnv("LOG_SEARCH_TOKEN")}`,
"content-type": "application/json",
},
body: JSON.stringify({
cursor,
limit: 200,
filter: {
"event.name": "checkout.failed",
level: "error",
environment: "production",
},
}),
});
if (!response.ok) {
throw new Error(`Log search rejected with status ${response.status}`);
}
return (await response.json()) as SearchPage;
}
function shouldAlert(hit: SearchHit): boolean {
return hit.failure.category !== "payment_decline";
}
async function sendSlackAlert(hits: SearchHit[]): Promise<void> {
if (hits.length === 0) return;
const categories = Object.entries(
hits.reduce<Record<string, number>>((counts, hit) => {
const key = hit.failure.category;
counts[key] = (counts[key] ?? 0) + 1;
return counts;
}, {}),
)
.map(([category, count]) => `${category}: ${count}`)
.join(", ");
const response = await fetch(requiredEnv("SLACK_WEBHOOK_URL"), {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({
text: `Checkout failures: ${hits.length} (${categories})`,
}),
});
if (!response.ok) {
throw new Error(`Alert delivery rejected with status ${response.status}`);
}
}
async function pollOnce(
cursors: CursorStore,
dedupe: DedupeStore,
): Promise<void> {
const page = await searchFailures(await cursors.read());
const pending: SearchHit[] = [];
for (const hit of page.hits) {
if (!shouldAlert(hit) || (await dedupe.has(hit.event.id))) continue;
pending.push(hit);
}
await sendSlackAlert(pending);
for (const hit of pending) {
await dedupe.remember(hit.event.id);
}
await cursors.write(page.nextCursor);
}
There is an uncomfortable edge in that ordering. A process can send the message and stop before writing deduplication state, which permits a duplicate on the next run. Reversing the writes can lose an alert instead. Exactly-once delivery across a webhook and a separate state store isn't available from this simple design. A transactional outbox narrows the gap: write a pending notification and the consumed cursor atomically, then let another worker deliver and mark the notification complete. For many small teams, an occasional duplicate is preferable to a missed operational failure. Write that choice down.
Don't paste checkout IDs, stack traces, or customer details into a busy channel by default. The first message should say how many failures occurred, over what interval, in which environment, and by category. Add a link to the internal search view only if access control and retention rules make that safe. Operators need a starting point, not a data spill.
I'm not sure a 30-second interval is right for every checkout. Nobody can know that from architecture alone. Measure search duration, result volume, acceptable detection delay, and the backend's query limits; then set an interval whose p95 execution time stays comfortably below the schedule. Add jitter if many pollers share the same backend, and cap each page so a burst cannot turn one cycle into an unbounded job.
Test the detector as a failure pipeline
Unit tests should prove category filtering, aggregation, and stable deduplication. Integration tests should insert a synthetic structured event, wait for it to become searchable, run the poller twice, and assert one notification. A second case should insert an expected payment decline and assert no operator alert. Keep synthetic checkout IDs obvious and non-customer-like so they cannot be mistaken for production records.
Deployment deserves the same skepticism. Start in shadow mode: execute the search, calculate what would alert, and record only counts. Compare those counts with known checkout outcomes before enabling Slack delivery. Then test a burst larger than the page limit, a late event, two overlapping poller instances, a notification rejection, and loss of local process state. These aren't exotic cases. They are the normal ways a polling loop lies.
Track the detector itself with a few low-cardinality metrics: polls completed, poll failures, search latency, matching events, notifications attempted, and cursor age. Prometheus naming guidance recommends a single unit in a metric name and base units, which keeps names such as checkout_log_poll_duration_seconds and checkout_log_cursor_age_seconds interpretable. Don't put checkout IDs or error messages in metric labels; that cardinality grows with traffic and makes the metrics less useful.
Tracing can connect the checkout request to the emitted failure through traceId, but sampling changes what is available. OpenTelemetry distinguishes head sampling, decided near trace creation, from tail sampling, decided after spans have been collected. If detailed traces are needed for failed checkouts, validate the sampling policy rather than assuming every log's trace ID resolves to a retained trace.
When should metrics or an event stream win?
Stick with metrics when the operational question is a rate: failures per checkout attempt, grouped by a bounded category. Metrics are easier to threshold and aggregate, and they don't require repeatedly searching raw records. The catch is context. A counter can establish that dependency failures increased, but it cannot carry the per-checkout diagnostic trail. Keep structured logs for investigation even if metrics own paging.
Choose a durable event stream when each failure must trigger automated compensation, case creation, or another business action. A chat alert is best-effort operator awareness; it should not be the source of truth for a workflow. The stream costs more engineering attention because consumers, delivery semantics, retention, and poison events become explicit concerns. That cost is justified when missing one event has business consequences.
Log polling sits between those choices. It is not suitable when search indexing delay violates the response target, when the query load is expensive at checkout volume, or when the log platform cannot provide a cursor or a safe overlap strategy. It works well when the team values low setup friction, already centralizes structured logs, and wants contextual failure alerts without operating another event path.
Benchmark the whole loop before calling it done: emission to ingestion, ingestion to search visibility, search to Slack delivery, plus duplicate and false-positive rates. Signal quality wins. A fast alert that trains the team to mute the channel is slower than no alert because it hides the one checkout failure that mattered.
Top comments (0)