Short answer: capture exceptions at the Express boundary and at every agent-job boundary, turn them into low-cardinality metrics, then poll grouped error events inside a short window before alerting. For an e-commerce AI agent loop, keep latency, estimated cost, and the rollback decision in the same event.
The useful mental model is small. Before, one failed inventory lookup creates a stack trace, a log line, and perhaps three notifications after retries. After, the application records the failure once per attempt, groups the same failure by a stable key, and sends one alert when the group crosses a threshold. The alert should answer: what failed, how often, which checkout or recommendation path is affected, and can the last release be rolled back?
That last question changes the design. An error counter without deployment and latency context is a siren with no map.
How should an Express Node.js server capture exceptions and alert?
Capture at two boundaries. Express error middleware sees request exceptions and can attach route, status, request ID, and release. The agent loop needs a second boundary around its worker or scheduled poll, because a failure after the HTTP response has ended will never reach Express middleware. Normalize unknown thrown values into an Error, but preserve the original error name and stack when they exist.
Do not use a raw message as the grouping key. Messages often contain an order ID, URL, or provider request ID. Those values create a new group for every event. Use a stable failure class plus a route or operation name, such as inventory_lookup:TimeoutError, and put volatile detail in fields that are useful for investigation but excluded from metric labels. Prometheus recommends naming metrics for the thing being measured and keeping labels bounded; that is the difference between a useful counter and a cardinality leak.
For the agent loop, record a histogram for duration and a counter for failed attempts. A cost estimate can be an event field or a carefully bounded metric, but it must not become the alert's only reason. A slow successful agent can damage conversion before it throws, while a quick failure in payment authorization may deserve attention immediately.
Here is a deliberately tiny capture and polling contract. The storage adapter could call a log service, a metrics gateway, or an error API. The alert rule stays in application code, where the rollback policy is visible and testable.
type FailureEvent = {
groupKey: string;
operation: string;
release: string;
latencyMs: number;
estimatedCostUsd: number;
occurredAt: number;
detail: string;
};
type FailureGroup = {
groupKey: string;
operation: string;
release: string;
count: number;
latestLatencyMs: number;
estimatedCostUsd: number;
};
interface ErrorApi {
capture(event: FailureEvent): Promise<void>;
groupsSince(sinceMs: number): Promise<FailureGroup[]>;
}
const WINDOW_MS = 5 * 60 * 1_000;
const ALERT_AFTER = 3;
const COOLDOWN_MS = 15 * 60 * 1_000;
const lastAlertAt = new Map<string, number>();
function toError(value: unknown): Error {
return value instanceof Error ? value : new Error(String(value));
}
function groupKey(error: Error, operation: string): string {
return `${operation}:${error.name}`;
}
async function captureFailure(
api: ErrorApi,
error: unknown,
operation: string,
release: string,
startedAt: number,
): Promise<void> {
const normalized = toError(error);
await api.capture({
groupKey: groupKey(normalized, operation),
operation,
release,
latencyMs: Date.now() - startedAt,
estimatedCostUsd: 0,
occurredAt: Date.now(),
detail: normalized.message,
});
}
export async function pollRepeatedErrors(
api: ErrorApi,
notify: (group: FailureGroup) => Promise<void>,
now = Date.now(),
): Promise<void> {
const groups = await api.groupsSince(now - WINDOW_MS);
for (const group of groups) {
const lastAlert = lastAlertAt.get(group.groupKey) ?? 0;
if (group.count < ALERT_AFTER || now - lastAlert < COOLDOWN_MS) continue;
await notify(group);
lastAlertAt.set(group.groupKey, now);
}
}
Register the Express handler after routes, and wrap the agent's job entry point with captureFailure in a catch block. A polling worker can call pollRepeatedErrors every 60 seconds. The ErrorApi adapter should treat a non-success response as a poll failure, apply bounded retry or Retry-After, and expose that poller's own health separately. Never let a slow notification request block the customer request that is already failing.
The sample uses an in-memory cooldown map to keep the rule readable. It is not suitable for multiple poller replicas. Persist alert state in a shared store, or make notifications idempotent with a key such as groupKey + release + windowStart.
Start with the rollback contract, not the notification channel
Imagine an agent that checks stock, recommends a substitute, and calculates a delivery promise. A single request can make several model or API calls. Capturing only the final exception loses the useful sequence: inventory took 820 ms, recommendation took 1,900 ms, the loop retried once, and the estimated request cost crossed the team budget.
The event stream should look more like a compact timeline than a pile of text:
agent.request.started -> inventory.lookup.failed -> retry.started -> recommendation.completed -> agent.request.failed
Attach a correlation ID to every event, but keep it out of metric labels. Use logs or traces for the ID. Use metrics for aggregate questions: how many failures per operation, what is the latency distribution, and which release changed the rate? OpenTelemetry sampling can reduce trace volume, but sampling strategy is a trade-off: head sampling decides early, while tail sampling can keep traces after their outcome is known. A rollback investigation needs enough unsampled evidence to compare releases, so document what your sampling policy may omit.
Rollback safety also needs a release marker. If three errors occur in five minutes, that is an alert condition. If all three belong to the current release and the previous release had a normal rate, it is a rollback candidate. If both releases fail equally, rolling back may only move the symptom. This is why the alert payload should contain release, operation, count, latestLatencyMs, and a linkable correlation field.
One practical rule: alert on a repeated error group, but page on a repeated error group plus customer impact. Customer impact can be a failed checkout step, a high latency budget breach, or an agent fallback rate. The exact threshold is workload-specific; I'm not sure a universal value exists. Start with a five-minute window and three occurrences as an example, then tune it from traffic and business risk.
Where should the polling worker stop?
Polling is a good fit when the error API already groups events and the team wants a small, explicit alert worker. It is easy to run beside a Node.js service, easy to disable during a migration, and easy to test with a fake ErrorApi. It also gives the team a clean rollback boundary: deploy the poller rule separately from the request path, then compare alert behavior before changing capture.
| Method | Best use | Main trade-off |
|---|---|---|
| Poll a grouped error API | A small worker with a clear time-window rule | Detection waits for the next poll and shared alert state is your responsibility |
| Metrics alert | Rates, latency, saturation, and release comparisons | It needs labels and dashboards that explain the underlying exception |
| Trace sampling | One slow or failed agent request across several calls | Sampling can omit the trace you need unless the policy preserves failures |
These are complementary signals, not competing brands. Polling answers “which grouped failure crossed the rule?” Metrics answer “is the service getting worse?” Traces answer “where did this request spend its time?”
The catch is freshness and ownership. A 60-second poll can miss a brief burst unless the API retains events for the full window. A crashed poller delays notification. Multiple replicas can duplicate messages unless shared state or idempotency exists. Polling is not suitable when the requirement is sub-second detection, continuous stream processing, or complex incident correlation; use a streaming or managed alerting system that is designed for that job.
It is also the wrong tool for silent failures. If an order-sync worker stops starting, no exception is captured. Add a heartbeat metric and alert when the expected run does not arrive. Metrics remain the right signal for rates and saturation, while traces explain one slow agent loop and logs preserve the detailed exception. Each signal answers a different question.
Keep the implementation generic. A vendor-neutral adapter lets the application change the backing error API without rewriting Express middleware, grouping policy, or rollback logic. That separation is more valuable than a long integration example: the operational contract remains yours.
What does a rollback-safe setup verify?
Before shipping, test these paths with a fixed clock and a fake API:
- One exception creates one event with a stable group key.
- Three events inside the window create one notification.
- A fourth event during cooldown creates no second notification.
- The same group in an older window creates no alert.
- A new release appears in the alert payload.
- A notification retry cannot send a duplicate.
- A poller failure is visible as its own health signal.
Then test the dangerous path: release a change that makes the inventory operation fail, confirm the alert identifies that release, and verify that rollback stops new failures without erasing the evidence needed for review. Do not count a successful rollback as proof that the original alert was well designed.
Your mileage may vary on the threshold. The shape is stable: capture at the right boundaries, group without unbounded labels, poll a bounded window, deduplicate notification state, and connect the alert to a reversible deployment action.
References
- Prometheus metric naming best practices: https://prometheus.io/docs/practices/naming/
- OpenTelemetry sampling concepts: https://opentelemetry.io/docs/concepts/sampling/
Top comments (0)