A customer-support AI agent needs three urgent signals: failed conversations, latency that breaches the support promise, and abnormal cost per resolved conversation. Error tracking can evaluate threshold rules and send notifications or webhooks when the service supports those features. When it does not, API polling can approximate the same loop, but it cannot provide instant delivery, perfect event coverage, or unlimited history.
TL;DR: use error tracking for exceptions and broken outcomes, add explicit latency and cost events, and page only on a sustained customer-impact condition. Prefer a signed push event when one is available. Poll a documented API otherwise, with durable cursors, overlap, deduplication, and a separate stale-poller alarm. A free plan is practical only after its retention, rate, rule, and destination limits survive a workload test.
For a one-person SaaS, this is a revenue-per-hour decision. A noisy page steals a feature-shipping block; a missed failure burns customer trust. The useful question is not whether a tool has an Alerts checkbox. It is whether the whole path preserves a high-quality signal from the support conversation to the person who can act.
What changed the design?
A support agent loop can succeed at the HTTP layer and still fail the customer. The model may return too late, a tool call may fail after a retry, or a long exchange may consume far more tokens than the outcome justifies. Exception counts alone miss those cases. Raw request latency is noisy because it mixes short FAQ answers with conversations that need several tool calls.
The unit of alerting should therefore be the completed support conversation, not every internal span or model request. Record one outcome event with a stable conversation identifier, duration, resolution state, estimated usage or cost, and deployment version. Keep the detailed steps for diagnosis, but do not turn each one into a paging candidate.
The first tempting rule is usually "alert on any error." It feels safe. It is mostly a noise generator once retries and expected tool failures enter the loop. A useful rule needs both customer impact and enough volume to mean something. For example, evaluate a rolling window only after a minimum number of completed conversations, then page when the unresolved ratio or tail latency stays above the service's own objective for more than one evaluation. Those values must come from the support promise and observed traffic; there is no universal threshold hidden in an error-tracking product. Consider a five-step conversation in which the second tool call times out and succeeds on retry: paging the timeout reports internal turbulence, while recording the completed conversation reports the outcome the customer actually received. If that same retry pushes the whole exchange beyond the support objective, the conversation-level latency signal catches the harm without pretending every recovered operation was an incident.
This changes the data model. Good.
| Signal | Event scope | Urgency | Why |
|---|---|---|---|
| Unresolved conversation | Completed conversation | Page after a sustained breach | Direct customer impact |
| End-to-end latency | Whole agent loop | Page when the objective is breached | Internal step timing is diagnostic noise |
| Cost per resolution | Aggregated outcome | Review or ticket first | Spikes rarely require waking someone |
| Individual retry | Internal operation | Store for diagnosis | A recovered retry is not an incident |
That split is the core signal-quality choice. Pages protect the customer promise. Tickets protect margins. Stored details explain both.
Can error tracking threshold alerts use webhooks instead of polling?
Not completely. A webhook and a poller are delivery mechanisms, not proof that a threshold was evaluated correctly. The rule engine, event retention, aggregation window, notification destinations, and delivery guarantees are separate capabilities. A free plan may expose some and constrain others, so its label tells you less than a short end-to-end test.
Push is the cleaner path for urgent signals because the receiver doesn't perform repeated read traffic and can react when the sender delivers. The receiver still needs signature verification when the sender provides signing, replay protection, idempotency, timeouts, and a dead-letter path. Delivery can be duplicated or delayed. Treat the event identifier as a deduplication key and acknowledge only after durable acceptance.
Polling is useful when outbound webhooks or notification destinations are unavailable. It introduces a detection delay of up to the polling interval under normal operation, plus processing and network time. Shortening the interval increases API traffic and the chance of hitting a rate limit. Lengthening it delays detection. Pagination, retention, clock skew, and late-arriving records can create gaps unless the poller uses a cursor and deliberately rereads an overlap window.
There is one more failure mode: the poller can stop. Add a heartbeat evaluated somewhere outside the polling process. Otherwise silence means either "no incidents" or "the detector died," and those states look identical.
Silence isn't health.
Notifications sit downstream of all this. Email, chat, or an incident system can be a fine destination, but destination count is not signal quality. Route a page to one accountable place, and put lower-urgency cost drift into the weekly operating review. Shipping weekly is hard enough without three channels repeating the same alert.
The smallest working implementation
The useful boundary is a small adapter. It accepts normalized conversation outcomes, applies a local rule, and sends an idempotent incident message. The event source can later change from an error-tracking API to a queue or a metrics backend without rewriting the decision logic.
type ConversationOutcome = {
id: string;
finishedAt: string;
durationMs: number;
resolved: boolean;
estimatedCostUsd: number;
};
type Page = { key: string; summary: string; observedAt: string };
function evaluateWindow(events: ConversationOutcome[]): Page | null {
const minimumSample = 20;
if (events.length < minimumSample) return null;
const unresolved = events.filter((event) => !event.resolved).length;
const unresolvedRatio = unresolved / events.length;
const latencies = events.map((event) => event.durationMs).sort((a, b) => a - b);
const p95Ms = latencies[Math.ceil(latencies.length * 0.95) - 1];
// Examples only: replace these with objectives derived from the support promise.
if (unresolvedRatio < 0.1 && p95Ms < 15_000) return null;
const end = events.at(-1)?.finishedAt ?? new Date().toISOString();
return {
key: `support-window:${end}`,
summary: `Support degraded: unresolved=${unresolvedRatio.toFixed(2)}, p95Ms=${p95Ms}`,
observedAt: end,
};
}
The numbers demonstrate the mechanism; they are not operational recommendations. A production rule should use two consecutive breached windows or another explicit persistence condition. It should also distinguish low traffic from healthy traffic. Twenty quiet conversations cannot justify the same confidence as thousands, and zero conversations may itself be a collection failure.
A poller needs durable state. A timestamp by itself is fragile because several events can share it, so prefer an opaque server cursor when the API supplies one. If it does not, persist a compound high-water mark and reread a bounded overlap. Deduplicate by event ID before evaluation.
type Cursor = { token?: string; lastSeenAt?: string };
type PageResult = { events: ConversationOutcome[]; next: Cursor };
interface OutcomeSource {
read(cursor: Cursor, signal: AbortSignal): Promise<PageResult>;
}
interface IncidentSink {
send(page: Page, idempotencyKey: string): Promise<void>;
}
async function pollOnce(
source: OutcomeSource,
sink: IncidentSink,
cursor: Cursor,
): Promise<Cursor> {
const result = await source.read(cursor, AbortSignal.timeout(8_000));
const unique = [...new Map(result.events.map((event) => [event.id, event])).values()];
const page = evaluateWindow(unique);
if (page) await sink.send(page, page.key);
return result.next;
}
Persist the returned cursor only after the events and any resulting incident have been accepted durably. Retry with bounded exponential backoff and jitter. A rate-limit response should postpone the next read rather than launch parallel retries. If the source cannot promise stable pagination, test inserts during pagination before trusting it for paging.
The receiver side is equally small: validate authenticity, reject stale deliveries, deduplicate the event ID, enqueue the work, and respond quickly. Business logic does not belong in the request handler. The queue gives retries somewhere durable to live.
Keep it boring.
How do you test a free plan without trusting its feature grid?
Run a synthetic incident through the complete path before routing real pages. Generate known outcome events, cross the proposed threshold, and measure whether exactly one actionable notification arrives. Then repeat with an event on the aggregation-window boundary, a duplicated delivery, a delayed event, and enough pages to exercise pagination.
Check the practical limits in the account and current documentation: event retention, API rate limits, query granularity, number of rules, minimum evaluation interval, available destinations, webhook signing, retry behavior, payload size, and export access. These details can change, which is why hard-coding a claim about a free allowance into the architecture is brittle. Record the tested limits and date in an internal runbook instead.
Also test privacy controls. Support conversations may contain sensitive customer text, while alert evaluation usually needs only identifiers, timings, states, and usage totals. Keep message bodies out of the alert payload. Collect the minimum diagnostic context, define deletion behavior, and make telemetry choices explicit. The DO_NOT_TRACK convention is one public example of an opt-out signal for command-line telemetry; it is useful evidence that collection preferences should be straightforward to express, though it does not define an alerting protocol.
A pass means more than "the message arrived." The event must be attributable to the right deployment, deduplicated, delivered inside the response objective, and useful without opening five dashboards. Test the recovery message too. Alerts that never resolve train the operator to ignore them.
What I would change at scale
At higher support volume, move window aggregation out of the polling worker and into a metrics or stream-processing layer. Keep error tracking as diagnostic evidence, not the only ledger of customer outcomes. Evaluate service-level indicators over consistent windows, attach conversation IDs for investigation, and retain a small audit record explaining why each incident opened.
I would also split detection from routing. Detection emits a vendor-neutral incident event; routing decides who gets paged, deduplicates related symptoms, applies maintenance windows, and escalates. That boundary makes it possible to replace either side without changing the customer-support application. It costs another component, so a low-volume solo operation should earn that complexity first.
The trade-off remains signal quality versus noise. More instrumentation is cheap to add and expensive to attend to. Start with the three conversation-level signals, review false positives every week, and promote a cost warning to a page only when delayed action creates immediate customer harm. Everything else can wait for working hours.
Error tracking can participate in this system, and in some deployments it can host the rules and delivery path. It should not define the reliability model. Own the outcome schema, the threshold rationale, and the delivery test. Outsource the undifferentiated storage and transport only after proving that their limits fit the support promise.
Top comments (0)