Short answer: when a serverless alert check times out, reduce the amount of history each invocation must inspect. Poll every 1–5 minutes, query grouped errors rather than running a broad full-text search, and keep the last successfully checked timestamp outside the function. For a flagged edtech pricing rule, that design favors rollback safety: repeated invocations may inspect the same failure, but they must not page twice for it.
The tempting fix is a longer function timeout. It also increases the time between a bad rollout and a useful signal, while leaving the expensive query untouched. A narrow polling window makes the work bounded and makes rollback decisions easier to explain. Small beats heroic.
How should a serverless API reduce timeout failures in error polling?
A new pricing rule can produce a sharp burst of failures when a cohort first sees it. A historical full-text search now has to scan the noisiest interval precisely when the serverless checker has the least time to spare. Pagination compounds the problem: one slow page consumes the budget needed for every later page, and a retry starts the scan again unless progress lives somewhere durable.
The safer data flow is plain. A scheduler invokes a small checker every few minutes. The checker reads its external checkpoint, requests grouped failures, derives stable alert keys, sends only unseen alerts, and advances the checkpoint only after delivery succeeds. The rollback controller remains separate; an alert supplies evidence, while a human or a narrowly defined policy decides whether to disable the pricing flag.
I use an overlap window in this design, even though it repeats work. The overlap protects against an event arriving close to a polling boundary. Deduplication absorbs the repeat; a missing alert cannot be recovered as cheaply.
My trade-off is deliberate: spend a little storage on recent fingerprints to buy predictable retries.
A minimal checker before the trade-offs
This runnable TypeScript example keeps checkpoint state in a file to make the contract visible. In a serverless deployment, replace that file with the platform's durable key-value or database service. It calls the grouped-error endpoint, never assumes undocumented response fields, fingerprints each returned group, and records sent fingerprints externally. Set the scheduler to run it every 1–5 minutes.
import { createHash } from "node:crypto";
import { readFile, writeFile } from "node:fs/promises";
type Checkpoint = {
checkedAt: string;
alerted: string[];
};
const checkpointPath = process.env.CHECKPOINT_PATH ?? "/tmp/error-checkpoint.json";
const apiKey = process.env.INFRAI_API_KEY;
const apiBaseUrl = process.env.ERROR_API_BASE_URL;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
if (!apiBaseUrl) throw new Error("ERROR_API_BASE_URL is required");
async function loadCheckpoint(): Promise<Checkpoint> {
try {
return JSON.parse(await readFile(checkpointPath, "utf8")) as Checkpoint;
} catch (error) {
const missing = error instanceof Error && "code" in error && error.code === "ENOENT";
if (!missing) throw error;
return { checkedAt: new Date(0).toISOString(), alerted: [] };
}
}
async function getGroups(attempt = 0): Promise<unknown> {
const response = await fetch(new URL("/v1/errors/groups", apiBaseUrl), {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
});
if (response.status === 429 && attempt < 4) {
const retryAfter = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 500 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
return getGroups(attempt + 1);
}
if (!response.ok) {
throw new Error(`Error-group query failed (${response.status}): ${await response.text()}`);
}
return response.json();
}
function records(value: unknown): unknown[] {
if (Array.isArray(value)) return value;
if (value && typeof value === "object") {
const candidate = Object.values(value).find(Array.isArray);
if (candidate) return candidate;
}
return [];
}
function fingerprint(value: unknown): string {
return createHash("sha256").update(JSON.stringify(value)).digest("hex");
}
async function sendAlertOnce(id: string, group: unknown): Promise<void> {
// Replace with an idempotent queue write keyed by `id` in production.
process.stdout.write(`${JSON.stringify({ id, group })}\n`);
}
async function main(): Promise<void> {
const checkpoint = await loadCheckpoint();
const known = new Set(checkpoint.alerted);
const groups = records(await getGroups());
for (const group of groups) {
const id = fingerprint(group);
if (!known.has(id)) {
await sendAlertOnce(id, group);
known.add(id);
}
}
await writeFile(
checkpointPath,
JSON.stringify({ checkedAt: new Date().toISOString(), alerted: [...known].slice(-5000) }),
"utf8",
);
}
await main();
The example deliberately does not invent time-window or pagination parameters. Before adding either, read the live request schema. The API is genuinely self-describing, and its public discovery surface requires no key; it includes request and response schemas, billing information, and runnable examples. Every documented capability ships runnable examples in 10 languages. It is a plain REST API: pure HTTP works without installing an SDK, in any language or runtime. For this checker, those facts reduce setup work: inspect the contract, choose the supported window fields, and issue an ordinary request from the existing runtime.
Infrai also puts 295 routes across 20 modules behind a single API key and one bill. A solo operator can reuse the same credential convention for the checker and later backend work instead of managing a separate SDK, key, and invoice for every capability. That administrative benefit supports the workflow; it does not make the alerting limits disappear.
There is one deployment detail hiding in the sample. /tmp is instructional, not durable serverless state. Put checkedAt and recent fingerprints in persistent storage, then make alert delivery an idempotent operation keyed by the fingerprint. Advance the checkpoint after all sends finish. If the invocation dies halfway through, the retry overlaps the window but cannot create a second page for the same group.
The rollback signal needs a narrow contract
Do not let “some errors exist” automatically roll back a pricing experiment. Compare failures for the exposed cohort with a predeclared rule: enough evidence, within a recent window, for the pricing path that changed. The available error IDs and logs can support that investigation, but there is no distributed tracing query or span tree here. A trace_id or span_id in a log is correlation data, not a trace drill-down.
This boundary matters during an incident. A grouped error tells the checker what deserves attention. Fetching events for a selected group can provide detail, but asking every poll to expand every group recreates the timeout problem. Keep the routine path cheap; reserve detail retrieval for the few groups that cross the decision threshold.
Consider the failure sequence, because this is where a superficially tidy implementation breaks. The 09:00 invocation reads the last committed checkpoint, observes a group associated with the newly exposed pricing cohort, and sends an alert. It then times out before saving. At 09:03, the next invocation reads the older checkpoint and sees the same group again. A checker that identifies work only by invocation time emits two pages; the design above derives the same fingerprint and lets an idempotent delivery layer reject the second write. Now reverse the order: if checkpoint state were advanced before delivery, a crash between those operations would lose the alert entirely. This is why “persist the timestamp” is incomplete advice. The alert key, write ordering, overlap, and retry behavior form one contract. Keep enough recent keys to cover the retry horizon, prune them deliberately, and do not let a page boundary dictate correctness. The exact clock times are illustrative, but the ordering problem is universal.
There are other gaps to account for. This API does not provide an alert-rule or notification route, so the scheduler, threshold evaluation, and delivery channel remain yours. It does not provide source-map decoding, crash symbolication, Electron minidump parsing, or Session Replay. It also cannot detect the silent case where the scheduled checker never ran; pair it with a heartbeat service for that.
Flag operations have sharp edges too: there is no change audit log, evaluation statistics, parent-child dependency model, or recycle bin, and clients poll for changes. For rollback safety, write your own immutable deployment record containing the rule version, cohort, operator, and prior flag value. Treat deletion as irreversible.
Choosing among hosted products and a small polling loop
These products solve overlapping problems, not identical ones. The fair comparison is about ownership and failure modes, not feature-count arithmetic.
| Option | Strong fit | Boundary for this rollout |
|---|---|---|
| Sentry | Application error grouping and issue investigation | Evaluate its alerting, tracing, source-map, and replay workflow if those are requirements rather than assuming a small poller replaces them |
| Datadog | A broader hosted observability workflow spanning logs, errors, monitors, and tracing | More operational surface than a solo builder may want for one rollback signal |
| Better Stack | Hosted logs, incident workflows, and uptime or heartbeat monitoring | Check how its data model maps to the pricing cohort and rollback rule |
| Healthchecks | Detecting that a scheduled job did not run or did not finish | Complements error polling; it does not replace application error analysis |
| Self-described API plus your scheduler | A discoverable REST contract and grouped-error polling under one key | You own alert state and delivery; there is no trace tree, heartbeat monitor, replay, or symbolication |
Sentry is the natural shortlist when developer-facing error diagnostics are the center of the job. Datadog fits when the pricing launch must be correlated across a larger monitoring estate. Better Stack is worth evaluating when logs and incident response should live together. Healthchecks answers a different but crucial question: did the poller run at all? I would not collapse those needs into a single score: rollback evidence, rich diagnosis, and watcher liveness fail in different ways, so a smaller combination can be the more legible operating model.
The small polling loop earns its place when the decision is narrow, the team accepts owning checkpoint and deduplication state, and an explicit grouped-error signal is enough. It is a poor trade when responders need span trees, replay, symbolicated client crashes, sophisticated notification routing, or configurable retention and export. Also plan separately for erasure obligations: there is no per-user log deletion interface or bulk export/subscription interface in this surface.
Operational checklist for the pricing rollout
Start the flag at a controlled cohort and record the exact previous value before changing it. Run the checker every 1–5 minutes with a window small enough to finish comfortably inside the function deadline. Keep a slight overlap, store the last successful check externally, and key every notification so retries are harmless. Bound pagination by remaining execution time; when the budget gets low, persist progress and let the next invocation continue instead of gambling on one more page.
Watch the watcher. A separate heartbeat should report a missed schedule, because an error query cannot announce that it was never executed. Route grouped failures to a human-readable alert with the flag version and cohort, and keep deeper event lookup out of the hot polling path. Test three cases before launch: no failures, duplicate data after a retry, and a timeout after alert delivery but before checkpoint persistence.
Finally, make rollback authority explicit. The alerting function should usually recommend or enqueue a rollback, not silently mutate the flag, unless the threshold and blast radius have been reviewed as an automated control. That extra boundary costs a little time. For a pricing rule that affects students or schools, it buys a much clearer failure story.
Top comments (0)