Persist one small, immutable completion record for every scheduled import, then alert on the absence of an expected record rather than repeatedly asking an error-tracking API to prove that nothing happened. This design reduces long polling queries and serverless timeouts because the critical question becomes a bounded lookup against a checkpoint ledger: did import merchant-catalog-eu produce a terminal result for the expected schedule slot?
TL;DR: divide each API scan into closed time windows, follow pagination inside each window, and commit the cursor only after every page has been processed. Use those scans to enrich incidents, not as the sole liveness signal. The alert should carry the schedule slot, checkpoint state, query interval, page cursor, and correlation identifier needed to reconstruct what the detector knew at decision time.
How should serverless error tracking API polling avoid a timeout?
An absence detector has an awkward burden of proof. An exception proves that some code reported an exception; no exception does not prove that a scheduled job started, reached its source, emitted rows, or committed its result. A process can be killed before error reporting, a scheduler can omit the invocation, and a tracking query can time out before reaching the page containing the relevant event. Silence is ambiguous.
That ambiguity is the problem.
The first invariant is therefore explicit: each schedule slot has one stable identity, such as merchant-catalog-eu/2026-10-08T02:00:00Z, and reaches at most one accepted terminal state. Retries may submit the same completion again, but the state transition is idempotent. This is an exactly-once mindset implemented over at-least-once delivery: uniqueness belongs in durable storage, not in a hopeful assumption about invocation count.
The second invariant is evidentiary. A detector records the cutoff time and the checkpoint revision it evaluated before it emits an alert. If an operator cannot later distinguish “the import was late” from “the detector queried an incomplete interval,” the alert is operationally loud but forensically weak. The failure boundaries matter too: the scheduler, importer, checkpoint store, detector, notification path, and optional error-event API are separate domains, so a healthy detector cannot certify its own notification path and a dead-man signal or synthetic test must watch the watcher. Likewise, an import should acknowledge completion only after its business result is durable; recording completion before the final write creates a false success that reconciliation cannot explain. Consider a slot due at 02:00 with a ten-minute grace period: an evaluation at 02:11 can prove that no accepted completion existed at its cutoff, but an event query that stopped on page four can prove only that four pages were read. Retaining the cutoff, decision, and revision preserves that distinction for later reconciliation.
Three records are enough for the basic audit trail:
-
expected: the immutable schedule slot and its deadline. -
completed: the terminal outcome, row count, result digest, and correlation ID, inserted with a uniqueness constraint on the slot. -
evaluated: the detector's cutoff, decision, and checkpoint revision.
Keep payloads out of those records. Data minimization makes erasure and retention work tractable, particularly where imported source data can contain personal information and deletion obligations apply under GDPR Article 17. Identifiers should be scoped, access controlled, and retained according to a documented purpose rather than copied wholesale into alerts.
Compare the decision paths
The architectural choice is less about which query language is available than about which evidence survives an incident. The table treats reconstruction as the primary axis.
| Decision path | Evidence when the job never starts | Timeout exposure | Reconstruction quality | Appropriate use |
|---|---|---|---|---|
| Query error events over one long interval | None | High because duration and result volume grow together | Weak; a failed or partial query resembles an empty result | Ad hoc investigation of a known interval |
| Query fixed windows with committed page cursors | Indirect only | Bounded per invocation | Good if interval, cursor, and deduplication key are retained | Backfill and incident enrichment |
| Evaluate durable schedule checkpoints | Direct missing-slot evidence | One bounded key or range lookup | Strong; expected, completed, and evaluated records form a timeline | Primary scheduled-import alert |
The checkpoint path wins for primary detection because its negative result has defined scope. It asks for one expected slot after a grace period, rather than inferring system health from a potentially unbounded search through errors. Fixed-window pagination still has value, but it belongs off the alert's critical path or behind a strict execution budget.
There is a trade-off.
The checkpoint design has limitations: it requires control over the importer or its orchestration layer, a durable write with uniqueness semantics, and a migration plan for schedules that change. It is not suitable when the only observable surface is a third-party event API and no completion hook can be added. In that case, fixed-window polling is the defensible alternative, provided each window has a completeness marker, pagination state is durable, and an inconclusive timeout is reported as detector failure rather than import failure.
This separation also clarifies severity. RFC 5424 defines severity numerically from Emergency (0) through Debug (7), while leaving operational interpretation to the application. A missing production import after its contractual grace period might warrant Error severity; a delayed noncritical feed might be Warning. Do not label every retry as an incident. The terminal missed deadline is the auditable event.
Implement the checkpoint and bounded scan
The following Go program shows the critical path with generic interfaces. The checkpoint store enforces idempotency, while the event client exposes opaque pagination so the detector does not invent offsets or assume that an empty page means a complete query. Production implementations must make PutCompletion an atomic insert-or-confirm operation keyed by Slot.ID.
package main
import (
"context"
"errors"
"fmt"
"time"
)
type Slot struct {
ID string
Deadline time.Time
}
type Completion struct {
SlotID string
FinishedAt time.Time
RowCount int64
ResultDigest string
CorrelationID string
}
type CheckpointStore interface {
// PutCompletion atomically inserts the first completion or confirms an identical retry.
PutCompletion(context.Context, Completion) error
GetCompletion(context.Context, string) (Completion, bool, error)
RecordEvaluation(context.Context, string, time.Time, string) error
}
type Event struct {
ID string
OccurredAt time.Time
Severity int
}
type Page struct {
Events []Event
NextCursor string
}
type EventClient interface {
ListEvents(ctx context.Context, from, to time.Time, cursor string) (Page, error)
}
var ErrMissingCompletion = errors.New("scheduled import missed its deadline")
func EvaluateSlot(ctx context.Context, store CheckpointStore, slot Slot, now time.Time) error {
if now.Before(slot.Deadline) {
return nil
}
completion, found, err := store.GetCompletion(ctx, slot.ID)
if err != nil {
return fmt.Errorf("read checkpoint %s: %w", slot.ID, err)
}
decision := "missing"
if found && !completion.FinishedAt.After(now) {
decision = "complete"
}
if err := store.RecordEvaluation(ctx, slot.ID, now, decision); err != nil {
return fmt.Errorf("record evaluation %s: %w", slot.ID, err)
}
if decision == "missing" {
return fmt.Errorf("%w: slot=%s cutoff=%s", ErrMissingCompletion, slot.ID, now.UTC().Format(time.RFC3339))
}
return nil
}
func ScanWindow(ctx context.Context, client EventClient, from, to time.Time, visit func(Event) error) error {
if !from.Before(to) {
return errors.New("scan window must be non-empty")
}
cursor := ""
seen := make(map[string]struct{})
for {
page, err := client.ListEvents(ctx, from, to, cursor)
if err != nil {
return fmt.Errorf("list events [%s,%s) cursor=%q: %w", from, to, cursor, err)
}
for _, event := range page.Events {
if _, duplicate := seen[event.ID]; duplicate {
continue
}
if event.OccurredAt.Before(from) || !event.OccurredAt.Before(to) {
return fmt.Errorf("event %s outside requested half-open window", event.ID)
}
if err := visit(event); err != nil {
return err
}
seen[event.ID] = struct{}{}
}
if page.NextCursor == "" {
return nil
}
if page.NextCursor == cursor {
return errors.New("pagination cursor did not advance")
}
cursor = page.NextCursor
}
}
func main() {
fmt.Println("checkpoint detector interfaces compiled")
}
There is a deliberate half-open interval, [from, to). Adjacent scans can meet at the same timestamp without assigning a boundary event to both windows. Event IDs still require deduplication because APIs can provide overlapping pages while data is arriving, and the durable high-water mark must advance only after the entire page has been handled. Commit it earlier and a crash loses events. Commit it later and a retry repeats work, which is acceptable only when the visitor is idempotent.
Do not let pagination consume the whole serverless execution allowance. Give the scan a shorter context deadline than the platform invocation, stop requesting pages when that budget expires, and persist the last fully processed cursor together with the exact window. The next invocation resumes the same window. It does not widen the query to “catch up,” because widening recreates the timeout and makes an empty response impossible to interpret.
One detail is easy to miss: the in-memory seen map in this compact example protects only a single invocation. A production backfill that can resume across invocations needs durable event deduplication, normally a uniqueness constraint on the source event ID plus the scan namespace. Correctness lives at the write boundary.
Test the evidence before deployment
Unit tests should fix time rather than sleep. Cover completion before the deadline, no completion after the deadline, a store read failure, an evaluation write failure, an identical completion retry, and a conflicting second completion. The last case must fail closed and preserve both attempted digests in an audit event without accepting the new result.
Integration tests need uglier pagination. Return duplicate event IDs across two pages, a cursor that repeats itself, an event exactly at to, a transient page error, and a final page with no next cursor. Then kill the worker after processing a page but before committing its cursor. On restart, the expected outcome is repeated input with one durable effect, not skipped input.
The deployment test is shorter: create a synthetic schedule slot whose importer deliberately emits a completion, verify that no alert fires, then create an authorized test slot with no completion and verify one alert after the configured grace period. Record the test correlation ID so compliance review can distinguish synthetic evidence from production activity.
Observe the detector itself with counts for expected slots, terminal completions, missing decisions, evaluation failures, scan windows, pages, duplicate events, and cursor stalls. Cardinality control is mandatory: slot IDs and correlation IDs belong in structured logs or traces, not metric labels. Log severity should follow a stable policy, and secrets or imported row contents should never be placed in diagnostic fields.
Short tests catch this.
Why reject one long polling query?
The rejected design runs a single error-tracking query from the last successful poll until now and alerts when it finds failure events. It is appealing because it adds no checkpoint store, but it joins three independent questions into one fragile operation: did the import run, did it report an error, and did the query enumerate all matching data before the function deadline? A timeout answers none of them.
It also makes recovery progressively harder. After one failed invocation, the next query covers a longer period and may return more pages, increasing the chance of another timeout. Advancing the cursor despite the failure loses evidence; refusing to advance it creates an expanding retry loop. A fixed maximum window breaks that feedback cycle, while a checkpoint lookup avoids it for primary detection.
The long query still has a valid use case: an operator performing a bounded, interactive investigation where partial results are clearly marked and no alerting decision depends on completeness. Keep it there. Scheduled detection needs deterministic boundaries, idempotent writes, and an audit trail that can explain both the import's state and the detector's judgment without depending on the continued availability of a separate event search service.
For the final operational rule, alert once per missing slot after its deadline and grace period, attach only reconstruction metadata, and resolve the alert when an accepted completion appears. Reconciliation can then compare expected slots with completions and evaluations as three independently countable sets. That structure is less glamorous than a clever query, but it makes silence testable.
References
- RFC 5424, The Syslog Protocol: https://datatracker.ietf.org/doc/html/rfc5424
- GDPR Article 17, Right to erasure: https://gdpr-info.eu/art-17-gdpr/
Top comments (0)