Short answer: To prevent a serverless timeout in error tracking, replace long-query polling with small error-group checks every one to five minutes, keep the last checked timestamp and alert fingerprint in durable storage, and reserve broad search for incident reconstruction.
For a fintech experiment split across tenant cohorts, the alert loop has one job: tell the on-call engineer that failures changed without timing out or paging twice. The longer query belongs after the page, when logs and error IDs can reconstruct which cohort was affected.
Keep those paths separate.
Why broad searches turn an alert into a capacity problem
A serverless checker has a hard execution budget. A broad historical search consumes that budget in proportion to the history it asks the backend to inspect, while an alert decision usually needs only the newest grouped failures. If the function times out, increasing its timeout merely moves the cliff; it doesn't bound query work, and it makes the detection SLO depend on an increasingly expensive scan.
The safer capacity plan is deliberately boring: schedule a small poll every one to five minutes, record the completed logical window outside the function, and ask the grouped-error endpoint for the compact signal. There is an important constraint here: the discovery schema does not declare filters for the broad logs search, so don't invent a timestamp, window, pagination, or tenant parameter for that route. The window is a client-side scheduling and checkpoint boundary unless the discovered request schema explicitly says otherwise.
A timeout must leave the checkpoint unchanged. The retry then covers the same logical window, while an alert fingerprint prevents the retry from notifying twice. A successful query followed by a failed notification is the awkward case — the state needs separate query and notification markers if the notifier cannot accept an idempotency key. This is where a two-minute poll can still produce a bad night: timing alone is not correctness.
For cohort comparison, attach tenant and experiment identifiers to the errors when they are captured, then retain the error IDs used by the alert. There is no distributed tracing query or span tree here; logs may carry trace_id and span_id, but incident reconstruction still starts from logs and error IDs rather than a trace drill-down.
How should a Node.js serverless alert reduce timeout failures from long query polling?
Use the Node.js scheduler only as the trigger; the control loop is runtime-independent. Set a request deadline comfortably below the function deadline, call the grouped-error route rather than full-text search, retry HTTP 429 with exponential backoff while honoring Retry-After, and commit the checkpoint only after the response has been processed. The Go reference below makes those boundaries visible even if the production function is Node.js.
Do not fake pagination.
If the returned collection can exceed the response or execution budget and discovery exposes no cursor, the correct response is a capacity decision: shorten the interval, narrow what is captured upstream, or choose a service with a documented cursor. Guessing page, limit, or since produces a request that looks plausible in review and has no verified contract. I'm not sure a five-minute interval will meet every team's detection SLO; the function duration distribution, failure arrival rate, and on-call error budget decide whether the interval should be closer to one minute.
Safe implementation with a durable checkpoint
This program performs one poll, which fits a serverless invocation model. It uses exactly one verified route, always sends an explicit method, places the key in the Bearer header, bounds the request, handles 429 without a tight loop, checks non-success responses, and writes state atomically. ERROR_API_BASE_URL, INFRAI_API_KEY, and STATE_PATH must come from the runtime; STATE_PATH must point to durable storage shared across invocations, not ephemeral function storage. Standard output is the handoff to an existing notification pipeline.
package main
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
"time"
)
type checkpoint struct {
LastChecked time.Time `json:"last_checked"`
LastDigest string `json:"last_digest"`
}
func main() {
baseURL := strings.TrimRight(required("ERROR_API_BASE_URL"), "/")
apiKey := required("INFRAI_API_KEY")
statePath := required("STATE_PATH")
previous, err := load(statePath)
must(err)
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
body, err := getWithBackoff(ctx, baseURL+"/v1/errors/groups", apiKey)
must(err)
sum := sha256.Sum256(body)
digest := hex.EncodeToString(sum[:])
checkedAt := time.Now().UTC()
if previous.LastDigest != "" && previous.LastDigest != digest {
alert := map[string]string{
"kind": "error_groups_changed",
"window_start": previous.LastChecked.Format(time.RFC3339),
"window_end": checkedAt.Format(time.RFC3339),
"fingerprint": digest,
}
encoded, err := json.Marshal(alert)
must(err)
fmt.Println(string(encoded))
}
must(save(statePath, checkpoint{LastChecked: checkedAt, LastDigest: digest}))
}
func getWithBackoff(ctx context.Context, url, apiKey string) ([]byte, error) {
client := &http.Client{}
delay := time.Second
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return body, nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return nil, fmt.Errorf("error groups request returned %d: %s", resp.StatusCode, body)
}
wait := delay
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
wait = time.Duration(seconds) * time.Second
}
select {
case <-time.After(wait):
delay *= 2
case <-ctx.Done():
return nil, ctx.Err()
}
}
return nil, fmt.Errorf("rate limit persisted after 4 attempts")
}
func load(path string) (checkpoint, error) {
data, err := os.ReadFile(path)
if os.IsNotExist(err) {
return checkpoint{}, nil
}
if err != nil {
return checkpoint{}, err
}
var value checkpoint
return value, json.Unmarshal(data, &value)
}
func save(path string, value checkpoint) error {
data, err := json.Marshal(value)
if err != nil {
return err
}
temp := path + ".tmp"
if err := os.MkdirAll(filepath.Dir(path), 0700); err != nil {
return err
}
if err := os.WriteFile(temp, data, 0600); err != nil {
return err
}
return os.Rename(temp, path)
}
func required(name string) string {
value := os.Getenv(name)
if value == "" {
panic(name + " is required")
}
return value
}
func must(err error) {
if err != nil {
panic(err)
}
}
The digest is an alert deduplication key, not proof that every changed group is new. If the response schema exposes stable group IDs and timestamps through discovery, store those IDs as the finer-grained deduplication set. Until then, a response-level fingerprint is conservative and contract-safe — it avoids fabricated fields — but it may notify on any representation change. Your mileage may vary with response volume.
Verification, failure injection, and rollback
Deploy this as a shadow check first. For at least one alerting interval, record invocation duration, deadline cancellations, 429 count, checkpoint age, response bytes, and emitted fingerprints without paging anyone. Capacity approval should require p95 execution time to remain below the request budget, plus enough remaining function time to persist state; no measured threshold is universal, so set it from the platform's own SLO and timeout.
Then test four transitions: an unchanged response emits nothing; a changed response emits one fingerprint; a retry with the same response does not emit again after a committed checkpoint; and a timed-out request does not advance last_checked. Inject 429 with both an integer Retry-After and no header. A client that makes four attempts should stop cleanly instead of consuming the whole concurrency pool.
Watch the silent-failure path too. This platform has no alert or notification route and no heartbeat monitoring, so a separate Healthchecks.io-style dead-man check should page when the scheduled poll fails to run. It is complementary: error polling detects reported failures, while heartbeat monitoring detects absence.
Rollback is operational, not architectural: disable the new scheduler, leave the prior alert active, and preserve the checkpoint for later inspection. Don't delete state during rollback. If paging noise rises, route the new output to a non-paging sink while checking fingerprints and cohort tags; if deadline cancellations rise, return to the shorter interval or previous grouped-error check rather than switching the page path back to a broad historical search.
Buy versus build for incident reconstruction
The decision is mostly about who owns alert state, delivery, and the 3 a.m. failure mode. A compact comparison keeps the recommendation honest.
| Option | Operating choice | Best fit | Catch |
|---|---|---|---|
| Infrai | Build the poller around a plain REST API | Teams consolidating backend services behind one key and one bill, while avoiding SDK sprawl | No native alert delivery, distributed trace query, span tree, source-map decoding, crash symbolication, Session Replay, heartbeat monitor, log deletion by user, or bulk log export/subscription |
| Sentry | Buy an error-tracking workflow | Teams that want a packaged error workflow instead of owning this polling loop | Validate tenant-cohort fields, retention, notification routing, and export needs in a proof of concept |
| Datadog | Buy into a broader managed observability platform | Teams already standardizing incident operations in that platform | Evaluate ingestion governance, lock-in, and query behavior at the expected cohort cardinality |
| Grafana Loki | Operate or procure a log-oriented stack | Teams prioritizing log control and query ownership | The platform team retains more capacity planning and on-call responsibility when self-hosting |
| Healthchecks.io | Add dead-man monitoring | Scheduled jobs whose main failure is that they never ran | It complements error tracking; it does not reconstruct application errors by itself |
The consolidated REST option is attractive when key sprawl and month-end invoice reconciliation are already platform problems; its second useful property here is that a Go or Node.js function can call the same HTTP contract without installing a vendor SDK. The catch is real: it is not suitable when the team expects built-in paging, distributed trace drill-down, source-map processing, or replay. Stick with Sentry when a packaged error workflow is the primary requirement, Datadog when the organization already wants its wider managed platform, or Grafana Loki when log control justifies operating more of the stack. Add Healthchecks.io or an equivalent dead-man service whenever silent scheduler failure is inside the paging SLO.
For the stated fintech experiment, start with grouped polling only if tenant-cohort context is already captured and durable checkpoint ownership is acceptable. Otherwise, buying the workflow is cheaper in on-call attention even before anyone opens a pricing sheet.
References
- RFC 5424, The Syslog Protocol: https://datatracker.ietf.org/doc/html/rfc5424
- GDPR Article 17, Right to erasure: https://gdpr-info.eu/art-17-gdpr/
- Sentry alert documentation: https://docs.sentry.io/product/alerts/
- Datadog monitor documentation: https://docs.datadoghq.com/monitors/
- Grafana Loki documentation: https://grafana.com/docs/loki/latest/
- Healthchecks.io documentation: https://healthchecks.io/docs/
Top comments (0)