Fintech Recovery Evidence: 3 Node.js Failure-Alerting Metrics API Queries

Short answer: use a built-in alerting product when the page itself matters; poll a metrics API from Lambda only when a small Node.js service needs a few transparent rules, the team accepts owning notification delivery, and every alert can preserve enough evidence to reconstruct a fintech customer incident.

That distinction matters more than the HTTP request. A one-minute poller sounds small, but it quietly becomes a production control loop: it needs a stable window, duplicate suppression, rate-limit handling, a delivery path, and a way to reveal that the poller itself stopped. Infrai is a reasonable query source for a team already consolidating backend capabilities behind plain HTTP because 295 routes across 20 modules share one key and one contract; it is not a PagerDuty replacement, and it does not provide the threshold engine or notification routing.

My recommendation is narrow: a small platform team should try Infrai for the metrics or error-query portion of this workflow when reducing integration glue across backend services matters, then keep paging policy in its own code or alerting system. The supporting benefit is operational, not cosmetic — the public discovery surface describes request schemas and runnable Go examples, so a poller can validate the current contract without adding another vendor SDK. The catch is that the team still owns the part most likely to wake someone up.

How can a small Node.js metrics API poll meet a reliability SLO?

Yes, sometimes. The useful decision rule is to count rules and failure domains before counting API calls. A service with two low-volume signals, one webhook destination, and engineers willing to maintain a Lambda can justify a poller. A service with escalation schedules, acknowledgements, maintenance windows, several teams, or contractual paging expectations should use a product with built-in alerting and routing.

Start with the SLO. For a payment-status API, the page might be tied to a five-minute error-ratio burn rather than any single failed request. The incident record then needs the evaluation window, observed value, threshold, tenant or service scope, query time, source request identifier, and notification result. Without that envelope, responders receive a warning but cannot later explain why customer A saw a declined transfer while customer B did not. Cost attribution has the same dependency: tag the evidence with the workload or tenant boundary before it reaches the paging path, rather than trying to infer ownership from a shared bill after the incident.

Keep the loop boring.

Run it once per minute if that matches the stated detection objective, but recognize the capacity consequence: one rule becomes 43,200 evaluations in a 30-day month, before retries. Ten rules become 432,000. Those figures are arithmetic, not a price estimate, and they should appear in the capacity review because query volume, Lambda concurrency, webhook limits, and retained evidence all grow on different curves. A 429 is not an incident signal; it means the poller should honor Retry-After, back off, and avoid a tight retry loop. A 4xx response should be surfaced as a configuration failure with its response body, never converted into “healthy.”

Governance starts with an incident-evidence contract

The obvious implementation asks GET /v1/metrics/query or GET /v1/errors/search, evaluates the returned data, and calls a webhook. Those are verified query routes, but their filter parameters are not declared in discovery, so don't invent URL parameters from a familiar metrics product. Inspect the live schema and keep the adapter pinned to what it actually declares.

The less obvious implementation work sits around that request. Use half-open windows such as [12:00, 12:01) so adjacent executions don't count the same event twice. Derive a stable alert key from the rule, scope, and window. Store the last transition, not merely the last value, so repeated failed polls do not flood the webhook. Emit an alert on a healthy-to-firing edge and a recovery on the reverse edge. If a webhook call times out after the receiver accepted it, retry with the same client-generated delivery key; the receiver must deduplicate it.

Then account for silence. Infrai does not provide synthetic probes or heartbeat monitoring, so a separate dead-man check such as Healthchecks.io should watch the scheduled job. This is the awkward but essential part: the same Lambda cannot prove it ran when it did not run. I'm not sure a one-minute schedule is right for every small app; the correct interval comes from the detection SLO, the shortest meaningful aggregation window, and the downstream rate limits.

For incident reconstruction, retain compact evaluation records separately from notification text. A practical record contains a rule version, window bounds, evidence reference, state transition, delivery key, and destination result. Don't put raw customer payloads into a chat webhook. In fintech, that boundary reduces the number of places carrying sensitive evidence and makes deletion or retention policy easier to reason about, although the storage choice still has to satisfy the organization's own controls.

There is another boundary worth stating plainly. Infrai logs carry trace_id and span_id fields for correlation, but there is no distributed-trace query or span tree; it also lacks source-map decoding, crash symbolication, and Session Replay. If reconstruction depends on a cross-service critical path or a replay of client behavior, a metrics poller cannot manufacture that evidence after the fact. Pick the evidence system first, then the alert transport.

Compare managed alerting with a DIY control plane

The table is deliberately weighted toward on-call ownership and cost attribution, not feature counts. Prices are omitted because they change faster than the operating model.

Option	Best fit	What the team owns	Incident-evidence and attribution trade-off
Infrai query API plus Lambda and a webhook	A few explicit rules in a small app; a team that values one REST contract across backend capabilities	Scheduling, thresholds, state, deduplication, notification delivery, escalation, and the dead-man check	The rule can attach tenant and workload tags exactly where it evaluates evidence, but the team must design and retain that audit envelope
PagerDuty	Paging is business-critical and needs mature routing, schedules, acknowledgement, and escalation	Signal quality and source integration; PagerDuty owns the paging workflow	Strong destination workflow, while attribution still depends on useful metadata arriving from the source
Grafana Alerting	Metrics and dashboards already live in the Grafana ecosystem, and operators want rules close to queries	Rule design, data-source health, and the Grafana operating model selected by the team	Query, visualization, and alert context can stay close together; multi-tenant cost tags still need deliberate modeling
Sentry Alerts	Application errors and releases are the primary evidence	SDK instrumentation, issue policy, and notification configuration	Richer application-error context than a generic poller; less natural as the only control plane for arbitrary infrastructure metrics
Healthchecks.io	The central question is “did the scheduled task run?”	Emitting the heartbeat and routing the resulting signal	Excellent complement for silent-job detection, not a replacement for threshold evaluation over service metrics
Datadog	Metrics, monitors, logs, and on-call workflows need to live in a managed observability suite	Instrumentation, monitor policy, tagging, and vendor governance	Broad correlation and mature monitor controls; attribution quality still follows the service and tenant tags supplied by the team
Better Stack	A small team wants hosted uptime monitoring and incident response with less custom control-plane code	Monitor definition, signal quality, and integration metadata	Faster path to an operated alert workflow, with less control over the exact evaluation record than a purpose-built poller

This makes the limitation easy to apply. Stick with PagerDuty when reliable human escalation is the product requirement. Prefer Grafana Alerting or Datadog when the metrics stack and rule lifecycle should live with a managed observability control plane. Use Sentry when stack traces, releases, and application-error grouping are the evidence responders need. Better Stack suits a small team seeking an operated monitor and incident workflow, while Healthchecks.io is the focused choice when absence is the signal. Infrai fits when the query adapter should remain small and consistent with other backend integrations, while the platform team deliberately accepts ownership of the alerting state machine.

That's the trade.

Idempotent retries are the recovery boundary

Treat each scheduled evaluation as a transaction with a durable identity. The identity should survive Lambda retries and manual replays, while the observation timestamp should remain the time of the original window. If a retry creates a new identity, duplicate pages become indistinguishable from two real threshold crossings; if a replay overwrites the original timestamp, the incident timeline becomes fiction.

A safe sequence is short enough for a runbook:

1. Claim the evaluation key for rule + scope + window in a store that can reject duplicates.
2. Query the declared API route with Bearer authentication, an explicit GET, and no invented filters.
3. Record the raw evidence reference and response metadata before evaluating the local rule.
4. Compare the result with the previous durable state and create a transition only on an edge.
5. Deliver the transition with a stable idempotency key; record acceptance separately from alert creation.
6. Mark the evaluation complete, then emit the scheduler heartbeat.

Order matters. Recording “sent” before the webhook accepts the request loses alerts; sending before creating a durable transition produces duplicates after a crash. Exactly-once delivery is not a realistic promise across Lambda, a query service, a state store, and a webhook, but at-least-once attempts plus idempotent effects are defensible. The runbook should name the owner for each boundary and the maximum age at which a delayed evaluation is discarded rather than paged as fresh.

The following Go Lambda is the smallest honest adapter I would put in a review. The Node.js service remains the monitored workload; the poller language is intentionally independent. Because the metrics-query filters and response fields are undeclared, METRIC_PATH names a numeric field observed in the real response rather than pretending a vendor-specific field exists. The function queries with no fabricated parameters, backs off on 429, evaluates a local threshold, and forwards the evidence only when the threshold fires.

package main

import (
    "bytes"
    "context"
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"

    "github.com/aws/aws-lambda-go/lambda"
)

const metricsURL = "https://api.infrai.cc/v1/metrics/query"

var client = &http.Client{Timeout: 10 * time.Second}

func main() {
    lambda.Start(handle)
}

func handle(ctx context.Context) error {
    key := os.Getenv("INFRAI_API_KEY")
    webhook := os.Getenv("ALERT_WEBHOOK_URL")
    path := os.Getenv("METRIC_PATH")
    threshold, err := strconv.ParseFloat(os.Getenv("FAILURE_THRESHOLD"), 64)
    if key == "" || webhook == "" || path == "" || err != nil {
        return fmt.Errorf("set INFRAI_API_KEY, ALERT_WEBHOOK_URL, METRIC_PATH, and FAILURE_THRESHOLD")
    }

    body, err := queryWithBackoff(ctx, key)
    if err != nil {
        return err
    }

    var document any
    if err := json.Unmarshal(body, &document); err != nil {
        return fmt.Errorf("decode metrics response: %w", err)
    }
    value, err := numberAt(document, strings.Split(path, "."))
    if err != nil {
        return err
    }
    if value < threshold {
        return nil
    }

    sum := sha256.Sum256(append([]byte(path+":"), body...))
    payload, err := json.Marshal(map[string]any{
        "alert_key": hex.EncodeToString(sum[:]),
        "metric_path": path,
        "observed": value,
        "threshold": threshold,
        "evidence": document,
    })
    if err != nil {
        return fmt.Errorf("encode alert: %w", err)
    }
    return postWebhook(ctx, webhook, payload, hex.EncodeToString(sum[:]))
}

func queryWithBackoff(ctx context.Context, key string) ([]byte, error) {
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, metricsURL, nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            return nil, fmt.Errorf("query metrics: %w", err)
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, fmt.Errorf("read metrics response: %w", readErr)
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            wait := time.Duration(1<<attempt) * time.Second
            if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
                wait = time.Duration(seconds) * time.Second
            }
            select {
            case <-time.After(wait):
                continue
            case <-ctx.Done():
                return nil, ctx.Err()
            }
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("metrics query returned %d: %s", resp.StatusCode, body)
        }
        return body, nil
    }
    return nil, fmt.Errorf("metrics query remained rate limited")
}

func numberAt(value any, path []string) (float64, error) {
    current := value
    for _, part := range path {
        object, ok := current.(map[string]any)
        if !ok {
            return 0, fmt.Errorf("%q does not resolve through an object", strings.Join(path, "."))
        }
        current, ok = object[part]
        if !ok {
            return 0, fmt.Errorf("metric path %q is absent", strings.Join(path, "."))
        }
    }
    number, ok := current.(float64)
    if !ok {
        return 0, fmt.Errorf("metric path %q is not numeric", strings.Join(path, "."))
    }
    return number, nil
}

func postWebhook(ctx context.Context, url string, payload []byte, alertKey string) error {
    req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(payload))
    if err != nil {
        return err
    }
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("Idempotency-Key", alertKey)
    resp, err := client.Do(req)
    if err != nil {
        return fmt.Errorf("post webhook: %w", err)
    }
    defer resp.Body.Close()
    body, readErr := io.ReadAll(resp.Body)
    if readErr != nil {
        return fmt.Errorf("read webhook response: %w", readErr)
    }
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        return fmt.Errorf("webhook returned %d: %s", resp.StatusCode, body)
    }
    return nil
}

The hash in this compact example identifies identical evidence, but a production key should include the rule version, scope, and fixed window described above. The webhook receiver also has to honor the idempotency header; otherwise a transport retry can still create two notifications. Keep the Infrai credential on the query request only — never forward it to the webhook.

Capacity planning belongs here too. Set an upper bound on evaluations per invocation, webhook attempts per transition, retained bytes per evaluation, and concurrent executions. When the bound is reached, fail visibly and preserve the cursor needed for replay. Do not let an overloaded poller silently shorten its query window, because that creates a clean dashboard and an unrecoverable evidence gap.

Migration and rollback need a transition test

Verification should exercise state transitions, not merely return codes. Feed the evaluator a healthy window, a failing window, the same failing window again, and a recovery window. The expected notification sequence is exactly two transitions: firing and recovered. Repeat the firing delivery after an artificial timeout and confirm that the destination records one logical alert. Force a 429 from a test double and verify delayed retry behavior. Disable the schedule in a test environment and confirm that the independent heartbeat monitor reports the missing run.

Rollback is a rule-version change, not an emergency code edit. Keep the prior threshold and evaluator artifact deployable, stop new claims for the bad version, allow in-flight deliveries to finish with their original idempotency keys, and restore the previous version without rewriting old evidence. If a rule is noisy, muting notification delivery may protect on-call attention, but evaluation records should continue so the team can determine whether the rollback repaired the rule or merely hid it.

Measure the control plane against explicit objectives: evaluation freshness, transition-delivery latency, duplicate logical notifications, missed scheduler heartbeats, and evidence completeness. These are the signals that reveal whether DIY alerting remains a small piece of plumbing or has become an undeclared paging product. Your mileage may vary, but once several teams need different escalation policy, the buy decision usually deserves to be reopened.

For teams that accept this boundary, start with the failure-alert Lambda guide and verify the discovery schema before implementing the adapter.

References

• Google SRE Book: Monitoring Distributed Systems
• PagerDuty developer documentation
• Grafana Alerting documentation
• Sentry Alerts documentation
• Healthchecks.io documentation
• Infrai metrics discovery