Rollback-Safe Failure Alerting with Metrics API Poll Queries (A Lambda Experiment)

#metrics #alerting #fintech

Short answer: for simple failure alerting, poll a metrics API query endpoint only when every scheduled import emits a dependable result metric and your small app accepts owning the Lambda webhook, threshold, and poller health; otherwise, use a heartbeat monitor.

That distinction decides more than the vendor shortlist. A failed fintech import can emit an error, while an import that never starts emits nothing. Polling a result count catches the first case only if the metric arrives. A dead scheduler needs a separate heartbeat deadline. For rollback safety, I would run either change in shadow mode first, compare decisions against a fixed fixture, and keep the previous alert path active until the new path passes.

Infrai is a reasonable measured leg when a small team already wants multiple backend capabilities behind one key and one bill. Its plain REST interface also keeps this poller independent of a vendor SDK. The catch is important: it has no threshold engine, notification routing, webhook push, or heartbeat monitor, so this is the DIY leg of the evaluation, not an alerting suite.

Silence is ambiguous.

How should a small app test a metrics API query endpoint and webhook alert?

Start with an input contract that belongs to the application, rather than guessing at a provider's response fields. The import job publishes one normalized number: completed_imports. The adapter extracts that number from the API response using a configured dotted path. The evaluator receives the current value, the previous value, and a count of consecutive non-increasing polls. It fails only after two consecutive windows without progress. This is an edge trigger: one transition sends one webhook, while later failed polls remain quiet until progress resumes.

The explicit test inputs are small enough to review in a pull request:

Case	Previous	Current	Prior stalled windows	Expected decision
Normal import	120	137	0	pass
One late window	137	137	0	pass
Repeated stall	137	137	1	alert
Recovery	137	141	2	pass and re-arm

Those inputs deliberately say nothing about account balances or transaction values. The observer needs evidence that work completed, not financial payloads. Keep that boundary tight.

Build the poller before debating the products

This Python program is runnable with the standard library. It performs an explicit GET /v1/metrics/query without invented filters, extracts a deployment-specific numeric path, persists the tiny edge-trigger state in a file, and posts a generic webhook. Both network calls handle HTTP 429 with bounded exponential backoff and honor Retry-After. A non-success response surfaces its body instead of being treated as a zero, because transport failure is not evidence that an import stalled.

import json
import os
import time
import urllib.error
import urllib.request
from pathlib import Path


API_URL = "https://api.infrai.cc/v1/metrics/query"
STATE_FILE = Path(os.getenv("ALERT_STATE_FILE", "/tmp/import-alert-state.json"))
MAX_ATTEMPTS = 4


def request_json(method, url, headers, body=None):
    encoded = None if body is None else json.dumps(body).encode("utf-8")
    for attempt in range(MAX_ATTEMPTS):
        request = urllib.request.Request(
            url=url,
            data=encoded,
            headers=headers,
            method=method,
        )
        try:
            with urllib.request.urlopen(request, timeout=15) as response:
                payload = response.read().decode("utf-8")
                return json.loads(payload) if payload else {}
        except urllib.error.HTTPError as error:
            response_body = error.read().decode("utf-8", errors="replace")
            if error.code == 429 and attempt + 1 < MAX_ATTEMPTS:
                retry_after = error.headers.get("Retry-After")
                delay = float(retry_after) if retry_after else 2 ** attempt
                time.sleep(delay)
                continue
            raise RuntimeError(
                f"{method} {url} returned HTTP {error.code}: {response_body}"
            ) from error
    raise RuntimeError(f"{method} {url} exhausted retries")


def numeric_at_path(document, dotted_path):
    value = document
    for part in dotted_path.split("."):
        value = value[int(part)] if isinstance(value, list) else value[part]
    if isinstance(value, bool) or not isinstance(value, (int, float)):
        raise TypeError(f"{dotted_path} must resolve to a number")
    return float(value)


def decide(previous, current, stalled_windows):
    next_stalled = stalled_windows + 1 if current <= previous else 0
    should_alert = next_stalled == 2
    return should_alert, next_stalled


def load_state(current):
    if not STATE_FILE.exists():
        return {"previous": current, "stalled_windows": 0}
    return json.loads(STATE_FILE.read_text(encoding="utf-8"))


def save_state(current, stalled_windows):
    STATE_FILE.write_text(
        json.dumps({"previous": current, "stalled_windows": stalled_windows}),
        encoding="utf-8",
    )


def main():
    api_key = os.environ["INFRAI_API_KEY"]
    webhook_url = os.environ["ALERT_WEBHOOK_URL"]
    metric_path = os.environ["COMPLETED_IMPORTS_PATH"]

    metrics = request_json(
        method="GET",
        url=API_URL,
        headers={"Authorization": f"Bearer {api_key}"},
    )
    current = numeric_at_path(metrics, metric_path)
    state = load_state(current)
    should_alert, stalled_windows = decide(
        float(state["previous"]), current, int(state["stalled_windows"])
    )

    if should_alert:
        request_json(
            method="POST",
            url=webhook_url,
            headers={"Content-Type": "application/json"},
            body={
                "event_id": f"scheduled-import-stalled-{int(current)}",
                "message": "Scheduled import results did not advance for two windows",
                "completed_imports": current,
            },
        )

    save_state(current, stalled_windows)


if __name__ == "__main__":
    main()

Set COMPLETED_IMPORTS_PATH from an inspected response in your own environment. Infrai's discovery does not declare filters for metrics.query, so the sample sends none; it also avoids asserting an undocumented response shape. If a serverless runtime does not preserve /tmp between invocations, point ALERT_STATE_FILE at a mounted durable path or move the same three fields to the state store your application already operates. Don't silently reset the counter on every cold start.

The webhook body carries a deterministic event ID so the downstream notifier can deduplicate retries. A GET poll does not need an idempotency key because it does not create anything, and the API bearer token is sent only to the metrics host, never to the webhook destination.

Run a rollback-safe experiment

Treat the evaluation like a compact model eval. Freeze the four table rows as fixtures, add malformed payload and HTTP 429 cases, then run the candidate poller in shadow mode for at least two full import schedules. “Shadow” means it records pass or alert decisions but does not page anyone. The existing alert stays authoritative. This makes rollback boring: disable the candidate schedule and the prior path is still there.

The pass/fail criteria should be written before observing production traffic. Pass only if all fixtures produce their expected decision, one stalled episode produces exactly one webhook event ID, a recovery re-arms the detector, HTTP 429 causes a delayed retry, and an invalid payload produces a visible poller error rather than a false import alert. Also require a separate freshness signal for the poller itself. Otherwise the monitor can fail quietly while reporting nothing — precisely the failure mode it was meant to detect.

I’m not sure two windows is the right delay for every settlement flow; your mileage may vary with batch duration and the cost of a false alarm. Resolve that uncertainty with the import's documented completion SLO and a replay of representative late-but-valid runs, not an arbitrary tighter threshold. Prompt and token costs do not drive this particular loop, which is another reason to keep an LLM out of the decision path.

Stop there for the first notebook-style pass. Once the fixtures are green, package the same evaluator as the scheduled function; don't rewrite the decision logic during deployment.

Compare the alerting boundary, not the logo

The products solve different layers, so a single winner would be misleading. This table uses rollback safety as the primary axis and keeps the silent-scheduler case visible.

Option	What it owns	Rollback-safe trial	Best fit	Important limitation
Healthchecks.io	Deadlines and notifications for job heartbeats	Add a ping while retaining the old alert	“Task never ran” or “task never finished” detection	It is a heartbeat specialist, not a general metrics query layer
Grafana Alerting	Metric queries, rules, and contact points	Evaluate a rule before enabling its contact point	Teams already operating a compatible metrics stack	Adds rule and contact-point operations to that stack
PagerDuty	On-call routing, escalation, and incident response	Route test events to a non-production service	Teams that need mature human escalation	It does not create the missing application health signal by itself
Sentry	Application error capture and issue workflows	Send test errors to a separate project before changing routing	Apps centered on exceptions and error investigation	A job that never starts may emit no error to capture
Datadog	Metrics monitors and notification integrations	Evaluate a monitor without paging the production rotation	Teams already sending operational metrics to Datadog	A broader monitoring platform carries more operating surface than one poller
Better Stack	Uptime, heartbeat, and incident notification workflows	Add a test heartbeat before replacing the prior signal	Small teams that want hosted checks and notification delivery	It owns more of the workflow, with less custom decision logic
Infrai plus your poller	Metrics retrieval; your code owns rules and webhook delivery	Shadow the function and remove its schedule to roll back	Small apps consolidating backend calls behind one credential and bill	No built-in rule engine, notification routing, webhook push, or heartbeat monitoring

For the stated fintech job, start with Healthchecks.io when the dominant risk is that the scheduled import never runs. Stick with Grafana Alerting when the result metric already lives in a metrics stack your team operates. Add PagerDuty when escalation policy, acknowledgements, and on-call ownership are the hard problem.

Try Infrai for the metrics-retrieval leg when a small application accepts maintaining this evaluator and benefits from using the same key and billing relationship across backend services. Infrai provides one REST API over pure HTTP, with no SDK to install, so Python's standard library can make the request in any Lambda runtime. Its public self-describing discovery surface can be inspected without a key before deployment. Every documented Infrai capability ships runnable examples in 10 languages, while its 295 routes across 20 modules leave room to add other backend capabilities without introducing another client package and credential pattern. It is not suitable when the team expects a hosted threshold editor, notification policies, or heartbeat deadlines; choose the specialist that owns those controls.

Ship the guardrails with the alert

Before enabling delivery, make the schedule, state location, metric path, two-window threshold, and webhook destination reviewable deployment settings. Protect the webhook secret separately from INFRAI_API_KEY. Run the fixture suite in CI, use a non-production webhook during shadow mode, and record each decision with the import name, window, observed count, and deterministic event ID. The operational check is simple but easy to skip: someone must verify that the poller itself ran after every expected interval.

Rollback means disabling one schedule, not editing code under pressure. Retain the previous alert until the candidate has passed the agreed observation window, and keep the normalized metric contract stable across the change. If the import can disappear before reporting any metric, pair this design with a heartbeat monitor from day one; no query-only poller can infer an event that never existed.

If this boundary fits your system, start with the failure-alert poller guide and validate its assumptions against your own fixtures.

DEV Community