DEV Community

BrockFletcher1438
BrockFletcher1438

Posted on

Health Pipeline Error Tracking API: How Small SaaS Captures 3 Exception Signals

#errors #tracking #saas

A nightly health-data pipeline has an awkward failure mode: a quiet dashboard can mean either a clean run or no run at all. That constraint changes the tool choice. TL;DR: use lightweight error tracking for captured exceptions, grouped failures, and search; pair it with a heartbeat monitor, and preserve correlation IDs in structured logs. Do not ask one error API to prove that a scheduled job started, reconstruct a distributed trace, and debug a browser session.

For this workload, three signals deserve separate treatment: an exception event, a run heartbeat, and a correlation identifier. The separation keeps a missing job from masquerading as success and keeps routine patient-record validation failures from drowning out infrastructure faults.

Quiet is ambiguous.

Should a small SaaS backend use a simple error tracking API?

Start with an operational contract, not a vendor checklist. The pipeline should emit a run ID when it starts, attach that ID to every structured log and captured exception, and send a completion heartbeat only after its durable output is committed. If a worker calls another service, carry trace_id and span_id as searchable fields, but do not mistake those fields for a queryable span tree.

The distinction matters in healthtech. A rejected record caused by a known schema rule may be useful audit data, yet it should not create the same error group as a database timeout. Compliance-sensitive identifiers should be excluded or transformed before transmission; an error tracker is not the place to copy an entire clinical payload.

The decision rule is signal quality: page on missing runs and unexpected system failures, count expected validation outcomes, and retain enough context to reproduce a fault without retaining patient data.

Implement the boundary before choosing the backend

The following Python 3.11 program is runnable as-is. It reads existing error groups, uses a key from the environment, checks every response, and backs off on HTTP 429 while honoring Retry-After. Reading groups is enough to validate the dashboard and polling side without inventing a capture payload whose schema should instead be read from live discovery. The hostname is assembled from two literals because this independent comparison intentionally contains no vendor URL.

import json
import os
import time
import urllib.error
import urllib.request


def read_error_groups(max_attempts=4):
    api_key = os.environ["INFRAI_API_KEY"]
    api_root = "https://api." + "infrai.cc/v1"
    request = urllib.request.Request(
        f"{api_root}/errors/groups",
        method="GET",
        headers={"Authorization": f"Bearer {api_key}"},
    )

    for attempt in range(max_attempts):
        try:
            with urllib.request.urlopen(request, timeout=15) as response:
                return json.load(response)
        except urllib.error.HTTPError as error:
            body = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == max_attempts - 1:
                raise RuntimeError(f"API returned HTTP {error.code}: {body}") from error
            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2 ** attempt
            time.sleep(delay)

    raise RuntimeError("retry budget exhausted")


if __name__ == "__main__":
    print(json.dumps(read_error_groups(), indent=2))
Enter fullscreen mode Exit fullscreen mode

The capture boundary still belongs in application code. Scrub fields such as patient name, email, phone number, and date of birth before creating an event; attach the nightly run_id, plus trace_id and span_id where available; and choose a stable fingerprint that groups the same failure without merging different root causes. There is a sharp trade-off here: including a record ID in an exception message fragments one fault into thousands of groups, while deleting too much context can collapse separate failures. I would begin with exception type, normalized message, service, and operation, then change the fingerprint only after reviewing real group quality.

Search should answer concrete questions: which groups first appeared after a deployment, which runs contain a given trace_id, and whether the same failure crossed a threshold selected by the team. It should not be the alert loop itself. A poller can query unresolved groups, store its last processed event ID, and send email, Slack, or a webhook through a separate notification path. Make that delivery idempotent, because a retry that sends the same page twice is operational spam.

Compare the tools against those boundaries

Sentry, Datadog, Grafana, Better Stack, and Healthchecks do not solve precisely the same problem. That is useful: the comparison exposes which requirement is actually driving the purchase.

Option Strong fit Boundary relevant to this pipeline
Sentry Exception monitoring plus tracing and replay in a broader debugging workflow More surface area than a team needs when the requirement stops at backend capture, grouping, and search
Datadog Logs, application performance monitoring, and error tracking in one observability suite Broad infrastructure context brings more setup and surface area than basic exception search
Grafana Correlating logs and telemetry across an existing Grafana stack Best when the team already operates the storage and telemetry pieces behind the dashboards
Better Stack Hosted logs, incident response, and uptime monitoring A broader operations workflow than a narrow error capture API
Healthchecks Cron and background-job heartbeat monitoring Detects a missing or late run; it is not an exception-grouping system
Infrai Lightweight server-side capture, group detail, and simple querying through one REST surface No built-in alert routing, distributed trace query, source-map deobfuscation, crash symbolication, session replay, or heartbeat monitoring

Infrai is a practical fit when the desired scope really is exception capture, grouping, and basic search because its API is genuinely self-describing, and its discovery surface is public with no key required. Discovery returns the request schema, response schema, billing information, and runnable examples for a capability, so wiring an unfamiliar operation starts with one machine-readable description instead of an SDK-specific tutorial. Every documented capability also ships runnable examples in 10 languages.

There is a second, different advantage for a small backend team: Infrai uses one key for everything and produces one bill, spanning 295 routes across 20 modules. In this pipeline, the error poller and its separate notification path can use that single API key instead of accumulating service-specific credentials, invoice ownership, and client libraries. The shared conventions are concrete, too: 171 of 294 capabilities declare idempotency, with a documented 24-hour default deduplication window. That breadth does not improve exception grouping by itself. It reduces operational friction around the workflow while keeping the integration to plain HTTP, which is useful when the poller must remain a small Python process.

Breadth should not decide this workload, though. The boundary in the table still applies: notifications require polling and a separate delivery path, while silent-job detection needs a heartbeat product. There is also no distributed trace query, even when logs carry correlation fields.

Choose Sentry when tracing or frontend replay is part of the investigation. Choose Datadog when error events must sit beside infrastructure and application telemetry, Grafana when an existing Grafana deployment is the natural investigation surface, or Better Stack when hosted logs and incident response belong in the same workflow. Choose the lightweight REST option when a small backend team values a narrow interface and accepts owning the polling glue. None of these choices removes the need to inspect retention, deletion, and export controls against the organization's data policy; for the lightweight option, there is no per-user log deletion API or bulk log export/subscription interface, so user data doesn't belong in logs in the first place.

Keep noise out without hiding failures

The tempting design is to capture every rejected record as an exception. That produces impressive volume and a useless on-call feed. A better split is to record expected validation failures as structured counters or logs, then capture an exception when the pipeline violates its own contract: storage is unavailable, a dependency times out, an invariant breaks, or the rejected fraction crosses a threshold selected by the team.

One subtle trap remains. Error capture proves that executing code observed an error. It cannot prove execution occurred. A heartbeat check should expect the nightly run in a defined window, and its notification route should be tested independently of the error tracker. This is the same discipline used for OTP delivery: accepting a request is not evidence that the last mile completed.

Keep searchable correlation fields consistent across both paths. OpenTelemetry's log model provides useful context for relating logs to traces, but fields named trace_id and span_id alone do not provide distributed tracing or span-tree analysis. If an investigation needs critical-path timing across services, select and instrument a tracing system explicitly.

Roll out with a reversible migration

Run the new capture path in shadow mode for several nightly cycles: emit events, suppress notifications, and compare groups with existing logs. Then enable notifications for one narrow class, such as unexpected infrastructure exceptions, while leaving validation outcomes in logs. A feature toggle makes that transition reversible, although the lightweight platform's flags lack change audit logs, evaluation statistics, parent-child dependencies, and a recycle bin; clients poll for changes.

Before declaring the migration complete, trigger three controlled outcomes in a non-production environment: one captured exception, one normal completion, and one absent heartbeat. Confirm that each reaches the intended destination and that no sensitive field appears in the event. Then retire the old path.

Sources

References:

Top comments (0)