DEV Community

EmersonPrice3718
EmersonPrice3718

Posted on

Small SaaS Observability Stack for Health Endpoint Monitoring (Without Pager Alerts)

#observability #monitoring #python

Short answer: a small SaaS observability stack can use one store for health endpoint monitoring, logs, metrics, and errors, but a public service with US or EU availability commitments should put an external uptime monitor in front of it. A store that has no native threshold rules, notifications, or synthetic checks cannot be the system that wakes someone up. It can still be the system that explains, with evidence, why last night's logistics pipeline stopped moving consignments.

That distinction decides the architecture. I would accept a plain REST API for a small team's internal health history because it can receive signals without an SDK or client-library lifecycle, and because exceptions, logs, and availability metrics can live behind one interface. One credential across the platform also reduces secret rotation and configuration drift in the poller, ingestion worker, and investigation tooling. I would not call that a complete uptime stack. The missing alert path is a failure boundary, not a minor feature gap.

What should a small observability stack monitor at its health endpoint?

The workload is a nightly pipeline: it imports depot scans, validates shipment records, calls dependencies, and publishes a completion result. Its /health response can say the process is alive while the scheduled import never started. Green is ambiguous.

An architecture decision record therefore needs invariants, not a shopping list. These are mine:

  1. Every pipeline run has a stable run_id carried into structured logs and exception records.
  2. Health samples include an observation time and region, so a responder can distinguish an old result from a current one.
  3. Availability is measured outside the application process for public production endpoints.
  4. A failed notification system cannot erase the evidence needed for reconstruction.
  5. Duplicate samples and retried writes are expected; analysis must tolerate them.

The last invariant is easy to neglect. A polling worker can time out after its write succeeds, retry, and produce two observations for one check. Deduplicate on a client-generated observation ID or aggregate by the intended check timestamp. Otherwise a transient network failure can improve the apparent sample count while making the incident timeline less trustworthy.

There are also hard boundaries. Logs with trace_id and span_id can correlate records, but they do not create a distributed trace query or span tree. Error grouping is useful for recurring outage-causing exceptions, but there is no source-map decoding, crash symbolication, Electron minidump parsing, or Session Replay. A nightly job that never runs produces neither an exception nor a log, which is precisely why a dead-man's-switch service belongs outside this store.

The decision, compared on the failure path

The useful comparison is not feature count. It is which independent system still observes the service after the service, scheduler, or primary telemetry path fails.

Option Strong fit Incident-reconstruction value Boundary that matters here
Infrai Small internal systems wanting logs, metrics, and errors through one REST API under one key Central evidence with no SDK dependency; public self-describing discovery exposes request and response schemas before integration No native threshold rules, webhook/email/SMS/phone notifications, synthetic checks, or span-tree queries
Grafana Cloud Teams already using Prometheus-style metrics and Loki logs Broad dashboards and alerting can connect metric changes to logs More components and data-model decisions than a minimal health-history store
Better Stack Public HTTP services needing hosted uptime checks and incident notification External checks establish an independent availability timeline Nightly job completion still needs an explicit heartbeat or job signal
Healthchecks.io Cron jobs and pipelines where silence is the primary failure mode A missing completion ping directly represents “the task did not run” It is deliberately narrower than a unified log, metric, and error analysis system
Sentry Applications where grouped exceptions and release context dominate investigation Strong exception-centered diagnosis It is not, by itself, proof that an endpoint was reachable from US and EU vantage points

This is not a ranking. Grafana Cloud is the more natural choice when metrics, log querying, and alert rules already form the team's operating model. Better Stack is a credible external checker when HTTP reachability and notification delivery are central. Healthchecks.io maps unusually well to the silent nightly-run failure. Sentry earns its place when stack traces and error recurrence are the investigation's starting point.

The trade-off is blunt.

For the stated small-SaaS case, I would pair the unified store with Better Stack for public endpoint checks, or with Healthchecks.io when completion of the nightly import is the invariant. Keeping the evidence store and the alarm source independent is intentional: one damaged path should not blind both detection and diagnosis.

How does the critical path preserve reconstructable evidence?

The reader below calls Infrai's verified log-search route after a monitor reports a failure. It deliberately supplies no undocumented filters: the discovery metadata does not declare the filtering parameters for this route. Set INFRAI_BASE_URL to the service API origin and keep the key in INFRAI_API_KEY; a scheduler or external monitor still owns notification delivery. The client uses an explicit method, reports non-2xx bodies, and backs off on rate limiting.

#!/usr/bin/env python3
import json
import os
import time
import urllib.error
import urllib.request
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime


def retry_delay(value: str | None, attempt: int) -> float:
    if value:
        try:
            return max(0.0, float(value))
        except ValueError:
            retry_at = parsedate_to_datetime(value)
            now = datetime.now(retry_at.tzinfo or timezone.utc)
            return max(0.0, (retry_at - now).total_seconds())
    return min(2**attempt, 30)


def search_logs(base_url: str, api_key: str) -> object:
    url = f"{base_url.rstrip('/')}/v1/logs/search"
    for attempt in range(5):
        request = urllib.request.Request(
            url,
            method="GET",
            headers={
                "Accept": "application/json",
                "Authorization": f"Bearer {api_key}",
            },
        )
        try:
            with urllib.request.urlopen(request, timeout=10) as response:
                if not 200 <= response.status < 300:
                    raise RuntimeError(f"unexpected HTTP status {response.status}")
                return json.load(response)
        except urllib.error.HTTPError as exc:
            body = exc.read().decode("utf-8", errors="replace")
            if exc.code == 429 and attempt < 4:
                time.sleep(retry_delay(exc.headers.get("Retry-After"), attempt))
                continue
            raise RuntimeError(f"log search failed: HTTP {exc.code}: {body}") from exc
    raise RuntimeError("log search exhausted retries")


def main() -> int:
    result = search_logs(
        os.environ["INFRAI_BASE_URL"],
        os.environ["INFRAI_API_KEY"],
    )
    print(json.dumps(result, indent=2))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
Enter fullscreen mode Exit fullscreen mode

Run the health probe outside the application process, then use this reader during investigation. For the nightly pipeline, emit separate pipeline_started, pipeline_completed, and pipeline_failed records with the same run_id; a client-generated observation_id handles retry duplication, while the pipeline identifier ties the import stages to exceptions and dependency context. Metrics should answer trend questions such as duration and success ratio. Logs should answer which depot, dependency, or validation stage failed. Error groups should answer whether the exception is recurring. The platform's public discovery surface reports 295 capabilities across 20 modules, but breadth does not remove the need to verify each route's declared schema before sending data.

Do not quietly turn the log store into a compliance archive. Its limitations include no per-user log deletion API, bulk export or subscription API, and no exposed configuration entry point for retention or cold storage, so a system subject to GDPR erasure requirements needs a separately designed data-lifecycle path. That may disqualify it before observability features are discussed; choose a system with explicit deletion and export controls instead.

Failure boundaries and operating rules

A polling worker can approximate an internal alert by querying stored logs or metrics and then calling a separately operated notification channel. It is acceptable for a low-risk internal environment if the worker itself is monitored, query lag is bounded by policy, and missed polls are visible. It is circular for a public SLA: when the shared network or account path fails, the poller and the application may disappear together.

No amount of dashboarding repairs that correlation.

US and EU checks also need explicit semantics. Decide whether both regions must fail before paging, how long a failure must persist, and which endpoint proves useful service rather than process liveness. The Google SRE guidance on latency, traffic, errors, and saturation is a sound starting vocabulary, but a health endpoint alone usually covers only a thin slice of those signals. Record dependency state carefully; do not return secrets, customer identifiers, or raw exception messages from a public health route.

During reconstruction, begin with the externally observed failure window. Then query availability metrics for the trend, locate health and pipeline records by time and run_id, and inspect grouped errors for recurrence. If two independent regions failed while the application emitted no completion event, the evidence supports a different hypothesis than one regional probe failing while the pipeline completed normally.

Rejected option, and when it becomes correct

I reject the single-stack design in which the application writes health results to its own observability store, an internal worker polls that same store, and that worker is the only alarm source. It is compact, but its shared dependencies undermine the claim that it monitors availability. The design also cannot detect a nightly task that produces no signal unless some independent clock evaluates the absence.

There is a valid use case. For an internal staging pipeline with no pager requirement and operators who review a dashboard during business hours, the unified REST store alone is reasonable: it minimizes integration surface, preserves useful evidence, and avoids installing a client SDK. The decision changes as soon as missed overnight processing has a customer-facing deadline. At that point, use an external uptime checker for reachability and a heartbeat monitor for expected job completion, while retaining logs, metrics, and grouped errors for the slower and more demanding work of explaining the incident.

References

Top comments (0)