IronspireDraven77

Posted on Aug 21

Edtech Delivery — Pingdom and UptimeRobot Endpoints, Healthchecks Cron Heartbeats

#sre #observability #monitoring

Short answer: use Pingdom or UptimeRobot to page on public endpoint failure, use Healthchecks to catch missing cron heartbeats, and keep internal logs, errors, and metrics for the evidence needed to decide whether an edtech notification release should be rolled back.

Don't ask one signal to do three jobs. A green /healthz response doesn't prove that a lesson reminder reached its delivery provider, a log record doesn't page anyone by itself, and a worker that has silently stopped leaves no error to collect. For a small SaaS, the useful design is a short chain: an outside observer detects reachability, a dead-man's switch detects absent scheduled work, and application telemetry explains what changed.

The page must have an owner and a rollback decision attached to it. Otherwise it's dashboard furniture.

Start with the page that should fire

For a notification service, “uptime” is too broad to be an actionable condition. The public API can answer while a queue consumer is stalled; a cron scheduler can run while every delivery attempt is rejected; the application can emit rich error events while nobody is listening for them. The first runbook question is therefore blunt: what page fired? Its source tells the responder which claim has actually failed.

An endpoint monitor establishes that an external client can reach a chosen URL and receive the response your service defines as healthy. A heartbeat monitor establishes something different: a scheduled job checked in within its expected window. Internal telemetry records the failed probe, timeout, or worker exception and supplies context for investigation. These signals overlap a little, but none substitutes for the other two.

That separation matters during rollback. If only the deployment-specific readiness check changes state after a release, rollback is a defensible first move. If the cron heartbeat is missing but the public endpoint remains healthy, reverting the web process may add risk without restarting the scheduler or worker that actually stopped. If the page comes from a downstream delivery symptom, the responder needs release identity and error grouping before touching production.

Fast is good. Specific is better.

How should a small SaaS combine endpoint uptime monitoring and cron healthchecks?

Choose by failure mode, not by the number of charts in the product. Pingdom and UptimeRobot fit the public-check role in this design. Healthchecks fits the scheduled-job role. Treat the choice between the first two as an operational evaluation: run the same representative endpoint checks, then verify the notification path, escalation ownership, maintenance behavior, and the US or EU handling requirements that apply to your organization. I'm not sure a generic vendor label can settle residency or on-call fit; current contracts, regions, and product documentation have to settle those points.

Component	Signal it owns	Useful rollback evidence	What it cannot establish alone
Pingdom	Public endpoint check	The release boundary and first failed outside probe	Whether a scheduled worker checked in
UptimeRobot	Public endpoint check	The release boundary and first failed outside probe	Why an internal delivery attempt failed
Healthchecks	Missing cron or worker heartbeat	The last successful run around a deployment	Whether the public API is reachable
Internal telemetry	Failed probes, timeout errors, worker exceptions, availability summaries, and response timing summaries	Release ID, component, failure class, and correlated application context	Threshold evaluation or alert delivery unless you build it

Datadog, Grafana Cloud, and Better Stack also deserve evaluation when the team is choosing a broader monitoring stack rather than filling only these two detection gaps. Datadog is the candidate to test when an organization already centralizes managed monitoring there; Grafana Cloud is the candidate when the operating model already revolves around Grafana; Better Stack is a candidate when uptime and incident workflow are being assessed together. Those names don't remove the need to test the actual page path, current regional terms, and rollback evidence. A familiar dashboard with an untested notification route is still an untested notification route.

The catch is that none of those rows should become a universal recommendation. Stick with a dedicated endpoint service when immediate incident notification is the job, and keep Healthchecks-style monitoring when “the task should have run but didn't” is the risk. An internal observability API is not suitable as the only pager here because threshold evaluation and webhook, SMS, phone, or email delivery must be built outside it.

Infrai can be a reasonable internal-telemetry option when a team expects to add other backend capabilities because it combines one key and one bill with one REST API over pure HTTP, which any language or runtime can call without installing an SDK, while its public, self-describing discovery contract covers 295 routes across 20 modules and lets the notification worker and a small incident utility share request definitions instead of maintaining separate client integrations. For this incident path, though, that breadth doesn't turn logs or metrics into an external uptime checker or heartbeat monitor. The dedicated services still own detection and paging.

Build a rollback-safe health signal

A health endpoint used for rollback needs a narrow contract. It should answer whether this instance can accept notification work, identify the running release without exposing secrets, and avoid declaring the system healthy merely because the HTTP server started. Don't fold every distant dependency into one check: a volatile dependency can then turn a local readiness signal into a noisy global alarm. The exact dependency set is a service-level decision, but it must be written down before the first page.

Use /healthz for process liveness and /readyz for notification readiness. The release string in each response lets an outside probe be aligned with deployment records, while HTTP 503 gives the monitor an unambiguous failure response. In a real service, readiness should use the smallest checks that prove the process can safely accept delivery work.

Once the external page has fired, the following runnable Go utility retrieves internal error groups through the verified GET /v1/errors/groups route. It deliberately supplies no filters because that route's discovery parameters do not declare any. The utility uses an environment variable for the key, sets the method explicitly, honors an integer Retry-After on HTTP 429, applies bounded exponential backoff otherwise, and surfaces non-success response bodies instead of assuming a 200.

package main

import (
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        log.Fatal("INFRAI_API_KEY is required")
    }
    baseURL := strings.TrimRight(os.Getenv("OBSERVABILITY_API_BASE"), "/")
    if baseURL == "" {
        log.Fatal("OBSERVABILITY_API_BASE is required")
    }

    client := &http.Client{Timeout: 15 * time.Second}
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequest(
            http.MethodGet,
            baseURL+"/v1/errors/groups",
            nil,
        )
        if err != nil {
            log.Fatal(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            log.Fatal(err)
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            log.Fatal(readErr)
        }

        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            fmt.Println(string(body))
            return
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            log.Fatalf("request failed: status=%d body=%s", resp.StatusCode, strings.TrimSpace(string(body)))
        }

        delay := time.Second << attempt
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
            delay = time.Duration(seconds) * time.Second
        }
        time.Sleep(delay)
    }

    log.Fatal("request remained rate-limited after 5 attempts")
}

Run the utility with INFRAI_API_KEY set to the key from your environment and OBSERVABILITY_API_BASE set to the API base configured for the service. Keep the public probe outside the service's own network boundary, because a self-check can stay green through a routing or certificate problem that blocks students and instructors. The error-group response is investigation evidence, not the alert source and not a rollback command.

The cron side uses a different contract: the scheduled process sends its heartbeat only after the notification batch reaches the completion point the team has defined. Sending it at job start hides crashes halfway through the batch. Set the expected window from the actual schedule plus tolerated runtime variance, then record the job name and release alongside internal telemetry so a missing heartbeat can be matched to the deployed worker.

There is no clever shortcut here.

Preserve evidence without making telemetry the pager

When a probe or worker fails, record the application facts that will survive a rollback: release ID, component, failure class, operation name, and a correlation identifier that can connect related log and error records. Keep high-cardinality values such as student IDs out of metric labels; Prometheus's instrumentation guidance warns that each unique label set creates another time series. Sensitive identifiers also complicate deletion obligations, and the internal log surface described here has no per-user deletion route.

Metrics can hold availability percentages and response-time summaries, but somebody still has to evaluate thresholds and deliver the alert. Logs can carry trace_id and span_id for correlation, but that isn't a distributed trace query or span tree. There is also no source-map decoding, crash symbolication, Electron minidump parsing, or Session Replay in this path. Those are capability boundaries, and a team that needs them should select dedicated tooling rather than pretending a correlation field closes the gap.

This is where dashboards become dangerous. A dashboard is evidence after a human arrives; it is not proof that a human will arrive. The external endpoint and heartbeat services should own the initial page, while internal errors, logs, and metrics answer the next questions: did failures begin at the release boundary, are they isolated to the notification worker, and did they stop after rollback?

Verify the page, then rehearse rollback

Before calling the setup complete, deploy a harmless canary release and test each failure independently. First, make readiness false and confirm that the public checker observes 503, the intended page reaches the intended owner, and the alert identifies the endpoint and release. Restore readiness and verify recovery. Second, suppress one test job's completion heartbeat and confirm that the heartbeat alert names the job rather than the web endpoint. Third, emit a controlled application failure and verify that its release ID and correlation identifier are searchable in the internal evidence store without expecting that store to deliver the page.

The rollback rule should be short enough to use at 3 a.m.: roll back when a new release is the common boundary for a customer-facing check failure and the rollback itself is known to be safe. Pause when the signals disagree. A missing heartbeat isolated to an old worker, for example, calls for worker ownership and scheduler evidence; a blind web rollback may change nothing while erasing the clean comparison point.

After rollback, don't stop at a green endpoint. Confirm that the outside check recovered, the next scheduled heartbeat arrived, and notification failures stopped accumulating in internal telemetry. Preserve timestamps from first failure through recovery for the postmortem. If any of those checks remains red, the original change was not the whole cause, and the incident commander needs a new hypothesis rather than repeated reversions.

DEV Community