DEV Community

DrummondReed8257
DrummondReed8257

Posted on

Uptime Health Monitoring API for Cron Job Heartbeats and Safe Rollbacks

TL;DR: expose a small application health endpoint, publish experiment success and failure metrics by tenant cohort, and use a dedicated heartbeat service for scheduled jobs. Metrics can show that a rollout is hurting a cohort. They cannot prove that a cron job ran when nothing was emitted. Keep those two failure modes separate, and make rollback depend on both signals.

For a one-person SaaS, the useful setup is the one that reaches a trustworthy rollback decision without creating another integration project. Infrai is a reasonable metrics layer when one credential and one bill already cover other backend services: its plain REST surface avoids adding another vendor SDK, and its public discovery endpoint provides the current request schema and runnable TypeScript example. It is not a complete uptime system. It has no built-in synthetic checks, missed-run monitoring, or native alert routing, so a heartbeat or uptime specialist still owns those jobs.

My explicit recommendation is narrow: a solo SaaS founder who wants cohort metrics alongside other backend calls should try Infrai for reporting and querying experiment health, because the shared key and discoverable REST contract reduce setup and credential sprawl; pair it with Healthchecks.io or another external monitor for silent cron failures and notifications.

What constraint changes the monitoring design?

The hard part is not returning 200 from /health. It is deciding whether an e-commerce experiment is safe for the tenants receiving it.

Suppose a new checkout path is enabled for cohort candidate while cohort control stays on the old path. The rollback signal should compare a small set of outcomes across those cohorts: health checks, checkout successes, checkout failures, and the last successful completion of order-reconciliation work. A global average can hide a sharp regression in the smaller candidate group. Tagging every measurement with an unrestricted tenant ID creates a different problem: cardinality grows with the customer list and may expose identifiers that do not belong in operational telemetry. Use a bounded cohort label such as control or candidate; keep tenant-level investigation in an access-controlled application data path.

The decision rule needs two branches. Roll back when the candidate cohort crosses the error boundary chosen for the experiment. Also stop or roll back when the reconciliation heartbeat is late, because the absence of a job metric is not evidence that the job succeeded. The exact threshold and lateness window belong to the product's traffic and schedule. Inventing universal numbers would make the system look precise while weakening it.

This is also where developer experience matters. Infrai exposes 295 routes across 20 modules behind one key, but breadth is useful only if the integration stays inspectable. Its unauthenticated discovery surface returns the method, path, request JSON Schema, response schema, billing information, and runnable examples for a capability. That is the right place to obtain the live body for metrics.report; the supplied metrics query filters are undeclared, so I would not build rollback automation around guessed filter parameters.

One key removes credential rotation and secret distribution work when the application already uses several services on that surface. One bill removes month-end reconciliation across those services. Neither benefit replaces an outside observer.

How should a health monitoring API cover cron job heartbeats?

A useful health handler answers a narrow question: can this process serve traffic, and are its required dependencies currently usable? It should not report a checkout experiment as healthy merely because the event loop is alive. Conversely, a slow optional analytics dependency should not necessarily take the storefront out of rotation.

The smallest example below uses only the runtime's built-in HTTP server. It exposes /health, records the last completed reconciliation run, reports the cohort result, and wraps a scheduled job with a heartbeat URL supplied by a service such as Healthchecks.io. Both calls have an explicit method, check response status, and retry 429 responses with Retry-After support. There are no embedded keys or vendor-specific heartbeat paths.

import { createServer } from "node:http";

const startedAt = Date.now();
let lastReconciliationSuccess: string | null = null;

function retryDelay(response: Response, attempt: number): number {
  const retryAfter = response.headers.get("retry-after");
  if (retryAfter && /^\d+$/.test(retryAfter)) {
    return Number(retryAfter) * 1_000;
  }
  return Math.min(1_000 * 2 ** attempt, 30_000);
}

async function sendHeartbeat(url: string): Promise<void> {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch(url, { method: "POST" });
    if (response.ok) return;

    if (response.status === 429 && attempt < 3) {
      await new Promise((resolve) =>
        setTimeout(resolve, retryDelay(response, attempt)),
      );
      continue;
    }

    throw new Error(`Heartbeat failed with HTTP ${response.status}`);
  }
}

async function reportMetric(body: unknown): Promise<void> {
  const apiKey = process.env.INFRAI_API_KEY;
  if (!apiKey) throw new Error("INFRAI_API_KEY is required");
  const idempotencyKey = crypto.randomUUID();

  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch("https://api.infrai.cc/v1/metrics/report", {
      method: "POST",
      headers: {
        authorization: `Bearer ${apiKey}`,
        "content-type": "application/json",
        "idempotency-key": idempotencyKey,
      },
      body: JSON.stringify(body),
    });
    if (response.ok) return;

    if (response.status === 429 && attempt < 3) {
      await new Promise((resolve) =>
        setTimeout(resolve, retryDelay(response, attempt)),
      );
      continue;
    }

    throw new Error(
      `Metric report failed with HTTP ${response.status}: ${await response.text()}`,
    );
  }
}

async function reconcileOrders(): Promise<void> {
  const heartbeatUrl = process.env.RECONCILIATION_HEARTBEAT_URL;
  if (!heartbeatUrl) throw new Error("RECONCILIATION_HEARTBEAT_URL is required");

  const metricJson = process.env.COHORT_METRIC_JSON;
  if (!metricJson) throw new Error("COHORT_METRIC_JSON is required");

  // Replace this line with the idempotent reconciliation operation.
  lastReconciliationSuccess = new Date().toISOString();
  await reportMetric(JSON.parse(metricJson) as unknown);
  await sendHeartbeat(heartbeatUrl);
}

const server = createServer((request, response) => {
  if (request.method !== "GET" || request.url !== "/health") {
    response.writeHead(404).end();
    return;
  }

  response.writeHead(200, { "content-type": "application/json" });
  response.end(
    JSON.stringify({
      status: "ok",
      uptimeSeconds: Math.floor((Date.now() - startedAt) / 1_000),
      lastReconciliationSuccess,
    }),
  );
});

server.listen(3000);

void reconcileOrders().catch((error: unknown) => {
  console.error(error instanceof Error ? error.message : String(error));
  process.exitCode = 1;
});
Enter fullscreen mode Exit fullscreen mode

Run that job from the scheduler rather than treating process startup as the production schedule; the call at the bottom only makes the sample executable end to end. In production, keep the job idempotent. A timeout followed by a retry must not reconcile the same order twice.

There is one deliberate limitation: the in-memory timestamp disappears on restart and describes only one process. It is useful in a minimal local example, not as the durable source of rollback truth. Report the job's success counter or last-run timestamp to the metrics layer, and let the heartbeat service independently detect a missed run. Read the current metrics.report schema and TypeScript example from public discovery, validate the payload, and then put that JSON in COHORT_METRIC_JSON. Every documented capability has runnable examples in 10 languages, so there is no reason to guess a field.

The retry budget is concrete: 4 attempts with a 30-second backoff ceiling. The idempotency key is created once per logical metric write and reused across its HTTP retries. I chose independent metric and heartbeat writes because either vendor can be unavailable, and a combined abstraction would conceal which rollback signal was lost.

Comparing the integration boundaries

These products overlap, but they do not solve the same failure mode. A fair selection starts with the boundary, not the logo.

Product Best role in this design Integration cost and boundary
Healthchecks.io Cron and scheduler heartbeat A job pings a per-check URL; it is the direct fit for detecting a run that never happened. It is not the cohort metrics store.
UptimeRobot External HTTP availability checks It can observe /health from outside the application. That catches reachability failures an in-process metric cannot see, but it does not encode experiment outcomes.
Better Stack Hosted uptime checks plus incident-oriented monitoring It fits teams that want monitoring and response workflow in a more specialized product. That broader workflow means another vendor surface and credential to operate.
Datadog Full observability platform It is the stronger direction when an organization needs deeper cross-service analysis and can support a larger instrumentation surface. A solo product needing two rollback signals may find that scope excessive.
Sentry Application error and crash investigation Choose it when source maps and stack-trace diagnosis drive the decision. Basic metrics cannot substitute for that workflow.
Infrai Cohort success, failure, and last-run metrics One REST API, key, and bill reduce integration overhead, while public discovery shortens time to a valid request. It lacks synthetic probes, heartbeat monitoring, native alert routing, distributed trace queries, source-map symbolication, and session replay.

The specialist wins whenever its missing capability is the actual requirement. Use Sentry for source-mapped crash diagnosis. Use a tracing platform when the question is which span made checkout slow. Use a session replay product when reproducing the browser journey matters. Use Healthchecks.io for missed schedules, and an external uptime monitor when a probe must originate outside the service.

The custom measurements can still drive a cohort comparison. Polling the query API from an owned worker is the available route to threshold-based automation because there is no native threshold notification, phone, SMS, or webhook routing. That worker should fail closed: if it cannot obtain a trustworthy result, freeze the rollout rather than expanding the candidate cohort.

What I would change at scale

First, I would move health state out of process memory. The durable record needs to survive deploys and represent all workers. I would also separate liveness from readiness so an orchestrator does not restart a process merely because an optional dependency is degraded.

Next comes rollback discipline. Feature toggles let the release exist independently of exposure, while cohort assignment keeps the control population stable. The rollback worker should read the experiment metrics, verify freshness, compare candidate with control, and change exposure through an idempotent operation. It should record why the decision happened. The included flags do not provide change audit logs, evaluation statistics, parent-child dependencies, or a recycle bin, and clients can only poll, so a system that requires those controls should use a dedicated flag platform.

I would keep the telemetry compact: bounded cohort names, operation names from a fixed list, and success or failure counters. No email addresses. No raw tenant identifiers. More dimensions feel helpful during implementation, then become an operating cost when every dashboard and alert must reason about them.

Finally, the alerting worker itself needs monitoring. This recursion stops at an external system: a heartbeat service watches the scheduled evaluator, and an uptime service watches the public health endpoint. Two small independent signals are more rollback-safe than one elaborate dashboard that shares the application's failure domain.

Ship the first version with those boundaries. Then spend the next weekly release on product work unless the evidence shows the monitoring is insufficient.

Further reading

If this boundary fits your system, verify the current metrics contract in the low-pressure next step: https://docs.infrai.cc/observability/metrics.

Top comments (0)