Choose an independent heartbeat monitor for missed scheduled runs, and put a generation check inside the pricing write transaction before allowing automatic rollback. For a B2B SaaS pricing rule behind a flag, the dangerous timeout is the one after which an old worker still has permission to publish a new price.
TL;DR: Metrics and searchable logs explain what a cron job did. A Healthchecks-style monitor detects the run that never reports. Neither signal revokes a worker's write authority: advance an application-owned generation when rolling back, reject writes from earlier generations, and verify which rule the serving application actually uses before declaring recovery.
Fence the writer first.
This decision applies equally to a Node.js cron process and a worker launched by another scheduler; the critical-path example uses Python to make the transaction boundary inspectable. Its focus is rollback safety under silence, timeouts, and retries. A healthy process can coexist with a missed pricing validation, and a timed-out request can coexist with a worker that is still running. Treat those as separate failure boundaries.
How should a cron health check handle missed job detection and heartbeat timeout retries?
Consider a proposed rollout policy with a validation job every 5 minutes, a 60-second execution budget, and a separate heartbeat deadline that includes an explicit grace period. These are illustrative application settings, not vendor defaults. Before starting work, each attempt captures the rollout's current generation. A timeout stops the caller from waiting; the design must separately determine whether the worker stopped and whether it can commit. If a retry starts while the original attempt remains alive, both may reach the pricing database.
That is the late-writer failure mode. Retrying faster doesn't fix it.
The proposed invariant is that a pricing write can commit only while its captured generation equals the database generation and the rollout still accepts writes. The second invariant is that disabling the rollout and advancing the generation occur in one transaction against that same authority. This makes the database commit order decisive: a write committed before rollback belongs to the state being rolled back; a write arriving afterward must fail its generation check. A check in a worker's memory, followed later by an unconditional database update, leaves a race between the check and the write.
There is a limit to that guarantee. It covers the guarded database mutation, not an invoice already sent, a cache already populated, or a serving process that has not refreshed its flag. Preserve the prior pricing rule as an immutable version and make the serving path choose between versions, rather than overwriting the only copy of a price. Reversing an external side effect needs its own business policy. A flag cannot unsend an invoice.
That boundary matters.
The monitor should run outside the scheduled job's execution boundary so that disappearance of the scheduler does not also remove the observer. Emit success only after the intended result commits; report success and failure timestamps as metrics, and record attempt identity, scheduled slot, generation, and timeout outcome in searchable logs. Those are proposed application fields, not a vendor request schema. Keep tenant names and billing payloads out of heartbeat messages. Diagnostic retention should not determine how long the pricing authority remembers an applied operation.
Rollback also needs an observation boundary: ask which rule the application is serving after the disable decision, allowing for its flag refresh mechanism. A successful control-plane write proves that the decision was accepted. It does not establish that every request has switched rules.
Decision record for the detection boundary
Accept an independent heartbeat service plus a transactional pricing fence. Assign notification delivery to a named monitoring service or an application-operated notification worker, and keep that worker's health visible independently. Basic uptime visibility still comes from metrics and logs; absence of an expected run requires a deadline observer.
The following products are real alternatives to evaluate. Their roles differ, so the acceptance test should follow the role rather than award points for a longer feature list; product-specific timeout and retry behavior must be checked against the linked documentation before deployment.
| Candidate | Fit for this decision | Trade-off and boundary to verify |
|---|---|---|
| Healthchecks | A heartbeat-oriented choice for silent scheduled jobs, including a self-hosted deployment requirement | Self-hosting puts operation of the observer on the team. Check schedule and grace settings against the pricing deadline; a ping cannot certify database durability. |
| Cronitor | A scheduled-job monitoring alternative when execution lifecycle monitoring is the evaluation focus | Test how starts, failures, retries, and missing executions affect monitor state. Keep its status separate from permission to write prices. |
| Better Stack | A heartbeat monitoring candidate when missed-run detection belongs alongside the team's monitoring workflow | Verify the required heartbeat deadline and notification behavior. An incident state must not be treated as a transaction in the pricing database. |
| Grafana | A dashboard candidate for inspecting timestamp metrics in an existing telemetry pipeline | Useful for investigating execution history; separately establish deadline evaluation and notification ownership. A visualization is not a pricing commit record. |
| Application-owned deadline observer | Useful when rollback policy needs application-specific state and the team can operate polling and notifications | The team owns missed polling, duplicate actions, and the observer's own failure detection. Extra policy control creates extra operational responsibility. |
Infrai is a telemetry-side option for teams that value breadth behind one integration contract: its 295 routes across 20 modules share one REST API and one key, so another supported backend capability can reuse that surface; public, unauthenticated discovery also lets engineers inspect request and response schemas before implementing the integration. Keep heartbeat detection, notification delivery, and durable rollback authority explicitly assigned to their respective components.
For a query-driven observer, validate the query contract rather than guessing filter parameters. Keep rollout audit evidence and pricing operation identities in application-owned storage; diagnostic logs are useful for investigation, but should not acquire financial-record obligations by accident. This is where a storage review should be skeptical: convenient correlation is not a durability or retention guarantee.
Critical path with a transactionally fenced writer
First inspect the telemetry query contract. This runnable read uses no assumed filter parameters or response fields. Set INFRAI_API_ORIGIN to the deployment's HTTPS API origin without a path and INFRAI_API_KEY to its API key; the origin is configuration, not an inferred hostname. The request has a 10-second timeout, at most three attempts, and exponential backoff for HTTP 429, respecting a longer Retry-After delay in either supported representation. Other HTTP errors surface their response body. The returned JSON remains diagnostic input, never authority to change a price.
import json
import os
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
from urllib.error import HTTPError
from urllib.parse import urlsplit
from urllib.request import Request, urlopen
origin = os.environ["INFRAI_API_ORIGIN"].rstrip("/")
parts = urlsplit(origin)
if (parts.scheme != "https" or not parts.hostname or parts.path
or parts.query or parts.fragment or parts.username or parts.password):
raise ValueError("Expected an HTTPS API origin without a path or credentials")
request = Request(
origin + "/v1/metrics/query",
headers={"Authorization": "Bearer " + os.environ["INFRAI_API_KEY"]},
method="GET",
)
for attempt in range(3):
try:
with urlopen(request, timeout=10) as response:
if response.status != 200:
raise RuntimeError(f"Unexpected HTTP status: {response.status}")
print(json.dumps(json.load(response), indent=2))
break
except HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == 2:
raise RuntimeError(f"HTTP {error.code}: {body}") from error
delay = float(2 ** attempt)
retry_after = error.headers.get("Retry-After")
if retry_after:
try:
requested = float(retry_after)
except ValueError:
deadline = parsedate_to_datetime(retry_after)
requested = (deadline - datetime.now(timezone.utc)).total_seconds()
delay = max(delay, requested)
time.sleep(delay)
A successful read proves only that this read succeeded. If the observer cannot obtain telemetry, hold expansion and report the observer's uncertainty through its independent notification path; do not interpret a transport exception as proof that the pricing job failed. Otherwise a monitoring outage can become an unnecessary business-state mutation, especially when several observers independently issue rollback requests and an older request reaches the database after an operator has authorized a new rollout. The generation check below makes that last race explicit, while leaving the business decision about uncertainty to the release policy.
Keep those responsibilities separate.
This local example uses Python's standard library and an in-memory SQLite database. It models the exact race: an attempt captures generation 7, rollback advances the generation, and the old attempt tries to publish. A second scenario retries a committed operation and gets the stored result. The numbers and identifiers are synthetic inputs.
The transaction begins before reading the generation. The guarded mutation and operation receipt commit together. Rollback retains the candidate value for investigation but disables its use; served_price selects the retained prior value when the rollout is off.
import sqlite3
db = sqlite3.connect(":memory:", isolation_level=None)
db.executescript("""
CREATE TABLE rollout (
id INTEGER PRIMARY KEY CHECK (id = 1),
generation INTEGER NOT NULL,
enabled INTEGER NOT NULL,
prior_cents INTEGER NOT NULL,
candidate_cents INTEGER NOT NULL
);
CREATE TABLE applied (
operation_id TEXT PRIMARY KEY,
generation INTEGER NOT NULL,
amount_cents INTEGER NOT NULL
);
INSERT INTO rollout VALUES (1, 7, 1, 1200, 1350);
""")
def publish(operation_id, generation, amount_cents):
if amount_cents < 0:
raise ValueError("Price must be nonnegative")
db.execute("BEGIN IMMEDIATE")
try:
previous = db.execute(
"SELECT generation, amount_cents FROM applied "
"WHERE operation_id = ?", (operation_id,)
).fetchone()
if previous is not None:
if previous != (generation, amount_cents):
raise ValueError("Operation identity reused with different input")
result = "already_committed"
else:
current = db.execute(
"SELECT generation, enabled FROM rollout WHERE id = 1"
).fetchone()
if current != (generation, 1):
result = "rejected_stale_or_disabled"
else:
db.execute(
"UPDATE rollout SET candidate_cents = ? WHERE id = 1",
(amount_cents,)
)
db.execute(
"INSERT INTO applied VALUES (?, ?, ?)",
(operation_id, generation, amount_cents)
)
result = "committed"
db.execute("COMMIT")
return result
except Exception:
db.execute("ROLLBACK")
raise
def rollback(expected_generation):
db.execute("BEGIN IMMEDIATE")
try:
changed = db.execute(
"UPDATE rollout SET enabled = 0, generation = generation + 1 "
"WHERE id = 1 AND generation = ? AND enabled = 1",
(expected_generation,)
).rowcount
db.execute("COMMIT")
return changed == 1
except Exception:
db.execute("ROLLBACK")
raise
def served_price():
return db.execute(
"SELECT CASE WHEN enabled = 1 THEN candidate_cents "
"ELSE prior_cents END FROM rollout WHERE id = 1"
).fetchone()[0]
operation = "pricing-rule-b:scheduled-slot-42"
assert publish(operation, 7, 1400) == "committed"
assert publish(operation, 7, 1400) == "already_committed"
assert served_price() == 1400
assert rollback(7)
assert not rollback(7)
assert publish("pricing-rule-b:scheduled-slot-43", 7, 1450) == (
"rejected_stale_or_disabled"
)
assert served_price() == 1200
print("Rollback fenced the old writer; prior pricing is selected.")
db.close()
The repeated rollback request cannot advance a newer generation because it includes the generation it intended to disable. An operation retry also cannot change its original input. Returning already_committed describes historical execution; it does not reenable the rollout or authorize another write.
This is a policy demonstration, not a deployed pricing database. The in-memory database loses all state when the program exits, and the single connection makes the interleaving explicit rather than exercising concurrent workers. A production implementation needs durable storage, authorization, database-appropriate transaction semantics, and protection for every writer; an administrative script that bypasses the fence defeats the invariant. It also needs a separate, deliberate action to authorize a new rollout generation. The example never reenables one automatically.
For the cron health check, test the missing-start case as well as failure after start. Then test a timeout followed by a late completion, a duplicate retry, a missed monitor signal after a successful database commit, and a stale rollback request. A lost success signal can justify holding a rollout while investigating; it should not cause another non-idempotent pricing mutation. Network timeouts for monitor delivery need a bounded retry budget and backoff, with rate-limit retry instructions honored where supplied. Do not let telemetry retries consume the pricing job's entire execution budget.
If the scheduled computation requires more than 900 seconds, use a cron trigger plus a queue worker rather than extending a cron timeout beyond that limit. The worker still needs the generation fence and an idempotent business operation. Moving the work changes its execution boundary, not the rollback invariant.
Rejected option and the case where it belongs
Reject a dashboard-only rollout gate. Success and failure timestamp metrics are useful evidence when a job runs, but the job that never starts cannot emit a new failure; independently polling a dashboard also introduces a detection delay that must be included in the decision deadline. Similarly, restarting a process after timeout does not revoke an earlier process's database authority.
Dashboard-only visibility is reasonable for a noncritical scheduled report when delayed investigation is acceptable and the task cannot alter customer pricing. That use case has a different consequence for silence. For a pricing rollout, retain the diagnostic telemetry, add the independent heartbeat observer, and require the guarded write path before permitting automated rollback. The acceptance criterion is concrete: after rollback commits, a late attempt from the disabled generation cannot publish a price through any authorized writer.
Top comments (0)