The hard trade-off is evidence versus simplicity: a failure alert is easy to add, but it cannot explain a silent scheduler or prove which tenant cohort was actually processed. For a property-management experiment running across EU and US cohorts, preserve a small, append-only run ledger outside the scheduled task and derive alerts from missing or contradictory evidence. That gives responders something they can reconstruct, not merely a red light.
TL;DR: assign every planned cohort run a unique identity, record its deadline before execution, and accept started, failed, and completed evidence against that identity. Alert when the evidence violates the run contract. Keep tenant data out of the ledger; retain the region, cohort, experiment version, counts, attempts, and UTC timestamps needed to replay the decision.
How should monitoring catch a missed cron job run without a heartbeat?
Start with the incident review and work backward. A responder needs to determine whether the EU treatment run existed, which experiment revision it used, how much work it claimed, and whether a later retry duplicated effects. A generic process heartbeat answers none of those questions reliably. It says that one process was alive at one instant.
The useful unit is a planned run. For example, lease-msg-v4:eu:treatment:2026-09-25 identifies one cohort decision, while an attempt number distinguishes retries without pretending that each retry is new scheduled work. The ledger can remain compact: planned time, deadline, region, cohort, experiment version, policy version, expected count, processed count, attempt, terminal status, and timestamps. Tenant names, addresses, messages, and model prompts do not belong there.
This restraint matters because operational records can become another copy of sensitive application data. NIST SP 800-66r2 addresses HIPAA-regulated entities rather than property managers, but its risk-analysis and information-system-activity-review framing is a useful reference when deciding what evidence to retain, who may read it, and how reviews are performed. The practical rule is narrower: store enough to explain the run, then link to separately controlled detail when an authorized investigation requires it.
Here is the evaluator I would put in a notebook first. It is deliberately pure. The scheduler, persistence layer, and alert transport can change without changing the decision contract.
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from enum import Enum
class Verdict(str, Enum):
WAITING = "waiting"
MISSING = "missing"
FAILED = "failed"
INCOMPLETE = "incomplete"
COMPLETE = "complete"
CONFLICT = "conflict"
@dataclass(frozen=True)
class RunContract:
run_id: str
region: str
cohort: str
due_at: datetime
grace: timedelta
expected_items: int
@dataclass(frozen=True)
class RunEvidence:
run_id: str
attempt: int
status: str
processed_items: int
recorded_at: datetime
def reconstruct(
contract: RunContract,
events: tuple[RunEvidence, ...],
observed_at: datetime,
) -> Verdict:
relevant = tuple(event for event in events if event.run_id == contract.run_id)
deadline = contract.due_at + contract.grace
if not relevant:
return Verdict.MISSING if observed_at > deadline else Verdict.WAITING
terminal = tuple(event for event in relevant if event.status in {"failed", "completed"})
if not terminal:
return Verdict.MISSING if observed_at > deadline else Verdict.WAITING
latest = max(terminal, key=lambda event: (event.attempt, event.recorded_at))
same_attempt = tuple(event for event in terminal if event.attempt == latest.attempt)
if len({event.status for event in same_attempt}) > 1:
return Verdict.CONFLICT
if latest.status == "failed":
return Verdict.FAILED
if latest.processed_items != contract.expected_items:
return Verdict.INCOMPLETE
return Verdict.COMPLETE
contract = RunContract(
run_id="lease-msg-v4:eu:treatment:2026-09-25",
region="eu",
cohort="treatment",
due_at=datetime(2026, 9, 25, 1, 0, tzinfo=timezone.utc),
grace=timedelta(minutes=20),
expected_items=480,
)
before = datetime(2026, 9, 25, 1, 19, tzinfo=timezone.utc)
after = datetime(2026, 9, 25, 1, 21, tzinfo=timezone.utc)
assert reconstruct(contract, (), before) is Verdict.WAITING
assert reconstruct(contract, (), after) is Verdict.MISSING
The dates, 20-minute grace, and count of 480 are example policy data, not measured recommendations. In a real deployment, define grace from the job's own runtime evidence, upstream timing, and the remaining intervention window. Version that policy alongside the run contract so an old incident can be replayed under the rule that actually produced its alert.
Small function.
Big boundary.
Read silence, failure, and partial work differently
Silence has several possible causes: the schedule was absent, its host was unavailable, the process never started, or it stopped before recording a terminal event. From the worker's point of view, these cases are unreachable. An exception handler therefore cannot replace an independent deadline evaluation.
An explicit failed event is different. It proves that an attempt reached code capable of recording failure, and it can carry a bounded reason code for triage. incomplete is different again: the task reached a nominal completion path, yet its processed count disagreed with its contract. Collapsing all three into “cron unhealthy” loses the first decision a responder has to make.
| Verdict | Evidence pattern | First reconstruction question |
|---|---|---|
| Missing | Deadline passed with no terminal event | Did the schedule or execution environment disappear? |
| Failed | Attempt recorded terminal failure | Which attempt and experiment version failed? |
| Incomplete | Completion count violates the contract | Which part of the cohort was omitted? |
| Conflict | One attempt has incompatible terminal evidence | Was delivery duplicated or state mutated incorrectly? |
| Complete | Terminal count matches the contract | Did downstream analysis use this exact run? |
Do not let a late completion erase a missing interval. It may resolve an active page, but the ledger should retain when the deadline was breached and which attempt later completed. This is the difference between current health and historical evidence.
Retries need the same discipline. Alert identity belongs to the planned run ID; attempt identity belongs to execution. Otherwise one late retry can look like a fresh successful schedule, or two deliveries can create two incidents for one business obligation. Idempotent ingestion prevents duplicate evidence from multiplying, while conflicting evidence remains visible rather than being overwritten.
The page is a view; the ledger is the record.
Keep the regional comparison honest
EU and US cohorts need separate run contracts even when they are part of one experiment. Their intended schedules and grace policies may differ, and a shared heartbeat could stay green while one cohort never runs. Record timestamps as UTC instants, while retaining region and intended local schedule as explanatory metadata. That combination lets a responder reason about elapsed time without discarding the schedule operators intended.
Only compare experiment outcomes after both cohort contracts reach acceptable terminal states. Operational completion still does not prove that the experiment improved anything; it proves that the intended slice was processed under a recorded version. Analysis rules belong in a separate eval harness. If the outcome uses Core Web Vitals, for example, web.dev defines LCP, INP, and CLS assessment at the 75th percentile and recommends segmenting mobile and desktop. Preserve that definition in analysis fixtures instead of substituting an average during incident pressure.
This separation also keeps prompt cost under control. If an AI component summarizes an incident, give it the bounded ledger rows and restricted evidence links after deterministic code has classified the run. A model can compress context for a reviewer. Comparing a deadline or two integer counts does not need tokens, probabilistic output, or a prompt revision.
The correction I would make early in a notebook-to-production path is to stop treating “completed” as a trustworthy scalar. A completion claim needs a run identity and an invariant. Here, the invariant is that the terminal evidence belongs to the planned region and cohort and its count matches the contract. Another job may need a checksum or a committed output reference instead. The mechanism changes; the demand for reconstructable evidence does not.
Operate the witness, then rehearse the review
The witness must sit outside the scheduled worker's failure domain. Materialize run contracts independently, persist evidence before acknowledging it, enforce uniqueness for duplicate submissions, and retry alert delivery. Monitor the witness from another chosen boundary; otherwise the component responsible for detecting silence can disappear silently too.
Promotion should follow the evals. Turn the notebook cases into parameterized tests covering an on-time completion, explicit failure, no start, late completion, incomplete count, duplicate event, conflicting terminal state, wrong run ID, and regional schedule boundary. Add a property that evidence for run A can never satisfy run B. Then deploy the evaluator unchanged behind the chosen storage and delivery adapters.
The operational checklist works better as a rehearsal than as a wall of boxes. Before enabling pages, confirm that every region and cohort has a future contract. Stop one worker and observe the contract cross its documented deadline; force a controlled exception and confirm that it becomes failed, not missing. Replay the same terminal evidence, then submit incompatible evidence for the same attempt and make sure the conflict remains visible. Finally, take one alert and trace it through the planned run, attempts, policy version, experiment revision, and cohort counts without opening raw tenant content.
There is real overhead here: another data store, another clock, retention decisions, and a witness that needs its own care. A best-effort cache refresh with no meaningful deadline may justify only scheduler logs and failure notification. Use a run ledger when silence changes a business decision and when responders must explain which cohort received which experiment revision.
That is the decision line. For cross-region property experiments, incident reconstruction has value because a missed slice can invalidate the comparison. Build the evidence chain before adding richer alert presentation, and keep the deterministic evaluator boring enough to test exhaustively.
Top comments (0)
Some comments may only be visible to logged-in visitors. Sign in to view all comments.