Short answer: use an externally observed completion heartbeat to detect a cron job's silent failure, and use structured app logs to explain what happened. For a nightly marketplace import, send completion only after the output is committed and validated. Do not ask a log search to prove that a process which never started is missing.
This split also makes cost attribution legible for a beginner SaaS team operating in Europe and the US. One small completion event belongs to the monitoring control plane. Detailed events belong to the pipeline and carry stable team, job, and run_id fields. You can retain each according to its value instead of indexing every item-level event forever.
How should app logs monitor a cron job's silent failure?
Logs describe code that executed far enough to emit them. If a scheduler never launches the process, the host disappears, or execution stalls before initialization, there may be no application event. Searching for level=error then returns an immaculate empty result. Silence is ambiguous.
Nothing ran.
Heartbeat monitoring reverses the question: a receiver expects evidence inside a defined window and alerts when it does not arrive. The receiver must be outside the job's failure domain. A local success file beside the process cannot report that its own machine is unavailable.
Send the decisive signal at completion, not startup. A startup ping proves scheduling, but it does not prove that marketplace offers were published. Completion should mean the new dataset was committed and inexpensive business invariants passed. If runtime visibility matters, add a start or progress signal without allowing it to clear the missing-completion alert.
Build the smallest useful completion contract
The flow is short. The scheduler launches the importer, which assigns a unique run ID used in every structured log. After commit and validation, the process posts a compact completion document to an independent receiver. The receiver owns the expected cadence and grace period; the log backend remains the investigation surface.
This runnable Python example uses only the standard library. HEARTBEAT_URL is a deployment-supplied HTTPS endpoint, not a provider-specific API route.
import json
import os
import time
import uuid
from datetime import datetime, timezone
from urllib.request import Request, urlopen
JOB = "nightly-marketplace-offer-import"
RUN_ID = str(uuid.uuid4())
def emit(event: str, **fields: object) -> None:
record = {
"timestamp": datetime.now(timezone.utc).isoformat(),
"event": event,
"job": JOB,
"run_id": RUN_ID,
"team": "catalog-data",
**fields,
}
print(json.dumps(record, separators=(",", ":")), flush=True)
def send_completion(duration_ms: int, rows_written: int) -> None:
payload = json.dumps({
"job": JOB,
"run_id": RUN_ID,
"status": "complete",
"duration_ms": duration_ms,
"rows_written": rows_written,
}).encode("utf-8")
request = Request(
os.environ["HEARTBEAT_URL"],
data=payload,
headers={"Content-Type": "application/json"},
method="POST",
)
with urlopen(request, timeout=10) as response:
if not 200 <= response.status < 300:
raise RuntimeError(f"heartbeat returned status {response.status}")
def import_offers() -> int:
# Replace with the real transaction or atomic publish operation.
time.sleep(0.2)
return 12_480
started = time.monotonic()
emit("run_started")
try:
rows = import_offers()
if rows <= 0:
raise ValueError("validated output must contain rows")
elapsed_ms = round((time.monotonic() - started) * 1000)
emit("output_committed", rows_written=rows, duration_ms=elapsed_ms)
send_completion(elapsed_ms, rows)
emit("completion_reported")
except Exception as error:
emit("run_failed", error_type=type(error).__name__)
raise
The 12_480 rows are sample data, not a benchmark or suggested threshold. A real rule comes from the marketplace's input contract and recent successful runs.
Ordering is the important part: commit, validate, log, then report completion. This example treats failed delivery as a failed run. That trade-off favors visible uncertainty over a false success. Production retries should reuse the run ID, apply bounded backoff, and preserve a failing exit status; the receiver should deduplicate by job and run ID.
Never place credentials, seller data, offer text, or raw prompts in either signal. The run ID is enough to join an alert to diagnostics.
Separate paging evidence from diagnostic evidence
The signals answer different questions, so each has limits.
| Signal | Question | Useful fields | Main limitation |
|---|---|---|---|
| Completion heartbeat | Did the outcome arrive on time? | job, run ID, completion time | Cannot explain the failed stage |
| Run summary log | What did this run produce? | counts, duration, version | Exists only if code executes |
| Detailed app logs | Where did execution fail? | run ID, stage, safe context | Higher volume and retention cost |
Use stable ownership dimensions such as team and pipeline to attribute ingestion and retention. Keep run_id for log correlation, but not as a metrics label: the Prometheus data model states that every unique label combination creates a new time series.
For an AI-assisted enrichment stage, one run_summary can record accepted and rejected counts, a bounded rejection reason, model or ruleset version, duration, and token usage returned by the serving system. This connects evaluation and prompt cost to an immutable run without indexing every prompt. It is less useful when per-item forensic replay is a contractual requirement; in that case, retain protected item-level evidence in purpose-built storage and link it by run ID.
An analytical store can query structured logs and apply retention rules, but its limits still matter. Large unbounded text fields cost space, high-cardinality filters need deliberate schema design, and a storage cluster does not replace an independent absence detector. OpenTelemetry's Logs Data Model offers common fields for timestamp, severity, body, resource, and attributes, reducing translation when collectors or stores change.
How long should the monitor wait?
An alert window combines the expected schedule with a grace period. Derive grace from observed successful durations, scheduler jitter, and upstream availability, then revisit it. Five minutes copied from another job may be noisy for a 90-minute import and dangerously slow for a ten-minute feed.
Store timestamps in UTC with an RFC 3339 representation. When business rules require local time, configure an IANA time zone and test daylight-saving transitions in both European and US deployments. Kubernetes documents that a CronJob can set .spec.timeZone; it also explains that Job creation is approximate and that scheduling can sometimes create two Jobs or none. The import therefore needs idempotency as well as monitoring.
The heartbeat receiver has its own failure mode. Monitor its ingestion and alert route separately. If every job stops reporting at once, receiver health is a stronger first hypothesis than simultaneous pipeline failure.
That gap is the signal.
Operate the contract
Before deployment, test with a fake receiver and assert that completion follows commit and validation. Exercise three negative paths: prevent launch, terminate before commit, and reject heartbeat delivery. The first two should become missing-completion alerts. The third should fail the process and leave scheduler-captured logs that identify delivery as the failing stage.
On call, begin with the expected window. No start event points toward scheduling or the execution environment. A start without an output commit points into the pipeline. A commit without recorded completion requires reconciling the immutable output before retrying, because a blind rerun can duplicate marketplace state.
Assign one team to the schedule, receiver, notification route, runbook, and retention review. Track late successful runs and false alerts, then adjust grace from evidence. For Europe and US deployments, place logs and alert data according to contractual residency and response requirements rather than assuming one global stream is acceptable.
The durable design stays small: one external proof of completion, one correlation ID, and enough structured evidence to explain the result. Healthchecks-style absence detection catches true silence; logs retain the rich context needed for debugging and cost attribution.
Top comments (0)