Reconstruct failed nightly cron jobs with searchable structured logs, a small backend metrics dashboard, and a separate heartbeat check. Start the retention discussion with the bill's actual components: ingestion, searchable storage, query work, and operational maintenance. Keeping every intermediate record searchable indefinitely is a design decision, and it needs a stronger justification than the possibility that somebody might want it later.
TL;DR: Use metrics for cron-job success counts, API failures, durations, backlog sizes, and business-event trends; retain structured run evidence for reconstruction, and add Healthchecks-style heartbeat monitoring for a job that never runs. For a small B2B SaaS operation, that is the least complex useful division of responsibility. A dashboard showing zero failures cannot establish that the nightly pipeline started.
Consider an explicitly hypothetical sizing exercise: 100 tenant imports each emit 10,000 structured records each night, averaging 1 KB per record. That produces 1 GB per night before compression, indexes, replicas, or service-specific accounting, using decimal units. Thirty nights retain 30 GB of raw records; seven retain 7 GB. These are arithmetic assumptions, not production measurements or vendor billing units. They isolate the retained-volume term before a product recommendation obscures it; identifying the dominant dollar charge would still require the actual ingest, storage, and query rates.
Keep the evidence.
The question is which evidence, for how long, and what an investigator can still prove after deletion.
What should a backend metrics dashboard retain for cron jobs?
For a B2B SaaS pipeline, the incident question is usually more specific than whether the backend was healthy: which tenant import failed, which attempt encountered the upstream API error, and whether the business outcome was eventually committed. Metrics make the shape of the night visible. Structured records preserve the distinctions needed to explain it. A count of successful imports cannot identify the tenant whose data stayed stale.
Define an application-owned evidence contract before choosing the dashboard. A run identifier should identify the scheduled run; an attempt identifier should distinguish retries; a tenant reference should connect the incident to its business scope without copying the imported payload into telemetry. Add event time, the pipeline stage, and an explicit outcome. These are proposed application fields, not parameters for a vendor's search API. Their usefulness depends on the producer assigning them consistently across records.
The failure mode to name here is retry ambiguity. If the same run has a failed attempt followed by a successful attempt, counting every completion record as a completed business operation turns a reconstruction aid into a misleading success chart. Count business outcomes at the application boundary that defines completion, and retain attempt evidence separately. The dashboard and the log search should describe the same execution without pretending they have identical granularity.
A missing record proves less than it seems.
An absent completion can mean unfinished work, a missing telemetry write, or a search window that excludes the relevant event. Establishing that the data became durable requires evidence from the data layer's own commit boundary; a log message announcing success should not be treated as a storage durability guarantee. Similarly, a shared trace identifier can help correlate records, but correlation alone does not establish a complete distributed trace or a span tree.
Move the retention term without destroying the explanation
Under the sizing assumptions above, shrinking the searchable window from 30 nights to seven removes 23 GB of raw records from that window. That difference says nothing by itself about money saved: compression, indexing, replication, ingest accounting, and query charging can change the bill. The useful engineering conclusion is narrower. Searchable retention grows with records per run, record size, run frequency, and days retained; reducing one of those factors moves the volume term directly.
Before shortening retention, remove unnecessary payload duplication at the producer. The investigator usually needs the import stage, attempt, outcome, and a usable reference to the affected operation. Copying the same input document into every intermediate event increases retained bytes without necessarily improving the explanation. However, replacing payloads with references introduces its own boundary: the reference is useful only while the corresponding source evidence remains available and authorized for the investigator to read. A pointer to deleted data isn't an archive.
The next table is a proposed evidence policy, not a claim about any service's configurable retention or storage tiers. The seven-night and 30-night windows are deliberately illustrative; the operation's actual investigation delay must decide them.
Retention has limits.
| Evidence class | Proposed treatment | What it supports | What it sacrifices |
|---|---|---|---|
| Intermediate attempt records | Searchable for seven nights | Stage-by-stage reconstruction of recent failures | Detailed diagnosis of older attempts |
| Application completion records | Retain for 30 nights in a chosen evidence store | Connecting a scheduled run to its recorded business outcome | Intermediate context after the shorter window expires |
| Success, failure, duration, and backlog metrics | Keep a deliberate trend window | Comparing nightly behavior over time | Individual tenant and attempt explanations |
| Heartbeat evidence | Track expected run arrival separately | Recognizing a scheduled run that did not arrive | Explaining the cause of the missing run |
Do not describe a second copy as durable just because it is outside the searchable system. Its write acknowledgement, access controls, retention, and ability to be read during an incident deserve separate acceptance checks. If the team cannot demonstrate retrieval after the primary search window expires, the architecture has shortened its evidence horizon regardless of what the diagram calls the second store.
This is also where search acceptance belongs. Test a known failed run, its later retry, and its completion record against the intended workflow before deleting older searchable records. Require the investigation to recover the relationship among them. Don't assume a product supports a particular filter syntax merely because its interface advertises search.
Compare tools by the question they must answer
A fair comparison starts with the role a product would occupy. Prometheus, Grafana Loki, and Healthchecks are real options to evaluate, but treating them as interchangeable dashboard backends hides the evidence problem. Their primary documentation offers distinct starting points: metric time series, logs, and cron monitoring. The table names those roles and the additional question an evaluator should ask; it does not claim equivalent retention controls, delivery guarantees, or measured performance.
| Option | Role in this pipeline | Useful evaluation task | Boundary to preserve |
|---|---|---|---|
| Prometheus | Metrics and time-series investigation | Review success counts, failure counts, durations, and backlog trends | An aggregate does not preserve the underlying structured execution record |
| Grafana Loki | Log investigation | Recover related structured records for a known failed import | Searchable records need an explicit retention and retrieval policy |
| Healthchecks | Cron-job monitoring | Check whether the expected nightly execution reported its presence | A heartbeat does not explain the upstream API failure or prove a business commit |
For a small operations dashboard, Infrai offers a plain REST API with one key and one bill. An HTTP-capable producer can call it without installing a service-specific SDK or maintaining that client library's version. The single key covers 295 routes across 20 modules, including metrics and error APIs. This unified API means the nightly producer and the dashboard reader don't need separate service credentials merely to combine those signals; unified billing also removes separate invoice reconciliation for those capabilities. That is a workflow advantage, not evidence of stronger durability. Its public, self-describing discovery requires no key and exposes request and response schemas, letting an evaluator inspect the contract before connecting the pipeline. Structured-log investigation and heartbeat completeness still need explicit acceptance criteria in the overall design.
The trade-off is evidence granularity. Metrics alone are not suitable for reconstructing individual attempts; choose the log investigation workflow for that requirement, and choose Healthchecks for missed-run monitoring. These limitations belong in the acceptance criteria, even when the charts look convincing.
The following read-only Python example retrieves metrics without inventing filter parameters or a response schema. Set INFRAI_API_ORIGIN to the service's documented HTTPS API origin, without a trailing path, and set INFRAI_API_KEY in the environment. It uses only Python's standard library. Inspect the returned JSON before building widgets; this request checks the metrics read path, not log retention or heartbeat coverage.
import json
import os
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
from urllib.error import HTTPError
from urllib.request import Request, urlopen
origin = os.environ["INFRAI_API_ORIGIN"].rstrip("/")
if not origin.startswith("https://"):
raise ValueError("An HTTPS API origin is required")
request = Request(
origin + "/v1/metrics/query",
headers={"Authorization": "Bearer " + os.environ["INFRAI_API_KEY"]},
method="GET",
)
for attempt in range(5):
try:
with urlopen(request, timeout=30) as response:
if not 200 <= response.status < 300:
raise RuntimeError(f"Unexpected status: {response.status}")
print(json.dumps(json.load(response), indent=2))
break
except HTTPError as error:
detail = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == 4:
raise RuntimeError(f"HTTP {error.code}: {detail}") from error
delay = float(2 ** attempt)
retry_after = error.headers.get("Retry-After")
if retry_after:
try:
delay = max(delay, float(retry_after))
except ValueError:
retry_at = parsedate_to_datetime(retry_after)
if retry_at.tzinfo is None:
retry_at = retry_at.replace(tzinfo=timezone.utc)
delay = max(
delay,
(retry_at - datetime.now(timezone.utc)).total_seconds(),
)
time.sleep(delay)
No write occurs.
Choose the boundary deliberately. If the primary question is which attempt failed, prioritize the log investigation; if it is how the pipeline behaved over several nights, prioritize metrics; if it is whether the expected run appeared at all, prioritize the heartbeat. A small team can combine these roles without claiming complete monitoring coverage.
There are further procurement questions that should stay separate from that recommendation. Requirements for notifications, complete distributed traces, source-map decoding, native crash symbolication, or session replay need their own verified coverage. So do per-user deletion, export, and retention administration. A metrics chart is insufficient evidence that any of those requirements has been met.
Decide what an older incident will cost to explain
The proposed policy deliberately stops keeping intermediate attempt records searchable after seven nights. It also avoids copying imported payloads into every event. The reason is to preserve a manageable reconstruction window without treating every byte as equally valuable evidence; the cost is losing the detailed sequence of older retries and depending on still-readable source records when a payload becomes relevant.
Deletion is irreversible evidence loss unless an independently verified copy remains. Under the illustrative policy, an investigation on night eight might still establish that an import committed, but it cannot promise to explain every intermediate API failure. After the completion-record window expires, even that claim needs another authoritative source. Trend metrics and heartbeat history cannot recreate discarded records.
Accept that boundary explicitly, or extend the evidence window before adopting the policy. Retaining less is defensible only when the team can say which future questions it has chosen to leave unanswered.
References
The primary sources below document the different signal roles used in the comparison. They don't establish equivalent retention behavior or a shared durability guarantee across products.
Top comments (0)