Short answer: for scheduled B2B SaaS imports, keep a small internal view for diagnosis, but page from an independent check of the last successful result. An Express /health response proves that one process can answer now; it does not prove that the 02:00 customer import read its source, committed rows, and advanced its reconciliation cursor. The least complex reliable design therefore has three signals: a cheap process-health probe, a durable success timestamp per schedule, and an external evaluator that alerts only after the agreed completion window.
Start with the bill because retention, rather than probe traffic, usually determines what this design becomes. If 2,000 tenants run one import per hour, retaining one compact completion record per run produces 48,000 records per day, or about 1.44 million in a 30-day month. Retaining 20 log lines for every run produces about 28.8 million lines before retries, debug context, or stack traces. The important change is to retain the small audit record longer while expiring verbose logs sooner. What disappears is line-by-line forensic context for an old run; when an incident surfaces after that window, the team can prove whether and when the run completed, but may have to reproduce the detailed cause.
Should an internal health dashboard or external uptime monitor page us?
An HTTP liveness check answers a narrow question: can the service accept and answer a request? Readiness may additionally test whether the process should receive traffic. Neither condition establishes scheduled work completion. A worker can remain responsive while a scheduler stops dispatching, credentials reject every source request, a queue accumulates work, or a transaction repeatedly rolls back. Green is not evidence of progress.
Silence is data.
Define success at the business boundary instead. For an import, that boundary might require a source snapshot identifier, an idempotency key, committed row counts, a terminal status, and a reconciliation cursor written in the same transactional outcome. The evaluator should compare last_success_at with expected_by, plus an explicit grace period. This is an audit assertion, not a guess derived from the presence or absence of log text. Consider a schedule due at 02:30: a process probe at 02:31 may be green while the latest committed cursor still belongs to yesterday's occurrence; the internal dashboard should expose both facts, but the external uptime decision should derive from the overdue occurrence. If the run commits at 02:42, the incident retains that timestamp and the logical import key, allowing reconciliation to distinguish a late success from a missing one without parsing a message stream.
The distinction matters for exactly-once thinking. Most distributed execution is retryable rather than magically exactly once, so the durable claim should be precise: a particular logical import key produced one committed outcome, and subsequent attempts found or reproduced that outcome without duplicating effects. An append-only attempt history can coexist with a single current outcome. That arrangement supports reconciliation without asking operators to infer state from interleaved messages.
| Evidence | Answers | Keep | Alert role |
|---|---|---|---|
| Process probe | Is this instance answering? | Minutes to days | Routing and immediate availability |
| Import outcome | Did schedule tenant/time commit? |
One or more audit periods | Primary missing-result signal |
| Metrics | Is delay or failure rate changing? | Aggregated longer than raw events | Trend and capacity warning |
| Detailed logs | Why did an attempt fail? | Shortest useful diagnostic window | Investigation, not primary truth |
Compliance obligations determine the actual retention periods; there is no universal number. Access to outcome records should be controlled, changes attributable, and tenant-sensitive source data excluded unless it is necessary for the audit purpose. The record needs enough evidence to reconcile the event, not a copy of the customer payload.
Make absence measurable
Absence alerts are difficult because silence has several meanings. A tenant may have paused a schedule. A regional holiday may mean the upstream publishes nothing. A long import may still be making legitimate progress. Model those states explicitly: enabled, expected_by, started_at, heartbeat_at, completed_at, and a terminal outcome. Do not turn every missed heartbeat into a page.
Keep it boring.
Use two thresholds. The first marks a run late and creates a visible, non-paging condition; the second pages after the completion deadline plus grace. Set both from the schedule contract and observed duration distribution, not from an arbitrary five-minute interval. A daily import expected by 02:30 with a 30-minute grace should be evaluated against that contract even if the API itself has answered every second.
One edge case deserves disproportionate attention: a successful run that writes its result and then crashes before acknowledging its work item. A retry must use the same logical key and discover the committed result. Otherwise the monitoring system may report recovery while the data plane applies the import twice. The alert can close only from durable outcome state, never merely because a retrying worker emitted done.
Here is a small, vendor-neutral evaluator. It is Go because the checker is easier to review as a separate control-plane program than as another branch inside the Node.js service.
package liveness
import "time"
type Schedule struct {
Enabled bool
ExpectedBy time.Time
Grace time.Duration
LastSuccess time.Time
}
type State string
const (
OnTime State = "on_time"
Late State = "late"
Page State = "page"
)
func Evaluate(now time.Time, s Schedule) State {
if !s.Enabled || !s.LastSuccess.Before(s.ExpectedBy) {
return OnTime
}
if now.Before(s.ExpectedBy.Add(s.Grace)) {
return Late
}
return Page
}
The compact example omits calendars, time-zone changes, and runs already in progress; production policy must define them. Keep the function pure, feed it persisted schedule state, and test times immediately before, at, and after each boundary. That is less exciting than another dashboard. It is also reviewable.
This architecture has a real limitation: it is not suitable when the organization cannot maintain authoritative schedule state or an independent notification path. In that case, begin with a process probe and a manually reviewed completion report while the data contract is established. The trade-off is delayed detection in exchange for avoiding confident but false pages. At the other extreme, teams with continuously arriving work rather than explicit schedules should evaluate age and backlog distributions; manufacturing synthetic deadlines for every message would create noise without adding audit evidence.
Signal quality without alert multiplication
Page on symptoms that require action. A missed committed outcome is such a symptom; a single application error generally is not. Logs can preserve error details, metrics can reveal rising duration, and traces can connect an import attempt across services, but each should enrich the incident keyed by tenant and logical schedule rather than create an independent page.
Sampling needs care. OpenTelemetry distinguishes head sampling, decided near the beginning of a trace, from tail sampling, decided after the trace completes. Tail decisions can retain failed or unusually slow executions more selectively, but they require the sampling system to observe completed trace data. Neither technique should sample away the durable import outcome. Audit evidence and diagnostic telemetry have different loss budgets.
Keep cardinality bounded in metrics. A counter labeled by outcome class and import type is useful; a tenant identifier or raw idempotency key can create a separate time series for every customer or run. Put those identifiers in access-controlled outcome records and logs, then link from an alert using an opaque run identifier. The dashboard shows system shape. The audit store answers which run.
Quiet alerts also require deduplication. Use the logical schedule occurrence as the incident key, notify once when it crosses the paging boundary, and resolve only after a later committed success or an attributable operator decision such as disabling the schedule. Re-evaluating every minute must not produce a new notification every minute.
Deployment, toggles, and failure containment
The evaluator and the writer of outcome records form a contract, so deploy schema changes compatibly. Add fields before requiring them, allow both representations during migration, and remove the old interpretation only after all writers have moved. A malformed new record should become a visible validation failure; it should not silently look like an absent import.
Feature toggles can limit rollout by cohort, but the toggle itself becomes operational state. Martin Fowler's taxonomy separates release, experiment, ops, and permissioning toggles because their lifetimes and decision logic differ. For this workflow, record which policy version and toggle state evaluated a deadline. Otherwise an auditor can see that an alert changed state but cannot reconstruct why.
Test the negative paths deliberately: disable dispatch while leaving Express healthy, reject source authentication, hold a transaction open past the deadline, commit and then redeliver the work item, and pause a tenant schedule. The expected results should differ. Only the cases that violate the enabled schedule's completion contract should page.
Deploy the external evaluator in a failure domain that does not share the import service's scheduler, process, or primary notification path. Independence need not mean a commercial uptime product. It means the mechanism asking “did work finish?” cannot disappear with the mechanism doing the work.
The retention decision
Retain three layers according to purpose. The current schedule state must remain available for evaluation. Immutable completion outcomes should last for the audit and reconciliation period set by the organization's contractual and regulatory obligations. High-volume logs and spans should have a shorter diagnostic window, with aggregation or selective sampling preserving trends.
This deliberately sacrifices old debugging detail. It does not sacrifice proof of completion.
The resulting operating rule is concise: an internal view helps engineers explain a late import, while an independent evaluator detects the absence of a committed result. A Node.js health endpoint remains useful for traffic management, but it is the wrong ledger for scheduled progress. Build the alert around the evidence contract, deduplicate by logical run, and let retention follow the distinct purposes of audit and diagnosis.
Further reading
- Martin Fowler, “Feature Toggles”: https://martinfowler.com/articles/feature-toggles.html
- OpenTelemetry, “Sampling”: https://opentelemetry.io/docs/concepts/sampling/
Top comments (0)