Short answer: use error tracking for failed code, uptime monitoring for reachable services, and a cron heartbeat for scheduled work that never started or finished.
For a small edtech SaaS comparing an experiment across tenant cohorts, the heartbeat is the decisive addition. A green API and an empty error inbox cannot prove that last night's aggregation ran. If the job emits a completion heartbeat only after it commits attributed usage to Postgres, silence becomes evidence rather than an absence of evidence.
| Signal | Question it answers | Useful cohort evidence | Blind spot |
|---|---|---|---|
| Error tracking | Did executed code fail? | Exception plus tenant and experiment context | A scheduler that never invokes the code |
| Uptime probe | Can a known path answer now? | API and dependency availability | A healthy web process with a missing background run |
| Cron heartbeat | Did expected work report by its deadline? | Run ID, cohort counts, and completion time | Correctness inside a run unless the payload is validated |
Recommendation: keep all three, but page on the signal closest to the promise being made. Probe the learner-facing path for availability, capture exceptions from executed code, and require a completion heartbeat for the nightly cohort rollup. Do not make one signal impersonate the other two.
How should error tracking, uptime monitoring, and a cron heartbeat differ?
An uptime check makes a request on a schedule and judges the response. That is useful evidence that the selected path can respond. It says nothing about a separate scheduler unless that request exercises the scheduler itself. A health response can be green while the nightly aggregation has not run.
Error tracking starts later in the causal chain. The runtime must execute instrumented code and encounter a captured failure. If a deployment omits the scheduler entry, a queue trigger is disabled, or the process exits before installing its handlers, there may be no application exception to report. No execution, no captured error.
That is the silent-failure boundary.
Nothing crashed.
A heartbeat reverses the test. Instead of waiting for a failure event, monitoring waits for evidence of success within an expected window. For the experiment job, a start event proves only that execution began. A completion event sent after the database transaction proves more: this run reached the point at which its attributed records were committed. It still does not prove the numbers are correct, so validation remains part of the job.
This distinction matters to a solo operator because every alert spends attention. I would rather define one narrow promise per signal than debug a vague "background processing looks odd" alarm between feature releases. Ship weekly, but make the scheduled promises explicit.
One promise. One deadline.
Put evidence after the business invariant
The heartbeat position should follow the invariant, not the last convenient line in the scheduler wrapper. For this edtech experiment, the invariant might be: every eligible tenant cohort has one idempotent aggregate for a particular experiment date, and the usage assigned to those rows reconciles with the run total.
Send a start event if run duration is operationally useful. Send completion only after validation and commit. If validation fails, throw an error with the run identifier and structured cohort context; do not send completion. Error tracking then explains the executed failure, while the missing completion independently detects that the scheduled promise was missed.
There is a trap here. A finally block feels reliable, so it is tempting to put the success heartbeat there. But finally runs after both success and failure. That turns a failed aggregation into apparent success. The correct location is deliberately boring: after the commit, after reconciliation, and outside the failure path.
Placement wins.
The identifiers need stable meanings. A run_id identifies one attempted execution. An experiment_id identifies the comparison. A pseudonymous tenant_id or cohort key supports attribution without putting student data into telemetry. W3C Trace Context defines a standard format for propagating trace identity across process boundaries; OpenTelemetry's semantic conventions provide common names for telemetry attributes. Neither standard decides which business dimensions are safe to collect. That remains an application decision.
Attribute cost where work is accepted
A cohort experiment is not useful if the result says treatment beat control but the cost ledger cannot say which tenants produced the work. Cost attribution should therefore be recorded beside the durable aggregate, using the same run and experiment identifiers. Logs alone are a weak ledger: retention can differ, retries duplicate events, and a query over messages does not automatically share the transaction that accepted the result.
Use an append-only usage record with an idempotency key. The actual unit can be tokens, task executions, storage bytes, or another measurable driver. Do not force unlike units into a fictional dollar value inside the observability layer. Finance rules and provider rates can change; raw quantities plus an effective timestamp preserve the inputs needed to recalculate.
The decision rule is simple. Alert on missing evidence, but compare cohorts from durable, reconciled records. Monitoring tells you whether to investigate. Postgres holds the data used for the experiment and its cost attribution.
Alerts are not ledgers.
This also controls cardinality. Tenant and experiment dimensions are valuable on a bounded run record, but copying every learner or request identifier into metric labels can create an unbounded series set. Keep high-cardinality detail in traces, structured events, or database rows with explicit retention and access rules. Use metrics for bounded operational dimensions such as job name, environment, and result.
A TypeScript completion boundary
This sketch uses generic interfaces. The quantities are application measurements, not prices. The important part is the order: calculate, validate, commit the aggregate and usage record, then emit completion.
type Cohort = "control" | "treatment";
type CohortResult = {
tenantId: string;
experimentId: string;
cohort: Cohort;
eligibleLearners: number;
completedLessons: number;
computeMillis: number;
};
interface ExperimentStore {
saveRun(input: {
runId: string;
results: CohortResult[];
usage: Array<{
idempotencyKey: string;
tenantId: string;
experimentId: string;
unit: "compute_ms";
quantity: number;
}>;
}): Promise<void>;
}
interface Signals {
started(runId: string): Promise<void>;
completed(runId: string, cohortCount: number): Promise<void>;
capture(error: unknown, context: Record<string, string>): void;
}
export async function aggregateExperiment(
runId: string,
experimentId: string,
store: ExperimentStore,
signals: Signals,
calculate: () => Promise<CohortResult[]>,
): Promise<void> {
await signals.started(runId);
try {
const results = await calculate();
const tenants = new Set(results.map((row) => row.tenantId));
if (results.length === 0 || tenants.size === 0) {
throw new Error("Experiment aggregation produced no tenant cohorts");
}
await store.saveRun({
runId,
results,
usage: results.map((row) => ({
idempotencyKey: `${runId}:${row.tenantId}:${row.cohort}`,
tenantId: row.tenantId,
experimentId,
unit: "compute_ms",
quantity: row.computeMillis,
})),
});
await signals.completed(runId, results.length);
} catch (error) {
signals.capture(error, { runId, experimentId });
throw error;
}
}
saveRun should implement one database transaction and reject duplicate idempotency keys. The heartbeat adapter should use a short timeout and bounded retry policy. Decide failure semantics explicitly: if the data committed but heartbeat delivery fails, retrying the whole aggregation must not double-charge usage. Idempotency makes that recovery path tolerable.
Notice what the code does not do. It does not send tenant IDs to the heartbeat channel. A cohort count is enough to diagnose missing or suspicious completion at that layer; authorized operators can join the runId to the durable records when deeper inspection is warranted.
That boundary is intentional.
Test the boundaries, not just the happy function. Run one test where calculate throws, one where the transaction rejects, one where completion delivery times out, and one where the same runId is replayed. Then deploy a canary schedule with a short-lived test job and verify that removing its completion signal produces an alert after the configured grace period.
When is the runner-up signal enough?
A heartbeat is unnecessary for work that has no independent schedule or completion promise. If a learner's request synchronously performs the whole operation, an uptime probe against that representative path plus error tracking may cover the relevant boundaries. The probe exercises availability; captured exceptions explain failures. Adding a cron contract where no cron exists creates noise.
This design has limits. A heartbeat cannot prove statistical validity, detect biased cohort assignment, or explain an exception, and a completion-only contract is a poor fit for continuously running consumers with no discrete deadline. For those consumers, measure queue age or lag instead. For immediate learner-facing availability, choose a representative uptime probe. For an executed code failure, choose error tracking with enough structured context to diagnose it. These are trade-offs between evidence types, not interchangeable product features.
Uptime monitoring is the better primary alert when the user-facing promise is immediate reachability. A completed nightly job does not prove that the lesson API can answer now. Conversely, error tracking should be the primary diagnostic signal for a thrown exception with stack and request context. A heartbeat can say the deadline passed, but it cannot reconstruct the failing line.
The signals overlap on purpose, yet their evidence differs. Keep the overlap when the consequence of silence is material. For a low-stakes cleanup task, a database timestamp checked during routine operations may be enough. For a cohort experiment that informs rollout and allocates tenant cost, delayed discovery can contaminate the decision, so an explicit completion deadline earns its maintenance cost.
Operate the contract without building a monitoring project
Start with one scheduled promise: the aggregation must complete by a defined time after its schedule. Add a grace period based on normal runtime variability and upstream dependencies, not a round number copied from another service. Document who receives the alert, what run record to inspect, and how replay preserves idempotency.
Then measure the monitoring path itself. A synthetic canary heartbeat can verify that missing-signal detection and notification still work. Review stale schedules after deployment changes; an intentionally retired job should have its heartbeat contract retired in the same change. Otherwise the monitoring layer teaches the operator to ignore alerts.
Keep this small. The revenue-per-hour lens favors a generic Signals port, a transactional store, and managed or self-hosted adapters chosen on operational fit. The adapter is undifferentiated plumbing. Outsource it when that buys back feature time, but retain the evidence model and identifiers in application code so a tool change does not rewrite the business invariant.
The final check is concrete: can one runId connect the schedule expectation, captured failure, committed cohort outputs, and raw usage quantities without exposing learner data? If yes, the three signals form a useful chain. If not, another dashboard will not repair the missing relationship.
Top comments (0)