DEV Community

RadcliffBarrett4718
RadcliffBarrett4718

Posted on

Node.js Cron Cost Attribution: Alerting on Missed Scheduled Runs

#node #observability #cron

For a Node.js scheduled job, a failed run needs an alert, but a job that never starts needs a different alert. The cheapest reliable design for an edtech experiment is a small run ledger, a completion heartbeat, and ordinary application telemetry. The ledger says what should have happened for each tenant cohort. The heartbeat watches the deadline. Logs and metrics explain an error after a process actually starts, giving cost attribution a stable place in the same incident record.

Decision table

Design Detects a thrown error Detects a missing run Handles cohort cost Pick this when
Application logs Yes No Only with structured fields Diagnosis is the immediate need
Counters and duration metrics Usually No, without a schedule owner Useful for aggregates A metrics platform already owns operations
Completion heartbeat Indirectly Yes With run labels A small watchdog is enough
Scheduler deadline Yes Yes With a durable run ledger The scheduler already controls dependencies and retries
Reconciliation query Eventually Yes Best for late or duplicate work Accounting correctness matters more than instant paging

The key distinction is a missing fact. A failed process can emit an error. A process that never launches cannot. A heartbeat is therefore a clock check, not an error collector.

No magic.

For the cost axis, create one immutable key per experiment, cohort, and scheduled window. A display label is not enough: cohort names change, while an accounting key must remain stable. Keep estimated cost, rows processed, and the allocation-rule version beside that key. A missed heartbeat can then identify the affected budget without claiming that silence is an invoice.

How can a Node.js cron task report a missed run without hiding cohort costs?

Model the lifecycle before writing an alert rule. First, the scheduler creates a planned window. Next, the worker records an attempt. Finally, it records completion after the durable export, write, or handoff that defines success. A missed run is a planned window with no attempt or completion by its deadline.

That gives four useful states:

  • planned with no attempt: inspect scheduling, deployment, host, or queue ownership.
  • started with no completion: inspect the application and its dependencies.
  • completed without a received heartbeat: reconcile monitoring delivery before rerunning work.
  • two completions for one window: block an automatic rerun until the external effect is understood.

The state is more actionable than a generic failed counter. It also keeps a late cohort export from being charged twice just because the alert arrived first.

RFC 5424 provides useful severity semantics for explicit log events. It does not define what silence means. Keep schedule deadlines outside the log severity field, and route them using experiment priority and tenant impact.

Implement the success boundary in TypeScript

The completion call belongs after the durable operation. A startup ping proves that a scheduler launched a process; it does not prove that a cohort export exists. Short rule. Signal success last.

import { appendFile } from "node:fs/promises";

const heartbeatUrl = process.env.HEARTBEAT_URL;
const experimentId = process.env.EXPERIMENT_ID ?? "reading-time-a";
const cohortId = process.env.COHORT_ID ?? "tenant-cohort-03";
const windowId = process.env.WINDOW_ID ?? new Date().toISOString().slice(0, 10);
const runId = `${experimentId}:${cohortId}:${windowId}`;

if (!heartbeatUrl) {
  throw new Error("HEARTBEAT_URL is required");
}

async function writeCohortExport(): Promise<{ rows: number; estimatedCostUsd: number }> {
  // Replace this with the durable operation that defines success.
  const rows = 2_400;
  const estimatedCostUsd = 0.18;
  await appendFile("experiment-exports.log", `${runId},${rows}\n`, "utf8");
  return { rows, estimatedCostUsd };
}

async function sendHeartbeat(url: string): Promise<void> {
  const response = await fetch(url, {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify({ runId, experimentId, cohortId, windowId }),
  });

  if (!response.ok) {
    throw new Error(`Heartbeat returned HTTP ${response.status}`);
  }
}

async function main(): Promise<void> {
  const startedAt = new Date().toISOString();

  try {
    const result = await writeCohortExport();
    console.log(JSON.stringify({
      level: "info",
      event: "cohort_export_completed",
      runId,
      experimentId,
      cohortId,
      windowId,
      startedAt,
      completedAt: new Date().toISOString(),
      rows: result.rows,
      estimatedCostUsd: result.estimatedCostUsd,
    }));
    await sendHeartbeat(heartbeatUrl);
  } catch (error: unknown) {
    const message = error instanceof Error ? error.message : String(error);
    console.error(JSON.stringify({
      level: "error",
      event: "cohort_export_failed",
      runId,
      experimentId,
      cohortId,
      windowId,
      message,
      occurredAt: new Date().toISOString(),
    }));
    process.exitCode = 1;
  }
}

void main();
Enter fullscreen mode Exit fullscreen mode

The local append is only a readable example. Production accounting needs a durable store and uniqueness on (experimentId, cohortId, windowId). That constraint turns duplicate delivery into a visible reconciliation case. It also gives an operator something better than guessing from an alert.

Set the monitor's expected cadence and grace period from this job's normal queue delay and completion distribution. I'm not sure a universal five-minute deadline exists. A large tenant cohort can have a different normal runtime from a small one; your mileage may vary. Detect absence early enough to matter, but do not make ordinary variance page.

Run two tests. Force the export to throw: the structured event should contain the cost fields, the process should exit nonzero, and no completion heartbeat should be sent. Then do not launch the worker at all: the deadline should fire even though there is no application exception. Those tests exercise separate failure paths.

Logs own exception details, dependency names, and run context. Metrics own rate, latency, and size distributions. The heartbeat owns one question: did this labeled completion arrive before the window closed? The ledger owns the durable answer used for cost reconciliation.

Keep the heartbeat credential in the runtime environment and out of logs. Use opaque experiment and cohort IDs rather than learner data. A payload can be useful for routing without becoming a second data warehouse.

There is an awkward but normal race: the export may commit while the heartbeat request is still in flight. Bounded retries can improve delivery, but no HTTP request makes the export and the monitor one transaction. When a later reconciliation sees completed but not heartbeat_received, suppress an automatic rerun and investigate delivery. A repeated external effect is often worse than a late notification. This is where the ledger earns its keep: it lets an operator compare the durable export, the heartbeat receipt, the scheduled window, and the estimated allocation rule over one concrete run instead of inferring financial state from a page that arrived at an arbitrary time.

Limits and the final choice

This approach is not a distributed trace, session replay, or native-crash symbolication system. Electron's crashReporter flow is the relevant path for native crashes and minidumps. A log field such as trace_id helps correlation; it does not create a span tree.

It is not suitable when a job's external effect cannot be made retry-safe and a duplicate is more harmful than a delayed alert. Add idempotency and reconciliation first. For overlapping jobs, decide whether the deadline means “started by” or “completed by,” then name the signal accordingly.

The recommendation is deliberately narrow: use a completion heartbeat for absence, structured telemetry for explanation, and a run ledger for tenant-level cost attribution. Stick with a scheduler deadline when it already owns workflow state. Use reconciliation when financial correctness outranks immediate notification.

References

Further reading

Top comments (0)