Alerting on cron and background jobs that never ran, hung, or did nothing (in 6 languages)

Gökhan Ertoğan — Tue, 10 Nov 2026 22:48:01 +0000

Our HTTP alerting was in good shape. 500s, slow endpoints, error rates: all covered. Yet the incidents that hurt most kept coming from somewhere else:

The 02:00 invoice job never started, because the server restarted and the scheduler got stuck.
A queue consumer was up, but processed nothing for 20 minutes.
The bank statement file didn't arrive, so the import "succeeded" with 0 rows.
A worker loop deadlocked, and its exception was swallowed in a catch.

None of these has an HTTP status code. Ping-based monitors (Healthchecks.io, Cronitor and friends) cover part of it, but they add another SaaS, a ping URL in every job, and they can't tell "succeeded" from "succeeded with nothing done".

So I built RunSignal: cron and background-job monitoring on OpenTelemetry, with alerts as code, for .NET, Java, Go, Node.js, Python and Rust.

👉 GitHub: github.com/gokhanertogan/RunSignal (MIT)

How it works

Record. An SDK records each run of a workload (success, failure, cancellation, duration), the items it handled, and heartbeats from long-running loops, as OpenTelemetry metrics. Every SDK uses the same metric names.
Describe. Each workload declares what it should do: a cron schedule, a max duration, a heartbeat interval, a minimum output, a failure rate. The running app writes these to a JSON manifest.
Provision. In CI, runsignal provision turns the manifest into alert conditions and a dashboard in New Relic, or runsignal export prometheus writes Prometheus alerting rules (plus a Grafana dashboard).

Alerts are evaluated by the backend, never in your process, because a process that has died cannot report that it is missing. Because the metrics are identical in every language, one CLI provisions alerts for a whole polyglot system.

The same idea in six languages

Where a scheduler exists, there is no code in the jobs: the integration records every run and registers the job with its cron at startup, so a job that never fires still gets a missed alert.

.NET: Hangfire (Quartz.NET works the same way):

builder.Services.AddRunSignal().AddHangfire();
builder.Services.AddHangfire((sp, c) => c.UseSqlServerStorage(cs).UseRunSignal(sp));

[Workload("nightly-invoice", MaxDuration = "30m", MinProcessed = 100)]
public async Task GenerateInvoices(PerformContext? context = null) { /* ... */ }

Java: Spring @Scheduled (and Spring Batch), through Spring's own observations, with no proxies:

@Scheduled(cron = "0 0 2 * * *")
@RunSignal(value = "nightly-invoice", maxDuration = "30m", minProcessed = 100)
void generateInvoices() { ... }

Python: Celery tasks and beat schedules (APScheduler too), or a plain with block:

from runsignal import default_registry

invoices = default_registry.workload("nightly-invoice", schedule="0 2 * * *", max_duration="30m", min_processed=100)

with invoices.run():
    for order in unbilled_orders():
        create_invoice(order)
        invoices.processed()

Node.js: node-cron, croner, cron or node-schedule (BullMQ too):

import { cronJob } from '@runsignal/node/cron';

const invoices = cronJob('nightly-invoice', { schedule: '0 2 * * *', maxDuration: '30m' }, async (w) => {
  for (const order of await unbilledOrders()) { await createInvoice(order); w.processed(); }
});
cron.schedule(invoices.schedule, invoices.run);

Go: robfig/cron, or Run with context cancellation and panic handling:

rscron.AddFunc(c, runsignal.Default, "nightly-invoice", "0 2 * * *",
    func(ctx context.Context) error { return generateInvoices(ctx) },
    runsignal.MaxDuration(30*time.Minute))

Rust: sync and async runs; a dropped future is a cancelled run:

let invoices = registry.workload("nightly-invoice", Options::new()
    .schedule("0 2 * * *")
    .max_duration(Duration::from_secs(30 * 60)))?;

invoices.run(|| generate_invoices())?;

The alerts

Alert	Fires when
`missed`	No success within the schedule plus grace
`failed`	A run failed
`overrun`	A finished run took longer than its max duration
`hung`	A run is still running past its max duration
`stalled`	No heartbeat in time (a loop that is alive but stuck)
`low output`	Runs processed fewer items than expected on average
`failure rate`	Too many runs or items failed
`not reporting`	The service sent nothing at all

hung is new in 0.3.0. A duration histogram only sees runs that finish, so a run stuck forever never shows up there. Every SDK now reports the age of the oldest unfinished run as a gauge, so the alert fires while the run is still hanging.

Jobs with no SDK at all

Since 0.3.0, runsignal exec wraps any command (a shell script, a crontab entry, a container in a Kubernetes CronJob) as one monitored run:

runsignal exec --workload nightly-export --service billing -- ./export.sh

Outcome: exit code 0 is a success, anything else a failure with its exit code.
Signals: stop signals are forwarded and count as cancelled, and the command's exit code passes through.
Packaging: it's a 10 MB native executable with no runtime dependency. Copy it into any image with COPY --from=ghcr.io/gokhanertogan/runsignal.
Kubernetes: runsignal discover kubernetes writes the manifest straight from your CronJobs, including schedules, time zones and activeDeadlineSeconds.

Design choices

OpenTelemetry-native. These are plain OTel instruments, so the data goes to any OTLP backend. Alert provisioning targets New Relic and Prometheus today.
One contract, six SDKs. A shared conformance test defines the exact metrics a scenario must produce, and every SDK runs it in CI. An SDK that drifts fails its own build.
Cheap on the hot path. Recording a run doesn't allocate in .NET and Go (enforced by tests), and costs well under a microsecond in Rust and Node.
Idempotent provisioning. provision changes only what changed and never touches conditions it didn't create. Run it on every deploy.

Status

It's MIT-licensed and pre-1.0. Every alert kind has fired on a real New Relic account, and the Prometheus rules are covered by promtool test rules.

GitHub: github.com/gokhanertogan/RunSignal
Packages:
- NuGet
- Maven Central
- PyPI
- npm
- Go
- crates.io