delay.run

Posted on Nov 7 Edited on Nov 7

Your background jobs need SLOs too. 'It eventually ran' isn't good enough

#sre #backend #postgres #opinion

Most teams can tell you their API's p95 latency. Few can tell you how late their background jobs run.

Ask "how long after a user signs up does the welcome email go out?" and you'll get guesses. "Pretty quickly." "A few seconds?" "Should be instant." Then one day the queue backs up, welcome emails arrive four hours late, password resets take twenty minutes, and nobody knows when it started, because nobody was measuring.

My opinion: background jobs deserve service level objectives just as much as your API does. Not because SLOs are fashionable, but because users experience job latency directly, and "it eventually ran" hides a lot of pain.

Users feel job latency, they just can't see the job

Look at what typically runs in the background:

password reset and magic-link emails
payment confirmations and receipts
file exports and report generation
notifications ("someone replied to you")
webhooks to your customers' systems

Every one of these has an implicit promise. A password reset that arrives in 30 seconds is fine; one that arrives in 15 minutes is a broken login flow. A webhook delivered an hour late may be worse than useless if the receiving system made decisions in the meantime.

From the user's perspective, there's no difference between "the API was slow" and "the job was slow". There's only "the product was slow".

The metric that matters: lateness

For background work, the most useful single number isn't how long a job takes to run. It's how long after it should have started it actually started:

lateness = started_at - scheduled_for

For immediate jobs, scheduled_for is when it was enqueued. For delayed jobs, it's the time it was due. Either way, lateness captures queue backlog, worker shortages, scheduler drift and crashed workers in one number.

If you store both timestamps (and you should), measuring it is a query. In Postgres:

SELECT
  job_type,
  count(*) AS jobs,
  percentile_cont(0.50) WITHIN GROUP (
    ORDER BY extract(epoch FROM started_at - scheduled_for)) AS p50_s,
  percentile_cont(0.95) WITHIN GROUP (
    ORDER BY extract(epoch FROM started_at - scheduled_for)) AS p95_s,
  max(extract(epoch FROM started_at - scheduled_for)) AS max_s
FROM jobs
WHERE started_at > now() - interval '24 hours'
GROUP BY job_type
ORDER BY p95_s DESC;

The first time you run this, something will surprise you. It always does.

What an SLO for jobs looks like

Keep it simple and tied to the user promise:

Password reset emails: 99% start within 10 seconds of being requested.
Outgoing webhooks: 99% of first attempts start within 30 seconds of the event.
Scheduled reminders: 99% start within 60 seconds of their due time.
Nightly exports: 100% complete by 06:00 local time.

Different job types get different targets because they carry different promises. That's the whole point: lumping everything into one queue with one vague "should be fast" expectation is how your marketing digest ends up delaying your password resets.

"But our jobs are async, latency doesn't matter"

Async means the user doesn't wait on the HTTP request. It doesn't mean they don't wait. They wait in their inbox, on the confirmation screen, in their own system for your webhook. Async moved the waiting; it didn't remove it.

"But it's extra work"

It's two timestamps and a query. If your job system doesn't store scheduled_for and started_at, add them; you'll want them during the next incident anyway. The alerting can start crude:

-- anything due more than 5 minutes ago that hasn't started?
SELECT job_type, count(*)
FROM jobs
WHERE started_at IS NULL
  AND scheduled_for < now() - interval '5 minutes'
GROUP BY job_type;

Run that every minute and alert if it returns rows for a critical job type. That single check would have caught most of the background-job incidents I've seen discussed in postmortems, usually hours earlier than the customer complaints did.

Don't forget the jobs that never start

Lateness only measures jobs that eventually ran. A job that was never enqueued at all (a bug in the code that schedules it, a cron entry deleted during a migration) produces no row and no lateness. For critical recurring work, pair the lateness query with a simple expectation: "at least one nightly export completed in the last 26 hours". Absence is a failure mode too.

What SLOs change in practice

Once lateness has a target, decisions get easier:

Separate queues for urgent and bulk work stop being a nice-to-have; the numbers tell you when one is starving the other.
Capacity planning becomes concrete: "at Monday 09:00 the p95 for reminders hits 4 minutes; we need more workers or spread-out scheduling".
Deploys can be judged: did lateness spike during the rollout because workers were restarted all at once?
Incidents get a clear start time from the graph rather than from the first support ticket.

Where we stand

We care about this a lot. Delivering work at the right time is the whole job of delay.run, and "did it run on time?" is the question we think every scheduled or background task should be able to answer. But you don't need any particular tool for this opinion to apply. Add the timestamps. Write the query. Pick a target for the three job types users would miss most.

"It eventually ran" was never really a reliability standard. It just felt like one because nobody was measuring.

DEV Community