What happens to the cron job that's running when you deploy? Graceful shutdown for scheduled work

The nightly export job took 40 minutes. The team deployed at 02:15. Kubernetes sent SIGTERM, waited the default 30 seconds, then sent SIGKILL. The export stopped halfway, a partial file got uploaded, and the downstream system imported half a day of data as if it were complete.

Nobody had ever thought about what a deploy does to a job that's already running. Most teams haven't.

TL;DR

• On deploy or scale-down, your process gets SIGTERM, then SIGKILL after a grace period (30 s by default in Kubernetes, 10 s in Docker).
• If you don't handle SIGTERM, long jobs are cut off at a random point.
• Handle it by stopping new work, finishing or checkpointing the current item, and exiting cleanly.
• Make long jobs resumable and outputs atomic, so a kill is a delay rather than corruption.
• My opinion: "don't deploy while the job runs" is not a strategy. Jobs should survive being interrupted.

The shutdown sequence

deploy starts
  -> SIGTERM to your process
  -> grace period (terminationGracePeriodSeconds / docker stop -t)
  -> SIGKILL (cannot be caught)

Two common surprises:

• Your app may not receive SIGTERM at all. If the container runs sh -c "python job.py", the shell is PID 1 and may not forward signals. Use exec form (CMD ["python", "job.py"]) or a small init such as tini.
• Classic cron jobs on a VM are often unaffected by app deploys, but they get killed by reboots and by deploy scripts that restart everything. Check what your deploy actually does.

Step 1: catch SIGTERM and stop taking new work

Python:

import signal, sys

stopping = False

def on_term(signum, frame):
    global stopping
    stopping = True
    print("SIGTERM received, finishing current item", flush=True)

signal.signal(signal.SIGTERM, on_term)

for batch in batches():
    if stopping:
        save_checkpoint(batch.start_id)
        sys.exit(0)
    process(batch)
    save_checkpoint(batch.end_id)

Node:

let stopping = false;
process.on('SIGTERM', () => { stopping = true; });

for await (const batch of batches()) {
  if (stopping) { await saveCheckpoint(batch.startId); process.exit(0); }
  await processBatch(batch);
  await saveCheckpoint(batch.endId);
}

The loop checks a flag between units of work. It never tries to abort in the middle of one, because that's where partial writes come from.

Step 2: give it enough time

Set the grace period to longer than your largest unit of work, not longer than the whole job:

spec:
  terminationGracePeriodSeconds: 120

If one batch takes 20 seconds, 120 seconds is plenty. If the job is one giant 40-minute transaction, no grace period is long enough. That's the real bug, so split it into batches.

Step 3: make the job resumable

A checkpoint is just "where did I get to":

CREATE TABLE job_checkpoints (
  job_name   text PRIMARY KEY,
  run_key    text NOT NULL,       -- e.g. the scheduled date
  last_id    bigint NOT NULL,
  updated_at timestamptz NOT NULL DEFAULT now()
);

On start, the job reads the checkpoint for today's run_key and continues from last_id. A killed run becomes "slightly late", not "half done". Processing the same row twice around the checkpoint is still possible, so per-item work needs to be idempotent. The at-least-once post covers how.

Step 4: make outputs atomic

Never write the final file or table in place:

tmp=$(mktemp /data/export.XXXXXX)
generate_export > "$tmp"
mv "$tmp" /data/export-$(date +%F).csv     # rename is atomic on the same filesystem

For object storage, upload under a temporary key and write a small "manifest" or "_SUCCESS" marker last. Consumers only read exports that have the marker. For databases, load into a staging table and swap in one transaction.

Step 5: separate jobs from web deploys

If jobs run inside the same process as your web app, every web deploy interrupts them. Options:

• run scheduled work in its own deployment or worker pool, deployed less often
• trigger jobs over HTTP from an outside scheduler, so each run is a short request instead of a long-lived process inside your app (the setTimeout post compares these options)
• on Kubernetes, use CronJob, where each run is its own pod and app deploys don't touch it

With HTTP-triggered jobs, from cron and curl or a scheduler such as delay.run, the request should hand the work to a durable worker and return quickly. A deploy then only affects the worker, which handles SIGTERM as above.

How to know a run was cut off

The scary part of killed jobs is silence. Two cheap signals:

• log a clear "finished" line with counts, and alert when a run starts without finishing, which is exactly what Heartbeat checks: how to know when a scheduled job didn't run describes
• keep job output in logs you can read after the pod is gone. How to log cron job output so you can debug it later has the setup

Test it on purpose: start the job and send kill -TERM <pid> halfway through. Then run it again and check that it resumes and that the output is complete.

My take

Every scheduled job will be killed mid-run eventually: deploys, node drains, OOM kills, spot instance reclaims. The question is only whether that's a non-event or a data incident. Batches, checkpoints and atomic outputs make it a non-event, and they cost less than one incident to build.

Over to you

Do you pause deploys while important jobs run, or make jobs interruption-safe? I'd like to hear how people handle jobs that genuinely can't be split, like one huge third-party export.