DEV Community: Farooq Hussain

Your Kubernetes Deployments Are Killing Sidekiq Jobs

Farooq Hussain — Wed, 21 Oct 2026 12:00:00 +0000

How a preStop hook, one Unix signal, and the right grace period stopped Kubernetes from killing our long-running Sidekiq jobs in the middle of their work.

Every deployment looked clean. Green pipeline, new pods up, old pods gone. And every deploy, a handful of background jobs quietly died halfway through.

We ran a big part of our product on Sidekiq workers in Kubernetes. Short jobs were fine. The long ones, the jobs that needed a few minutes to finish, were the problem. If a deploy landed while one of them was running, it got cut off. Sometimes Sidekiq retried it, and it finished later. Sometimes it left work half done, and someone had to clean up by hand.

Nothing was "broken." Kubernetes and Sidekiq were both doing exactly what they were told. They just weren't told the same thing.

This article walks through why it happens, the fix we put in place, and every piece of config you need to do the same: the preStop script, the Deployment, the Sidekiq config, the Dockerfile, and how to test it before you trust it.

The symptom

If you're running Sidekiq on Kubernetes with default settings, you've probably seen some of these without connecting them to deploys:

Jobs show up in the Retry set right after a release, with errors that don't make sense for the job.
Long jobs that "ran twice": emails sent twice, records processed twice, partial exports.
Jobs that were running before the deploy and then simply vanished. No retry, no error, no trace.
Sidekiq logs on the old pod that end abruptly, without the usual Bye! shutdown line.

The pattern is the giveaway: it only happens around deploys, scale-downs, and node drains. In other words, it happens whenever Kubernetes stops a pod.

What actually happens when Kubernetes stops a pod

When a rolling update replaces a pod, Kubernetes runs this sequence:

Marks the pod as Terminating and removes it from Service endpoints.
Runs the container's preStop hook, if one is defined, and waits for it to finish.
Sends SIGTERM to PID 1 in the container.
Waits until terminationGracePeriodSeconds runs out.
Sends SIGKILL. Whatever is still running dies on the spot.

Two details matter a lot here:

The default terminationGracePeriodSeconds is 30 seconds.
The grace period clock starts at step 1, not after the preStop hook. The hook's time is spent out of the same budget.

Here's the timeline with defaults:

t=0s     Pod marked Terminating
t=0s     preStop hook (none by default)
t=0s     SIGTERM -> Sidekiq starts shutting down
t=25s    Sidekiq shutdown timeout expires, unfinished jobs pushed back to Redis
t=30s    SIGKILL if the process is still alive

What Sidekiq does with signals

Sidekiq listens for a few Unix signals. Two of them matter for deploys:

TSTP (quiet): stop fetching new jobs. Keep working on the jobs already running.
TERM (shutdown): stop fetching, wait up to timeout seconds for running jobs, push unfinished jobs back to Redis, and exit.

Older Sidekiq versions used USR1 for quiet. Since Sidekiq 5, it's TSTP. The default shutdown timeout is 25 seconds.

So, out of the box, Kubernetes sends SIGTERM, and Sidekiq gives running jobs 25 seconds, pushes the rest back to the queue, and exits. Kubernetes allows 30 seconds total before SIGKILL.

We were running both defaults. For our long jobs, neither number was anywhere close.

Here's what happens to a job that's still running at the deadline:

Best case: Sidekiq pushes it back to Redis. Another worker picks it up and starts again from zero. If the job isn't idempotent, it does half its work twice.
Worst case: the process gets SIGKILL before it can push the job back. With open-source Sidekiq, which uses basic fetch, that job is simply gone. Sidekiq Pro's reliable fetch can recover it, but most teams run the open-source version.

The fix: quiet first, then wait, then stop

The idea we landed on is simple. Before Kubernetes sends SIGTERM, tell Sidekiq to go quiet so it stops taking new work. Then give the jobs it already has enough time to finish. Only then let the normal shutdown happen.

The new timeline looks like this:

t=0s      Pod marked Terminating
t=0s      preStop: kill -TSTP 1      -> Sidekiq stops fetching new jobs
t=0s..N   preStop waits while running jobs finish (new pod already takes new work)
t=N       preStop exits
t=N       SIGTERM -> Sidekiq has nothing left to do, exits in seconds
t<600s    Pod gone. No SIGKILL, no lost jobs.

It takes four pieces to make that work:

A preStop script that sends TSTP and waits for running jobs to finish.
A terminationGracePeriodSeconds longer than your longest job.
A Sidekiq config with a sensible shutdown timeout.
A container where Sidekiq is actually PID 1, so the signal reaches it.

Let's go through each one.

Piece 1: the preStop drain script

This script goes into your app image, for example at bin/sidekiq-prestop.sh:

#!/bin/sh
# bin/sidekiq-prestop.sh
#
# Called by the Kubernetes preStop hook.
# 1. Puts Sidekiq into quiet mode (stop fetching new jobs).
# 2. Waits until this pod has no busy jobs, or until MAX_WAIT runs out.
#
# MAX_WAIT must be lower than terminationGracePeriodSeconds minus the
# Sidekiq shutdown timeout, or Kubernetes will SIGKILL us mid-drain.

set -u

MAX_WAIT="${SIDEKIQ_DRAIN_SECONDS:-540}"
INTERVAL=5

log() {
  # Write to PID 1's stdout so the message shows up in `kubectl logs`.
  echo "[prestop] $(date -u +%H:%M:%S) $*" > /proc/1/fd/1
}

log "sending TSTP to Sidekiq (quiet mode)"
kill -TSTP 1

waited=0
while [ "$waited" -lt "$MAX_WAIT" ]; do
  # Prints "<quiet> <busy>", for example "true 1".
  # Prints nothing if the check fails (Redis down, process not found).
  state=$(bundle exec ruby -e '
    require "sidekiq/api"
    me = Sidekiq::ProcessSet.new.find { |p| p["hostname"] == ENV["HOSTNAME"] }
    puts "#{me["quiet"]} #{me["busy"].to_i}" if me
  ' 2>/dev/null)

  quiet=$(echo "$state" | cut -d" " -f1)
  busy=$(echo "$state" | cut -d" " -f2)

  if [ "$quiet" != "true" ]; then
    # No heartbeat since TSTP yet, or the check failed. Don't trust busy.
    log "waiting for a fresh heartbeat, waited ${waited}s of ${MAX_WAIT}s"
  elif [ "$busy" = "0" ]; then
    log "no busy jobs after ${waited}s, letting shutdown continue"
    exit 0
  else
    log "still ${busy} busy job(s), waited ${waited}s of ${MAX_WAIT}s"
  fi

  sleep "$INTERVAL"
  waited=$((waited + INTERVAL))
done

log "drain timeout reached with jobs still running, continuing shutdown"
exit 0

A few notes on this script:

kill -TSTP 1 only works because Sidekiq is PID 1 in the container. More on that in piece 4.
The busy check uses Sidekiq's own API. Each Sidekiq process registers in Redis with its hostname, and in Kubernetes the hostname is the pod name, so ENV["HOSTNAME"] finds this exact pod.
It waits for a fresh heartbeat. The busy count comes from Sidekiq's heartbeat in Redis, which is a few seconds old. Right after TSTP, it can still say 0 while a job that was just fetched is running. So the script only trusts busy once the process reports quiet=true, which means a heartbeat ran after TSTP. If the check fails, it keeps waiting instead of assuming the pod is idle.
It exits as soon as the pod is idle. A quiet pod with nothing running shuts down in about five seconds. A pod in the middle of a long job waits for it. You don't pay the full wait on every pod.
It always exits 0. If the hook fails, Kubernetes records a FailedPreStopHook event and moves straight on to SIGTERM. Exiting cleanly keeps the shutdown predictable.
Logs go to /proc/1/fd/1. The preStop output doesn't show up in kubectl logs by default. Writing to PID 1's stdout puts the drain messages right next to Sidekiq's own logs, which makes debugging much easier.

Make it executable in your image:

COPY bin/sidekiq-prestop.sh /app/bin/sidekiq-prestop.sh
RUN chmod +x /app/bin/sidekiq-prestop.sh

The one-line version

If you don't want the Ruby check, there's a simpler version:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "kill -TSTP 1 && sleep 540"]

It works, but it's blunt. Every pod waits the full 540 seconds, even an idle one, so rollouts and scale-downs get slow. It's fine to start with. The script version is better once you run more than a couple of replicas.

Piece 2: the grace period

spec:
  terminationGracePeriodSeconds: 600

The rule is:

terminationGracePeriodSeconds > preStop max wait + Sidekiq shutdown timeout + buffer

With the numbers in this article:

600 > 540 (SIDEKIQ_DRAIN_SECONDS) + 25 (Sidekiq :timeout:) + 35 (buffer)

Pick the drain time from your real job durations, not from a guess. If your longest job can run for 9 minutes, a 540-second drain covers it. If you have a job that runs for an hour, a preStop hook is the wrong tool. Split that job into smaller chunks instead.

Piece 3: the Sidekiq config

# config/sidekiq.yml
:concurrency: 10
:timeout: 25
:queues:
  - [critical, 3]
  - [default, 2]
  - [low, 1]

By the time SIGTERM arrives, the preStop hook has already drained the pod. The 25-second :timeout: only has to cover whatever is left, which is usually nothing. Keep it well under the grace period minus the drain time.

Piece 4: make sure Sidekiq is PID 1

This is the gotcha that quietly breaks the whole thing. kill -TSTP 1 sends the signal to whatever process is PID 1. If that's a shell, Sidekiq never hears it.

This is wrong:

# Shell form: /bin/sh is PID 1, Sidekiq is a child process
CMD bundle exec sidekiq -C config/sidekiq.yml

This is right:

# Exec form: Sidekiq is PID 1 and receives signals directly
CMD ["bundle", "exec", "sidekiq", "-C", "config/sidekiq.yml"]

If you need an entrypoint script for setup work, end it with exec so Sidekiq replaces the shell:

#!/bin/sh
# docker-entrypoint.sh
set -e

# ...any setup you need (wait for Redis, load secrets, etc.)

exec "$@"

You can check what's running as PID 1 inside a pod:

kubectl exec -it deploy/sidekiq-worker -- cat /proc/1/cmdline | tr '\0' ' '

If the output starts with sidekiq or bundle exec sidekiq, you're good. If it starts with /bin/sh, fix that first.

The full Deployment

Here's everything put together:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sidekiq-worker
  labels:
    app: sidekiq-worker
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: sidekiq-worker
  template:
    metadata:
      labels:
        app: sidekiq-worker
    spec:
      terminationGracePeriodSeconds: 600
      containers:
        - name: sidekiq
          image: registry.example.com/app:1.42.0
          command: ["bundle", "exec", "sidekiq", "-C", "config/sidekiq.yml"]
          env:
            - name: SIDEKIQ_DRAIN_SECONDS
              value: "540"
            - name: REDIS_URL
              valueFrom:
                secretKeyRef:
                  name: app-secrets
                  key: redis-url
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "/app/bin/sidekiq-prestop.sh"]
          resources:
            requests:
              cpu: "250m"
              memory: "512Mi"
            limits:
              memory: "1Gi"

Two things are worth calling out:

maxUnavailable: 0 with maxSurge: 1 means Kubernetes starts a new worker before an old one begins draining. The queue keeps moving during the deploy, and you never drop below your normal worker count.
Watch your liveness probe. If you have one that checks Sidekiq, make sure a quiet, draining pod still passes it. Otherwise Kubernetes may restart the container in the middle of the drain.

Protect it during node drains too

Deploys aren't the only time pods get stopped. Cluster upgrades and kubectl drain evict pods as well. A PodDisruptionBudget stops Kubernetes from evicting too many workers at once:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: sidekiq-worker-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: sidekiq-worker

Evicted pods still run the preStop hook and respect the grace period, so the same drain logic applies. The PDB just makes sure they drain one at a time.

How to test it before you trust it

Don't wait for the next real deploy to find out if this works. Test it on purpose in a non-production environment.

1. Add a slow test job:

# app/jobs/slow_test_job.rb
class SlowTestJob
  include Sidekiq::Job

  def perform(seconds = 120)
    Sidekiq.logger.info "SlowTestJob starting, will run #{seconds}s"
    seconds.times { sleep 1 }
    Sidekiq.logger.info "SlowTestJob finished cleanly"
  end
end

2. Enqueue it:

kubectl exec -it deploy/sidekiq-worker -- \
  bundle exec rails runner 'SlowTestJob.perform_async(120)'

3. Find the pod that picked it up, then delete it:

kubectl get pods -l app=sidekiq-worker
kubectl delete pod sidekiq-worker-7d9f8c6b5-abcde

4. Watch the logs of the terminating pod:

kubectl logs -f sidekiq-worker-7d9f8c6b5-abcde

You should see something like this:

[prestop] 10:14:02 sending TSTP to Sidekiq (quiet mode)
INFO: Received TSTP, no longer accepting new work
[prestop] 10:14:07 still 1 busy job(s), waited 0s of 540s
[prestop] 10:14:12 still 1 busy job(s), waited 5s of 540s
...
INFO: SlowTestJob finished cleanly
[prestop] 10:15:57 no busy jobs after 115s, letting shutdown continue
INFO: Shutting down
INFO: Bye!

If you see SlowTestJob finished cleanly and Bye! before the pod disappears, the drain works. If the logs stop abruptly with no Bye!, the pod was killed. Check the grace period and the PID 1 setup.

5. Check events for hook failures:

kubectl get events --field-selector reason=FailedPreStopHook

An empty result is what you want.

Jobs still need to be idempotent

The preStop hook handles planned shutdowns: deploys, scale-downs, node drains. It doesn't handle everything. Nodes crash. Containers get OOM-killed. Someone runs kubectl delete pod --force --grace-period=0. In all of those cases, no hook runs.

So the real safety net is still a job that can safely run twice. A simple pattern is to record that the work is done and check before doing it again:

class SendInvoiceJob
  include Sidekiq::Job

  def perform(invoice_id)
    invoice = Invoice.find(invoice_id)
    return if invoice.sent_at.present?   # already done, safe to skip

    InvoiceMailer.with(invoice: invoice).deliver_now
    invoice.update!(sent_at: Time.current)
  end
end

This still leaves one gap. If the worker dies after the email goes out but before sent_at is saved, the retry sends the invoice again. No hook can close that window. For a plain notification, a rare duplicate may be fine. For anything tied to money, save a delivery record with a stable key before sending, pass that key to the provider if it supports idempotency, and check the record before retrying. Test it by killing the worker between the send and the database update.

For long jobs that loop over many records, save progress as you go, so a retry picks up where it stopped instead of starting over:

class BackfillJob
  include Sidekiq::Job

  def perform(batch_id)
    batch = Batch.find(batch_id)

    batch.items.where(processed: false).find_each do |item|
      process(item)
      item.update!(processed: true)   # checkpoint after each item
    end
  end
end

The drain makes deploys clean. Idempotency makes everything else survivable.

Checklist

Before you call this done:

Sidekiq runs as PID 1 (exec form CMD, or exec in the entrypoint)
preStop hook sends TSTP and waits for busy jobs
terminationGracePeriodSeconds > drain time + Sidekiq timeout + buffer
Drain time is based on your longest real job
maxUnavailable: 0 so new workers start before old ones drain
PodDisruptionBudget for node drains
Liveness probe still passes while the pod is quiet
Tested with a slow job and a manual pod delete
Jobs are idempotent for the cases no hook can catch

What changed

After this went in, deploys stopped killing jobs. Long jobs finished on the old pod while new pods picked up new work. Nobody had to check the Retry set after a release, and nobody had to clean up half-processed work by hand.

The whole fix is a shell script, a couple of numbers, and one signal. The hard part was noticing that Kubernetes and Sidekiq each had a perfectly reasonable idea of "shutting down gracefully," and the two didn't match.

Updated: thanks to Raza Hussain for spotting the stale heartbeat in the busy check, and to Tim Urista for pointing out the crash window in the invoice example. Both are fixed above.

I write up real production fixes like this one. If your team runs Sidekiq on Kubernetes, go check your terminationGracePeriodSeconds right now. If it still says 30, you're probably losing jobs.

I Built a Kubernetes Incident Manager That Explains Broken Pods With a Local AI Model

Farooq Hussain — Mon, 19 Oct 2026 10:19:28 +0000

It's 3 AM. Your phone buzzes. A pod is in CrashLoopBackOff.

You open your laptop, run kubectl describe, scroll through events, pull the logs from the crashed container, and try to work out what actually went wrong. Ten minutes later you find it: a missing Secret. Or a memory limit. Or a database that isn't reachable.

AI is good at exactly this kind of reading. But most teams can't paste production logs into a hosted AI chat. Those logs are full of connection strings, tokens, customer emails and internal IPs.

So I built a small incident manager that runs inside the cluster and explains broken pods with an open-weight model running on my own machine. Nothing leaves the infrastructure, and it costs nothing per incident.

Code: github.com/farooqhussain82/incident-explainer

What it does

Every 15 seconds it lists the pods in the cluster.
When one breaks (crash loop, OOMKilled, image pull error, missing Secret or ConfigMap, failing readiness probe, unschedulable), it opens an incident.
It collects the evidence: a kubectl describe-style summary, events, and logs, including the logs of the previous crashed run.
It redacts secrets from that text.
It asks a local model in Ollama (Qwen 3 8B by default) what went wrong and what to check next.
When the pod is healthy for four checks in a row, the incident resolves itself.

There is a small web page for the incidents, and it can post new incidents to Slack, Teams or Mattermost through an incoming webhook.

The demo: four pods, four different failures

To test it, I made a throwaway kind cluster and deployed four pods that break in four common ways. All four were detected and explained on my laptop (Apple M4, 16 GB RAM) in 20 to 45 seconds each.

Pod	What I broke	What the model said
`payments-api`	Refers to a Secret that doesn't exist	The pod can't start because Secret `payments-secrets` is missing. Check it exists in the namespace.
`report-worker`	Memory limit of 64Mi, process uses far more	OOMKilled, exit code 137, because usage goes over the 64Mi limit.
`checkout-web`	App can't reach its database	Connection refused to PostgreSQL on port 5432. Check the database is running and reachable.
`orders-db`	Postgres started without a password	Postgres won't initialise because `POSTGRES_PASSWORD` is not set.

Each answer quotes the exact log or event lines it is based on, so you can check it in seconds instead of trusting it blindly:

Then I created the missing Secret for payments-api. About a minute later the incident moved to "Recently resolved" on its own.

One detail I'm happy with: an out-of-memory container also shows CrashLoopBackOff in Kubernetes, because it keeps restarting. The detector looks at the last termination reason and reports it as OOMKilled, which is the real cause and the thing you need to fix.

Why an open, local model

This was the main reason for the project, not an extra.

The data stays inside. The model runs next to the cluster. Logs are never sent to a third party.
No bill per incident. A crash loop that fires 200 times a night doesn't turn into an API invoice.
You choose the model. Qwen 3 8B runs fine on a laptop. Use a bigger one on a GPU node, or one tuned on your own runbooks. It's one environment variable.

I also made the model answer in a fixed JSON format (summary, likely cause, evidence, next steps, confidence). That keeps the output predictable enough to show in a UI and send to chat.

Redaction, even though the model is local

"Local" should not mean "careless". Before anything reaches the model, the text is cleaned:

private keys, AWS keys, JWTs and bearer tokens
passwords in URLs like postgres://user:password@host
values of password=, token=, api_key= and similar
email addresses and single IP addresses

In the checkout-web demo, the log contained a database URL with a password, a JWT and IP addresses. The model only saw this:

connecting to postgres://[REDACTED-URL-CREDS]@[REDACTED-IP]:5432/checkout
Authorization: Bearer [REDACTED-JWT]

Each incident page has a "Show exactly what the model saw" section, so anyone can check.

One thing I had to fix during testing: my first version also masked 0.0.0.0/0 in an infrastructure change. That hid the most important fact, which was a database being opened to the whole internet. Now CIDR ranges and 0.0.0.0 are kept, and only single host IPs are masked.

Security of the thing itself

A tool that reads every pod in your cluster has to be boring and safe:

Read-only RBAC: only get, list and watch on pods, pod logs and events. I checked it with kubectl auth can-i: it cannot read Secrets and cannot delete pods.
Env var values are never collected, only their names and where they come from (for example, "comes from Secret payments-secrets").
A 13 MB FROM scratch image: no shell, no package manager, runs as non-root with a read-only filesystem and all Linux capabilities dropped.

It's written in Go with only the standard library, so there are no third-party packages to keep patched.

Honest limits

It runs as one replica and stores incidents in a JSON file. Fine for a team cluster, not for a huge fleet.
It explains, it does not fix. Any action is still a human decision.
An 8B model gets the common failures right, but small details can be off. In one run it said the OOM test "writes to /dev/zero", when tail /dev/zero actually reads from it. The quoted evidence makes slips like that easy to spot.
Redaction is pattern based. It catches the common shapes of secrets, not every possible one. There's a -dry-run mode that prints exactly what would be sent.

Try it

You need Docker, kind, kubectl, Go and Ollama.

ollama pull qwen3:8b
git clone https://github.com/farooqhussain82/incident-explainer
cd incident-explainer
make kind-up    # local cluster
make deploy     # build the image, load it, apply the manifests
make demo       # four broken pods
make ui         # open http://localhost:8080

It also works without a cluster, on a single log file or an infrastructure change plan:

kubectl logs my-pod --previous | ./bin/incident-explainer explain -

What I'd add next

A Helm chart and an option to run Ollama inside the cluster.
Grouping, so ten replicas of one broken Deployment become one incident, not ten.
Using node events and recent rollouts as extra evidence.

If you run Kubernetes and you've ever read crash loop logs at 3 AM, I'd love to hear what failures you'd want this to understand. And if you try it, tell me what it got wrong.