DEV Community: Farooq Hussain

Stop Running Database Migrations in initContainers

Farooq Hussain — Tue, 10 Nov 2026 08:00:00 +0000

How we moved Flyway into a Helm hook that runs once, before the deploy, and works the same in Argo CD

Our app ran 6 replicas. On every deploy, 6 pods tried to migrate the same MySQL database.

We had put Flyway in an initContainer. It looked clean: the migration runs before the app starts, so the app always sees the right schema. It's a common pattern, and for a single pod, it's fine.

With 6 replicas, it wasn't.

Here's why we moved away from it, the Helm hook we replaced it with, and what happens when a migration fails.

We didn't want to write this from memory. So we rebuilt the setup on a local kind cluster with MySQL, Helm, and Argo CD, reproduced the problems we'd had in production, and then tested the fix. Every YAML file and every error message you'll see below comes from that test setup.

Our example app is a Spring service on MySQL. Nothing here is Spring-specific, though. It works for any app that uses Flyway.

Why the initContainer had to go

1. Every replica ran the migration.

Not once per deploy. Once per pod. And it wasn't only deploys: every restart, every rescheduled pod, and every HPA scale-up ran Flyway again.

Flyway protects itself with a database lock, so the pods didn't corrupt anything. They waited for each other instead. That leads to the second problem.

2. New pods waited on the database lock.

If one pod is running a slow migration, every other new pod waits for the lock before its app can start. That includes pods the HPA adds because traffic just went up.

In our test setup, we reproduced it: a pod with nothing to migrate started 5 seconds after another pod began a 40-second migration. It waited 36 seconds before it could finish, even though it had no work to do.

Flyway itself isn't slow. With nothing to migrate, it finished in about a second in our tests. The problem is that scaling now depends on the database being free.

3. Failures were hard to see.

When a migration failed, we didn't get one clear "migration failed" step. We got pods stuck in Init:CrashLoopBackOff. Someone had to find the right pod and read the init container logs to see why.

The fix: one Job, before the deploy

We wanted the migration to run once per release, not once per pod. Kubernetes already has a tool for that. It's called a Job.

Helm can run a Job as a hook before it touches anything else in the release. If the Job fails, Helm stops the upgrade, and the old pods keep running.

The setup has three parts:

A small migration image with the SQL files inside
A Job with Helm hook annotations
The app, with its own Flyway turned off

Part 1: The migration image

Bake the SQL files into an image built on the official Flyway image:

FROM flyway/flyway:11
COPY sql /flyway/sql

migrations/
  Dockerfile
  sql/
    V1__create_users.sql
    V2__create_orders.sql
    V3__add_order_status.sql

Build it and tag it with the same version as the app. The migrations for app version 1.4.0 live in migration image 1.4.0. You don't need volume mounts or ConfigMaps, and you always know exactly which SQL ran in production.

Part 2: The migration Job as a Helm hook

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ .Release.Name }}-db-migrate
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-weight: "0"
    helm.sh/hook-delete-policy: before-hook-creation
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 600
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: flyway
          image: "{{ .Values.migrations.repository }}:{{ .Values.migrations.tag }}"
          args: ["migrate"]
          env:
            - name: FLYWAY_URL
              value: {{ .Values.db.url | quote }}
            - name: FLYWAY_USER
              valueFrom: {secretKeyRef: {name: {{ .Values.db.secretName }}, key: username}}
            - name: FLYWAY_PASSWORD
              valueFrom: {secretKeyRef: {name: {{ .Values.db.secretName }}, key: password}}
            - name: FLYWAY_CONNECT_RETRIES
              value: "10"

What each part does:

pre-install,pre-upgrade: the Job runs before Helm creates or updates anything else in the release, on the first install and on every upgrade.
before-hook-creation: Helm deletes the old Job only when the next release starts. A failed Job stays in the cluster, so you can still read its logs. Don't use hook-succeeded alone; you'll want those logs.
backoffLimit: 0: a failed migration fails once and stops. Retrying a half-applied migration on MySQL won't fix it (more on that below).
activeDeadlineSeconds: a stuck migration doesn't hang the release forever.
FLYWAY_CONNECT_RETRIES: if the database is briefly unreachable, Flyway retries the connection instead of failing straight away.

One thing to check: the Secret with the DB credentials must exist before the hook runs. In our test setup, on a fresh install where the Secret didn't exist yet, the Job sat in CreateContainerConfigError until Helm timed out. The app was never deployed. Create the Secret outside the app chart, or make sure it exists before the release.

Part 3: The app stops migrating itself

Spring Boot runs Flyway on startup by default when Flyway is on the classpath. If you leave that on, you're back to every pod migrating. Turn it off in the Deployment:

env:
  - name: SPRING_FLYWAY_ENABLED
    value: "false"

Now your app just connects. By the time it starts, the database is already at the right version.

What a deploy looks like now

The order with helm upgrade --wait is:

Helm creates the migration Job.
Flyway applies any new migrations and exits.
Only after the Job succeeds does Helm update the Deployment.
The rolling update starts, and the new pods start straight away.

We checked this on the test cluster. On the first install, the Deployment was created only after the Job had completed. On an upgrade with a new V3 migration and a new app version, V3 was applied first, then the pods rolled.

New pods from the HPA don't touch Flyway at all anymore.

When a migration fails

This is the part that matters most. So we broke a migration on purpose:

-- V4__broken.sql
ALTER TABLE orderz ADD COLUMN note VARCHAR(50);

Helm stopped the upgrade:

Error: UPGRADE FAILED: pre-upgrade hooks failed:
resource Job/shop/webapp-db-migrate not ready. status: Failed

Helm never touched the Deployment. Our 3 test pods kept running the old version and kept serving traffic. That's the whole point.

Then we fixed the SQL and deployed again. It still failed:

ERROR: Validate failed: Migrations have failed validation
Detected failed migration to version 4 (broken).
Please remove any half-completed changes then run repair to fix the schema history.

That's MySQL, not a Flyway bug. MySQL commits DDL statements on its own, so Flyway can't roll back a failed migration. It marks it as failed in its history table and won't go any further until someone looks at it. Flyway's documentation says the same.

The recovery is:

Check what the failed migration changed and undo any half-applied part by hand.
Run flyway repair. On MySQL, this removes the failed row from the history table.
Fix the migration file and deploy again.

After the repair, our next deploy applied the fixed V4 and updated the app.

We'd call that manual step a feature. If a schema change fails on production, you want a person to look at it before anything retries.

The same chart in Argo CD

We deploy through Argo CD, and the chart didn't need to change.

Argo CD reads Helm hook annotations and maps pre-install and pre-upgrade hooks to its PreSync phase. We tested this on the same cluster: Argo CD ran the migration Job as a PreSync hook, and the Deployment was synced only after the Job succeeded.

With the broken migration, the sync failed and Argo CD retried it, while the app stayed on the old version with all pods ready. If you use automated sync, the retries won't fix a failed MySQL migration, so expect the same repair step as above.

So one chart works with Argo CD and with plain helm upgrade from a pipeline.

Summary

An initContainer runs your migration once per pod, not once per release.
Move it to a Job with helm.sh/hook: pre-install,pre-upgrade.
Bake the SQL into a Flyway image versioned with the app.
Turn off Flyway inside the app.
Use before-hook-creation so failed Jobs stay and you can read their logs.
On MySQL, a failed migration needs flyway repair before the next deploy.
Argo CD runs the same hook as PreSync, with no chart changes.

Our 6 replicas don't race for the database anymore. Each release runs its migrations once, and if that fails, nothing else changes.

How do you run migrations in your clusters today: in the app, in an initContainer, or as a Job?

Flyway Migration: From Legacy Database to CI/CD

Farooq Hussain — Wed, 28 Oct 2026 12:00:00 +0000

Most teams think of Flyway only when they need to manage ongoing database schema changes. But what if your project already has a complete, stable schema and you’re only now introducing version control? Or what if you want to incorporate your schema into a CI/CD pipeline for reliable, reproducible deployments?

In this guide, I’ll walk you through a real-world scenario where we used Flyway to baseline an existing MySQL database and then extended it to support version-controlled schema migrations in CI/CD, without disrupting production.

🚀 What is Flyway?

Flyway is a simple but powerful tool for managing database migrations. It applies versioned SQL scripts to your database and tracks them in a special table called flyway_schema_history.

Typical script names:

V1__initial_schema.sql
V2__add_orders_table.sql
V3__rename_column.sql

Flyway ensures these scripts run only once and in the correct order. It also integrates well with CI/CD tools, making it ideal for automated environments.

📘 Use Case: Existing Schema with No Migrations (Yet)

In our project, we had:

A production MySQL database with dozens of tables
No migration history
No current need for structural changes
A desire to version control the database and enable future migrations via CI/CD

🧱 Step 1: Baselining an Existing Schema

🔹 Dump the schema

mysqldump -u root -p --no-data your_database > V1__baseline_schema.sql

Clean the output: remove auto-increments, user references, and environment-specific SQL.

🔹 Create directory structure

mkdir -p db/sql
mv V1__baseline_schema.sql db/sql/

🔹 Configure flyway.conf

flyway.url=jdbc:mysql://your-db-host:3306/your_database
flyway.user=db_user
flyway.password=secret_password
flyway.locations=filesystem:db/sql
flyway.baselineOnMigrate=true

🔹 Run baseline

flyway baseline

This creates the flyway_schema_history table and marks version 1 as the current state without running any script.

🧪 Step 2: Add a Real Migration

After baselining, we added a real change in a new script

-- db/sql/V2__add_status_to_orders.sql
ALTER TABLE orders ADD COLUMN status VARCHAR(20) DEFAULT 'pending';

Then we ran:

flyway migrate

Result: Flyway applied version 2, recorded it in history, and our database was now safely versioned.

⚙️ Step 3: Integrate Flyway into CI/CD

You can integrate Flyway with GitHub Actions, GitLab CI, Bitbucket Pipelines, or Jenkins.

✅ Example GitHub Actions Workflow

name: Flyway Migration
on:
  push:
    branches: [ main ]

jobs:
  migrate:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3
      - name: Set up Java (Flyway needs it)
        uses: actions/setup-java@v3
        with:
          distribution: 'temurin'
          java-version: '17'
      - name: Install Flyway
        run: |
          curl -L https://repo1.maven.org/maven2/org/flywaydb/flyway-commandline/10.8.1/flyway-commandline-10.8.1-linux-x64.tar.gz | tar xvz
          sudo ln -s $PWD/flyway-10.8.1/flyway /usr/local/bin/flyway
      - name: Run Flyway Migrations
        env:
          FLYWAY_URL: jdbc:mysql://your-host:3306/your_database
          FLYWAY_USER: ${{ secrets.DB_USER }}
          FLYWAY_PASSWORD: ${{ secrets.DB_PASSWORD }}
        run: |
          flyway -url=$FLYWAY_URL \
                 -user=$FLYWAY_USER \
                 -password=$FLYWAY_PASSWORD \
                 -locations=filesystem:./db/sql \
                 migrate

📦 Benefits of This Approach

✅ You gain auditability for all database changes

✅ Developers can build local environments using the same scripts

✅ CI/CD ensures consistency across all environments

✅ Flyway’s validate command catches missing or changed scripts

✅ Schema management becomes part of your software delivery lifecycle

🧱 Potential Challenges

Problem → Fix

Flyway re-applies the baseline script → Use baselineOnMigrate=true and never modify a baseline
Passwords in config files → Use secrets and environment variables in CI/CD
Manual schema drift → Disable direct DB access in prod or run flyway validate periodically
Need rollback support → Write manual undo scripts or use point-in-time backups

🔁 Alternatives (and Why Flyway Wins)

🔹 Liquibase
✅ Pros: Advanced features, rollback support
❌ Cons: More complex, steeper learning curve

🔹 Django ORM / Alembic
✅ Pros: Built-in for app devs
❌ Cons: Not cross-language or DB-agnostic

🔹 Manual SQL or DB dumps
✅ Pros: Simple
❌ Cons: No versioning, no tracking, no automation

Flyway is a great middle-ground: powerful enough for teams, simple enough for solo devs, and perfect for CI/CD.

🧠 Final Thoughts

Flyway isn’t just for teams writing migrations every day. It’s also ideal for bringing an existing database under version control, giving your project a foundation for safe schema evolution.

We started with a baseline, added our first real migration, and then built it into our CI/CD pipeline — all in a day’s work.

If you’re managing an existing DB without schema tracking, now’s the time to fix that. Start with a baseline. Your future self will thank you.

📣 Let’s Connect!

Have you tried Flyway for baseline-only use cases or integrated it into your CI/CD pipeline? I’d love to hear your experience in the comments.

If you found this helpful, follow me here on DEV for more DevOps and database engineering content.

Your Kubernetes Deployments Are Killing Sidekiq Jobs

Farooq Hussain — Wed, 21 Oct 2026 12:00:00 +0000

How a preStop hook, one Unix signal, and the right grace period stopped Kubernetes from killing our long-running Sidekiq jobs in the middle of their work.

Every deployment looked clean. Green pipeline, new pods up, old pods gone. And every deploy, a handful of background jobs quietly died halfway through.

We ran a big part of our product on Sidekiq workers in Kubernetes. Short jobs were fine. The long ones, the jobs that needed a few minutes to finish, were the problem. If a deploy landed while one of them was running, it got cut off. Sometimes Sidekiq retried it, and it finished later. Sometimes it left work half done, and someone had to clean up by hand.

Nothing was "broken." Kubernetes and Sidekiq were both doing exactly what they were told. They just weren't told the same thing.

This article walks through why it happens, the fix we put in place, and every piece of config you need to do the same: the preStop script, the Deployment, the Sidekiq config, the Dockerfile, and how to test it before you trust it.

The symptom

If you're running Sidekiq on Kubernetes with default settings, you've probably seen some of these without connecting them to deploys:

Jobs show up in the Retry set right after a release, with errors that don't make sense for the job.
Long jobs that "ran twice": emails sent twice, records processed twice, partial exports.
Jobs that were running before the deploy and then simply vanished. No retry, no error, no trace.
Sidekiq logs on the old pod that end abruptly, without the usual Bye! shutdown line.

The pattern is the giveaway: it only happens around deploys, scale-downs, and node drains. In other words, it happens whenever Kubernetes stops a pod.

What actually happens when Kubernetes stops a pod

When a rolling update replaces a pod, Kubernetes runs this sequence:

Marks the pod as Terminating and removes it from Service endpoints.
Runs the container's preStop hook, if one is defined, and waits for it to finish.
Sends SIGTERM to PID 1 in the container.
Waits until terminationGracePeriodSeconds runs out.
Sends SIGKILL. Whatever is still running dies on the spot.

Two details matter a lot here:

The default terminationGracePeriodSeconds is 30 seconds.
The grace period clock starts at step 1, not after the preStop hook. The hook's time is spent out of the same budget.

Here's the timeline with defaults:

t=0s     Pod marked Terminating
t=0s     preStop hook (none by default)
t=0s     SIGTERM -> Sidekiq starts shutting down
t=25s    Sidekiq shutdown timeout expires, unfinished jobs pushed back to Redis
t=30s    SIGKILL if the process is still alive

What Sidekiq does with signals

Sidekiq listens for a few Unix signals. Two of them matter for deploys:

TSTP (quiet): stop fetching new jobs. Keep working on the jobs already running.
TERM (shutdown): stop fetching, wait up to timeout seconds for running jobs, push unfinished jobs back to Redis, and exit.

Older Sidekiq versions used USR1 for quiet. Since Sidekiq 5, it's TSTP. The default shutdown timeout is 25 seconds.

So, out of the box, Kubernetes sends SIGTERM, and Sidekiq gives running jobs 25 seconds, pushes the rest back to the queue, and exits. Kubernetes allows 30 seconds total before SIGKILL.

We were running both defaults. For our long jobs, neither number was anywhere close.

Here's what happens to a job that's still running at the deadline:

Best case: Sidekiq pushes it back to Redis. Another worker picks it up and starts again from zero. If the job isn't idempotent, it does half its work twice.
Worst case: the process gets SIGKILL before it can push the job back. With open-source Sidekiq, which uses basic fetch, that job is simply gone. Sidekiq Pro's reliable fetch can recover it, but most teams run the open-source version.

The fix: quiet first, then wait, then stop

The idea we landed on is simple. Before Kubernetes sends SIGTERM, tell Sidekiq to go quiet so it stops taking new work. Then give the jobs it already has enough time to finish. Only then let the normal shutdown happen.

The new timeline looks like this:

t=0s      Pod marked Terminating
t=0s      preStop: kill -TSTP 1      -> Sidekiq stops fetching new jobs
t=0s..N   preStop waits while running jobs finish (new pod already takes new work)
t=N       preStop exits
t=N       SIGTERM -> Sidekiq has nothing left to do, exits in seconds
t<600s    Pod gone. No SIGKILL, no lost jobs.

It takes four pieces to make that work:

A preStop script that sends TSTP and waits for running jobs to finish.
A terminationGracePeriodSeconds longer than your longest job.
A Sidekiq config with a sensible shutdown timeout.
A container where Sidekiq is actually PID 1, so the signal reaches it.

Let's go through each one.

Piece 1: the preStop drain script

This script goes into your app image, for example at bin/sidekiq-prestop.sh:

#!/bin/sh
# bin/sidekiq-prestop.sh
#
# Called by the Kubernetes preStop hook.
# 1. Puts Sidekiq into quiet mode (stop fetching new jobs).
# 2. Waits until this pod has no busy jobs, or until MAX_WAIT runs out.
#
# MAX_WAIT must be lower than terminationGracePeriodSeconds minus the
# Sidekiq shutdown timeout, or Kubernetes will SIGKILL us mid-drain.

set -u

MAX_WAIT="${SIDEKIQ_DRAIN_SECONDS:-540}"
INTERVAL=5

log() {
  # Write to PID 1's stdout so the message shows up in `kubectl logs`.
  echo "[prestop] $(date -u +%H:%M:%S) $*" > /proc/1/fd/1
}

log "sending TSTP to Sidekiq (quiet mode)"
kill -TSTP 1

waited=0
while [ "$waited" -lt "$MAX_WAIT" ]; do
  # Prints "<quiet> <busy>", for example "true 1".
  # Prints nothing if the check fails (Redis down, process not found).
  state=$(bundle exec ruby -e '
    require "sidekiq/api"
    me = Sidekiq::ProcessSet.new.find { |p| p["hostname"] == ENV["HOSTNAME"] }
    puts "#{me["quiet"]} #{me["busy"].to_i}" if me
  ' 2>/dev/null)

  quiet=$(echo "$state" | cut -d" " -f1)
  busy=$(echo "$state" | cut -d" " -f2)

  if [ "$quiet" != "true" ]; then
    # No heartbeat since TSTP yet, or the check failed. Don't trust busy.
    log "waiting for a fresh heartbeat, waited ${waited}s of ${MAX_WAIT}s"
  elif [ "$busy" = "0" ]; then
    log "no busy jobs after ${waited}s, letting shutdown continue"
    exit 0
  else
    log "still ${busy} busy job(s), waited ${waited}s of ${MAX_WAIT}s"
  fi

  sleep "$INTERVAL"
  waited=$((waited + INTERVAL))
done

log "drain timeout reached with jobs still running, continuing shutdown"
exit 0

A few notes on this script:

kill -TSTP 1 only works because Sidekiq is PID 1 in the container. More on that in piece 4.
The busy check uses Sidekiq's own API. Each Sidekiq process registers in Redis with its hostname, and in Kubernetes the hostname is the pod name, so ENV["HOSTNAME"] finds this exact pod.
It waits for a fresh heartbeat. The busy count comes from Sidekiq's heartbeat in Redis, which is a few seconds old. Right after TSTP, it can still say 0 while a job that was just fetched is running. So the script only trusts busy once the process reports quiet=true, which means a heartbeat ran after TSTP. If the check fails, it keeps waiting instead of assuming the pod is idle.
It exits as soon as the pod is idle. A quiet pod with nothing running shuts down in about five seconds. A pod in the middle of a long job waits for it. You don't pay the full wait on every pod.
It always exits 0. If the hook fails, Kubernetes records a FailedPreStopHook event and moves straight on to SIGTERM. Exiting cleanly keeps the shutdown predictable.
Logs go to /proc/1/fd/1. The preStop output doesn't show up in kubectl logs by default. Writing to PID 1's stdout puts the drain messages right next to Sidekiq's own logs, which makes debugging much easier.

Make it executable in your image:

COPY bin/sidekiq-prestop.sh /app/bin/sidekiq-prestop.sh
RUN chmod +x /app/bin/sidekiq-prestop.sh

The one-line version

If you don't want the Ruby check, there's a simpler version:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "kill -TSTP 1 && sleep 540"]

It works, but it's blunt. Every pod waits the full 540 seconds, even an idle one, so rollouts and scale-downs get slow. It's fine to start with. The script version is better once you run more than a couple of replicas.

Piece 2: the grace period

spec:
  terminationGracePeriodSeconds: 600

The rule is:

terminationGracePeriodSeconds > preStop max wait + Sidekiq shutdown timeout + buffer

With the numbers in this article:

600 > 540 (SIDEKIQ_DRAIN_SECONDS) + 25 (Sidekiq :timeout:) + 35 (buffer)

Pick the drain time from your real job durations, not from a guess. If your longest job can run for 9 minutes, a 540-second drain covers it. If you have a job that runs for an hour, a preStop hook is the wrong tool. Split that job into smaller chunks instead.

Piece 3: the Sidekiq config

# config/sidekiq.yml
:concurrency: 10
:timeout: 25
:queues:
  - [critical, 3]
  - [default, 2]
  - [low, 1]

By the time SIGTERM arrives, the preStop hook has already drained the pod. The 25-second :timeout: only has to cover whatever is left, which is usually nothing. Keep it well under the grace period minus the drain time.

Piece 4: make sure Sidekiq is PID 1

This is the gotcha that quietly breaks the whole thing. kill -TSTP 1 sends the signal to whatever process is PID 1. If that's a shell, Sidekiq never hears it.

This is wrong:

# Shell form: /bin/sh is PID 1, Sidekiq is a child process
CMD bundle exec sidekiq -C config/sidekiq.yml

This is right:

# Exec form: Sidekiq is PID 1 and receives signals directly
CMD ["bundle", "exec", "sidekiq", "-C", "config/sidekiq.yml"]

If you need an entrypoint script for setup work, end it with exec so Sidekiq replaces the shell:

#!/bin/sh
# docker-entrypoint.sh
set -e

# ...any setup you need (wait for Redis, load secrets, etc.)

exec "$@"

You can check what's running as PID 1 inside a pod:

kubectl exec -it deploy/sidekiq-worker -- cat /proc/1/cmdline | tr '\0' ' '

If the output starts with sidekiq or bundle exec sidekiq, you're good. If it starts with /bin/sh, fix that first.

The full Deployment

Here's everything put together:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sidekiq-worker
  labels:
    app: sidekiq-worker
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: sidekiq-worker
  template:
    metadata:
      labels:
        app: sidekiq-worker
    spec:
      terminationGracePeriodSeconds: 600
      containers:
        - name: sidekiq
          image: registry.example.com/app:1.42.0
          command: ["bundle", "exec", "sidekiq", "-C", "config/sidekiq.yml"]
          env:
            - name: SIDEKIQ_DRAIN_SECONDS
              value: "540"
            - name: REDIS_URL
              valueFrom:
                secretKeyRef:
                  name: app-secrets
                  key: redis-url
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "/app/bin/sidekiq-prestop.sh"]
          resources:
            requests:
              cpu: "250m"
              memory: "512Mi"
            limits:
              memory: "1Gi"

Two things are worth calling out:

maxUnavailable: 0 with maxSurge: 1 means Kubernetes starts a new worker before an old one begins draining. The queue keeps moving during the deploy, and you never drop below your normal worker count.
Watch your liveness probe. If you have one that checks Sidekiq, make sure a quiet, draining pod still passes it. Otherwise Kubernetes may restart the container in the middle of the drain.

Protect it during node drains too

Deploys aren't the only time pods get stopped. Cluster upgrades and kubectl drain evict pods as well. A PodDisruptionBudget stops Kubernetes from evicting too many workers at once:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: sidekiq-worker-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: sidekiq-worker

Evicted pods still run the preStop hook and respect the grace period, so the same drain logic applies. The PDB just makes sure they drain one at a time.

How to test it before you trust it

Don't wait for the next real deploy to find out if this works. Test it on purpose in a non-production environment.

1. Add a slow test job:

# app/jobs/slow_test_job.rb
class SlowTestJob
  include Sidekiq::Job

  def perform(seconds = 120)
    Sidekiq.logger.info "SlowTestJob starting, will run #{seconds}s"
    seconds.times { sleep 1 }
    Sidekiq.logger.info "SlowTestJob finished cleanly"
  end
end

2. Enqueue it:

kubectl exec -it deploy/sidekiq-worker -- \
  bundle exec rails runner 'SlowTestJob.perform_async(120)'

3. Find the pod that picked it up, then delete it:

kubectl get pods -l app=sidekiq-worker
kubectl delete pod sidekiq-worker-7d9f8c6b5-abcde

4. Watch the logs of the terminating pod:

kubectl logs -f sidekiq-worker-7d9f8c6b5-abcde

You should see something like this:

[prestop] 10:14:02 sending TSTP to Sidekiq (quiet mode)
INFO: Received TSTP, no longer accepting new work
[prestop] 10:14:07 still 1 busy job(s), waited 0s of 540s
[prestop] 10:14:12 still 1 busy job(s), waited 5s of 540s
...
INFO: SlowTestJob finished cleanly
[prestop] 10:15:57 no busy jobs after 115s, letting shutdown continue
INFO: Shutting down
INFO: Bye!

If you see SlowTestJob finished cleanly and Bye! before the pod disappears, the drain works. If the logs stop abruptly with no Bye!, the pod was killed. Check the grace period and the PID 1 setup.

5. Check events for hook failures:

kubectl get events --field-selector reason=FailedPreStopHook

An empty result is what you want.

Jobs still need to be idempotent

The preStop hook handles planned shutdowns: deploys, scale-downs, node drains. It doesn't handle everything. Nodes crash. Containers get OOM-killed. Someone runs kubectl delete pod --force --grace-period=0. In all of those cases, no hook runs.

So the real safety net is still a job that can safely run twice. A simple pattern is to record that the work is done and check before doing it again:

class SendInvoiceJob
  include Sidekiq::Job

  def perform(invoice_id)
    invoice = Invoice.find(invoice_id)
    return if invoice.sent_at.present?   # already done, safe to skip

    InvoiceMailer.with(invoice: invoice).deliver_now
    invoice.update!(sent_at: Time.current)
  end
end

This still leaves one gap. If the worker dies after the email goes out but before sent_at is saved, the retry sends the invoice again. No hook can close that window. For a plain notification, a rare duplicate may be fine. For anything tied to money, save a delivery record with a stable key before sending, pass that key to the provider if it supports idempotency, and check the record before retrying. Test it by killing the worker between the send and the database update.

For long jobs that loop over many records, save progress as you go, so a retry picks up where it stopped instead of starting over:

class BackfillJob
  include Sidekiq::Job

  def perform(batch_id)
    batch = Batch.find(batch_id)

    batch.items.where(processed: false).find_each do |item|
      process(item)
      item.update!(processed: true)   # checkpoint after each item
    end
  end
end

The drain makes deploys clean. Idempotency makes everything else survivable.

Checklist

Before you call this done:

Sidekiq runs as PID 1 (exec form CMD, or exec in the entrypoint)
preStop hook sends TSTP and waits for busy jobs
terminationGracePeriodSeconds > drain time + Sidekiq timeout + buffer
Drain time is based on your longest real job
maxUnavailable: 0 so new workers start before old ones drain
PodDisruptionBudget for node drains
Liveness probe still passes while the pod is quiet
Tested with a slow job and a manual pod delete
Jobs are idempotent for the cases no hook can catch

What changed

After this went in, deploys stopped killing jobs. Long jobs finished on the old pod while new pods picked up new work. Nobody had to check the Retry set after a release, and nobody had to clean up half-processed work by hand.

The whole fix is a shell script, a couple of numbers, and one signal. The hard part was noticing that Kubernetes and Sidekiq each had a perfectly reasonable idea of "shutting down gracefully," and the two didn't match.

Updated: thanks to Raza Hussain for spotting the stale heartbeat in the busy check, and to Tim Urista for pointing out the crash window in the invoice example. Both are fixed above.

I write up real production fixes like this one. If your team runs Sidekiq on Kubernetes, go check your terminationGracePeriodSeconds right now. If it still says 30, you're probably losing jobs.

I Built a Kubernetes Incident Manager That Explains Broken Pods With a Local AI Model

Farooq Hussain — Mon, 19 Oct 2026 10:19:28 +0000

It's 3 AM. Your phone buzzes. A pod is in CrashLoopBackOff.

You open your laptop, run kubectl describe, scroll through events, pull the logs from the crashed container, and try to work out what actually went wrong. Ten minutes later you find it: a missing Secret. Or a memory limit. Or a database that isn't reachable.

AI is good at exactly this kind of reading. But most teams can't paste production logs into a hosted AI chat. Those logs are full of connection strings, tokens, customer emails and internal IPs.

So I built a small incident manager that runs inside the cluster and explains broken pods with an open-weight model running on my own machine. Nothing leaves the infrastructure, and it costs nothing per incident.

Code: github.com/farooqhussain82/incident-explainer

What it does

Every 15 seconds it lists the pods in the cluster.
When one breaks (crash loop, OOMKilled, image pull error, missing Secret or ConfigMap, failing readiness probe, unschedulable), it opens an incident.
It collects the evidence: a kubectl describe-style summary, events, and logs, including the logs of the previous crashed run.
It redacts secrets from that text.
It asks a local model in Ollama (Qwen 3 8B by default) what went wrong and what to check next.
When the pod is healthy for four checks in a row, the incident resolves itself.

There is a small web page for the incidents, and it can post new incidents to Slack, Teams or Mattermost through an incoming webhook.

The demo: four pods, four different failures

To test it, I made a throwaway kind cluster and deployed four pods that break in four common ways. All four were detected and explained on my laptop (Apple M4, 16 GB RAM) in 20 to 45 seconds each.

Pod	What I broke	What the model said
`payments-api`	Refers to a Secret that doesn't exist	The pod can't start because Secret `payments-secrets` is missing. Check it exists in the namespace.
`report-worker`	Memory limit of 64Mi, process uses far more	OOMKilled, exit code 137, because usage goes over the 64Mi limit.
`checkout-web`	App can't reach its database	Connection refused to PostgreSQL on port 5432. Check the database is running and reachable.
`orders-db`	Postgres started without a password	Postgres won't initialise because `POSTGRES_PASSWORD` is not set.

Each answer quotes the exact log or event lines it is based on, so you can check it in seconds instead of trusting it blindly:

Then I created the missing Secret for payments-api. About a minute later the incident moved to "Recently resolved" on its own.

One detail I'm happy with: an out-of-memory container also shows CrashLoopBackOff in Kubernetes, because it keeps restarting. The detector looks at the last termination reason and reports it as OOMKilled, which is the real cause and the thing you need to fix.

Why an open, local model

This was the main reason for the project, not an extra.

The data stays inside. The model runs next to the cluster. Logs are never sent to a third party.
No bill per incident. A crash loop that fires 200 times a night doesn't turn into an API invoice.
You choose the model. Qwen 3 8B runs fine on a laptop. Use a bigger one on a GPU node, or one tuned on your own runbooks. It's one environment variable.

I also made the model answer in a fixed JSON format (summary, likely cause, evidence, next steps, confidence). That keeps the output predictable enough to show in a UI and send to chat.

Redaction, even though the model is local

"Local" should not mean "careless". Before anything reaches the model, the text is cleaned:

private keys, AWS keys, JWTs and bearer tokens
passwords in URLs like postgres://user:password@host
values of password=, token=, api_key= and similar
email addresses and single IP addresses

In the checkout-web demo, the log contained a database URL with a password, a JWT and IP addresses. The model only saw this:

connecting to postgres://[REDACTED-URL-CREDS]@[REDACTED-IP]:5432/checkout
Authorization: Bearer [REDACTED-JWT]

Each incident page has a "Show exactly what the model saw" section, so anyone can check.

One thing I had to fix during testing: my first version also masked 0.0.0.0/0 in an infrastructure change. That hid the most important fact, which was a database being opened to the whole internet. Now CIDR ranges and 0.0.0.0 are kept, and only single host IPs are masked.

Security of the thing itself

A tool that reads every pod in your cluster has to be boring and safe:

Read-only RBAC: only get, list and watch on pods, pod logs and events. I checked it with kubectl auth can-i: it cannot read Secrets and cannot delete pods.
Env var values are never collected, only their names and where they come from (for example, "comes from Secret payments-secrets").
A 13 MB FROM scratch image: no shell, no package manager, runs as non-root with a read-only filesystem and all Linux capabilities dropped.

It's written in Go with only the standard library, so there are no third-party packages to keep patched.

Honest limits

It runs as one replica and stores incidents in a JSON file. Fine for a team cluster, not for a huge fleet.
It explains, it does not fix. Any action is still a human decision.
An 8B model gets the common failures right, but small details can be off. In one run it said the OOM test "writes to /dev/zero", when tail /dev/zero actually reads from it. The quoted evidence makes slips like that easy to spot.
Redaction is pattern based. It catches the common shapes of secrets, not every possible one. There's a -dry-run mode that prints exactly what would be sent.

Try it

You need Docker, kind, kubectl, Go and Ollama.

ollama pull qwen3:8b
git clone https://github.com/farooqhussain82/incident-explainer
cd incident-explainer
make kind-up    # local cluster
make deploy     # build the image, load it, apply the manifests
make demo       # four broken pods
make ui         # open http://localhost:8080

It also works without a cluster, on a single log file or an infrastructure change plan:

kubectl logs my-pod --previous | ./bin/incident-explainer explain -

What I'd add next

A Helm chart and an option to run Ollama inside the cluster.
Grouping, so ten replicas of one broken Deployment become one incident, not ten.
Using node events and recent rollouts as extra evidence.

If you run Kubernetes and you've ever read crash loop logs at 3 AM, I'd love to hear what failures you'd want this to understand. And if you try it, tell me what it got wrong.