DEV Community

Cover image for How a missing DEFAULT column locked my production DB for 20 minutes (and why your CI won't catch it)
Rajesh Bhanushali
Rajesh Bhanushali

Posted on

How a missing DEFAULT column locked my production DB for 20 minutes (and why your CI won't catch it)

Let’s be brutally honest. As developers, we don’t fear database downtime. We fear the Post-Mortem incident meeting where we have to explain to the CTO why a simple ALTER TABLE statement brought the entire API down.

A while back, we had a classic 2 AM outage. A developer merged a migration that added a new NOT NULL column to a massive production table, but forgot to provide a DEFAULT value.

If you know PostgreSQL, you know what happened next. Postgres slapped an Access Exclusive Lock on the table while it verified the constraint across 10 million rows. The app went down, queues backed up, and support tickets flooded in.

I decided we needed an automated check in our CI/CD pipeline to catch this. But when I looked at the existing SQL linters, I realized they are completely broken.

The Problem with Most SQL Linters (Alert Fatigue)

Most SQL linters are dumb. They just use Regex to hunt for keywords like DROP TABLE or ALTER TABLE ... NOT NULL.

Because they suffer from "Context Blindness", they throw false positives everywhere. For example, if you write a migration that creates an ephemeral (temporary) table, and then drops it in the very next file, a dumb linter will flag the DROP TABLE as a critical danger and block your CI.

Developers get annoyed, they start ignoring the linter warnings, or worse, they bypass the CI checks entirely.

Building a "Context-Aware" Shield

I got tired of the false positives, so I built migra-guard.

Instead of dumb Regex, it uses an AST (Abstract Syntax Tree) engine and tracks the chronological state of your migrations across your Pull Request. It remembers what it has seen.

  • Drop an ephemeral table created in the same PR? migra-guard knows it's a new table. It stays quiet. (SAFE)
  • Add NOT NULL to a brand new table? Safe.
  • Drop an existing production table? CI blocked. 🚨
  • Add NOT NULL to a prod table without a DEFAULT? CI blocked. 🚨

It acts as an automated Senior DBA that lives inside your GitHub Actions. It doesn't scream at you for local development patterns, it only screams when you are actually about to nuke production.

The Ultimate Insurance Policy

As a Tech Lead, you can't manually review every single SQL migration from junior developers. You need a system you can blame, an automated scapegoat that catches Postgres lock quirks before they merge.

It's completely zero-config and open-source.

If you are using PostgreSQL, drop this in your CI pipeline. It’s cheaper than a DBA, and it's the best insurance policy your team can buy against human error.

Check out the repo here: turfin-logic/migra-guard

Let me know if I missed any other ridiculous Postgres locking edge cases, I'm adding more AST rules this weekend!

Top comments (0)