DEV Community

hwu121380-afk
hwu121380-afk

Posted on

Tests pass but the agent dropped a column: what a green diff can still contain

If you let Claude Code, Codex or Cursor write a migration, you've probably seen a PR that's all green: tests pass, CI is happy. Tests usually run against a fresh database, so they rarely notice a change that deletes data. A dropped column, a DELETE with no WHERE, or rm -rf $OUT_DIR/ can sit in a diff that looks routine.

Here are the patterns I watch for, and the cheap checks I use for each one. You can do all of this with git, a grep, and (if you're on Rails) a free gem.

1. The migration drops or renames something

Here are the kind of lines that look routine in a migration:

remove_column :users, :email
Enter fullscreen mode Exit fullscreen mode
DROP TABLE sessions;
ALTER TABLE orders DROP COLUMN total;
Enter fullscreen mode Exit fullscreen mode

Django, Alembic, Knex, Laravel, Sequelize and Prisma can all do the same thing in their own dialects. A prisma migrate reset or rails db:drop in a script is the blunt version of the same problem.

Cheap check: read the added lines of migration files first, before the rest of the PR.

git diff main -- 'db/migrate' '**/migrations/**' 'prisma/**' '*.sql' \
  | grep -E '^\+' | grep -vE '^\+\+\+'
Enter fullscreen mode Exit fullscreen mode

Look for drop, remove_column, renames, type changes, and reset/flush commands. If the app is Rails-only, strong_migrations is free and checks migrations in more depth than a grep.

2. DELETE or UPDATE with no WHERE

DELETE FROM users;
UPDATE orders SET status = 'cancelled';
Enter fullscreen mode Exit fullscreen mode
User.objects.all().delete()
Enter fullscreen mode Exit fullscreen mode
db.users.deleteMany({})
Enter fullscreen mode Exit fullscreen mode

Cheap check: scan added SQL, including SQL strings in code, not just .sql files. A simple grep for DELETE and UPDATE is a start, but a WHERE on the next line is easy to miss, so read the surrounding lines before you panic or before you ignore it.

3. A "cleanup" script or CI step that deletes too much

rm -rf $OUT_DIR/
git push --force
terraform destroy
kubectl delete namespace staging
docker compose down -v
Enter fullscreen mode Exit fullscreen mode

rm -rf $OUT_DIR/ is fine until OUT_DIR is empty. ${OUT_DIR:?} fails closed if the variable is unset.

Cheap check:

git diff main -- '*.sh' '*.yml' '*.yaml' 'Makefile' 'Dockerfile' 'package.json' \
  | grep -E '^\+' | grep -vE '^\+\+\+' \
  | grep -Ei 'rm -rf|push --force|reset --hard|terraform destroy|kubectl delete|compose down -v|migrate reset|db:drop|db reset|FLUSHALL'
Enter fullscreen mode Exit fullscreen mode

4. An env var quietly disappears

A removed line in .env.example, or a deleted key under env: in GitHub Actions or docker-compose, often won't fail tests. It can fail in production. Review those files the same way you review migrations: added and removed lines.

git diff main -- '.env*' '*.yml' '*.yaml' | grep -E '^[+-]' | grep -vE '^[+-]{3}'
Enter fullscreen mode Exit fullscreen mode

Putting it together

The common thread: a green test run is not evidence that the diff is safe for a real database. Review the added lines in migrations, scripts and CI first. If you want guardrails inside the agent session, AgentLint is free and checks risky file changes and shell commands as the agent proposes them.

I wanted the same idea at commit and PR time, over the git diff itself, whoever wrote it, so I built Destructive Diff Gate (one offline Python file, standard library + git):

  • It runs 30 rules over the lines the diff adds: DROP / TRUNCATE, DELETE and UPDATE with no WHERE, destructive migrations (Rails, Django, Alembic, Knex, Laravel, Sequelize, Prisma), rm -rf, force push, terraform destroy, kubectl delete, and removed env vars.
  • A git pre-commit hook, plus a GitHub Actions pull-request check that compares the branch with its base. Local hooks can be skipped with --no-verify, so the CI check is the real enforcement.
  • When a destructive change is intended, you allow it with a reason.
  • 48 tests pass on Linux.

To be clear about what it is: a pattern check on this diff, not a security audit. It doesn't look for secrets, injection or vulnerable dependencies. Regex over diff lines can miss SQL built at runtime, or a dangerous helper defined elsewhere, and it can flag harmless changes. It doesn't replace backups, code review or a staging environment.

It's $9 here if you want that scan in the repo: https://payhip.com/b/jtBER

Even if you don't use it, steal the greps above. The migration diff and the unquoted rm -rf $VAR are the ones I'd add first.

Not affiliated with Anthropic, Cursor, OpenAI, GitHub, or the makers of AgentLint and strong_migrations.

Top comments (3)

Collapse
 
aiapikey profile image
HAOHAOnovai •

The green diff is honest about what it checked: the tests that exist. A dropped column is invisible to a test that never selects it, so the only gate that catches it is a schema diff - columns added, removed, or type-changed - computed against the previous version and reviewed as its own artifact, not as a side effect of reading the code diff.

A cheap version: snapshot the schema before and after, and fail on any removal unless the same change set also touches every reader of that column. That turns "did the agent drop something" from a review question into a mechanical one.

Collapse
 
hwu121380afk profile image
hwu121380-afk •

Agreed, the tests only vouch for what they actually touch. I like the "a removal has to come with every reader touched" rule, it turns a judgment call into a check.

One gap I've hit with it: readers that don't live in the repo. A BI dashboard, a cron job in another service, a weekly CSV export someone downloads. The schema diff flags the drop, but "every reader touched" can still pass while an outside consumer breaks.

What's worked for me is keeping a short list of known external readers next to the migrations, and making any column removal carry a one-line note for each of them. Low-tech, but it forces someone to ask the question before merge.

Collapse
 
aiapikey profile image
HAOHAOnovai •

That external-reader gap is the one that actually bites, because the list only contains readers you already know about. The failure is the reader nobody remembered.

Two ways to discover the unknown ones. First, grep the query logs for the column name over the last 30 days - most warehouses keep them, and it finds the BI dashboard and the cron job without asking anyone. Second, for readers that never show up in logs, rename instead of dropping: keep the old name as a view over the new column for one full billing cycle and watch whether anything still reads it.

The rename step is the part I would make mandatory. A removal that only fails inside a repo test is a removal that fails in production.