We found a data-loss bug that alembic downgrade ran clean

croc100 — Fri, 05 Jun 2026 20:18:55 +0000

alembic downgrade -1 ran clean. No errors. Monitoring went green.

The users' phone numbers were gone. The column came back. The data didn't.

Why standard migration tests miss this

Most CI pipelines verify:

alembic upgrade head exits 0
alembic downgrade -1 exits 0

Neither checks that data survived the round-trip. The SQL can be perfectly valid, the migration fully reversible in schema terms, and rows still disappear silently.

Two ways to catch it

1. Static analysis — no database needed

pip install pytest-mrt
mrt check migrations/versions/

30 patterns checked: op.drop_column() in upgrade, no-op downgrade(), PostgreSQL ENUM adds that can't be rolled back, op.execute() without reverse, NOT NULL without server_default, and more. Runs in ~22ms for 10 migrations.

2. Dynamic verification — seed rows, roll back, check they survived

# conftest.py
from pytest_mrt import MRTConfig

def pytest_configure(config):
    config._mrt_config = MRTConfig(
        alembic_ini="alembic.ini",
        db_url=os.environ["TEST_DATABASE_URL"],
    )

# tests/test_migrations.py
def test_migrations_are_safe(mrt):
    mrt.assert_all_reversible()

The mrt fixture seeds rows before each migration, runs downgrade, asserts nothing was lost. Catches the case above — column exists after downgrade, rows don't.

When to use each

	Static (`mrt check`)	Dynamic (`mrt` fixture)
Speed	~22ms / 10 migrations	~330ms / migration
Needs DB	No	Yes
Catches logic bugs	No	Yes
Best for	every commit	PR merge / nightly

Databases

PostgreSQL, MySQL/MariaDB, SQLite, Oracle, SQL Server.

GitHub: https://github.com/croc100/pytest-mrt
Docs: https://croc100.github.io/pytest-mrt

The incident at the top is a reconstruction of a pattern I've seen across production systems and open-source repos.