We found a data-loss bug that alembic downgrade ran clean

#python #django #testing #devops

alembic downgrade -1 ran clean. No errors. Monitoring went green.

The users' phone numbers were gone. The column came back. The data didn't.

Why standard migration tests miss this

Most CI pipelines verify:

alembic upgrade head exits 0
alembic downgrade -1 exits 0

Neither checks that data survived the round-trip. The SQL can be perfectly valid, the migration fully reversible in schema terms, and rows still disappear silently.

Two ways to catch it

1. Static analysis — no database needed

pip install pytest-mrt
mrt check migrations/versions/

30 patterns checked: op.drop_column() in upgrade, no-op downgrade(), PostgreSQL ENUM adds that can't be rolled back, op.execute() without reverse, NOT NULL without server_default, and more. Runs in ~22ms for 10 migrations.

2. Dynamic verification — seed rows, roll back, check they survived

# conftest.py
from pytest_mrt import MRTConfig

def pytest_configure(config):
    config._mrt_config = MRTConfig(
        alembic_ini="alembic.ini",
        db_url=os.environ["TEST_DATABASE_URL"],
    )

# tests/test_migrations.py
def test_migrations_are_safe(mrt):
    mrt.assert_all_reversible()

The mrt fixture seeds rows before each migration, runs downgrade, asserts nothing was lost. Catches the case above — column exists after downgrade, rows don't.