The deployment window was Sunday night specifically because it was supposed to be quiet, the lowest-traffic slot in the week, the safest possible time to ship a schema change nobody expected to notice. What went out that night was a rename - one column, legacy_flag becoming status_code, part of a larger cleanup that had been sitting in the backlog for months. The migration script was correct. The problem was that it never ran.
The properties file behind it was meant for staging. It had been copied into the production configuration months earlier during an environment setup nobody fully audited afterward, and it carried two settings that should never have left staging: spring.flyway.enabled=false and spring.jpa.hibernate.ddl-auto=update. The first meant the migration was never executed. The second meant Hibernate filled the gap. With update active, Hibernate inspects the mapped domain objects at startup and alters the schema to match what the code expects. The application redeployed, Hibernate saw an entity mapped to status_code, found no such column, and quietly added it, empty. legacy_flag stayed exactly where it was with its data intact, but nothing read from it or wrote to it anymore. No migration ran and no rename happened. The table now had two columns: one holding all the history, and one the application actually used.
Nobody noticed for the first eleven minutes, because nothing threw an exception. legacy_flag was nullable, so inserts that no longer populated it went through without complaint. Reads against status_code returned null, which is a valid value, not an error. Reports depending on that column started showing empty fields instead of failing loudly, and it took a downstream team asking why customer statuses looked wrong before anyone connected it to the deployment. The data wasn't lost, it was stranded. Every write since the deploy had gone to a column the old rows didn't have, while legacy_flag held values that were now stale. Copying one column into the other was a single statement. Deciding which value to trust for every row touched in between took far longer than the fix itself.
What made ddl-auto=update dangerous here wasn't that it did something wrong. It did exactly what it's designed to do: keep the schema in sync with the domain objects, with zero migration tooling in the loop. That's a reasonable convenience for a local environment where the database resets constantly and nobody cares about history. In any environment holding real data, the same behavior means the schema is one refactor away from being rewritten by inference instead of by an explicit, reviewed script. Renaming a field in code isn't a rename to Hibernate. It's a new, empty column appearing next to an old one that is silently orphaned, and Hibernate does that at startup, without asking.
The fix wasn't just re-enabling Flyway and flipping the setting to validate, though both happened within the hour. With validate, the same deploy would have refused to start on a missing column instead of quietly creating one. The real fix was auditing every environment's configuration against a single source of truth, instead of trusting that a staging properties file, once copied, stays where it was copied to. If your production configuration was ever bootstrapped from a staging template, it's worth checking whether that inheritance is still intentional or just inherited.
More like this?
I write short production war stories like this one, plus a deep-dive architecture series. Subscribe here if you want both in your inbox.
Recommended Resources for Java & Spring Boot Engineers
If you are preparing for Senior/Lead Java interviews or looking to solidify your Spring Boot & Architecture skills, check out these highly-rated resources from the Javarevisited publication (Use promo code friends20 for an exclusive 20% discount automatically applied at checkout):
- Grokking the Java Interview Prepare for core Java, concurrency, JVM internals, and design pattern questions. Get Full Book (Paid) | Download Free Sample Copy
- Grokking the Spring Boot Interview Master Spring Core, Auto-configuration, Spring Data JPA, Security, and Microservices. Get Full Book (Paid) | Download Free Sample Copy
- Grokking the SQL Interview Deep-dive into query optimization, indexing, joins, and complex SQL window functions. Get Full Book (Paid) | Download Free Sample Copy
- The Complete Java + Spring + SQL Interview Bundle Get all three interview guides in a single heavily discounted package. Get the Ultimate Interview Bundle
- Spring Professional Certification Practice Questions (250+ Questions) Validating your skills? Practice with real exam-style questions before taking the Spring Professional certification. Get the Spring Professional Questions | Download Free Sample Copy
Top comments (0)