Most conversations about data warehouse migration jump straight to tooling: which platform, which conversion utility, which orchestration framework. That's rarely where migrations actually succeed or fail. They succeed or fail based on a handful of decisions made before any code gets touched, decisions that often never make it into the project plan because nobody flagged them as decisions in the first place.
Here's the engineering reasoning behind the ones that matter most.
The problem: estimates built on what's visible, not what's true
A legacy SQL Server, Teradata, or Oracle environment that's been in production for ten-plus years accumulates undocumented logic the way old buildings accumulate structural quirks nobody remembers the reason for. Some of it is genuinely load-bearing. Some of it is dead weight nobody ever removed. You can't tell which is which by counting objects, you have to actually look, and how deep you look is a decision with real consequences downstream.
Teams that decide to look shallow (object counts, average complexity assumptions, move fast into planning) end up making every subsequent decision on incomplete information. Teams that look deep first tend to move slower initially and faster overall, because they aren't re-litigating the plan every time a surprise turns up mid-execution.
Why it happens: discovery gets treated as a phase, not a decision
Discovery isn't something you complete and move past. It's an ongoing decision about how much you're willing to know before you commit to an estimate. Treating it as a fixed phase with a start and end date is what causes teams to stop looking the moment the calendar says discovery is done, whether or not the estate has actually been understood by that point.
The business consequence: sequencing decisions inherit the gap
Here's where this compounds. The order in which systems get migrated often gets decided by organizational convenience, which business unit is easiest to coordinate with, which team has bandwidth first, rather than by actual technical dependency. If one table feeds three downstream jobs that two other tables depend on, migrating those two before the first is fully validated is asking for a cascade of broken pipelines nobody predicted, because nobody traced the dependency graph before sequencing the waves.
Getting sequencing right requires knowing, concretely, what depends on what. Not "we think this team's systems are mostly self-contained." Actually tracing the lineage. This is unglamorous work, and it's exactly the work that determines whether wave two of a migration goes smoothly or triggers three unplanned fire drills.
The insight: complexity is not a single number
A common shortcut is assigning one complexity multiplier to an entire category of objects. All stored procedures are "medium complexity." All ETL jobs are "high complexity." This feels efficient and it's almost always wrong in ways that matter.
The realistic distribution in most legacy estates looks something like this: a small number of genuinely complex objects carrying embedded business logic or platform-specific functions, a larger number of straightforward objects that convert with minimal review, and a middle tier that needs a human to actually look at them before anyone can say which bucket they belong in. Treating everything as equally complex means overbuilding effort for the easy majority or underbuilding it for the hard minority, and the hard minority is usually where the real schedule risk lives.
The part that's easy to skip: reading the business logic, not just counting it
A four hundred line stored procedure might contain three lines of business critical logic buried inside error handling and historical scaffolding nobody's cleaned up. Or it might be four hundred lines where every section encodes something a specific business unit depends on. You can't know which without reading it, and deciding how much of that reading happens before conversion versus after is itself an architecture decision with real risk attached.
Convert first and validate later, and you risk shipping subtly wrong business logic into production, the kind of error that doesn't surface until a quarterly report doesn't reconcile. Understand the logic first, and conversion becomes faster and safer, because the team knows what it's actually preserving rather than guessing at it under deadline pressure.
The engineering approach: automation for extraction, judgment for decisions
There's a version of this conversation where the assumption is that AI-powered tooling now just handles all of this automatically. It handles a meaningful part of it: extracting metadata at scale, flagging dependency patterns, doing bulk code conversion on the objects that are genuinely straightforward. That's real, useful acceleration.
What it doesn't do is make the judgment calls. Deciding whether a piece of business logic is safe to simplify during conversion, deciding how much modernization to layer onto a migration versus doing a clean lift and shift, deciding what the actual risk tolerance is for a given system: those remain engineering and architecture decisions. The tooling should be doing the repetitive extraction and classification work fast enough that engineers can spend their time on the decisions that actually require judgment, instead of spending months manually cataloguing an estate before anyone gets to the interesting part.
The evidence: overconfidence in automation is a documented pattern, not a hunch
This isn't just an engineering intuition. Gartner predicted in June 2026 that more than 70% of mainframe exit projects initiated that year would fail to produce their intended benefits specifically because organizations overestimated what generative AI tooling could do with complex legacy code. That prediction is scoped to mainframe exits, not data warehouse migrations, so it's worth being precise about what it does and doesn't cover. But the pattern it describes, trusting automation before testing it against a real sample, isn't specific to mainframes. It shows up anywhere a team is under schedule pressure and a tool offers a plausible shortcut around discovery work that hasn't been done yet.
Validation as a design decision, not a final step
One more piece that's easy to defer: validation often gets treated as something you do at the end, right before cutover. Framed that way, it's a pass-or-fail gate with high stakes and no room to fix what it finds. Framed as a design decision made early, validation becomes something built into each migration wave, defining what "correct" means for a dataset before you migrate it, comparing legacy and target outputs at the row level and the business logic level while there's still time to investigate discrepancies.
The implication: none of this is a tooling decision
Every decision above is an engineering judgment call about how much you're willing to know, when you're willing to know it, and how much risk you're willing to carry forward into production. Get those right and the platform choice, Snowflake, Databricks, Fabric, matters a lot less than most teams assume going in.
This is roughly the model we work from at 3X Data Engineering: use automation for the repetitive discovery work so engineering time goes toward the decisions that need real judgment. The reverse engineering approach we use for undocumented legacy systems is a reasonable illustration of where that line between automation and human judgment actually sits.
What's the biggest sequencing or dependency surprise you've hit mid-migration? I'd be curious whether it traces back to one of these decisions getting made by default instead of on purpose.
Top comments (0)