DEV Community

Cover image for Materialized Lake Views: Can MLVs Replace Silver and Gold Tables?
Sai Prudhvi Neelakantam
Sai Prudhvi Neelakantam

Posted on Originally published at geekindata.com AI-assisted

Materialized Lake Views: Can MLVs Replace Silver and Gold Tables?

I used AI tools to help draft and edit this post. I checked the facts and the wording, and the original lives at geekindata.com. The diagrams are on that page.

Bronze to silver to gold with gates

Key takeaways

  • Full view inventory and dependency graph gate any bulk MLV conversion.
  • Schema and shortcut resolution rules are documented before silver/gold waves.
  • A vertical POC must pass parity and runtime before long-tail batches.
  • One notebook (or small set) per MLV; shared logic in libraries only.
  • Success = parity checks + runtime budget, not portal object existence.
  • Promote in layer waves with stop criteria and a named owner per batch.

Moving a handful of warehouse views to Materialized Lake Views is a rewrite. Moving a large set is a program. Teams that treat the long tail like a weekend notebook usually discover dependency order the hard way: gold breaks, silver thrashes, and bronze shortcuts were never registered as the real source of truth.

Frame the work as bronze to silver to gold with gates. Count the views. Order by dependency. Prove a representative POC. Then convert with per-view notebooks so blast radius stays local. Success is parity plus runtime, not "the portal shows an MLV."

Named model: bronze to silver to gold with gates

  • Inventory and order: every warehouse view, its references, and its consumers on a graph before anyone rewrites SQL.
  • Bronze wave: shortcut definitions and schema resolution stabilize first so names resolve the same way for every downstream MLV.
  • Silver wave: transforms become MLVs in isolatable notebooks; parity and runtime are tracked per batch.
  • Gold wave: consumers and known measures move last, when keys are stable.
  • Gates: a vertical POC and explicit stop criteria sit between waves. Portal object count is not a gate.
  • Done: agreed parity against a pinned baseline, plus wall-clock inside the refresh window you will actually run.

Named model: bronze, silver, and gold waves with gates between layers; done means parity plus runtime

Figure 1. Named model: bronze to silver to gold with gates. Inventory and order first. Waves earn promotion. Portal green is not acceptance.

Context and constraint

A warehouse view graph often looks flat on a slide and nests deeply in reality. One mega-notebook that converts "almost everything" leaves the team owning silent grain drift. The portal was green. The business measures were not.

Schema-enabled lakehouses and bronze shortcut layouts change how names resolve. Views that assumed a warehouse catalog need an explicit path story. A representative POC must clear parity and runtime before the long tail starts. Per-view notebooks keep failures isolatable. "Views exist" is a status report. "Views match and finish in budget" is a release.

Figure 2 is the Learn medallion framing. Layers are a shared meaning. Packaging still needs gates when you convert many views.

OneLake medallion lakehouse architecture example

Figure 2. MLV migration is a program across bronze, silver, and gold. Gates belong between layers, not as a weekend cutover wish. Source: Microsoft Learn: medallion lakehouse architecture.

Design

1. Count and dependency-order first

Inventory every warehouse view, its references, and its consumers (reports, downstream views, notebooks). Draw edges before rewriting SQL. Bulk conversion without a graph is how you convert leaves before trunks and spend a week chasing missing objects. Numbers beat optimism: count the views, state the number, and plan batches.

2. Redesign resolution for schema-enabled lakehouses

Shortcuts and schemas are not cosmetic. Bronze paths may move from flat warehouse objects to schema-qualified lakehouse tables. Update the mental model and the SQL together. Document how MLVs read shortcut-backed bronze versus managed silver. Migration is partly a namespace migration pretending to be a syntax migration.

3. Gate a representative POC

Pick a vertical slice that touches bronze shortcuts, a non-trivial silver transform, and a gold consumer with known measures. Prove row counts, key parity, critical column aggregates, and wall-clock under realistic concurrency. Only then open the floodgates. A POC that only converts three isolated bronze views teaches almost nothing about your real graph.

Vertical POC clears bronze shortcuts, silver transform, and gold measures before long-tail batches

Figure 3. Vertical POC before the long tail. Happy-path bronze-only slices are POC theater.

4. Prefer per-view notebooks over a mega-notebook

One notebook per MLV (or small tightly coupled set) limits blast radius, clarifies ownership, and makes CI diffs readable. Mega-notebooks hide which object failed and invite "re-run everything" as the only recovery. Shared helpers belong in libraries; object definitions belong in isolatable units.

Per-view notebooks keep blast radius local; a mega-notebook blocks every view on one failure

Figure 4. Per-view notebooks beat a mega-notebook. Register notebook path, batch, parity, runtime, and owner for each MLV.

5. Define success as parity plus runtime

Portal presence is not done. Done means agreed parity checks pass against the warehouse baseline (or a frozen snapshot), and runtime fits the refresh window you will actually run. Track both metrics per batch. A fast wrong MLV is a regression factory.

6. Promote in layer waves with explicit stop criteria

Bronze shortcut definitions first when shortcuts must stabilize. Silver next for transforms. Gold last when consumers depend on stable keys. Each wave has exit criteria and a rollback story. Skipping waves to "save time" usually spends it twice in incident mode.

What the migration owner should track

Keep a living register: view name, layer, dependencies, notebook path, POC/batch id, parity status, runtime vs budget, owner. Review it in the same meeting that prioritizes feature work. Migration debt becomes invisible the moment it is only tracked in chat. If a batch slips, slip the date in the register; do not silently widen the parity exception list.

Dual-run without lying to yourself

If you keep warehouse views alive beside MLVs, pin the comparison to the same business day and the same key set. Dual-run that compares Tuesday warehouse to Wednesday lakehouse will manufacture false failures and false confidence. Record which baseline snapshot the parity job used. When stakeholders ask why a batch is late, show the register and the failing tier, not a vague sense that "Spark is slow."

Common mistakes

Rewrite-first inventory-never.
Clever SQL, unknown dependents, surprise report breaks.

One notebook to rule them all.
A single failure blocks many unrelated objects.

POC theater.
Happy-path views only; no gold consumer; no runtime budget.

Assuming warehouse names map one-to-one.
Schemas and shortcuts change resolution; silent wrong-table reads follow.

Declaring victory on object count.
A batch of MLVs with several parity failures is not a migration.

Converting gold before bronze shortcuts exist.
You materialize fiction, then blame Spark.

Use as-is or adapt

Use as-is

  • Full view inventory and dependency graph gate any bulk MLV conversion.
  • Schema and shortcut resolution rules are documented before silver/gold waves.
  • A vertical POC must pass parity and runtime before long-tail batches.
  • One notebook (or small set) per MLV; shared logic in libraries only.
  • Success = parity checks + runtime budget, not portal object existence.
  • Layered promotion with stop criteria and a named owner per batch.

Adapt before you use

Program overhead feels slow next to "just convert it." It is faster than a partial cutover that freezes reporting. Per-view notebooks add files; they subtract mystery. Strict POC gates annoy stakeholders who want a date more than a proof. Give them both: a date contingent on gate metrics. Schema redesign may force temporary dual-run cost. Budget it. Dual-run without parity jobs is theater.

Primary references: Materialized Lake Views and lakehouse schemas on Microsoft Learn. Pair them with your shortcut inventory and a written parity harness.

Count the graph. Prove a slice. Convert in isolatable units. Call it done when parity and runtime agree, not when the portal looks busy.

Related

Top comments (0)