DEV Community: laura williams

Unity Catalog Is Becoming the Operating Layer for Enterprise Data Governance

laura williams — Wed, 16 Sep 2026 08:24:41 +0000

For many enterprises, the Databricks journey starts with a simple goal: give data teams a scalable platform for analytics and AI. Then the environment grows. New workspaces appear across business units, teams adopt different security practices, and workloads spread across AWS, Azure, and GCP. What initially looked like one data platform gradually becomes a collection of independently governed environments. The problem is rarely that any individual workspace is poorly managed. The problem is that governance itself has become fragmented. Metadata, permissions, lineage, and audit trails stop at workspace boundaries, making enterprise-wide control increasingly difficult. This is the problem Databricks Unity Catalog is designed to address. By moving governance from the workspace level to an account-level model, Unity Catalog creates a common foundation for managing data and AI assets across clouds, workspaces, and increasingly diverse data environments.

From Workspace Governance to Enterprise Governance

The most important idea behind Unity Catalog is not simply that it centralizes metadata. It changes where governance happens. In a traditional Databricks environment, Apache Hive Metastore instances can exist independently across workspaces, with each team maintaining its own access controls and metadata structures. That model works when environments are relatively isolated, but becomes difficult to manage as organizations scale. The same data may be replicated across workspaces, security policies may be implemented differently, and cross-workspace collaboration can require additional configuration. Unity Catalog introduces a standardized three-level namespace - catalog.schema.object - that provides a common structure for governed assets. Catalogs can establish business or environment boundaries, schemas organize related entities, and objects such as tables, views, volumes, models, and functions sit beneath them. Above this hierarchy, the account-level metastore provides the common governance layer that connects identities, storage credentials, security policies, and metadata across workspaces.

This architectural shift becomes particularly important when access control moves beyond individual tables. Enterprise governance has historically relied heavily on table-level permissions, manually created views, and exceptions that accumulate as data estates grow. Unity Catalog introduces Attribute-Based Access Control, allowing organizations to define policies around governed tags and apply those policies across assets. A sensitive dataset, for example, can be classified through tags representing confidentiality or PII, while a policy determines how that information should be accessed or masked. The value is not simply automation. It is consistency. Instead of relying on every data owner to remember the same security rule whenever a new dataset arrives, governance teams can define the policy centrally and have it apply to appropriately classified assets. Governance therefore starts moving away from a collection of permissions toward a policy-driven operating model.

Governance Cannot Stop at the Databricks Boundary

Modern enterprises rarely run their entire data estate through a single processing engine. Spark may power large-scale engineering workloads while Trino, Starburst, DuckDB, Snowflake, or other technologies serve different analytical requirements. A governance system that only works inside Databricks would eventually create another form of fragmentation. Unity Catalog addresses this through its open interfaces, including the Unity Catalog open-source protocol and Iceberg REST Catalog API, allowing external engines to interact with governed data without requiring another physical copy. Credential vending provides scoped, short-lived access to underlying cloud storage, while governance decisions remain centralized in the catalog. Universal Format, or UniForm, extends this interoperability by making Apache Iceberg metadata available for Delta tables without requiring duplicated Parquet data. For enterprises operating heterogeneous data stacks, this creates a useful separation between where data is governed and where it is processed.

The same principle applies when data does not physically live inside Databricks. Lakehouse Federation allows organizations to connect external systems such as PostgreSQL, MySQL, Snowflake, Redshift, Azure SQL Database, and BigQuery without turning every integration into another extraction and replication project. The external schema can be represented within the Unity Catalog namespace, while query processing can remain closer to the source. Delta Sharing extends governed access to external organizations, while Clean Rooms provide an approach for sensitive collaborative analysis. The broader architectural idea is straightforward: governance should follow the data and the use case rather than forcing organizations to create additional copies simply to make governance possible.

Governance Is Also a FinOps Problem

Security is usually the first thing that comes to mind when discussing data governance, but enterprise governance increasingly includes financial accountability. Databricks system tables such as system.billing.usage, system.access.audit, and lineage tables make information about consumption, access, and data movement available within a queryable environment. This creates an opportunity to connect platform governance with FinOps. Instead of knowing only that Databricks spending has increased, teams can investigate which jobs, identities, workloads, and infrastructure contributed to that increase. That visibility can support chargebacks, forecasting, cluster rightsizing, and anomaly detection. In this model, governance is no longer simply about deciding who can access data. It also helps establish who is responsible for using the platform and what that usage costs.

The scope of governance is expanding further as data platforms become AI platforms. Tables are no longer the only assets that require control. Enterprises now have ML models, semantic metrics, functions, document volumes, and AI agent tools operating alongside their analytical workloads. Unity Catalog extends its governance model across these assets, bringing them into the same namespace and lineage framework. Semantic metrics are particularly important because AI systems and BI applications increasingly depend on consistent definitions of business concepts. A metric such as revenue or customer churn can be defined once and reused across dashboards, notebooks, SQL workloads, and AI applications instead of being independently interpreted by every team. As organizations move toward AI-driven decision-making, preventing semantic drift can be just as important as preventing unauthorized access.

The Migration Is Where Strategy Matters

Understanding Unity Catalog is relatively straightforward. Moving a live enterprise estate onto it is not. A migration involves identities, existing Hive metastores, table permissions, storage locations, pipelines, external engines, and business-critical workloads. Databricks Labs' Unity Catalog Migrator, or UCX, provides a structured framework around this work, beginning with assessment and identity synchronization before moving through metastore provisioning, schema mapping, table upgrades, and policy enforcement. The sequencing matters. Identity and SCIM synchronization need to be established before ABAC policies can be reliably applied. Catalog boundaries need to be designed before large numbers of assets are migrated. Existing permissions need to be translated into the new governance model before cutover rather than after it.

A real multi-cloud implementation illustrates why this sequencing matters. Modak recently led a Unity Catalog migration for an enterprise running Databricks across AWS, Azure, and GCP, moving from workspace-isolated Hive metastores to a unified governance model. The engagement followed the UCX approach, using SYNC-based table upgrades rather than turning the migration into a large-scale data-copy exercise, enabling UniForm during the table upgrade phase for cross-engine access, and translating legacy ACLs into tag-based ABAC policies before the final cutover. The engagement reported a 75% reduction in security administration overhead and a 30% reduction in platform runtime cost, while establishing unified governance across structured data, unstructured assets, ML models, and AI workloads.

The Bigger Shift Behind Unity Catalog

Unity Catalog is easy to describe as a data governance technology, but its significance is broader. It provides an operating layer for an enterprise data and AI environment that is increasingly distributed across clouds, workspaces, engines, and workloads. Metadata, identity, security, lineage, interoperability, cost visibility, and AI governance can begin operating from the same foundation instead of being managed as separate systems. That does not make governance automatic, and it certainly does not eliminate the complexity of migration. What it does is give enterprises a consistent architecture in which that complexity can be managed.

The organizations getting the most value from Unity Catalog will therefore not be the ones that treat it as another Databricks feature to enable. They will be the ones that use the migration as an opportunity to rethink how governance should work at enterprise scale. The technology provides the common layer. The real advantage comes from designing the operating model around it.

The original blog was published in https://modak.com/blog/databricks-unity-catalog-enterprise-data-governance-guide

Databricks vs Snowflake: Two Different Philosophies for the Same Data Future

laura williams — Tue, 08 Sep 2026 11:20:39 +0000

At some point, almost every enterprise data leader faces the same question: Databricks or Snowflake? The question sounds straightforward, but the answer rarely is. Both platforms now compete across analytics, data engineering, AI, governance, and increasingly similar enterprise workloads. Yet beneath that convergence are two fundamentally different philosophies about how data platforms should work.

Snowflake was designed around a simple proposition: make cloud data warehousing easier to operate, easier to scale, and easier for analysts to use. Databricks started from a different place, evolving from Apache Spark into a lakehouse platform designed to bring data engineering, analytics, machine learning, and AI together.

That distinction matters. Choosing between them based purely on feature comparisons or vendor benchmarks can lead enterprises toward an architecture that looks impressive on paper but creates friction for the teams actually using it. The better question is not which platform is better. It is which platform aligns with the organization's workloads, skills, operating model, and long-term data strategy.

Two Platforms, Two Starting Points

Snowflake began as a cloud-native data warehouse. Its architecture separates storage, compute, and cloud services into independently managed layers. Virtual warehouses can scale according to workload requirements, while much of the traditional operational work associated with data warehouses is abstracted away.

For SQL-first organizations, that simplicity is powerful. Analysts can query structured and semi-structured data without worrying about cluster management, infrastructure patching, or extensive performance tuning. Snowflake's model is built to make analytics feel like a service rather than an infrastructure project.

Databricks took a different path. Built around Apache Spark and the lakehouse model, it brings data engineering, streaming, analytics, data science, and machine learning onto a shared foundation. Data can remain in open formats such as Delta Lake and Apache Iceberg while different workloads operate against the same underlying data.

In practical terms, Snowflake tends to optimize for the analyst, while Databricks tends to optimize for the data engineer and data scientist.

Where Snowflake Has the Advantage

Snowflake's greatest strength is still simplicity. For organizations whose primary requirement is enterprise BI, reporting, and SQL analytics, the platform can deliver significant value without demanding extensive infrastructure expertise.

Its separation of compute and storage also makes workload isolation straightforward. Independent virtual warehouses allow different teams and workloads to operate without one department's heavy queries necessarily disrupting another's dashboards. For enterprises running large numbers of concurrent BI workloads, that operational model can be particularly attractive.

Snowflake also has a strong story around governed data sharing. Secure Data Sharing and its broader ecosystem make it possible to share governed data across business units and organizations without repeatedly creating physical copies.

This combination of usability, concurrency, and managed operations explains why Snowflake remains compelling for SQL-first enterprises. If the primary goal is to get analysts productive quickly while minimizing platform administration, simplicity becomes a strategic advantage rather than merely a convenience.

Where Databricks Pulls Ahead

Databricks becomes increasingly compelling when the data platform needs to do more than serve SQL queries.

Its lakehouse architecture brings data engineering, streaming, analytics, machine learning, and AI into a common environment. Instead of moving data between a warehouse, a separate data lake, and an ML platform, teams can build pipelines, train models, and develop AI applications against the same governed data foundation.

That matters as enterprises move from traditional analytics toward generative AI and agentic applications. AI initiatives depend on more than models. They require reliable pipelines, high-quality data, lineage, governance, feature engineering, experimentation, and production deployment. Having these capabilities closer together can reduce duplication and simplify the path from raw data to production AI.

Databricks also provides more control over complex workloads. Engineering teams can tune compute, caching, cluster configurations, and execution strategies to balance performance and cost. That flexibility comes with a trade-off: more control creates a larger tuning surface.

For organizations with strong data engineering capabilities, however, that complexity can become an advantage.

Performance Is About Workloads, Not Marketing Slides

The Databricks-versus-Snowflake performance debate is difficult to settle with a single benchmark. Snowflake generally delivers strong SQL performance with minimal tuning, making it particularly effective for structured analytics and BI. Databricks provides greater configurability and is naturally suited to demanding ETL, streaming, unstructured data, and ML workloads.

Both vendors publish benchmarks designed to demonstrate their strengths. Enterprises should therefore treat those comparisons as directional rather than definitive.

The more useful test is the organization's own workload. Data volume, query patterns, concurrency, transformation complexity, streaming requirements, and AI workloads should all be evaluated through a proof of concept using representative data.

AI Is Changing the Decision

The AI layer is where the two platforms have moved closest together.

Snowflake has expanded beyond traditional warehousing through Snowpark and Cortex, enabling organizations to bring Python-based workloads and AI-assisted analytics closer to their data. For companies that primarily operate a SQL-centric analytics environment, this can be an effective way to introduce AI capabilities without adding another major platform.

Databricks, however, has deeper roots in machine learning and AI. MLflow, AutoML, feature management, and the Mosaic AI ecosystem give organizations a broader environment for experimenting with, deploying, and managing ML and generative AI workloads.

The distinction is therefore less about whether either platform supports AI. Both do. The more important question is what kind of AI an enterprise intends to build.

Adding AI-assisted analysis to an existing analytics environment may favor Snowflake. Building and operationalizing custom ML and GenAI applications at scale may favor Databricks.

The Enterprise Decision Is Not Always Either-Or

This is where the debate becomes more interesting.

Some enterprises do not need to choose one platform exclusively. They use Databricks for large-scale engineering, data science, and AI while Snowflake serves business users through a highly governed analytics layer.

That architecture is not necessarily a compromise. In organizations with substantial BI requirements and equally significant AI ambitions, using each platform where it is strongest can be a deliberate architectural decision.

The important part is interoperability. Governance, lineage, access controls, and data-sharing strategies need to be designed with the possibility of multiple platforms in mind rather than treated as an afterthought.

Start With the Workload, Not the Vendor

The most reliable way to approach the Databricks versus Snowflake decision is to start with the organization rather than the product.

How much of the workload is BI and reporting? How much involves ETL, streaming, or unstructured data? How important are machine learning and GenAI? What skills exist within the data team? How much operational complexity can the organization realistically manage?

Those questions usually reveal more than a feature comparison ever will.

Snowflake remains a strong choice for enterprises prioritizing turnkey SQL analytics, concurrency, and operational simplicity. Databricks is particularly powerful when data engineering, AI, machine learning, streaming, and open data architecture are strategic priorities.

And for some enterprises, the strongest architecture may involve both.

The real winner in the Databricks versus Snowflake debate is therefore not the platform with the longer feature list. It is the architecture that aligns technology with how the enterprise actually works—and gives its data teams room to evolve as analytics becomes increasingly inseparable from AI.

AI-First vs AI-Native: The Architecture Decision Most Enterprises Get Wrong

laura williams — Wed, 12 Aug 2026 09:58:18 +0000

There’s a conversation happening in every enterprise technology leadership meeting right now. It goes something like this: the AI pilot worked, the board is excited, and now the question is why scaling feels so much harder than building did.

The models didn’t change. The talent didn’t leave. The budget didn’t shrink. Yet somewhere between proof-of-concept and production, the momentum stalled.

Here’s what nobody in that meeting is saying out loud: the architecture was wrong from the start.

AI-first vs AI-native. What is the difference?

The enterprise technology world has been using “AI-first” and “AI-native” interchangeably. They are not the same thing, and conflating them is quietly responsible for a lot of expensive disappointment.

AI-first means you’ve added AI to existing systems. Your core infrastructure — data pipelines, application logic, workflows — stayed exactly where it was. You layered a model on top, integrated it into a specific task, and called it transformation. The thinking is model-centric: which model, which task, which workflow gets improved.

AI-native means your system was designed around AI from day one. Data pipelines, application logic, and models are built to co-evolve. The thinking is system-centric: how does the system improve over time, what feedback loops are embedded, what does every user interaction teach the next version of the product.

AI-first optimizes for speed of adoption. AI-native optimizes for compounding intelligence. Both sound good. Only one of them scales.

The Real Tell Is the Data Architecture

You don’t need a technical audit to know which category your organization falls into. One question is usually enough.

If your model stopped being updated tomorrow, how would your system degrade?

An AI-first system wouldn’t. It would keep performing exactly as it did on day one — because it was never designed to learn from what happens in production. Feedback from users rarely loops back into training. Models improve on project schedules driven by ML teams, not by the system’s own operational behavior.

An AI-native system would start to deteriorate — because it was built to continuously improve. Training and serving share the same data layer. Every production interaction is a signal. Model updates are operational events, not quarterly projects.

That deterioration, counterintuitively, is evidence of good architecture.

The gap between these two approaches is fundamentally a data engineering problem. AI-first data engineering produces isolated pipelines where training and inference are decoupled. AI-native data engineering is built around continuous data flow — lakehouse-style architecture where interaction data is a first-class input to the learning system, not an afterthought.

Where AI-First Companies Hit the Wall

The ceiling on AI-first architecture arrives faster than most organizations expect — and it arrives quietly.

Each new use case in an AI-first organization requires nearly as much effort as the last. Separate pipelines, separate feature engineering, separate model serving, separate monitoring. Two teams building AI applications on the same underlying data can produce contradictory outputs because they each made different assumptions in their isolated stacks.

You end up with a portfolio of AI projects. Not an AI-capable organization. There’s a meaningful difference.

The most consistent failure pattern I see across data engineering companies attempting to scale AI: they mistake the successful pilot for proof that the architecture works. The pilot worked because it was controlled, scoped, and staffed with the best people. The architecture reveals itself at use case number six, not use case number one.

But AI-Native Isn’t a Safe Destination Either

Here’s where most AI-native advocates lose credibility — they sell the destination without describing the road conditions.

AI-native fails in four specific situations, and being honest about them matters.

It fails when data foundations are immature. Organizations with inconsistent pipelines and poor data quality that attempt AI-native data engineering too early build feedback loops that amplify bad data at scale. The system learns, just the wrong things, faster.

It fails when organizational readiness is overestimated. AI-native requires integrated teams with shared ownership across data engineering, ML, and application development. Most enterprises have siloed structures that took years to build. The architecture is achievable well before the organization is.

It fails when upfront investment can’t be sustained. Rebuilding data infrastructure, standing up shared feature stores, centralizing governance — the time-to-value gap is long. For companies with short runways or immediate ROI pressure, AI-native is the wrong choice regardless of its long-term merits.

And it fails when it becomes a reason not to ship. Teams that over-index on building the “right” AI-native foundation can spend eighteen months on infrastructure and deliver nothing. The architecture becomes institutional perfectionism.

One more structural risk worth naming: in AI-first systems, a broken pipeline breaks one use case. In AI-native systems, a broken shared layer can degrade every use case simultaneously. Higher ceiling, higher blast radius.

So What Should You Actually Do?

The honest answer for most organizations is a sequenced path, not a binary choice.

Start AI-first if you’re early in adoption, if you need to build internal confidence, or if your data foundations aren’t ready for shared infrastructure. AI-first data engineering delivers real value and there’s nothing wrong with using it as a starting point.

But build with the transition in mind. The organizations that get this right aren’t the ones that went AI-native from day one — they’re the ones that started AI-first deliberately, understood its ceiling clearly, and planned the architectural evolution before they hit the wall rather than after.

The risk isn’t starting AI-first. The risk is two things: staying there too long, and moving to AI-native before the foundations can support it. Both are expensive. The second one is less discussed but equally common.

The Question That Actually Matters

Every data engineering company is using AI in some form now. That’s table stakes. The question that separates the ones building durable advantage from the ones building a sequence of well-executed experiments is simpler than it sounds.

Is your architecture designed to improve with use?

If every production interaction automatically generates data that makes the next version of the system better — if feedback loops are embedded, not bolted on — you’re building something that compounds. If your models improve on a schedule rather than from operational signals, you’re building something that plateaus.

The gap between those two outcomes widens every quarter. And it started with an architectural decision that most organizations made without fully realizing they were making it.

Building AI-native data engineering infrastructure is one of the most consequential technical decisions an enterprise can make. Getting the sequencing right matters as much as getting the architecture right.

Everyone Knows What Data Engineering Is. So Why Do AI Initiatives Still Fail?

laura williams — Thu, 06 Aug 2026 17:15:41 +0000

Ask any CDO to define data engineering, and the answer comes quickly: pipelines, governance, moving data from source to something usable. The definition is no longer in dispute.

And yet forecasts are still wrong. Two departments still report different revenue figures from the same quarter. The AI pilot approved last spring is, in most organizations, still a pilot.

This is worth examining, because if the discipline is well understood, the reason it keeps failing at the moment it matters most cannot be a knowledge gap. It is something else: understanding a discipline and holding it to a standard are not the same thing, and most organizations have stopped at the first.

The awareness problem is solved. The trust problem is not

A few years ago, explaining what data engineering was took most of the conversation. That stage is complete. Leadership teams now understand the chain: acquisition, pipelines, integration, governance.

Understanding that the chain exists, however, is not the same as trusting what it produces.

Most enterprises have built the pipelines. Dashboards load. Reports ship on schedule. By every technical measure, the infrastructure is functioning.

What is frequently missing is less visible than infrastructure: whether a decision-maker would act on a figure without independently verifying it first. That instinct to verify, present in most organizations even when the numbers are technically correct, is the signal that the function has not fully matured. It has simply stopped failing in ways that are easy to see.

Why this belongs on the leadership agenda, not the engineering backlog

When that gap in trust surfaces, it rarely presents as a data engineering issue. It presents as a business issue.

A forecast that turns out to be wrong. A regulator's question the organization cannot answer with confidence. An AI initiative that is quietly deprioritized, not because the model underperformed, but because no one was willing to stand behind the data it was trained on.

Data engineering warrants the same organizational attention as cybersecurity and financial controls. It is infrastructure risk that surfaces as business risk, and it draws little attention when functioning and significant attention when it fails, by which point the cost has already shifted from technical to commercial.

The organizations most exposed here are not the ones that misunderstand data engineering. They are the ones that understood it, implemented the fundamentals, and treated that as the endpoint rather than a starting position.

Agentic AI is compressing the timeline for this problem

Historically, a human reviewing a dashboard served as a check against inconsistent or incomplete data before a decision was made.

Agentic AI removes that checkpoint. Autonomous agents now retrieve, join, and act on enterprise data without human review at each step. When the underlying data is technically connected but not fully governed, the consequence is not a flawed report.

It is an action, executed at operational speed, with the error potentially surfacing only after it has already had an effect.

This represents a materially different category of risk than an inaccurate quarterly figure, and one that most governance frameworks were not designed to anticipate.

A more accurate diagnostic than "do we have data engineering"

The more useful question for leadership is not whether the organization has a data engineering function. Nearly every enterprise does, in some form. The relevant question is where that function actually sits on a maturity curve.

Four stages describe most organizations. Ad hoc, where every report is a custom build and no shared source of truth exists. Centralized, where data has been consolidated but ownership remains undefined. Governed, where data is treated as a shared asset with clear accountability, and teams act on it without independent verification. AI-ready, where governance is consistent enough to support automated decision-making without additional oversight.

Many organizations that consider themselves AI-ready are, on closer examination, still operating at the centralized stage. Consolidation was completed years earlier and treated as the final milestone.

That gap between perceived and actual maturity is where most AI initiatives stall.

The question that matters now

For CDOs, CTOs, and CEOs who can already articulate what data engineering is, that is no longer the relevant exercise.

The relevant exercise is determining whether the data engineering function has earned the right to be trusted without verification, whether its current maturity level matches what the business now requires as AI initiatives raise expectations, and who is accountable for closing that gap where it exists.

Organizations that answer this honestly tend to move faster across reporting, analytics, machine learning, and agentic AI, because velocity follows trust rather than producing it.

Organizations that do not tend to continue acquiring new platforms in an effort to resolve an ownership and governance problem that no platform can address.