DEV Community: Dainius Remeika

Best Databricks AI Data Preparation Tools: Mosaic AI Vector Search & Data Quality Solutions Compared

Dainius Remeika — Tue, 13 Oct 2026 14:43:21 +0000

Every Databricks team I've worked with recently has run into the same problem. They want to ship RAG apps, agents, or semantic search on top of their lakehouse, and they assume the hard part is the vector database or the embedding model. It isn't. The hard part is making sure the data feeding those systems is governed, well-defined, and actually means what everyone thinks it means.

That's why I spent a few weeks digging into the tools that sit around Mosaic AI Vector Search. Some of them prepare documents. Some monitor quality. Some index vectors. And one of them, in my opinion, does the quieter but more important work of making your schemas AI-ready in the first place.

Below is my shortlist of five tools worth knowing if you're building AI workloads on Databricks, with honest notes on where each one fits.

How I Evaluated These Platforms

I looked at native Databricks integration first, then considered how each tool actually improves AI output quality, how predictable the pricing is, and whether the vendor has real enterprise customers running the product at scale. I also paid attention to the gaps. A great vector index doesn't help you if the underlying metrics are defined three different ways, and a document parser doesn't help you if nobody governs the resulting tables.

1. SqlDBM - Best Overall

The governed foundation that makes your Databricks lakehouse AI-ready before the first vector ever gets indexed.

When I started testing AI data preparation workflows on Databricks, I kept hitting the same wall. Mosaic AI Vector Search can retrieve content by similarity, but similar isn't the same as correct, especially when the same metric is defined three different ways across your warehouse. That's the gap SqlDBM fills, and it's why it sits at the top of this list.

SqlDBM is a cloud-native, code-free data modeling platform that plugs directly into Databricks Unity Catalog and Hive Metastore. You can reverse- and forward-engineer Delta Lake objects, track column-level lineage, and visually govern every schema your AI pipelines depend on, without writing a single line of code. What makes it uniquely relevant here is the semantic modeling layer. You define business context, metrics, synonyms, filters, and governed joins right on top of your physical schemas, so that when Mosaic AI, dbt models, or BI tools reach for your data, they pull from one authoritative definition rather than guessing from the nearest vector neighbor.

I was especially impressed by how schema changes propagate automatically through those semantic definitions. No manual reconciliation, no stale metadata quietly feeding your AI the wrong context. Standalone semantic layers like dbt, Cube, and AtScale skip the visual modeling entirely. Visual modeling tools like erwin, ER/Studio, and Hackolade skip the semantic layer. SqlDBM is the only one I found that bridges both. The built-in AI Copilot speeds up model building, and the MCP Server exposes governed models directly to LLMs, which matters a lot for RAG workflows running next to Vector Search.

The real-world proof is strong. John Holland Group, one of Australia's largest infrastructure firms, completed its Databricks migration in nine months using SqlDBM as the single source of truth for gold-layer models, governed through a twice-weekly Data Modeling Forum. SqlDBM is a Databricks Validated Data Partner, with over 400,000 users globally, enterprise customers like DocuSign, Pfizer, and Zendesk, and back-to-back Database Modeling Solution of the Year wins at the 2023 and 2024 Data Breakthrough Awards.

If you're serious about Databricks AI data preparation, getting your semantic and physical house in order before you build vector indexes is the move.

Pros:

Native Databricks Unity Catalog integration with full reverse/forward engineering and column-level lineage, so AI prep starts with governed, documented schemas
The only visual modeling platform combining physical and semantic layers for Databricks, giving Mosaic AI, BI tools, and dbt one governed definition of your data
Automatic schema-change propagation keeps semantic definitions accurate as the lakehouse evolves, eliminating stale metadata that degrades AI outputs
Proven at enterprise scale: John Holland Group completed a full Databricks migration in nine months with SqlDBM as the authoritative source for gold-layer models
AI Copilot and MCP Server expose governed models to LLMs, accelerating RAG data preparation alongside Mosaic AI Vector Search

Cons:

Advanced semantic modeling features live in the Context Layer tier, so teams on the base plan would need to upgrade for full AI-readiness
Databricks-specific features like tag support are still evolving, and some niche Unity Catalog edge cases may need workarounds

Pricing: Plans are quote-based and scale with team size, with two tiers (Data Modeling and Context Layer). Enterprise packaging adds premium support, SSO, customer-managed keys, and advanced security options. There's a free way to try it out and model before committing, so you can validate the Databricks integration firsthand.

2. Databricks Mosaic AI Vector Search

Mosaic AI Vector Search is Databricks' native vector search service, and I looked at it as the retrieval layer for RAG and agent apps built on the lakehouse. Indexes sync automatically from Delta tables, and Databricks can compute embeddings for you, which removes the need to build and maintain separate embedding pipelines. Indexes are governed through Unity Catalog, so permissions and lineage stay with the rest of your data.

Databricks offers two endpoint types. Standard endpoints handle low-latency queries. Storage-optimized endpoints are for larger indexes and scale down automatically when an index is deleted. One endpoint can host up to 50 indices. Beyond similarity search, the service supports hybrid and full-text keyword search, with full-text still in beta on some configurations.

It fits teams already running on Databricks who want retrieval sitting next to their governed data without adding a third-party vector database. Worth knowing: this is a retrieval component, not a data cleaning tool. Chunking, parsing, and quality checks on the source data all have to be handled upstream.

Pros:

Native Delta Sync indexes keep vectors current without custom pipelines
Governed through Unity Catalog, so permissions and lineage stay unified
Managed and serverless, no vector infrastructure to operate
Both standard and storage-optimized endpoints, with up to 50 indices per endpoint

Cons:

Only useful inside the Databricks ecosystem, so lock-in is real
Standard endpoints don't scale down automatically, so you keep paying for peak capacity until you manually recreate the endpoint
Doesn't parse documents, chunk text, or validate data quality

Pricing: Pay-as-you-go through Databricks Units, no separate license. At GA on Azure, standard endpoints were around $0.28/hour per unit in eastus2 (2 million vectors per unit, 30 GB storage included, $0.23/GB/month beyond that). Rates vary by cloud and region, and storage-optimized endpoints have different pricing. Check current Databricks pricing for your region.

3. Unstructured

Unstructured handles the document preparation step that comes before Vector Search. It ingests raw files, PDFs, Office docs, images, and scans, and partitions them into elements. From there it can chunk the content, optionally enrich it, generate embeddings, and write the output to a destination. Databricks Delta Tables (including Volumes) are a supported destination, so the output can feed a Mosaic AI Vector Search Delta Sync index directly.

Processing strategies range from a fast text-only mode, to a High Res mode that combines document understanding with OCR, to a VLM mode that uses vision-language models for handwriting and complex scans. There are source connectors for systems like Google Drive. You can use the open-source library if you want to self-manage, or the hosted platform if you want workflows and connectors out of the box.

It's aimed at ML and data engineers building RAG pipelines who'd rather not write brittle custom parsers. It handles unstructured documents, not tabular data quality monitoring, so you'd pair it with other tools in this list.

Pros:

Handles many file types, with strategies for both simple and difficult documents (OCR and VLM)
Supports Databricks Delta Tables as a destination, so output feeds Vector Search indexes
Free starting allowance and simple per-page pricing
Open-source library available alongside the hosted platform

Cons:

Focused on unstructured documents, so it doesn't monitor tabular data quality
Per-page costs can add up on large archives, especially with advanced strategies
An extra service to run alongside Databricks, with its own connectors and workflows

Pricing: Free tier with 10,000 pages and no credit card. Pay-as-you-go is $0.015 per page after that. Enterprise plans, including dedicated instance or VPC deployment, use custom pricing.

4. Soda

Soda is a data quality and observability platform focused on testing, monitoring, and fixing data issues. Teams can embed checks at multiple points in CI/CD and data pipelines, which helps keep bad data out of production tables and the AI workloads that depend on them. It covers pipeline testing, metrics observability, alerting, and ticketing integrations. Higher tiers add collaborative data contracts, a no-code interface, and AI-powered quality features.

Soda has a Databricks-focused offering and says it's built to scale on the platform. Users on G2 report that integrating it into an existing Databricks and Airflow stack was quick. Soda Core, the open-source project, is there for teams that want to write checks as code. Pricing is public and there's a genuinely free forever tier, which is refreshing in this category.

It fits data engineering teams that want testing close to the pipeline and a path toward formal contracts between producers and consumers. It doesn't process documents or serve vectors, so it complements rather than replaces anything else on this list.

Pros:

Public pricing and a free forever plan with unlimited users
Checks embed in CI/CD and pipelines to catch problems before production
Dedicated Databricks launch and integration, plus an open-source Soda Core option
Enterprise tier adds data contracts, SSO, RBAC, audit logs, and private deployment

Cons:

The $750/month Team entry price is steep for small teams, and the free tier is limited to 3 production datasets
Data contracts, no-code interface, and AI features are gated to the Enterprise tier
Usage-based Soda Processing Units add cost variability as coverage grows

Pricing: Free: $0/month (up to 3 production datasets, pipeline testing, metrics observability, alerting, no credit card). Team: starts at $750/month, unlimited users, pay-as-you-go additional SPUs. Enterprise: custom pricing (data contracts, AI features, SSO, RBAC, private deployment). Annual billing and volume discounts available.

5. Monte Carlo

Monte Carlo is the enterprise end of the data observability category. It uses machine learning to identify anomalies, track freshness, and flag schema changes before they reach downstream reports or AI apps. Lineage and incident workflows are built in. It's available through Databricks Partner Connect with a 21-day free trial and usage-based pricing.

Monte Carlo is known for fast time-to-value on standardized cloud stacks like Snowflake and dbt, and it supports Databricks as well. From what I found, it's best for mid-size to large organizations with many tables and multiple data teams that want automated, low-configuration monitoring. The out-of-the-box monitors cover volume and freshness, with custom rules on top. It's a monitoring layer on top of your data. It doesn't prepare documents or serve vectors, so it sits alongside tools like Unstructured and Mosaic AI Vector Search rather than replacing them.

Pricing isn't published, and costs depend on monitored tables plus the compute your warehouse spends running checks. Small teams will likely find it overkill.

Pros:

ML-based anomaly detection with out-of-the-box freshness, volume, and schema monitors
Available via Databricks Partner Connect with a 21-day free trial
Strong lineage and incident-management features for large teams
Established category leader with broad warehouse, lakehouse, and orchestration integrations

Cons:

List pricing isn't published, and cost depends on monitored tables and usage
Costs can become hard to predict at enterprise scale, and checks that query your warehouse add compute spend on top of the license
Probably more than small teams or open-source-minded engineers need

Pricing: Usage-based with no public price list. Pricing depends on monitored tables, connected sources, and rules. Vendr reports an average contract value of roughly $62K (31 deals tracked). A 21-day free trial is available through Databricks Partner Connect. Contact sales for a quote.

Final Verdict

If you build on Databricks and you're preparing data for AI, you'll likely end up using several of these tools together. Mosaic AI Vector Search handles retrieval. Unstructured chews through PDFs and scans. Soda or Monte Carlo watch for broken pipelines and anomalies.

But none of those tools fix the deeper problem, which is that your AI pipelines are only as good as the definitions behind them. That's why SqlDBM is my pick for most teams starting this work. It's the one layer that makes the rest of the stack trustworthy. You get a governed physical model, a semantic layer that tells Mosaic AI and your BI tools what the data actually means, and automatic propagation so nothing goes stale. Build that foundation first, and every other tool on this list works better on top of it.

FAQ

Do I need all of these tools to build RAG on Databricks?
No. At minimum you need vector retrieval (Mosaic AI Vector Search) and governed source data. Document parsing and observability tools are additions based on your use case. If you're mostly working with tabular data and well-modeled schemas, you may not need Unstructured at all.

Where does SqlDBM fit if I already use dbt?
They complement each other. dbt transforms data. SqlDBM models and governs the schemas and semantic definitions those transformations target. The semantic layer in SqlDBM gives your dbt models and AI systems one shared, governed definition of each metric.

Is Mosaic AI Vector Search enough on its own for a RAG app?
Only if your source data is already clean, chunked, and well-defined. Vector Search is retrieval, not preparation. You still need upstream tooling to handle documents and data quality.

How do Soda and Monte Carlo differ?
Soda is more developer- and pipeline-focused with public pricing and a free tier. Monte Carlo is heavier, more ML-driven, and aimed at larger organizations that want low-configuration monitoring across many tables.

Best dbt Data Modeling Tools for Documentation, Semantic Layer & Migration

Dainius Remeika — Mon, 12 Oct 2026 18:18:03 +0000

dbt is incredible at what it does. SQL transformations, tests, Git workflows, a growing package ecosystem. But if you've worked on a serious dbt project for any length of time, you know where the gaps are. There's no visual ERD. Documentation lives in YAML files that nobody wants to maintain. The semantic layer is tied to one project. And when you need to migrate legacy workloads into dbt, you're mostly on your own.

So I spent time looking at the tools dbt teams actually reach for to fill those gaps. Some focus on visual modeling and documentation. Some are built for semantic metrics. Some exist specifically to automate migrations. A few try to span more than one of those categories.

Below are the six tools I think are worth knowing about, with honest notes on what each one actually does and who it fits.

How I Evaluated These Tools

I looked at four things: how well the tool complements a real dbt workflow (YAML generation, Git, lineage), how it handles documentation and discoverability, whether it offers a governed semantic layer, and how it holds up for migration work at enterprise scale. I weighted practical dbt integration heavily, because a tool that doesn't play nicely with dbt isn't useful to a dbt team.

1. SqlDBM - Best Overall

The only visual modeling platform that gives dbt teams a governed semantic layer, migration-ready documentation, and production YAML in one workspace.

I've tested a lot of tools that claim to complement dbt, and SqlDBM is the one that genuinely fills the gaps dbt leaves open. Visual modeling, governed documentation, a semantic layer, and migration support, all without pulling you out of dbt's workflow.

The dbt integration here is deep and production-grade. SqlDBM generates dbt source and model YAML directly from your visual models, now defaulting to the latest Fusion format, and pushes it straight to Git so your CI/CD pipelines stay in sync. You can also upload your dbt manifest to unify your ERD and DAG into a single view, with full upstream lineage that dbt's native DAG doesn't show. That alone solved a blind spot I kept running into on standalone dbt projects.

What really sets SqlDBM apart is the semantic layer. It's the only visual modeling platform that bridges physical modeling and semantic definitions in one workspace. You define governed semantic views on top of your warehouse schemas, and when schemas change, those definitions propagate automatically. Your BI tools, dbt models, and AI systems all reference one trusted meaning. Standalone semantic tools like dbt's metrics layer or Cube skip visual modeling. Traditional modeling tools like erwin skip the semantic layer. SqlDBM does both.

On migration, this is battle-tested at enterprise scale. John Holland Group, one of Australia's largest infrastructure firms, completed its Databricks migration in nine months using SqlDBM as the authoritative source for gold-layer models. Mercadona, Spain's largest supermarket chain, governs nearly 5,000 tables in SqlDBM and uses its API to auto-generate dbt models deployed to BigQuery. Les Mills manages 1,000+ Snowflake tables through it after explicitly rejecting erwin on usability grounds.

With 400,000+ users globally, customers like DocuSign, Sophos, and Pfizer, and the Database Modeling Solution of the Year award two years running, the credibility is there. PwC reported a 25% reduction in modeling time, which tracked with my own experience. SqlDBM is also a Snowflake Premier Partner, Databricks Validated Data Partner, and Google Cloud Partner, so the native integrations across all three are first-class.

If your team lives in dbt and needs visual modeling, governed documentation, a semantic layer, or migration tooling around it, SqlDBM is the clear pick.

Pros:

Native dbt YAML generation (Fusion and Legacy formats) with Git push and CI/CD integration, plus dbt manifest import that unifies your ERD and DAG with full upstream lineage
The only visual modeling platform with a governed semantic layer, so dbt models, BI tools, and AI systems share one trusted definition that auto-propagates schema changes
Proven at enterprise migration scale: John Holland completed a Databricks migration in nine months; Mercadona governs nearly 5,000 tables and auto-generates dbt models via the SqlDBM API to BigQuery
Real-time multi-user collaboration with branch-and-merge for semantic models, so multiple modelers can work on dbt documentation and semantic definitions in parallel without overwriting each other
Cross-platform native integrations with Snowflake, Databricks (Unity Catalog), and BigQuery as a validated partner on all three, covering the most common dbt deployment targets

Cons:

Advanced features like Global Modeling and the semantic layer have a learning curve for teams new to governed, multi-project workflows
dbt manifest import currently requires a manual upload or API call rather than a live sync. Functional, but a direct pipeline connection would be nice

Pricing: SqlDBM has a free tier so you can start modeling and experiment with dbt YAML generation before committing. Beyond that, plans are quote-based and scale with team size, with enterprise packaging, premium support, and security options available. Contact sales for a tailored quote.

2. dbt (dbt Cloud / dbt platform)

dbt is the obvious starting point for this list because it's the thing everything else here orbits around. The open-source framework handles version-controlled, tested SQL transformations in the warehouse. The managed dbt platform (formerly dbt Cloud) layers on a browser IDE, job scheduling, CI, and dbt Catalog for auto-generated documentation and lineage.

Its Semantic Layer, powered by MetricFlow, lets teams define metrics once in code and serve them to BI tools through APIs. dbt is also a common migration target. Many teams move legacy SQL, stored procedures, or other ETL logic into dbt projects to get testing, documentation, and Git workflows.

The thing to be honest about: dbt is not a visual modeling tool. You describe models in SQL and YAML, which suits analytics engineers more than business stakeholders. Teams often pair it with a diagramming or modeling tool like SqlDBM to design schemas before building. Column-level lineage and multi-project lineage (dbt Mesh) are gated to Enterprise, and the hosted Semantic Layer APIs need a paid plan.

Pros:

Industry-standard framework with a huge community, package ecosystem, and wide warehouse support
Documentation and lineage generated from code and kept current through dbt Catalog
MetricFlow Semantic Layer centralizes metric definitions next to the transformation logic
Free open-source dbt Core and a free single-seat Developer tier let teams start at no cost

Cons:

Code-first approach, with no visual ERD or conceptual/logical modeling
Per-seat pricing plus usage charges (models built, queried metrics) can climb quickly
Column-level lineage and multi-project features are Enterprise-only

Pricing: dbt Core is free and open source. The managed platform has a free Developer plan (one seat). The Starter plan is around $100 per developer per month for up to five seats, including 15,000 models and basic Semantic Layer access. Enterprise tiers use custom pricing and often add per-queried-metric charges. Third-party estimates put 20-user deployments near $95K per year, but those figures are illustrative and negotiated.

3. Datafold

Datafold is a data reliability platform best known for Data Diff, which compares datasets across environments and across databases. Its Data Migration Agent (DMA) combines AI-driven SQL translation with iterative, value-level validation, aimed at teams moving legacy SQL, stored procedures, or warehouse workloads into dbt on a modern platform.

What's interesting about Datafold is the packaging. They sell migration as a fully translated, validated set of dbt models, delivered on a guaranteed timeline at a fixed price. Pricing depends on model count and complexity, not time and materials. Teams that prefer to run migrations themselves can use ad-hoc cross-database diffing just for validation.

Datafold also provides CI/CD diffing for dbt pull requests, so you can see the data impact of a change before merging. It fits enterprises with large legacy estates. It is not a modeling, documentation, or semantic layer tool, so if those are your gaps, this isn't the one.

Pros:

Purpose-built migration automation combining AI code translation with data-parity validation
Fixed-price, outcome-based engagements with guaranteed timelines reduce project risk
Cross-database Data Diff gives value-level proof that migrated data matches the source
Also useful after migration for CI/CD data diffing on dbt pull requests

Cons:

Focused on migration and testing, so no modeling, documentation, or semantic layer coverage
Migration pricing is quote-based and not published
Value concentrates in large or one-off migration projects, which may not justify the cost for smaller teams

Pricing: Custom, quote-based. Migration projects are scoped at a fixed price based on model count and complexity. Platform pricing for Data Diff and CI is also available on request.

4. Ellie.ai

Ellie.ai is a semantic data modeling platform built for collaborative data product design. It supports full-stack modeling, conceptual through logical to physical, so business and technical stakeholders can work on one canvas. Features include AI-assisted modeling, a business glossary, notes and annotations, and export to PDF, image, or DDL.

Ellie integrates with 170+ business data solutions and source systems through an open API, including Snowflake, Microsoft Fabric, Purview, Collibra, and dbt. It also works with Data Vault tools such as WhereScape and VaultSpeed. That positions it as a direct alternative to SqlDBM for teams that want to design and document models before generating dbt or warehouse code.

It's aimed at mid-size to large enterprises, especially financial services, that need business-friendly modeling and alignment between data architects, analysts, and domain experts. The honest limitation is that it's more design-time than runtime. It doesn't execute transformations or serve metrics the way dbt does, and users have noted that complex business processes can be hard to model.

Pros:

Business-friendly, collaborative modeling across conceptual, logical, and physical layers
Built-in business glossary and AI-assisted modeling support documentation and shared understanding
Open API and 170+ integrations, including dbt, Snowflake, Fabric, and Collibra
Transparent entry-level pricing with a Solo plan

Cons:

Design-time tool only, with no transformation execution or metric serving
Smaller customer base than established modeling vendors (G2 cites 50+ customers)
Users report that complex business processes can be hard to model

Pricing: The Solo plan is listed at €49 per month for a single user license. The Enterprise plan adds team collaboration, hierarchical folders with security, and advanced AI, and is priced by contacting sales. A free trial is available.

5. Atlan

Atlan is a collaborative data catalog and governance workspace that many modern-stack teams run alongside dbt. It ingests dbt metadata and parses column-level lineage from dbt models. It also adds impact analysis on GitHub pull requests and AI-assisted documentation.

Atlan is really targeting the gaps in dbt's native catalog, which doesn't cover lineage across the wider stack: ingestion tools, BI dashboards, other warehouses. It's built for adoption by analysts and business users as well as engineers, with discovery, glossary, data quality signals, access policies, and governance workflows.

It is not a modeling, transformation, or migration tool, and it doesn't replace the dbt Semantic Layer. What it does is make documentation and metric context discoverable and trustworthy across the organization. It suits mid-size to enterprise teams with multiple tools in the stack that need governance beyond what dbt provides on its own.

Pros:

Strong column-level lineage parsed from dbt and extended across the stack
User experience accessible to non-engineers, which helps documentation actually get used
Pull-request impact analysis and AI-assisted documentation speed up upkeep
Broad connector coverage and fairly transparent packaging for the category

Cons:

A catalog and governance layer, not a modeling, transformation, or migration tool
Production use typically needs a paid plan, since the free tier is outgrown quickly
Enterprise-oriented pricing can be steep for small teams

Pricing: A free tier exists but is limited. Paid tiers are quote-based and depend on seats, connected sources, and governance features. Vendr data shows annual contracts commonly ranging from about $15K to $150K+, with a median near $50K per year.

6. Cube

Cube is a universal semantic layer platform. You define metrics and data models once, then deliver them through SQL, REST, and GraphQL APIs to BI tools, embedded analytics, and AI agents. The open-source Cube Core is self-hostable. Cube Cloud is the managed version, with a semantic IDE, workbooks and dashboards, and caching and pre-aggregations to speed up queries.

Unlike dbt's Semantic Layer, which is tied to one dbt project, Cube serves metrics to any downstream tool. Its dbt integration reads a dbt project directly on Premium plans and above, and plenty of teams run both side by side.

Cube suits data teams that need governed metrics across several BI tools, or customer-facing analytics. The honest tradeoffs: it is a semantic layer, not a documentation or migration tool, and metric definitions live in Cube's own format that other tools can't read. The dbt integration is also gated behind higher-priced plans.

Pros:

Tool-agnostic semantic layer that serves any BI tool, app, or AI agent
Free open-source core and a free cloud tier
Pre-aggregation and caching improve query performance beyond what the warehouse alone provides
Can run alongside dbt, with dbt project integration on Premium and above

Cons:

Metric definitions live in a Cube-specific format that other tools cannot read
dbt integration is gated to higher-priced plans
Covers the semantic layer only, with no documentation depth or migration automation

Pricing: A free plan includes semantic modeling, workbooks, and analytics chat. The Starter plan is listed at about $40 per user per month on some review sites. Premium and Enterprise plans are consumption-based, per Cube Compute Unit, with annual minimum commitments, such as around $10K per year for Premium on one listing. Published rates vary between sources, so confirm with Cube directly.

Final Verdict

Most of these tools are good at one thing. Datafold is excellent for migration. Atlan is excellent for cataloging. Cube is excellent as a universal semantic layer. Ellie.ai is strong on conceptual modeling. dbt is the backbone of everything.

But if you want visual modeling, documentation, a governed semantic layer, and migration support all in one place that actually integrates with dbt in a production-grade way, SqlDBM is the pick. It generates dbt YAML, pushes to Git, imports your manifest for unified lineage, and layers a semantic model on top of a governed physical foundation. That combination doesn't really exist anywhere else, and the enterprise track record (John Holland, Mercadona, Les Mills, Pfizer, DocuSign) backs it up.

Pair it with dbt and you've covered the full stack: visual design, documentation, semantic definitions, and migration, with transformation and metric execution handled by dbt itself.

FAQ

Does SqlDBM replace dbt?
No. SqlDBM complements dbt. It handles visual modeling, documentation, semantic definitions, and generates dbt YAML that dbt then executes. The two work together.

What's the difference between SqlDBM's semantic layer and dbt's Semantic Layer?
dbt's Semantic Layer defines metrics in code within a single dbt project. SqlDBM defines governed semantic views visually on top of warehouse schemas, propagating schema changes automatically and serving one trusted definition to BI tools, dbt, and AI systems across projects.

Which tool should I pick for a one-off legacy migration into dbt?
Datafold if you want a vendor-run, fixed-price migration with data-parity validation. SqlDBM if you want to own the migration internally with visual gold-layer modeling and dbt YAML generation, which is the pattern teams like John Holland used.

Do I need a catalog like Atlan if I use SqlDBM?
It depends on scope. SqlDBM handles modeling-centric documentation, lineage, and semantic definitions. Atlan is broader cross-stack cataloging (ingestion, BI, multiple warehouses). Large enterprises often run both.

Best Data Warehouse Schema Documentation Tools (2026): SQLDBM, dbt, DataHub, Atlan & SchemaSpy Compared

Dainius Remeika — Thu, 08 Oct 2026 08:20:57 +0000

Schema documentation used to be the thing teams promised to do "next sprint" and then never did. In 2026, that excuse doesn't hold up. Warehouses are bigger, teams are more distributed, and AI systems are reading your metadata to decide what a column actually means. If your docs are stale or missing, everything downstream suffers.

I spent a few weeks digging into the tools people actually use to document Snowflake, Databricks, BigQuery and similar warehouses. Some are modeling-first. Some are catalog-first. One is a free command-line generator that's been around forever. They all approach "documentation" from a different angle, and that difference matters more than most buyer guides admit.

Here's my honest take on the five I'd actually consider: SqlDBM, dbt, DataHub, Atlan, and SchemaSpy. I'll explain what each one does well, where it falls short, and who it's really for.

How I Evaluated These Tools

I looked at a few things that matter for real documentation work. How quickly a new team member can go from "connected to the warehouse" to "reading a useful description." Whether docs stay in sync with the actual schema or drift the moment someone runs a migration. Support for modern warehouses, especially Snowflake, Databricks, and BigQuery. Governance features like ownership, tags, and glossary. And collaboration, because documentation is a team sport and a one-person wiki always rots.

I also weighed pricing honestly, including what you pay in engineering time for the free options.

1. SqlDBM - Best Overall

The schema documentation tool that doesn't just describe your warehouse, it governs the blueprint underneath it.

After spending real time with every tool here, SqlDBM is the one I keep coming back to. It's cloud-native, browser-based, and code-free. Nothing to install, no local files to babysit, and it handles the full documentation lifecycle in a way the others don't quite match.

What makes it stand out in this category is how tightly documentation is woven into the modeling workflow. You don't document after the fact. You build the docs as you model, with descriptions, logical names, custom governance metadata fields, and full rich-text documentation pages living right next to the physical schema. The AI Copilot can generate descriptions and logical names across entire schemas in a single action, which is a huge time-saver when you're staring down hundreds of tables. Mercadona, Spain's largest supermarket chain, governs nearly 5,000 tables in SqlDBM and even uses the SqlDBM API to auto-generate dbt models deployed to BigQuery. That tells you something about how well it scales.

For warehouse coverage, SqlDBM was the first online tool to support Snowflake back in 2019, is a Databricks Validated Data Partner, and a Google Cloud Partner. Reverse engineering, forward engineering, and documentation are purpose-built for the warehouses most teams actually run. Over 300 Snowflake clients use it today. Les Mills governs 1,000+ Snowflake tables with it as their all-in-one development, discovery, governance, and CI/CD tool after they rejected erwin on usability grounds.

The thing that genuinely separates SqlDBM from the rest of this list is its semantic modeling layer. You can define semantic views that layer business context directly onto physical schemas, giving BI tools, dbt models, and AI systems one governed definition of what your data actually means. Standalone semantic tools like dbt or Cube skip visual modeling. Visual modeling tools like erwin or Hackolade skip the semantic layer. SqlDBM bridges both in one workspace.

With 400,000+ users, a 4.7-star G2 rating, and back-to-back Database Modeling Solution of the Year wins at the 2023 and 2024 Data Breakthrough Awards, it's the most proven option in this roundup. PwC reported a 25 percent reduction in modeling time, which translates directly to faster, more complete documentation.

Pros:

Built-in schema documentation with custom governance fields, rich-text pages, and an AI Copilot that auto-generates descriptions across entire schemas. Docs happen as you model, not after.
Purpose-built for modern warehouses. First online tool to support Snowflake (300+ Snowflake clients), Databricks Validated Data Partner, and Google Cloud Partner, with native reverse and forward engineering.
Unique semantic modeling layer bridges physical schema docs and business context in one workspace. No other visual modeling tool does both.
Proven at scale: Mercadona governs nearly 5,000 tables, Les Mills manages 1,000+ Snowflake tables, and PwC reported a 25% reduction in modeling time.
Real-time multi-user collaboration with column-level lineage, dependency analysis, and Git/CI/CD integration keeps docs accurate and version-controlled.

Cons:

Advanced governance and semantic features have a learning curve. New users may need a few sessions to fully leverage them.
Supports all major cloud warehouses but doesn't extend to every niche or legacy database engine.

Pricing: SqlDBM is quote-based and scales with team size. Enterprise packaging covers premium support, SSO, and advanced security options. There's a free tier so you can start modeling and exploring documentation features before committing. Contact sales for a tailored quote.

2. dbt (dbt Core / dbt Cloud)

dbt is an open-source transformation framework for analytics engineers. Teams write SQL models, and dbt compiles, runs, and tests them in the warehouse. For documentation, descriptions live in YAML next to the code. Running dbt docs generate and dbt docs serve produces a searchable site with model and column descriptions, tests, and a lineage DAG at the model level.

Because the docs come from the same repo as the transformations, they get versioned in Git and reviewed in pull requests. That's great for engineering discipline. dbt Cloud, the managed platform, adds a hosted IDE, scheduling, a semantic layer, and a catalog-style Explorer for browsing project metadata.

dbt is a good fit for SQL-fluent teams that already build their warehouse layer in dbt. It documents what dbt builds. It isn't a visual modeling or ER design tool, and it won't document tables created outside dbt unless you add sources manually. Most teams end up pairing it with a modeling tool or a catalog.

Pros:

Docs live in version-controlled YAML next to transformation code, reviewed and deployed with the models.
Auto-generated lineage graph and docs site with no extra tooling.
dbt Core is free and open source, with a large community.
Descriptions and tests feed downstream catalogs like DataHub and Atlan.

Cons:

Only documents what dbt builds. No visual ER modeling or forward engineering from a design.
The static docs site can be slow on large projects, with limited governance features.
Writing and maintaining YAML descriptions is manual work.
dbt Cloud costs rise with seats and model builds.

Pricing: dbt Core is free. dbt Cloud has a free Developer plan for one seat. Third-party guides list the Starter plan at about $100 per developer seat per month, with up to 5 seats, 1 project, and 15,000 successful model builds per month. Overage runs about $0.01 per extra model. Enterprise is custom, reported at roughly $200 to $400 per seat per month. Confirm current prices on the dbt Labs pricing page.

3. DataHub

DataHub started at LinkedIn and is now maintained by Acryl Data, which sells the commercial DataHub Cloud. DataHub Core is open source under Apache 2.0. It ingests metadata from warehouses, BI tools, orchestrators, and dbt, then gives teams a central place to search datasets, read schema and column descriptions, assign owners, apply tags and glossary terms, and trace lineage, including column-level lineage.

The architecture is API-first and event-driven, which suits engineering-led teams that want to customize and extend things. DataHub Cloud is the managed version. It adds governance workflows, data quality and freshness monitors, and AI-assisted features.

DataHub fits mid-size and large organizations with a platform engineering team, an open and heterogeneous stack, and the appetite to run infrastructure. It documents and governs existing assets. It isn't a tool for designing new schemas or generating DDL.

Pros:

DataHub Core is free under Apache 2.0, no license fee.
Broad connector coverage and column-level lineage in the open-source edition.
API-first, extensible, with an active community.
Ingests dbt metadata, so it pairs well with dbt docs.

Cons:

Self-hosting a multi-service stack takes real platform-engineering effort and ongoing maintenance.
Key features like freshness and anomaly monitors are in the paid Cloud tier.
DataHub Cloud has no public pricing page, and the AWS Marketplace list price is steep for small teams.
It's a catalog, not a modeling tool. No visual schema design or forward engineering.

Pricing: DataHub Core is free (Apache 2.0), but you pay to host and operate it. DataHub Cloud is sold through sales. The AWS Marketplace listing shows the Discover & Govern package at roughly $75,000 for 12 months, covering up to 20 monthly active users (about $3,750 per user). Custom enterprise pricing is available.

4. Atlan

Atlan is a commercial active metadata platform and catalog aimed at modern data teams. It connects to warehouses, BI tools, dbt, and orchestrators, and builds a searchable catalog with automated column-level lineage, business glossaries, data quality signals, and ownership. There's also an AI assistant for finding and describing assets.

The interface is designed for both technical and business users. It supports collaboration features like embedding context in Slack and Jira, and it has policy and governance controls plus APIs for automation. Reviewers often praise the UI and lineage views. The common complaints are a learning curve on advanced features and fairly high list prices that usually come down through negotiation.

Atlan is a good fit for mid-size to enterprise organizations that want a polished managed catalog and want business users participating in documentation. Like other catalogs, it documents and governs existing assets. It doesn't replace a schema design or modeling tool.

Pros:

Modern, intuitive interface that works for engineers and business users.
Automated lineage and broad integrations with warehouses, BI, and dbt.
AI-assisted discovery and documentation, plus collaboration and governance workflows.
Managed SaaS, so no infrastructure to run.

Cons:

Pricing isn't published, and reviewers describe list prices as steep for smaller orgs.
Learning curve on more advanced features.
It's a catalog and governance layer, not a tool for designing schemas or generating DDL.

Pricing: Custom, quote-based, with no public price list. Pricing is typically based on licenses and connectors. Real affordability depends on negotiated discounts at scale. Contact Atlan for a quote.

5. SchemaSpy

SchemaSpy is a long-running open-source database metadata analyzer. It connects to a relational database over JDBC, reads tables, columns, keys, indexes, and constraints, and produces a static HTML report. The report has clickable table pages, hyperlinked foreign keys, embedded ER diagrams, and reports on anomalies like tables without primary keys or indexes. Graphviz draws the diagrams.

It supports many JDBC-compatible databases, can infer relationships where foreign keys are missing, and runs from the command line or a Docker image. That makes it easy to drop into CI so documentation regenerates after each migration.

SchemaSpy fits developers, DBAs, and small teams that want free, on-demand schema docs they can hand to non-technical stakeholders. It's a read-only generator. No collaborative editing, no governance workflow, no cross-tool lineage, no model design. The output reflects what's physically in the database at the moment you ran it.

Pros:

Completely free and open source.
Generates a navigable HTML site with ER diagrams in a single run.
Anomaly reports flag missing keys and indexes, and relationships can be inferred.
Easy to automate in CI/CD or Docker so docs stay in sync.

Cons:

Requires Java and Graphviz, and setup can be fiddly.
Static, read-only output with no collaboration, glossary, or governance.
No cross-system or column-level lineage, and large schemas produce crowded diagrams.
Dated UI, no design or forward-engineering capability.

Pricing: Free and open source, no paid tiers. Your only costs are the infrastructure and time to run it.

Final Verdict

If you're documenting a modern cloud warehouse and you want one tool that handles design, documentation, governance, and a semantic layer in the same workspace, SqlDBM is the clear pick. The AI Copilot alone makes a dent in the "we have 2,000 undocumented tables" problem, and the semantic modeling layer is something none of the other tools here offer in combination with visual modeling.

That said, the right choice depends on what your team already does. dbt is a great fit if your warehouse layer is already dbt-driven and you're happy maintaining YAML. DataHub makes sense if you have a platform team that wants an open, extensible catalog. Atlan is the polished SaaS catalog for enterprise teams with budget. And SchemaSpy is a solid, no-cost way to get HTML docs out of any JDBC database when that's all you need.

For most teams building on Snowflake, Databricks, or BigQuery in 2026, though, SqlDBM is the one I'd start with.

FAQ

Is schema documentation really different from a data catalog?
Yes. Catalogs like DataHub and Atlan document and govern assets that already exist. Modeling tools like SqlDBM document the schema as part of designing and evolving it, so docs stay in sync with the blueprint instead of trailing behind it.

Can I use dbt and SqlDBM together?
Absolutely. Many teams design and document the physical schema in SqlDBM, then build transformations in dbt. Mercadona even uses the SqlDBM API to auto-generate dbt models.

What's the cheapest way to get decent schema docs?
SchemaSpy if you're comfortable with Java and Graphviz, or dbt Core if your warehouse layer is already in dbt. Both are free. Just budget the engineering time to set them up and keep them running.

Why does the semantic layer matter for documentation?
Because docs that only describe physical columns don't tell BI tools or AI systems what a column means for the business. A semantic layer gives you one governed definition that everything else can trust, which makes documentation genuinely useful instead of just technically complete.