Learn how to choose what version of dbt to adopt
Photo by Pietro Jeng on Unsplash
What is dbt?
Because my background is SRE/DevOps, I like to equate dbt to terraform for data. It is a great data transformation tool, and it allows you to recreate your data warehouse from scratch with proper version control.
The team dbt team believes that data should be built like any other application powering work like any engineer and focused on SQL. A low barrier of entry language.
If you do not know what dbt is, then you have not been in the modern data stack in the last few years. Check out the dbt tag in medium https://medium.com/tag/dbt, which has many great articles. Or you can try this article from dbt on the basics of dbt.
dbt versions
dbt has an open-source version called dbt core. The commercial offering is dbt Cloud, which comes in three editions: Developer, Team and Enterprise.
I will focus my comparison between dbt core and dbt enterprise, with bits for Team.
Supports: Postgres, Redshift, BigQuery, Snowflake, Spark, Databricks, Starburst/Trino, Microsoft SQL Server, Microsoft Azure Synapse DW, Exasol Analytics, Oracle Database, Dremio
dbt core
dbt core is the open-source transformation workflow that you can modularize all the analytical code, while also giving to test the transformations using proper testing. It provides a single source of truth with full metrics coverage.
This gives you the ability to define tests, which helps to reduce errors as you evolve the model logic. You can even create monitoring and alerting.
I like to compare dbt core to terraform in the world of infrastructure as core, and it is widely adopted by the data community and used across many of the data warehousing. The study State of Datasurvey shows dbt is getting close to pandas for transformations, but it is also the most positively looked at by the industry. For us at Infostrux, it seems to be around 90% of engagements use dbt.
dbt cloud — enterprise
Please note that this comparison is current as of January 2024 and it can evolve. I will try to keep updated as I receive new information but always best to contact the dbt team for latest information.
Here are some features that you can gain with dbt cloud:
- Integrated Development Environment (IDE) with integrated Continous Integration (CI)
- dbt Semantic Layer, centrally define your metrics, ensure consistency, eliminated duplicate coding as you can define the metrics on top of existing models
- dbt explorer, this tool allows a view into models, tests, and metrics and their lineage. You get a clear view into your data production current state
- Job scheduling, it provides you with a simple workflow engine (
- Jobs Runs, successful models built per month (in deployment) 3,000 for developer, 15,000 for teams and Custom for enterprise
- Single Sign On (SSO), for enterprise only, it gives the ability use your own SSO accounts to login to dbt cloud
- Multi-region deployment (https://docs.getdbt.com/docs/cloud/about-cloud/regions-ip-addresses), for enterprise only, it provides the ability to choose your deployment zone
- Native support for GitHub, GitLab, and Azure DevOps, note that all version of dbt cloud support GitHub and GitLab but only Enterprise support Azure DevOps
- Source freshness reporting, takes snapshots of the data to report on how fresh data is and alert if out of SLAs
- Outbound webhooks, send events with HTTP web calls
- Full set of API’s to drive dbt cloud (Administrative, Discovery and Semantic APIs)
- Compliant environment with many common frameworks, ISO 27001:2013, ISO 27001:2019, SOC2 Type II, PCI, GDPR, and HIPAA
- finally, you gain an Service Level Agreements (SLAs) in the Enterprise plan, professional services and support
Visual Comparison
Page 1
Page 2
Conclusion
At Infostrux, we are using dbt for most of our client and often companies do choose dbt core; however, many also chose dbt cloud principally if they are initially trying to run their data warehouse through a full SaaS model. I hope this article does give you some visibility into the differences.
I think the dbt business model has been evolving, and they did not have a clear explanation of the benefits of DBT Enterprise.
Having a dbt workflow and run engine is benefit of dbt cloud Teams, but often companies use their CI/CD like GitHub Actions or a real workflow engine like Apache Airflow.
You can see that the benefits of dbt Enterprise are becoming much stronger now, and I believe they will continue to improve as they add new features that are only in Enterprise. However, dbt core is still a very compelling software and predict that majority of clients will continue to use it for the next while.
I’m Augusto Rosa, VP of Engineering for Infostrux Solutions. Thanks for reading my blog post. You can follow me on LinkedIn. Subscribe to Infostrux Medium Blogs https://medium.com/infostrux-solutions for the most interesting Data Engineering and Snowflake news.
Top comments (0)