DataEngineeringZoomcamp #dbt #AnalyticsEngineering #DataModeling
Part 1: Introduction to Analytics Engineering & dbt Fundamentals 🎯
What is Analytics Engineering?
The Evolution of Data Roles
Traditionally, there were two main roles in data:
| Role | Focus | Skills |
|---|---|---|
| Data Engineer | Building pipelines, infrastructure, data movement | Python, Spark, Airflow, cloud services |
| Data Analyst | Creating reports, dashboards, insights | SQL, Excel, BI tools |
But there was a gap! Who transforms the raw data into clean, analysis-ready tables? Enter the Analytics Engineer.
What Does an Analytics Engineer Do?
An Analytics Engineer sits between Data Engineering and Data Analytics:
┌─────────────────┐ ┌──────────────────────┐ ┌─────────────────┐
│ Data Engineer │ ──► │ Analytics Engineer │ ──► │ Data Analyst │
│ │ │ │ │ │
│ • Pipelines │ │ • Transform data │ │ • Dashboards │
│ • Infrastructure│ │ • Data modeling │ │ • Reports │
│ • Data movement│ │ • Quality tests │ │ • Insights │
└─────────────────┘ │ • Documentation │ └─────────────────┘
└──────────────────────┘
Key responsibilities:
- 📊 Transform raw data into clean, modeled datasets
- 🧪 Write tests to ensure data quality
- 📝 Document everything so others can understand
- 🔗 Build the "T" in ELT (Extract, Load, Transform)
The Kitchen Analogy 🍳
Think of a data warehouse like a restaurant:
| Restaurant | Data Warehouse | Who accesses it |
|---|---|---|
| Pantry (raw ingredients) | Staging area (raw data) | Data Engineers |
| Kitchen (cooking happens) | Processing area (transformations) | Analytics Engineers |
| Dining Hall (served dishes) | Presentation area (final tables) | Business users, Analysts |
Raw ingredients (data) come in, get processed (transformed), and are served as polished dishes (analytics-ready tables).
What is dbt? 🛠️
dbt stands for data build tool. It's the most popular tool for analytics engineering.
The Problems dbt Solves
Before dbt, data transformation was messy:
- ❌ SQL scripts scattered everywhere with no organization
- ❌ No version control (changes got lost)
- ❌ No testing (errors discovered too late)
- ❌ No documentation (nobody knew what anything meant)
- ❌ No environments (changes went straight to production!)
dbt brings software engineering best practices to analytics:
- ✅ Version control - Your SQL lives in Git
- ✅ Modularity - Reusable pieces instead of copy-paste
- ✅ Testing - Automated data quality checks
- ✅ Documentation - Generated from your code
- ✅ Environments - Separate dev and prod
How dbt Works
dbt follows a simple principle: write SQL, dbt handles the rest.
┌─────────────────────────────────────────────────────────────┐
│ Your dbt Project │
│ │
│ ┌───────────────┐ ┌───────────────┐ ┌────────────┐ │
│ │ models/*.sql │───►│ dbt compile │───►│ SQL Queries│ │
│ │ (your logic) │ │ dbt run │ │ (executed) │ │
│ └───────────────┘ └───────────────┘ └────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ Data Warehouse │ │
│ │ (views/tables) │ │
│ └──────────────────┘ │
└─────────────────────────────────────────────────────────────┘
- You write SQL files (called "models")
- dbt compiles them (adds warehouse-specific syntax)
- dbt runs them against your data warehouse
- Views/tables are created automatically!
dbt Core vs dbt Cloud
| Feature | dbt Core | dbt Cloud |
|---|---|---|
| Cost | Free (open source) | Free tier + paid plans |
| Where it runs | Your machine/server | Cloud-hosted |
| Setup | Manual installation | Browser-based IDE |
| Scheduling | Need external tool | Built-in scheduler |
| Best for | Local development, cost savings | Teams, ease of use |
💡 For this course: You can use either! Local setup uses DuckDB + dbt Core (free). Cloud setup uses BigQuery + dbt Cloud.
Top comments (0)