DEV Community: Anthony Gicheru

Why Your Code Breaks in Production (and How Docker Fixes It)

Anthony Gicheru — Tue, 12 May 2026 05:28:52 +0000

1. Why This Matters

You write your code.
You test it locally.
Everything works perfectly.

Then it goes to production… and breaks.

You spend hours debugging, only to realize:
nothing is wrong with your code — the environment is the problem.

In data engineering, this happens all the time:

A Spark job runs locally but fails in production
Airflow works on Ubuntu but breaks on macOS
Kafka pipelines behave differently across environments

At its core, the issue is simple:

Your environment is not consistent.

Containerization solves this by packaging everything your application needs into a single, portable unit that runs the same way anywhere.

2. Core Concept — What is Containerization?

Let’s simplify it with an analogy.

Analogy: A Fully Equipped House

Imagine being placed in an empty field with nothing around you.

No food.
No water.
No electricity.
No shelter.

You might survive for a while, but functioning properly would be difficult.

Now imagine being placed inside a fully equipped house.

Everything you need is already there:

food
water
electricity
furniture
internet
a bed

No matter where that house is moved, you can still live comfortably because your essentials move with you.

Applications work the same way.

An application needs certain things to function:

libraries
runtime versions
system tools
environment variables
dependencies

Without them, the application breaks.

Containerization solves this problem by packaging the application together with everything it needs to run.

Think of a container as:

a fully equipped house for your application.

Inside the container, the app already has:

its dependencies
configurations
runtime environment
required tools

So whether the container runs on:

your laptop
a cloud server
a teammate’s machine

…the application still behaves the same way.

The Mental Model

Containerization gives your application its own portable environment with everything it needs to survive and run consistently.

3. Docker Basics

Key Components

Image - A blueprint/template
Container - A running instance of that image
Dockerfile - Instructions to build the image

Let’s Make It Real

Here’s the smallest possible Docker setup for a Python app.

app.py

print("Hello from Docker!")

Dockerfile

FROM python:3.10-slim

WORKDIR /app
COPY app.py .

CMD ["python", "app.py"]

Build and Run

docker build -t my-python-app .
docker run my-python-app

Notice what we didn’t do:

Install Python manually
Manage versions
Configure anything

The environment is fully defined in the Dockerfile.

4. Why Docker is Useful in Data Engineering

In real-world data systems, you work with tools like:

Apache Airflow
Spark / PySpark
PostgreSQL or another data warehouse
Reporting tools or dashboards

Each of these has:

Different dependencies
Different configurations
Different runtime requirements
Different ports
Different environment variables

Without Docker, they often conflict.

For example:

Airflow may require specific Python packages
PySpark may need Java and Spark installed
PostgreSQL may need database credentials and storage
Dashboard tools may need access to the processed data

With Docker:

each tool runs in its own isolated environment — no conflicts, no surprises.

This is especially useful in batch data pipelines because the entire workflow can be reproduced across different machines and environments.

5. Docker Compose — Managing Multiple Containers

Real systems are never just one container.

A Dockerized data engineering pipeline may include:

An Airflow webserver
An Airflow scheduler
A PostgreSQL database
A Spark / PySpark processing service
Shared folders for DAGs, logs, scripts, and data

Running each service manually quickly becomes painful.

Docker vs Docker Compose

Docker - runs one container
Docker Compose - runs an entire system made up of multiple containers

The Key Insight

Without Docker Compose:

Multiple terminals
Manual startup order
Constant configuration issues
Harder networking between services

With Docker Compose:

one command starts everything.

Example: Multi-Service Setup

A simplified Docker Compose setup for a batch pipeline may include Airflow and PostgreSQL.

docker-compose.yml

services:
  airflow-webserver:
    image: apache/airflow:3.2.1
    container_name: airflow_webserver
    command: airflow webserver
    ports:
      - "8080:8080"
    environment:
      AIRFLOW__CORE__EXECUTOR: LocalExecutor
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
    volumes:
      - ./dags:/opt/airflow/dags
      - ./logs:/opt/airflow/logs
      - ./jobs:/opt/airflow/jobs
    depends_on:
      - postgres

  airflow-scheduler:
    image: apache/airflow:3.2.1
    container_name: airflow_scheduler
    command: airflow scheduler
    environment:
      AIRFLOW__CORE__EXECUTOR: LocalExecutor
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
    volumes:
      - ./dags:/opt/airflow/dags
      - ./logs:/opt/airflow/logs
      - ./jobs:/opt/airflow/jobs
    depends_on:
      - postgres

  postgres:
    image: postgres:16
    container_name: postgres_db
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    ports:
      - "5433:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:

8. Common Mistakes

Using localhost inside containers

This breaks almost everyone at first.

Inside a container:

localhost refers to the container itself, not your machine.

Forgetting environment variables

Missing configs often cause silent failures.

Not persisting data

Containers are temporary. Without volumes, your data disappears.

  volumes:
    - postgres_data:/var/lib/postgresql/data

Rebuilding unnecessarily

Poor Dockerfile structure can slow builds significantly.

9. Best Practices

Use lightweight images

  FROM python:3.10-slim

Add a .dockerignore

  node_modules
  .git
  .env

Avoid latest in production

Use fixed versions to keep builds predictable.

Separate dev and production setups

They have different requirements.

Use Docker Compose for local development

It helps simulate real systems easily.

Use clear service names

Examples:

kafka
postgres
airflow

This simplifies networking and debugging.

10. Conclusion

Containerization changes how you think about environments.

Docker packages your application into a portable unit.
Docker Compose runs entire systems with one command.
Your pipelines become reproducible and consistent.

The real shift is this:

You stop debugging environments — and start defining them as code.

And once you reach that point:

You’re no longer just writing code — you’re building systems.

Data Warehouses, Data Marts, Data Lakes, and Lakehouses - Explained Like You’re Building Them in Real Life

Anthony Gicheru — Sun, 03 May 2026 08:10:27 +0000

If you’ve been around data engineering long enough, you’ve probably heard these terms thrown around in meetings:

“Just dump it in the data lake”
“We’ll expose it through the warehouse”
“That goes into the mart”
“We’re moving to a lakehouse architecture”

And honestly… it can sound like four different ways of saying the same thing.

They’re not.

Each one solves a slightly different problem in the data ecosystem. Once you understand the “why” behind each, the architecture suddenly feels a lot less like buzzwords and more like a clean system design.

Let’s break it down in a practical, engineer-first way.

1. The Big Picture (Why all these systems exist)

In most companies, data doesn’t come from one place — it flows in from everywhere:

User clicks from web/mobile apps
Payments and transactions
Logs from servers
Third-party APIs (Stripe, Shopify, etc.)
IoT or streaming data (Kafka, sensors, etc.)

Now here’s the problem:

Raw data is messy. Business users don’t want messy.

So we build systems that progressively refine data from:

Raw → Clean → Structured → Business-ready

That’s where these four concepts come in:

Data Lake → store everything raw
Data Warehouse → structured analytics-ready data
Data Mart → department-specific slices of warehouse data
Lakehouse → hybrid of lake + warehouse

2. Data Lake — “Store everything first, figure it out later”

A data lake is basically a massive storage system where you dump raw data in its original format.

Think of it like:

A giant warehouse where you throw every box in as-is, without opening it.
Or even better: a farm storage system where everything is stored right after harvest, unprocessed and mixed together.

Characteristics:

Stores structured, semi-structured, and unstructured data
Cheap storage (usually object storage like S3)
Schema is applied when reading, not writing (schema-on-read)

Example tools:

Amazon S3
Azure Data Lake Storage
Google Cloud Storage

Example:

You might store:

/events/clicks/2026/05/01.json
/logs/api/2026/05/01.log
/payments/stripe/2026/05/01.parquet

No transformations. No enforcement. Just storage.

Here’s the catch:

If you’re not careful, a data lake becomes a data swamp — lots of data, zero usability.

3. Data Warehouse — “Clean, structured, and business-ready”

A data warehouse is where data goes after it has been cleaned, transformed, and modeled for analytics.

Think of it like:

A well-organized supermarket where everything is cleaned, packaged, labeled, and placed on the right shelves.
You don’t pick raw potatoes from the soil — you get them washed, sorted, and priced.

Characteristics:

Structured data only
Schema-on-write (you define structure before loading)
Optimized for analytics queries (OLAP systems)
Highly curated and trustworthy

Example tools:

Amazon Redshift
Snowflake
Google BigQuery

Typical workflow:

Extract data from sources
Transform (clean, join, aggregate)
Load into warehouse tables

Example SQL model:

CREATE TABLE sales_fact (
    order_id INT,
    customer_id INT,
    product_id INT,
    amount DECIMAL(10,2),
    order_date DATE
);

Now business analysts can run queries like:

SELECT product_id, SUM(amount)
FROM sales_fact
GROUP BY product_id;

4. Data Marts — “Department-specific mini warehouses”

A data mart is a subset of a data warehouse focused on a specific business domain.

Think of it like:

A grocery store or specialty shop — like a bakery, butcher, or vegetable shop.
It doesn’t sell everything. It only sells what its customers actually need.

Characteristics:

Smaller scope than a warehouse
Built for a specific team (finance, marketing, sales)
Faster queries for targeted use cases

Example:

A marketing data mart might include:

Campaign performance
Customer acquisition metrics
Ad spend data

Example structure:

CREATE TABLE marketing_campaign_performance AS
SELECT
    campaign_id,
    SUM(clicks) AS total_clicks,
    SUM(impressions) AS total_impressions
FROM ad_events
GROUP BY campaign_id;

Why it exists:

Instead of everyone querying a massive warehouse, teams get pre-optimized datasets.

5. Data Lakehouse — “Best of both worlds”

Now this is where things get interesting.

A lakehouse combines:

The flexibility of a data lake
The structure and performance of a data warehouse

Think of it like:

A modern retail system where the warehouse and supermarket are combined into one smart facility.
Raw goods arrive, but they are immediately tracked, organized, and made queryable without losing flexibility.

Characteristics:

Uses low-cost storage (like a lake)
Adds structure, ACID transactions, and governance
Supports both analytics and ML workloads

Example tools:

Apache Spark + Delta Lake
Apache Iceberg
Apache Hudi

Why it matters:

In traditional setups:

Data lakes = flexible but messy
Warehouses = clean but expensive and rigid

Lakehouses try to remove that tradeoff.

6. How They Work Together (Real Architecture Flow)

A modern data pipeline often looks like this:

[ Data Sources ]
      ↓
   DATA LAKE (raw storage)
      ↓
ETL / ELT pipelines (Airflow, Spark)
      ↓
DATA WAREHOUSE (modeled data)
      ↓
DATA MARTS (team-specific views)
      ↓
Dashboards / BI tools

Or in a lakehouse setup:

[ Data Sources ]
      ↓
DATA LAKEHOUSE (single system)
      ↓
BI + ML + Analytics directly

7. Practical Example (Mini Pipeline)

Let’s say we’re processing e-commerce data.

Step 1: Raw data in S3 (Data Lake)

{
  "order_id": 101,
  "user_id": 55,
  "amount": 250,
  "timestamp": "2026-05-01T10:00:00Z"
}

Step 2: Spark transformation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("etl").getOrCreate()

df = spark.read.json("s3://datalake/raw/orders/")

clean_df = df.dropna() \
             .withColumnRenamed("amount", "order_amount")

clean_df.write.mode("overwrite").parquet("s3://warehouse/sales_fact/")

Step 3: Load into warehouse (Redshift example)

COPY sales_fact
FROM 's3://warehouse/sales_fact/'
IAM_ROLE 'arn:aws:iam::123456:role/RedshiftRole'
FORMAT AS PARQUET;

Step 4: Create a data mart

CREATE TABLE sales_summary AS
SELECT
    DATE(order_date) AS date,
    SUM(order_amount) AS revenue
FROM sales_fact
GROUP BY DATE(order_date);

8. Common Pitfalls (Where most teams mess up)

1. Turning the data lake into a swamp

Dumping everything without metadata or structure leads to chaos.

2. Over-modeling too early

Trying to build perfect schemas upfront slows everything down.

3. Duplicating logic across marts

You end up with inconsistent metrics like “Revenue_v1”, “Revenue_final”, “Revenue_real_final”.

4. No governance layer

Without access control and cataloging, nobody trusts the data.

9. Best Practices

1. Use layered architecture

Raw (lake)
Cleaned (staging)
Modeled (warehouse)
Aggregated (marts)

2. Standardize transformations

Use tools like:

dbt
Apache Airflow
Spark jobs with clear ownership

3. Define a single source of truth

One metric definition per business KPI. No duplicates.

4. Treat data like software

Version it, test it, document it.

5. Monitor everything

Pipeline failures
Data freshness
Schema changes

10. Conclusion — The mental model that matters

If you strip away the jargon, it’s really simple:

Data Lake → store everything
Data Warehouse → clean and organize it
Data Mart → tailor it for teams
Lakehouse → unify storage and analytics

The real skill in data engineering isn’t memorizing definitions.

It’s knowing:

When to keep data raw, when to structure it, and when to specialize it.

Once that clicks, designing data systems becomes a lot more intuitive — and honestly, more fun to build.

Refactoring Airflow Pipelines: From PythonOperator to TaskFlow

Anthony Gicheru — Fri, 24 Apr 2026 10:52:39 +0000

Actually Embracing TaskFlow After a Year of Doing It the “Old Way”

1. Introduction: This Isn’t New… But It Feels New

If you’ve been using Airflow for a while-like I have-you probably didn’t start with the TaskFlow API.

You likely started with the classic Airflow 2.x style:

PythonOperator
**kwargs
ti.xcom_push() and ti.xcom_pull()
Explicit task chaining with >>

I spent over a year building pipelines this way. And to be clear-it works. It’s stable, production-ready, and widely used.

But here’s the interesting part:

The TaskFlow API has existed since Airflow 2.0. I just didn’t fully adopt it.

Honestly, I ignored TaskFlow for a long time because I thought it was just ‘syntactic sugar’. That’s more common than people admit.

Most production systems and tutorials still rely on operators, so you naturally stay in that pattern. It’s only later-when readability and maintainability start to matter-that TaskFlow becomes interesting.

And once it clicks, it changes how you think about Airflow.

2. Core Concepts: Same Engine, Different Experience

TaskFlow doesn’t replace Airflow concepts-it abstracts them.

You still work with:

Tasks
DAGs
Scheduling
XComs

The difference is how you express them.

Traditional Approach

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def extract(**kwargs):
    data = [1, 2, 3]
    kwargs['ti'].xcom_push(key='data', value=data)

def transform(**kwargs):
    ti = kwargs['ti']
    data = ti.xcom_pull(task_ids='extract', key='data')
    return [x * 2 for x in data]

with DAG('traditional_dag', start_date=datetime(2023, 1, 1)) as dag:

    t1 = PythonOperator(task_id='extract', python_callable=extract)
    t2 = PythonOperator(task_id='transform', python_callable=transform)

    t1 >> t2

This works, but it introduces a lot of orchestration boilerplate into your business logic.

TaskFlow Approach

from airflow.sdk import dag, task
from datetime import datetime

@dag(start_date=datetime(2023, 1, 1), schedule='@daily', catchup=False)
def taskflow_dag():

    @task
    def extract():
        return [1, 2, 3]

    @task
    def transform(data):
        return [x * 2 for x in data]

    transform(extract())

dag = taskflow_dag()

This feels simpler because it is.

TaskFlow removes explicit XCom handling and lets function returns define data flow.

3. The Real Shift: From Wiring Tasks to Modeling Data Flow

With the traditional approach, your mental model looks like this:

Task A - XCom - Task B - XCom - Task C

With TaskFlow, it becomes:

data = extract()
result = transform(data)

Same execution engine. Different abstraction.

The shift is from task orchestration to data flow composition.

4. XComs: Manual vs Automatic

Manual XComs

def extract(**kwargs):
    kwargs['ti'].xcom_push(key='data', value=[1, 2, 3])

def transform(**kwargs):
    ti = kwargs['ti']
    data = ti.xcom_pull(task_ids='extract', key='data')

You manage everything explicitly.

TaskFlow XComs

@task
def extract():
    return [1, 2, 3]

@task
def transform(data):
    return [x * 2 for x in data]

Airflow handles:

serialization
storage
retrieval

You focus on logic.

When You Still Need Control

TaskFlow still allows explicit control when needed:

from airflow.models.xcom_arg import XComArg

@task
def extract():
    return {"numbers": [1, 2, 3]}

@task
def transform(data):
    return [x * 2 for x in data["numbers"]]

transform(XComArg(extract()))

5. Real-World Example: Gas Prices ETL Refactor

I didn’t build two versions of this pipeline at once.

I originally built it using the traditional Airflow 2.x approach and later refactored it using TaskFlow.

That’s when the difference became clear.

Pipeline Overview

API - Extract gas prices - Transform - Store in PostgreSQL

GitHub Reference

Full project: Github Link to the project

It includes both the original DAG and the TaskFlow refactor.

Traditional Version

def fetch_gas_prices(**kwargs):
    kwargs['ti'].xcom_push(key='raw_gas_data', value=decoded_data)

def transform_gas_prices(**kwargs):
    raw_data = kwargs['ti'].xcom_pull(
        task_ids='fetch_gas_prices',
        key='raw_gas_data'
    )

This approach tightly couples logic with Airflow internals.

Data must be serialized manually:

json_data = df.to_json(orient='records')

TaskFlow Version

@task
def fetch_gas_prices():
    return decoded_data

@task
def transform_gas_prices(raw_data: str):
    return df.to_json(orient='records')

And the pipeline becomes:

raw = fetch_gas_prices()
cleaned = transform_gas_prices(raw)
store_gas_prices(cleaned)

This reads like standard Python.

What Changed

The logic stayed the same. The structure changed completely.

Instead of manually managing XComs, data flows naturally between functions.

Before vs After

Aspect	Traditional	TaskFlow
Task definition	PythonOperator	@task
Data passing	Manual XCom	Automatic
Readability	Medium	High
Boilerplate	High	Low
Mental model	Wiring tasks	Data flow

6. Lessons From the Refactor

1. TaskFlow doesn’t remove XComs

It only hides them.

You still need to respect serialization limits:

return big_dataframe  # still not ideal

2. Passing data is easier-but not always better

TaskFlow makes it easy to pass data between tasks, but large payloads should still live in external storage.

3. Refactoring was mostly structural

Most of the work was:

removing **kwargs
replacing XCom logic with returns
simplifying task boundaries

4. The biggest change is mental

The shift was not technical-it was conceptual.
From:

How do I connect tasks?
to:

How does data flow through this pipeline?

7. Pitfalls to Avoid

Don’t push large objects through XCom
Don’t mix styles without intention
Don’t overuse TaskFlow just because it’s cleaner
Don’t forget serialization still exists

8. Conclusion

TaskFlow isn’t new-but adopting it after using the traditional approach makes its benefits clearer.

It moves you from writing orchestration-heavy DAGs to writing clean Python workflows.

And that shift improves:

readability
maintainability
reasoning about pipelines

Key Takeaways

TaskFlow simplifies DAG structure without changing Airflow’s core engine
XComs still exist but are abstracted
The real improvement is cleaner data flow modeling
Refactoring old DAGs is one of the best ways to understand it

Data Pipelines Explained Simply (and How to Build Them with Python)

Anthony Gicheru — Fri, 17 Apr 2026 07:34:55 +0000

Data pipelines are the backbone of modern data-driven organizations. They automate the movement, transformation, and storage of data - from raw sources to actionable insights.

Python has become the go-to language for building scalable pipelines because of its rich ecosystem, flexibility, and ease of use.

This guide walks through the fundamentals, tools, and best practices for building robust data pipelines using Python.

Understanding Data Pipelines

Imagine you need to supply clean water to a village. The process involves collecting water from different sources (rivers, wells, rain), purifying it, transporting it, and storing it so people can access it whenever they need it.

A data pipeline works in a very similar way.

It automates the journey of raw, unstructured data from multiple sources (like databases, APIs, or IoT devices) and transforms it into clean, usable data stored in a destination (like a data warehouse) for analysis.

Components of a Data Pipeline

Let’s break it down using the same analogy:

1. Collecting Water (Data Ingestion)

Just like gathering water from lakes or wells, a pipeline starts by extracting data from sources such as databases, APIs, spreadsheets, or sensors.

The goal here is simple: get all the data into one system, no matter how scattered it is.

2. Filtering and Purifying (Data Transformation)

Raw water isn’t clean—and neither is raw data.

At this stage, the pipeline:

Removes duplicates
Handles missing values
Standardizes formats
Enriches data

This is where messy data becomes usable.

3. Transporting Through Pipes (Data Movement)

Once cleaned, water flows through pipes. In data pipelines, this represents the movement of data between systems.

This can involve:

ETL processes
Message queues (like Kafka)
Cloud data transfer services

The goal is to move data efficiently without delays or bottlenecks.

4. Storing in Tanks (Data Storage)

Clean water is stored in tanks. Similarly, processed data is stored in:

Data warehouses (like Snowflake)
Data lakes (like AWS S3)
Databases

This is where data becomes ready for use.

5. Accessing on Demand (Data Consumption)

Finally, people use the water.

In the same way, data is consumed through:

Dashboards
APIs
Machine learning models

This is where insights actually happen.

Essential Python Libraries and Tools

Python supports every stage of a pipeline:

Data Ingestion

requests - API calls
pandas - handling CSV/JSON files

Transformation

pandas - cleaning and aggregation
PySpark - large-scale distributed processing

Storage

SQLAlchemy - database interaction
boto3 - AWS S3 integration

Orchestration

Apache Airflow - workflow scheduling and automation
Dagster - modern pipeline orchestration with observability

Best Practices

Error Handling

Implement retries and proper logging to avoid silent failures.

Monitoring

Track pipeline health using tools like Airflow’s UI.

Documentation

Keep clear documentation for:

Code
Dependencies
Workflow logic

Testing

Test each stage of the pipeline using:

Unit tests
Sample datasets

Popular Frameworks for Advanced Use Cases

Apache Airflow - Best for complex workflows with dependencies
Dagster - Strong focus on testing and data asset visibility
Prefect - Simplifies building fault-tolerant pipelines
Luigi - Good for batch processing and dependency management

ETL vs ELT: Which One Should You Use and Why?

Anthony Gicheru — Sun, 12 Apr 2026 21:36:22 +0000

When I first started learning data engineering, ETL and ELT honestly felt like the same thing with just swapped letters. Everyone kept mentioning them like they were obvious concepts, but I had to sit down and really break them apart before it made sense.

If you’re in the same place, don’t worry, you’re not alone.

Let’s make it simple.

First things first: what do ETL and ELT even mean?

Both ETL and ELT are ways of moving and processing data from one place to another.

ETL (Extract, Transform, Load)

Extract data from a source (like an API or database)
Transform it before storing it (cleaning, filtering, joining, etc.)
Load the final cleaned data into a target system (like a data warehouse)

The key idea: you clean the data before storing it.

ELT (Extract, Load, Transform)

Extract data from the source
Load it directly into the storage system first
Transform it inside the database/warehouse later

The key idea: you store raw data first, then clean it inside the system.

So what’s the real difference?

The biggest difference is where the transformation happens.

ETL → Transform happens outside the warehouse
ELT → Transform happens inside the warehouse

That one shift changes a lot more than you’d think.

When ETL makes sense

ETL is usually used when:

You have smaller datasets
You need strict data control before loading
Your system can’t handle heavy processing
Data quality must be enforced early

Think of it like cleaning your room before putting things in storage.

You don’t want messy data entering your system at all.

When ELT makes sense

ELT is more common in modern systems, especially with cloud platforms.

It works well when:

You have large volumes of data
You’re using powerful cloud warehouses (like Snowflake or BigQuery)
You want flexibility in how data is transformed
You want to keep raw data for future use

Think of it like dumping everything into a warehouse first, then organizing it later when needed.

A simple real-world example

Imagine you’re building a dashboard for an e-commerce app.

With ETL:

You:

Pull order data
Clean it (remove duplicates, fix missing values)
Then load it into your database ready for reporting

Everything is neat before it even arrives.

With ELT:

You:

Pull raw order data
Load everything into a data warehouse
Later write SQL transformations to clean and structure it

This gives you more flexibility if business rules change later.

My key takeaway

When I first learned this, I thought ETL was “old” and ELT was “new,” but that’s not really true.

They both still matter.

Here’s a simple way I now remember it:

ETL = Clean first, store later
ELT = Store first, clean later

Common mistakes beginners make

A few things that confused me at the start:

Thinking ELT means “no cleaning” (it still involves transformation!)
Mixing up where SQL transformations happen
Assuming one is always better than the other (it depends on the system)

So… which one should YOU use?

There’s no universal winner.

If you’re working with traditional systems → ETL is common
If you’re in modern cloud data engineering → ELT is more popular

Most real companies actually use a mix of both, depending on the pipeline.

To make this even more practical, here are some common tools used in real ETL and ELT workflows

ETL Tools (Transformation happens before loading)

Apache Airflow – for scheduling and orchestrating ETL workflows
Informatica PowerCenter – widely used in enterprise ETL pipelines
Talend – open-source tool for data integration and transformation
Apache NiFi – good for real-time data flow and routing
SSIS (SQL Server Integration Services) – Microsoft-based ETL tool

These tools usually handle data cleaning and transformation before sending data to a warehouse.

ELT Tools (Transformation happens after loading)

Snowflake – modern cloud data warehouse with strong ELT support
Google BigQuery – popular for serverless ELT workflows
Amazon Redshift – widely used in AWS-based data stacks
dbt (Data Build Tool) – one of the most popular tools for transformations inside the warehouse
Databricks (Apache Spark) – used for large-scale ELT processing

In ELT setups, tools like dbt handle transformation using SQL after data is loaded.

Final thoughts

Once I understood this difference, a lot of other concepts like data pipelines, warehouses, and analytics started to make way more sense.

If you’re learning data engineering right now, don’t rush it. Build a small pipeline, try both approaches, and you’ll see the difference quickly.

That’s where it really clicks.