DEV Community: Anthony Gicheru

Kafka Partitioning Strategies: How to Get It Right Before It Costs You

Anthony Gicheru — Sun, 28 Jun 2026 15:43:49 +0000

Most engineers don't think seriously about Kafka partitioning until something breaks in production. A topic that worked fine at low volume starts falling behind. Events that should be in order aren't. All of it traces back to a partitioning decision that was made quickly and never revisited.

Why Partitioning Actually Matters

Partitions are the unit of parallelism in Kafka. Every consumer in a group is assigned one or more partitions, and it processes those partitions alone. No two consumers in the same group share a partition. That means your partition count sets a hard ceiling on how many consumers can work in parallel: if you have 6 partitions, the 7th consumer in your group sits idle no matter how much load you're under.

Partitioning also controls ordering. Within a single partition, events are strictly ordered. Across partitions, there are no guarantees. So how you distribute events across partitions determines what ordering guarantees your consumers can actually rely on. Get this wrong and you'll spend a long time debugging why events from the same user are being processed out of sequence.

The partition key controls both of these things. It determines which partition an event lands in, and that decision has consequences that are expensive to reverse.

Partitioning Strategies

Partition by Key

This is the most common strategy and the right default when ordering matters. You supply a key when producing an event, Kafka hashes it using the murmur2 algorithm, and takes the modulo against the partition count to decide where it lands.

producer.send('orders', key=b'user_4821', value=event)

Every event with the same key always lands in the same partition. That's what guarantees ordering within a key. All events for user_4821 go to partition 3 (or wherever the hash resolves), and your consumer reads them in the exact sequence they were produced. I default to this for almost everything I build now and only go keyless when I have a specific reason to.

Use key-based partitioning when the sequence of events for a specific entity matters: order lifecycle events, user activity streams, account transactions, IoT device readings. Basically any time you need to reconstruct what happened to a thing, in order.

Where this goes wrong is key distribution. If you use country_code as a key and 80% of your traffic comes from one country, 80% of your events land in the same partition. That partition becomes a bottleneck; the consumer handling it is overwhelmed while others sit mostly idle. This is a hot partition, and it's one of the most common Kafka performance problems I see on teams that are new to it. Keys with high cardinality and relatively even distribution are what you want: user_id, order_id, device_id. The reason those work is simple: they have millions of possible values, so traffic spreads naturally across partitions. Keys like status, region, and event_type tend to cause skew because they only have a handful of possible values, which means a handful of partitions carry almost everything.

No Key, No Order

Skip the key and Kafka spreads events evenly across partitions on its own, no configuration needed. Throughput is good, distribution is even, and you've traded away ordering entirely. That's a fair trade for logs, metrics, anything you're aggregating where sequence doesn't matter. It's a bad trade the moment you need to reconstruct a sequence: an order moving from pending to confirmed to shipped needs those three events read in order, and round-robin gives you no such guarantee.

The Default Kafka Doesn't Advertise

This is what Kafka does by default since version 2.4, and almost nobody knows it until they're confused by something in their metrics. Instead of switching partitions with every record, Kafka batches records to the same partition until the batch fills or the linger.ms timeout hits, then moves to the next one. It's more efficient than scattering tiny batches across brokers, and it costs you nothing to configure.

The only reason to care: if you're watching a keyless topic and partition distribution looks lumpy in the short term, that's sticky partitioning, not a sign of skew. Give it time and it evens out.

Rolling Your Own

When the built-in strategies don't fit, you write your own. A custom partitioner takes the key, value, and topic, and returns a partition number directly.

from kafka import KafkaProducer

def custom_partitioner(key, all_partitions, available_partitions):
    region = key.decode().split(':')[0]  # key format: "region:entity_id"
    if region == 'EU':
        return all_partitions[0]  # partitions 0-2 for EU
    elif region == 'US':
        return all_partitions[3]  # partitions 3-5 for US
    return all_partitions[0]

producer = KafkaProducer(
    bootstrap_servers='localhost:9092',
    partitioner=custom_partitioner
)

The use cases are narrow: geographic routing so region-specific consumers don't have to filter, customer tiering to give enterprise accounts dedicated partitions, compliance rules that pin certain event types to specific partitions for audit purposes. I've built exactly one of these in production, for a compliance requirement nobody was willing to negotiate on, and I wouldn't reach for it again unless forced to. Your routing logic now lives in producer code, every change means a redeploy, and the next engineer debugging a partition imbalance has to learn your custom logic before they can learn anything else. A well-chosen key solves almost everything else.

Hot Partitions: What Key Skew Actually Looks Like

A hot partition is one partition taking significantly more traffic than the rest. The consumer assigned to it is CPU-bound and falling behind; everyone else is comfortable. Latency spikes for whatever's stuck in that partition while the rest of the pipeline looks completely healthy on every dashboard that matters.

I watched a team chase a phantom performance bug for two days because of exactly this. Their aggregate consumer lag sat at a few hundred messages, well within their alert threshold. CPU across the consumer group looked fine. On paper, everything was healthy. But one partition out of twelve was carrying sixty percent of the volume, and the consumer assigned to it was quietly drowning. The key was region_code, and most of their customer base happened to be in one country. Nobody noticed because nobody was looking at lag per partition, only the average. Once we split it out, the problem was obvious in about thirty seconds.

The root cause is nearly always the same: a key whose values don't spread evenly across your traffic. Geography does this constantly. So do status fields, where most events sit in one or two states, and timestamps, which cluster during business hours and go quiet overnight.

The fix depends on how bad the skew is. If it's moderate, adding a random suffix to the key helps: user_id becomes user_id_0, user_id_1, or user_id_2 assigned randomly at write time. This redistributes load but breaks per-key ordering, so your consumer logic needs to handle events from the same user arriving out of sequence. If skew is severe and you need ordering, the options get harder: a custom partitioner that maps high-volume keys to multiple partitions, reordering logic in your consumer that buffers and sequences events before processing them, or rethinking the data model so the skewed key isn't the partition key at all. None of those are fun to retrofit. That's why catching it in pre-production matters.

The best place to catch this is before you go to production. Sample your actual key distribution and simulate what percentage of traffic would land in each partition. If any partition would receive more than 3-4x the average, change the key before you deploy.

Changing Partition Count After the Fact

You can increase the number of partitions on an existing topic. You can't decrease it. And increasing it is less safe than it sounds the moment you're using key-based partitioning.

The partition a key lands in is determined by hash(key) % partition_count. Change the partition count and the mapping changes. A user_id that was routing to partition 3 with 6 partitions now routes to partition 9 with 12 partitions. Any stateful consumer that was building per-user state by reading partition 3 now has an incomplete picture.

I've watched a team burn the better part of a week on exactly this. They went from 6 partitions to 12 on a Friday afternoon to handle a traffic spike, called it a routine scaling change, and went home. By Monday, a stateful aggregation job that built per-user session windows was producing numbers that didn't reconcile with anything, and nobody connected it to the partition change for three more days, because on paper, adding partitions is supposed to be safe. It wasn't, because the moment the partition count changed, every key's hash landed somewhere new, and the consumer's in-memory state for partition 3 was now watching a completely different slice of users.

The safer path when you genuinely need more partitions: create a new topic with the target count, run dual-write from your producer during a migration window, let consumers drain the old topic, cut over, then retire the old topic completely. That last step gets skipped more often than it should. A zombie topic with stale data sitting around is its own category of future confusion. More work upfront, but silent data integrity problems are worse than loud failures, and this approach avoids both.

If your topic is stateless and consumers don't maintain any per-partition state, increasing partition count is lower risk. But write that assumption down somewhere. It's easy to forget, and the next person to touch the pipeline may not know it matters.

Monitoring Partition Health

Aggregate consumer lag hides imbalances by design. It's an average, and averages smooth out exactly the spikes you need to see. If one partition out of six is 200,000 events behind, the aggregate number can still look fine while that partition is functionally broken. Watch lag per partition, not the total, and watch partition size on disk too. Uneven sizes across partitions of the same topic is key skew showing up in a different form, often before lag becomes visible at all.

Rebalance frequency is worth tracking too. A consumer stuck on a hot partition will sometimes start timing out, which triggers rebalances across the whole group. If you're seeing frequent rebalances and can't explain why, check whether one partition is running a consumer ragged. Both lag and size metrics live in Kafka's JMX interface and surface cleanly in Prometheus with the Kafka exporter. Confluent's built-in monitoring shows per-partition lag directly, no extra setup required.

Before You Create the Topic

Get this right before the topic exists, not after. Does ordering matter for this data? If yes, you need a key. What does your actual key distribution look like in production traffic, not the sample data on your laptop? And what's the largest number of consumers you'll realistically want running in parallel against this topic? That last answer is your partition count.

Set it to 2-3x your initial consumer count with room for traffic growth, not just where you are today but where you expect to be in a year. Write down why you chose the number. Treat the decision the way you'd treat a schema change. You can walk it back later if you really have to. You don't want to.

The reason this section exists at the end of an article full of incidents is that every single one of those incidents had a cleaner solution available before the topic was created than after. The region_code hot partition was fixable with a better key upfront. The Friday partition count disaster was avoidable with a higher initial count or a documented assumption about statefulness. The problems aren't hard to prevent. They're just easy to deprioritize when you're moving fast and the topic is one line of setup in a larger ticket. It takes about ten minutes to think through these questions properly. The incidents in this article took days.

Conclusion

I've never seen a team get burned by picking the wrong partitioning strategy. I've seen plenty get burned by picking a key without checking how it actually distributes, or bumping a partition count without asking what depends on the old mapping. Ask the key distribution question before you write the first producer, and you'll skip most of the incidents in this article entirely.

Apache Kafka Explained: A Practical Beginner Guide for Data Engineers

Anthony Gicheru — Tue, 09 Jun 2026 05:38:19 +0000

If you're learning data engineering, you'll probably meet Apache Kafka very early. You'll see it in job descriptions, system design diagrams, real-time analytics projects, fraud detection systems, and streaming pipelines. But Kafka can feel confusing at first because most explanations jump straight into brokers, partitions, and consumer groups before explaining the actual problem Kafka solves.

So let's start there.

Why Kafka Exists

Traditional batch pipelines are built to answer one question: What does the data look like right now? Pull from the source, transform it, load it somewhere. Run it every hour, or every night. That works fine until you need to react to something the moment it happens.

A payment system that catches fraud after the transaction clears isn't catching fraud. A recommendation engine that updates overnight isn't personalizing anything in real time. A location tracking system that refreshes every hour is useless for logistics. These systems don't just need data. They need to know what's happening now, as events occur.

That's the gap Kafka fills. It's a system built to capture events and move them to wherever they're needed, reliably and at scale.

The Core Concepts

Events

An event is a record of something that happened. Not a current state, not an aggregation. Just a fact, timestamped and immutable. A user placed an order. A payment failed. A package changed location. A temperature sensor crossed a threshold.

In Kafka, events are stored as key-value records, typically in JSON or Avro:

{
  "user_id": 4821,
  "action": "purchase",
  "item_id": "SKU-009",
  "timestamp": "2026-05-05T14:32:00Z"
}

Once an event is written to Kafka, it doesn't change. It becomes part of a permanent, ordered history.

Topics

A topic is a named stream of related events, basically a category. You might have an orders topic that stores every order placed on your platform, a payments topic for every transaction, and a user-activity topic for clicks, logins, and page views. Producers write to topics; consumers read from them. That's the core of how Kafka organizes data.

If you're coming from a relational database background, the closest mental model is a table. An orders topic is roughly analogous to an orders table: it holds a collection of related records, each with the same general shape. But there are two important differences. First, you can only append to a Kafka topic. No UPDATE, no DELETE. Second, a table stores current state (what the order looks like right now), while a topic stores history (every event that ever happened to every order, in the order it happened). A more accurate comparison is an append-only audit log table, not a regular transactional one. If you've ever used event sourcing patterns or tracked changes in a created_at/updated_at log table, that's the right mental model.

Producers and Consumers

A producer is any application that sends events into Kafka. Your e-commerce backend produces an order event every time a customer checks out. It doesn't need to know who's going to use that event. It just writes it to the orders topic and moves on.

A consumer is any application that reads events from Kafka. Your fraud detection service, your analytics pipeline, and your notification system are all consumers. Here's the part that surprises most people coming from traditional message queues: Kafka doesn't delete an event after a consumer reads it. Each consumer tracks its own position in the log independently. One topic can serve ten different consumer systems simultaneously, and none of them interfere with each other.

Partitions

Topics are split into partitions. Each partition is an ordered, append-only log stored on disk on one of Kafka's servers. Splitting a topic across multiple partitions is what gives Kafka its scalability: different consumers can read different partitions in parallel, and partitions can be spread across multiple servers for throughput and fault tolerance.

Topic: orders
  ├── Partition 0  →  [event][event][event]...
  ├── Partition 1  →  [event][event][event]...
  └── Partition 2  →  [event][event][event]...

When a producer sends an event, Kafka uses a partition key to decide which partition it lands in. If you use user_id as the key, every event from the same user always goes to the same partition, which guarantees they'll be read in order. That's key-based partitioning, and it's what you'll use most of the time when the sequence of events for a specific entity matters.

If you don't supply a key, Kafka distributes records across partitions evenly. That's fine when ordering doesn't matter, like application logs or metrics — but you lose the ordering guarantee entirely.

Choosing the right partitioning strategy goes deeper than just picking a key, and the consequences of getting it wrong aren't always obvious until you're in production. That's worth a dedicated article on its own.

Offsets

Every event inside a partition is assigned a sequential integer called an offset, starting at zero and incrementing forever.

Partition 0:  [0][1][2][3][4][5]...

Consumers use offsets to track their progress. When a consumer crashes and restarts, it picks up from the last offset it committed. This is also what makes Kafka replayable: you can reset a consumer's offset back to zero and reprocess every event from the beginning. That's not something you get with traditional queues where messages disappear on consumption. It's one of Kafka's most useful features in practice. You can backfill a new table, reprocess events after fixing a bug, or onboard a new downstream system against months of historical data.

Brokers, Clusters, and the Metadata Layer

Brokers and Clusters

A broker is a single Kafka server: one machine running the Kafka process, storing partitions on disk, and handling reads and writes. When your producer sends an event, it's connecting to a broker. When your consumer reads, it's reading from a broker.

In production, you don't run one broker. You run a cluster, which is just a group of brokers working together. More brokers means more storage, more throughput, and fault tolerance. When a topic has a replication factor of 3, Kafka stores each partition on 3 different brokers. One of those copies is the leader, the broker that handles all reads and writes for that partition. The other two are followers that stay in sync. If the leader broker goes down, Kafka promotes a follower automatically. Producers and consumers reconnect to the new leader, and the cluster keeps running. Done correctly, your consumers don't even notice.

Replication factor isn't just about data safety, it's about availability. A replication factor of 1 means any single broker failure takes that partition offline completely.

ZooKeeper and KRaft: The Part Everyone Finds Confusing

If you've read any Kafka tutorials from the last few years, you've probably seen Apache ZooKeeper mentioned alongside Kafka. ZooKeeper's job was to manage cluster metadata: tracking which brokers were alive, which broker was the leader for each partition, and coordinating elections when a broker went down.

It worked, but it was operationally awkward. Running Kafka in production meant also running, monitoring, and scaling a completely separate ZooKeeper cluster. Two systems to deploy, two systems to debug, two failure surfaces to worry about. If ZooKeeper had a problem, Kafka had a problem.

Kafka 3.x introduced KRaft (Kafka Raft) to replace it. Instead of depending on an external system, a subset of brokers act as controllers and manage cluster metadata internally using a built-in consensus protocol. One system, not two.

The practical takeaway: if you're setting up Kafka today, use KRaft mode. It's the default. If you're following an older tutorial and there's a step to configure ZooKeeper separately, that tutorial is out of date. Kafka 4.0 dropped ZooKeeper support entirely, so this isn't a future transition you need to plan for. It's already done. The reason it still comes up is that a lot of teams are running older Kafka versions in production and haven't migrated yet, so you'll still see ZooKeeper in the wild. Now you know what it is and why it's being replaced.

How Kafka Fits Into a Real Pipeline

In production, Kafka typically sits between your source systems and everything downstream:

[App / Services]
      │
      ▼
[Kafka: orders topic]
      │
      ├──▶ [Fraud Detection Service]  ──▶  [PostgreSQL: flagged orders]
      ├──▶ [Spark Streaming]  ──▶  [Snowflake]  ──▶  Analytics Dashboard
      └──▶ [Notification Service]  ──▶  [Email / SMS]

Your application writes the order event once. Every downstream system consumes it independently, at its own pace, with its own offset. Adding a fifth consumer (like a new inventory service) means creating a new consumer group, not touching the producer or any existing consumers.

A Practical Example

You're building the data infrastructure for an e-commerce platform. Every user action generates an event. Your fraud detection service needs to score transactions in under a second. Your analytics pipeline needs to update a Snowflake dashboard every few minutes. Your notification service sends confirmation emails.

Here's a minimal Python producer writing order events to Kafka:

from kafka import KafkaProducer
import json

producer = KafkaProducer(
    bootstrap_servers='localhost:9092',
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)

event = {"user_id": 4821, "action": "purchase", "item_id": "SKU-009"}
producer.send('orders', key=b'user_4821', value=event)
producer.flush()

And a consumer for the fraud detection service:

from kafka import KafkaConsumer
import json

consumer = KafkaConsumer(
    'orders',
    bootstrap_servers='localhost:9092',
    group_id='fraud-detection-group',
    value_deserializer=lambda m: json.loads(m.decode('utf-8'))
)

for message in consumer:
    score_transaction(message.value)

If the fraud detection service goes down for 20 minutes, it resumes from its last committed offset when it restarts. The analytics pipeline and notification service never knew there was a problem. They have their own offsets, their own pace, their own position in the log.

Common Mistakes (And How to Avoid Them)

Treating Kafka Like a Queue

This is the most common mental model mistake. In a traditional queue, a message is consumed and gone. Kafka doesn't work that way. Events stay in a partition for your configured retention period (often 7 days), regardless of how many consumers have read them. That means your consumers need to be idempotent: processing the same event twice shouldn't produce incorrect results. The practical fix is simple: use a unique event ID or order ID when writing to your final database, so duplicate processing is harmless.

Picking Partition Count Arbitrarily

Most tutorials say "set it to 3" and don't explain why. Here's what they don't tell you: you can increase partition count later, but you can't decrease it. And increasing it after you're already using partition keys breaks your key-to-partition mapping. An event from user_id: 4821 that used to land in partition 3 might now land in partition 7. Ordering guarantees break, and stateful consumers get incomplete data.

Think about partition count before you create the topic. A reasonable starting point is 2–3× your initial consumer count, with room to grow. Write down why you chose it. Treat it like a schema decision; once it's set, it's very hard to undo.

Ignoring Consumer Lag

Consumer lag is the gap between the latest offset in a partition and the offset your consumer has actually processed. A small, stable lag is normal. A lag that keeps growing is a silent production incident.

If your fraud detection service is 200,000 events behind, it isn't doing real-time fraud detection. It's doing very delayed batch processing with extra steps. Set up monitoring and alerts on consumer lag before you go to production, not after something breaks.

Auto-Committing Offsets for Critical Pipelines

Auto-commit sounds convenient. It commits your offset on a timer, which means it can commit before your consumer has actually finished processing an event. If your service crashes in that window, the event is silently skipped. For payments, orders, or anything where losing an event matters, use manual offset commits and only commit after processing is confirmed complete.

Best Practices Worth Knowing

Use meaningful topic names. A topic called events is going to cause you problems in six months. orders, payments, user-activity. Specific names, clear ownership.

Choose partition keys deliberately. If ordering matters by user, use user_id. If it matters by account, use account_id. Don't use random or null keys unless you genuinely don't care about ordering.

Set replication factor to 3 in production. A replication factor of 1 means a single broker failure takes your data with it. In production, 3 is the baseline, not optional.

Use schemas for serious pipelines. JSON is easy to start with and easy to break. If a producer team renames a field, every downstream consumer breaks silently. Avro with a Schema Registry enforces a contract between producers and consumers and lets schemas evolve safely.

Don't use Kafka when you don't need it. If your pipeline runs once a day and has one consumer, Airflow and a batch job are simpler and easier to maintain. Kafka earns its operational overhead when you need real-time throughput, multiple consumers, or replayability. Reach for it when those requirements are real, not just anticipated.

Conclusion

The mental shift Kafka requires isn't about learning a new API. It's about designing systems around events rather than state. Capture what happened once, and let every system that cares about it consume that record independently, at its own pace, with the ability to replay.

The best way to make that concrete: spin up Kafka locally with Docker, create a topic with 3 partitions, write the producer and consumer from the example above, and then reset your consumer offset to zero and watch it replay every event from the beginning. That one experiment will do more than any amount of reading.

Slowly Changing Dimensions Explained: How Data Warehouses Keep History Accurate

Anthony Gicheru — Sun, 17 May 2026 07:11:45 +0000

1. Why Slowly Changing Dimensions Matter

In data engineering, not all data changes the same way.

Some data changes constantly, like transactions, clicks, payments, and sensor readings. These are usually facts: events that happen at a specific point in time.

But other data changes slowly.

A customer changes their address.
A product changes category.
An employee moves to a new department.
A supplier changes region.
A user upgrades from a free plan to a premium plan.

These changes do not happen every second, but when they happen, they matter a lot.

Imagine you are building a sales report. A customer originally lived in Nairobi, then moved to Mombasa. If you simply update the customer record, all their historical sales may suddenly appear as if they happened in Mombasa.

That is a problem.

The business may ask:

“How much revenue did we make from Nairobi customers last year?”

But if you overwrote the customer’s location, your report may give the wrong answer.

This is the exact problem Slowly Changing Dimensions solve.

Slowly Changing Dimensions help data teams manage changes in descriptive data over time while keeping analytics accurate.

2. What Is a Slowly Changing Dimension?

A Slowly Changing Dimension, often shortened to SCD, is a technique used in data warehousing to manage changes in dimension tables over time.

A dimension table stores descriptive information.

For example:

customer_id	customer_name	city	customer_type
101	Mary Wanjiku	Nairobi	Regular

This is not a transaction. It describes the customer.

Now imagine Mary moves from Nairobi to Kisumu.

The question becomes:

Should we overwrite Nairobi with Kisumu, or should we keep a history of both?

That decision is what SCD is all about.

This is where Slowly Changing Dimensions become useful.

They give data teams a structured way to decide how changes should be stored.

Sometimes we only care about the latest value.

Sometimes we want to preserve the original value.

Sometimes we need the full history of every meaningful change.

And sometimes we only need a simple previous-and-current comparison.

3. How Slowly Changing Dimensions Work

In a data warehouse, data is usually organized into fact tables and dimension tables.

Fact Tables

Fact tables store business events.

Examples:

Sales
Orders
Payments
Website clicks
Deliveries

A sales fact table might look like this:

sale_id	customer_key	product_key	amount	sale_date
5001	1	20	3000	2025-01-10

Dimension Tables

Dimension tables describe the facts.

A customer dimension might look like this:

customer_key	customer_id	name	city	customer_type
1	101	Mary Wanjiku	Nairobi	Regular

The fact table tells us what happened.

The dimension table tells us who, what, where, or how it happened.

The challenge is that dimension data changes.

When Mary moves from Nairobi to Kisumu, we need to decide how to store that change.

There are different SCD types, but the most commonly used are:

SCD Type 0
SCD Type 1
SCD Type 2
SCD Type 3

Let’s go through them practically.

4. SCD Type 0: Keep the Original Value

SCD Type 0 means a value does not change in the data warehouse.

Once the value is loaded, it stays the same, even if the source system changes later.

In simple terms, Type 0 says:

“Keep the original value as it was first recorded.”

In real data warehouse work, Type 0 appears often for fields that should represent the original state of something. But many teams do not always call it “SCD Type 0” explicitly.

They may simply say:

“This field should never be updated.”

Or:

“Preserve the original value.”

So conceptually, Type 0 is common. The name “Type 0” is just less commonly emphasized.

Good examples of Type 0 fields are:

Original signup date
First purchase date
Original registration country
Original acquisition channel
Original product launch date
Original employee hire date

For example, imagine Mary Wanjiku first registered as a customer while living in Kenya through an Instagram campaign.

customer_id	name	original_signup_date	original_country	acquisition_channel
101	Mary Wanjiku	2025-01-01	Kenya	Instagram Ads

Later, Mary may move cities, upgrade her customer type, or start coming through email campaigns.

But the original acquisition channel should still remain Instagram Ads.

Why?

Because it tells the business how Mary was first acquired.

If we overwrite that value, we lose the ability to answer questions like:

“Which marketing channel originally brought us our best customers?”

That is where Type 0 is useful.

It protects values that describe the original state of a record.

5. SCD Type 1: Overwrite the Old Value

SCD Type 1 is the simplest approach.

When a value changes, you overwrite the old value with the new one.

Before:

customer_id	name	city
101	Mary Wanjiku	Nairobi

After Mary moves:

customer_id	name	city
101	Mary Wanjiku	Kisumu

The old city is gone.

When Type 1 Makes Sense

SCD Type 1 is useful when history does not matter.

For example:

Fixing a spelling mistake
Correcting wrong data
Updating an email address
Updating a phone number
Correcting a product name typo

If the original value was wrong, you usually do not want to preserve it.

Example

Imagine a customer’s name was loaded as:

Mary Wanjikuuu

Then later corrected to:

Mary Wanjiku

You do not need historical tracking for the typo. You just update the record.

That is SCD Type 1.

The Risk

The risk with Type 1 is that it destroys history.

If city changes from Nairobi to Kisumu, all past reports will now treat Mary as a Kisumu customer, even if she lived in Nairobi when the sales happened.

So Type 1 is simple, but dangerous when historical accuracy matters.

6. SCD Type 2: Keep Full History

SCD Type 2 is the most important and most commonly used SCD technique in analytics.

Instead of overwriting the old record, you create a new row when important attributes change.

Before Mary moves:

customer_key	customer_id	name	city	start_date	end_date	is_current
1	101	Mary Wanjiku	Nairobi	2024-01-01	NULL	true

After Mary moves to Kisumu:

customer_key	customer_id	name	city	start_date	end_date	is_current
1	101	Mary Wanjiku	Nairobi	2024-01-01	2025-03-15	false
2	101	Mary Wanjiku	Kisumu	2025-03-15	NULL	true

Notice something important.

The customer_id stays the same because it represents the real-world customer.

But the customer_key changes because each historical version gets its own unique warehouse key.

This is usually called a surrogate key.

Why This Matters

Now, if Mary made a purchase while living in Nairobi, the fact table can point to the Nairobi version of her customer record.

If she made another purchase after moving to Kisumu, that sale can point to the Kisumu version.

This allows historical reports to stay accurate.

Example Fact Table

sale_id	customer_key	amount	sale_date
5001	1	3000	2025-02-10
5002	2	4500	2025-04-20

The first sale belongs to Mary when she was in Nairobi.

The second sale belongs to Mary when she was in Kisumu.

That is the power of SCD Type 2.

7. SCD Type 3: Store Limited History in Columns

SCD Type 3 keeps limited history by adding extra columns.

For example:

customer_id	name	current_city	previous_city
101	Mary Wanjiku	Kisumu	Nairobi

This lets you see the current value and one previous value.

When Type 3 Makes Sense

SCD Type 3 is useful when you only care about a small amount of history.

For example:

Previous region and current region
Previous plan and current plan
Previous department and current department

But it does not scale well if changes happen many times.

What happens if Mary moves from Nairobi to Kisumu, then Nakuru, then Eldoret?

You would need more columns:

previous_city_1
previous_city_2
previous_city_3

That becomes messy quickly.

So Type 3 is useful, but only for very specific cases.

8. Practical Example in a Data Warehousing Project

Let’s say you are building a sales analytics warehouse for an e-commerce company.

You have data coming from:

PostgreSQL for application data
Kafka for order events
Airflow for orchestration
dbt for transformations
Snowflake, BigQuery, Redshift, or PostgreSQL as the warehouse

Your source customer table in PostgreSQL looks like this:

customer_id	name	city	customer_type	acquisition_channel	updated_at
101	Mary Wanjiku	Nairobi	Regular	Instagram Ads	2025-01-01

Later, the same customer changes:

customer_id	name	city	customer_type	acquisition_channel	updated_at
101	Mary Wanjiku	Kisumu	Premium	Instagram Ads	2025-03-15

Now the data team must decide:

Do we overwrite the old record?

Or do we preserve the old version?

And what do we do with the original acquisition channel?

In this example:

acquisition_channel can be treated as Type 0 because it represents how Mary was originally acquired.
city and customer_type can be treated as Type 2 because they affect historical reporting.

For analytics, this is why we often combine different SCD behaviors in the same dimension table. Some fields preserve the original value, while others keep full history.

A Type 2 customer dimension may look like this:

customer_key	customer_id	name	city	customer_type	acquisition_channel	valid_from	valid_to	is_current
1	101	Mary Wanjiku	Nairobi	Regular	Instagram Ads	2025-01-01	2025-03-15	false
2	101	Mary Wanjiku	Kisumu	Premium	Instagram Ads	2025-03-15	NULL	true

Now your reports can answer questions like:

How many sales came from Nairobi customers in February?
How much revenue came from Premium customers after March?
What was the customer type at the time of purchase?
How many customers upgraded from Regular to Premium?

Without SCD Type 2, these questions become difficult or inaccurate.

9. A Simple SCD Type 2 Flow

A typical SCD Type 2 pipeline works like this:

Step 1: Load the Latest Source Data

You extract the latest customer data from the source system.

This could come from PostgreSQL, an API, a CSV file, or CDC events from Kafka.

Step 2: Compare Source Data With Current Dimension Records

You compare the incoming record with the current active record in the warehouse.

For example, compare:

source.city
source.customer_type

against:

dim_customer.city
dim_customer.customer_type

Step 3: Detect Changes

If nothing changed, do nothing.

If important attributes changed, expire the old record.

For example:

customer_key	customer_id	city	valid_to	is_current
1	101	Nairobi	2025-03-15	false

Step 4: Insert a New Current Record

Then insert the new version:

customer_key	customer_id	city	valid_from	valid_to	is_current
2	101	Kisumu	2025-03-15	NULL	true

Step 5: Use the Correct Dimension Version in Fact Tables

When loading fact data, join the fact date to the correct dimension record using the validity period.

For example:

SELECT
    f.order_id,
    d.customer_key,
    f.order_date,
    f.amount
FROM staging_orders f
JOIN dim_customer d
    ON f.customer_id = d.customer_id
   AND f.order_date >= d.valid_from
   AND (
        f.order_date < d.valid_to
        OR d.valid_to IS NULL
   );

This ensures the order connects to the correct version of the customer.

10. Common Mistakes Beginners Make

Mistake 1: Using Type 1 When History Matters

This is probably the most common mistake.

Beginners often overwrite dimension records because it feels simple.

But later, when the business asks historical questions, the warehouse cannot answer correctly.

For example:

“What was revenue by customer region last year?”

If you overwrote all customer regions with the latest value, the report will be wrong.

How to Avoid It

Before choosing Type 1, ask:

“Will the business ever need to know what this value was in the past?”

If yes, consider Type 2.

Mistake 2: Tracking Every Column as Type 2

Not every change deserves a new historical version.

For example, do you really need a new customer dimension row when the phone number changes?

Maybe not.

If you track every small change, your dimension table can grow unnecessarily large and become harder to manage.

How to Avoid It

Classify columns carefully.

For example:

Column	SCD Type
customer_name typo fix	Type 1
city	Type 2
customer_type	Type 2
phone_number	Type 1
email	Type 1 or Type 2 depending on business need

The decision should be based on reporting needs, not just technical preference.

Mistake 3: Not Using a Surrogate Key

A big mistake is using the source system ID as the primary key for the dimension table.

For example, using customer_id as the only key.

That becomes a problem in Type 2 because the same customer can have multiple historical versions.

Better Approach

Use:

customer_id as the business key
customer_key as the warehouse surrogate key

Example:

customer_key	customer_id	city	is_current
1	101	Nairobi	false
2	101	Kisumu	true

The surrogate key uniquely identifies each version.

Mistake 4: Forgetting the Current Flag

Without an is_current column, it becomes harder to query the latest version of each record.

You would have to check for:

valid_to IS NULL

That works, but is_current makes queries easier and clearer.

A good SCD Type 2 table usually has:

valid_from
valid_to
is_current

Some teams also add:

created_at
updated_at
record_hash

Mistake 5: Poor Date Handling

SCD Type 2 depends heavily on dates.

If your valid_from and valid_to values are wrong, your historical joins will be wrong.

Common problems include:

Overlapping date ranges
Gaps between versions
Incorrect timezone handling
Using load date instead of actual business effective date
Not handling late-arriving data

How to Avoid It

Be very intentional about what your dates mean.

For example:

valid_from: when this version became valid
valid_to: when this version stopped being valid
loaded_at: when the data entered the warehouse

These are not always the same thing.

11. Best Practices for SCD

Use SCD Type 2 for Business-Critical History

If a change affects reporting, segmentation, revenue analysis, or compliance, preserve history.

Examples:

Customer region
Customer plan
Product category
Sales territory
Employee department
Account status

These are usually worth tracking with Type 2.

Use Hashing to Detect Changes

Instead of comparing many columns one by one, you can create a hash from the important attributes.

Example:

MD5(CONCAT(city, customer_type, region))

Then compare the source hash with the current dimension hash.

If the hash changes, something important changed.

This makes SCD pipelines easier to maintain, especially when there are many columns.

Keep Your SCD Logic Clear

SCD logic can become confusing quickly.

Use clear column names like:

valid_from
valid_to
is_current

Avoid unclear names like:

start
end
flag

Your future self and your teammates will thank you.

Document Which Columns Are Tracked

Do not leave SCD behavior hidden inside SQL code only.

Document which columns are:

Type 0
Type 1
Type 2
Type 3
Ignored
Used for change detection

This is especially important in team environments.

Avoid Duplicates in Current Records

For a Type 2 dimension, each business key should have only one current record.

This should be true:

One customer_id = one current record

You can test this with SQL:

SELECT
    customer_id,
    COUNT(*) AS current_records
FROM dim_customer
WHERE is_current = true
GROUP BY customer_id
HAVING COUNT(*) > 1;

If this query returns rows, your dimension has a problem.

Test for Overlapping Validity Periods

For each business key, the date ranges should not overlap.

Bad example:

customer_id	city	valid_from	valid_to
101	Nairobi	2025-01-01	2025-04-01
101	Kisumu	2025-03-15	NULL

These overlap between March 15 and April 1.

That means a sale on March 20 could match both records.

That is dangerous.

12. When to Use Slowly Changing Dimensions

Use SCD when dimension data changes and those changes affect analysis.

Good use cases include:

Customer address history
Product category history
Subscription plan changes
Employee department changes
Supplier region changes
Account status changes
Sales territory changes

SCD is especially useful in data warehouses and analytics systems where historical accuracy matters.

For example:

“Show revenue by the customer’s region at the time of purchase.”

That is a classic SCD Type 2 problem.

13. When SCD May Not Be the Best Choice

SCD is powerful, but not every situation needs it.

Do Not Use Type 2 for Every Small Change

If a field changes often and does not matter historically, Type 2 may create unnecessary complexity.

For example:

Last login timestamp
Profile picture URL
Phone number
Minor spelling corrections
Temporary status fields

For these, Type 1 may be enough.

Be Careful With Very High-Volume Changes

If a dimension changes too frequently, Type 2 can grow very fast.

At that point, you may need a different modeling approach, such as:

Event sourcing
Audit tables
Snapshot tables
Data lake versioning
Change Data Capture history

SCD is best for slowly changing descriptive attributes, not every event that happens in the system.

14. SCD in Modern Data Tools

SCD is not limited to traditional warehouses.

You can implement SCD patterns in many modern data stacks.

In dbt

dbt supports snapshots, which are commonly used to implement SCD Type 2.

A dbt snapshot can track when records change and automatically create historical versions.

In Airflow

Airflow can orchestrate SCD pipelines by scheduling extraction, staging, comparison, and dimension loading tasks.

In Spark

Spark is useful when you are handling large-scale dimension updates.

You can compare source and target datasets, detect changes, and write updated records to a lakehouse or warehouse.

In Kafka and CDC

Kafka can stream changes from source systems.

For example, using CDC tools, you can capture customer updates from PostgreSQL and send them into Kafka.

From there, you can process those changes and update your dimension tables.

In Warehouses

SCD can be implemented in:

Snowflake
BigQuery
Redshift
PostgreSQL
Databricks
SQL Server

The concept stays the same, even if the syntax differs.

15. Final Summary

Slowly Changing Dimensions help data engineers manage changes in dimension data over time.

They solve an important problem:

How do we keep analytics accurate when descriptive data changes?

The most common SCD types are:

Type	Meaning	Best For
Type 0	Keep the original value unchanged	Original values that should not change
Type 1	Overwrite old values	Corrections and non-historical changes
Type 2	Keep full history using new rows	Historical reporting
Type 3	Keep limited history in columns	Simple previous/current comparisons

In real data engineering projects, SCD Type 2 is especially important because it allows the warehouse to answer historical questions correctly.

Without SCD, reports can quietly become wrong.

A customer’s current city may overwrite their past city.
A product’s new category may rewrite old sales history.
An employee’s new department may change old performance reports.

That is why SCD matters.

The practical takeaway is simple:

Whenever a dimension value changes, ask whether the business needs to remember the old value.

If the value should never change, Type 0 may be the right choice.

If the old value does not matter, Type 1 may be enough.

If the business needs full history, Type 2 is usually the best choice.

If the business only needs a simple previous-and-current comparison, Type 3 may work.

Why Your Code Breaks in Production (and How Docker Fixes It)

Anthony Gicheru — Tue, 12 May 2026 05:28:52 +0000

1. Why This Matters

You write your code.
You test it locally.
Everything works perfectly.

Then it goes to production… and breaks.

You spend hours debugging, only to realize:
nothing is wrong with your code — the environment is the problem.

In data engineering, this happens all the time:

A Spark job runs locally but fails in production
Airflow works on Ubuntu but breaks on macOS
Kafka pipelines behave differently across environments

At its core, the issue is simple:

Your environment is not consistent.

Containerization solves this by packaging everything your application needs into a single, portable unit that runs the same way anywhere.

2. Core Concept — What is Containerization?

Let’s simplify it with an analogy.

Analogy: A Fully Equipped House

Imagine being placed in an empty field with nothing around you.

No food.
No water.
No electricity.
No shelter.

You might survive for a while, but functioning properly would be difficult.

Now imagine being placed inside a fully equipped house.

Everything you need is already there:

food
water
electricity
furniture
internet
a bed

No matter where that house is moved, you can still live comfortably because your essentials move with you.

Applications work the same way.

An application needs certain things to function:

libraries
runtime versions
system tools
environment variables
dependencies

Without them, the application breaks.

Containerization solves this problem by packaging the application together with everything it needs to run.

Think of a container as:

a fully equipped house for your application.

Inside the container, the app already has:

its dependencies
configurations
runtime environment
required tools

So whether the container runs on:

your laptop
a cloud server
a teammate’s machine

…the application still behaves the same way.

The Mental Model

Containerization gives your application its own portable environment with everything it needs to survive and run consistently.

3. Docker Basics

Key Components

Image - A blueprint/template
Container - A running instance of that image
Dockerfile - Instructions to build the image

Let’s Make It Real

Here’s the smallest possible Docker setup for a Python app.

app.py

print("Hello from Docker!")

Dockerfile

FROM python:3.10-slim

WORKDIR /app
COPY app.py .

CMD ["python", "app.py"]

Build and Run

docker build -t my-python-app .
docker run my-python-app

Notice what we didn’t do:

Install Python manually
Manage versions
Configure anything

The environment is fully defined in the Dockerfile.

4. Why Docker is Useful in Data Engineering

In real-world data systems, you work with tools like:

Apache Airflow
Spark / PySpark
PostgreSQL or another data warehouse
Reporting tools or dashboards

Each of these has:

Different dependencies
Different configurations
Different runtime requirements
Different ports
Different environment variables

Without Docker, they often conflict.

For example:

Airflow may require specific Python packages
PySpark may need Java and Spark installed
PostgreSQL may need database credentials and storage
Dashboard tools may need access to the processed data

With Docker:

each tool runs in its own isolated environment — no conflicts, no surprises.

This is especially useful in batch data pipelines because the entire workflow can be reproduced across different machines and environments.

5. Docker Compose — Managing Multiple Containers

Real systems are never just one container.

A Dockerized data engineering pipeline may include:

An Airflow webserver
An Airflow scheduler
A PostgreSQL database
A Spark / PySpark processing service
Shared folders for DAGs, logs, scripts, and data

Running each service manually quickly becomes painful.

Docker vs Docker Compose

Docker - runs one container
Docker Compose - runs an entire system made up of multiple containers

The Key Insight

Without Docker Compose:

Multiple terminals
Manual startup order
Constant configuration issues
Harder networking between services

With Docker Compose:

one command starts everything.

Example: Multi-Service Setup

A simplified Docker Compose setup for a batch pipeline may include Airflow and PostgreSQL.

docker-compose.yml

services:
  airflow-webserver:
    image: apache/airflow:3.2.1
    container_name: airflow_webserver
    command: airflow webserver
    ports:
      - "8080:8080"
    environment:
      AIRFLOW__CORE__EXECUTOR: LocalExecutor
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
    volumes:
      - ./dags:/opt/airflow/dags
      - ./logs:/opt/airflow/logs
      - ./jobs:/opt/airflow/jobs
    depends_on:
      - postgres

  airflow-scheduler:
    image: apache/airflow:3.2.1
    container_name: airflow_scheduler
    command: airflow scheduler
    environment:
      AIRFLOW__CORE__EXECUTOR: LocalExecutor
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
    volumes:
      - ./dags:/opt/airflow/dags
      - ./logs:/opt/airflow/logs
      - ./jobs:/opt/airflow/jobs
    depends_on:
      - postgres

  postgres:
    image: postgres:16
    container_name: postgres_db
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    ports:
      - "5433:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:

8. Common Mistakes

Using localhost inside containers

This breaks almost everyone at first.

Inside a container:

localhost refers to the container itself, not your machine.

Forgetting environment variables

Missing configs often cause silent failures.

Not persisting data

Containers are temporary. Without volumes, your data disappears.

  volumes:
    - postgres_data:/var/lib/postgresql/data

Rebuilding unnecessarily

Poor Dockerfile structure can slow builds significantly.

9. Best Practices

Use lightweight images

  FROM python:3.10-slim

Add a .dockerignore

  node_modules
  .git
  .env

Avoid latest in production

Use fixed versions to keep builds predictable.

Separate dev and production setups

They have different requirements.

Use Docker Compose for local development

It helps simulate real systems easily.

Use clear service names

Examples:

kafka
postgres
airflow

This simplifies networking and debugging.

10. Conclusion

Containerization changes how you think about environments.

Docker packages your application into a portable unit.
Docker Compose runs entire systems with one command.
Your pipelines become reproducible and consistent.

The real shift is this:

You stop debugging environments — and start defining them as code.

And once you reach that point:

You’re no longer just writing code — you’re building systems.

Data Warehouses, Data Marts, Data Lakes, and Lakehouses - Explained Like You’re Building Them in Real Life

Anthony Gicheru — Sun, 03 May 2026 08:10:27 +0000

If you’ve been around data engineering long enough, you’ve probably heard these terms thrown around in meetings:

“Just dump it in the data lake”
“We’ll expose it through the warehouse”
“That goes into the mart”
“We’re moving to a lakehouse architecture”

And honestly… it can sound like four different ways of saying the same thing.

They’re not.

Each one solves a slightly different problem in the data ecosystem. Once you understand the “why” behind each, the architecture suddenly feels a lot less like buzzwords and more like a clean system design.

Let’s break it down in a practical, engineer-first way.

1. The Big Picture (Why all these systems exist)

In most companies, data doesn’t come from one place — it flows in from everywhere:

User clicks from web/mobile apps
Payments and transactions
Logs from servers
Third-party APIs (Stripe, Shopify, etc.)
IoT or streaming data (Kafka, sensors, etc.)

Now here’s the problem:

Raw data is messy. Business users don’t want messy.

So we build systems that progressively refine data from:

Raw → Clean → Structured → Business-ready

That’s where these four concepts come in:

Data Lake → store everything raw
Data Warehouse → structured analytics-ready data
Data Mart → department-specific slices of warehouse data
Lakehouse → hybrid of lake + warehouse

2. Data Lake — “Store everything first, figure it out later”

A data lake is basically a massive storage system where you dump raw data in its original format.

Think of it like:

A giant warehouse where you throw every box in as-is, without opening it.
Or even better: a farm storage system where everything is stored right after harvest, unprocessed and mixed together.

Characteristics:

Stores structured, semi-structured, and unstructured data
Cheap storage (usually object storage like S3)
Schema is applied when reading, not writing (schema-on-read)

Example tools:

Amazon S3
Azure Data Lake Storage
Google Cloud Storage

Example:

You might store:

/events/clicks/2026/05/01.json
/logs/api/2026/05/01.log
/payments/stripe/2026/05/01.parquet

No transformations. No enforcement. Just storage.

Here’s the catch:

If you’re not careful, a data lake becomes a data swamp — lots of data, zero usability.

3. Data Warehouse — “Clean, structured, and business-ready”

A data warehouse is where data goes after it has been cleaned, transformed, and modeled for analytics.

Think of it like:

A well-organized supermarket where everything is cleaned, packaged, labeled, and placed on the right shelves.
You don’t pick raw potatoes from the soil — you get them washed, sorted, and priced.

Characteristics:

Structured data only
Schema-on-write (you define structure before loading)
Optimized for analytics queries (OLAP systems)
Highly curated and trustworthy

Example tools:

Amazon Redshift
Snowflake
Google BigQuery

Typical workflow:

Extract data from sources
Transform (clean, join, aggregate)
Load into warehouse tables

Example SQL model:

CREATE TABLE sales_fact (
    order_id INT,
    customer_id INT,
    product_id INT,
    amount DECIMAL(10,2),
    order_date DATE
);

Now business analysts can run queries like:

SELECT product_id, SUM(amount)
FROM sales_fact
GROUP BY product_id;

4. Data Marts — “Department-specific mini warehouses”

A data mart is a subset of a data warehouse focused on a specific business domain.

Think of it like:

A grocery store or specialty shop — like a bakery, butcher, or vegetable shop.
It doesn’t sell everything. It only sells what its customers actually need.

Characteristics:

Smaller scope than a warehouse
Built for a specific team (finance, marketing, sales)
Faster queries for targeted use cases

Example:

A marketing data mart might include:

Campaign performance
Customer acquisition metrics
Ad spend data

Example structure:

CREATE TABLE marketing_campaign_performance AS
SELECT
    campaign_id,
    SUM(clicks) AS total_clicks,
    SUM(impressions) AS total_impressions
FROM ad_events
GROUP BY campaign_id;

Why it exists:

Instead of everyone querying a massive warehouse, teams get pre-optimized datasets.

5. Data Lakehouse — “Best of both worlds”

Now this is where things get interesting.

A lakehouse combines:

The flexibility of a data lake
The structure and performance of a data warehouse

Think of it like:

A modern retail system where the warehouse and supermarket are combined into one smart facility.
Raw goods arrive, but they are immediately tracked, organized, and made queryable without losing flexibility.

Characteristics:

Uses low-cost storage (like a lake)
Adds structure, ACID transactions, and governance
Supports both analytics and ML workloads

Example tools:

Apache Spark + Delta Lake
Apache Iceberg
Apache Hudi

Why it matters:

In traditional setups:

Data lakes = flexible but messy
Warehouses = clean but expensive and rigid

Lakehouses try to remove that tradeoff.

6. How They Work Together (Real Architecture Flow)

A modern data pipeline often looks like this:

[ Data Sources ]
      ↓
   DATA LAKE (raw storage)
      ↓
ETL / ELT pipelines (Airflow, Spark)
      ↓
DATA WAREHOUSE (modeled data)
      ↓
DATA MARTS (team-specific views)
      ↓
Dashboards / BI tools

Or in a lakehouse setup:

[ Data Sources ]
      ↓
DATA LAKEHOUSE (single system)
      ↓
BI + ML + Analytics directly

7. Practical Example (Mini Pipeline)

Let’s say we’re processing e-commerce data.

Step 1: Raw data in S3 (Data Lake)

{
  "order_id": 101,
  "user_id": 55,
  "amount": 250,
  "timestamp": "2026-05-01T10:00:00Z"
}

Step 2: Spark transformation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("etl").getOrCreate()

df = spark.read.json("s3://datalake/raw/orders/")

clean_df = df.dropna() \
             .withColumnRenamed("amount", "order_amount")

clean_df.write.mode("overwrite").parquet("s3://warehouse/sales_fact/")

Step 3: Load into warehouse (Redshift example)

COPY sales_fact
FROM 's3://warehouse/sales_fact/'
IAM_ROLE 'arn:aws:iam::123456:role/RedshiftRole'
FORMAT AS PARQUET;

Step 4: Create a data mart

CREATE TABLE sales_summary AS
SELECT
    DATE(order_date) AS date,
    SUM(order_amount) AS revenue
FROM sales_fact
GROUP BY DATE(order_date);

8. Common Pitfalls (Where most teams mess up)

1. Turning the data lake into a swamp

Dumping everything without metadata or structure leads to chaos.

2. Over-modeling too early

Trying to build perfect schemas upfront slows everything down.

3. Duplicating logic across marts

You end up with inconsistent metrics like “Revenue_v1”, “Revenue_final”, “Revenue_real_final”.

4. No governance layer

Without access control and cataloging, nobody trusts the data.

9. Best Practices

1. Use layered architecture

Raw (lake)
Cleaned (staging)
Modeled (warehouse)
Aggregated (marts)

2. Standardize transformations

Use tools like:

dbt
Apache Airflow
Spark jobs with clear ownership

3. Define a single source of truth

One metric definition per business KPI. No duplicates.

4. Treat data like software

Version it, test it, document it.

5. Monitor everything

Pipeline failures
Data freshness
Schema changes

10. Conclusion — The mental model that matters

If you strip away the jargon, it’s really simple:

Data Lake → store everything
Data Warehouse → clean and organize it
Data Mart → tailor it for teams
Lakehouse → unify storage and analytics

The real skill in data engineering isn’t memorizing definitions.

It’s knowing:

When to keep data raw, when to structure it, and when to specialize it.

Once that clicks, designing data systems becomes a lot more intuitive — and honestly, more fun to build.

I Avoided Airflow's TaskFlow API for a Year. I Shouldn't Have.

Anthony Gicheru — Fri, 24 Apr 2026 10:52:39 +0000

I knew TaskFlow existed. It's been in Airflow since 2.0, and I'd scrolled past it in the docs more than once. I just assumed it was decorator syntax layered over the same execution model, and I had actual pipelines to ship. So I kept writing PythonOperator tasks, wiring xcom_push and xcom_pull by hand between them, and calling it fine.

That was about a year of production work before I finally refactored something into TaskFlow and understood what I'd been missing.

The Part Traditional Operators Make You Do Yourself

The classic pattern goes like this. You define a Python callable, wrap it in a PythonOperator, and when one task needs to pass data to the next, you reach for XCom. XCom, if you're newer to Airflow, is how tasks store small values in the metadata database so a downstream task can retrieve them later. The upstream task calls ti.xcom_push(key='data', value=something) and the downstream task calls ti.xcom_pull(task_ids='upstream_task_name', key='data'), where you write both strings yourself, by hand, every time.

def extract(**kwargs):
    data = [1, 2, 3]
    kwargs['ti'].xcom_push(key='raw_data', value=data)

def transform(**kwargs):
    data = kwargs['ti'].xcom_pull(task_ids='extract', key='raw_data')
    return [x * 2 for x in data]

The failure mode is quiet. Get the task ID slightly wrong, forget to update a key string after renaming a task, and you get None downstream with no error at parse time, no warning in the logs. Just None showing up somewhere two steps later in a task that expected a list. I've wasted more debugging time on that specific bug class than I want to admit.

And beyond the debugging, there's just the volume of it. In a DAG with six or eight tasks, you're writing xcom_push and xcom_pull pairs constantly. None of it has anything to do with the logic you're trying to express. It's plumbing Airflow makes you wire yourself, and it accumulates fast.

What TaskFlow Changes

TaskFlow replaces that wiring with two decorators. You put @task on each function, and when one task's return value gets passed as an argument to the next, Airflow creates the XCom push and pull automatically. The @dag decorator wraps the pipeline itself, and you instantiate it at the bottom of the file. That's it.

from airflow.decorators import dag, task
from datetime import datetime

@dag(start_date=datetime(2024, 1, 1), schedule='@daily', catchup=False)
def my_pipeline():

    @task
    def extract():
        return [1, 2, 3]

    @task
    def transform(data):
        return [x * 2 for x in data]

    transform(extract())

my_pipeline()

The execution engine underneath is the same and XCom is still happening, but you're not managing it — no key strings to mistype, no task_ids arguments to keep in sync with task names. The data just flows.

What I didn't expect was how much easier TaskFlow functions are to unit test. Call transform([1, 2, 3]) directly in a test and it works like any Python function. Try that with a callable that expects ti in **kwargs and you're mocking Airflow internals before you've tested a single line of business logic.

The mental model shifts too. With traditional operators, I was always thinking about wiring: task A pushes to XCom with this key, task B pulls from task A using that key, and I have to keep all of that consistent across the file. With TaskFlow, I think about data moving through functions, which is actually how the pipeline works.

The Gas Price Pipeline That Made It Concrete

I had a small job running in production: pull daily gas price data from a public API, clean it up, and load it into PostgreSQL for a downstream dashboard. Three tasks, nothing complicated. I'd originally built it with the traditional approach, and it had been sitting there working fine for months.

The fetch task had this in it:

json_data = df.to_json(orient='records')
kwargs['ti'].xcom_push(key='raw_gas_data', value=json_data)

That to_json call wasn't there because the transform step needed JSON. It was there because you can't reliably push a Pandas DataFrame through XCom — it won't serialize cleanly. So I was converting to JSON in fetch, pushing the string, then immediately parsing it back into a DataFrame at the top of transform. The whole round trip existed to satisfy Airflow's plumbing, not anything related to the pipeline's purpose.

After the refactor:

@task
def fetch_gas_prices():
    return decoded_data  # list or dict, serializes fine

@task  
def transform_gas_prices(raw_data):
    # work directly with raw_data, no JSON parsing needed
    return df.to_json(orient='records')

raw = fetch_gas_prices()
cleaned = transform_gas_prices(raw)
store_gas_prices(cleaned)

The logic inside each function barely changed. What disappeared was the JSON round trip, all the xcom_push and xcom_pull calls, and the **kwargs threading through everything. The code now describes the pipeline instead of describing how to configure Airflow to run it. That might sound like a small thing until you try to onboard someone to a DAG written the old way and watch them spend twenty minutes tracing XCom keys through four functions.

XCom Size Limits Don't Disappear With TaskFlow

TaskFlow can mislead you on this. Because passing data between tasks now looks like passing a variable between two Python functions, it starts to feel like it's free. It isn't.

Every value a TaskFlow task returns still gets serialized and written to Airflow's metadata database as an XCom. Exactly the same as before, just automatic. Return a DataFrame with a few million rows from your extract task and that entire object gets pickled into a database that was never designed to hold payloads that size. I've seen this go wrong slowly: the metadata database bloats over weeks, and eventually DAG parsing and scheduling starts degrading across the whole environment, not just the one DAG responsible. By the time you notice, it's not obvious what caused it.

The fix is to pass references once your data gets beyond trivially small. Write the DataFrame to S3 or a staging table in the extract task, return the file path or table name, and have the next task read from storage. A rough rule: if the value wouldn't fit comfortably in a Slack message, it probably doesn't belong in an XCom. That constraint was always there with the traditional approach — TaskFlow just makes it easier to forget.

The Fastest Way to Understand It

Pick a DAG you've already written in the traditional style — something with three or four tasks where you can see the whole thing at once — and refactor it into TaskFlow. The point is to do it side by side with something you already understand, so the structural difference is obvious rather than buried under unfamiliar logic. Strip out the PythonOperator wrappers, add the decorators, replace the xcom_push and xcom_pull calls with return values and function arguments. The business logic inside each task barely moves. What changes is whether the file reads like a data pipeline or like a set of Airflow instructions that happen to contain one.

Most people who do this refactor want to go back and do it to everything else they've written. That's probably the clearest signal that it was worth understanding.

Data Pipelines Explained Simply (and How to Build Them with Python)

Anthony Gicheru — Fri, 17 Apr 2026 07:34:55 +0000

Data pipelines are the backbone of modern data-driven organizations. They automate the movement, transformation, and storage of data - from raw sources to actionable insights.

Python has become the go-to language for building scalable pipelines because of its rich ecosystem, flexibility, and ease of use.

This guide walks through the fundamentals, tools, and best practices for building robust data pipelines using Python.

Understanding Data Pipelines

Imagine you need to supply clean water to a village. The process involves collecting water from different sources (rivers, wells, rain), purifying it, transporting it, and storing it so people can access it whenever they need it.

A data pipeline works in a very similar way.

It automates the journey of raw, unstructured data from multiple sources (like databases, APIs, or IoT devices) and transforms it into clean, usable data stored in a destination (like a data warehouse) for analysis.

Components of a Data Pipeline

Let’s break it down using the same analogy:

1. Collecting Water (Data Ingestion)

Just like gathering water from lakes or wells, a pipeline starts by extracting data from sources such as databases, APIs, spreadsheets, or sensors.

The goal here is simple: get all the data into one system, no matter how scattered it is.

2. Filtering and Purifying (Data Transformation)

Raw water isn’t clean—and neither is raw data.

At this stage, the pipeline:

Removes duplicates
Handles missing values
Standardizes formats
Enriches data

This is where messy data becomes usable.

3. Transporting Through Pipes (Data Movement)

Once cleaned, water flows through pipes. In data pipelines, this represents the movement of data between systems.

This can involve:

ETL processes
Message queues (like Kafka)
Cloud data transfer services

The goal is to move data efficiently without delays or bottlenecks.

4. Storing in Tanks (Data Storage)

Clean water is stored in tanks. Similarly, processed data is stored in:

Data warehouses (like Snowflake)
Data lakes (like AWS S3)
Databases

This is where data becomes ready for use.

5. Accessing on Demand (Data Consumption)

Finally, people use the water.

In the same way, data is consumed through:

Dashboards
APIs
Machine learning models

This is where insights actually happen.

Essential Python Libraries and Tools

Python supports every stage of a pipeline:

Data Ingestion

requests - API calls
pandas - handling CSV/JSON files

Transformation

pandas - cleaning and aggregation
PySpark - large-scale distributed processing

Storage

SQLAlchemy - database interaction
boto3 - AWS S3 integration

Orchestration

Apache Airflow - workflow scheduling and automation
Dagster - modern pipeline orchestration with observability

Best Practices

Error Handling

Implement retries and proper logging to avoid silent failures.

Monitoring

Track pipeline health using tools like Airflow’s UI.

Documentation

Keep clear documentation for:

Code
Dependencies
Workflow logic

Testing

Test each stage of the pipeline using:

Unit tests
Sample datasets

Popular Frameworks for Advanced Use Cases

Apache Airflow - Best for complex workflows with dependencies
Dagster - Strong focus on testing and data asset visibility
Prefect - Simplifies building fault-tolerant pipelines
Luigi - Good for batch processing and dependency management

The Real Difference Between ETL and ELT

Anthony Gicheru — Sun, 12 Apr 2026 21:36:22 +0000

ETL and ELT look like the same acronym with the letters reshuffled, and most introductions treat them that way — a terminology distinction worth memorizing, not a decision worth thinking carefully about. The actual difference is architectural: where transformation happens in your pipeline shapes how you debug failures, how you handle business rule changes, and whether you can answer the question "what did the raw data look like?" six months after a pipeline has been running.

What the terms actually mean

Both describe the same three stages: you extract data from a source system, you transform it — cleaning, filtering, reshaping — and you load it into somewhere you can query it. The difference is the order of the last two steps, and that sequence determines where the heaviest processing work in your pipeline actually lives.

In ETL, transformation happens before the data lands anywhere permanent. You pull raw records from your source, run cleaning logic in an intermediate layer, and only write the finished result into your data warehouse. The warehouse receives data that's already been processed.

In ELT, you flip it. You extract raw data and push it directly into your warehouse first, then write transformation logic that runs inside the warehouse after the fact. Your storage layer gets the raw data immediately; the cleaning happens later, against data that's already sitting in a table.

Where transformation happens determines what infrastructure you need, how flexible your pipeline is, and what your options are when something breaks.

Why ETL was the default for decades

ETL made sense when storage was expensive and warehouse compute was slow. If you were paying enterprise licensing fees for an on-premise data warehouse and running queries against spinning disks, you had a strong incentive to minimize what went in. Storing raw data — with all its duplicates, malformed rows, and irrelevant columns — was genuinely costly. So teams invested in dedicated transformation servers: ETL tools like Informatica or SSIS that cleaned and shaped data before it ever touched the warehouse.

The real cost was rigidity. When a business rule changed — say, the definition of "active customer" shifted from 90-day activity to 30-day — you had to update transformation logic in a separate system, re-run historical data through the pipeline, and verify the results across two different environments. Debugging a transformation failure meant chasing logs through an ETL server and a warehouse that had no shared context, and any business rule change required updating logic in both systems before you could trust the numbers again.

Why ELT took over

Cloud warehouses changed the underlying economics. Storage in Snowflake, BigQuery, or Redshift is cheap enough that holding raw data isn't a budget risk. More importantly, these systems are built to run heavy SQL transformations efficiently at scale — so doing your transformation work inside the warehouse is often faster than running it on an external server anyway.

ELT takes direct advantage of that. You land raw data first, which keeps your ingestion layer simple and fast. Transformation logic lives in SQL — or in dbt, which compiles down to SQL — and runs directly inside the warehouse against data that's already there. When a business rule changes, you update a SQL model and re-run it against the raw data you've already stored. You don't re-ingest anything.

There's also a recovery argument that's easy to overlook. In ELT, raw data is always sitting in your warehouse. If a transformation has a bug, you fix the logic and re-run. In a pure ETL setup where only transformed data is stored, a bad transformation step can silently corrupt your historical record — unless you kept the source data somewhere else, which teams frequently don't.

What this looks like on a real pipeline

Say you're building a daily reporting pipeline for an e-commerce company. The source is a PostgreSQL transactional database with an orders table seeing roughly 50,000 inserts per day. Finance needs a reporting table showing revenue, refunds, and volume by region.

In an ETL setup, you write a script — probably Python — that queries new orders since the last run, applies cleaning logic (null shipping addresses, test orders in a dummy currency code, timestamps in the wrong timezone), and writes cleaned rows directly into the warehouse reporting table. Finance sees clean data immediately. But if someone asks six months later what the raw order record looked like before your script touched it, you can't answer that from the warehouse.

In an ELT setup, your ingestion job does one thing: copy raw order rows from PostgreSQL into a raw schema in your warehouse, unchanged. A dbt model then runs the transformation — filtering test orders, correcting timezones, calculating normalized revenue — and writes the result into the schema finance actually queries. The raw schema stays intact. When a revenue number gets questioned, you can trace the final figure back to the exact source row and show precisely which transformation logic produced it.

-- dbt model: orders_cleaned.sql
select
    order_id,
    customer_id,
    convert_timezone('UTC', order_timestamp) as order_timestamp_utc,
    amount_usd,
    region
from {{ source('raw', 'orders') }}
where currency_code != 'TEST'
  and shipping_address is not null

That model runs inside the warehouse, against raw data that's already there. Change the filter logic, run dbt run, and your entire history updates.

The mistake that quietly breaks ELT pipelines

The most common ELT failure I've seen isn't a technical one — it's treating "load first, transform later" as if it means transformation is optional. Teams land raw data into the warehouse, call the ingestion pipeline done, and leave analysts writing the same cleaning logic in four different dashboards. Revenue numbers start disagreeing across reports. Nobody can explain which number is right because the transformation step was never formalized.

ELT doesn't mean raw data is analysis-ready. It means transformation happens inside the warehouse — not never. If your raw orders table has null addresses and no model handles them, every analyst touches that table differently, and your data drifts apart over time. The transformation layer in ELT needs to be version-controlled, documented, and owned by someone. That's the gap dbt fills — it brings software engineering practices to the part of ELT that's easiest to skip.

Where to start

If you're building on a cloud warehouse today, ELT is almost certainly the right default. Keep ingestion simple, land raw data, and define transformations in SQL models you can test and version. ETL still makes sense when data genuinely cannot be stored raw — PII that needs masking before it touches your warehouse, for example — or when you're working within a legacy stack built around ETL tooling that isn't worth migrating yet.

The next time you're designing a pipeline, ask one concrete question first: does the raw data need to be cleaned before it enters storage, or can I afford to store it and clean it there? That answer will tell you which pattern fits.