DEV Community

Cover image for Understanding Kafka Complete Architecture Flow - Advanced🎯
Ajinkya Singh
Ajinkya Singh

Posted on • Edited on

Understanding Kafka Complete Architecture Flow - Advanced🎯

#programming #kafka #distributedsystems #go

Please read the last two articles in my Kafka series before this one — this part gets serious.

🌐 The Big Picture: Two Brains Working Together

Think of Kafka as a well-organized company with two main components:

┌─────────────────────────────────────────────┐
│           KAFKA CLUSTER                     │
│                                             │
│  ┌────────────────────────────────────┐    │
│  │  CONTROLLERS (The Brain 🧠)        │    │
│  │  • Manage who does what            │    │
│  │  • Track what's happening          │    │
│  │  • Make decisions                  │    │
│  └──────────────┬─────────────────────┘    │
│                 ↓ Commands & Updates       │
│  ┌────────────────────────────────────┐    │
│  │  BROKERS (The Workers 💪)          │    │
│  │  • Store the actual data           │    │
│  │  • Serve producers and consumers   │    │
│  │  • Follow controller's instructions│    │
│  └────────────────────────────────────┘    │
└─────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

💡 Simple Analogy: Controllers are like managers who plan and coordinate, while Brokers are employees who do the actual work.

🚀 Part 1: SETUP - Controllers Organize Everything

Step 1: Controllers Start and Elect Leader

When Kafka starts, multiple controllers use the Raft Election algorithm to choose one leader:

Controller-1    Controller-2    Controller-3
     │               │               │
     └───────Raft Election───────────┘
                     ↓
          One becomes LEADER ⭐
                     │
            ┌────────────────┐
            │ Controller-1   │
            │   (LEADER) ⭐   │ ← Makes all decisions!
            └────────────────┘
Enter fullscreen mode Exit fullscreen mode

💡 Simple: Like choosing a class monitor who manages everything.

Step 2: Controllers Create Metadata Registry

The Controller Leader creates a comprehensive "notebook" 📒 of everything happening in the cluster:

┌──────────────────────────────────────────┐
│        METADATA REGISTRY                 │
├──────────────────────────────────────────┤
│                                          │
│ TOPICS:                                  │
│  • "orders" → 3 partitions               │
│  • "payments" → 2 partitions             │
│                                          │
│ BROKERS:                                 │
│  • Broker-1 → Alive ✅ (IP: 192.168.1.10)│
│  • Broker-2 → Alive ✅ (IP: 192.168.1.11)│
│  • Broker-3 → Alive ✅ (IP: 192.168.1.12)│
│                                          │
│ WHO'S IN CHARGE (Leaders):               │
│  • orders-partition-0 → Broker-1         │
│  • orders-partition-1 → Broker-2         │
│  • orders-partition-2 → Broker-3         │
│                                          │
│ BACKUPS (Followers):                     │
│  • orders-partition-0 → Broker-2, Broker-3│
│  • orders-partition-1 → Broker-3, Broker-1│
│  • orders-partition-2 → Broker-1, Broker-2│
└──────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

💡 Simple: Like a school register showing which classes exist, which teachers are present, who teaches which subject, and who are the substitute teachers.

Step 3: Controller Tells Brokers Their Jobs

The controller assigns specific roles to each broker:

Controller ⭐ → Broker-1: "You are the LEADER for orders-partition-0"
                         "You are a BACKUP for orders-partition-2"

Controller ⭐ → Broker-2: "You are the LEADER for orders-partition-1"
                         "You are a BACKUP for orders-partition-0"

Controller ⭐ → Broker-3: "You are the LEADER for orders-partition-2"
                         "You are a BACKUP for orders-partition-1"
Enter fullscreen mode Exit fullscreen mode

💡 Simple: Like a manager assigning tasks to employees.

📤 Part 2: PRODUCER SENDS DATA

Step 4: Producer Wants to Send Message

Your application (Producer) has data to send:

┌──────────────────────────┐
│  I have a message:       │
│  • Topic: "orders"       │
│  • Key: "user_123"       │
│  • Value: {order data}   │
│                          │
│  "Where do I send this?" │
└──────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

💡 Simple: Like having a letter but not knowing which post office to use.

Step 5: Producer Asks ANY Broker for Information

The producer can connect to any broker to get routing information:

Producer ──────────► Broker-1
"Where do I send     "Let me check
messages for         my copy of
'orders' topic?"     the registry..."
Enter fullscreen mode Exit fullscreen mode

Key Point: Every broker has a copy of the controller's metadata!

┌───────────────────────────────────┐
│  Broker-1's Copy of Metadata:     │
│  • orders-partition-0 → Broker-1  │
│  • orders-partition-1 → Broker-2  │
│  • orders-partition-2 → Broker-3  │
└───────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

💡 Simple: Like asking a postman for directions - he has a map (copy of registry).

Step 6: Producer Calculates Which Partition

The producer's client library automatically determines the target partition:

1. Hash the key: hash("user_123") = 456789
2. Divide by partitions: 456789 % 3 = 0
3. Result: Goes to Partition 0
Enter fullscreen mode Exit fullscreen mode

💡 Simple: Like a sorting machine that knows exactly which box to put each item in.

Step 7: Producer Sends to Correct Broker

Now the producer sends directly to the partition leader:

Producer ───────────► Broker-1 (Leader for P0)

Broker-1 receives:
1. ✅ Validates it's the leader for P0
2. 💾 Writes to disk: /kafka/data/orders-0/
3. 📝 Assigns offset: 1251
4. ✅ Sends "OK!" back to producer
Enter fullscreen mode Exit fullscreen mode

💡 Simple: Like mailing a letter to the correct post office that handles your area.

Step 8: Broker Replicates to Followers (Background)

After writing, the leader broker automatically replicates to followers:

Broker-1 (Leader P0) ───COPY MESSAGE───► Broker-2 (Follower P0)
                                          ✅ Copied!
         │◄──────────ACK (Copied!)───────┤

Broker-1 (Leader P0) ───COPY MESSAGE───► Broker-3 (Follower P0)
                                          ✅ Copied!
Enter fullscreen mode Exit fullscreen mode

Result: Data now exists on 3 brokers! 💪

💡 Simple: Like making photocopies of important documents and storing them in different safes.

📥 Part 3: CONSUMER READS DATA

Step 9: Consumer Wants to Read Messages

Your application (Consumer) wants to read data:

┌──────────────────────────┐
│  I want to read from:    │
│  • Topic: "orders"       │
│  • Group: "my-group"     │
│                          │
│  "How do I start?"       │
└──────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

💡 Simple: Like wanting to read a book but not knowing which library has it.

Step 10: Consumer Connects and Gets Metadata

The consumer connects to any broker to get cluster information:

Consumer ────────────► Broker-2
"Tell me everything    "Here's the full
about 'orders'"        cluster map!"
Enter fullscreen mode Exit fullscreen mode

Broker returns:

┌───────────────────────────────────┐
│  Topic "orders" info:             │
│  • Partition 0 → Leader: Broker-1 │
│  • Partition 1 → Leader: Broker-2 │
│  • Partition 2 → Leader: Broker-3 │
└───────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

💡 Simple: Like getting a mall directory showing which stores are on which floor.

Step 11: Consumer Joins Group (Controller Coordinates)

The consumer joins a consumer group for coordinated reading:

Consumer → Broker → Controller ⭐
"I want to join      "Let me assign
group 'my-group'"    partitions..."

Controller decides:
┌───────────────────────────────────┐
│  Group "my-group" has 1 consumer  │
│  Topic "orders" has 3 partitions  │
│                                   │
│  Assignment:                      │
│  Consumer-A → [P0, P1, P2]        │
│  (gets all 3 partitions)          │
└───────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

💡 Simple: Like a teacher assigning homework to students.

Step 12: Consumer Reads and Tracks Progress

How Consumer Manages Reading Position:

STARTUP (Once):
Consumer → Broker: "Where did I leave off?"
Broker → Consumer: "P0: offset 1250, P1: offset 890, P2: offset 2100"
Consumer stores in memory ✅

CONTINUOUS READING:
Consumer fetches using local memory (no broker queries!)
• Fetch from P0: offset 1250 → 1300 → 1350 (tracked in memory)
• Fetch from P1: offset 890 → 940 → 990 (tracked in memory)
• Fetch from P2: offset 2100 → 2150 → 2200 (tracked in memory)

PERIODIC COMMIT (Every 5 seconds or after batch):
Consumer → Broker: "Save progress: P0=1350, P1=990, P2=2200"
Enter fullscreen mode Exit fullscreen mode

🔥 KEY POINTS:

  • Consumer reads position from broker ONCE at startup
  • Tracks current position IN MEMORY while reading
  • Saves progress back to broker PERIODICALLY (every 5 seconds by default)

💡 Simple: Like checking your bookmark when opening a book, remembering your page while reading, and updating the bookmark occasionally.

Step 13: Consumer Pulls Data from Brokers

Consumer fetches data in parallel from all partition leaders:

Consumer ───────► Broker-1 (P0) → Returns 50 events
        ├────────► Broker-2 (P1) → Returns 50 events
        └────────► Broker-3 (P2) → Returns 50 events

Total: 150 events fetched!
         ↓
   Process all events
   (Your business logic)
         ↓
   All done! ✅
Enter fullscreen mode Exit fullscreen mode

💡 Simple: Like reading from multiple books at the same time, keeping track of your progress in each.

🔧 Part 4: FAILURE HANDLING

SCENARIO A: Broker Fails (Controller Handles) 

Broker-1 crashes! 💥

STEP 1: Controller Detects Failure
Controller ⭐: "Broker-1 stopped responding!"

STEP 2: Controller Checks Metadata
┌───────────────────────────────────┐
│  Partition 0 (orders):            │
│  • Leader: Broker-1 💀            │
│  • Followers: Broker-2 ✅, Broker-3 ✅│
│                                   │
│  Need new leader for P0!          │
└───────────────────────────────────┘

STEP 3: Controller Elects New Leader
Controller ⭐ → Broker-2: "You are now LEADER for P0!"
Broker-2: "OK! I'm the new leader!"

STEP 4: Controller Updates Metadata
┌───────────────────────────────────┐
│  Partition 0 (orders):            │
│  • Leader: Broker-2 ⭐ (NEW!)     │
│  • Followers: Broker-3 ✅         │
│  • Broker-1: 💀 (removed)         │
└───────────────────────────────────┘

STEP 5: Controller Notifies Everyone
Controller → All: "Metadata changed! P0 leader is now Broker-2!"

Total time: 2-3 seconds! ⚡
Enter fullscreen mode Exit fullscreen mode

💡 Simple: Like when a teacher is absent, the principal quickly assigns a substitute.

SCENARIO B: Consumer Fails (Controller Rebalances) 

Consumer-A crashes! 💥

STEP 1: Group Coordinator Detects
Coordinator: "Consumer-A missed 3 heartbeats!"

STEP 2: Coordinator Triggers Rebalance
Coordinator → Controller: "Group 'my-group' needs rebalancing!"
Controller → Other Consumers: "Stop! Rebalancing..."

STEP 3: Controller Reassigns Partitions
Before:
Consumer-A: [P0, P1] 💀 (dead)
Consumer-B: [P2] ✅

After:
Consumer-B: [P0, P1, P2] ✅ (takes over all!)

STEP 4: Controller Notifies Consumer-B
Controller → Consumer-B: "You now read P0, P1, P2"

Consumer-B resumes reading:
✅ Loads last saved positions
✅ Continues from where Consumer-A left off
✅ No messages lost!
Enter fullscreen mode Exit fullscreen mode

💡 Simple: Like when one waiter is sick, another takes over their tables.

🎨 The Complete Picture 

┌────────────────────────────────────────────────────────┐
│                   KAFKA CLUSTER                        │
│                                                        │
│  ┌───────────────────────────────────────────────┐    │
│  │     CONTROLLERS (The Managers 🧠)             │    │
│  │                                               │    │
│  │  Controller-1 ⭐  Controller-2  Controller-3  │    │
│  │    (Leader)      (Follower)    (Follower)    │    │
│  │                                               │    │
│  │  Maintains METADATA REGISTRY                  │    │
│  │  • Who's alive?                               │    │
│  │  • Who's the leader?                          │    │
│  │  • Who reads what?                            │    │
│  └──────────────────┬────────────────────────────┘    │
│                     ↓ Commands & Notifications        │
│  ┌───────────────────────────────────────────────┐    │
│  │     BROKERS (The Workers 💪)                  │    │
│  │                                               │    │
│  │  Broker-1      Broker-2      Broker-3        │    │
│  │  P0 (L)⭐      P1 (L)⭐      P2 (L)⭐         │    │
│  │  P1 (F)        P2 (F)        P0 (F)          │    │
│  │  P2 (F)        P0 (F)        P1 (F)          │    │
│  │                                               │    │
│  │  Stores & Serves Data                         │    │
│  └───────────────────────────────────────────────┘    │
│         ↑              ↑              ↑               │
└─────────┼──────────────┼──────────────┼───────────────┘
          │              │              │
          ↓              ↓              ↓
    Producer-1     Producer-2     Producer-3
          │              │              │
          └──────────────┴──────────────┘
                         │
         All query metadata from any broker
                         │
          ┌──────────────┴──────────────┐
          ↓                             ↓
    Consumer-A                    Consumer-B
    Group: g1                     Group: g1
    Reads: P0,P1                  Reads: P2
Enter fullscreen mode Exit fullscreen mode

📋 Key Roles Summary

Controller (The Boss 👔)

MANAGES:

  • ✅ Who's alive? (broker health)
  • ✅ Who's in charge? (partition leaders)
  • ✅ Who reads what? (consumer assignments)
  • ✅ What exists? (topics, partitions)

DECIDES:

  • ✅ New leader when broker fails
  • ✅ Partition assignments for consumers
  • ✅ Where new partitions go

NOTIFIES:

  • ✅ Tells brokers their jobs
  • ✅ Updates everyone on changes
  • ✅ Coordinates rebalancing

Broker (The Worker 👷)

STORES:

  • ✅ Actual data on disk
  • ✅ Log files for partitions
  • ✅ Copy of metadata (from controller)

SERVES:

  • ✅ Producer write requests
  • ✅ Consumer read requests
  • ✅ Metadata queries

REPLICATES:

  • ✅ Copies data to followers
  • ✅ Syncs with leader
  • ✅ Reports status to controller

Producer (The Sender 📤)

DOES:

  • ✅ Creates messages
  • ✅ Queries metadata (from any broker)
  • ✅ Calculates partition (key hash)
  • ✅ Sends to correct broker

DOESN'T CARE ABOUT:

  • ❌ Controllers (transparent)
  • ❌ Followers (writes only to leader)
  • ❌ Other producers

Consumer (The Receiver 📥)

DOES:

  • ✅ Joins consumer group
  • ✅ Gets partition assignment
  • ✅ Tracks reading position in memory
  • ✅ Pulls from leader brokers
  • ✅ Saves progress periodically

DOESN'T CARE ABOUT:

  • ❌ How controller assigns partitions
  • ❌ Follower replicas
  • ❌ Other consumer groups

🎯 The Magic: Why This Works So Well

1. Separation of Concerns

  • CONTROLLERS think 🧠
  • BROKERS work 💪
  • Controllers don't handle data
  • Brokers don't make decisions
  • Like: Managers plan, workers execute

2. Everything Has a Backup

  • Controllers: 3 copies (1 leader + 2 followers)
  • Partitions: 3 copies (1 leader + 2 followers)
  • Metadata: All brokers have a copy
  • Result: If anything fails, backups take over!

3. Distributed = Fast + Reliable

  • Multiple brokers = Parallel processing
  • Multiple partitions = Load distribution
  • Multiple replicas = No data loss
  • Like: Many checkout lanes at a store

4. Automatic Recovery

  • Broker fails → Controller elects new leader (seconds)
  • Consumer fails → Controller reassigns partitions (seconds)
  • Controller fails → Another controller becomes leader (seconds)
  • All automatic! No human intervention needed!

I also took some help from Claude to understand and make a few concepts more visual.

Top comments (1)

Collapse
 
jjbb profile image
Jason Burkes •

This breakdown of Kafka's roles oddly reminds me of how a film set runs—directors, assistant directors, crew, and backup plans for every shot so the production never really stops even if something goes wrong.