DEV Community: Sarah Muriithi

Apache Kafka Deep Dive: Core Concepts, Data Engineering Applications and Real-World Production Practices.

Sarah Muriithi — Wed, 10 Sep 2025 05:25:56 +0000

Apache Kafka is an open source distributed streaming platform. It is designed to handle real-time streams of data at scale, in a fault-tolerant way.

1. Core Concepts.

Clusters.

A collection of brokers(servers) working together to provide fault tolerance, scalability and high throughput.
They handle million of messages per second in distributed systems.

Topic.

A topic is a logical channel where messages are produced and consumed.
Each topic is split into partitions for parallel processing.
Writing a Kafka topic:

bin/kafka-topics.sh --create \
  --topic my-first-topic \
  --bootstrap-server localhost:9092 \
  --partitions 3 \
  --replication-factor 1

Listing all topics:

bin/kafka-topics.sh --list --zookeeper localhost:2181

Partition.

It is an ordered append-only sequence of records inside a topic.
They enable parallel consumption and allow horizontal scaling.

Brokers.

A Kafka server that stores partitions and serves clients.
Kafka brokers manage topic partitions, mess<age replication and data storage and retrieval.

Producers.

They send messages (events) to Kafka topics.
They ensure that messages with the same key go to the same partition.
Writing messages to a Kafka topic:

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic customer_orders

Consumers.

They read messages from Kafka topics.
Consumers belong to consumer groups, thus allowing parallel processing.
Reading messages from a topic:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic customer_orders --from-beginning

Offset.

A unique identifier for each message within a partition.
Consumers use offsets to track which messages they’ve already read.

Zookeeper vs Kraft.

Zookeeper is an external system used by Kafka for metadata management and cluster coordination.
Kraft is a zookeeper-free mode where Kafka manages it's own metadata using the Raft consensus algorithm.
Starting a zookeeper.

bin/zookeeper-shell.sh localhost:2181

Kafka Connect.

Kafka Connect is used to stream data between Kafka and external data systems like databases, file systems, and cloud storage.
Running a Kafka Connect:

bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties

Replication.

Kafka keeps multiple copies of partitions across brokers which provides fault tolerance.
One broker hosts the leader partition, others host replicas (followers).
Producers and consumers talk to the leader.

Retention.

Kafka doesn’t delete messages immediately after consumption.
Messages are kept based on time (e.g., 7 days) or size (e.g., 1GB).
Allows consumers to re-read messages later (useful for replaying events).

2. Data Engineering Applications.

Real-Time Data Ingestion.

Streams data from multiple sources (databases, APIs, IoT devices, apps).
Example: Collecting clickstream events from a website.
Tools: Kafka Connect + Source Connectors (Debezium).

ETL/ELT Pipelines.

Kafka acts as the transport layer in ETL workflows.
Data can be cleaned/transformed on the fly using Kafka Streams.

Data Lake/ Warehouse Ingestion.

Kafka feeds batch & streaming data into storage systems.

Examples:

- Write data to S3 (data lake).
-Send cleaned data into Snowflake, BigQuery, or Redshift.

Change Data Capture.

Tools like Debezium integrate with Kafka to capture changes from relational databases (MySQL, Postgres, Oracle).
This enables real-time ETL, replication, and synchronization across systems.

Machine Learning Pipelines.

Kafka streams feed real-time features into ML models, powering use cases like fraud detection, dynamic pricing, or recommendation systems.

3. Real-World Production Practices.

Netflix

Netflix uses Kafka for real-time monitoring, event sourcing and recommendations.
Every playback event or error is streamed to Kafka for analysis.

LinkendIn

LinkendIn uses Kafka to process 1 trillion messages per day for activity tracking, search indexing and fraud-detection.

Uber

Uber relies on Kafa to match riders with drivers, hanles surge pricing and provides real-time Estimated Time of Arrival.

DOCKER

Sarah Muriithi — Wed, 27 Aug 2025 05:09:13 +0000

GETTING STARTED WITH DOCKER AND DOCKER COMPOSE: A BEGINNER GUIDE

1. What is Docker?

Docker is a tool used for building, running and managing containers.
A Container is a lightweight, portable package that has everything an application needs to run-code, libraries, system tools and settings.

2. Install Docker

On Ubuntu/Linux

 sudo apt install gnome-terminal
 sudo apt update
 sudo apt install -y docker.io
 sudo systemctl enable -now docker

On Windows/Mac
Install Docker Desktop
After installation, run docker version to verify that docker is working.

3. Basic Docker Concepts

- Image

It is a blueprint that contains everything needed to run an app.(OS libraries, dependencies and your code)
They are immutable.

- Container

It is a running instance of an image.
Containers are ephemeral -> you can start,stop, delete, and recreate them anytime.

- Dockerfile

It is a text file with instructions to build a custom image.
An example:

FROM python:3.12
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["python", "app.py"]

- Docker Hub/Registry

A registry is where images are stored.
Docker hub is the default public registry and one can also have private registries.

- Docker Engine

The underlying runtime that builds and runs containers.
Includes:
Deamon(dockerd) -> background services.
CLI(docker) -> command-line tool.
API -> programmatic access.

4. Basic Docker Commands

List images docker images
List running containers docker ps
Run a container docker run -d -p 8080:80 nginx
Then visit http://localhost:8080 in your browser
Stop a container docker stop <container id>

5. What is Docker Compose

Docker Compose lets you define and run multi-container applications.
Instead of starting containers one by one, you define everything in a docker-compose.yml file and run: docker compose up

6. Install Docker Compose

If you installed Docker Desktop, Compose is already included.
On Linux: sudo apt install docker-compose-plugin docker compose version

7. Useful Docker compose Commands

Start containers:
docker compose up -d
Stop containers:
docker compose down
View logs:
docker compose logs -f
Restart a service:
docker compose restart web