DEV Community: Nanditha Vuppunuthula

Kafka Architecture at Uber: Powering Real-Time Mobility at Scale

Nanditha Vuppunuthula — Sat, 14 Jun 2025 18:29:17 +0000

Uber’s meteoric growth and global reach depend on the ability to process, analyze, and react to massive streams of data in real time. At the heart of this capability is Apache Kafka, which Uber has transformed into a highly customized, resilient, and scalable backbone for its data infrastructure. Here’s a deep dive into how Kafka powers Uber’s core systems, from ride requests to dynamic pricing.

Why Kafka?

Uber’s business hinges on real-time data: rider and driver locations, trip events, payments, and more. Kafka was chosen for its ability to:

Handle trillions of messages and petabytes of data daily

Provide high throughput and low latency

Guarantee durability and fault tolerance

Support both batch and real-time processing

Key Architectural Innovations

1. Federated Kafka Clusters

Scalability & Reliability: Instead of one monolithic Kafka cluster, Uber operates many federated clusters, each with around 150 nodes. This makes scaling easier and reduces operational risk.

Cross-Cluster Replication: Uber developed uReplicator, a tool to synchronize data across clusters and data centers, ensuring global data availability and disaster recovery.

2. Tiered Storage

Local & Remote Storage: Kafka brokers store recent data on fast local disks (SSDs) for quick access, while older data is offloaded to remote, cost-effective storage. This two-tier approach decouples storage from compute, reducing hardware costs and enabling longer data retention without performance trade-offs.

3. Consumer Proxy Layer

Simplified Client Management: With hundreds of microservices in different languages, Uber built a proxy layer that standardizes Kafka consumption, handles retries, and manages errors (like poison pill messages) via dead-letter queues (DLQ). This keeps the system robust and easy to maintain.

4. Security & Authorization

End-to-End Encryption: Mutual TLS (mTLS) secures all producer-broker and consumer-broker connections. Uber’s internal PKI (uPKI) system manages certificates for both brokers and clients.

Fine-Grained Access Control: Requests are authorized via Uber’s IAM framework, ensuring only permitted services can produce or consume from specific topics.

Kafka in Action: Dynamic Pricing
Uber’s surge pricing is a textbook example of Kafka’s power:

Data Ingestion: Millions of GPS and event messages per second flow from rider and driver apps into Kafka.

Stream Processing: Tools like Apache Flink consume these streams, analyzing supply and demand in real time.

Decision Making: Pricing models update fares every few seconds, with results published back to Kafka for downstream systems and user notifications.

Benefits Realized
Real-Time Responsiveness: Kafka’s low latency enables Uber to match riders and drivers and adjust prices instantly.

Reliability: Features like partitioning, replication, and DLQ ensure data is never lost and the system remains operational even during failures.

Operational Efficiency: Tiered storage and federated clusters keep costs manageable while supporting massive scale.

Security: End-to-end encryption and strict authorization protect sensitive data and maintain user trust.

Conclusion

Uber’s Kafka architecture is a masterclass in building a real-time, resilient, and scalable data backbone. Through innovations like federated clusters, tiered storage, consumer proxies, and custom replication, Uber has pushed Kafka to its limits—enabling everything from seamless ride matching to dynamic pricing and global business continuity. For any organization looking to build real-time, data-driven applications at scale, Uber’s Kafka journey offers invaluable lessons in both technology and strategy.

Apache Kafka and Spring Boot: A Simple Example

Nanditha Vuppunuthula — Fri, 13 Jun 2025 18:29:47 +0000

Part 1: Install and Run Kafka on Windows

Prerequisites

Java 8+

Kafka 3.x (includes Zookeeper)

A terminal like Command Prompt, Git Bash, or PowerShell

Step 1: Download Kafka

Go to: https://kafka.apache.org/downloads

Choose a binary (e.g., Kafka 3.6.0 with Scala 2.13)

Extract it to C:\kafka

Step 2: Start Zookeeper

Kafka uses Zookeeper for managing brokers. In a terminal:

cd C:\kafka bin\windows\zookeeper-server-start.bat config\zookeeper.properties
Keep this terminal open.

Step 3: Start Kafka Server

Open a new terminal:

cd C:\kafka bin\windows\kafka-server-start.bat config\server.properties
Kafka is now running on localhost:9092.

Part 2: Kafka Commands

Create a Topic

bin\windows\kafka-topics.bat --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

Check the Topic

bin\windows\kafka-topics.bat --list --bootstrap-server localhost:9092

Part 3: Spring Boot Kafka Project

Step 1: Create a Spring Boot Project

You can use Spring Initializr with the following settings:

Dependencies: Spring Web, Spring for Apache Kafka

Name: kafka-demo

Package: com.example.kafkademo

Project Structure

src/
└── main/
├── java/com/example/kafkademo/
│ ├── KafkaDemoApplication.java
│ ├── config/KafkaConfig.java
│ ├── controller/MessageController.java
│ ├── service/KafkaProducerService.java
│ └── listener/KafkaConsumerListener.java
└── resources/
└── application.yml
🛠 application.yml
spring: kafka: bootstrap-servers: localhost:9092 consumer: group-id: my-group auto-offset-reset: earliest key-deserializer: org.apache.kafka.common.serialization.StringDeserializer value-deserializer: org.apache.kafka.common.serialization.StringDeserializer producer: key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.apache.kafka.common.serialization.StringSerializer

KafkaProducerService.java

@Service
public class KafkaProducerService {
    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    public void sendMessage(String message) {
        kafkaTemplate.send("test-topic", message);
    }
}

KafkaConsumerListener.java

@Component
public class KafkaConsumerListener {
    @KafkaListener(topics = "test-topic", groupId = "my-group")
    public void listen(String message) {
        System.out.println("Received Message: " + message);
    }
}

MessageController.java

@RestController
@RequestMapping("/api/messages")
public class MessageController {

    @Autowired
    private KafkaProducerService producerService;

    @PostMapping
    public ResponseEntity<String> sendMessage(@RequestBody String message) {
        producerService.sendMessage(message);
        return ResponseEntity.ok("Message sent to Kafka");
    }
}

Main Application

@SpringBootApplication
public class KafkaDemoApplication {
    public static void main(String[] args) {
        SpringApplication.run(KafkaDemoApplication.class, args);
    }
}

Part 4: Run and Test

Run the App

./mvnw spring-boot:run

Test with Postman or Curl

curl -X POST http://localhost:8080/api/messages -H "Content-Type: text/plain" -d "Hello Kafka!"## Output
Your consumer will show:

Received Message: Hello Kafka!

Part 5: Clean Up

Stop the Spring Boot app --> Ctrl+C

Stop Kafka and Zookeeper --> Ctrl+C (Ctrl+C in each terminal)

Conclusion

You’ve just created a complete Kafka setup with:

A running Kafka instance on Windows

A Spring Boot REST API to send messages

A Kafka listener to consume them

Kafka MQ

Nanditha Vuppunuthula — Thu, 12 Jun 2025 18:28:31 +0000

What is Kafka?

Kafka is a publish/subscribe (pub/sub) messaging system that provides data streaming capabilities while also taking advantage of distributed computing.

What is a pub/sub messaging system?

A pub/sub messaging system contains two components that relay some form of data or information between each other. One component publishes data while the other component subscribes to the publisher to receive the published data.

Kafka follows this pattern with its own set of components and features.

Producers

The first component in a pub/sub messaging system is the publisher which is referred to as a Producer in Kafka. The producer is a data source that publishes or produces a message into Kafka. One of the great features of Kafka is that it is data type independent. This means that Kafka does not care about what type of data is being produced, whether it’s the GPS signal of a car, application metrics from front-end servers, or even images!

Consumers

The second component in a pub/sub messaging system is the subscriber, which is referred to as a Consumer in Kafka. The consumer can subscribe or listen to a data stream and consume messages from that stream while having no relationship or knowledge about the producers.

Consumers can subscribe to multiple streams of data regardless of the type of data being consumed. In other words, you can have a single application that takes in data from as many different sources as you’d like. Kafka makes it easy to access the data you need while leaving the processing steps entirely in your control.

High-level Architecture

Now that you know where messages come from (producers) and how messages can be retrieved (consumers), let’s discuss what happens in between.

Simple Kafka flow
Lets imagine a simple Kafka flow with three producers and two consumers. Each producer must specify a destination for the message and each consumer must specify from where it needs to consume. This middle ground between the producer and consumer, where the Kafka message is stored, is called a Topic.

Topics, Partitions, and Offsets

Topics can be thought of as a table in a database, where producers can write to and where consumers can read from. Each topic contains Partitions which are essentially logs that commit and append Kafka messages as they arrive. To identify messages, partitions use an auto-incrementing integer called an Offset, which is unique within partitions.

Offsets provide consumers the flexibility of reading messages when and from where they want, this is done by committing the offset. A commit from a consumer is like checking items on a list, once a message has been consumed, the commit tells Kafka to mark that offset as processed for that consumer.

As a consumer, you have the ability to read partitions from a specified offset or from the last committed message. How can this be useful? Consider an application that receives data every 2 hours. In this case, having the application continuously running and waiting for messages can be very expensive. By reading from the last committed message, you could have the application go live every 8 hours to simply consume all new messages and commit the offset of the latest message. This can reduce costs and usage of resources significantly.

Brokers and Clusters

Now, you know about producers, consumers, and how Kafka messages flow within Kafka, but one of the most important components remain, the Kafka Broker. The broker is what ties the whole system together; it is the Kafka server that is responsible for dealing with all communications involving producers, consumers, and even other brokers. Producers rely on the broker to correctly accept and store the incoming Kafka message to its appropriate topic. Consumers rely on the broker to handle their fetch and commit requests while consuming from topics.

A group of brokers is called a Kafka Cluster. One of the biggest perks of using Kafka is its use of distributed computing. A distributed system shares its workload among many other computers called nodes. These nodes all work together and communicate to complete the work rather than having all the work assigned to a single node. When we have multiple Kafka brokers and clusters dealing with large amounts of data, distributed computing saves resources and increases overall performance; making Kafka a desirable choice for big data applications.

Benefits of Apache Kafka

There are four key benefits of using Kafka:

Reliability: Kafka distributes, replicates, and partitions data. Additionally, Kafka is fault-tolerant.
Scalability: Kafka’s design allows you to handle enormous volumes of data. And it can scale up without any downtime.
Durability: Messages received are persisted as quickly as possible to storage. So, we can say Kafka is durable.
Performance: Finally, Kafka maintains the same level of performance even under extreme loads of data (many terabytes of message data). Kafka can perform up to two million writes per second. So, you can see that Kafka can store large amounts of data with zero downtime and no data loss.

Disadvantages of Apache Kafka

After discussing the advantages, let’s take a look at the disadvantages:

Limited flexibility: Kafka doesn’t support rich queries. For example, it’s not possible to filter for specific asset data in messages. (Functions like this are the responsibility of the consumer application reading the messages.) With Kafka, you can simply retrieve messages from a particular offset. The messages will be ordered as Kafka received them from the message producer. Not designed for holding historical data: Kafka is great for streaming data, but the design doesn’t allow you to store historical data inside Kafka for more than a few hours. Additionally, data is duplicated, which means storage can quickly become expensive for large amounts of data. You should use Kafka as transient storage where data gets consumed as quickly as possible.
No wildcard topic support: Last on the list of disadvantages is that it’s not possible to consume from multiple topics with one consumer. If you want to consume, for example, both log-2019-01 and log-2019-02, it’s not possible to use a wildcard for topic selection like log-2019-*. The above disadvantages are design limitations intended to improve Kafka’s performance. For some use cases that expect more flexibility, the above limitations can constrain an application consuming from Kafka.

Summary

Kafka is a great tool when it comes to handling and processing data especially with big data applications. It's a reliable platform that provides low-latency and high throughput with its data-streaming capabilities and gives an ample amount of helpful features and services to make your application better.

Mastering Java Streams API: A Modern Way to Process Data

Nanditha Vuppunuthula — Mon, 02 Jun 2025 18:29:42 +0000

If you're tired of writing clunky for-loops to manipulate collections in Java, then the Streams API is your best friend. Introduced in Java 8, the Stream API brings functional programming to Java, making your code more expressive, readable, and concise.

In this blog post, we’ll break down what streams are, how they work, and explore real-world examples to get you comfortable with this powerful tool.

What is a Stream?

A Stream in Java represents a sequence of elements supporting sequential and parallel aggregate operations. Think of it as a pipeline of data where you can apply a series of transformations and computations in a fluent, functional style.

⚠️ Streams don’t store data. They simply convey elements from a source (like a List) through a pipeline of operations.

Stream Operations: The Basics

Stream operations are typically chained and categorized into two types:

Intermediate Operations – Return another stream (e.g., map, filter, sorted)

Terminal Operations – Produce a result or side-effect (e.g., collect, forEach, reduce)

Here’s the general syntax:

streamSource
    .intermediateOperation()
    .intermediateOperation()
    .terminalOperation();

Practical Examples

1. Filtering a List

List<String> names = Arrays.asList("Alice", "Bob", "Charlie", "David");
List<String> filtered = names.stream()
                             .filter(name -> name.startsWith("A"))
                             .collect(Collectors.toList());

System.out.println(filtered); // Output: [Alice]

2. Mapping Elements (Transforming)

List<String> lowerCase = Arrays.asList("apple", "banana");
List<String> upperCase = lowerCase.stream()
                                  .map(String::toUpperCase)
                                  .collect(Collectors.toList());

System.out.println(upperCase); // Output: [APPLE, BANANA]

3. Sorting Elements

List<Integer> nums = Arrays.asList(5, 3, 8, 1);
List<Integer> sorted = nums.stream()
                           .sorted()
                           .collect(Collectors.toList());

System.out.println(sorted); // Output: [1, 3, 5, 8]

4. Reducing a List to a Single Value

List<Integer> nums = Arrays.asList(1, 2, 3, 4);
int sum = nums.stream()
              .reduce(0, Integer::sum);

System.out.println(sum); // Output: 10

5. Finding the First Element

Optional<String> first = Stream.of("cat", "dog", "elephant")
                               .findFirst();

first.ifPresent(System.out::println); // Output: cat

6. Counting Elements

long count = Stream.of("a", "b", "a", "c")
                   .filter(s -> s.equals("a"))
                   .count();

System.out.println(count); // Output: 2

7. Working with Custom Objects

class Product {
    String name;
    double price;
    Product(String name, double price) {
        this.name = name; this.price = price;
    }
}

List<Product> products = Arrays.asList(
    new Product("Laptop", 999.99),
    new Product("Phone", 499.99),
    new Product("Tablet", 299.99)
);

List<String> expensiveProducts = products.stream()
    .filter(p -> p.price > 300)
    .map(p -> p.name)
    .collect(Collectors.toList());

System.out.println(expensiveProducts); // Output: [Laptop, Phone]

Bonus: Parallel Streams

Want to speed up operations on large datasets? Use parallelStream:

List<Integer> bigList = IntStream.rangeClosed(1, 1_000_000)
                                 .boxed()
                                 .collect(Collectors.toList());

long sum = bigList.parallelStream()
                  .mapToLong(Integer::longValue)
                  .sum();

System.out.println(sum); // Output: 500000500000