DEV Community: abharan gupta

Kafka 4.0 Unleashed: How to Upgrade and Thrive in a ZooKeeper-Free World

abharan gupta — Mon, 21 Apr 2025 08:24:23 +0000

Understanding the Differences Between Older Kafka Versions and Kafka 4.0 — And How to Upgrade Smoothly

Apache Kafka has been a game-changer in the world of real-time data streaming for years. But with the release of Kafka 4.0, there are some big changes that everyone should know about—especially if you’re running older versions. Whether you’re a developer, a DevOps engineer, or a tech enthusiast, this blog will break down what’s new, what’s gone, and how to prepare your Kafka cluster for the upgrade.

What’s New in Kafka 4.0? The Big Picture

Kafka 4.0 isn’t just another update; it’s a major milestone. The biggest headline is that Kafka no longer uses ZooKeeper for managing cluster metadata and coordination. This is a huge architectural change that simplifies Kafka’s operation but also means the upgrade path requires some planning.

Let’s walk through the main differences between older Kafka versions (like 2.x or 3.x) and Kafka 4.0 in simple terms.

1. Goodbye ZooKeeper, Hello KRaft!

Older Versions:
Kafka used ZooKeeper as a separate system to keep track of cluster metadata — things like which brokers are alive, topic configurations, and partition leadership. This meant you had to run and manage a ZooKeeper ensemble alongside your Kafka brokers.

Kafka 4.0:
ZooKeeper is completely removed. Kafka now uses a built-in consensus system called KRaft (Kafka Raft Metadata mode) to handle all metadata internally. This means fewer moving parts, easier setup, and faster cluster management.

Why it matters:
No more ZooKeeper means less complexity and fewer components to maintain. But it also means you can’t just upgrade Kafka and keep ZooKeeper running—you need to migrate your cluster metadata to KRaft first.

2. Smarter Consumer Rebalancing

Older Versions:
When Kafka consumers (applications that read data) join or leave a group, Kafka pauses message consumption during rebalancing. This could cause noticeable delays or downtime, especially in large clusters.

Kafka 4.0:
Introduces a new consumer group protocol (called KIP-848) that makes rebalancing faster and less disruptive. Consumers can continue processing most partitions even while the group is rebalancing.

Why it matters:
Your streaming applications will experience fewer interruptions and better performance during scaling or recovery events.

3. Updated Java Requirements

Older Versions:
Kafka brokers and clients supported Java 8 and Java 11, which were the industry standards for many years.

Kafka 4.0:
Requires Java 17 for brokers, Connect, and tools, and Java 11 for clients and Kafka Streams applications.

Why it matters:
You’ll need to upgrade your Java runtime environments before moving to Kafka 4.0. This may require coordination with your infrastructure or development teams.

4. MirrorMaker 1 Is Gone

Older Versions:
MirrorMaker 1 was a basic tool for replicating data between Kafka clusters.

Kafka 4.0:
MirrorMaker 1 has been removed. Only MirrorMaker 2 remains, which is more robust, scalable, and easier to manage.

Why it matters:
If you’re still using MirrorMaker 1, you’ll need to migrate to MirrorMaker 2 before upgrading.

5. Logging Changes

Older Versions:
Kafka used Log4j for logging.

Kafka 4.0:
Kafka has fully switched to Log4j2, which is more secure and modern.

Why it matters:
You may need to update your logging configurations and tools.

6. API Clean-Up
Kafka 4.0 removes APIs that have been deprecated for over a year. This helps keep the codebase clean and encourages users to adopt the latest, supported features.

7. Early Access to Queues for Kafka
Kafka traditionally uses a publish-subscribe messaging model. Kafka 4.0 introduces early access to queue semantics (via KIP-932), enabling point-to-point messaging patterns.

This is exciting for use cases that require strict message ordering and processing guarantees.

How to Upgrade to Kafka 4.0: A Step-by-Step Guide

Upgrading to Kafka 4.0 isn’t as simple as flipping a switch—especially because of the ZooKeeper removal. Here’s a clear, step-by-step approach to make your upgrade smooth and safe.

Step 1: Check Your Current Setup and Prepare
Know your Kafka version: If you’re running anything older than 3.9.x, you’ll first need to upgrade to 3.9.x before moving to 4.0.

Check Java versions: Make sure your brokers and clients can run on the required Java versions (Java 17 for brokers, Java 11 for clients).

Backup everything: Always back up your Kafka data, configurations, and metadata before starting the upgrade.

Test in a staging environment: Don’t upgrade your production cluster first. Run tests on a staging or dev cluster.

Step 2: Upgrade to Kafka 3.9.x (The Bridge Version)
Kafka 3.9.x supports both ZooKeeper and KRaft modes and includes tools to migrate metadata from ZooKeeper to KRaft.

Upgrade your brokers to 3.9.x.

Make sure everything is stable and running as expected.

Start planning your migration to KRaft.

Step 3: Migrate from ZooKeeper to KRaft
This is the most critical step.

Use Kafka’s official migration tools and follow the detailed migration guide.

The migration process moves all cluster metadata from ZooKeeper into Kafka’s internal KRaft quorum.

Validate that your cluster is healthy and all topics, partitions, and configurations are intact after migration.

Step 4: Upgrade to Kafka 4.0
Once your cluster is running in KRaft mode on 3.9.x:

Upgrade your brokers to Kafka 4.0.

Upgrade clients and tools, making sure they’re compatible with Kafka 4.0 and the new Java versions.

Monitor your cluster closely for any issues.

Step 5: Post-Upgrade Cleanup and Optimization
Remove any deprecated APIs or features.

Switch from MirrorMaker 1 to MirrorMaker 2 if you haven’t already.

Update your logging configurations to use Log4j2.

Explore new features like improved consumer rebalancing and queue semantics.

Final Thoughts

Kafka 4.0 is a powerful, modernized release that simplifies cluster management and boosts performance. But because it removes ZooKeeper and updates core dependencies, it requires a thoughtful upgrade plan.

By following the steps above—starting with upgrading to Kafka 3.9.x, migrating to KRaft, and then moving to Kafka 4.0—you’ll ensure a smooth transition with minimal downtime.

If you’re running Kafka in production, don’t rush the upgrade. Test thoroughly, involve your teams, and take advantage of Kafka’s detailed documentation and tools.

Kafka 4.0 is the future of streaming platforms—embrace it to unlock new possibilities and keep your data pipelines running smoothly!

Have you started your Kafka 4.0 upgrade? Feel free to share your experiences or ask questions in the comments below!

Dive into Jackson for JSON in Java: Understanding JsonNode, ArrayNode, and ObjectMapper

abharan gupta — Tue, 22 Oct 2024 11:53:24 +0000

Hey, fellow Java devs! 🚀

Ever find yourself staring at JSON data and thinking, "How on earth do I work with this in Java?" Don't worry - you're not alone! Whether it's building APIs, consuming them, or just handling data, dealing with JSON is almost inevitable. But here's the good news: Jackson has your back!

In this article, I'm going to walk you through some Jackson basics, like JsonNode, ArrayNode, and the ObjectMapper. We'll keep it simple, with easy code examples and outputs to show you exactly how things work.

Sound good? Let's dive in! 🔍

Setting Up Jackson in a Spring Boot Project

Before we dive into the examples, let's quickly cover how to set up Jackson in a Spring Boot project. Good news: Spring Boot has Jackson built-in, so there's minimal setup required. 🎉

When you create a new Spring Boot project, Jackson comes as the default JSON library for serialization and deserialization. If you want to explicitly add Jackson, ensure the following dependency is in your pom.xml:

<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.18.0</version> <!-- You can use the latest version -->
</dependency>

This will add Jackson's core functionality, including ObjectMapper, for JSON handling.

Bonus: Spring Boot Configuration
Spring Boot provides an out-of-the-box setup for Jackson, but you can also customize it via the application.properties or application.yml file.

For example, to configure pretty-printing of JSON, you can add:

spring.jackson.serialization.indent_output=true

Or in application.yml:

spring:
  jackson:
    serialization:
      indent_output: true

Now, whenever your Spring Boot app serializes JSON, it will be nicely formatted!

With this setup done, you're ready to work with JSON in your Spring Boot app using Jackson.

So, What Is Jackson?

Jackson is like a Swiss Army knife for working with JSON in Java. You can use it to:

1. Convert Java objects to JSON (serialization).
2. Convert JSON to Java objects (deserialization).
3. Handle JSON in a tree-like structure with JsonNode.

We're going to explore some of these features today, so get ready to make JSON handling feel a lot less scary!

JsonNode: Your First Step into JSON

Think of JsonNode as a magical key that lets you explore and manipulate JSON data. It's a way of representing different parts of a JSON structure in Java.

Imagine you've got this simple JSON data:

{
  "name": "Samarth",
  "age": 30,
  "city": "New York"
}

How do you read this in Java? Let's see!

Here's the code:

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;

public class JsonNodeExample {
    public static void main(String[] args) throws Exception {
        String jsonString = "{\"name\":\"Samarth\", \"age\":30, \"city\":\"New York\"}";

        // Step 1: Create an ObjectMapper
        ObjectMapper objectMapper = new ObjectMapper();

        // Step 2: Parse the JSON string into a JsonNode
        JsonNode jsonNode = objectMapper.readTree(jsonString);

        // Step 3: Access values from the JsonNode
        System.out.println("Name: " + jsonNode.get("name").asText());
        System.out.println("Age: " + jsonNode.get("age").asInt());
        System.out.println("City: " + jsonNode.get("city").asText());
    }
}

And the output:

Name: Samarth  
Age: 30  
City: New York

What's happening here?

ObjectMapper is Jackson's main helper. It's the one that translates between JSON and Java.
readTree() reads the JSON and converts it into a JsonNode object.
We use .get() to access the individual fields-"name", "age", and "city"-from the JSON.

Pretty cool, right? Now you're starting to see how easy it is to work with JSON in Java!

ArrayNode: Handling JSON Arrays

But what if your JSON is an array? Don't worry, Jackson has that covered too! Let's say you have this JSON array:

[
  {"name": "Samarth"},
  {"name": "Ujjwal"},
  {"name": "Gaurav"}
]

We can use ArrayNode to read and work with this array of objects.

Here's the code:

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ArrayNode;

public class ArrayNodeExample {
    public static void main(String[] args) throws Exception {
        String jsonArrayString = "[{\"name\":\"Samarth\"}, {\"name\":\"Ujjwal\"}, {\"name\":\"Gaurav\"}]";

        // Step 1: Create an ObjectMapper
        ObjectMapper objectMapper = new ObjectMapper();

        // Step 2: Parse the JSON array into an ArrayNode
        ArrayNode arrayNode = (ArrayNode) objectMapper.readTree(jsonArrayString);

        // Step 3: Loop through each element in the array
        for (JsonNode jsonNode : arrayNode) {
            System.out.println("Name: " + jsonNode.get("name").asText());
        }
    }
}

And the output:

Name: Samarth
Name: Ujjwal
Name: Gaurav

What's happening here?

ArrayNode is a special type of JsonNode that represents an array of JSON objects.
We loop through each element in the array and print out the "name" of each person.

Easy, right? With ArrayNode, Jackson makes handling JSON arrays a breeze!

ObjectMapper: The Heart of Jackson

Now, let's talk about ObjectMapper - the heart and soul of Jackson. It's your go-to tool for converting Java objects to JSON and vice versa.

Serializing Java Objects to JSON
Serialization is just a fancy way of saying, "turn my Java object into a JSON string." Let's say you have a simple Personclass:

Code:

import com.fasterxml.jackson.databind.ObjectMapper;

public class SerializationExample {
    public static void main(String[] args) throws Exception {
        ObjectMapper objectMapper = new ObjectMapper();

        // Step 1: Create a Java object
        Person person = new Person("Samarth", 30, "New York");

        // Step 2: Convert the Java object to JSON (serialization)
        String jsonString = objectMapper.writeValueAsString(person);

        // Output the JSON
        System.out.println("Serialized JSON: " + jsonString);
    }
}

class Person {
    private String name;
    private int age;
    private String city;

    public Person(String name, int age, String city) {
        this.name = name;
        this.age = age;
        this.city = city;
    }

    public String getName() { return name; }
    public int getAge() { return age; }
    public String getCity() { return city; }
}

Output:

Serialized JSON: {"name":"Samarth","age":30,"city":"New York"}

What's happening here?

ObjectMapper takes the Personobject and converts it into a JSON string using writeValueAsString().
The method writeValueAsString() creates a JSON representation of the Java object.
The result is a valid JSON string you can send to an API or store in a database.

Deserializing JSON to a Java Object

And it works the other way around too! You can take JSON and turn it back into a Java object. This is called deserialization.

Here's the code:

import com.fasterxml.jackson.databind.ObjectMapper;

public class DeserializationExample {
    public static void main(String[] args) throws Exception {
        String jsonString = "{\"name\":\"Samarth\", \"age\":30, \"city\":\"New York\"}";

        ObjectMapper objectMapper = new ObjectMapper();

        // Step 1: Convert the JSON string to a Java object (deserialization)
        Person person = objectMapper.readValue(jsonString, Person.class);

        // Output the Java object
        System.out.println("Deserialized Object: " + person.getName() + ", " + person.getAge() + ", " + person.getCity());
    }
}

And the output:

Deserialized Object: Samarth, 30, New York

What's happening here?

We use ObjectMapper again, but this time it reads a JSON string and converts it back into a Person object.
This is done using readValue(), and the result is a full Java object ready to be used in your code.

Wrapping Up

And there you have it! We've covered a lot of ground:

JsonNode: How to read and manipulate JSON data.
ArrayNode: How to handle JSON arrays.
ObjectMapper: How to serialize and deserialize Java objects to and from JSON.

I hope this guide makes Jackson a little less intimidating and a lot more fun to use! Once you get the hang of it, you'll be handling JSON like a pro in no time.

But hey, don't stop here! Keep an eye out for my next article where we'll dive deeper into more advanced Jackson features and best practices for real-world applications.

See you next time! Happy coding! 😄