Connecting Kafka to Spring Boot: A Step-by-Step Guide Using KafkaTemplate

Hello, Tech enthusiasts! Today, we're going to learn how to connect to one of the most widely used distributed streaming platforms, Kafka, from Spring Boot. This will be a brief write-up as we'll be utilizing KafkaTemplate, which makes our lives easier. Let's dive in.

If you like to learn in action, below is the video version that goes over the implementation.

Overview

We will create a Rest endpoint that will be responsible for posting the message into the Kafka topic and the consumer who is also part of the same project listening to the topic will print the received message on the console.

Project Set-up

Let's create a spring boot application with the following dependencies.
Add "spring-boot-starter-web" for the rest endpoint creation.

<dependency>             
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
</dependency>

Here comes the Kafka dependency will do the magic of producing & consuming from Kafka via KafkaTemplate.

<dependency>
        <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

As the next step will add below configuration to establish the connection with Kafka broker. Spring boot will auto-configure the KafkaTemplate with these properties.

Make sure to update the server and group-id below.

spring:
  kafka:
    consumer:
      bootstrap-servers:
        - localhost:9092
      group-id: ytube-group
#      auto-offset-reset: earliest
#      key-deserializer:
#      value-deserializer:
    producer:
      bootstrap-servers:
        - localhost:9092

Consumer Configuration:

• bootstrap-servers: Specifies the Kafka server(s) that the consumer will connect to. In this case, it's set to a single server at localhost:9092.

• group-id: Identifies the consumer group to which this consumer belongs. It helps Kafka keep track of the progress of different consumer groups.

• auto-offset-reset: (Commented out) This property determines what happens when there is no initial offset or the current offset does not exist. The earliest option means it will start reading from the beginning of the topic.

• key-deserializer and value-deserializer: (Commented out) These properties define the deserializer classes for the key and value of the messages. Depending on your use case, you might need to specify appropriate deserializer classes.

Producer Configuration:

• bootstrap-servers: Similar to the consumer, this property specifies the Kafka server(s) that the producer will connect to. In this case, it's also set to a single server at localhost:9092.

KafkaMessageProducer

With KafkaTemplate, it's as simple as writing a letter. Create a KafkaProducer service, autowire KafkaTemplate, and start sending messages.

@Component
public class KafkaMessageProducer {

    @Autowired
    KafkaTemplate<String, String> kafkaTemplate;

    public void sendMessage(String topic, String message){
        kafkaTemplate.send(topic, message);
    }
}

KafkaMessageConsumer

Of course, every good story has two sides. Create a KafkaConsumer service to catch the messages on the other end. KafkaTemplate sends, and KafkaConsumer receives – it's the yin and yang of our messaging world.

We configure the consumer with the help of KafkaListener.

@Component
public class MessageConsumer {

    @KafkaListener(topics = "ytube-topic1", groupId = "ytube-group")
    public void consumerMessage(String message){
        System.out.println("Received Message "+ message);
    }
}

Let's make the rest endpoint to post the message into the Kafka topic.

We have autowired the kafka message producer and passed the topic name and message.

    @Autowired
    KafkaMessageProducer messageProducer;

    @PostMapping("/sendMessage")
    public String sendMessage(@RequestParam("message") String message){
        messageProducer.sendMessage("ytube-topic1", message);
        return "Message has been sent successfully "+message;
    }

Let's execute the application and see the message posted to Kafka topic and received by the consumer. In real world, the consumer can be another microservice that consumes the message to process it further.

Kudos, you made it to the end! Thanks for hanging out with us. If you have more questions or just enjoyed the ride, come back soon. Happy exploring!

Checkout the completed code here

amaialth / kafkademo

Demo/Starter project to connect Spring boot with Kafka

Simple Kafka Demo

The application demonstrates the use of KafkaTemplate. We will use Kafka template to produce and consume the message.

View on GitHub