Setup Kafka for development

Introduction

When developing applications using Apache Kafka, I had the need to make quick tests like publishing/consuming messages, check its format or just observe what was inside a topic.

In this article I will show how I setup a local productive environment for Apache Kafka development using Docker. There's a demo application in Github.

Install dependencies

To run our local Kafka and its tools locally we need to install:

• Docker CE
• Docker Compose

If you are on Mac OS or Windows you need to install also:

• Virtual Box
• Docker Machine

Docker image

For running Kafka, its dependencies and tools, I've chosen the fast-data-dev image, because it already contains Kafka, Zookeeper and Kafka Topics UI.

Exposing Kafka through Docker

Our Kafka instance will run outside Docker internal network, so we need to expose an IP address for our application connection. If you're on Linux you can use the localhost address directly.

The chosen Docker image mentionend above already contains the steps necessary to create the docker-machine and set the environment variables:

docker-machine create --driver virtualbox --virtualbox-memory 4096 landoop

eval $(docker-machine env landoop)

By running docker-machine ip you can see the machine's IP (usually 192.168.99.100).

Docker compose

I like to use docker-compose files to register docker commands on a git repository so I don't need to type and remember what to type everytime I want to run a docker image.

For example, instead of running:

docker run --rm -p 2181:2181 -p 3030:3030 -p 8081-8083:8081-8083 \
       -p 9581-9585:9581-9585 -p 9092:9092 -e ADV_HOST=192.168.99.100 \
       landoop/fast-data-dev:latest

I just need to create a file called docker-compose.yml containing:

version: '3'
services:
  kafka:
    image: landoop/fast-data-dev:latest
    ports:
      - "9092:9092"
      - "8081:8081"
      - "8082:8082"
      - "8083:8083"
      - "2181:2181"
      - "3030:3030"
      - "9581-9585:9581-9585"
    environment:
      - ADV_HOST=192.168.99.100 // Use your 'docker-machine ip' or 'localhost' if linux

And run it by:

docker-compose up

Test

I've provided a sample application to test the Kafka connection using the same docker configuration above. It's written in Java using Spring Kafka project, but it should work for any language or library. You can also use the command line interface to publish simple messages.

After running the demo application provided, you just need to access the endpoint http://localhost:8080/publish?message=sample_message to publish a message with content sample_message in test topic.

Now to check your topic messages, you can access the address where Kafka Topics UI is running on address <docker-ip>:3030/kafka-topics-ui (i.e.: http://192.168.99.100:3030/kafka-topics-ui/):

Conclusion

With this simple setup and a single docker-compose.yml file, you are ready to publish and consume Kafka messages and start developing your application.