When developing applications using Apache Kafka, I had the need to make quick tests like publishing/consuming messages, check its format or just observe what was inside a topic.
In this article I will show how I setup a local productive environment for Apache Kafka development using Docker. There's a demo application in Github.
To run our local Kafka and its tools locally we need to install:
If you are on Mac OS or Windows you need to install also:
For running Kafka, its dependencies and tools, I've chosen the fast-data-dev image, because it already contains Kafka, Zookeeper and Kafka Topics UI.
Our Kafka instance will run outside Docker internal network, so we need to expose an IP address for our application connection. If you're on Linux you can use the
localhost address directly.
The chosen Docker image mentionend above already contains the steps necessary to create the
docker-machine and set the environment variables:
docker-machine create --driver virtualbox --virtualbox-memory 4096 landoop
eval $(docker-machine env landoop)
docker-machine ip you can see the machine's IP (usually 192.168.99.100).
I like to use docker-compose files to register
docker commands on a
git repository so I don't need to type and remember what to type everytime I want to run a docker image.
For example, instead of running:
docker run --rm -p 2181:2181 -p 3030:3030 -p 8081-8083:8081-8083 \ -p 9581-9585:9581-9585 -p 9092:9092 -e ADV_HOST=192.168.99.100 \ landoop/fast-data-dev:latest
I just need to create a file called
version: '3' services: kafka: image: landoop/fast-data-dev:latest ports: - "9092:9092" - "8081:8081" - "8082:8082" - "8083:8083" - "2181:2181" - "3030:3030" - "9581-9585:9581-9585" environment: - ADV_HOST=192.168.99.100 // Use your 'docker-machine ip' or 'localhost' if linux
And run it by:
I've provided a sample application to test the Kafka connection using the same docker configuration above. It's written in Java using Spring Kafka project, but it should work for any language or library. You can also use the command line interface to publish simple messages.
After running the demo application provided, you just need to access the endpoint http://localhost:8080/publish?message=sample_message to publish a message with content
With this simple setup and a single
docker-compose.yml file, you are ready to publish and consume Kafka messages and start developing your application.