This week we explored using redpanda as a drop in for kafka.
Using redpanda can be ran from a docker container.
Steps:
Copy the docker-compose.yml file into your install directory (this is a replica of what we used for the course).
version: '3.7'
services:
# Redpanda cluster
redpanda-1:
image: docker.redpanda.com/vectorized/redpanda:v22.3.5
container_name: redpanda-1
command:
- redpanda
- start
- --smp
- '1'
- --reserve-memory
- 0M
- --overprovisioned
- --node-id
- '1'
- --kafka-addr
- PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
- --advertise-kafka-addr
- PLAINTEXT://redpanda-1:29092,OUTSIDE://localhost:9092
- --pandaproxy-addr
- PLAINTEXT://0.0.0.0:28082,OUTSIDE://0.0.0.0:8082
- --advertise-pandaproxy-addr
- PLAINTEXT://redpanda-1:28082,OUTSIDE://localhost:8082
- --rpc-addr
- 0.0.0.0:33145
- --advertise-rpc-addr
- redpanda-1:33145
ports:
# - 8081:8081
- 8082:8082
- 9092:9092
- 28082:28082
- 29092:29092
Then run
docker-compose up
and voila!
References:
Data engineering zoomcamp week 6 course and homework notes:
https://github.com/DataTalksClub/data-engineering-zoomcamp/tree/main/cohorts/2024/06-streaming
Top comments (0)