DEV Community

Cover image for Getting Started With Apache Kafka
Clivern
Clivern

Posted on

12 1

Getting Started With Apache Kafka

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

Let's first run a single kafka node with docker and docker-compose.

  • Create a docker-compose.yaml file
---
version: '3'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.1.1
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-server:6.1.1
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:29092
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'true'
      CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'

Enter fullscreen mode Exit fullscreen mode
  • After installing docker and docker-compose, Start kafka and zookeeper containers with the following command
$ docker-compose up -d
Enter fullscreen mode Exit fullscreen mode
  • Get kafka container id to run some commands inside the container.
$ docker ps

$ docker exec -it $ID bash
Enter fullscreen mode Exit fullscreen mode

To create a kafka topic clivern

$ cd /bin
$ kafka-topics --create \
  --bootstrap-server localhost:9092 \
  --topic clivern
Enter fullscreen mode Exit fullscreen mode

To run a consumer

$ cd /bin
$ kafka-console-consumer \
    --bootstrap-server localhost:9092 \
    --topic clivern
Enter fullscreen mode Exit fullscreen mode

To run a producer

$ cd /bin
$ kafka-console-producer \
    --bootstrap-server localhost:9092 \
    --topic clivern
Enter fullscreen mode Exit fullscreen mode

To list all topics

$ cd /bin
$ kafka-topics --bootstrap-server localhost:9092 --list
Enter fullscreen mode Exit fullscreen mode

Sentry image

See why 4M developers consider Sentry, “not bad.”

Fixing code doesn’t have to be the worst part of your day. Learn how Sentry can help.

Learn more

Top comments (0)

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs