DEV Community

eidher
eidher

Posted on

2

Kafka Connect: FileStreamSourceConnector in distributed mode

1) Source connectors: Start our kafka cluster. This is the docker-compose.yml file:

version: '2'

services:

  kafka-cluster:
    image: landoop/fast-data-dev:cp3.3.0
    environment:
      ADV_HOST: 127.0.0.1
      RUNTESTS: 0
    ports:
      - 2181:2181                 # Zookeeper
      - 3030:3030                 # Landoop UI
      - 8081-8083:8081-8083       # REST Proxy, Schema Registry, Kafka Connect
      - 9581-9585:9581-9585       # JMX Ports
      - 9092:9092                 # Kafka Broker
Enter fullscreen mode Exit fullscreen mode

Create and start the container:

docker-compose up kafka-cluster
Enter fullscreen mode Exit fullscreen mode

2) Create the topic we're going to write to:

docker run --rm -it --net=host landoop/fast-data-dev:cp3.3.0 bash
kafka-topics --create --topic demo-2-distributed --partitions 3 --replication-factor 1 --zookeeper 127.0.0.1:2181
Enter fullscreen mode Exit fullscreen mode

3) In a browser go to 127.0.0.1:3030 -> Connect UI
Create a new connector -> File Source
Paste the next configuration and click the Create button:

name=file-stream-demo-distributed
connector.class=org.apache.kafka.connect.file.FileStreamSourceConnector
tasks.max=1
file=demo-file.txt
topic=demo-distributed
key.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=true
Enter fullscreen mode Exit fullscreen mode

4) Create the source file:
Execute docker ps to see the containerId.

docker exec -it <containerId> bash
touch demo-file.txt
echo "add any content" >> demo-file.txt
Enter fullscreen mode Exit fullscreen mode

3) In a browser go to 127.0.0.1:3030 -> Kafka Topics UI
Click the topic defined previously (demo-distributed) and validate the content:

Image description

Image of Datadog

Create and maintain end-to-end frontend tests

Learn best practices on creating frontend tests, testing on-premise apps, integrating tests into your CI/CD pipeline, and using Datadog’s testing tunnel.

Download The Guide

Top comments (0)

Postmark Image

Speedy emails, satisfied customers

Are delayed transactional emails costing you user satisfaction? Postmark delivers your emails almost instantly, keeping your customers happy and connected.

Sign up