DEV Community

Cover image for 🎯 Sending flat files to Kafka
Daniel Santos for opt-nc

Posted on

5

🎯 Sending flat files to Kafka

Hello 🖐️ !

Following the mechanisms described here to send CSV files into Kafka in JSON using kcat through shell pipelines, we packed 📦 all of that in a new home-made tool named producer-file. Here are the features :

  • 💨 Sending line by line
  • 💪 CSV to JSON conversion
  • 🔦 Support message headers by file and/or by line
  • ⚡️ Delta-changes mode (only send changes occurred after last send)

☝️ More about the shell way :

Motivations

Using jq + kcat with shell pipelines do the job well, but this doesn't fit the company's requirements today about software packaging and logging capabilities 🤷🏻‍♂️.

📽️ Demo

Short video demonstrating how to use this tool to send a CSV file in JSON into Kafka within a Docker environnement :

📖 Ressources

AWS GenAI LIVE image

Real challenges. Real solutions. Real talk.

From technical discussions to philosophical debates, AWS and AWS Partners examine the impact and evolution of gen AI.

Learn more

Top comments (2)

Collapse
 
adriens profile image
adriens

Here is a related and worhtwhile content :

GitHub logo opt-nc / atelier-spring-kafka

Projet d'exemple d'utilisation de la lib `org.springframework.kafka:spring-kafka`

Project Website Confluent tutorial

A propos

Ce projet explique comment développer un producer/consumer sans la lib interne historique au profit de celle développée et maintenue par Spring.

Pourquoi Kafka

image

📝 Description

Projet d'exemple d'utilisation de la lib org.springframework.kafka:spring-k...

Pré-requis

  • Instance kafka
  • docker & docker-compose (si utilisation de kafka.yml)

👶 Kafka ?! WTF

Avant de passer à la suite, prendre soin de consulter les ressources ci-dessous:

🧑‍🎓 Supports custome

📜 Scripts

1️⃣ Initialiser le topic demo.user

Entrer dans le container kafka :

docker exec -ti kafka bash

Créer le producer :

kafka-console-producer.sh --broker-list kafka:9092 --topic demo.user --property "parse.key=true" --property "key.separator=|"

Copier les lignes :

112233|{"phoneNumber":"112233", "firstName":"Hubert", "lastName":"Bonisseur de la Bath"}
998877|{"phoneNumber":"998877", "firstName":"Jean", "lastName":"Soudajman"}
446655|{"phoneNumber":"446655", "firstName":"Henri", "lastName":"Tathan"}

2️⃣ Ecriture automatique dans le topic demo.sms

Initialiser

Collapse
 
adriens profile image
adriens

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs