🎯 Sending flat files to Kafka

Hello 🖐️ !

Following the mechanisms described here to send CSV files into Kafka in JSON using kcat through shell pipelines, we packed 📦 all of that in a new home-made tool named producer-file. Here are the features :

• 💨 Sending line by line
• 💪 CSV to JSON conversion
• 🔦 Support message headers by file and/or by line
• ⚡️ Delta-changes mode (only send changes occurred after last send)

☝️ More about the shell way :

⌨️ Pipe xlsx files into/from Kafka... From cli with (k)cat 🙀

adriens for opt-nc ・ Oct 10 '22 ・ 2 min read

#tutorial #productivity #kafka #linux

Motivations

Using jq + kcat with shell pipelines do the job well, but this doesn't fit the company's requirements today about software packaging and logging capabilities 🤷🏻‍♂️.

📽️ Demo

Short video demonstrating how to use this tool to send a CSV file in JSON into Kafka within a Docker environnement :

📖 Ressources

• 🐳 producer-file
• 🥷 Killercoda about the "shell way"
• 🪶 Kafka

Comments

[adriens]

Here is a related and worhtwhile content :

opt-nc / atelier-spring-kafka

Projet d'exemple d'utilisation de la lib `org.springframework.kafka:spring-kafka`

❔ A propos

Ce projet explique comment développer un producer/consumer sans la lib interne historique au profit de celle développée et maintenue par Spring.

✋Pourquoi Kafka

📝 Description

Projet d'exemple d'utilisation de la lib org.springframework.kafka:spring-k...

✅ Pré-requis

• Instance kafka
• docker & docker-compose (si utilisation de kafka.yml)

👶 Kafka ?! WTF ❔

Avant de passer à la suite, prendre soin de consulter les ressources ci-dessous:

• 🎥 Comprendre ce qu'est Kafka en 6 minutes
• 📝 The Apache Kafka Handbook (freecodecamp)

🧑‍🎓 Supports custome

• ⌨️ Pipe xlsx files into/from Kafka... From cli with (k)cat 🙀

📜 Scripts

1️⃣ Initialiser le topic demo.user

Entrer dans le container kafka :

docker exec -ti kafka bash

Créer le producer :

kafka-console-producer.sh --broker-list kafka:9092 --topic demo.user --property "parse.key=true" --property "key.separator=|"

Copier les lignes :

112233|{"phoneNumber":"112233", "firstName":"Hubert", "lastName":"Bonisseur de la Bath"}
998877|{"phoneNumber":"998877", "firstName":"Jean", "lastName":"Soudajman"}
446655|{"phoneNumber":"446655", "firstName":"Henri", "lastName":"Tathan"}

2️⃣ Ecriture automatique dans le topic demo.sms

Initialiser…

View on GitHub

[adriens]

Adrien SALES

@rastadidi

📢 We have just released a new #tools for #productivity around #apachekafka 🎉
ℹ️ It comes in addition to #opensource #kafkacat (#kcat) #devops tool 🤓
👉 If you need to send file content into #kafka, this is for you 👇
dev.to/optnc/sending-…
#devopstools #devopsculture

20:36 PM - 03 May 2023