DEV Community

Cover image for Beautify kcat consumer output by piping to jq
Francesco Tisiot
Francesco Tisiot

Posted on • Originally published at ftisiot.net

4 2

Beautify kcat consumer output by piping to jq

I was working on some demos recently on the Apache Kafka source connectors (hi #KafkaSummit!), and trying to display the stream of changes in the resulting Apache Kafka topic.

My standard approach is to vertically divide the terminal and have on the left the database connection where I can issue commands and on the right the output of kcat. This setup works really well for simple messages, but when using it with a Debezium source connector, the amount of fields pushed in a single message makes it hardly parsable by a live audience watching a screen.

Unparsable JSON

In the above image you can notice that the kcat output is a JSON format without syntax highlight making it almost unreadable.

jq to the rescue

I knew and was using since long time jq a tool that allows you to beautify and parse JSON outputs. After installing you can just use it as

echo '{"name":"Francesco", "preferences":["pasta","pizza","espresso"]}' | jq '.'
Enter fullscreen mode Exit fullscreen mode

and you get your amazing JSON displayed nicely

{
  "name": "Francesco",
  "preferences": [
    "pasta",
    "pizza",
    "espresso"
  ]
}
Enter fullscreen mode Exit fullscreen mode

jq and kcat

Now, how can I apply jq to the kcat consumer output? I tried a direct pipe like:

kcat -F kcat.config -C \
    -t my_pgsource.public.players | jq 
Enter fullscreen mode Exit fullscreen mode

But unfortunately I only got the output comments without the actual data being displayed

% Reading configuration from file kcat.config
% Reached end of topic my_pgsource.public.players [0] at offset 3
Enter fullscreen mode Exit fullscreen mode

Seen on screen as

No output of JSON

I struggled long time, but then reached out to a friend (kudos @rmoff) who pointed me to the right solution: the -u flag ( Unbuffered output) in kcat. Once added the -u flag I'm now able to see my beautified JSON!

Beautified JSON

AWS Security LIVE!

Tune in for AWS Security LIVE!

Join AWS Security LIVE! for expert insights and actionable tips to protect your organization and keep security teams prepared.

Learn More

Top comments (1)

Collapse
 
milkcoke profile image
milkcoke

You saved me bro!
-u flag for streaming!

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs

👋 Kindness is contagious

Dive into an ocean of knowledge with this thought-provoking post, revered deeply within the supportive DEV Community. Developers of all levels are welcome to join and enhance our collective intelligence.

Saying a simple "thank you" can brighten someone's day. Share your gratitude in the comments below!

On DEV, sharing ideas eases our path and fortifies our community connections. Found this helpful? Sending a quick thanks to the author can be profoundly valued.

Okay