Why I wrote Kafka Streamyard
I built Kafka Streamyard to make Kafka data investigations and manual testing less repetitive. It preserves context between topics and environments, keeps workflow-related topics and filters organised, combines related topics in Live Views, and supports reusable test-data samples and development playbooks. It is a local-first desktop app for macOS, Windows, and Linux.
I've been developing Kafka applications for about 10 years, and I started an early version of Kafka Streamyard about six or seven years ago. It is an Electron application built with React and a local Java backend.
After working closely with developers, testers, and DevOps engineers, I found that the Kafka UIs we used were generally designed around administration. They did not fit our day-to-day data investigation and manual testing workflows particularly well. Lots of individually small inconveniences added up to a significant amount of repeated work.
One example is simply finding all the topics involved in a workflow. In a busy cluster, I have spent far too much time searching for the same groups of topics again and again. Kafka Streamyard lets me create and save reusable topic-name filters using wildcards, nested Boolean logic, and some Kafka metadata. I can tag each filter for the connections where it is relevant and quickly switch between different parts of a workflow during testing, investigations, or team workshops. AI assistance is also available when I need help building a filter.
I have also put a lot of effort into preserving context. I can investigate something in UAT, switch to Dev, and later return with my selected topic, results, filter, and even the selected record still available. That context is restored after restarting the application too. Optional Cloud Sync lets me reuse filters, custom columns, Live Views, Data Samples, and Playbooks when moving between my laptop and a VM, while Kafka connections and credentials stay local.
Live Views are probably my favourite part when working with a team. They combine related topics into one chronological view, so everyone can watch a process move from a UI to a database, through CDC, and into the topics that trigger the rest of a workflow. Each topic can use its own custom columns, making it easier to confirm that the important fields were set correctly at each stage. A Live View can also include a session filter, message counts, and an optional flowchart showing the process and data flow.
For repetitive manual testing, Data Samples can keep related keys, headers, and messages together. Session variables can carry a correlation ID across several samples, increment sequence values, generate timestamps or identifiers, and reset everything for the next test. Samples can then be sent in a controlled order, in bursts, or with delays between them.
Message filters use SL, or Streamyard Language, which is yet another Java-like filter language to learn. That is why I added autocomplete and optional AI assistance. SL can filter the message, key, headers, and Kafka metadata, including fields inside JSON, XML, and embedded JSON. The AI-generated filter remains visible and editable, so it can be checked before use. Users provide their own API key and can choose from several common providers.
Playbooks help me reset a narrow part of a development or local Docker environment into a known state before testing again. They are line-based scripts that can create, configure, clear, delete, or copy topics and import test data. They support variables, repeat blocks, topic patterns, validation, Dry Run, a builder wizard, and AI assistance. They are intended for development and testing, not as a replacement for production CI/CD or Infrastructure as Code.
There are also precise offset and time controls for historical investigations, background Live Feeds for catching infrequent records, automatic format detection, Schema Registry support, a hex editor for raw bytes, custom calculated columns, and CSV export when somebody needs a report or evidence from a set of topics.
Kafka Streamyard is not intended to replace a typical multi-user Kafka management platform. It includes some familiar management features, but its main focus is working directly with Kafka data during investigations, testing sessions, support work, and team workshops.
The website is still a work in progress, but you can try Kafka Streamyard on macOS with M1 or newer, Windows, or Linux: www.kafka-streamyard.com. The application works consistently across the three platforms and updates automatically when a new version is available.
If you try it, I would genuinely appreciate feedback and criticism. I am particularly interested in hearing how other people investigate Kafka data, prepare test scenarios, and follow workflows across topics. If there is a specific problem you would like it to solve, let me know. I enjoy turning useful feature requests around quickly.
AI disclosure: This is mostly in my own words, but I used AI to correct my grammar, spelling, structure, and word order.
Top comments (0)