DEV Community: Dave Klein

Getting Started as a Kafka Developer

Dave Klein — Mon, 30 Jan 2023 20:28:41 +0000

Intro – why then how

So, you’ve decided to pursue a career developing applications with Apache Kafka^®. You’ve made a wise decision. Kafka is used by over 80% of Fortune 500 companies, and Kafka development ranks as one of the highest-paying job skills in IT. And besides all the practical stuff, it’s a lot of fun to work with!

Now you’re probably wondering what’s the best way to get started in this new career. In this article, we’ll discuss some tips, strategies, and resources to get you off to a great start on this exciting journey.

Developer vs. administrator

First, you should be clear about what type of Kafka work you’d like to pursue. There are a variety of roles in this space, but they mainly fall into one of two camps: developer or administrator. Since I’m a developer, that will be the main focus of this article, but if you lean more toward the administrator or operator role, you may still be able to glean some insights.

For developers, there’s a range of opportunities, such as event-driven microservices, real-time analytics, data pipelines, stream processing, and more. You can get a good feel for some of the ways to use Kafka by perusing these use cases.

Once you have an idea of the type of work you’d like to do, it’s time to drill into more specifics, like programming languages, industries, and types of companies.

Lost in translation

When we talk about software development, we tend to think in terms of the language we’re most familiar with, but there is Kafka client support for many languages so I want to be careful not to make assumptions. I mean, you’re probably programming in Haskell, but you might not be 🙂. So we need to consider language options and opportunities. The client libraries that ship with Kafka are in Java and will work with most JVM languages, but there are also very good libraries available for Python, C/C++, .NET, JavaScript, Go, and others.

The level of support and popularity of these different Kafka clients varies, with Java being the most popular and having the strongest support, both from Confluent and the community (e.g., external libraries, books, tutorials). So, if you don’t already have a preference, Java might be the way to go. A related consideration is who, in your area, is hiring and what language(s) are they using. A search on a job site, like Indeed or Dice, can be helpful there.

Learning resources

Once you’ve decided on a language to focus on, it’s time to start filling those knowledge gaps. Don’t be discouraged by this step. We all have knowledge gaps, and filling them can be very rewarding. First, you’ll want to make sure you have a good understanding of Kafka basics. Fortunately, there are many resources to help you with this. A web search will turn up many great books and other resources. And, of course, Confluent Developer offers interactive courses ranging from introductory (Apache Kafka 101) to advanced (Kafka Internals), full documentation, and other content to help you get started.

The Kafka ecosystem

The Kafka ecosystem is continually growing, but there are some key components that anyone looking to work with Kafka should be familiar with.

Clients

We’ve already mentioned the clients. In whichever language you choose, there will be some code available for producing data to Kafka, consuming data from Kafka, and administering resources and configurations on Kafka brokers. Developers should be proficient with these. Aspiring administrators should be familiar with them, as they will affect how users deal with your Kafka cluster.

Kafka Connect

Kafka Connect is a framework built on top of the Kafka Producer and Consumer client libraries. It allows you to integrate external systems, such as databases, analytics engines, and SaaS applications with Kakfa using plugins, called connectors, and some configuration. There are hundreds of connectors available, but the best source of vetted connectors is the Confluent Hub. There you will find over 200 connectors with all the information you need to use them.

Kafka Streams

Kafka Streams is a Java library that provides powerful APIs and an easy-to-use DSL for building stateless and stateful event streaming applications. Kafka Streams can only be used with JVM languages, but even developers not working with a JVM language would benefit from understanding how it works and the role it plays. There are similar libraries in other languages, and an understanding of Kafka Streams will help you evaluate them.

Schema Registry

Kafka producers and consumers execute separately and don’t know anything about each other, but they do need to know about and agree upon the format of the data they are working with. This is where schemas come into play, and if you’re using schemas with Kafka, you really should be using the Confluent Schema Registry. Schema Registry provides a way to store and retrieve schemas, as well as a means of versioning your schemas as they evolve. It works well with most Kafka client libraries and can also be accessed directly via HTTP. You can learn more about it with the Schema Registry 101 course on Confluent Developer.

ksqlDB

As we discussed earlier, Kafka Streams is only available on the JVM, but there is another powerful tool for creating event-streaming applications with Kafka, and that is ksqlDB. ksqlDB is an application that runs in its own cluster that allows you to build streaming applications with SQL. ksqlDB supports filtering, aggregation, transformation, and joining of event streams and tables based on Kafka topics. It’s REST API allows you to interact with those applications from just about any programming language. For more information check out ksqlDB 101 or Inside ksqlDB

Command Line and Graphical interfaces

There are many ways to work with Kafka interactively, ranging from the shell scripts that come with Kafka to extensive graphical user interfaces. There isn’t space here to cover them all, but here are a few that are worth checking out.

Command Line

Confluent CLI - https://docs.confluent.io/confluent-cli/current/overview.html

kcat (formerly KafkaCat) - https://github.com/edenhill/kcat

kcctl (CLI for Kafka Connect) - https://github.com/kcctl/kcctl

GUI

Confluent Control Center - https://docs.confluent.io/platform/current/control-center/index.html

Conduktor - https://www.conduktor.io

Kafdrop - https://github.com/obsidiandynamics/kafdrop

akHQ - https://github.com/tchiotludo/akhq

Strategy

That’s a lot to learn, and it might seem overwhelming, so here’s some advice for how to build your knowledge without burning out.

Warming up

Before jumping into building Kafka applications, depending on your level of experience, it will be helpful to warm up with some introductory material, for example, the courses on Confluent Developer, such as Kafka 101, Kafka Connect 101, and Schema Registry 101. These courses have both video and text content that provide you with a gentle introduction to these technologies. They even include exercises that allow you to get hands-on.

Working out

A great next step is to get more hands-on experience with Kafka using quick-start guides and tutorials, where a problem is presented, and you can use Kafka and your favorite language to solve it. Confluent Developer provides plenty of these. Most of the tutorials are based on Java, but there are also getting-started exercises available for Python, .Net, Go, and other languages.

Building a project

Now that you’ve got some experience working with and solving problems with Kafka, you can continue and accelerate your learning journey by building a project. It can be something for work, a side project, or even a cool demo idea that you can show off at a meetup (more about that in the next section). But whatever project you choose, it should be end-to-end so that you get the broadest possible experience from it.

The reason that a complete project is so much more helpful than exercises or tutorials is that the problems you are trying to solve are your own. This will provide a much stronger context for what you are learning. Context is like a hook on which to hang knowledge. Things learned without context tend to fade quickly. Just ask someone who’s been cramming for an exam.

Another benefit of building a project with Kafka is that you can use it to show what you know by hosting your project in a GitHub repository. One of the advantages that we have in the technology space is that it is much easier to show prospective employers what we are capable of, and one of the great ways to do this is via source code repositories.

Contributing to Apache Kafka

There are many opportunities for involvement in the open source Apache Kafka project. This is a great way to learn as well as to put your learning into practice. Check out the official contributors' guide or watch this video from Kafka Summit 2020 for more information and inspiration.

Certification

Another way to demonstrate your knowledge is certification. While not a silver bullet, when used in conjunction with examples of actual code you’ve written, certification can provide an extra level of comfort to prospective employers. In truth, the more valuable aspect of getting certified is the incentive it provides for learning. Confluent provides certificates for both developers and administrators, along with suggestions for how to prepare for the exams.

Show your work

Chronicling your learning journey by way of a blog can help you to learn faster by better organizing your thoughts. It can also help you to get valuable feedback from those who read it. And it’s a great way to show prospective employers what you’ve learned and your ability to communicate it.

Don’t go it alone

As you go about building your first project or preparing for your certification exam, you will undoubtedly run into questions. You can find some good answers on Google or Stackoverflow, but there is a large, active, and helpful community of developers, administrators, and just all-around good people in the Kafka community.

Getting involved in the community will make your learning journey more enjoyable and more productive. It will also provide you with invaluable networking opportunities that can lead to your first job as a Kafka developer.

The Confluent Developer Forum and Confluent Community Slack are two great places to introduce yourself to the community, ask questions, and learn from reading others' questions and answers. You can also subscribe to the Apache Kafka developer and user mailing lists.

But the community is about more than just learning. It’s also about helping, inspiring, collaborating, and building each other up. So, don’t just lurk, or post a question when you’re stuck. Get involved. Try to answer some questions, tell us about what you are working on, or the cool new thing you learned.

Besides the online text venues, another great way to get to know people in the community is at meetups and conferences. The Confluent meetup hub will show you what meetups are coming up. There may be one in your area, or you can join one of the many online meetups from anywhere. As you learn, you may even consider presenting at a meetup. Most of these are recorded, and a link to your presentation could be a great addition to your CV.

Not only do members of the community help each other with technical issues, they often have valuable career advice. Here are some tips from a few Community Catalysts.

Olena Kutsenko is a Senior Developer Advocate.

“Starting to work with a new technology is often tough, especially if it is as complex as Apache Kafka. Here is a list of things that I'd recommend to those who want to become an Apache Kafka developer:

Don't be afraid to ask for help. The community around Apache Kafka is knowledgeable and very friendly. So if after reading the docs and trying different approaches, you're still struggling to find a working solution for your task, don't hesitate to ask a question. Who knows, maybe the perfect solution is not so far from what you have!
Don't get frustrated if the learning process is not as quick as you hoped. The Apache Kafka ecosystem is very wide and it is normal that a single person won't know all the nuances of the system. In fact, it is better to accept the fact that the learning process is never over! That's why it is so vital for us to share knowledge with each other (See the next point ;) ).
Share what you learned with others. This is the best way to solidify your knowledge and get different perspectives.”

Robert Zych is a Data Platform Engineer.

“My 1st tip for anyone (regardless of programming background) interested in using Kafka is to understand when it should and shouldn’t be used. My 2nd tip would be to get/develop experience with a JVM language such as Kotlin, Java, or Scala (in that order). My 3rd tip would be to take courses and CCDAK (Confluent Certified Developer for Apache Kafka). And my 4th tip would be to experiment and build something with Kafka (preferably with helpful friends like you and Neil 🙂)”

Neil Buesing is a Kafka and Kafka Streams expert. He shares a few tips from the perspective of a hiring manager.

“Show interest in Kafka’s improvements (e.g. up to date on some KIPs) seeing someone being current sets them apart from others in that they can be seen as someone that “enjoys Kafka” vs just using Kafka.
Operational Knowledge - I don’t need a developer to know how to manage a cluster, but understanding operational aspects is helpful; e.g. knowing that a compacted topic and how it gets compacted is useful; knowing performance issues when it comes to “commitSync()” after every reading a message — I was at a client that did a commitSync() after EVERY consumed message. This is a sign that Kafka is not well understood.
If you use an additional framework be sure you know what is part of that framework vs what is Apache Kafka (e.g. KafkaTemplate is Spring Kafka not Kafka). — ok, maybe this is just a pet peeve of mine.”

Along with the Confluent forum and slack group, you can also find many members of the Kafka community on Twitter, LinkedIn, and other social media outlets. Not only can you find help, inspiration, and potential job leads, but you can also make some great friends. This, far from complete, Twitter list can help you get started meeting some amazing people.

Practical Tips

When it comes to the actual process of looking for a job, there’s no substitute for good old-fashioned shoe leather. The more doors you knock on, the better your chances. I know that most shoes are soled with rubber these days, and you rarely walk or knock on doors when applying for jobs, but you get my meaning. Apply for as many opportunities as you can find. Even the ones that don’t turn out to be what you were looking for will provide you with valuable practice. Job hunting, and more specifically, interviewing, is a skill, and like all skills, it requires practice.

Another important point about interviews is to take careful notes of any technical questions that stumped you. Research these areas and, if possible, include what you learn in one of your projects. Remember the value of context.

As far as where to look, aside from your personal network, Indeed.com is a good place to start. As of the time of this writing, they are showing over 14,000 job openings that include Kafka as a desired skill. Though not quite as large of a database, Dice.com has also yielded good results for me in the past. Both of these sites allow you to post your CV and specify the types of opportunities you are looking for but don’t stop at that. Take the initiative to reach out to recruiters and potential employers.

Enjoy the journey

We’ve talked about a lot of things that you can do, and it may seem a bit overwhelming, but don’t look at it as a checklist, but rather a menu of opportunities for a fun adventure. Learning, experimenting, networking, blogging, presenting, and even interviewing can all be a lot of fun. There will be problems to solve, challenges to overcome, people to meet, and milestones to pass along the way. So enjoy the journey, and keep us posted. We’re rooting for you!

Helpful Tools for Apache Kafka Developers

Dave Klein — Thu, 08 Jul 2021 19:59:42 +0000

This blog post was originally published on the Confluent blog.

Apache Kafka^® is at the core of a large ecosystem that includes powerful components, such as Kafka Connect and Kafka Streams. This ecosystem also includes many tools and utilities that make us, as Kafka developers, more productive while making our jobs easier and more enjoyable. Below, we'll take a look at a few of these tools and how they can help us get work done.

kafkacat

We like to save the best for last, but this tool is too good to wait. So, we'll start off by covering kafkacat.

kafkacat is a fast and flexible command line Kafka producer, consumer, and more. Magnus Edenhill, the author of the librdkafka C/C++ library for Kafka, developed it. kafkacat is great for quickly producing and consuming data to and from a topic. In fact, the same command will do both, depending on the context. Check this out:

$  ~ echo "Hello World" | kafkacat -b localhost:29092 -t hello-topic
% Auto-selecting Producer mode (use -P or -C to override)

We’ve sent data to stdout with echo and piped it to kafkacat. We only needed two simple flags: -b for the broker and -t for the topic. kafkacat realizes that we are sending it data and switches into producer mode. Now, we can read that data with the exact same kafkacat command:

$  ~ kafkacat -b localhost:29092 -t hello-topic
% Auto-selecting Consumer mode (use -P or -C to override)
Hello World
% Reached end of topic hello-topic [0] at offset 1

If we want to send a record with a key, we just need to use a delimiter and tell kafkacat what it is with the -K flag. In this case, we'll use a colon:

$  ~ echo "123:Jane Smith" | kafkacat -b localhost:29092 -t customers -K:
% Auto-selecting Producer mode (use -P or -C to override)

Again, the same kafkacat command will read the record from the topic:

$  ~ kafkacat -b localhost:29092 -t customers -K:
% Auto-selecting Consumer mode (use -P or -C to override)
123:Jane Smith
% Reached end of topic customers [0] at offset 1

Alternatively, we can leave the -K flag off when reading, if we only want the value:

$  ~ kafkacat -b localhost:29092 -t customers
% Auto-selecting Consumer mode (use -P or -C to override)
Jane Smith
% Reached end of topic customers [0] at offset 1

Note that piping data from stdout to kafkacat, as we did above, will spin up a producer, send the data, and then shut the producer down. To start a producer and leave it running to continue sending data, use the -P flag, as suggested by the auto-selecting message above.

The consumer will stay running just as the kafka-console-consumer would. In order to consume from a topic and immediately exit, we can use the -e flag.

To consume data that is in Avro format, we can use the -s flag. This flag can be used for the whole record -s avro, for just the key -s key=avro, or just the value -s value=avro. Here's an example using the movies topic from the popular movie rating tutorial:

$  ~ kafkacat -C -b localhost:29092 -t movies -s value=avro -r http://localhost:8081
------------------------------------------------------
{"id": 294, "title": "Die Hard", "release_year": 1988}
{"id": 354, "title": "Tree of Life", "release_year": 2011}
{"id": 782, "title": "A Walk in the Clouds", "release_year": 1995}
{"id": 128, "title": "The Big Lebowski", "release_year": 1998}
{"id": 780, "title": "Super Mario Bros.", "release_year": 1993}

There is a lot of power packed into this little tool, and there are many other flags that can be used with it. Running kafkacat -h will provide the complete list. For more great examples of kafkacat in action, check out related posts on Robin Moffatt’s blog. One piece missing from kafkacat is the ability to produce data in Avro format. As we saw, we can consume Avro with kafkacat using the Confluent Schema Registry, but we can't produce it. This leads us to our next tool.

Confluent REST Proxy

Confluent REST Proxy is a feature-rich HTTP Kafka client. It can be used to provide Kafka support to applications written in a language without a native Kafka client. This is probably its most common use, but it can also be a handy developer tool. It can easily produce Avro data to a Kafka topic, as shown here:

$ ~ curl -X POST \
-H "Content-Type: application/vnd.kafka.avro.v2+json" \
-H "Accept: application/vnd.kafka.v2+json" \
--data @newMovieData.json "http://localhost:8082/topics/movies"
------------------------------------------------------------------------------
{"offsets":[{"partition":0,"offset":5,"error_code":null,"error":null}],"key_schema_id":null,"value_schema_id":3}

REST Proxy is part of the Confluent Platform under the Confluent Community License, but it can be used on its own with any Kafka cluster. It can do a lot more than what we’ll cover here, as you can see from the docs. As shown above, REST Proxy can be used from the command line with curl or something similar. It can also be used with tools such as Postman to build a user-friendly Kafka UI. Here's an example of producing to a topic with Postman (the Content-Type and Accept headers were set under the “Headers” tab): As we can see from both the curl and Postman versions, REST Proxy does require that the schema for Avro messages be passed in with each produce request. A tool like Postman, which allows you to build up a library of saved queries, can make this easier to manage. To consume from topics with REST Proxy, we first create a consumer in a consumer group, then subscribe to a topic or topics, and finally fetch records to our heart’s content. We'll switch back to curl so that we can see all the necessary bits at once. First, we POST to the consumer’s endpoint with our consumer group name. In this POST request, we will pass a name for our new consumer instance, the internal data format (in this case, Avro), and the auto.offset.reset value.

$ ~ curl -X POST  -H "Content-Type: application/vnd.kafka.v2+json" \
      --data '{"name": "movie_consumer_instance", "format": "avro", "auto.offset.reset": "earliest"}' \
      http://localhost:8082/consumers/movie_consumers
------------------------------------------------------------------------------
{"instance_id":"movie_consumer_instance","base_uri":"http://localhost:8082/consumers/movie_consumers/instances/movie_consumer_instance"}

This will return the instance id and base URI of the newly created consumer instance. Next, we'll use that URI to subscribe to a topic with a POST to the subscription endpoint.

$  ~ curl -X POST -H "Content-Type: application/vnd.kafka.v2+json" --data '{"topics":["movies"]}' \      http://localhost:8082/consumers/movie_consumers/instances/movie_consumer_instance/subscription

This doesn't return anything, but should get a 204 response. Now we can use a GET request to the records endpoint of that same URI to fetch records.

$  ~ curl -X GET -H "Accept: application/vnd.kafka.avro.v2+json" \      http://localhost:8082/consumers/movie_consumers/instances/movie_consumer_instance/records
----------------------------------------------------------------------
[{"topic":"movies","key":null,"value":{"id": 294, "title": "Die Hard", "release_year": 1988},"partition":0,"offset":0},
{"topic":"movies","key":null,"value":{"id": 354, "title": "Tree of Life", "release_year": 2011},"partition":0,"offset":1},{"topic":"movies","key":null,"value":{"id": 782, "title": "A Walk in the Clouds", "release_year": 1995},"partition":0,"offset":2},{"topic":"movies","key":null,"value":{"id": 128, "title": "The Big Lebowski", "release_year": 1998},"partition":0,"offset":3},{"topic":"movies","key":null,"value":{"id": 780, "title": "Super Mario Bros.", "release_year": 1993},"partition":0,"offset":4},{"topic":"movies","key":null,"value":{"id":101,"title":"Chariots of Fire","release_year":1981},"partition":0,"offset":5}]

The consumer that we created will remain, and we can make the same GET request anytime to check for new data. If we no longer need this consumer, we can DELETE it using the base URI.

$  ~ curl -X DELETE -H "Content-Type: application/vnd.kafka.v2+json" \    http://localhost:8082/consumers/movie_consumers/instances/movie_consumer_instance

We can also get information about brokers, topics, and partitions with simple GET requests.

$  ~ curl "http://localhost:8082/brokers"

$  ~ curl "http://localhost:8082/topics"

$  ~ curl "http://localhost:8082/topics/movies"

$  ~ curl "http://localhost:8082/topics/movies/partitions"

These requests can return quite a bit of JSON data, which we'll leave off for the sake of space. However, this does lead us nicely to our next tool.

`jq`: A command line processor for JSON

Though not specific to Kafka, jq is an incredibly helpful tool when working with other command line utilities that return JSON data. jq is a command line utility that allows us to format, manipulate, and extract data from the JSON output of other programs. Instructions for downloading and installing jq can be found on GitHub, along with links to tutorials and other resources.

Let's go back and take a look at the REST Proxy output from the GET call to our consumer above. It's not the largest blob of JSON out there, but it’s still a bit hard to read. Let's try again, this time piping the output to jq:

$  ~ curl -X GET -H "Accept: application/vnd.kafka.avro.v2+json" \
      http://localhost:8082/consumers/movie_consumers/instances/movie_consumer_instance/records | jq

[
  {
    "topic": "movies",
    "key": null,
    "value": {
      "id": 294,
      "title": "Die Hard",
      "release_year": 1988
    },
    "partition": 0,
    "offset": 0
  },
  {
    "topic": "movies",
    "key": null,
    "value": {
      "id": 354,
      "title": "Tree of Life",
      "release_year": 2011
    },
    "partition": 0,
    "offset": 1
  },
  {
    "topic": "movies",
    "key": null,
    "value": {
      "id": 782,
      "title": "A Walk in the Clouds",
      "release_year": 1995
    },
    "partition": 0,
    "offset": 2
  },
  {
    "topic": "movies",
    "key": null,
    "value": {
      "id": 128,
      "title": "The Big Lebowski",
      "release_year": 1998
    },
    "partition": 0,
    "offset": 3
  },
  {
    "topic": "movies",
    "key": null,
    "value": {
      "id": 780,
      "title": "Super Mario Bros.",
      "release_year": 1993
    },
    "partition": 0,
    "offset": 4
  },
  {
    "topic": "movies",
    "key": null,
    "value": {
      "id": 101,
      "title": "Chariots of Fire",
      "release_year": 1981
    },
    "partition": 0,
    "offset": 5
  }
]

It’s much easier to read now but still a bit noisy. Let's say we only want the movie titles and their release years. We can do that easily with jq:

$  ~ curl -X GET -H "Accept: application/vnd.kafka.avro.v2+json" \
      http://localhost:8082/consumers/movie_consumers/instances/movie_consumer_instance/records | jq \
      | jq '.[] | {title: .value.title, year: .value.release_year}'

{
  "title": "Die Hard",
  "year": 1988
}
{
  "title": "Tree of Life",
  "year": 2011
}
{
  "title": "A Walk in the Clouds",
  "year": 1995
}
{
  "title": "The Big Lebowski",
  "year": 1998
}
{
  "title": "Super Mario Bros.",
  "year": 1993
}
{
  "title": "Chariots of Fire",
  "year": 1981
}

Let's take a look at what we just did (and you can follow along with the live example at jqplay):

We piped the output from the REST Proxy to jq. The bit between the single quotes is a jq program with two steps. jq uses the same pipe character to pass the output of one step to the input of another.
In our example, the first step in jq is an iterator, which will read each movie record from the array and pass it to the next step.
The second step in jq creates a new JSON object from each record. The keys are arbitrary, but the values are derived from the input using jq's identity operator, '.'.

Pretty cool, huh? There is much more that can be done with jq, and you can read all about it in the documentation. The way that jq operates on a stream of JSON data, allowing us to combine different operations in order to achieve our desired results, reminds me of Kafka Streams—bringing us to our final tool.

Kafka Streams Topology Visualizer

The Kafka Streams Topology Visualizer takes the text description of a Kafka Streams topology and produces a graphic representation showing input topics, processing nodes, interim topics, state stores, etc. It's a great way to get a big-picture view of a complex Kafka Streams topology. The topology for the movie ratings tutorial is not all that complex, but it will serve nicely to demonstrate this tool.

Here's the text of our topology, which we captured with the Topology::describe method:

Topologies:
   Sub-topology: 0
    Source: KSTREAM-SOURCE-0000000000 (topics: [movies])
      --> KSTREAM-MAP-0000000001
    Processor: KSTREAM-MAP-0000000001 (stores: [])
      --> KSTREAM-SINK-0000000002
      <-- KSTREAM-SOURCE-0000000000
    Sink: KSTREAM-SINK-0000000002 (topic: rekeyed-movies)
      <-- KSTREAM-MAP-0000000001 Sub-topology: 1 Source: KSTREAM-SOURCE-0000000010 (topics: [KSTREAM-MAP-0000000007-repartition]) --> KSTREAM-JOIN-0000000011
    Processor: KSTREAM-JOIN-0000000011 (stores: [rekeyed-movies-STATE-STORE-0000000003])
      --> KSTREAM-SINK-0000000012
      <-- KSTREAM-SOURCE-0000000010 Source: KSTREAM-SOURCE-0000000004 (topics: [rekeyed-movies]) --> KTABLE-SOURCE-0000000005
    Sink: KSTREAM-SINK-0000000012 (topic: rated-movies)
      <-- KSTREAM-JOIN-0000000011 Processor: KTABLE-SOURCE-0000000005 (stores: [rekeyed-movies-STATE-STORE-0000000003]) --> none
      <-- KSTREAM-SOURCE-0000000004 Sub-topology: 2 Source: KSTREAM-SOURCE-0000000006 (topics: [ratings]) --> KSTREAM-MAP-0000000007
    Processor: KSTREAM-MAP-0000000007 (stores: [])
      --> KSTREAM-FILTER-0000000009
      <-- KSTREAM-SOURCE-0000000006 Processor: KSTREAM-FILTER-0000000009 (stores: []) --> KSTREAM-SINK-0000000008
      <-- KSTREAM-MAP-0000000007
    Sink: KSTREAM-SINK-0000000008 (topic: KSTREAM-MAP-0000000007-repartition)
      <-- KSTREAM-FILTER-0000000009

You may be adept at reading this kind of output, but most people will find a graphical representation very helpful: The Kafka Streams Topology Visualizer is a web app that you can host yourself (the source is available on GitHub). For occasional use, the public hosted version is probably sufficient.

A complex topology might be difficult to view all at once, so you can also visualize sub-topologies and then combine the images in a way that is easier to view. This can be a huge help in bringing new developers up to speed on an existing Kafka Streams application.

ksqlDB topologies

ksqlDB is an event streaming database that allows us to build complex topologies using syntax familiar to any SQL developer. Since ksqlDB is built on top of Kafka Streams, the Kafka Streams Topology Visualizer also works on these types of topologies.

We can get the topology description from ksqlDB with the EXPLAIN command. First, find the executing query:

ksql> SHOW QUERIES;

 Query ID              | Query Type | Status    | Sink Name      | Sink Kafka Topic | Query String
--------------------------------------------------------------------------------------------------------------------------
 CSAS_SHIPPED_ORDERS_0 | PERSISTENT | RUNNING:1 | SHIPPED_ORDERS | SHIPPED_ORDERS   | CREATE STREAM SHIPPED_ORDERS WITH 
 ...

Now we can use that generated query name, CSAS_SHIPPED_ORDERS_0, to get the topology:

ksql> EXPLAIN CSAS_SHIPPED_ORDERS_0;

This gives us a fair amount of output, so we won't show it all here, but toward the end, we see the topology description. Copying and pasting it into the visualizer results in this diagram:

Tip of the iceberg

We’ve looked at four helpful tools for Apache Kafka developers, but there are many others out there, which is one of the benefits of working in such a vibrant community. If there is a command line tool or a graphical application that you find helpful in getting the most out of Kafka, tell others about it. The Confluent Community Forum is a great place to share this kind of information. We look forward to continuing this discussion with you there!