DEV Community: KafmaKarma

confluent-kafka-go in 2026: What Still Works and What to Watch

KafmaKarma — Wed, 21 Oct 2026 10:01:38 +0000

confluent-kafka-go is Confluent's Kafka client for Go. It wraps librdkafka, Confluent's C client, through cgo, and it is still actively developed. v2.16.0 shipped on 7 October 2026. If you run a Go service on it, five things deserve a check: how it builds, what its Schema Registry packages pull in, where keyed records land, when offsets get committed, and which consumer group protocol it uses.

TL;DR: confluent-kafka-go covers transactions, admin operations, KIP-848 consumer groups, and Confluent's own Schema Registry serializers. It needs cgo, and Go turns cgo off when you cross-compile. The Schema Registry serializers need cgo too, even next to a pure Go client. The default partitioner does not match the Java client, but murmur2_random does. By default, offsets can be committed before your code finishes processing a message. The consumer group protocol is expected to default to KIP-848 in a later release, so set it explicitly.

For how confluent-kafka-go compares with franz-go, kafka-go, and Sarama on cgo, consumer groups, and Schema Registry, see Kafka Go Clients: franz-go vs kafka-go vs Sarama.

What still works

librdkafka underneath. Most protocol behavior comes from the bundled librdkafka, the same C library behind Confluent's Python, .NET, and JavaScript clients. v2.16.0 bundles librdkafka 2.16.0, with idempotent and transactional producers and an AdminClient for topics, configs, ACLs, and consumer groups.
No separate install on common platforms. The module ships prebuilt librdkafka for macOS (x64 and arm64), glibc Linux (x64 and arm64), musl Linux such as Alpine (x64 and arm64), and Windows x64. On those platforms, you do not install librdkafka on the build or target machine.
KIP-848 consumer groups. The next-generation consumer group protocol, in which the broker computes partition assignments, has been production-ready since v2.12.0 on Kafka 4.0+ brokers. You turn it on with group.protocol=consumer.
Confluent's Schema Registry serializers. The schemaregistry packages include serializers for Avro, Protobuf, and JSON Schema that register and cache schemas and write Confluent's wire format.
Active development. In the 90 days to 25 September 2026, the repository had 55 commits and 29 merged pull requests. Over the preceding year, merges were spread across 17 accounts, and the most active one handled 39% of them. The open backlog is older, with 74 open pull requests at a median age of 2.4 years. These figures come from the GitHub snapshot in the full comparison.

What to watch

cgo is required, and cross-compiling turns it off

The client calls librdkafka through cgo, so it builds only with CGO_ENABLED=1 and a C compiler. Go turns cgo off by default when you cross-compile, for example GOOS=linux go build on a Mac, and when it cannot find a C compiler, as in the golang:alpine image. With cgo off, a v2.16.0 build fails inside the library:

# github.com/confluentinc/confluent-kafka-go/v2/kafka
.../kafka/deserialized_message.go:28:17: undefined: TopicPartition
.../kafka/deserialized_message.go:34:17: undefined: TimestampType
.../kafka/deserialized_message.go:36:19: undefined: Header

Up to v2.15.1, the same build fails in your own code instead, with errors such as undefined: kafka.NewProducer and undefined: kafka.ConfigMap. To build for another platform, set CGO_ENABLED=1 and give cgo a C compiler for that platform, or build on the target platform itself, for example in a container for that architecture.

A few more build details matter once the service ships in a container:

Alpine needs -tags musl. The project README requires it for go get, go build, and similar commands on musl systems.
The binary is not static. It links the system C library dynamically, so the runtime image needs one. scratch and distroless/static have none, and a binary built against a newer glibc fails on an older one. For a fully static binary, the project README describes a musl build with external-linker flags.
Kerberos needs a system librdkafka. The bundled builds exclude GSSAPI. Install a librdkafka with GSSAPI on both the build and target systems, and build with -tags dynamic. Without librdkafka and pkg-config on the build machine, that build stops with exec: "pkg-config": executable file not found in $PATH.
v2.16.0 needs Go 1.26. The module's go directive moved from 1.25.0 to 1.26.0. With the default GOTOOLCHAIN=auto, Go 1.25 downloads a newer toolchain by itself. With GOTOOLCHAIN=local, you need to upgrade Go first.

Schema Registry serializers need cgo too

The Schema Registry client and serializers do not call librdkafka, so it seems natural to pair them with a pure Go client such as franz-go and build with CGO_ENABLED=0. For the serializers, that stopped working in v2.10.1 (June 2025). In that release, the serializer interface gained methods that can carry the schema ID in Kafka headers. Those methods use the kafka.Header type, and the kafka package is the one that needs cgo.

We built a small program that uses the Avro serializer (avrov2) with franz-go, and a second one that uses only the Registry client, with CGO_ENABLED=0 and Go 1.27.1 against each release:

Release	Registry client only	Avro serializer
v2.5.4 to v2.10.0	Builds	Builds
v2.10.1 to v2.13.0	Builds	`undefined: kafka.Header`
v2.13.2 to v2.15.1	`undefined: kafka.LibraryVersion`	Same error
v2.16.0	`undefined: TopicPartition`	Same error

The v2.16.0 release notes say this dependency was removed from the Registry client, and that change did ship. The same release, however, added an experimental SerializingProducer and DeserializingConsumer API. It brought a new import of the kafka package into the Registry client, and new files in the kafka package that do not compile without cgo.

Both problems are tracked in issue #1635, with a proposed fix in pull request #1634. The release notes also mark that new API as subject to change or removal, so pin the module version if you adopt it.

The default partitioner does not match Java

The default partitioner, consistent_random, hashes keys with CRC32, while the Java client and franz-go use Murmur2. If a Go service and a Java service write the same keys to one topic, a key can land on two partitions, which breaks per-key ordering. Setting partitioner to murmur2_random gives Java-compatible placement:

p, err := kafka.NewProducer(&kafka.ConfigMap{
    "bootstrap.servers": "localhost:9092",
    "partitioner":       "murmur2_random",
})

We checked it by producing with v2.16.0 to a local Kafka 4.3.1 broker and comparing the partition in each delivery report with the Java client 4.3.1 (BuiltInPartitioner.partitionForKey). We used the keys customer_42, order-1001, user-7, 21, foobar, a-little-bit-long-string, café, and an empty key, each at 3, 6, and 12 partitions.

murmur2_random matched Java in all 24 cases. The default matched in only 1 of the 21 cases with a non-empty key. For example, on a three-partition topic customer_42 goes to partition 1 by default and to partition 0 with murmur2_random, as in Java.

Empty keys behave differently too. The default sends an empty key to a random partition. Five sends to a 12-partition topic landed on partitions 4, 0, 9, 7, and 0. murmur2_random hashes it as Java does and sent it to partition 9 every time. Records with a null key go to a random partition under both settings.

After a switch, new records for a key can go to a different partition than its earlier records. Switch when other producers write the same keys to the same topic, and compare placements with your own keys first.

Offsets can be committed before processing finishes

By default, the consumer stores a message's offset as soon as Poll or ReadMessage hands the message to your code (enable.auto.offset.store=true). It then commits the stored offsets in the background every 5 seconds (enable.auto.commit=true, auto.commit.interval.ms=5000). If the process crashes after a commit but before your code finishes a message, that message is not delivered again.

For at-least-once processing, turn off the automatic store and store each offset yourself after the work is done. Auto-commit stays on and commits only what you stored:

c, err := kafka.NewConsumer(&kafka.ConfigMap{
    "bootstrap.servers":        "localhost:9092",
    "group.id":                 "orders",
    "enable.auto.offset.store": false,
})

// After processing msg:
if _, err := c.StoreMessage(msg); err != nil {
    // handle the error
}

This works when you process each partition's messages in order. If you process one partition's messages in parallel, store an offset only after every earlier message in that partition has succeeded. After a crash, messages that finished but were not yet committed are delivered again, so processing must tolerate duplicates.

KIP-848 is opt-in for now

The default group.protocol is still classic, but librdkafka's configuration reference says the default "will change to consumer in next releases."

The new protocol changes a few contracts. The client no longer uses partition.assignment.strategy, session.timeout.ms, or heartbeat.interval.ms. Rebalance callbacks receive only the partitions being added or revoked. If your callbacks manage assignments themselves, use IncrementalAssign and IncrementalUnassign instead of Assign and Unassign.

Set group.protocol explicitly, so that a future upgrade does not switch protocols for you. Before moving a group to consumer, read Confluent's KIP-848 migration guide.

If you need what it lacks

A service that builds and runs well on confluent-kafka-go can keep running it. If cgo is the problem, franz-go is a pure Go client that supports KIP-848. Its own Schema Registry package, pkg/sr, writes the same wire format, but you supply the Avro, Protobuf, or JSON Schema codec. The full comparison linked above covers what to test before switching, including how to keep key placement.

Check what your service actually does

Whether you change the partitioner, the offset store, or the build image, check the records themselves. Kafma, a Kafka GUI client, shows each record's key, partition, headers, and raw bytes, and a consumer group's committed offsets and lag. That lets you confirm that keys land where you expect. To check whether commits run ahead of processing, compare the committed offsets with your application logs. Rebalances, retries, and crash recovery still need tests in your application.

Written by the team behind Kafma.

KafkaJS in 2026: What Still Works and What to Watch

KafmaKarma — Wed, 14 Oct 2026 09:19:07 +0000

KafkaJS is still a widely used Kafka client for Node.js. npm counted about 3.6 million downloads in the week of 6–12 October 2026, and in August 2026 GitHub listed more than 32,000 public repositories depending on it. Its latest release, 2.2.4, shipped in February 2023. If you run KafkaJS today, or are about to inherit a service that does, here is what still works, what to watch, and how to stop the idle timer loop that newer Node.js versions now warn about.

TL;DR: In our test against a local Kafka 4.3.1 broker, KafkaJS 2.2.4 still produced, consumed, and ran Admin calls. It has had no release since February 2023 and does not support KIP-848 or GSSAPI. It also keeps a 1 ms timer running while a connection is idle, on every Node.js version we tested; Node.js 23 and later expose it through a warning. A small patch stops the timer.

For how KafkaJS compares with Platformatic Kafka, confluent-kafka-javascript, and node-rdkafka on Node.js support, consumer models, and Schema Registry, see Kafka JavaScript & Node.js Clients: KafkaJS vs Platformatic.

What still works

No native addon. KafkaJS is pure JavaScript and declares support for Node.js 14+, so there is no librdkafka binary to build or ship for each platform.
A familiar Promise API. Producers, consumers with eachMessage and eachBatch handlers, transactions, headers, and an Admin client.
Common authentication. PLAIN, SCRAM, and OAUTHBEARER. For Amazon MSK IAM, AWS's aws-msk-iam-sasl-signer-js README shows KafkaJS using the signer as an OAUTHBEARER token provider.
Kafka 4.x brokers. Against a local Kafka 4.3.1 broker, KafkaJS 2.2.4 created a topic, produced records, consumed them in a consumer group, and deleted the topic and group without errors.

What to watch

No releases since February 2023

The latest npm release is 2.2.4, published on 27 February 2023, and the last commit on the default branch was on 16 May 2024. As of 28 August 2026, the repository had 355 open issues and 62 open pull requests, and no pull request had been merged in the previous 90 days. Fixes proposed as pull requests, including the two below, are not being merged or released.

An idle 1 ms timer behind TimeoutNegativeWarning

On Node.js 23 and later, including the Node.js 24 LTS line, connecting a KafkaJS producer, consumer, or admin client prints a warning like this:

(node:95450) TimeoutNegativeWarning: -1791964353628 is a negative number.
Timeout duration was set to 1.

The warning is new, but the behavior behind it is not. After a connection receives its first response, KafkaJS's request queue schedules a check for pending requests. When the connection is not throttled, the delay it computes is negative (-1 - Date.now() by default), and Node.js runs a timer with a negative delay after 1 ms. Each check schedules the next one, so the loop keeps running while the connection is idle. The loop runs on Node.js 20 and 22 as well; of the versions we tested (20, 22, 23, 24, and 26), only 23 and later printed the warning. The report is issue #1751, open since May 2025.

The cost is small but constant. We ran the script below three times per configuration on an Apple Silicon Mac, with one idle, connected producer and a single-node Kafka 4.3.1 broker. It counts timers created with a delay below 1 ms and the process's CPU time over 10 seconds:

Node.js	KafkaJS 2.2.4	With the patch below
22.20.0	about 8,760 short timers, 253–303 ms CPU	0 short timers, 43–44 ms CPU
24.10.0	about 8,760 short timers, 202–218 ms CPU	0 short timers, 14–15 ms CPU

That is roughly 2–3% of one core for a process that is doing nothing.

Reproduction script

// node repro.js localhost:9092   (kafkajs@2.2.4, a test broker running)
const { Kafka, logLevel } = require('kafkajs')

// Count timers created with a delay below 1 ms; Node.js runs them after 1 ms
let shortTimers = 0
const realSetTimeout = global.setTimeout
global.setTimeout = (fn, ms, ...args) => {
  if (ms < 1) shortTimers++
  return realSetTimeout(fn, ms, ...args)
}

const main = async () => {
  const kafka = new Kafka({ brokers: [process.argv[2] || 'localhost:9092'], logLevel: logLevel.NOTHING })
  const producer = kafka.producer()
  await producer.connect()

  // Measure 10 seconds of an idle, connected producer
  shortTimers = 0
  const start = process.cpuUsage()
  await new Promise(resolve => realSetTimeout(resolve, 10_000))
  const { user, system } = process.cpuUsage(start)

  console.log(`${process.version}: ${shortTimers} short timers, ${Math.round((user + system) / 1000)} ms CPU in 10 s`)
  await producer.disconnect()
}

main()

Suppressing the warning or staying on Node.js 22 hides the message but keeps the loop. The two open pull requests for this issue, #1755 and #1768, clamp the delay to 1 ms or 0 ms. In our test, both removed the warning, but the timer kept firing at the same rate.

A patch posted in the issue thread stops the loop instead: when no request is pending and the connection is not throttled, it skips scheduling the check. In node_modules/kafkajs/src/network/requestQueue/index.js, scheduleCheckPendingRequests() becomes:

scheduleCheckPendingRequests() {
  let scheduleAt = this.throttledUntil - Date.now()
  if (!this.throttleCheckTimeoutId) {
    if (this.pending.length > 0) {
      scheduleAt = scheduleAt > 0 ? scheduleAt : CHECK_PENDING_REQUESTS_INTERVAL
    } else if (scheduleAt <= 0) {
      // Nothing is pending and the connection is not throttled
      return
    }
    this.throttleCheckTimeoutId = setTimeout(() => {
      this.throttleCheckTimeoutId = null
      this.checkPendingRequests()
    }, scheduleAt)
  }
}

To keep the change across installs with npm, use patch-package:

Run npm install patch-package. Keep it a regular dependency if production installs run npm install or npm ci, because the postinstall script runs there too.
Add "postinstall": "patch-package" to your package.json scripts. If you already have a postinstall script, append && patch-package to it.
Edit the file above, then run npx patch-package kafkajs. This writes patches/kafkajs+2.2.4.patch.
Commit the patches directory.

pnpm and Yarn 2+ have built-in equivalents, pnpm patch and yarn patch.

Skipping the check is safe for queued requests because push() calls scheduleCheckPendingRequests() again whenever a request has to wait. With the patch, a producer and a consumer group still produced and consumed records against the same broker. It is still an unofficial change to a dependency, so test it under your own load, including broker throttling.

Classic consumer groups only

KafkaJS implements the classic consumer group protocol with eager rebalancing only. When the group rebalances, each member stops fetching from all of its partitions and rejoins, and consumption resumes only after the new assignment arrives, so every rebalance pauses the whole group (stop-the-world). Cooperative rebalancing (KIP-429), which lets members keep consuming partitions that are not moving, needs support in the client's rebalance handling, not only in the assigner. A custom assigner can change which member gets which partition, but it cannot remove the pause. KafkaJS also does not support KIP-848, the broker-driven protocol that became generally available in Kafka 4.0. Kafka 4.x brokers still support the classic protocol.

Authentication and compression gaps

No GSSAPI/Kerberos. If your cluster requires Kerberos, KafkaJS cannot connect to it.
Only GZIP is built in. Snappy, LZ4, and ZSTD come from separate codec packages, kafkajs-snappy, kafkajs-lz4, and @kafkajs/zstd, last published in 2018, 2020, and 2021. Register the codec in every process that produces or consumes those records.
Schema Registry is a separate package. @kafkajs/confluent-schema-registry is still released (4.1.0 in May 2026), and its 4.x line requires Node.js 22+.

If you need what KafkaJS lacks

A stable service that does not need these features can keep running KafkaJS. If you need KIP-848, Kerberos, or active maintenance, two clients publish KafkaJS migration guides:

confluent-kafka-javascript keeps eachMessage and eachBatch through a KafkaJS-compatible API and is built on librdkafka, so it supports KIP-848. It adds a native addon to your deployment. For GSSAPI, the maintainers note that you need a librdkafka build with GSSAPI support, which also depends on system libraries, so verify it in your target environment.
Platformatic Kafka stays pure JavaScript and supports KIP-848. It requires Node.js 22.22+ or 24.6+, and consumers move to Node.js streams.

For what changes in each case, see the migration paths in the full comparison.

Check what your service actually does

Whether you patch KafkaJS or replace it, compare before and after. A Kafka desktop client such as Kafma shows each record's key, headers, partition, and raw bytes, and a consumer group's committed offsets and lag, which helps you spot changes in what your service writes and how its offsets move. Rebalances, retries, and crash recovery still need tests in your application.

Written by the team behind Kafma.

kafkactl: kubectl-Style Commands for Kafka

KafmaKarma — Sat, 10 Oct 2026 06:58:58 +0000

If you already think in kubectl get and kubectl config use-context, kafkactl will feel familiar. It is a single binary that covers producing, consuming, topics, consumer groups and offset resets, and ACLs, with the same verb-first command style.

TL;DR: Choose kafkactl if you want one maintained binary for records and administration, especially if your team already works in Kubernetes. Look at kcl, a pure-Go CLI from the author of franz-go, instead if AWS MSK IAM without a plugin or Protobuf decoding through Schema Registry is a daily requirement.

For how kafkactl compares with kcat, kaf, kcl, rpk, and Confluent CLI across authentication, Schema Registry formats, and dated GitHub activity, see Best Kafka CLI Tools in 2026: kcat vs kafkactl vs kaf.

Install

kafkactl installs through Homebrew, winget, Docker, or a release binary:

brew install kafkactl      # Homebrew
winget install kafkactl    # Windows

Configure contexts

kafkactl keeps each cluster as a named context, but there is no command to add one. Contexts live in a YAML file, by default ~/.config/kafkactl/config.yml, which kafkactl generates on first run if it finds no config file:

contexts:
  local:
    brokers:
      - localhost:9092
  prod:
    brokers:
      - prod-kafka-1:9092
      - prod-kafka-2:9092

Then list and switch contexts:

kafkactl config get-contexts
kafkactl config use-context prod

If no current context is set, kafkactl uses the first context in the file.

Everyday commands

kafkactl get topics
kafkactl describe consumer-group my-group

describe consumer-group shows a snapshot of the group, including its lag. Reading and writing records looks like this, using a topic named orders:

# Read history and exit at the current end
kafkactl consume orders --from-beginning --exit

# Start at a timestamp
kafkactl consume orders --from-timestamp 2026-09-28T00:00:00Z

# Stop after 10 records, printed as JSON
kafkactl consume orders --from-beginning --max-messages 10 -o json

# Send one keyed record with a header
kafkactl produce orders --key=order-1 --value=hello --header source:manual

Coming from kcat

Task	kcat	kafkactl
Follow a topic from the beginning	`kcat -b localhost:9092 -t orders -C -o beginning`	`kafkactl consume orders --from-beginning`
Read history and exit	`kcat -b localhost:9092 -t orders -C -o beginning -e`	`kafkactl consume orders --from-beginning --exit`
Print records as JSON	`kcat -b localhost:9092 -t orders -C -J`	`kafkactl consume orders -o json`
List topics	`kcat -b localhost:9092 -L`	`kafkactl get topics`

kafkactl reads brokers from the current context, so there is no -b. Also, kafkactl -o json does not follow kcat's -J field layout. Before you switch a script that parses kcat output, check how kafkactl prints null values, binary payloads, and headers.

Two features worth knowing

Kubernetes mode. If your brokers are reachable only from inside a Kubernetes cluster, kafkactl can run each command in a pod through kubectl. You keep typing the same commands on your laptop.

Passwords in the OS keyring. If a required credential is missing from the config file and environment, kafkactl prompts for it in the terminal. By default, after the command succeeds, it stores the credential in Keychain on macOS, GNOME Keyring on Linux, or Credential Manager on Windows, so it does not end up in plaintext in the config file. After a password rotation, run the command with --clear-keyring to clear the stored credential and be prompted again.

Check these two gaps first

Protobuf decoding needs local files. kafkactl decodes Avro and JSON Schema through Schema Registry, but Protobuf needs local .proto files or a protoset.
AWS MSK IAM needs a plugin. It works through the separate AWS plugin rather than out of the box.

If either one is central to your work, kcl supports both natively.

Is the project active?

As of 28 September 2026, kafkactl had shipped 8 releases in the previous 12 months, the latest being v5.20.0 on 30 July 2026, with 87 commits from 8 authors over the same period. It had 2,394 Homebrew installs in the year to that date and is licensed under Apache-2.0.

One caveat: a single account merged all of its pull requests in those 12 months, so review capacity rests on one maintainer. For the same metrics across kcat, kaf, kcl, and Confluent CLI, see the GitHub repository metrics in the full comparison.

When a terminal is not enough

kafkactl is the right tool for scripts, CI jobs, and quick checks. When an investigation means reading many records and moving between topics, schemas, and consumer groups, a Kafka GUI client such as Kafma connects to the same clusters, including over SASL, TLS, and AWS MSK IAM, and decodes Avro, Protobuf, and JSON Schema through Schema Registry without local .proto files.

Written by the team behind Kafma.

Choosing a Python Kafka Consumer: confluent-kafka vs. kafka-python vs. aiokafka

KafmaKarma — Mon, 17 Aug 2026 08:44:37 +0000

Choosing a Python Kafka consumer is mainly about how it must fit into your application. confluent-kafka-python, kafka-python, and aiokafka can all consume records and participate in consumer groups, but they differ in API model, packaging, Schema Registry integration, and lifecycle management.

Start with the application's runtime. A synchronous worker has different needs from an asyncio service, while a restricted deployment environment may rule out native dependencies before API design even matters.

Versions checked: confluent-kafka 2.15.0, kafka-python 3.0.10, and aiokafka 0.14.0.

Quick decision

Consumer requirement	Best starting point
Synchronous processing or built-in Schema Registry support	`confluent-kafka-python`
Asyncio-native consumption	`aiokafka`
Pure-Python installation or PyPy support	`kafka-python`
Existing consumer with no unmet requirement	Keep the current library

For the full feature comparison, repository metrics, and producer and consumer tests, see Kafka Python Clients 2026: Which Should You Choose?.

confluent-kafka-python

confluent-kafka-python, installed from PyPI as confluent-kafka, uses the native librdkafka client. Its established synchronous Consumer API reads records through poll(), giving the application direct control over polling, processing, and offset commits.

It is a strong default for synchronous consumers and applications that deserialize Avro, Protobuf, or JSON Schema through Schema Registry. Newer AsyncIO APIs are also available, although applications should evaluate them separately from the long-established synchronous API.

Prebuilt wheels cover common Python, operating-system, and CPU combinations. Test installation in the production container or host because less common targets may require a source build.

kafka-python

kafka-python provides the synchronous KafkaConsumer. Applications can iterate over records or call poll() to receive batches, and the 3.x line supports cooperative rebalancing.

Its base package is implemented in Python and distributed as a universal wheel. This makes it useful when the deployment must avoid a native Kafka library or support PyPy. Schema Registry support is not built in, so schema-aware consumers need separate deserialization tooling.

Choose kafka-python when packaging is the deciding requirement or when an existing synchronous consumer already runs reliably. Switching libraries without a missing capability usually adds risk without changing the application's result.

aiokafka

aiokafka provides the asyncio-native AIOKafkaConsumer. Startup and shutdown use coroutines, and applications can process records with async iteration without wrapping a synchronous consumer in an executor.

This model fits FastAPI services and other applications that already manage network I/O and cancellation through an event loop. It does not include Schema Registry integration, and blocking processing must still be moved off the event loop.

Choose aiokafka when the complete consumer lifecycle needs to use async and await. Using it only because async syntax looks familiar is not enough if the rest of the processing path remains synchronous.

Before changing a consumer library

A consumer migration can change offset commits, retry behavior, partition assignment, rebalancing, and shutdown. Test those behaviors instead of checking only whether the new consumer receives a record.

Use a separate consumer group while comparing implementations. Verify deserialized keys, values, and headers, then test restart behavior and partition revocation before moving production traffic. Two consumers in the same group will divide partitions; they will not both receive every record.

Final recommendation

Start with confluent-kafka-python for a new synchronous consumer or built-in Schema Registry deserialization. Choose aiokafka when the application is asyncio-native. Choose kafka-python when pure-Python packaging or PyPy support is the deciding requirement.

Written by the team behind Kafma, a desktop Kafka UI.

A New Kafka Viewer for a Better Debugging Workflow

KafmaKarma — Fri, 07 Aug 2026 01:40:40 +0000

Kafma is a new Kafka UI client for macOS, Windows, and Linux, built to keep the debugging workflow in one workspace. Each topic opens as a persistent Console tab that keeps its message stream, producer, topic details, schema context, and consumer state together.

A Kafka browser can help you find and read a record, but debugging usually continues beyond it. You need to inspect its metadata and schema, check what a consumer group has processed, reproduce the input, and observe what happens next.

Each step is straightforward on its own. The friction comes from moving between separate consumer, producer, Schema Registry, topic, and consumer-group views and rebuilding the same context each time. Kafma keeps those parts of the investigation together, so the debugging loop becomes shorter: find the records, inspect their context, create a controlled test, and watch the result.

Kafka debugging loses context easily

Consider a common debugging sequence:

Consume a bounded set of records around the time an issue occurred.
Find the relevant key or payload and inspect its headers, partition, offset, and timestamp.
Resolve the schema and check whether the record was encoded as expected.
Check the consumer group’s position and whether the record is still in its backlog.
Reuse the record as a test case or produce a modified version.
Watch the new input arrive, then check whether the target consumer group processes it and advances its committed position.

This is one investigation, but it is often spread across several commands and screens. Copying only the payload between them is not enough: message keys, headers, schemas, partitions, and consumer positions are all part of the context.

The missing piece is not another isolated view. It is a workspace that keeps the message, schema context, topic details, consumer state, and producer together throughout the investigation.

A persistent workspace for each topic

In the Console, each topic opens in its own tab. The message stream occupies the upper part of the tab, the producer stays available below it, and a topic-details drawer opens beside them. Several topic tabs can remain open at once, so one investigation does not need to replace another.

The Console also remembers the open tabs for each cluster connection. If you switch from a production cluster to a local cluster and later return, the topics you were investigating are still open. Each cluster has its own tab set, which helps prevent local and production work from being mixed together while keeping both workspaces ready.

This matters when a problem crosses environments: you can keep the source cluster’s context intact while reproducing the issue in a local cluster.

Find and inspect the relevant messages

A busy topic may contain far more data than one investigation needs. Start with a bounded snapshot from the newest records, the beginning, a specific offset, or a timestamp. Then narrow the loaded records by key, value, or partition.

Kafma reads these records directly from the brokers. Opening or refreshing a Console tab does not join an application consumer group, trigger a rebalance, or commit offsets, so inspection does not change the consumer state you are trying to understand.

Expand a record to see its key, value, headers, partition, offset, timestamp, size, and compression. When Schema Registry is connected, Kafma resolves the schema ID and automatically decodes Avro, Protobuf, and JSON Schema records. The original bytes remain available in the Raw view, which is useful when the serializer, schema, or wire format may be part of the problem.

For the individual controls and decoding behavior, see Consuming messages and Decoding Avro and Protobuf messages.

Inspect the topic without leaving the Console

A suspicious record often raises questions about the topic around it. Is a partition unhealthy? What is the retention policy? Which consumer groups use the topic? How much data is currently retained?

The topic-details drawer answers those questions without replacing the message stream. It shows topic health, message count, retention, partitions, and the consumer groups associated with the topic. This keeps the topic details, selected record, and producer visible in the same tab.

Keeping the details beside the stream avoids a common break in the debugging flow: navigating to a separate topic or consumer page, finding the same resource again, and then returning to the original message.

Watch a consumer group in the message stream

Lag is more useful when you can connect it to the records waiting to be processed. In Watch Group mode, the Console shows a bounded window around a selected consumer group’s committed positions and marks records as consumed, pending, or unassigned.

The same view includes member status, total lag, and the lag trend. A divider separates the remaining backlog from the most recently consumed record, so you can inspect the messages at the front of the lag rather than looking at a number in isolation.

Kafma reads the group’s committed positions without joining the group. Watching it therefore does not add a member or trigger a rebalance in the application you are debugging.

This closes an important gap in the workflow: the message contents and the consumer’s progress are visible in the same investigation.

Reuse messages across topics and clusters

Once you find a representative record, you should not have to reconstruct it by hand.

Replay to topic opens a producer for the current topic or another topic in the same cluster and fills in the record’s key, value, and headers. The result is an editable draft, not an immediate write. You can change the payload, choose a partition or schema version, and review everything before producing it.

For a quick cross-cluster test, Forward to cluster sends one record or the currently loaded, filtered set to the same topic name on another cluster. Kafma blocks forwarding to clusters marked as Production. Schema-backed records are forwarded as decoded JSON or text rather than retaining their original Schema Registry encoding. Export is also available when the current records need to be saved as JSON or CSV for a test or shared investigation.

Replay and forwarding solve different problems. Replay creates a controlled, editable test case; forwarding moves messages directly. Forwarding does not copy Schema Registry subjects or topic configuration. When the destination needs the topic’s schemas, configuration, and a selected range of source data, use Data Clone instead.

See Replay and export Kafka messages for the exact behavior of each action.

Keep a producer loop running while you debug

Some consumer behavior is difficult to test with a single record. An application may aggregate a stream, process periodic events, or fail only after several inputs arrive.

The Console producer can send one message manually or run on a timed Loop. A loop can repeat the current key, value, and headers at a fixed interval. If the key or value is bound to a Schema Registry subject, the loop can regenerate that field on every tick with a fresh schema-compatible payload.

The loop continues while you switch to other Console tabs or pages in the same cluster. You can keep generating test traffic while inspecting another topic, checking a consumer group, or following a different part of the debugging flow. Return to the source tab to see the sent count or stop the loop. Closing the source tab stops the loop.

This makes Loop useful for simulating a producer that reports continuously while you work on the consumer side. It is intended for repeatable functional testing, not throughput benchmarking.

See Producing messages for payload encodings, schema generation, partitions, acknowledgments, and Loop controls.

More than a Kafka viewer

The value of one workspace is not simply that several controls fit on one screen. It is that the investigation remains intact:

The topic stays open as a tab.
The message stream and producer share the same context.
Topic details and consumer state are available without leaving the Console.
A producer loop can continue while you inspect other resources.
Each cluster preserves its own set of open investigations.

A conventional Kafka viewer can show which records are in a topic. For debugging, the useful unit is larger: Kafma keeps the topic, message metadata, schema context, consumer state, producer, and active investigation together even when you switch between clusters.

Download Kafma for macOS, Windows, or Linux, or read the Console documentation for a complete reference.

Introducing Kafma: A Desktop Kafka UI for Faster Debugging

KafmaKarma — Sat, 01 Aug 2026 13:55:37 +0000

Kafma is a desktop Kafka GUI client and IDE for macOS, Windows, and Linux. It keeps the consume–inspect–produce loop in one workspace and lets you clone remote topics—with their configurations, schemas, and a selected range of messages—into a local cluster for testing, with optional field masking.

Disclosure: I'm the creator of Kafma. It is closed-source software with Free and Pro plans.

Debug Kafka in one workspace

A topic opens as a tab in Kafma. Messages appear at the top, while the producer stays open at the bottom. You can send a message, inspect the result, change the payload, and send it again without leaving the topic tab.

Each topic keeps its own state, so several investigations can remain open at the same time. Live tails new messages as they arrive. Watch Group opens a bounded window around a consumer group's committed positions, showing consumed and pending records alongside member status, lag, and backlog.

Kafma automatically decodes Avro and Protobuf records using the configured Schema Registry. Expand a record to inspect its decoded key and value, headers, timestamp, partition, offset, and schema metadata. From there, you can replay it to a topic in the same cluster. Pro also supports forwarding records to another cluster and exporting results as JSON or CSV.

The producer handles JSON, text, Avro, and Protobuf. It can send a single record or run on a timed loop. With schema-based auto-generation enabled, each interval creates a fresh schema-compatible payload. That makes it possible to exercise a consumer and inspect the results from the same topic tab.

Clone remote topics for local testing

Some failures depend on the configuration, schemas, and records in a remote environment. Recreating that context locally often requires exports, scripts, and manual setup.

With Pro, Kafma's Data Clone makes a one-time, bounded copy between connected clusters. The Free plan supports same-cluster clones. Select one or more topics, choose a time window or offset range, and copy the topic structure, the schema versions required by the selected records, and the records themselves. You can also copy structure and schemas without data when you want to produce your own test records.

This is intended for focused debugging and test setup, not continuous cluster replication. Data Clone does not join a consumer group, commit offsets, or modify topics, messages, or schemas on the source. On the target, Kafma can create missing topics and register the required schemas; copied records are appended rather than overwriting existing records.

Optional masking rules transform selected fields in schema-backed message values before records are written to the target cluster. A pre-flight plan shows the selected range, schemas, masking rules, and planned target operations before the clone begins. Together, those controls make it practical to bring the context around a remote failure into a local or test cluster without writing back to the source.

Review schema changes before publishing

Kafma includes a Schema Registry workspace for Avro, JSON Schema, and Protobuf. When you create a new version, the selected registered version appears beside an editable draft. The diff updates as you type, making added and removed lines visible before the new version reaches the registry.

You can browse version history, generate mock data from a registered schema, and validate a JSON payload against a selected version in Schema Lab. When you publish a draft, Kafma runs the Schema Registry compatibility check as part of the same workflow and shows the result alongside the changes.

More than a message browser

The same desktop app also covers the operational work around a debugging session. You can inspect consumer-group members, assignments, committed offsets, and partition lag; reset or delete offsets; and manage topics while viewing broker and topic configuration.

Connections can be marked read-only to block Kafka and Schema Registry writes. Selecting Production turns on read-only mode automatically, though you can turn it off when writes are required. A Data Clone into a writable production target also requires typed confirmation. Kafma keeps a local activity log for operations performed in the app and can revert supported configuration and offset changes.

Kafma connects to self-hosted Kafka, Confluent Cloud, and Amazon MSK, with support for SASL, mTLS, OAuth, and AWS IAM authentication.

Available on macOS, Windows, and Linux

Kafma connects directly from your device to the Kafka clusters and Schema Registries you configure. Kafka data does not pass through Kafma's servers, and the app has no accounts, analytics, or telemetry.

The Free plan includes the core produce, consume, live-tail, schema-decoding, topic and consumer-group management, Schema Registry workflows, and same-cluster cloning. Pro adds unlimited clusters and tabs, cross-cluster cloning and forwarding, message export, and a higher Console message limit. Download Kafma to get started with the Free plan, or see pricing for Pro.

The Kafma 1.0 changelog contains the complete feature list and more screenshots.

What is the most frustrating part of your Kafka debugging workflow today?