DEV Community

Cover image for kafka-core 2.5: two weeks of hardening a TypeScript Kafka client
Mykhailo Toporkov 🇺🇦
Mykhailo Toporkov 🇺🇦

Posted on

kafka-core 2.5: two weeks of hardening a TypeScript Kafka client

#kafka #typescript #node #opensource

Over the last two weeks I spent most of my time on the unglamorous side of kafka — a TypeScript Apache Kafka client for Node.js that talks the wire protocol directly (no Java client, no native bindings). Instead of features, this round was about edge cases: what happens when a broker disappears mid-refresh, when a transaction commit fails, or when a consumer group rebalances while a seek is pending.

The result is kafka-core 2.5.1, kafka-cli 1.10.1, kafka-studio 1.2.1 and kafka-config 1.2.1 — 60+ fixes in total. Here are the highlights.

Coming from kafkajs? There's now a migration guide

kafkajs hasn't shipped a release since 2023. kafka-core keeps the same API shape — new Kafka({ clientId, brokers }), producer(), consumer(), eachMessage — and adds Kafka 4.x, KIP-848, share groups, and built-in Snappy/LZ4/ZSTD compression. Most code moves over with an import change:

// before
import { Kafka } from 'kafkajs'
// after
import { Kafka } from '@cookiemonsterdev/kafka-core'

const kafka = new Kafka({ clientId: 'my-app', brokers: ['localhost:9092'] })
Enter fullscreen mode Exit fullscreen mode

The full guide covers the differences: Migrating from kafkajs.

Producer: transactions and idempotence that hold up under failure

Most of the producer work went into making exactly-once semantics behave correctly when things go wrong:

  • Sequence numbers survive retries. Produce retries now keep their original sequence numbers, and when the broker reports an unknown sequence the producer bumps its epoch instead of reusing stale sequences.
  • Epoch bumps wait for in-flight sends to drain, so no request is sent with a half-updated producer state.
  • Retries stop at the delivery deadline rather than continuing past the user's timeout.
  • Failed sends make a transaction abort-only, and sends rejected in that state now explain why, instead of a generic error.
  • EndTxn is retried, a failed commit can be resent, and a rejected EndTxn no longer wedges the transaction.
  • Fenced producers reject further calls with a clear error.

Consumer: KIP-848 and rebalances

  • Offset commits use the latest KIP-848 member epoch, avoiding stale-epoch rejections after heartbeats.
  • Retained partitions keep their offsets across reassignment.
  • A seek() is held until its partition's current batch finishes, so it isn't overwritten by an in-progress batch.
  • Prefetched batches behind the current position are dropped.
  • With read_committed, every aborted transaction a record reaches is opened, so aborted data is filtered correctly across batch boundaries.

Share groups (KIP-932)

Share consumers got a lot of attention:

  • Only acquired records are delivered.
  • ShareFetch partition errors are handled per partition — a fatal error on one partition releases the healthy records instead of losing them.
  • Retriable fetch errors back off; non-retriable ones crash loudly. ShareFetch retries on a fresh session when needed.

Networking and cluster metadata

  • Happy Eyeballs fixes: remaining addresses are tried on error, and the next attempt starts immediately on fast failure.
  • Concurrent connection attempts to the same broker are deduplicated.
  • Sends are rejected while disconnecting, with a bounded drain wait; errors thrown by instrumentation listeners are isolated.
  • The receive buffer is compacted once per socket read and right-sized — fewer copies on hot paths.
  • Broker set updates are serialized, refreshes can no longer resurrect removed brokers, and metadata staleness is re-checked inside the update queue to avoid redundant refreshes.
  • withBroker falls through to the next broker when one is unreachable.

Protocol and admin

  • Tagged fields are skipped by their raw size, ShareAcknowledge partition tagged fields are read, and ListOffsets v4 decodes the leader epoch.
  • Empty partition lists stay distinct from null.
  • Admin retries are smarter: group deletes retry on coordinator errors, describe re-resolves coordinators, and deleteTopicRecords re-resolves leaders on retry.

CLI, config and Studio

  • kafka-cli: SIGINT/SIGTERM exits right away (a second signal forces it), confirmation prompts require a TTY on stdin, integer flags reject non-integers, and completion/init/doctor skip config loading.
  • kafka-config: config discovery stops at untrusted directories — a small but meaningful security fix — and load errors now include the cause.
  • kafka-studio: one profile is pinned per request, tail reconnect loops stop after an error, connections and jobs close cleanly on shutdown, and message reads respect the requested limit.

Try it 

npm install @cookiemonsterdev/kafka-core
Enter fullscreen mode Exit fullscreen mode

Issues, feedback and stars are all very welcome. If you're stuck on kafkajs and need Kafka 4.x features, I'd love to hear how the migration goes.

Top comments (0)