A pure-Go Kafka broker: protocol implementation, consumer groups, and transactional semantics

I’ve always found Kafka to be one of the most interesting distributed systems to study because the actual complexity is not in the “message bus” idea itself, but in the protocol semantics, state transitions, and client compatibility requirements underneath it.

This repository stood out to me because it is a pure-Go Kafka broker that implements the Kafka wire protocol directly, rather than wrapping a JVM-based broker. It is a concrete implementation of many of the internals that Kafka clients normally hide behind a simpler API.

Repository: https://github.com/sloth-os/kafka

Why this matters

Kafka is often treated as a black box: producers write, consumers read, groups coordinate, offsets commit, and the broker handles replication and retention. But the real engineering challenge is in the details:

• request/response framing
• version negotiation
• flexible vs non-flexible protocol encoding
• message batch encoding and decoding
• consumer group state machines
• transactional producer semantics
• offset management and isolation behavior
• compatibility with external clients and protocol expectations

This repo implements a substantial subset of the Kafka protocol in Go and does so in a way that is explicit and inspectable. That makes it valuable not only as a broker implementation but as a reference for understanding Kafka internals.

The project supports a broad range of Kafka APIs, including:

• Produce / Fetch
• Metadata / ListOffsets
• OffsetCommit / OffsetFetch
• FindCoordinator / JoinGroup / Heartbeat / LeaveGroup / SyncGroup
• DescribeGroups / ListGroups
• ApiVersions
• CreateTopics / DeleteTopics / CreatePartitions
• InitProducerId / AddPartitionsToTxn / AddOffsetsToTxn / EndTxn / TxnOffsetCommit

It also includes:

• a full in-memory consumer-group coordinator implementing the classic group state machine
• transactional produce flow support
• in-memory topic metadata and partition logs
• handling of both legacy and flexible protocol encodings
• compatibility with real Kafka clients

The testing story is also important

The project does not only rely on unit tests or mocks. It uses end-to-end tests against real Kafka client libraries:

• Sarama
• kafka-go
• confluent-kafka-go

That is especially valuable in a protocol-heavy system like Kafka. A lot of correctness issues do not appear in isolated tests—they appear when real clients negotiate protocol versions, deserialize responses, and exercise edge cases under actual producer/consumer patterns.

Running end-to-end tests against real client libraries gives much stronger confidence that the implementation matches real-world Kafka behavior, not just a simplified internal model.