DEV Community: sloth

A pure-Go Kafka broker: protocol implementation, consumer groups, and transactional semantics

sloth — Fri, 23 Oct 2026 07:30:53 +0000

I’ve always found Kafka to be one of the most interesting distributed systems to study because the actual complexity is not in the “message bus” idea itself, but in the protocol semantics, state transitions, and client compatibility requirements underneath it.

This repository stood out to me because it is a pure-Go Kafka broker that implements the Kafka wire protocol directly, rather than wrapping a JVM-based broker. It is a concrete implementation of many of the internals that Kafka clients normally hide behind a simpler API.

Repository: https://github.com/sloth-os/kafka

Why this matters

Kafka is often treated as a black box: producers write, consumers read, groups coordinate, offsets commit, and the broker handles replication and retention. But the real engineering challenge is in the details:

request/response framing
version negotiation
flexible vs non-flexible protocol encoding
message batch encoding and decoding
consumer group state machines
transactional producer semantics
offset management and isolation behavior
compatibility with external clients and protocol expectations

This repo implements a substantial subset of the Kafka protocol in Go and does so in a way that is explicit and inspectable. That makes it valuable not only as a broker implementation but as a reference for understanding Kafka internals.

The project supports a broad range of Kafka APIs, including:

Produce / Fetch
Metadata / ListOffsets
OffsetCommit / OffsetFetch
FindCoordinator / JoinGroup / Heartbeat / LeaveGroup / SyncGroup
DescribeGroups / ListGroups
ApiVersions
CreateTopics / DeleteTopics / CreatePartitions
InitProducerId / AddPartitionsToTxn / AddOffsetsToTxn / EndTxn / TxnOffsetCommit

It also includes:

a full in-memory consumer-group coordinator implementing the classic group state machine
transactional produce flow support
in-memory topic metadata and partition logs
handling of both legacy and flexible protocol encodings
compatibility with real Kafka clients

The testing story is also important

The project does not only rely on unit tests or mocks. It uses end-to-end tests against real Kafka client libraries:

Sarama
kafka-go
confluent-kafka-go

That is especially valuable in a protocol-heavy system like Kafka. A lot of correctness issues do not appear in isolated tests—they appear when real clients negotiate protocol versions, deserialize responses, and exercise edge cases under actual producer/consumer patterns.

Running end-to-end tests against real client libraries gives much stronger confidence that the implementation matches real-world Kafka behavior, not just a simplified internal model.

Build One Bot, Reach Every Chat: A Universal WebSocket Bridge for IM Channels

sloth — Wed, 19 Aug 2026 03:47:57 +0000

Every chat platform reinvents the wheel. WeChat has its own API, Lark has another, Telegram another still — and if you're building a bot, an agent, or any service that needs to talk to people across IM channels, you end up writing and maintaining a separate integration for each one. The plumbing differs, the auth flows differ, the message formats differ, and every new channel means another chunk of bespoke code in your app.

I got tired of that, so I built OpenClaw Bridge — a universal WebSocket bridge server that sits between your client and any number of IM backends. The idea is simple: your app speaks one protocol over a single WebSocket, and the bridge translates that into whatever each backend channel expects. Here's the shape of it:

Client ──[WebSocket]──▶ Bridge Server ──[ChannelPlugin API]──▶ Backend Channels

The key design decision is that the bridge contains zero plugin-specific code. It doesn't know how WeChat logs in, how Lark formats media, or how Telegram handles reply threading. Instead, it works with any channel that implements a standard ChannelPlugin interface. Install a plugin, drop its credentials into config.json, and it just works — WeChat, Lark, Telegram, Mattermost, IRC, QQ, WhatsApp, and whatever comes next.

In this post I'll walk through why a bridge like this is worth building, how the envelope protocol keeps clients channel-agnostic, and how a subscription model lets multiple clients share one set of connected accounts without tripping over each other. If you've ever copy-pasted your fifth nearly-identical IM integration, this one's for you.

For more information, visit https://github.com/sloth-os/openclaw-bridge

mm-gateway: One Provider-Neutral API for Image, Video, and Music Generation

sloth — Fri, 14 Aug 2026 09:32:30 +0000

Every generative-AI app I worked on hit the same wall. Pick a backend — OpenAI for images, Volcengine for video, Mureka for music — and before long the app is full of that provider's SDK quirks: its field names, its sync-vs-async polling loop, its error shapes. Then if want to swap one backend, or add a second one for failover, and it's a rewrite.

mm-gateway is an open-source Python gateway that sits in front of that mess. One provider-neutral contract — over 13 backends:
OpenAI · Google · xAI · DashScope · Volcengine · Flux · Stability · ElevenLabs · MiniMax · Mureka · ACE-Step · OpenRouter · UdioAPI

The idea is simple: provider wire formats never appear in application code. Every request goes through a strict, modality-specific envelope — an ordered list of typed input parts plus provider-neutral parameters — and each backend adapter translates that to its native SDK or REST shape.

For more information, visit https://github.com/sloth-os/mm-gateway