How I wrote a Go message broker with a throughput of a million messages per second

I built HermitMQ entirely in Go. The main feature is ditching heavy wrappers like JSON in favor of a custom 29 byte binary protocol. Additionally, data transmission over the network uses a direct file to socket copy mechanism. I will go into detail about the architecture, data storage approaches, benchmark numbers, and show how it is implemented in code.

The full source code for the HermitMQ project is available on GitHub: https://github.com/ekhidirov/hermitmq

The problem with standard brokers and the cost of serialization

When the message counter exceeds hundreds of thousands per second, the main problem for a Go developer is the garbage collector. If every message is parsed via standard JSON, the application starts allocating a massive number of small objects in memory. The GC wakes up too frequently, eating up CPU time and causing network latency spikes.

To avoid triggering the garbage collector at every turn, I completely abandoned standard serialization libraries. Every message is packed into a custom header of exactly 29 bytes. In code, the message structure looks extremely simple:

type Message struct {
    Magic       byte
    Timestamp   uint64
    Offset      uint64
    KeySize     uint32
    PayloadSize uint32
    RecordCount uint32
    Key         []byte
    Payload     []byte
}

The first byte is a magic number for version checking and instantly discarding bad packets. Next come 8 bytes for the timestamp in nanoseconds and 8 bytes for the offset, which the broker fills in itself to maintain message order. Then come the key and payload sizes, 4 bytes each. Finally, 4 bytes are reserved for the record count to support batching. The broker reads the stream using the binary package and reuses buffers via sync.Pool. As a result, under standard loads, we achieve practically zero memory allocation.

Being honest about allocations and plans for zero serialization

To be completely honest: although the broker is incredibly frugal under standard loads, a memory management compromise still remains.

An absolute victory over allocations and the GC has been achieved on the side of serving data to consumers, thanks to io.CopyN (the very zero copy approach I will discuss below). However, on the receiving end from producers, the Decode method still dynamically allocates slices for the payload size (make([]byte, m.PayloadSize)).

In future releases, I plan to completely eliminate this structure. We will move towards the concept of zero serialization (like in flatbuffers) and micro optimizations with branchless logic in hot loops. The broker will access byte offsets directly in the network buffer, without unpacking them into Go structs at all. This will finally eliminate the GC from the message processing path.

Direct data transfer at the kernel level

Copying data from the OS kernel into user space and back to the network interface is very expensive. It is double the work for the CPU and RAM. In HermitMQ, consumers receive data directly.

When a client requests data, the broker looks up the offset in the log file. Instead of reading the file into the program memory, it uses the built in io.CopyN function.

bytesToSend := int64(HeaderSize) + int64(keySize) + int64(payloadSize)
reader := io.NewSectionReader(walFile, position, bytesToSend)
_, err = io.CopyN(conn, reader, bytesToSend)
Data flows straight from the physical file on disk into the consumer network socket. Latencies remain minimal, and the CPU does not strain itself moving bytes around.

Data storage, mmap, and compaction

All data is written to a write ahead log (WAL). The log is divided into segments. Each segment has a .wal file with raw data and an .idx file to map offsets to physical bytes on the hard drive.

For fast searching, indices are mapped directly into RAM via mmap. The broker performs a binary search in memory and instantly finds the required byte. Background log compaction is also implemented here. It deduplicates messages by key, leaving only the most relevant data.

Surviving crashes: the torn page recovery mechanism

Any database developer knows: a server can crash at any millisecond due to a power failure or a hard kill 9 signal. If this happens during a log write, a half written chunk of data (a torn page) will be left at the end of the file.

HermitMQ implements an automatic recovery mechanism. On startup, the broker sequentially scans the WAL files. If it sees that the expected message size in the header exceeds the physical size of the file on disk, it understands that the write was interrupted. Instead of panicking and crashing the entire segment, the broker safely truncates this corrupted tail at the hardware level using os.Truncate.

if currentPos+msgSize > totalFileSize {
    log.Printf("PARTIAL MESSAGE DETECTED. Truncating tail.")
    break
}

// ...
if err := os.Truncate(walPath, currentPos); err != nil {
    return err
}

This guarantees that the broker will boot up with a completely consistent state, even after the most severe operating system crash.

Lock sharding and scaling

Storing consumer group offsets under a single global mutex is a bad idea when dealing with thousands of connections. Threads will start queuing up and kill all performance. Therefore, the offset store is split into 256 independent shards.

The broker takes a string key, runs it through a hashing algorithm, and routes it to the correct shard. Here is the implementation of this hash:

func getShardIndex(key string) uint8 {
    var hash uint32 = 2166136261
    for i := 0; i < len(key); i++ {
        hash ^= uint32(key[i])
        hash *= 16777619
    }
    return uint8(hash & (OffsetShardCount - 1))
}

Incoming requests are evenly spread across the allocated memory. Producers and consumers do not block each other, and the broker scales perfectly across CPU cores.

Developer experience: on demand routing

Working with hardcore systems is often painful due to excessive infrastructure bureaucracy, you usually have to hit an API in advance to configure topics and partitions. I solved this problem through on demand routing.

If a producer or consumer knocks on the broker and requests a non existent topic, the broker does not reject the connection with an error. Instead, it transparently creates the required folder and file structure on the fly under an RWMutex lock.

b.mu.RLock()
_, exists := b.topics[topicName]
b.mu.RUnlock()

if !exists {
    slog.Info("auto-creating new topic on demand", "topic", topicName)
    b.CreateTopic(topicName, 1)
}

The client simply needs to start writing data, the entire physical structure under the hood will spin up on its own.

Smart batching on the client

Throughput depends not only on the server but also on the clients. In the project, the test producer uses Go channels and tickers to group messages.

const (
    ProducersCount    = 10
    MessagesPerWorker = 100000
    BatchSize         = 100
    LingerTime        = 100 * time.Millisecond
)

Workers buffer messages and flush them to the TCP socket only when two conditions are met: a batch of 100 records has accumulated, or a 100 millisecond wait timer expires. This glues a hundred logical records into a single dense header and a single system write call.

Benchmark numbers and stress test results

To measure the real throughput of the pipeline, I set up a cluster of 10 parallel producers and a separate consumer. The test scenario pumps 1,000,000 messages through the broker.

Producers generate 10 threads of 100,000 messages each. The consumer fetches the incoming stream, parses the 29 byte headers, and calculates the net throughput over time.

totalMB := float64(logicalCount * 1812) / 1024 / 1024
fmt.Printf("Throughput: %.0f msgs/sec (Approx %.0f MB/sec)\n", 
    float64(logicalCount)/elapsed.Seconds(), 
    totalMB/elapsed.Seconds())

With a batch of 1,000,000 messages, the entire volume of data is pumped through in a matter of seconds. Thanks to zero copy and the complete absence of intermediate memory allocations, the broker delivers hundreds of thousands of messages per second and utilizes tens of megabytes of traffic per second on a single instance, bottlenecked solely by the physical speed of the disk bus and the local TCP stack.

Overload protection and security

The broker is protected against memory exhaustion attacks by strict limits. The payload size is strictly validated before memory is allocated; by default, the limit is set to 50 megabytes.

maxBytes := uint32(maxPayloadMB * 1024 * 1024)
if m.PayloadSize > maxBytes {
    return fmt.Errorf("security policy violation: payload too large")
}

If the size exceeds the limit or a packet arrives with a bad starting byte, the broker does not crash with an out of memory error; it simply drops the malicious connection.

Graceful shutdown

High delivery speed is meaningless if data stuck in OS buffers is lost during a restart. That is why the broker knows how to respond correctly to system interrupt signals.

Interception of SIGINT and SIGTERM signals is implemented in main.go. Upon receiving a signal, the broker stops accepting new connections, closes the TCP listener, and neatly iterates through all topics and partitions. For each open segment, Sync() and Close() are forcibly called, flushing all file descriptors to disk.

sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
<-sigChan
slog.Info("received shutdown signal, shutting down...")

listener.Close()
b.Close()

Conclusion

Writing your own message broker is excellent practice for understanding how high load systems work at a low level. The architecture allows implementing direct memory management, data streaming, and concurrency without unnecessary headaches. Ditching fat frameworks gives full control over every allocated byte and provides predictable performance under any load.