DEV Community: Erkin Khidirov

How I wrote a Go message broker with a throughput of a million messages per second

Erkin Khidirov — Tue, 18 Aug 2026 21:15:58 +0000

I built HermitMQ entirely in Go. The main feature is ditching heavy wrappers like JSON in favor of a custom 29 byte binary protocol. Additionally, data transmission over the network uses a direct file to socket copy mechanism. I will go into detail about the architecture, data storage approaches, benchmark numbers, and show how it is implemented in code.

The full source code for the HermitMQ project is available on GitHub: https://github.com/ekhidirov/hermitmq

The problem with standard brokers and the cost of serialization

When the message counter exceeds hundreds of thousands per second, the main problem for a Go developer is the garbage collector. If every message is parsed via standard JSON, the application starts allocating a massive number of small objects in memory. The GC wakes up too frequently, eating up CPU time and causing network latency spikes.

To avoid triggering the garbage collector at every turn, I completely abandoned standard serialization libraries. Every message is packed into a custom header of exactly 29 bytes. In code, the message structure looks extremely simple:

type Message struct {
    Magic       byte
    Timestamp   uint64
    Offset      uint64
    KeySize     uint32
    PayloadSize uint32
    RecordCount uint32
    Key         []byte
    Payload     []byte
}

The first byte is a magic number for version checking and instantly discarding bad packets. Next come 8 bytes for the timestamp in nanoseconds and 8 bytes for the offset, which the broker fills in itself to maintain message order. Then come the key and payload sizes, 4 bytes each. Finally, 4 bytes are reserved for the record count to support batching. The broker reads the stream using the binary package and reuses buffers via sync.Pool. As a result, under standard loads, we achieve practically zero memory allocation.

Being honest about allocations and plans for zero serialization

To be completely honest: although the broker is incredibly frugal under standard loads, a memory management compromise still remains.

An absolute victory over allocations and the GC has been achieved on the side of serving data to consumers, thanks to io.CopyN (the very zero copy approach I will discuss below). However, on the receiving end from producers, the Decode method still dynamically allocates slices for the payload size (make([]byte, m.PayloadSize)).

In future releases, I plan to completely eliminate this structure. We will move towards the concept of zero serialization (like in flatbuffers) and micro optimizations with branchless logic in hot loops. The broker will access byte offsets directly in the network buffer, without unpacking them into Go structs at all. This will finally eliminate the GC from the message processing path.

Direct data transfer at the kernel level

Copying data from the OS kernel into user space and back to the network interface is very expensive. It is double the work for the CPU and RAM. In HermitMQ, consumers receive data directly.

When a client requests data, the broker looks up the offset in the log file. Instead of reading the file into the program memory, it uses the built in io.CopyN function.

bytesToSend := int64(HeaderSize) + int64(keySize) + int64(payloadSize)
reader := io.NewSectionReader(walFile, position, bytesToSend)
_, err = io.CopyN(conn, reader, bytesToSend)
Data flows straight from the physical file on disk into the consumer network socket. Latencies remain minimal, and the CPU does not strain itself moving bytes around.

Data storage, mmap, and compaction

All data is written to a write ahead log (WAL). The log is divided into segments. Each segment has a .wal file with raw data and an .idx file to map offsets to physical bytes on the hard drive.

For fast searching, indices are mapped directly into RAM via mmap. The broker performs a binary search in memory and instantly finds the required byte. Background log compaction is also implemented here. It deduplicates messages by key, leaving only the most relevant data.

Surviving crashes: the torn page recovery mechanism

Any database developer knows: a server can crash at any millisecond due to a power failure or a hard kill 9 signal. If this happens during a log write, a half written chunk of data (a torn page) will be left at the end of the file.

HermitMQ implements an automatic recovery mechanism. On startup, the broker sequentially scans the WAL files. If it sees that the expected message size in the header exceeds the physical size of the file on disk, it understands that the write was interrupted. Instead of panicking and crashing the entire segment, the broker safely truncates this corrupted tail at the hardware level using os.Truncate.

if currentPos+msgSize > totalFileSize {
    log.Printf("PARTIAL MESSAGE DETECTED. Truncating tail.")
    break
}

// ...
if err := os.Truncate(walPath, currentPos); err != nil {
    return err
}

This guarantees that the broker will boot up with a completely consistent state, even after the most severe operating system crash.

Lock sharding and scaling

Storing consumer group offsets under a single global mutex is a bad idea when dealing with thousands of connections. Threads will start queuing up and kill all performance. Therefore, the offset store is split into 256 independent shards.

The broker takes a string key, runs it through a hashing algorithm, and routes it to the correct shard. Here is the implementation of this hash:

func getShardIndex(key string) uint8 {
    var hash uint32 = 2166136261
    for i := 0; i < len(key); i++ {
        hash ^= uint32(key[i])
        hash *= 16777619
    }
    return uint8(hash & (OffsetShardCount - 1))
}

Incoming requests are evenly spread across the allocated memory. Producers and consumers do not block each other, and the broker scales perfectly across CPU cores.

Developer experience: on demand routing

Working with hardcore systems is often painful due to excessive infrastructure bureaucracy, you usually have to hit an API in advance to configure topics and partitions. I solved this problem through on demand routing.

If a producer or consumer knocks on the broker and requests a non existent topic, the broker does not reject the connection with an error. Instead, it transparently creates the required folder and file structure on the fly under an RWMutex lock.

b.mu.RLock()
_, exists := b.topics[topicName]
b.mu.RUnlock()

if !exists {
    slog.Info("auto-creating new topic on demand", "topic", topicName)
    b.CreateTopic(topicName, 1)
}

The client simply needs to start writing data, the entire physical structure under the hood will spin up on its own.

Smart batching on the client

Throughput depends not only on the server but also on the clients. In the project, the test producer uses Go channels and tickers to group messages.

const (
    ProducersCount    = 10
    MessagesPerWorker = 100000
    BatchSize         = 100
    LingerTime        = 100 * time.Millisecond
)

Workers buffer messages and flush them to the TCP socket only when two conditions are met: a batch of 100 records has accumulated, or a 100 millisecond wait timer expires. This glues a hundred logical records into a single dense header and a single system write call.

Benchmark numbers and stress test results

To measure the real throughput of the pipeline, I set up a cluster of 10 parallel producers and a separate consumer. The test scenario pumps 1,000,000 messages through the broker.

Producers generate 10 threads of 100,000 messages each. The consumer fetches the incoming stream, parses the 29 byte headers, and calculates the net throughput over time.

totalMB := float64(logicalCount * 1812) / 1024 / 1024
fmt.Printf("Throughput: %.0f msgs/sec (Approx %.0f MB/sec)\n", 
    float64(logicalCount)/elapsed.Seconds(), 
    totalMB/elapsed.Seconds())

With a batch of 1,000,000 messages, the entire volume of data is pumped through in a matter of seconds. Thanks to zero copy and the complete absence of intermediate memory allocations, the broker delivers hundreds of thousands of messages per second and utilizes tens of megabytes of traffic per second on a single instance, bottlenecked solely by the physical speed of the disk bus and the local TCP stack.

Overload protection and security

The broker is protected against memory exhaustion attacks by strict limits. The payload size is strictly validated before memory is allocated; by default, the limit is set to 50 megabytes.

maxBytes := uint32(maxPayloadMB * 1024 * 1024)
if m.PayloadSize > maxBytes {
    return fmt.Errorf("security policy violation: payload too large")
}

If the size exceeds the limit or a packet arrives with a bad starting byte, the broker does not crash with an out of memory error; it simply drops the malicious connection.

Graceful shutdown

High delivery speed is meaningless if data stuck in OS buffers is lost during a restart. That is why the broker knows how to respond correctly to system interrupt signals.

Interception of SIGINT and SIGTERM signals is implemented in main.go. Upon receiving a signal, the broker stops accepting new connections, closes the TCP listener, and neatly iterates through all topics and partitions. For each open segment, Sync() and Close() are forcibly called, flushing all file descriptors to disk.

sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
<-sigChan
slog.Info("received shutdown signal, shutting down...")

listener.Close()
b.Close()

Conclusion

Writing your own message broker is excellent practice for understanding how high load systems work at a low level. The architecture allows implementing direct memory management, data streaming, and concurrency without unnecessary headaches. Ditching fat frameworks gives full control over every allocated byte and provides predictable performance under any load.

How Messengers Actually Encrypt Messages (End-to-End)

Erkin Khidirov — Thu, 29 May 2025 22:34:20 +0000

Hi everyone!

I've written an article and I'm publishing it prematurely. Initially, I planned to write it after the project was completed, but since there are still a couple of months left until the end, I decided not to waste time and write the article while the information is still fresh in my mind. Besides, I'm mostly writing this for myself. :) In one of my latest projects, which I'm developing as open source, I implemented end-to-end encryption—similar to how WhatsApp or Telegram do it, for example.

In this article, we'll dive into the implementation of client-side message encryption using JavaScript and the Web Crypto API, breaking down a practical example that will be at the very end of the article.

Let's start by saying that if you're a complete beginner in cryptography, understanding what's written here might not be easy. Even with 10 years of development experience, I had to scratch my head a bit—everything happening here is pure mathematics, which we won't be discussing in this article :) The easily impressed might think it's magic :)

If I were to briefly explain the essence of end-to-end encryption without complex words and terms:

The Magic of Encryption in Three Keys

Three keys are the foundation upon which end-to-end encryption is built. Pay attention to the key in the center – this is important.

The foundation on which everything rests is these three keys. Refer back to this section if something is unclear.

Private Key: Stored (in encrypted form).
Public Key: Accessible to everyone.
Shared Secret / Symmetric Key: Generated based on your private key + your contact's public key. This is the key used for the direct encryption and decryption of messages.

The combination of your private key + your contact's public key allows you to obtain the shared secret key (in our example below, this will be an AES key). Thanks to this shared secret key, you can encrypt and decrypt messages.

Private and public keys can be stored in a database, but there's a nuance with the private key. The private key itself is not recommended to be stored in plain text; it needs to be additionally encrypted with the user's password or any other keyword (within a messenger, this is typically the user's password). We store the public key in plain text.

The shared secret key (the one referred to as this.aesKey in the code below) is not stored in the database. It is generated (calculated) each time a chat with a specific contact is initialized. This might cause confusion: how will we decrypt messages if this key is not stored but generated anew? This is where the "magic" of asymmetric encryption and key exchange protocols lies.

The shared key is "Your private key" + "Contact's public key."

When you open a chat with a contact, your client recalculates this shared secret key, as you already know, like this: (Your private key + Your contact's public key = Shared secret key). With this key, you encrypt new messages and decrypt all previous messages in this chat, as they were encrypted with the same shared secret key. You can wrack your brains for a long time to no avail until you understand asymmetric encryption and key exchange protocols.

Asymmetric Encryption and ECDH

Asymmetric encryption uses a pair of keys: public and private. The public key can be freely distributed, while the private key must be kept secret by its owner, meaning in an encrypted form.

ECDH (Elliptic Curve Diffie-Hellman) is a key exchange protocol based on elliptic curve mathematics. It allows two parties, each possessing their own ECDH key pair (private and public), to establish a shared secret key over an insecure channel. Importantly, a third party, even if they intercept their public keys, cannot compute this shared secret. Our example uses the P-256 curve – a popular and reliable standard.

I think few understood what they just read. All you need to understand at this stage is that the technology works :) Later, the puzzle will come together, perhaps after a reread. And now, a bit about built-in browser technologies.

Web Crypto API

The Web Crypto API is a JavaScript interface built into browsers that provides access to low-level cryptographic primitives. It allows performing operations such as hashing, signature generation, encryption, and decryption. Using the Web Crypto API is preferable to third-party libraries for basic cryptographic operations, as it is often hardware-accelerated and thoroughly vetted for security. All Web Crypto API operations are asynchronous and return a Promise.

Now let's move on to the practical analysis. I've created a ChatCrypto class, which we'll examine in more detail:

class ChatCrypto {

  constructor(myPrivateKeyBase64, theirPublicKeyBase64) {
    this.myPrivateKeyBase64 = myPrivateKeyBase64;
    this.theirPublicKeyBase64 = theirPublicKeyBase64;
    this.aesKey = null; // The shared symmetric AES key will be stored here
  }

  static base64ToArrayBuffer(base64) {
    const binary = atob(base64);
    const bytes = new Uint8Array(binary.length);
    for (let i = 0; i < binary.length; i++) {
      bytes[i] = binary.charCodeAt(i);
    }
    return bytes.buffer;
  }

  static arrayBufferToBase64(buffer) {
    const bytes = new Uint8Array(buffer);
    let binary = '';
    for (let b of bytes) {
      binary += String.fromCharCode(b);
    }
    return btoa(binary);
  }

  init() {

    // Convert keys from Base64 to ArrayBuffer
    const privateRaw = ChatCrypto.base64ToArrayBuffer(this.myPrivateKeyBase64);
    const publicRaw = ChatCrypto.base64ToArrayBuffer(this.theirPublicKeyBase64);

    // Note: The following lines for parsing publicRaw into x and y coordinates,
    // and assembling uncompressedPoint might be specific to a particular format
    // for representing a "raw" public key. If publicRaw is already in SPKI format,
    // they might not be necessary, as crypto.subtle.importKey("spki", ...)
    // expects a standard structure.
    // const x = publicRaw.slice(0, publicRaw.byteLength / 2);
    // const y = publicRaw.slice(publicRaw.byteLength / 2);
    // const uncompressedPoint = new Uint8Array([0x04, ...new Uint8Array(x), ...new Uint8Array(y)]);

    // Import our private key
    return crypto.subtle.importKey(
      "pkcs8", // Private key format (standard)
      privateRaw,
      { name: "ECDH", namedCurve: "P-256" }, // Algorithm and parameters
      false, // Non-exportable
      ["deriveBits"] // Allowed usage: for deriving bits (shared secret)
    ).then(privateKey => {
      // Import the contact's public key
      return crypto.subtle.importKey(
        "spki", // Public key format (standard)
        publicRaw,
        { name: "ECDH", namedCurve: "P-256" },
        false, // Non-exportable
        [] // Specific uses are not needed here for the public key in ECDH
      ).then(publicKey => {
        // 4. Compute the shared secret (deriveBits)
        return crypto.subtle.deriveBits(
          { name: "ECDH", public: publicKey }, // Specify the contact's public key
          privateKey, // Our private key
          256 // Length of the derived secret in bits
        );
      });
    }).then(sharedBits => {
      // Hash the shared secret to obtain an AES key (using SHA-256 as KDF)
      return crypto.subtle.digest("SHA-256", sharedBits);
    }).then(hashed => {
      // Import the hashed secret as an AES-GCM key
      return crypto.subtle.importKey(
        "raw", // "Raw" byte format
        hashed, // Hashed secret
        { name: "AES-GCM" }, // Symmetric encryption algorithm
        false, // Non-exportable
        ["encrypt", "decrypt"] // Allowed uses: encryption and decryption
      );
    }).then(aesKey => {
      this.aesKey = aesKey; // Save the obtained AES key
      return true;         // Signal successful initialization
    });
  }

  encrypt(plaintext) {
    if (!this.aesKey) return Promise.reject("ChatCrypto not initialized");

    // Generate a unique initialization vector (IV)
    const iv = crypto.getRandomValues(new Uint8Array(12)); // 12 bytes (96 bits) is recommended for AES-GCM

    // Convert the text message to bytes (UTF-8)
    const encoded = new TextEncoder().encode(plaintext);

    // Encrypt the data
    return crypto.subtle.encrypt(
      { name: "AES-GCM", iv: iv }, // Algorithm and IV
      this.aesKey, // Our shared AES key
      encoded // Data to encrypt
    ).then(encrypted => {
      // 4. Return IV and encrypted data (in Base64 for convenient transmission)
      return {
        iv: ChatCrypto.arrayBufferToBase64(iv),
        data: ChatCrypto.arrayBufferToBase64(encrypted)
      };
    });
  }

  decrypt(cipherBase64, ivBase64) {
    if (!this.aesKey) return Promise.reject("ChatCrypto not initialized");

    // Convert ciphertext and IV from Base64 to ArrayBuffer
    const encrypted = ChatCrypto.base64ToArrayBuffer(cipherBase64);
    const ivBuffer = ChatCrypto.base64ToArrayBuffer(ivBase64);

    // Decrypt the data
    return crypto.subtle.decrypt(
      { name: "AES-GCM", iv: new Uint8Array(ivBuffer) }, // Algorithm and IV (must be a TypedArray)
      this.aesKey, // The same shared AES key
      encrypted // Encrypted data
    ).then(decrypted => {
      // Convert decrypted bytes back to a string
      return new TextDecoder().decode(decrypted);
    });
  }
}

constructor() and base64ToArrayBuffer() method:
The constructor accepts your private key and the contact's public key in Base64 format. Base64 is a way of encoding binary data into a text string, convenient for transmission or storage.

this.aesKey is initialized as null and will be populated after the init() method successfully completes.

The static methods base64ToArrayBuffer and arrayBufferToBase64 are used to convert data between Base64 strings and ArrayBuffer (the format Web Crypto API works with).

init() Method: Establishing the Shared AES Key
This is the heart of our class, where the "magic" of ECDH occurs and the shared key for symmetric encryption is created.

Breakdown of steps in init():

Key Conversion: Keys are converted from Base64 to ArrayBuffer.
Import Private Key: Your private key is imported in pkcs8 format. It's specified that this is an ECDH key on the P-256 curve and will be used for deriveBits (computing the shared secret).

Import Contact's Public Key: The contact's public key is imported in spki format.

Compute Shared Secret (deriveBits): This is the key ECDH step. Using your private key and the contact's public key, deriveBits computes a shared secret set of bits (sharedBits). This secret will be the same for you and your contact if they use their respective private keys and each other's public keys.

Hash Shared Secret (digest): sharedBits are hashed using SHA-256. This is a common practice to transform the output of deriveBits into a cryptographically strong key of the desired length for a symmetric cipher (AES in this case). This step also serves as a KDF (Key Derivation Function).

Import AES Key (importKey): The resulting hash (hashed) is imported as a "raw" key for the AES-GCM algorithm. This key (this.aesKey) is now ready to be used for encrypting and decrypting messages.
After successful execution, this.aesKey will contain a CryptoKey object, ready for use.

encrypt(plaintext) Method: Encrypting a Message
Breakdown of steps in encrypt():

Generate IV (Initialization Vector): A random 12-byte IV is created. As a reminder, it must be unique for each encryption with the same key.

Encode Text: The message is converted from a JavaScript string to a Uint8Array (a sequence of bytes in UTF-8 encoding) using TextEncoder.
Encryption: crypto.subtle.encrypt performs data encryption using AES-GCM, our this.aesKey, and the generated iv.

Return Result: The encrypted data and IV (both in Base64) are returned as an object. The IV must be transmitted to the recipient along with the ciphertext, as it will be required for decryption.
decrypt(cipherBase64, ivBase64) Method: Decrypting a Message
Breakdown of steps in decrypt():

Data Conversion: The received ciphertext and IV (in Base64) are converted back to ArrayBuffer. Note that for crypto.subtle.decrypt, the iv parameter must be a TypedArray (e.g., Uint8Array), so we pass new Uint8Array(ivBuffer).

Decryption: crypto.subtle.decrypt performs the decryption. Importantly, AES-GCM will not only decrypt the data but also verify its integrity and authenticity, using the same aesKey and iv that were used for encryption. If the data has been tampered with, the key is wrong, or the IV is wrong, the decrypt method will return an error (reject the Promise).

Decode Text: Successfully decrypted bytes are converted back into a readable string using TextDecoder.

How It Works Together: Conceptual Flow Key Pair Generation:

User A generates their ECDH key pair (public PkA and private SkA).
User B does the same (public PkB and private SkB). This step is not shown in the provided ChatCrypto code but is shown below in the examples section (it is performed once, for example, during user registration), and it precedes the use of the class. The Web Crypto API has a crypto.subtle.generateKey method for this. The private key Sk must be stored securely and encrypted with the user's password.
Public Key Exchange:

User A transmits their public key PkA to User B.
User B transmits their public key PkB to User A. This exchange must be secure to avoid Man-in-the-Middle (MitM) attacks. For example, via a secure server or by verifying key fingerprints.

ChatCrypto Initialization and Shared Secret Key Computation:

On User A's side: const cryptoA = new ChatCrypto(SkA_base64, PkB_base64); await cryptoA.init();
On User B's side: const cryptoB = new ChatCrypto(SkB_base64, PkA_base64); await cryptoB.init(); As a result, both (cryptoA.aesKey and cryptoB.aesKey) will have computed the same symmetric AES key.
Message Exchange:

User A encrypts a message for B: const { iv, data } = await chatCryptoA.encrypt("Hello, B!"); Then A sends the { iv, data } object to user B.

User B receives { iv, data } and decrypts: const message = await chatCryptoB.decrypt(data, iv); // message will be "Hello, B!"

Example: Encrypting Text encrypt()

// Initialize the class
let chatCrypto = new ChatCrypto( "My_private_key" , "Contact_public_key" );

// Run it
chatCrypto.init().then(() => {
   chatCrypto.encrypt("Text").then(result => {

     // The encrypted text will be output
      console.log(result);

  });
});

Example: Decrypting Text decrypt()

// Initialize the class
let chatCrypto = new ChatCrypto( "My_private_key" , "Contact_public_key" );

// Run it
chatCrypto.init()
  .then( () => chatCrypto.decrypt("Encrypted_text", "Vector_key_iv") ) // Note: "Vector_key" likely means IV
  .then(result => {

    // The decrypted text will be output
    console.log(result);

  })

Conclusion

We've looked at how to implement robust end-to-end message encryption in JavaScript using the Web Crypto API. The combination of ECDH for secure key exchange and AES-GCM for efficient and authenticated data encryption is a powerful and modern approach. The ChatCrypto class serves as a good starting example of such an implementation. Remember the importance of secure generation, storage of private keys, and reliable exchange of public keys to build a secure system.