I'm building a Redis Clone in Zig: A Deep Dive into Pub/Sub, and Memory Allocation

It's been a while since I started a project called Zedis (Redis written in Zig). It's been a great way to learn low-level programming.

I want to take you on a tour of three core features I've implemented: Pub/Sub and the memory allocation strategy.

Why Build a Redis Clone in Zig?

I have a personal goal of mastering Zig this year; this was the perfect way to accomplish that. Zig offers a number of intriguing features for high-performance systems, such as comptime, which allows code to run at compile time, and explicit memory management. Zedis has been my playground for exploring everything from network programming to custom allocators.

Feature Deep Dive: Pub/Sub

Implementing the Publish/Subscribe mechanism was a fun challenge. At its core, it's a messaging system that decouples senders (publishers) from receivers (subscribers).

Here's how it works in Zedis:

• Channels and Subscribers: When a client subscribes to a channel, they are added to a list of subscribers for that channel. I used a std.StringHashMap([]u64) to map channel names to a list of client IDs.
• Publishing a Message: When a message is published to a channel, the server iterates through the list of subscribers and writes the message to each of their connections.
• Entering Pub/Sub Mode: Once a client subscribes to a channel, they enter a special “Pub/Sub mode” where they can only receive messages and can't execute other commands.

Here is the subscribe function:

pub fn subscribe(client: *Client, args: []const Value) !void {
    var pubsub_context = client.pubsub_context;
    // Enter pubsub mode on first subscription
    if (!client.is_in_pubsub_mode) {
        client.enterPubSubMode();
    }

    var i: i64 = 0;
    for (args[1..]) |item| {
        const channel_name = item.asSlice();
        // Ensure channel exists
        pubsub_context.ensureChannelExists(channel_name) catch {
            try client.writeError("ERR failed to create channel");
            continue;
        };

        // Subscribe client to channel
        pubsub_context.subscribeToChannel(channel_name, client.client_id) catch |err| switch (err) {
            error.ChannelFull => {
                try client.writeError("ERR maximum subscribers per channel reached");
                continue;
            },
            else => {
                try client.writeError("ERR failed to subscribe to channel");
                continue;
            },
        };

        const subscription_count = i + 1;
        const response_tuple = .{
            "subscribe",
            channel_name,
            subscription_count,
        };
        // Use a generic writer to send the tuple as a RESP array.
        try client.writeTupleAsArray(response_tuple);
        i += 1;
    }
}

Memory Allocation Strategy

One of the most interesting aspects of this project has been designing the memory allocation strategy. I opted for a hybrid approach to balance performance and memory usage:

• KeyValueAllocator: This is a custom allocator I built for the main key-value store. It uses a fixed-size memory pool and has a simple eviction policy to stay within its budget. When the allocator runs out of memory, it can evict all keys to make space for new ones.
• Arena Allocator: For temporary, short-lived allocations (like parsing commands), I use an arena allocator. This is incredibly fast because it simply bumps a pointer for new allocations and frees all the memory at once when it's no longer needed.
• Fixed Pools: For objects that are frequently allocated and deallocated, like client connections, I use a fixed-size pool. This avoids the overhead of dynamic allocation and deallocation.

This approach allows Zedis to be memory-efficient while still being performant.

What's Next?

I'm excited to keep building on this foundation. Here's what's on the roadmap:

• Implement AOF (Append Only File) logging
• Add support for more data structures like lists and sets
• Implement key expiration
• Add clustering support

Check it out

I'd love for you to check out Zedis on GitHub, try it out, and let me know what you think. Contributions are always welcome!

Thanks for reading!

Comments

[csm]

Out of topic question:
How you learn something new that has less or no resources? How you figure out what to learn and from where?
Because i tried zig for 2 or 3 times and always gave up on it and went to langs that have more resources.

[Charles Fonseca]

It is definitely lacking in resources, particularly examples in the most recent version of the language (0.15.1).
It was mostly reading the Tigerbeetle code base at github.com/tigerbeetle/tigerbeetle, articles like openmymind.net/, and libraries like the ones Karl Seguin creates, github.com/karlseguin. I'm a database freak, so reading Tigerbeetle is my main resource for learning it.