Demystifying Redis: How I Built a Custom In-Memory Store in C++

Executive Summary & Roadmap

Rebuilding a system you rely on every day is the ultimate way to demystify its internal architecture.

In this article, we break down Redis-Lite—a multi-threaded, RESP-compliant in-memory key-value store built entirely from scratch in modern C++. We trace the exact execution path of a database command: from raw TCP byte stream ingestion over POSIX sockets, through protocol parsing, to concurrent in-memory execution and file persistence.

---

Topics to look forward to:

• Low-Level Socket Programming: How POSIX TCP sockets handle concurrent connections using a Thread-per-Connection model.

• Protocol Parsing: Safely deserializing the Redis Serialization Protocol (RESP) without buffer overflows.

• Concurrency & Memory Safety: Protecting std::unordered_map with C++ RAII mutex guards and comparing coarse-grained locking against single-threaded event loops (epoll).

• Real-World Application: Building a BullMQ-style Distributed Priority Task Queue on top of our custom database commands (HMSET, LPUSH, RPUSH, LPOP).

---

Table of Contents

1. Architectural Overview & System Design

2. The TCP Server Loop and Threading Model

3. Parsing the RESP Protocol

4. Memory Storage & Concurrency

5. Advanced Features: Expiration (TTL) & RDB State Persistence

6. Building a Distributed Task Queue

7. Performance Takeaways & Conclusion

1. Architectural Overview & System Design

Building an in-memory database requires carefully orchestrating network I/O, string parsing, and thread-safe data storage. Redis-Lite is designed around a straightforward, blocking architectural flow:

1. TCP Server Loop: The main server loop listens for incoming client connections.

2. Thread-per-Connection Execution: Upon accepting a connection, the server immediately hands off the socket to a newly spawned std::thread, preventing the main loop from blocking.

3. RESP Parser: The worker thread reads the raw network byte stream and parses it according to the Redis Serialization Protocol (RESP) to extract executable commands.

4. In-Memory Store: Parsed commands are routed to the database, which uses coarse-grained mutex locking to safely execute reads and writes against std::unordered_map data structures.

High-Level Architecture Flow

2. The TCP Server Loop and Threading Model

At its core, the server relies on low-level POSIX socket programming to communicate over the network. We initialize a TCP socket, bind it to a port, and begin listening. Because we aren't using high-level web frameworks, we manually manage the raw byte streams traveling across these socket file descriptors.

Handling multiple concurrent clients is the biggest challenge in a custom database. Unlike native Redis (which is single-threaded and uses an event loop), Redis-Lite uses a Thread-per-Connection concurrency model.

The server sits in an infinite loop blocking on the accept() system call. When a client connects, it spins up a detached POSIX thread via C++ std::thread to handle the entire lifecycle of that specific client socket.

Here is the exact implementation of it:

while(running) {
    int client_socket = accept(server_socket, nullptr, nullptr);
    if (client_socket < 0) {
        if (running) cerr << "Couldn't accept client connection\n";
        break;
    }

    // Spin up a new thread for this specific client connection
    threads.emplace_back([client_socket, &cmdHandler]() {
        char buffer[1024];
        while (true) {
            memset(buffer, 0, sizeof(buffer));
            // Prevent buffer overflow by restricting read size
            int bytes = recv(client_socket, buffer, sizeof(buffer) - 1, 0); 
            if (bytes <= 0) break;

            string request(buffer, bytes);
            string response = cmdHandler.processCommand(request);
            send(client_socket, response.c_str(), response.size(), 0);
        }
        close(client_socket);
    });
}

3. Parsing the RESP Protocol

Once the thread reads the raw socket bytes, it must translate the Redis Serialization Protocol (RESP). The protocol prefixes Arrays with * and Bulk Strings with $, followed by their respective lengths.

Instead of guessing where strings end, the parser uses these prefixes to perfectly extract data. Here is the core parsing logic:

size_t pos = 0;

// 1. Read the '*' to get the Array size
if(input[pos] != '*') return tokens; 
pos++;

size_t crlf = input.find("\r\n", pos);
int numElements = stoi(input.substr(pos, crlf - pos));
pos = crlf + 2; // Move past \r\n

// 2. Iterate and extract each Bulk String
for(int i = 0; i < numElements; i++) {
    if(pos >= input.size() || input[pos] != '$') break;
    pos++;

    crlf = input.find("\r\n", pos);
    int len = stoi(input.substr(pos, crlf - pos));
    pos = crlf + 2; 

    // Extract exact string safely using length 'len'
    string token = input.substr(pos, len);
    tokens.push_back(token);

    pos += len + 2; 
}

4. Memory Storage & Concurrency

Our primary storage mechanism leverages C++'s std::unordered_map to provide average O(1) time complexity for lookups and insertions. However, because our server uses a Thread-per-Connection model, multiple client threads might attempt to read or modify the hash map simultaneously.

To prevent memory corruption (race conditions), we use a global std::mutex paired with std::lock_guard.

// Storage definition: std::unordered_map<std::string, std::string> kv_Store;
// Global lock: std::mutex db_mutex;

void RedisDatabase::set(const std::string& key, const std::string& value) {
    // RAII wrapper automatically acquires lock here
    std::lock_guard<std::mutex> lock(db_mutex); 
    kv_Store[key] = value;
} // Lock is automatically released upon exiting function scope

bool RedisDatabase::get(const std::string& key, std::string& value) {
    std::lock_guard<std::mutex> lock(db_mutex);
    purgeExpired(); // Lazy expiration cleanup

    auto it = kv_Store.find(key);
    if(it != kv_Store.end()) {
        value = it->second;
        return true;
    }
    return false;
}

Preventing Race Conditions with RAII

By wrapping the mutex in std::lock_guard, we utilize the C++ RAII (Resource Acquisition Is Initialization) pattern. The lock is safely acquired when the function begins and is strictly guaranteed to be released when the scope ends—even if the function returns early or throws an exception. This ensures that only one thread can ever access the database at a time.

The Bottleneck of a Global Lock vs. Native Redis

While a single global db_mutex guarantees thread safety, it acts as a coarse-grained lock. If Thread A is reading a key, Thread B is entirely blocked from writing to a completely unrelated key. Under heavy traffic, client threads spend most of their time waiting for the lock to be released, severely bottlenecking throughput.

How production Redis solves this: Native Redis bypasses this locking problem entirely by being fundamentally single-threaded. Instead of spinning up threads for every client, it uses an asynchronous event loop (like epoll) to multiplex thousands of non-blocking sockets onto a single execution core. Because only one thread processes commands, race conditions simply do not exist, eliminating the need for slow mutexes altogether.

5. Advanced Features: Expiration (TTL) & RDB State Persistence

A production-grade database must do more than just store strings in memory. It must manage the lifecycle of volatile data (Time-To-Live) and ensure that critical data survives a server crash or reboot.

Core Data Stores Explained

Our database orchestrates four separate std::unordered_map structures to handle different workloads efficiently:

• kv_Store: Maps string to string. The fastest way to handle 1-to-1 caching.

• list_store: Maps string to std::vector<string>. Vectors provide incredibly cache-friendly memory locality and $O(1)$ appends (RPUSH).

• hash_Store: Maps string to an inner unordered_map. This nested map allows $O(1)$ lookup for specific fields inside large objects.

• expiry_map: Maps string to a std::chrono::time_point. This records the exact future timestamp when a key should be evicted.

Lazy Expiration (TTL)

To manage key expiration, we use a Lazy / On-Demand Expiration model. When a client sets an EXPIRE time, we calculate the absolute future time_point and store it in expiry_map. Instead of running a continuous background loop that scans for expired keys, we evaluate expiration synchronously during data access. Whenever a command like GET or SET is called, it first triggers purgeExpired():

//check all the expiry_amp timings with current time
void RedisDatabase::purgeExpired() {
    auto now = std::chrono::steady_clock::now();
    for (auto it = expiry_map.begin(); it != expiry_map.end(); ) {
        if (now > it->second) {
            // Remove from all stores
            kv_Store.erase(it->first);
            list_store.erase(it->first);
            hash_Store.erase(it->first);
            it = expiry_map.erase(it);
        } else {
            ++it;
        }
    }
}

Why this is efficient: We only pay the CPU cost for deletion when we actively interact with the dataset, completely eliminating the need for complex, lock-heavy background threads.

RDB-Style File Persistence

Because RAM is volatile, shutting down the server wipes all data. To solve this, we implemented a custom dump.my_rdb serialization protocol. When the server undergoes a graceful shutdown, it writes the in-memory state to a binary file on the disk.

bool RedisDatabase::dump(const std::string& filename) {
    std::lock_guard<std::mutex> lock(db_mutex);
    std::ofstream ofs(filename, std::ios::binary);

    // Serialize Key-Value Store
    for(const auto& kv : kv_Store) {
        ofs << "K " << kv.first << " " << kv.second << "\n";
    }

    // Serialize Lists
    for(const auto& kv : list_store) {
        ofs << "L " << kv.first;
        for(const auto& item : kv.second) ofs << " " << item;
        ofs << "\n";
    }

    // (Hashes are serialized similarly)
    return true; 
}

When the server boots up, the load() function reads this file line-by-line to repopulate the hash maps. This hybrid approach guarantees data survival across restarts while maintaining the blazing-fast speeds of an in-memory execution engine.

6. Building a Distributed Task Queue

To demonstrate the real-world utility of Redis-Lite, I built a Distributed Task Queue (similar to BullMQ) entirely on top of the custom Redis commands.

This system separates work into two independent processes: Producers that generate tasks, and Workers that asynchronously consume and process them.

Enqueuing and Dequeuing Logic

The task queue heavily utilizes Redis Hashes and Lists:

1. Producer: When a producer creates a task, it first uses HMSET to store the task's metadata (type, priority, timestamp) in a Redis Hash. Then, it pushes the taskId into a Redis List acting as a queue. For urgent tasks, it uses LPUSH (putting it at the front of the line), and for standard tasks, it uses RPUSH (putting it at the back).

2. Worker: The worker runs an infinite loop, constantly querying the server using LPOP. Because LPOP removes the item from the list and returns it in one atomic operation, it is impossible for two workers to accidentally grab the same task. Once popped, the worker updates the hash status to "processing", simulates the work, and finally marks it "completed".

Task Flow Architecture

Priority Queues & Worker Distribution

Priority Handling: We enforce priority natively by creating four separate list queues: critical, high, normal, and low. Inside the worker's getNextTask() method, the worker explicitly attempts to LPOP from the critical queue first. Only if critical returns null does it move on to high, and so on. This guarantees that background tasks never block urgent workloads.

7. Performance Takeaways & Conclusion

Building Redis-Lite revealed several critical performance realities about concurrent systems programming:

1. The Mutex Bottleneck: Despite having a multi-threaded architecture, throughput plateaus under heavy load. Because all operations share a single global db_mutex, threads constantly queue up waiting for access.A production-ready fix would be transitioning to fine-grained locking or sharding.

2. Context Switching Overhead: The Thread-per-Connection model is incredibly intuitive to program, but it scales poorly. If 10,000 clients connect, the OS attempts to schedule 10,000 threads. The CPU wastes massive amounts of time just context-switching between them, and many threads sit idle blocking on recv(). This perfectly illustrates why native Redis uses a single-threaded asynchronous event loop (like epoll) to multiplex connections instead.

3. Algorithmic Complexity Matters: I chose std::vector for lists because appending (RPUSH) is O(1) and cache-friendly. However, inserting at the front (LPUSH) forces the vector to shift all elements, degrading to O(n). Understanding these underlying C++ container mechanics is vital when building a database meant for microsecond latencies.

Final Thoughts

Rebuilding a tool you use every day is the ultimate way to demystify it. Redis-Lite successfully implements the RESP protocol, multiplexes connections, and safely stores data—but more importantly, it taught me the exact engineering hurdles that led to the architectural decisions behind the real Redis.