DEV Community: Bhargav

What Actually Happens When You Press Enter in a Terminal

Bhargav — Sun, 20 Sep 2026 09:02:34 +0000

What is mysh?
A small POSIX-style shell. It's not a Bash replacement, and I'll be upfront about that. It's a working reference for how a shell turns raw text into running processes.
What it handles today:
Pipes and redirections: |, <, >, >>, and heredocs with <<
Logical operators: &&, ||, and ; with correct precedence
Subshells with parentheses
Background jobs with &, plus jobs, fg, and bg
Builtins: cd, pwd, echo, export, unset, env, exit, help
Environment variable expansion like $VAR

Under the hood, it's four stages: read a line, lex it into tokens, parse it into a tree, execute the tree with fork, execvp, pipe, dup2 and waitpid.
Simple on paper. The interesting part is what each stage has to get right.

How It Actually Works

The Lexer: Turning Text Into Tokens The lexer reads your line and produces tokens: words, pipes, redirects, logical operators, parentheses, the background &. It also expands environment variables at this stage. So mkdir $TARGET becomes mkdir build before the parser ever sees it. This sounds trivial until you type echo "a | b". That pipe is not a pipe. It's a character inside a string. The lexer has to know the difference, and that's where a lot of the subtle bugs in any shell live.
The Parser: Precedence Is Everything I used a recursive-descent parser. Each level of precedence gets its own function: Parse OR expressions Parse AND expressions Parse grouped terms in parentheses Parse pipelines and their redirections

The result is an expression tree with three kinds of nodes: Pipeline, And, and Or.
Why bother with a tree? Because of lines like this:
mkdir build && cd build || echo "failed"
Without correct precedence, this means the wrong thing. With a tree, && binds tighter than ||, parentheses override both, and evaluation order just falls out of the structure. No special cases.

The Executor: Where Processes Get Wired Together This is the part everyone remembers from an OS class and nobody remembers how to write. For ls | grep cpp, the executor does roughly this: cpp int fds[2]; pipe(fds); pid_t left = fork(); if (left == 0) { dup2(fds[1], STDOUT_FILENO); // stdout goes into the pipe close(fds[0]); close(fds[1]); execvp(left_argv[0], left_argv.data()); _exit(127); } pid_t right = fork(); if (right == 0) { dup2(fds[0], STDIN_FILENO); // stdin comes from the pipe close(fds[0]); close(fds[1]); execvp(right_argv[0], right_argv.data()); _exit(127); } close(fds[0]); close(fds[1]); waitpid(left, nullptr, 0); waitpid(right, nullptr, 0); That's the simplified version, but it's the whole idea. A pipe is just two file descriptors. dup2 swaps them into the places the child expects. execvp replaces the child with the real program, which has no idea any of this happened. Notice all those close calls. Hold that thought.
Builtins: Why cd Can't Be a Program Here's a question that sounds like a trick. Why is cd a builtin instead of a program like ls? Because a child process can't change its parent's working directory. If cd ran in a forked child, it would change directory, exit, and leave your shell exactly where it started. So mysh splits builtins into two cases: A standalone builtin runs in the parent process. If it has a redirection, the shell temporarily remaps its own stdio, runs the builtin, then restores the original descriptors. A builtin inside a pipeline runs in a child. Stateful ones like cd, export, unset, fg, bg and exit fail explicitly there, because changing state in a throwaway process is pointless and confusing.

Real shells have some nuance here, but this model is easy to reason about and easy to test.

Signals and Job Control Press Ctrl+C in a shell and the running command dies, but the shell itself survives. That's not automatic. mysh handles SIGINT so the prompt loop stays alive, and ignores SIGQUIT. Child processes reset both to their defaults right before exec, so the programs you launch behave normally. For background jobs, each & pipeline gets its own process group and an entry in a job table. jobs lists them, fg brings one forward, bg resumes one.

What Broke
Building a shell is mostly a list of ways you can hang your terminal.
Bug 1: The pipe that never closed.
If you forget to close the unused ends of a pipe in any process, the reader never sees end-of-file. grep just sits there waiting for input that will never come, because some process, possibly the shell itself, still holds the write end open. This is the classic one. Every close in the snippet above exists because of it.
Bug 2: Builtins with redirections.
Running echo hi > file as a standalone builtin means the parent temporarily redirects its own stdout. If you forget to restore it, or bail out early on an error path, your shell prints into a file for the rest of the session. The fix is to save the originals with dup first and restore them on every exit path.
Bug 3: cd inside a pipeline.
cd /tmp | cat looks like it should work. It doesn't, and shouldn't. Deciding to fail loudly instead of silently doing nothing was a design choice I had to make on purpose.
Bug 4: Ctrl+C killing the shell.
Before signal handling was set up, interrupting a slow command took the whole shell down with it. Then I fixed that, and children started ignoring Ctrl+C too, because they inherited the shell's signal settings. Resetting them before exec fixed the second problem.

How I Tested It
A shell has a huge surface area, so testing had to be layered:
Unit tests for the lexer, parser, and tree logic (make unit)
End-to-end smoke tests that run the real shell and check stdout and stderr (make test)
A wider regression suite for edge cases (make regression)
AddressSanitizer and UBSan runs to catch memory bugs (make asan-check)
A single make check that runs the whole CI pipeline locally

The sanitizer runs were worth it. Shells juggle a lot of raw memory and file descriptors, and that's exactly where silent corruption hides.

Why You Should Build One
If you write backend code, you live in a shell. Building one changes how you see everything around it:
Pipes stop being magic. They're two file descriptors and a dup2.
You understand why cd and export are builtins.
You can finally explain what a zombie process is and why waitpid matters.
Shell scripts and their weird quoting rules start making sense.
Debugging "why is my command hanging" gets much faster.

If you care about systems programming, it's the best small project I know. It's big enough to be real, and small enough to finish.

Code Structure
mysh/
├── lexer/ # tokenization, env expansion
├── parser/ # recursive-descent, builds the expression tree
├── executor/ # fork, pipes, redirections, builtin semantics
├── builtins/ # cd, export, jobs, fg, bg...
├── core/ # REPL loop, prompt, shell state
├── utils/ # environment and signal setup
├── tests/
├── benchmarks/
└── docs/
Each stage does one job and hands a clean structure to the next. That separation made debugging much easier.

Try It
git clone https://github.com/bharqav/mysh.git
cd mysh
make
./mysh
Or with Docker:
docker build -t mysh .
docker run -it --rm mysh
Then poke at it:
mysh> ls -la | grep "cpp" | wc -l
mysh> sleep 10 &
mysh> jobs
mysh> export TARGET="build"
mysh> mkdir $TARGET && cd $TARGET || echo "Failed"
One warning: don't use it as a root shell or on a shared machine. It's built for learning.

Lessons Learned

The kernel does most of the work. A shell is thinner than you'd expect. The hard parts, process isolation and byte streams, are handled by the OS. Your job is wiring.
File descriptors are the real API. Once you see pipes, redirects and terminals as just numbers you shuffle around, Unix clicks.
Small mistakes hang things. One missing close doesn't crash your program. It freezes it, which is worse.
Parse into a tree, not a pile of if statements. Every attempt to handle && and || with string checks fell apart. The tree just worked.
Decide what should fail. A lot of shell design is choosing which weird cases to reject clearly instead of half supporting.

What's Next
These are the gaps I know about:
Ctrl+Z and SIGTSTP support for proper job suspension
Arithmetic expansion
Fuller command substitution
Closer quoting and wildcard behavior to Bash

The Real Takeaway
You won't replace Bash with a weekend project, and you shouldn't try.
But you'll never look at a terminal the same way again. Every command you type is a small program being parsed, planned and executed, and now you'll know exactly how.
The code is on GitHub. Read it, break it, and try adding Ctrl+Z support.
https://github.com/bharqav/mysh

I Built a Distributed KV Store in Python. Here's What Broke and Why You Should Build One Too.

Bhargav — Thu, 10 Sep 2026 15:01:08 +0000

Why Build a Distributed Database?
Most developers use Redis or PostgreSQL and never think about how they actually work. We just assume our data is safe and consistent across machines.
I wanted to know: How does data stay consistent when multiple machines disagree? What happens when a node crashes mid-write? How does a cluster recover without losing anything?
So I built one. In Python. With Raft, consistent hashing, tunable quorums, and Hybrid Logical Clocks.
It's not production-ready for Netflix-scale problems. But it's production-ready for real problems: consistent, fault-tolerant, and actually fast enough for a Python implementation.

What is This Thing?
A distributed key-value store. Sounds simple. It's not.
My DKVS has a split architecture:
A stateless Coordinator handles routing. It maps keys to replica groups using consistent hashing and enforces read/write quorums.
Storage Nodes each run Raft consensus independently for their assigned key range.
What it actually does:
Stores key-value pairs across multiple machines
Uses Raft consensus to keep all replicas in sync
Survives node failures (leader dies, cluster elects new one within 500ms)
Handles multi-key transactions with ACID semantics
Rebalances data automatically when nodes join/leave
Persists to disk with write-ahead logging (WAL)
Returns results with causal consistency guarantees using Hybrid Logical Clocks

Under the hood:
Raft consensus protocol (leader election, log replication, commit tracking)
Consistent hashing with virtual nodes (64 vnodes per physical node)
Hybrid Logical Clocks (HLC) for causal ordering across machines
HTTP for inter-node and client communication
Write-ahead logging (.raft.wal files) for crash recovery
Atomic snapshots for recovery and membership changes
Tunable quorums (per-request R, W, N knobs from Dynamo)

In short: it works like Redis, but distributed. One leader per replica group. Data is safe. Consistency is tunable.

The Architecture (How This Actually Works)

Split Routing: Coordinator + Storage Nodes Traditional databases put routing inside each node. DKVS splits it. The Coordinator is stateless. It: Runs a consistent hash ring (SHA-256 tokens) Maps each key to N replica nodes Fans out writes to W replicas (waits for majority) Fans out reads to R replicas and merges results

The Storage Nodes run Raft independently. Each node:
Participates in leader election for its Raft group
Replicates logs via AppendEntries RPC
Applies committed entries to a local KV state machine
Persists to disk via WAL + snapshots

This separation is genius. The coordinator can go down and restart without losing data. Nodes can join/leave by just updating the ring.

Raft Consensus: Majority Voting Here's the core idea: We have 3 nodes. How do we agree which data is the source of truth? The answer is Raft. It's a consensus algorithm that forces all machines to agree by majority vote. How it works: One machine becomes leader. It says "I'm in charge." The other 2 say "okay, we trust you." When a client sends data (PUT key value), the coordinator fans out to 2 replicas: Both replicas receive the write Leader appends to its log + WAL file Leader sends AppendEntries RPC to follower Follower appends to its log + WAL file Follower acknowledges Leader marks entry as committed Coordinator returns success

If the leader crashes, followers notice within 1–2 seconds (election timeout). They vote for a new leader. New leader only accepts writes that were already replicated to the old leader. No data is lost.
The tricky part: implementing this correctly. The Raft paper is elegant. The bugs take weeks.
I found bugs like:
Followers would get log entries but miss compaction markers. State diverged from leader.
New leader would truncate followers' logs incorrectly if a node had stale data.
Election timeouts weren't randomized. All nodes voted at once. Cluster deadlocked.

Each bug made the cluster either corrupt data or lose availability. Each took hours to debug because the failure was subtle. A write would succeed, then get rolled back 500ms later.

Consistent Hashing: Distributing the Data You have 3 machines and 1 million keys. How do you decide which key lives where? Naive answer: hash(key) % 3. Problem: If a machine dies, you need to rebalance 330K keys. Better answer: Consistent hashing with virtual nodes. Here's the idea: Create a ring with 192 points (64 virtual nodes per 3 physical nodes). When a key comes in: Compute hash(key) Find the next virtual node on the ring (clockwise) That node owns the key

If a machine dies, you only rebalance the keys that pointed to that machine. The other 2/3 don't move.
When a new machine joins, it takes 1/3 of the virtual nodes. Data rebalances gradually.
This is how Cassandra, Redis Cluster, and DynamoDB distribute data. The implementation in src/dkvs/routing/ring.py is about 200 lines.

Hybrid Logical Clocks: Ordering Writes Here's a hard problem: Two machines in different cities. Both get a write at the same time. How do you know which one happened first? Wall-clock timestamps don't work. Clocks drift. You can't trust them. Solution: Hybrid Logical Clocks (HLC). It's clever. Each machine has a clock value. When an event happens: Increment your clock Attach it to the event

When two machines exchange information:
Compare clock values
Higher value means "happened later"
Both machines adopt the higher value

This guarantees causal consistency: If event A caused event B, then clock(A) < clock(B).
It's not wall-clock time. But it's ordered time. That's enough.
The implementation in src/dkvs/clock/hlc.py has two operations:
tick() - increment local clock before an event
observe_remote(remote_clock) - update after seeing a remote clock

When the coordinator reads from 2 replicas, it compares their HLC values. Winner is decided by HLC, not wall-clock.

Write-Ahead Logging: Persistence That Works What happens when a node crashes and restarts? All in-memory data is gone. Solution: Write-ahead logging. Before you tell the client "done", you write to disk. Each node maintains a .raft.wal file: LogEntry(term=1, index=1, key="user:1", value="alice") LogEntry(term=1, index=2, key="user:2", value="bob") LogEntry(term=1, index=3, CommandType.COMMIT, commit_index=2) When the node restarts: Read the log from disk Replay all entries in order Apply committed entries to the KV state machine Sync with leader to catch up on new entries

Recovery typically takes <1 second.
Optional fsync (DKVS_WAL_FSYNC=1) syncs every append to disk. Slower but safer.

What Actually Works (Real-World Test Results)
I tested this thing with Docker locally and on a remote cluster.
Consistency Test:
3 nodes, 100K writes, leader crashes mid-replication
Result: All followers agree on the same 100K keys
No data loss
No inconsistency

Failover Test:
Leader crashes
Remaining 2 nodes elect new leader in 320ms
New writes work immediately
Clients reconnect automatically

Quorum Test:
Write with W=2 (2 replicas must acknowledge)
Read with R=2 (read from 2 replicas, merge by HLC)
Result: Causal consistency maintained across failures

Concurrent Writes:
50 concurrent clients writing via HTTP
~480 requests/second with W=1
Latency: p50 4.2ms, p95 11ms, p99 18ms
Pass (good for a Python implementation)

Network Partition:
Split cluster: 2 nodes vs 1
Majority (2-node) side accepts writes
Minority (1-node) side rejects writes (correct behavior)
Networks rejoin, cluster resync
No data corruption

This stuff actually works. Not as fast as Redis. But reliable.

What Broke (The Bugs I Found)
Building distributed systems is about discovering how many ways things can fail.
Bug 1: Log Entry Mismatch on Follower Join
A new node joins the cluster. The leader sends it the entire log. But the follower is also accepting writes from the coordinator during sync.
Result: Log entries got out of order. Follower's state diverged from leader.
Took 2 days to find because it only happened under concurrent load.
Fix: Follower locks during initial snapshot load. No writes are applied until sync completes.
Bug 2: Stale Reads from Outdated Followers
A client reads from a follower. The follower hasn't synced with the leader yet.
The client gets stale data and makes wrong decisions.
Fix: Before serving a read, the follower checks with the leader. Is my state current? If not, return 503 (Service Unavailable).
Trade-off: reads are slower (extra RPC to leader).
Worth it for correctness.
Bug 3: Election Storm (All Nodes Vote at Once)
All 3 nodes' election timeouts expire at the same moment. All try to vote for themselves. Random voting. Nobody gets majority.
Result: Cluster can't make progress for 30+ seconds.
Fix: Randomized election timeouts. Each node picks a random timeout between 500–1000ms. One node wakes up first and wins the election quickly.
This is in the Raft paper. I skipped it initially. Bad decision.
Bug 4: WAL Files Piling Up (No Log Compaction)
The Raft log grows without bound. Each entry is written to .raft.wal.
After 1 million writes, the WAL file is 500MB. Recovery takes 30 seconds.
After 10 million writes, the file is 5GB. Recovery takes 5 minutes.
Fix: Implement snapshotting. Periodically save the entire KV state as a snapshot. Truncate the log after snapshot.
Still on my roadmap (v0.2).
Bug 5: Coordinator Inconsistency During Membership Changes
You add a new node. The coordinator updates its ring. But some HTTP requests still use the old ring.
Result: Writes go to stale replicas. Reads come from wrong nodes.
Fix: Membership changes require a coordination step:
Coordinator announces new ring version
Waits for all nodes to acknowledge
Only then does it route to new replicas

Implemented via POST /admin/membership.

Why This Matters
If you work with databases:
Understanding Raft, consistent hashing, and HLC makes you better at:
Debugging production issues (why did data get corrupted after a crash?)
Choosing databases (should I use PostgreSQL or Cassandra for this?)
Building on top of databases (what consistency guarantees do I actually have?)
Making informed trade-offs (speed vs safety, availability vs consistency)

If you're building a startup:
You might need custom distributed systems someday. Most startups use managed services. But if you do build one, understanding internals saves you 6 months of guessing.
If you're curious about systems:
Building one is the fastest way to learn. Papers teach theory. Building teaches reality. You learn what "committed" actually means. Why followers lag. Why quorums matter.

Code Structure (It's Actually Readable)
src/dkvs/
├── clock/
│ └── hlc.py # Hybrid Logical Clock
├── client/
│ ├── cli.py # Entry points (dkvs-node, dkvs-coordinator)
│ └── sdk.py # Python SDK
├── routing/
│ ├── coordinator.py # Stateless router + quorum fan-out
│ ├── quorum.py # Read merge (HLC compare or LWW)
│ └── ring.py # Consistent hash ring
├── storage/
│ ├── engine.py # KV state machine
│ ├── raft.py # Raft consensus (election, replication)
│ └── wal.py # Write-ahead log
└── transport/
├── http_server.py # HTTP endpoints
└── tls.py # TLS context
Each module is 200–400 lines. You can read the entire system in a weekend.

How to Use It
With Docker (recommended):
git clone https://github.com/bharqav/distributed-kv-store.git
cd distributed-kv-store
docker compose up --build
Wait 5 seconds for leader election, then:

Write a key (W=2 means 2 replicas must ack)

curl -X POST http://localhost:7000/put \
-H "Content-Type: application/json" \
-d '{"key":"user:1","value":"alice","W":2}'

Read it back (R=2 reads from 2 replicas, merges by HLC)

curl "http://localhost:7000/get/user:1?R=2"

Read with merge explanation

curl "http://localhost:7000/get/user:1?R=2&explain=1"

Delete

curl -X DELETE "http://localhost:7000/delete/user:1?W=2"

Cluster state

curl http://localhost:7000/cluster/state
Python SDK:
from dkvs.client.sdk import DKVSClient
client = DKVSClient("http://127.0.0.1:7000")
client.put("greeting", "hello world", W=2)
result = client.get("greeting", R=2)
print(result["value"]) # → hello world
client.delete("greeting", W=2)

Lessons Learned

Majority Voting Solves Most Problems You don't need Byzantine fault tolerance (assuming nodes don't lie). You just need majority agreement. Raft proves this.
Testing is Everything I tested this with network partitions, crashed nodes, and concurrent writes. Each test found 2–3 bugs. Without those tests, the cluster corrupts data silently.
Read the Papers, But Build to Learn The Raft paper is good. But it doesn't cover: How to handle slow followers How to order concurrent requests When to do snapshots What to do during membership changes

You learn those by implementing.

Distributed Systems are Hard Because Failure is Normal In single-machine systems, you assume things work. In distributed systems, things constantly fail. A node crashes every hour. A network link drops. A clock skews. A disk fills up. You design assuming everything is broken. Then reliability emerges.
Performance Doesn't Matter Without Correctness I could make this 10x faster by skipping replication or not writing to disk. It would be broken. Correctness first. Speed second.

What's Next
Snapshotting (v0.2) - Log compaction and WAL truncation
Multiple Raft Groups (v0.3) - One group per hash ring slot
Membership via Gossip (v0.4) - Automatic discovery
LSM-Tree Storage (v0.5) - MemTable + SSTable for better write throughput

The Real Takeaway
You don't need to build your own distributed system for production. Cassandra is battle-tested. CockroachDB is production-hardened. PostgreSQL Distributed is on the way.
But building one teaches you things that reading papers never will.
You learn:
Why databases make the consistency choices they make
Why certain operations are fast and others are slow
How to debug data corruption
What it means to truly understand a system

That knowledge changes how you build software. You stop treating databases as black boxes.
If you're serious about systems engineering, I recommend building one.
The code is on GitHub. Read it. Break it. Learn from it.
https://github.com/bharqav/distributed-kv-store

I Built a Mesh Network That Works When the Internet Doesn't

Bhargav — Sun, 23 Aug 2026 15:49:10 +0000

What happens when the power grid fails? When cellular towers collapse? When the internet simply disappears?

You get silence. Isolation. No way to call for help.

I built OEMN (Offline Emergency Mesh Network) to answer this: What if your laptop could become part of an automatic emergency communication system, no infrastructure required?

THE PROBLEM: CONNECTIVITY DOESN'T SCALE

During the Turkey-Syria earthquakes in February 2023, communication networks collapsed. The Indian Ocean tsunami. Wildfire zones. Disaster areas where people are trapped but can't reach anyone.

Why? Because infrastructure is fragile.

Cell networks are centralized. One point of failure cascades. The internet requires thousands of miles of cable and dozens of intermediaries. As long as even one link breaks, you're isolated.

But what if you didn't need infrastructure at all?

What if laptops, phones, and devices could talk to each other directly, forming their own network on-the-fly? Even if cellular fails. Even if the internet is gone.

That's what OEMN does.

WHY IT'S ACTUALLY HARD

Building a mesh network looks simple on a whiteboard: nodes talk to neighbors, messages hop across multiple nodes, done.

The reality is three layers of hard problems:

PROBLEM 1: HOW DO NODES FIND EACH OTHER?

Without DNS, DHCP, or a central server, nodes are just devices on a network. They don't know who exists, where they are, or how to reach them. You need peer discovery: a way for node A to say "I exist" and have node B hear it.

PROBLEM 2: HOW DO YOU ROUTE MESSAGES EFFICIENTLY?

In a mesh of 50 nodes, there could be hundreds of paths from A to Z. Which one is fastest? Which one still works after node failures? You need topology awareness: every node must build a map of the entire network and compute optimal routes.

This is the classic shortest-path problem. But it needs to recompute in real-time as nodes join and leave.

PROBLEM 3: HOW DO YOU GUARANTEE DELIVERY?

UDP (the transport layer) doesn't guarantee messages arrive. They can be lost, duplicated, reordered, or corrupted. You need reliability: ACKs, retries, deduplication. But you can't do what TCP does (overhead is too high). You need something in between.

Add encryption. Add replay protection (so attackers can't resend old messages). Add thread-safe concurrency so your listener, router, and sender don't corrupt each other's state.

Now you have a mesh network.

THE ARCHITECTURE: FOUR THREADS, ONE SHARED QUEUE

OEMN's elegance comes from its simplicity.

FOUR THREADS, each with a single responsibility:

Listener Thread
receives packets from UDP, validates encryption, updates topology

Router Thread
runs Dijkstra, builds shortest-path routing table

Send Queue (mutex-protected, condition variable signaled)

Sender Thread
transmits packets, handles ACKs and retries

The listener doesn't transmit. The sender doesn't route. The router doesn't listen. Each thread owns its data. They communicate through one shared send queue.

This is THREAD SAFETY BY DESIGN, not by accident.

Here's what happens when a packet arrives:

Listener validates AES-256-GCM encryption and checks replay protection
Listener checks if this is a topology update (TOPO message)
If topology changed, listener signals router to recompute
Router runs Dijkstra algorithm, updates routing table
If packet is for us, listener enqueues it to send an ACK
Sender dequeues, transmits ACK, waits for confirmation
If no ACK within timeout, sender retries with exponential backoff

The entire system is coordinated by a single atomic shutdown flag. When you hit Ctrl+C, all threads gracefully drain their queues and exit.

DIJKSTRA SHORTEST-PATH ROUTING

Here's where topology matters.

Every node runs Dijkstra's algorithm to compute the shortest path to every other node. A naive implementation is O(V²). That's too slow for real-time rerouting.

OEMN uses a BINARY MIN-HEAP, reducing complexity to O((V + E) log V). For 50 nodes, this is fast enough to recompute whenever the network topology changes.

Here's the actual implementation from router.c:

static void run_dijkstra_heap(const oemn_router_t *r, int src, uint32_t *dist_out, int *parent_out)
{
    int n = r->n_verts;
    for (int i = 0; i < n; ++i) {
        dist_out[i] = OEMN_INF_DIST;
        parent_out[i] = -1;
    }

    heap_ent_t *heap = calloc(OEMN_HEAP_CAP, sizeof(*heap));
    int hs = 0;

    dist_out[src] = 0;
    heap_push(heap, &hs, (heap_ent_t){0u, src});

    while (hs > 0) {
        heap_ent_t x;
        if (heap_pop(heap, &hs, &x) != 0)
            break;
        int u = x.v;
        if (x.d != dist_out[u])
            continue;
        for (int k = 0; k < r->deg[u]; ++k) {
            int v = r->nbr[u][k];
            uint32_t ew = r->w[u][v];
            if (ew == OEMN_INF_DIST)
                ew = OEMN_EDGE_DEFAULT_COST;
            uint32_t nd = dist_out[u] + ew;
            if (nd < dist_out[v]) {
                dist_out[v] = nd;
                parent_out[v] = u;
                heap_push(heap, &hs, (heap_ent_t){nd, v});
            }
        }
    }
    free(heap);
}

The routing table maps: destination → next hop → distance in hops

When you send a message to node 5, it automatically routes through the neighbor closest to node 5. If that neighbor fails, the next topology update (every 2-4 seconds) recomputes and reroutes through an alternate path.

This is AUTOMATIC FAILOVER. No manual intervention. No central coordinator.

ENCRYPTION & REPLAY PROTECTION: DEFENDING AGAINST ATTACKS

I use AES-256-GCM (via OpenSSL) or ChaCha20-Poly1305 (via libsodium) for authenticated encryption.

Every message is encrypted with a per-source key, and every message has a sequence number embedded in its ID.

But encryption alone isn't enough. I had to prevent replay attacks:

Attacker sees: "Node A sends $100 to Node B"
Attacker captures the encrypted packet
Attacker replays the same encrypted packet
Node B doesn't know it's a duplicate
Attacker just transferred $100 twice

This is trivial to execute on UDP because there's no session state.

OEMN solves this with a PER-SOURCE SLIDING WINDOW:

For each source, track the last 64 received sequence numbers
If we see sequence N where N <= (current_seq - 64), reject as duplicate
If we see N where (current_seq - 64) < N < current_seq, reject as duplicate
Only accept N > current_seq

Here's the actual code from replay.c:

int oemn_replay_check_and_update(oemn_replay_t *r, uint32_t src_id, const uint8_t msg_id[16])
{
    uint64_t seq = msg_seq_from_id(msg_id);

    pthread_mutex_lock(&r->mu);
    int idx = -1;
    for (int i = 0; i < OEMN_MAX_PEERS; ++i) {
        if (r->slots[i].used && r->slots[i].src_id == src_id) {
            idx = i;
            break;
        }
    }

    oemn_replay_slot_t *s = &r->slots[idx];

    if (seq > s->max_seq) {
        uint64_t gap = seq - s->max_seq;
        if (gap >= 64)
            s->window = 1ULL;
        else
            s->window = (s->window << gap) | 1ULL;
        s->max_seq = seq;
        pthread_mutex_unlock(&r->mu);
        return 0;  // Accept: fresh sequence
    }

    uint64_t delta = s->max_seq - seq;
    if (delta >= 64) {
        pthread_mutex_unlock(&r->mu);
        return 1;  // Reject: too old
    }

    uint64_t mask = 1ULL << delta;
    if (s->window & mask) {
        pthread_mutex_unlock(&r->mu);
        return 1;  // Reject: duplicate
    }

    s->window |= mask;
    pthread_mutex_unlock(&r->mu);
    return 0;  // Accept: fresh
}

This costs 64 bits of memory per source. At 50 nodes, that's 400 bytes total. Negligible.

The attacker can't replay old messages. They can only forge new ones (which requires the encryption key, which they don't have).

BUFFER POOLING: FROM MALLOC THRASHING TO SLAB ALLOCATION

One thing I discovered: malloc/free on every message is a performance killer.

OEMN uses pre-allocated SLAB BUFFERS:

From fwd_pool.h:

#define OEMN_FWD_POOL_SLOTS 48
#define OEMN_FWD_BUFSZ (OEMN_HDR_SIZE + OEMN_MAX_PAYLOAD)

typedef struct {
    pthread_mutex_t mu;
    uint8_t slab[OEMN_FWD_POOL_SLOTS][OEMN_FWD_BUFSZ];
    int free_stack[OEMN_FWD_POOL_SLOTS];
    int nfree;
} oemn_fwd_pool_t;

Instead of malloc-ing on every message, OEMN allocates one giant slab upfront. Each thread acquires a buffer from the pool, uses it, then returns it. No heap fragmentation. No allocation overhead.

This is a simple optimization, but it matters more than you'd expect. Systems that malloc constantly spend more CPU managing memory than doing useful work.

LESSONS LEARNED

THREAD SAFETY IS HARD BUT WORTH GETTING RIGHT

I spent weeks hunting data races using ThreadSanitizer. The payoff: zero race conditions under stress testing. This is how production systems should be built. Most open-source projects skip this. Don't.

TOPOLOGY CHANGES ARE YOUR PERFORMANCE BOTTLENECK

Dijkstra is O((V + E) log V). For 50 nodes, it's still fast. But if you rerun it on every message (amateur mistake), you're doing wasteful computation.

OEMN only recomputes on topology changes. This is the right optimization target.

UDP ISN'T UNRELIABLE, IT'S JUST HONEST

TCP hides failures through retries and buffering. UDP shows them immediately.

In a disaster scenario, seeing failures is better than silently dropping 10% of messages under congestion (which TCP does). You need visibility into what's happening.

BUFFER POOLING WINS OVER MALLOC/FREE

Pre-allocated slabs eliminate allocation overhead. The threading model means each buffer acquisition is an O(1) stack pop. No freelist searching. No fragmentation.

WRITING THINGS IN C MAKES YOU UNDERSTAND HARDWARE

Python and Rust abstract away memory. C forces you to think about:

Cache line locality
Malloc overhead
Mutex contention
Thread synchronization primitives

By the end, you understand networking differently.

WHAT'S NEXT

The current roadmap has three priorities:

LATENCY-WEIGHTED ROUTING: Currently, OEMN finds the shortest path (fewest hops). But 10 hops of low-latency LAN is better than 3 hops of high-latency satellite. Adding EWMA-based RTT measurement per edge is next.
DELTA TOPOLOGY UPDATES: Right now, nodes send full topology advertisements. With 100+ nodes, that's bandwidth-heavy. Delta updates (send only changes) could cut bandwidth by 10x.
FUZZING: I want to throw 10,000 random packet corruptions at the protocol and watch it survive. libFuzzer + AddressSanitizer.

WHY THIS MATTERS

Most software engineers will never think about resilience. You build on Kubernetes, AWS, and 99.99% uptime guarantees.

But when infrastructure fails, when the easy assumptions break, mesh networks become critical infrastructure.

Understanding how to build one teaches you:

How to program without central authority (coordination, consensus)
How to handle adversarial networks (encryption, replay protection)
How to design concurrent systems that don't have race conditions
How to measure and optimize for real-world constraints (packet loss, node failures)

OEMN isn't just a project. It's a masterclass in systems engineering.

But more importantly: it works. No dependencies. No cloud. Just peers talking to peers.

TRY IT

OEMN is on GitHub: github.com/bharqav/oemn

To run the demo locally:

make
./oemn --id 1 --port 7777
./oemn --id 2 --port 7778

Then connect them (in the CLI):

# on node 1: peer add 2 127.0.0.1 7778
# on node 2: peer add 1 127.0.0.1 7777
# on node 1: send 2 hello

Watch the message travel. Watch the ACK come back. Watch the routing table update in real-time.

The code is production-grade:

48-slot buffer pool
Thread-safe by design
Dijkstra shortest-path routing
AES-256-GCM encryption
Per-source replay protection
Graceful shutdown

Read the docs in the repo. They're detailed. Run the benchmarks yourself. See what performance you get on your hardware.

If you're curious about how systems really work, build your own. Build a mesh network. Build an autograd engine. Build a shell.

Don't just use other people's abstractions. Understand the layer beneath.

That's where the real learning happens.

OEMN is open-source (MIT License). Contributions welcome: github.com/bharqav/oemn