DEV Community: Joshua Chi

d-engine Dev Notes (Part 1): The Hard Parts of Software Development

Joshua Chi — Wed, 28 Oct 2026 04:00:00 +0000

Want

To explain where d-engine really came from, I have to go back four or five years — to before d-engine itself existed.

The first time I came across Kafka — especially Kafka Streams and Kafka Connect — I got hooked on event streaming platforms: pulling data in different formats together and aggregating it solved a lot of real business problems. An event is an abstraction of the real world: something that has already happened, a plain fact. That idea of "event" goes back, in my memory, to reading how Martin Fowler talks about event-driven architecture. But actually standing up and running a real Kafka cluster eats up a lot of hardware and a lot of your ops team's time — and using Confluent, the commercial version, is expensive for SMEs (small and medium-sized businesses). My impression at the time was that it was billed by the hour.

What I really wanted, even back then, was to make distributed streaming cheap and simple enough for any developer to run on their own — not something only a large team with a dedicated ops org could afford. That's the same vision that later became d-engine's: distributed coordination should be cheap to run and simple to use.

My own background has always been tied to messaging — I built an XMPP-based “online presence” component in Erlang early on, and later worked on a push notification platform — the same kind of system behind things like Google's push notification services (GCM). So I spent a year and a half building my own Kafka-style distributed event streaming platform in Go, starting with my own stream core. I got stuck at the consumer group part — it was just too slow, and looking back, some of that was probably just my own inexperience at the time. For the underlying distributed storage layer, I used embedded etcd rather than a standalone etcd cluster — I really liked that design. There were a lot of detours and research along the way.

Before deciding to implement Raft myself, I looked at alternatives — TiKV, for example — but they all assumed a standalone cluster, and standing one up was just as expensive. That wasn't what I wanted. So after a while, I decided to start by implementing the Raft protocol myself. The hardest part wasn’t writing the code — it was proving the implementation actually matched the Raft paper, that it was correct, not just that it ran. So I spent a few months learning TLA+. TLA+ isn’t a programming language at all — it’s a formal specification language: you use math to abstract Raft's components and how they behave, then use a model checker to explore the reachable states of that model, verifying that things like leader election still hold up under concurrent events — within the bounded state space you define, not an exhaustive proof over every possible case. It's a white-box method, and later on I paired it with Jepsen, which tests the same kind of thing from the outside, as a black box.

I respected Leslie Lamport, the author of Paxos. He hosts The TLA+ Video Course. What drew me in was how he taught — every lecture filmed in a different natural setting, and he kept changing his outfit. Small things, but they gave a dry subject some warmth and wonder, and I found myself looking forward to the next lecture every time. A few months later I had a model-checked design for d-engine’s Raft core, and only then did I start actually coding.

I guess what I’m trying to say is: wanting to do something always has a cost. And paying that cost is exactly how you find out how real your interest actually was.

Follow Nature

If I had to point at one idea that quietly shapes most of d-engine's architecture, it's this: in nature, the things that go with the flow — that work with nature's own rules instead of fighting them — take the least energy. I've felt this pattern again and again, and it's the same principle d-engine's architecture tries to follow.

A good example is how d-engine drains its inbound event channel. I was stuck on a performance problem, and the moment I saw this pattern in the open-source world, I knew it was the answer — because I already carried an old story with me: Li Bing and the Dujiangyan irrigation project. He didn't try to tame the flood by damming it; he worked with the flow instead, channeling it. Draining the channel follows the same idea: instead of pulling one event at a time, or blocking until a batch is "full," just drain whatever is sitting in the channel right now, up to a cap:

/// Drain all pending inbound events (up to max_batch_size).
async fn drain_inbound_events(&mut self) -> Result<()> {
    let max = self.ctx.node_config.raft.batching.max_batch_size;
    let mut count = 0;
    while count < max {
        match self.event_rx.try_recv() {
            Ok(inbound_event) => {
                if self.buffered_inbound_event.is_empty() {
                    self.inbound_event_oldest_pending_since = Some(std::time::Instant::now());
                }
                self.buffered_inbound_event.push_back(inbound_event);
                count += 1;
            }
            Err(_) => break,
        }
    }
    Ok(())
}

try_recv never blocks — it just grabs what's already there. So when traffic is quiet, this loop exits almost instantly with a small batch. When traffic spikes, it keeps pulling, up to max_batch_size, and the batch naturally grows. Nothing is throttled; the batch is simply capped at max_batch_size, growing and shrinking with the flood.

See

Seeing the problem is hard too, especially in a distributed, async system. Inside a single millisecond, a huge number of things are happening at once — across machines, code paths, and the network. Concurrency has to preserve ordering and consistency, on top of coordinating Raft's own network layer. There's just too much going on at once. Making all of that complexity something you can actually see, simply — that's the core of how I deal with architecture debugging.

AI is being used more and more these days, and I've tested this myself: hand a pile of performance metrics to a model, and it's quick to catch small bugs — but in the face of this many variables moving at once, it tends to miss the one thing that actually matters most, the real design flaw. Spotting that still takes an experienced human eye; AI isn't there yet. So rather than wait for a model that can "see" for me, I built a few side tools that make all that complexity something I can see for myself — repeatably, almost boring in a good way.

A diagnostic screenshot: 3 nodes and the load generator on one 8-core machine, data on a RAM disk. It shows where one write spends its time, not a benchmark result.

This kind of work pays off many times over — just stare at this chart for a while, let your mind drift, and ideas start popping into your head on their own. This is the view that grounds me; it’s how I actually figure out what to fix next.

Closing Thoughts

The hardest things in this world are often the simplest ones — and that's exactly why they're so hard to write about. A post like this can't really capture something simple; you only get there by practicing it, by living inside it for a while.

I’ve been writing d-engine’s code for three or four years now, as a solo developer. It's been my way of putting how I understand the world into practice, one commit at a time. I hope it can give something back to the open-source community, even in a small way. Thanks for reading, and for the support.

d-lmdb is now available as a Docker image: a 3-node cluster and failover demo

Joshua Chi — Fri, 31 Jul 2026 01:00:00 +0000

I’m happy to share d-lmdb-server and the corresponding Docker image: deventlab/d-lmdb.

The code is here: github.com/deventlab/d-lmdb.

If you want to try it without reading the code first, you can now test it directly with Docker. It should take about 5 minutes to get started.

What I wanted to do here is simple: turn an experimental distributed LMDB setup into something easier to try, so people can validate it quickly and give feedback more easily.

If you try it and find problems, or think the docs, API, or UX can be improved, PRs and issues are very welcome.

1. You can now run d-lmdb-server directly with Docker

d-lmdb-server can now run as a standalone HTTP service.

It supports:

quick single-node startup
a 3-node HA cluster demo
continued reads and writes after leader failover

For me, the main point of this step is to let developers try it with the lowest possible cost, without embedding Rust code into their own applications.

2. Why build a Docker version

When I look at infrastructure projects, I usually ask one very practical question:

Can I run it in 5 minutes and see if it is a good fit for me?

The Docker version is built for this use case.

It is not meant to replace the library version, and it is not meant to add an unnecessary extra layer.

It is just a way to help people try it faster and start validating it sooner.

If you are:

not sure yet whether you want to embed the library
interested in a standalone service
trying to test multi-node behavior and failover

then the Docker version may be easier to start with.

3. One-sentence description

d-lmdb-server is a standalone HTTP service for distributed, LMDB-backed key-value storage.

It is not trying to replace LMDB itself.

It is more like a demo of how to add a replicated write path to an existing embedded storage engine.

If you prefer a shorter version:

Reads stay local. Writes go through Raft. Each node still stores the full dataset.

That means:

reads still use the local node by default
writes go through Raft consensus
every node keeps the full dataset, so there is no sharding

4. How to try it

Single node

If you just want to see whether the service starts correctly, you can try the single-node setup first:

docker rm -f dlmdb 2>/dev/null   # clean up from a previous run
docker run -d --name dlmdb -p 8080:8080 -v dlmdb-data:/data deventlab/d-lmdb
sleep 3                            # wait for startup
curl -s -o /dev/null -w 'PUT: %{http_code}\n' -X PUT localhost:8080/kv/hello -d world
curl localhost:8080/kv/hello

If everything works, you should see:

PUT returns 204
GET returns world

There is no extra config needed for the first start.

A single-node TOML file will be generated automatically.

If you want to use your own config, you can mount a custom config.toml.

3-node cluster

If you want to see how it behaves in a multi-node setup, you can try the 3-node cluster:

git clone https://github.com/deventlab/d-lmdb.git && cd d-lmdb
docker pull deventlab/d-lmdb:latest
docker tag deventlab/d-lmdb:latest d-lmdb-server:compose   # compose uses this local tag
docker compose down -v 2>/dev/null
docker compose up -d
sleep 10   # wait for leader election and health checks

Then you can check the cluster status:

docker compose ps
curl -s -o /dev/null -w 'node1: %{http_code}\n' localhost:18081/primary
curl -s -o /dev/null -w 'node2: %{http_code}\n' localhost:18082/primary
curl -s -o /dev/null -w 'node3: %{http_code}\n' localhost:18083/primary
curl -s -o /dev/null -w 'PUT: %{http_code}\n' -X PUT localhost:8080/kv/hello -d world
curl localhost:8080/kv/hello

If you want to test failover, you can kill the leader and keep writing:

docker compose kill -s KILL node1   # replace this with the actual leader node
sleep 5
curl -s -o /dev/null -w 'PUT: %{http_code}\n' -X PUT localhost:8080/kv/post-failover -d alive
curl localhost:8080/kv/post-failover

What I like most about this part is not that it looks complete, but that you can directly feel one thing:

after the leader goes down, the client does not need to know that anything changed internally.

compose/smoke-test.sh contains the full automated walkthrough, including leader crash and recovery checks.

5. What it roughly does

If I describe the architecture in one sentence:

Reads stay local. Writes go through Raft. Each node still stores the full dataset.

The goal here is not sharding or scaling out.

It is fault tolerance.

In a way, it turns LMDB, which is a local embedded storage engine, into a system that can tolerate failures across multiple nodes.

6. This is still 0.1.0, so feedback is very welcome

d-lmdb-server is still an early version.

What I hope to learn from real usage is:

whether the documentation is clear enough
whether the API feels natural
whether the failure cases are covered well enough
whether the read/write semantics are easy to understand

If you try it and find anything that can be improved, please open an issue.

If you find a bug or a clear way to improve the code or docs, PRs are also very welcome.

7. Closing

If you want to try it, you can pull the Docker image deventlab/d-lmdb and give it a try.

d-lmdb-server is still early-stage, and I hope it can be used more in real situations first, then improved based on feedback.

Project page:

d-lmdb

Issues, PRs, and direct feedback are all very welcome.

d-lmdb: A Distributed LMDB Built on d-engine (Experimental)

Joshua Chi — Sun, 26 Jul 2026 15:59:00 +0000

I’m happy to open source d-lmdb. This is still an experimental project. The goal is to try using d-engine’s distributed capability to “wrap” an existing KV store.

There are already many Raft-based distributed KV systems in the community (many written in Go). d-engine itself is not a KV database, but in my actual work I do need a simple and reliable KV store. So I chose LMDB, which is well-known and stable, to run this experiment. If this approach works, I can also use it directly.

After putting the architecture together, one thing became quite clear: adding a “consensus-based write path” to a single-node KV is actually feasible, and more straightforward than I initially expected. Of course, most of the complexity still comes from the nature of distributed systems vs single-node systems. More details will likely show up during real usage.

What d-lmdb is (and is not)

d-lmdb is an experimental project. The core path is working, and it can be used for evaluation and testing, but it is not production-ready.

It is also trying to answer a question: can d-engine be reused on top of an existing storage engine to provide a “replicated write path”? LMDB is the first working example. In theory, the same pattern should also work with other embedded storage engines (not fully verified yet).

Architecture

Simplified model

d-engine runs in embedded mode, without a separate server process (each node is a peer process). Reads bypass the consensus layer and go directly to the local LMDB instance:

Your process
    │
    ├─ db.get(key)      ────────────────► Local LMDB (sync, local read)
    │
    └─ db.put(key, val) ──► d-engine Raft ──► Local LMDB (async, replicated)
                             leader election
                             log replication
                             quorum commit

Core idea

This design adds a distributed write path to an existing storage engine.

By default, get() reads locally without linearizability guarantees. If you need linearizable reads, d-lmdb also provides get_linearizable(), which is backed by d-engine’s ReadIndex / lease-read and goes through the Raft layer.

For example:

// LMDB (before)
let val = txn.get(db, &key)?;

// d-lmdb (after)
let val = db.get(&key)?;   // still sync, still local read

The unavoidable change is that writes become async:

db.put(b"user:1", b"alice").await?;
db.delete(b"user:1").await?;

This pattern — “local reads, consensus writes” — is what d-engine is trying to demonstrate. It is not tied to a specific KV implementation, but can be reused on top of other storage engines.

What d-lmdb solves

Single point of failure → quorum fault tolerance
Writes go through Raft. As long as a majority of nodes are alive, the system can continue serving. Native LMDB has no replication, and recovery from node failure can be difficult.
Strong consistency across nodes (for writes)
All nodes share the same linearizable write history. The only additional dependency is d-engine itself.
Read performance is not affected
get() reads directly from local LMDB, so read latency is not impacted by replication. However, reads are not guaranteed to be linearizable.

What d-lmdb does NOT solve

Write throughput will not improve
A local LMDB write is one fsync. A d-lmdb write is a network round-trip plus quorum fsync. This trades latency and throughput for availability and consistency (which is expected in distributed systems).
No sharding
d-engine replicates data but does not partition it. Each node stores the full dataset.
Still just a KV store
Adding Raft does not add relational modeling or complex queries.
Operational complexity
Raft introduces real operational concerns: leader election, membership changes, snapshots, log compaction, etc.

Current status

d-lmdb is still in an early stage. The core functionality works and can be used for evaluation and experiments. Feedback is very welcome.

It has not been validated in production yet, and both API and internal behavior may change.

If you are looking for a production-ready distributed KV, this is not there yet. But if you want to see what it looks like to “add a replicated write path” to an existing storage engine, d-lmdb can serve as a concrete example.

Final note

d-lmdb is not the end goal. If you already have your own storage engine or application and want a similar model (“local reads + consensus writes”), you can use d-lmdb as a reference or starting point.

d-lmdb itself is just one concrete implementation of that pattern. The more general part lives in d-engine.

Code: https://github.com/deventlab/d-lmdb
Issues and PRs are welcome.

d-engine: Embedded Distributed Coordination for Rust Apps

Joshua Chi — Sat, 25 Jul 2026 15:30:00 +0000

If you're writing a backend service in Rust and need a bit of distributed consistency ability — leader election, state synchronization, simple configuration coordination — but don't want to pull in a separate system to deploy and operate, this post is for you.

Is d-engine right for you

If your service already runs in the Rust ecosystem and you want to embed coordination directly into your application, rather than deploying a separate piece of middleware, d-engine is worth a look. It fits leader election, state synchronization, lightweight coordination, and scenarios where you want to start with a single node and grow to a cluster only when you need to.

If your team values simplicity and low operational overhead over standing up a full distributed infrastructure stack from day one, d-engine's design is aimed at you.

What is d-engine

d-engine (crates.io) does one thing: it's a distributed coordination engine. It's not a message queue, not a database — it's a foundational component, and the goal is to make coordination clearer and more reliable.

DEventLab's founding motivation is to make distributed systems development cheap. "Cheap" here isn't just about running on inexpensive hardware — it's about low cost to adopt, low cost to set up, and low cost to maintain over time.

Making something "cheap" for developers to use isn't easy. The principles behind d-engine's development reflect that: keep the architecture simple, avoid complexity unless it's earned, keep long-term maintenance cost low, and keep the code and comments easy to follow.

Simple architecture: state changes are processed serially

d-engine's Raft core state machine runs on a single logical execution path. Every state change — term changes, log writes, commit index advancement — is serialized. Network I/O and disk I/O can happen concurrently, but the state-change step itself is always processed in order, one at a time. There's no scenario where two operations race to modify the same state.

That means you don't need to reason about complex multi-threaded interleavings in your head — most of the time, you can just follow the log and see where the queue currently is. That makes debugging and reasoning about consistency more direct.

How it embeds into your application

In embedded mode, d-engine compiles directly into your existing Rust application as a local library — no separate deployment, no extra distributed system to operate. From your application's point of view, it behaves more like an embedded coordination layer than an external service you have to babysit.

Your application is a Raft node. Coordination stops being an external dependency and becomes part of the service itself.

Developer architecture diagram

The diagram below shows the layers a developer actually touches: EmbeddedClient on top, the d-engine core in the middle, storage and consensus underneath. Strongly consistent reads/writes and weakly consistent fast reads take different paths — that's one of the key points in the development experience.

Starting with a single node

You can start with a single node to validate your logic, then scale out to a multi-node cluster once you actually need high availability — without rebuilding anything. For early-stage projects, this path tends to work better: validate the business logic first, then add system complexity incrementally.

If you want to see it in practice, check out the single-node-expansion example.

Multi-node deployment example

If you need a multi-node deployment, the examples demonstrate one reference pattern: a 3-node cluster behind a load balancer, with writes routed to the leader, reads spread across all nodes, and the load balancer detecting each node's role through health-check endpoints.

To be clear about the boundaries here: this is a pattern we demonstrate in the examples for reference — it hasn't been validated in production. Actual failover time and data-safety guarantees depend on your deployment environment, network conditions, and load, so you should test it in your own setting.

The division of labor in one sentence: the load balancer handles routing, d-engine handles consistency.

Not limited to Rust

If your team doesn't use Rust, standalone mode lets you connect to a d-engine cluster over gRPC. See the quick-start-standalone example (Go client + 3-node cluster). That way you don't have to pull Raft logic into your whole stack — you can use the coordination layer as an independent piece of infrastructure instead.

This tends to work better for cross-language teams, and it's easier to adopt incrementally into an existing system.

This is just the first step

d-engine is still growing, and we don't want to claim it's the answer to every distributed coordination problem. The more honest framing: this post explains what it is, how to use it, and what kind of developer it's a good fit for.

If you want to try it hands-on, start with the examples directory — everything from a 5-minute quick start to a 3-node cluster. If you're evaluating a similar distributed coordination solution, feel free to open an issue on GitHub and tell us about your use case.

d-engine: A Lightweight Distributed Coordination Engine for Rust

Joshua Chi — Fri, 16 Jan 2026 03:57:20 +0000

A lightweight Raft implementation designed for embedding into Rust applications — the consensus layer for building reliable distributed systems.

Built with a simple vision: make distributed coordination accessible - cheap to run, simple to use.

Built on a core philosophy: choose simple architectures over complex ones.

Why I Built This

In my experience, adding distributed coordination to applications was always expensive – existing solutions like etcd are either too slow when embedded (gRPC overhead) or require running separate 3-node clusters.

d-engine aims to change this: make coordination cheap enough that you don't hesitate to add it when you need it.

Architecture

Single-threaded event loop: No race conditions, strict ordering. CPU-bound on single core—scale horizontally.

Role separation (SRP): Leader/Follower/Candidate/Learner each handle their own logic. Main loop just routes events.

Standard design for Raft implementations.

Two Integration Modes

Embedded mode - runs inside your Rust process:

let engine = EmbeddedEngine::start().await?;
engine.wait_ready(Duration::from_secs(5)).await?;;

let client = engine.client();
client.put(b"key".to_vec(), b"value".to_vec()).await?;  // <0.1ms

Direct memory access via mpsc channels (no serialization)
<0.1ms latency for local operations
Single binary deployment
Benchmark: 203K writes/s, 279K reads/s (M2 Mac)

Standalone mode - separate server process with gRPC:

d-engine = { version = "0.2", features = ["client"], default-features = false }

Language-agnostic (Go, Python, Java, Rust)
Independent scaling
Benchmark: 64K writes/s, 12K reads/s

Choose based on your language and latency requirements.

Benchmarks

Embedded mode (3-node cluster, M2 Mac localhost):

203K writes/s, 279K reads/s (linearizable)

Standalone mode (gRPC):

64K writes/s, 12K reads/s (linearizable)

Lab numbers only—production performance varies by workload and hardware.
Full details in benches/.

Design: Traits for Extension

d-engine provides working implementations (RocksDB storage, KV operations, gRPC transport). When defaults don't fit, implement the traits:

pub trait StorageEngine {
    type LogStore: LogStore;
    type MetaStore: MetaStore;
}

pub trait StateMachine {
    async fn apply_chunk(&self, entries: Vec<Entry>) -> Result<()>;
}

Examples in repo show Sled storage backend, custom HTTP handlers with HAProxy for HA deployments.

Consistency Model

Three read policies (configurable per-operation or server-wide):

LinearizableRead: Strongest guarantee, verify quorum (~2ms)
LeaseRead: Leader lease-based, fast path (~0.3ms, requires NTP)
EventualConsistency: Local read, may be stale (~0.1ms)

Trade-offs documented. Defaults are sane.

Start Simple, Scale When Needed

// Start with 1 node (auto-elected leader)
let engine = EmbeddedEngine::start().await?;

// Scale to 3 nodes later: update config, zero code changes
// See examples/single-node-expansion

No Kubernetes, no complex setup. Just Rust + config file.

What It Is

Raft consensus implementation
Pluggable storage (RocksDB, Sled, custom)
Flexible consistency (linearizable/lease/eventual)
Production-ready core, API stabilizing toward v1.0

Current State & Direction

Version 0.2

Core Raft engine is solid (1000+ tests, d-engine 0.1.x version Jepsen validated).
APIs are stabilizing toward v1.0. Pre-1.0 means breaking changes are possible
if they improve design.

No production users yet – looking for early adopters to validate real-world use cases.

Future Direction:

Cloud-native storage backends (Cloudflare, AWS, GCP)
Exploring: etcd-compatible API layer
Timeline depends on community feedback

Try It

[dependencies]
d-engine = "0.2"

GitHub: https://github.com/deventlab/d-engine
Docs: https://docs.rs/d-engine
Examples: https://github.com/deventlab/d-engine/tree/main/examples

If you're building distributed systems in Rust and need consensus, the code is there. Examples show embedded mode, standalone mode, custom storage, HA deployments with HAProxy.

Looking for Early Adopters

I'm looking for Rust developers building distributed systems who have real problems to solve:

Coordination bottlenecks in your architecture (slow consensus, expensive etcd clusters)
Specific use cases where strong consistency matters (leader election, distributed locks, metadata stores)
Production deployments where you need cheap, simple, and reliable coordination

Particularly interested in:

Teams with real coordination problems (not just kicking the tires)
Cloud-native scenarios (Cloudflare Workers, AWS Lambda, serverless)
Cost-sensitive use cases where etcd feels like overkill

What I'm offering:

Free architecture review for your use case
Direct access to roadmap planning
Quick response to GitHub issues/questions

If you have a specific problem: Open an issue with your use case, or reach out directly.
Show me what's broken, what's expensive, what's too complex. Let's see if d-engine can solve it cheaply.

License: MIT or Apache-2.0 | Platforms: Linux, macOS | MSRV: Rust 1.88