On February 27, 2026, the Apache Iggy team published a blog post documenting one of the more honest engineering war stories to come out of the Rust ecosystem this year: a months-long effort to replace Tokio's work-stealing async runtime with a thread-per-core, io_uring-driven architecture. The post reads less like a victory lap and more like a field report from the trenches — full of dead ends, panics, and a redesign that eventually cracked, then partially failed, before landing somewhere workable.
Iggy is a high-performance message streaming platform written in Rust — think a leaner, Rust-native alternative in the same space as Kafka or Redpanda. Performance is the whole pitch, so when the team hit a wall with their existing architecture, they didn't reach for a quick fix. They went looking for a new foundation.
Why Tokio Wasn't Enough
Tokio, the dominant async runtime in Rust, uses a multi-threaded work-stealing scheduler. Worker threads — typically one per CPU core — continuously pick up and reschedule futures, and idle workers "steal" work from busy ones to balance load automatically. It's a genuinely good design for most applications. But it comes with a lack of fine-grained control: the scheduler decides where a future runs, which can mean task migration between cores, cache invalidation, and unpredictable execution paths.
The real dealbreaker, though, was block-device I/O. Tokio's model is readiness-based — it waits for a notification that a file descriptor is ready, then performs the operation. That works fine for network sockets. It does not work for regular files, because the Linux kernel always reports files as "ready," meaning the notification returns instantly and the actual read or write still blocks the thread on page-cache contention. Tokio's workaround is to offload blocking file I/O onto a background thread pool — one that can grow up to 512 threads by default. For a system trying to push hardware to its limits, spinning up hundreds of threads to work around a scheduling model is not a scalable answer.
The Thread-Per-Core Bet
The team's answer was a shared-nothing, thread-per-core architecture — pin one thread to each CPU core, partition data by a hashing scheme, eliminate shared state, and communicate exclusively via message passing. This is the same philosophy behind ScyllaDB and Redpanda, both built on the C++ Seastar framework. The shift is often summarized as trading work stealing for work steering — the system decides up front where work goes rather than rebalancing after the fact.
That approach only works, however, if I/O is genuinely asynchronous rather than quietly parked on a thread pool — which is what pulled the team toward io_uring. Unlike epoll's readiness model, io_uring is completion-based: the application submits an operation into a shared ring buffer, the kernel executes it, and the result lands in a completion ring — no blocking, no notification-then-act round trip.
Picking a Runtime — and Watching Two of Three Die
Rust doesn't have a mature Seastar-equivalent, so the team evaluated three io_uring-backed async runtimes: monoio, glommio, and compio. They prototyped first with monoio, but found its feature coverage lagging behind io_uring's fast-moving API surface, with maintenance activity slowing. glommio — originally built by Glauber Costa, who'd previously worked on Seastar at ScyllaDB — offered a richer, more opinionated API, including three separate io_uring instances per thread for different latency profiles. It too turned out to be effectively unmaintained; Costa has since moved on to the Turso database project.
They settled on compio, largely because its driver and executor are decoupled, its io_uring coverage keeps pace with new kernel features, and its maintainers merged the team's own patches within hours of submission.
*Where the Design Actually Broke
*
The most candid section of the post concerns a redesign that failed outright. With no shared state between shards, RefCell-based interior mutability looked like a natural fit — until the team ran into runtime panics from holding a RefCell borrow across an .await point, a known Rust async footgun with its own Clippy lint. They restructured their data model around an Entity Component System (borrowed from game engine design) to split in-memory state from storage and avoid holding borrows across awaits. That fixed the panics but exposed a deeper problem: broadcasting state-change events between shards introduced non-deterministic orderings that even a dedicated single-writer shard couldn't fully tame. The team ultimately labeled the pure shared-nothing experiment a failure.
The fix was a hybrid: strongly-consistent shared resources (streams, topics) now go through a left-right concurrent structure — a single-writer, multi-reader design where one shard owns writes and others hold read-only handles — while eventually-consistent, per-partition data stays sharded. They call this split a control-plane/data-plane division, and it's the architecture that shipped.
The Numbers, and the Dates Behind Them
The team benchmarked three released versions: v0.5.0 (the original Tokio-based build), v0.6.1 (the initial thread-per-core plus io_uring iteration), and v0.7.0 (the hybrid design). At 32 partitions with 32 producers and streams pushing 80 GB of data, v0.7.0 cut p95 latency by roughly 57% and p99999 by a similar margin compared to v0.5.0, with the gap widening as load increased — at 8 partitions the two versions performed almost identically, but by 32 partitions v0.7.0 was clearly ahead across every percentile. In strong-consistency mode with fsync enabled on every batch, throughput improved by around 18% and tail latencies dropped by 20–45%.
Why This Matters Beyond One Project
Iggy's story lands at a moment when io_uring is turning up across the stack — not just in Rust, but in Zig's new async I/O interface and in Microsoft's ongoing work adding io_uring socket support to .NET on Linux. What makes the Iggy post worth reading isn't the destination; it's the honesty about the dead ends along the way, and the closing observation that Rust's async ecosystem still lacks a true Seastar-equivalent framework — with glommio, the closest attempt, now largely dormant. The team's stated next step, previewed in the same post, is building out clustering support using Viewstamped Replication — a story they say is still being written.
this is my github profile
you can also find me on linkedin
until next time, peace, focus
Top comments (1)
the glommio situation is rough losing the closest thing Rust had to Seastar because the maintainer moved on is exactly the kind of ecosystem gap that makes thread-per-core adoption stall. curious if compio is actually ready to carry that torch or if Iggy is essentially betting on an unproven runtime for production workloads