Porting Concurrent C++ to Rust: Field Notes From a Few Migrations

Who should read this !

If you are a Rust engineer who has just been handed a concurrent C++ codebase — a network service, an ingestion daemon, a gateway, a media pipeline — and told "we're moving this to Rust", this is the post I wish someone had handed me the first time it happened to me.

A caveat before anything else: this is not the story of one migration. It's a composite of a few I've done or consulted on over the last several years. The systems were different (an edge proxy, a log ingestion daemon, something broker-shaped), the teams were different, and the outcomes were different. The incidents I describe happened, but I've deliberately blurred which codebase each one came from, and I've rounded the numbers. Where I generalise, treat it as pattern-matching over a small sample, not a law.

The thesis I'll keep coming back to is this: in a concurrent C++ codebase, the concurrency is the migration. Types port mechanically. Serialization ports mechanically. Business logic is tedious, but it is only tedious. The thing that eats your schedule, your sleep, and your relationship with the C++ engineers who wrote the original is the layer where threads, locks, io_contexts and atomics meet. That's where the implicit contracts live — "this handler never runs concurrently with that one", "this queue has exactly one producer", "this object is never destroyed while that thread is alive" — and Rust refuses to compile until every one of them is made explicit. That is the whole reason you're moving. It is also why it hurts.

The codebases I'm describing used three concurrency styles, usually all at once and often in the same file:

• classic std::thread + std::mutex + std::condition_variable, with a hand-rolled thread pool or two
• Boost.ASIO with completion handlers, and in the newer parts, C++20 coroutines on asio::awaitable
• a small number of hand-written lock-free structures on the hot paths: an SPSC ring, an MPSC queue, a seqlock, a few std::atomic flags doing load-bearing work

If your codebase looks like that, read on. If it's OpenMP/TBB numeric code, the audit and testing sections still apply, but the Rust side is rayon, not Tokio, and I don't cover it here.

I'll give both migration strategies — incremental via FFI, and the big-bang rewrite — equal time, because I've done both and I don't think either is the default. Which one is right depends mostly on whether the concurrency architecture you inherited is something you want to keep.

Step zero: audit the concurrency before you write any Rust

Every migration I've been part of that went badly skipped this step. The ones that went well spent two to four weeks on it before a single cargo new.

The output is a document, a boring one, that answers: what threads exist, who creates them, what they own, what they share, and how they talk. In C++ none of this is in the type system, so you have to find it by reading. The point of writing it down is that Rust is going to ask you every one of these questions at compile time anyway. You'd rather have the answers ready than discover them one E0277 at a time.

What I look for, concretely:

Thread inventory. Grep for std::thread, pthread_create, boost::thread, std::async, and every io_context::run. For each: when is it spawned, when (if ever) is it joined, and what is its lifetime relative to the object that spawned it. The "detached thread that references a member of the object that spawned it" pattern has shown up in every codebase I've audited. Each one of these is a Rust compile error, pre-announced, because std::thread::spawn demands 'static.

Lock inventory. Every std::mutex, shared_mutex, recursive_mutex, and homemade spinlock. For each: what data does it actually protect — not what the comment says, the comment is from 2017 — and what is its lock-order relationship with the others. Draw the lock-order graph on a whiteboard. If there's a cycle, you've found a latent deadlock that hasn't fired yet because of timing. Flag every recursive_mutex specially: Rust's std has no equivalent, and its presence almost always means re-entrancy the design never intended.

Condition variables. For each condition_variable, find the predicate, every notify site, and whether the wait is in a loop or uses the predicate overload. Spurious wakeups translate fine. What I'm really hunting is the lost wakeup: a state change and a notify that happen without the mutex held, racing a waiter that has checked the predicate but not yet slept. Those bugs survive a literal translation to Rust's Condvar perfectly intact.

ASIO topology. How many io_contexts, how many threads call run() on each, which handlers are wrapped in a strand and which are not. The unstranded handlers that touch shared state are your race conditions. Also list every shared_from_this() — that's C++'s answer to "the callback might outlive the object", and each one becomes a decision about ownership in Rust.

Lock-free structures. List them, list the memory orderings used, and find out why each one exists. In my experience about half exist because someone measured contention, and the other half exist because someone read a paper. The second half become channels.

Blocking calls hiding in async code. sleep_for inside a handler, synchronous file I/O, a blocking database client, a std::future::get(). In ASIO these quietly reduce throughput on one thread. In Tokio they can wedge the runtime. Flag every one.

Tooling that helps: clang-tidy with the concurrency checks enabled, and a ThreadSanitizer build of the existing C++ run against its own integration tests. You will find real bugs in the C++ before you've written any Rust. Budget time to decide, per bug, whether to fix it in C++ now or let the port fix it; don't let that decision happen by accident.

I always end up with a table shaped roughly like this (a few real-ish rows for flavour):

Thing	Kind	Owner	Shared with	Lifetime	Risk
Session::outbox_	deque under mu_	Session	io thread, flush thread	Session	lock held across async_write
stats_	plain struct, "io thread only"	Session	timer handler (unstranded!)	Session	data race, confirmed by TSan
g_ring	SPSC ring, acq/rel	main	capture thread → encoder thread	process	reclaim-on-shutdown ordering
Flusher thread	std::thread, detached	Store	Store::buf_	longer than Store	use-after-free on shutdown

The most useful artifact, though, is the list of contracts the table implies: "handler H and handler G are never concurrent", "g_ring has exactly one producer", "Store is never destroyed while Flusher is running". These are the invariants C++ enforced by convention and Rust will enforce by type. Having them written down is what lets you choose Mutex<T> vs. owning task vs. channel deliberately in the sections that follow, instead of reaching for Arc<Mutex<Everything>> and hoping.

The mental-model shift: data races are gone, race conditions are not

You already know Send and Sync. The shift I want to describe is about reading the C++ with the right lens, because the C++ and the Rust are closer than they look and that closeness is a trap.

The memory model is the same. Rust's atomics are the C++20 memory model, deliberately. memory_order_acquire is Ordering::Acquire, fences are fences, and the reasoning you did about happens-before in the C++ carries over word for word. What differs is enforcement. In C++, a data race on a non-atomic is undefined behaviour and the compiler trusts you not to have one. In Rust, a data race on a non-atomic is a compile error unless you've written unsafe to get there. That's the entire safety story for concurrency, and it's a big one, but it's narrower than people assume.

Rust eliminates data races. It does not eliminate race conditions. Deadlocks compile. Lost wakeups compile. Check-then-act across two separate lock acquisitions compiles. Ordering bugs between two independent tasks compile. This one, which I ported from C++ almost verbatim, compiled and shipped:

// Two locks, two acquisitions, one race.
if !registry.lock().unwrap().contains(&id) {
    // another task registers `id` right here
    registry.lock().unwrap().insert(id, conn);
}

The fix is the same as it would have been in C++ (hold the lock across the check and the insert, or use the entry API). The borrow checker had no opinion.

C++ mutexes protect code; Rust mutexes protect data. This is the single difference that reshapes architecture. A std::mutex mu_; next to three fields and a comment says nothing about which fields it covers. Mutex<T> is the field. Every time you port a struct, you are forced to answer "which lock protects this?" for every member, and the audit's contract list is where those answers come from. A C++ class that looks like this:

class Session {
  std::mutex mu_;
  std::deque<Msg> outbox_;       // protected by mu_
  std::atomic<bool> closed_{false};
  Stats stats_;                  // "only touched from the io thread" (a comment)
};

becomes something like this, and the comment becomes a type error the first time a second task touches stats:

struct Session {
    outbox: Mutex<VecDeque<Msg>>,
    closed: AtomicBool,
    stats: Stats, // Cell inside → !Sync → Arc<Session> won't cross threads. Good.
}

That error is the compiler doing the audit for you. Don't paper over it with a Mutex reflexively; sometimes the right answer is that stats belongs to the one task that owns the session, and the struct should be split.

&mut is a thing C++ cannot say. There is no way in C++ to express "I am the only one touching this right now"; it's all convention. In Rust every &mut self method is that claim, checked. A lot of C++ methods that were "safe to call from any thread, as long as you hold mu_" become methods on the guarded type that only exist once you've locked.

Thread lifetime is in the signature. std::thread::spawn requires 'static, so the detached-thread-references-a-stack-object bug from the audit won't compile. std::thread::scope is the tool for threads that must end before the function returns, and it's the right translation for most "spawn N workers, join them at the end" C++ blocks.

What the compiler still can't see. It's worth being explicit, because this list is where your bugs will be:

• lock ordering between unrelated Mutexes
• whether a Relaxed load is enough (memory ordering is still your job)
• every invariant inside an unsafe block, which is where your lock-free code is going to live
• cancellation, which is new: a C++ thread runs to completion unless you tell it otherwise, a Rust future can be dropped between any two .awaits

The practical rule I've landed on: translate the invariant, not the code. The C++ says mutex m; int a; int b;. The Rust has to say whether a and b are under the same lock, under different locks, or owned by different tasks. If you can't answer from the audit, that's a question for the original author, and it's worth a meeting.

Porting std::thread, mutex and condition_variable

This is the easiest of the three styles to port and the one where a literal translation does the most damage, because it looks like it worked.

Most hand-rolled thread pools are a channel

Every C++ codebase I've inherited had at least one of these:

class Pool {
  std::vector<std::thread> workers_;
  std::deque<std::function<void()>> q_;
  std::mutex mu_;
  std::condition_variable cv_;
  bool stop_ = false;
public:
  explicit Pool(size_t n) {
    for (size_t i = 0; i < n; ++i)
      workers_.emplace_back([this] {
        for (;;) {
          std::function<void()> job;
          {
            std::unique_lock lk(mu_);
            cv_.wait(lk, [&] { return stop_ || !q_.empty(); });
            if (stop_ && q_.empty()) return;
            job = std::move(q_.front());
            q_.pop_front();
          }
          job();
        }
      });
  }
  void submit(std::function<void()> f) {
    { std::lock_guard lk(mu_); q_.push_back(std::move(f)); }
    cv_.notify_one();
  }
  ~Pool() {
    { std::lock_guard lk(mu_); stop_ = true; }
    cv_.notify_all();
    for (auto& t : workers_) t.join();
  }
};

The mutex, the condvar and the stop flag are all there to implement one thing: a blocking MPMC queue that can be closed. Rust has that as a library type. The port I actually shipped:

use crossbeam_channel::{unbounded, Sender};
use std::thread::{self, JoinHandle};

type Job = Box<dyn FnOnce() + Send + 'static>;

struct Pool {
    tx: Option<Sender<Job>>,
    workers: Vec<JoinHandle<()>>,
}

impl Pool {
    fn new(n: usize) -> Self {
        let (tx, rx) = unbounded::<Job>();
        let workers = (0..n)
            .map(|_| {
                let rx = rx.clone();
                // `rx.iter()` ends when every Sender is dropped. That's the stop flag.
                thread::spawn(move || for job in rx.iter() { job() })
            })
            .collect();
        Pool { tx: Some(tx), workers }
    }

    fn submit(&self, f: impl FnOnce() + Send + 'static) {
        self.tx.as_ref().expect("pool closed").send(Box::new(f)).ok();
    }
}

impl Drop for Pool {
    fn drop(&mut self) {
        drop(self.tx.take()); // closes the channel; workers drain and exit
        for w in self.workers.drain(..) {
            let _ = w.join();
        }
    }
}

The condvar is gone, the stop flag is gone, and so is a bug the C++ version had: one of the pools could be destroyed from a worker thread (a job that dropped the last shared_ptr<Pool>), which made the destructor join() itself and deadlock. In the Rust version that's still possible, but JoinHandle::join from the same thread panics (the underlying pthread\_join\ reports a deadlock) instead of hanging silently, and the 'static bound on jobs makes it much harder to arrange by accident.

If the C++ pool is only ever used as "spawn N workers, do the work, join", don't even write a pool — std::thread::scope plus a channel is twelve lines and no 'static gymnastics.

The mutex family

• std::mutex → std::sync::Mutex<T>. Modern std Mutex is futex-based on Linux, so the old "use parking_lot, it's faster" advice is mostly obsolete. The remaining reasons to pick parking_lot are API: no poisoning, const fn new, and an optional deadlock_detection feature that's worth turning on in debug builds while you're porting.
• Poisoning is new. If a thread panics while holding a std Mutex, the lock is poisoned and lock() returns Err. C++ has no concept here (the mutex is simply unlocked by the lock_guard destructor during unwinding). Decide once, codebase-wide: either parking_lot, or .lock().unwrap_or_else(PoisonError::into_inner) everywhere, or treat poison as fatal. Mixing policies is how you get a service that half-works after a panic.
• std::shared_mutex → RwLock<T>. Measure before keeping it. In two of three cases where the C++ used a shared_mutex, the critical sections were short enough that a plain Mutex was faster in Rust and simpler to reason about.
• std::recursive_mutex → nothing. parking_lot::ReentrantMutex exists but only gives you &T, which tells you what the real fix is: the re-entrant call path usually exists because a callback re-enters the object that invoked it. Restructure so the inner function takes the guard (or the data) as a parameter rather than re-locking.
• std::call_once / function-local statics → OnceLock / LazyLock. These are the easy ones.
• thread_local → thread_local!. The difference is you access it through .with(|v| ...) and can't take a plain reference out, which is annoying for about a day and then prevents the bug it's designed to prevent.

Condition variables

The mapping is mechanical but the predicate inverts, which has bitten me more than once:

std::unique_lock lk(mu);
cv.wait(lk, [&] { return ready; });   // wait *until* ready

let guard = cv.wait_while(mu.lock().unwrap(), |ready| !*ready).unwrap(); // wait *while* not ready

Rust's Condvar::wait takes the guard and gives it back, so you can't accidentally wait on a condvar without the mutex. You can still write a lost wakeup: if the notifier changes the state without holding the mutex and then calls notify_one, a waiter that has checked the predicate but not yet parked misses it. Keep the rule from the audit: state change under the lock, notify after. Rust doesn't check this.

One more thing to carry over from the audit: lock ordering is still your problem. Two Mutex<T>s in a struct say nothing about which to take first. I keep the lock-order graph from the audit as a comment on the struct, and I make any function that takes two locks take them in one place, in one order, through one helper. Boring, effective.

Porting Boost.ASIO, completion handlers and C++20 coroutines to Tokio

This is where most of the calendar goes. ASIO and Tokio solve the same problem and share almost no semantics, and the places where they differ are exactly the places that don't show up until production.

The vocabulary map, and where it lies

ASIO	Tokio	Where the analogy breaks
io_context	Runtime	one io_context with N run() threads ≈ multi-thread runtime; one io_context per thread ≈ N current_thread runtimes (see below)
post(ctx, f) / co_spawn(..., detached)	tokio::spawn	a spawned Tokio task is detached by default too, but you get a JoinHandle and can abort() it
strand	a task	a strand serialises handlers that share state; a Tokio task owns its state, so the strand usually disappears
completion handler	.await	handlers are eager and always run (even on cancel); futures are lazy and may never run to completion
error::operation_aborted	Drop	there is no error delivered on cancellation; the future is dropped, and cleanup is whatever Drop does
steady_timer	tokio::time::sleep / interval	cancelling a timer = dropping the future
shared_from_this()	Arc<..> or task ownership	each one is a design decision, not a translation

Strands become tasks

The canonical ASIO session, which I've seen in every codebase with only the variable names changed:

class Session : public std::enable_shared_from_this<Session> {
  tcp::socket sock_;
  asio::strand<asio::any_io_executor> strand_;
  std::array<char, 4096> buf_;
public:
  void start() { do_read(); }
  void do_read() {
    auto self = shared_from_this();   // keep *this alive until the handler runs
    sock_.async_read_some(asio::buffer(buf_),
      asio::bind_executor(strand_, [this, self](std::error_code ec, size_t n) {
        if (ec) return;               // includes operation_aborted on shutdown
        handle(buf_.data(), n);
        do_read();
      }));
  }
};

Three things are doing work here: the strand (so handle never runs concurrently with itself), the self capture (so buf_ outlives the read), and the ec check (so cancellation is observed). The Tokio version:

async fn session(mut sock: TcpStream, shutdown: CancellationToken) {
    let mut buf = [0u8; 4096];
    loop {
        tokio::select! {
            _ = shutdown.cancelled() => break,
            res = sock.read(&mut buf) => match res {
                Ok(0) | Err(_) => break,
                Ok(n) => handle(&buf[..n]),
            },
        }
    }
    // `sock` and `buf` drop here. No self-capture, no strand, no operation_aborted branch.
}

The strand disappeared because the task is the strand: one task, one socket, sequential by construction. The self capture disappeared because sock.read(&mut buf) borrows buf for exactly the duration of the operation, and the compiler checks it. The ASIO rule "the buffer must outlive the async operation" was enforced by nothing in C++ and by the signature in Rust.

The decision you do have to make: what about state that several sessions share? In ASIO that was "a strand around the shared object". In Tokio you have three choices, and I'd pick them in this order: (1) give the state to one task and talk to it over an mpsc channel (the actor pattern — this is the honest translation of a strand), (2) Arc<Mutex<T>> with short critical sections and no .await inside, (3) tokio::sync::Mutex only if you must hold the lock across an .await, which you should be suspicious of.

C++20 coroutines look closest and are the most deceptive

asio::awaitable code reads almost like Rust:

asio::awaitable<void> session(tcp::socket sock) {
  std::array<char, 4096> buf;
  for (;;) {
    auto n = co_await sock.async_read_some(asio::buffer(buf), asio::use_awaitable);
    handle(buf.data(), n);
  }
}

The differences that matter when porting:

• Eager vs lazy. async_read_some starts the read when called; the co_await just waits. In Rust nothing happens until .await. If you port a pattern that starts several operations and then awaits them in turn, you'll need join!/JoinSet to get the concurrency back, or you'll silently serialise them.
• Heap vs state machine. A C++ coroutine frame is heap-allocated by default (the compiler may elide it, but rarely across a translation unit boundary). A Rust async fn is a state machine with a size known at compile time, stored wherever you put it, which is why recursion needs Box::pin and why a deeply nested async call tree can make a 40 KB future. Neither is better; they just fail differently.
• Cancellation — the next section, because it deserves one.

Cancellation is the thing that will bite you

In ASIO, cancelling an operation delivers operation_aborted to its handler, and the handler runs. Your cleanup is in the handler. In Tokio, cancelling a future means dropping it, and nothing in its body runs again. Your cleanup is in Drop, or it doesn't happen.

This sounds like a bookkeeping difference. It is the single largest source of bugs I've introduced while porting ASIO code. The failure mode is cancellation safety: a future dropped between two .awaits may have done half its work.

// NOT cancellation-safe. If `shutdown` wins the select after 3 of 8 bytes
// arrived, those 3 bytes are gone and the stream is desynchronised.
tokio::select! {
    _ = shutdown.cancelled() => {}
    r = sock.read_exact(&mut header) => { ... }
}

Tokio's docs mark each method as cancellation-safe or not; read them for every method you use inside a select!. The patterns that kept me out of trouble:

1. Cancel at task granularity, not operation granularity. Instead of select! around individual I/O calls, let the task run and abort() the whole thing (or signal it between messages) when you need to stop. One place to reason about, not fifty.
2. Make state transitions atomic with respect to .await. Do all your bookkeeping, then await, or await into a local and then commit. Never mutate shared state, await, and then mutate again as if nothing could have happened in between.
3. Use Drop as the cleanup hook, and keep it cheap. The ASIO operation_aborted branch that returned a buffer to a pool becomes an impl Drop on a guard type. If the cleanup needs to be async, you need a different design (a shutdown message to the owning task), because Drop can't await.
4. CancellationToken from tokio-util for structured shutdown, and JoinSet so the parent actually knows when its children are gone. Detached tasks are the Rust equivalent of detached threads.

Blocking the executor

ASIO is tolerant of a handler that blocks for five milliseconds: that one thread is busy, the others keep going. Tokio is less tolerant, and the trap when porting is more specific:

// Fine in ASIO: the handler runs later, after the lock_guard is gone.
std::lock_guard lk(mu_);
asio::async_write(sock_, asio::buffer(outbox_.front()), handler);

// Not fine: the guard lives across the await.
let g = self.outbox.lock().unwrap();
self.sock.write_all(&g.front().unwrap()).await; // compile error on a multi-thread runtime, deadlock on current_thread

A std::sync::MutexGuard is !Send, so this is a compile error if the future needs to be Send (i.e. if it's tokio::spawned on the multi-thread runtime). On a current_thread runtime, or inside block_on, it compiles — and then deadlocks the moment two tasks contend. The fix is to copy the data out, drop the guard, then await. Do that in every port, unconditionally. For real blocking work (synchronous file I/O, a blocking client library), spawn_blocking.

Runtime topology is a decision, not a default

If the C++ used one io_context with N threads and strands, the multi-thread Tokio runtime is the natural mapping and work-stealing will probably improve tail latency. If the C++ used one io_context per core with SO_REUSEPORT and no cross-thread posting — common in anything that was performance-tuned — then porting to a single work-stealing runtime can be a regression, because you've just introduced cross-core traffic the original avoided. The faithful port is N current_thread runtimes, one per core, with sharded state. (And if the C++ was on io_uring, that's the moment to look at the thread-per-core runtimes built around it, rather than forcing the model onto Tokio.)

For what it's worth, I got this wrong once in each direction.

Porting lock-free atomics and hand-rolled queues

The good news first: the memory orderings map one to one, and the happens-before reasoning you already did is still valid. The bad news is that Rust's unsafe has stricter rules than C++ in one specific place, and lock-free C++ code tends to live exactly there.

What maps directly

C++	Rust
std::atomic<int>, <bool>, <T*>	AtomicI32, AtomicBool, AtomicPtr<T>
memory_order_relaxed / acquire / release / acq_rel / seq_cst	Ordering::Relaxed / Acquire / Release / AcqRel / SeqCst
memory_order_consume	nothing — use Acquire (every C++ compiler already does)
std::atomic_thread_fence(order)	std::sync::atomic::fence(order)
std::atomic_signal_fence	compiler_fence
compare_exchange_weak / _strong	compare_exchange_weak / compare_exchange
alignas(64) to avoid false sharing	#[repr(align(64))] or crossbeam_utils::CachePadded

What doesn't

std::atomic<T> for arbitrary T. Rust only has atomics for integers, bool and pointers. A std::atomic<SomeStruct> in C++ compiled either to a real atomic (if T fits in 16 bytes and the platform has cmpxchg16b) or to a hidden lock. Check is_lock_free() on the C++ side; if it was lock-free and 8 bytes, pack it into an AtomicU64. If it was a hidden lock, it was never lock-free, and Mutex<T> is the honest port.

Aliasing. This is the one that matters. C++ lock-free code freely holds multiple pointers to the same slot and reads through them from different threads, relying on atomics for the synchronisation and on "we know what we're doing" for everything else. Rust has the same atomics but a stronger aliasing model: a &mut T must be the only live reference to that memory, and creating one while another thread holds a &T is undefined behaviour even if no access happens. The port of a ring buffer slot therefore can't be &mut [T; N] behind an UnsafeCell with plain indexing; it has to be raw pointer arithmetic on *mut MaybeUninit<T>, and the references you create must be as short-lived as the accesses they're for. The compact form of an SPSC ring push that survives Miri:

struct Ring<T, const N: usize> {
    buf: Box<[UnsafeCell<MaybeUninit<T>>]>,
    head: CachePadded<AtomicUsize>, // consumer position
    tail: CachePadded<AtomicUsize>, // producer position
}
unsafe impl<T: Send, const N: usize> Sync for Ring<T, N> {}

impl<T, const N: usize> Ring<T, N> {
    /// Caller contract: exactly one producer thread.
    pub fn push(&self, v: T) -> Result<(), T> {
        let tail = self.tail.load(Ordering::Relaxed);           // only we write tail
        let head = self.head.load(Ordering::Acquire);           // pair with consumer's Release
        if tail.wrapping_sub(head) == N { return Err(v); }
        // SAFETY: slot is unobserved by the consumer until the Release below.
        unsafe { (*self.buf[tail % N].get()).write(v); }
        self.tail.store(tail.wrapping_add(1), Ordering::Release); // publish
        Ok(())
    }
}

The orderings are exactly what the C++ had. What's new is the SAFETY: comment (write it, every time, it's the only documentation of the invariant that the type system can't express) and the unsafe impl Sync, which is you signing the contract that the C++ never had to sign.

Seqlocks. A seqlock reader reads the protected data while a writer may be writing it, and retries if the sequence number changed. In C++ that read is technically a data race and UB, and everyone does it anyway. In Rust it's just as much UB, and Miri will tell you so. The sound version reads the payload through atomics word by word (or uses the seqlock crate, which does that for you). It's slightly slower and it's correct.

Memory reclamation

The hardest part of the C++ lock-free code was never the queue, it was "when is it safe to free this node". The codebases I inherited had, respectively: hazard pointers, an epoch scheme, and "we never free, the process restarts nightly". The Rust ecosystem has these as crates — crossbeam-epoch for epoch-based reclamation, haphazard for hazard pointers, arc-swap for the very common "readers see a consistent snapshot, writers replace it wholesale" case — and I'd port to them rather than port the C++ scheme. Reclamation bugs are the ones you find at 3 a.m. two months after launch; use the implementation other people have already found them in.

Most of it becomes a channel

I want to repeat this because it saved the most time: of the lock-free structures in the audit, roughly half were MPSC or MPMC queues whose only job was moving work between threads. Every one of those became crossbeam-channel or tokio::sync::mpsc, and in no case could we measure the difference under production load. The SPSC rings on the real hot paths (capture → encode, NIC → parser) stayed hand-written, and that's where the loom and Miri time went. Spend your unsafe budget on the two structures that need it, not the nine that don't.

Tools you didn't have in C++

• loom replaces "we ran it for a week and it didn't crash". It exhaustively explores interleavings of a small model under the C11 memory model. It found a Relaxed load in one of my ports that was only correct in C++ because a mutex elsewhere in the call path — a mutex that no longer existed in Rust — happened to provide the ordering. Write the loom test before the port, against the C++ structure's semantics, so it's a spec and not a regression test.
• Miri runs your tests on an interpreter that checks for UB, including the aliasing violations above and data races in unsafe code. Slow, worth it, run it on the lock-free crate in CI.
• ThreadSanitizer works on Rust too (nightly, -Zsanitizer=thread, rebuild std with -Zbuild-std). Run it on both the old C++ and the new Rust against the same integration tests; it's one of the few tools that gives you a like-for-like comparison.

Strategy A: incremental migration over an FFI boundary

The pitch is obvious: keep shipping, move one module at a time, never have a month where the Rust version "isn't ready yet". I've done this twice. It works, and it has a cost that nobody prices in at the start: the FFI boundary is where every concurrency guarantee from both sides goes to die. Send and Sync stop at extern "C". Strands stop at extern "C". You are back to comments.

Picking the bridge

• bindgen turns C headers into Rust declarations. It handles C++ poorly (no templates, no exceptions, name mangling by hand) and is the right tool only if the C++ already exposes a C API or you're willing to write one.
• cxx is what I'd reach for by default. You declare the shared interface once in a #[cxx::bridge] module; it generates both sides and type-checks them against each other at build time. It knows about std::unique_ptr, std::shared_ptr, std::string, std::vector, and opaque types in both directions, and it converts C++ exceptions into Result for functions declared to return one. What it doesn't do is templates beyond the built-ins or arbitrary existing headers.
• autocxx is bindgen plus cxx, pointed at headers you already have. When it works it's magic. When it hits a template it doesn't understand, you're debugging generated code. Worth a day's spike; don't bet the plan on it.
• cbindgen for the other direction, when Rust is the library and C++ is calling in, which is almost always the first phase.

The shape of a boundary that doesn't hurt

Two rules I'd carve above the door: narrow, and the side that allocates frees. A cxx bridge for a decoder module ported early in one migration looked roughly like this:

#[cxx::bridge(namespace = "legacy")]
mod ffi {
    extern "Rust" {
        type Decoder;
        fn new_decoder() -> Box<Decoder>;
        // C++ exception on Err. `&mut self` is a *promise the C++ makes*, not a check.
        fn feed(self: &mut Decoder, bytes: &[u8]) -> Result<usize>;
    }
    unsafe extern "C++" {
        include!("legacy/io.h");
        // Post a completion back onto the ASIO strand that owns this session.
        fn post_to_io(session_id: u64, cb: fn(u64));
    }
}

Read the &mut self on feed carefully, because this is the whole problem in one line. On the Rust side it means "exclusive access". On the C++ side it's a Decoder& that any thread can call. The only thing stopping two threads from calling feed concurrently — which would be UB on the Rust side — is that the C++ caller is a strand. That guarantee is now a comment on the bridge. Either you trust it and document it in capital letters, or you wrap the Rust object in a Mutex and pay for a lock the C++ didn't need. I've done both; I now default to the lock in phase one and remove it when the strand is ported, because "we trusted a comment" is a bad incident report.

Threads crossing the boundary

This is the part that's specific to concurrent code and that the FFI tutorials skip.

C++ threads calling into Rust. The C++ thread calling feed() is an ASIO thread; Tokio has never heard of it. Handle::current() panics there. If the Rust side needs to do async work, you keep a tokio::runtime::Handle somewhere global and use handle.spawn(...) (fire and forget, completion posted back via post_to_io) or handle.block_on(...) (which blocks the ASIO thread, so only for things that are fast). Never block_on from a thread that is itself a Tokio worker; it will panic, and the panic will be crossing an FFI boundary, which is its own problem.

Two event loops in one process. During the middle phase of an incremental migration you have an io_context and a Tokio runtime alive at the same time. The pattern that worked: the Tokio runtime lives on its own dedicated threads; work flows C++ → Rust via a mpsc::Sender held behind the bridge; results flow Rust → C++ by posting a closure back onto the io_context through a C++ function the bridge exposes. The two loops never share a thread. The moment you let an ASIO handler run a Tokio future inline, or vice versa, you've lost the ability to reason about either.

Thread-locals. thread_local! on the Rust side is per-OS-thread, and C++ will call you from whichever thread it likes. Anything the Rust side cached per-thread (buffer pools, RNGs) was being seen from different threads than it expected on one of my ports. Don't use thread-locals in code reachable from FFI.

Panics and exceptions. A Rust panic unwinding into C++ is undefined behaviour; since Rust 1.71, unwinding out of an extern "C" function aborts the process instead. A C++ exception unwinding into Rust is UB with no safety net. cxx handles the exception → Result direction for you. For the panic direction, wrap every entry point in std::panic::catch_unwind and turn it into an error code or a Result, or accept that a Rust panic takes the whole process down (which, for a service with a supervisor, is often the right answer — but choose it).

Allocators. By default Rust uses the system allocator, so a free() in C++ on memory from Box::into_raw happens to work. The day you add jemalloc on the Rust side, it stops working. Don't rely on the accident: the side that allocates, frees, through a function the bridge exposes for exactly that.

Build system

corrosion lets CMake drive Cargo, which is what you want when the C++ build is the one your CI already understands. Going the other way (Cargo as the top level, with the cc or cmake crates building the C++) is better once the Rust half is the bigger one. Cross-language LTO with clang is real and gets you back the inlining you lost at the boundary; leave it for after the port is correct.

What to port first

Leaf modules with a narrow interface and a high bug density. In practice that meant: codecs and parsers first (pure functions over bytes, trivially differential-tested against the C++), then one worker type that used the classic thread-pool pattern, then the shared data structures, and the I/O layer last. The I/O layer is the runtime, and porting it is the moment the two-event-loops phase ends — which is a relief, but it's also the biggest single step, and by then you want everything else already in Rust so that it's the only thing changing.

Strategy B: the big-bang rewrite

Everyone has read the essay about why you should never do this. I've done it anyway, once by choice and once because the C++ build had rotted to the point where the incremental path would have meant resurrecting it first. The second time was the better outcome, and I think that's not a coincidence.

When it wins

The incremental path preserves the architecture. That's its whole virtue: you move code, not shape. So the question is whether the shape is worth keeping.

If the audit found that the io_context topology, the lock structure and the ownership graph are basically right — the C++ is slow or unsafe at the leaves but sound in the middle — then FFI and go module by module. If the audit found that the concurrency architecture is the problem (the god-object under one mutex, the thirteen strands that all touch the same map, the three thread pools that exist because nobody could figure out how to share one), then the incremental path makes you carry that architecture across the boundary and then dismantle it in Rust, which is strictly more work than building the right one.

The other condition is interface shape. A rewrite is tractable when the contract is the wire: a network protocol, a file format, a queue schema. Those are testable from outside. A rewrite of something whose contract is "the exact behaviour of 400 internal C++ functions that 30 other services link against" is not a rewrite, it's a port with extra steps.

The wire is the test

The thing that makes a rewrite survivable is treating the old C++ binary as an oracle and never trusting your reading of it.

Differential testing. Same input to both binaries, diff the outputs. For a server that meant a replay harness: capture real traffic (request logs, or pcaps where the protocol was binary), replay it against the C++ and the Rust, and compare responses after canonicalising the nondeterministic fields — timestamps, request IDs, anything derived from thread scheduling. The harness was the second-largest piece of code in the migration after the service itself, and it was worth every hour.

Property-based and fuzz testing with the C++ as oracle. proptest generating inputs, both implementations run, outputs compared. Ironically the easiest way to do this is to link the C++ into the test binary over cxx — the one place FFI shows up in a rewrite. cargo-fuzz on the parser with the C++ parser as the reference finds the input-tolerance differences (below) faster than any human.

Shadow traffic, then canary. Mirror production traffic to the Rust service, discard its responses, log the diffs. Run that for long enough to see the weekly pattern. Then canary with real responses for a small slice, with the diff still logging. The shadow phase is where every one of the following turned up.

Behavioural fossils

Hyrum's law, applied to a decade of C++: every observable behaviour is being depended on by someone, including the ones nobody designed. The shadow diff found, among others:

• a message ordering between two independent streams that clients had come to rely on, which existed only because both streams happened to be on the same strand
• a timeout that was documented as 30 s and was actually 60 s, because of a retry nobody remembered adding
• acceptance of a malformed header that one old client had been sending for six years
• a response field that was always empty but always present, which one consumer's parser required

The concurrency-flavoured fossils are the interesting ones. Ordering between independent connections, exact interleavings, which of two racing requests "wins" — these were never contracts, they were artefacts of thread scheduling, and the Rust service with a different runtime topology will differ. Don't try to match them. Write down the actual contract ("ordering is guaranteed within a stream, not across streams"), get the consumers to agree, and let the diff harness ignore it. Where the fossil is load-bearing, the decision to preserve or fix it has to be explicit and written down; keep a fossils.md and make it part of the review.

What a rewrite costs that the plan won't say

• It takes 1.5× to 2× the estimate. Both times. What blew the estimate was never Rust; it was fossils, and the time to decide what to do about each one.
• There's a long dark period with no shipped value, and management's patience with it is a resource you should budget as carefully as engineer-weeks.
• You need one of the original C++ authors in the loop for the whole duration, and "why does it do this?" is the most valuable question in the project. If nobody who wrote it is still around, add 30 %.
• Second-system effect is real. The rewrite is the moment everyone wants to fix everything. The discipline that worked: parity first, improvements behind flags, and the flags stay off until the diff harness is quiet.

Choosing between them

The honest answer is that I've picked wrong once and right twice, and the difference was whether I did the audit first. The audit tells you whether the concurrency architecture is an asset or a liability, and that's most of the decision.

Factor	Favours incremental (FFI)	Favours rewrite
Concurrency architecture	Sound; problems are at the leaves	The architecture is the problem
External interface	Wide, internal, many callers	Narrow: a wire protocol, a file format, a queue
C++ build health	Healthy, CI you trust	Rotted; resurrecting it is its own project
Shipping pressure	Must keep releasing throughout	A dark period is survivable
Lock-free / unsafe surface	Large; you want to move it piece by piece with the old code as oracle next door	Small, or already isolated behind a clean interface
Runtime topology	One io_context, N threads, strands → maps cleanly to Tokio	Per-core io_contexts, io_uring, custom scheduling → you'll be redesigning anyway
Original authors	Gone	Available for "why does it do this?"
Team's Rust depth	Mixed; FFI phase is a training ground	Strong; nobody needs the C++ as a crutch

Two hybrids that worked better than either pure form:

Rewrite the core, FFI the leaves. When the runtime and the ownership model are what you're replacing, but there's a battle-tested codec or a numerics kernel that is correct, boring and has no concurrency in it — keep it as C++ behind cxx and port it last, or never. The migration is about the concurrency; a pure function that works is not on the critical path.

Rewrite, with the C++ as test oracle only. Build the Rust service from scratch, but link the old C++ into the test binary over FFI so every parser, every state machine, every scheduler decision can be differentially tested against the original from day one. You get the architectural freedom of a rewrite and most of the safety net of incremental, at the cost of maintaining a build of the C++ you'll throw away at the end.

Whatever you pick, decide it in writing, with the audit attached, before the first line of Rust. Switching strategies halfway is the one outcome I'd rule out: you end up with the FFI tax and the dark period.

Testing the concurrency you just wrote

The compiler took data races off the table. Everything else on the list from the mental-model section — deadlocks, lost wakeups, ordering, cancellation — needs a test, and most of it needs a kind of test the C++ codebase didn't have. This is the tooling I'd set up in the first week, by layer.

Lock-free structures: loom, then Miri. Every hand-written unsafe structure gets a loom model of its core operations — a producer, a consumer, a close — with the model kept small enough to explore exhaustively. Then the same tests under Miri (cargo +nightly miri test) for aliasing and UB. Both in CI, both blocking. These are slow; put them in their own crate so the rest of the test suite doesn't pay for them.

Sync code: deadlock detection and sanitizers. parking_lot with the deadlock_detection feature in debug builds will catch lock-order cycles the first time they fire in a test rather than the first time they fire at 2 a.m. ThreadSanitizer on nightly (RUSTFLAGS="-Zsanitizer=thread" cargo +nightly test -Zbuild-std --target x86_64-unknown-linux-gnu) is worth the setup for the FFI phase specifically, because it's the only tool that sees races that cross the boundary.

Async code: deterministic time and deterministic networks. tokio::time::pause() and advance() make timeout and retry logic testable without sleeping; every timer in the port should have a test that advances the clock and asserts what fired. turmoil simulates a network with controllable partitions and latency, deterministically, on a single thread — this is where I'd test the reconnect, backpressure and shutdown paths, which in the C++ were "we tested it once on staging".

Cancellation tests, specifically. For every task that can be cancelled, a test that cancels it at each .await point and asserts the state afterwards is consistent. The cheap way to write this is a wrapper future that yields Pending a configurable number of times before completing, so you can drop the task after exactly N polls. It's tedious and it has found a bug in every port I've done it on. If a function is used inside a select!, it gets this test; no exceptions.

Differential tests as regression tests. The replay harness from the rewrite section doesn't retire when the rewrite ships. Keep it as the integration suite, with a corpus of recorded traffic that grows every time production finds a diff.

Soak tests, because the leaks are different now. C++ leaked memory and file descriptors. Rust leaks tasks: a tokio::spawn whose future never completes is invisible, costs a few hundred bytes, and accumulates. Run the service under constant-arrival-rate load for hours, not minutes, and watch RSS, FD count and — new — the live task count (tokio-console, or a tokio::runtime::RuntimeMetrics gauge exported to your metrics). A flat line is fine. A sawtooth is fine. A slow monotonic climb is a bug, and the one I had was a task waiting on an mpsc whose sender was held by a struct that lived forever.

And run the old tools on the old code. TSan on the C++ before you start gives you a baseline of what was already broken, which turns "the Rust version behaves differently here" into "the C++ version had a race here" surprisingly often. It's also the single best argument you'll ever have for the migration budget.

Things that burned me

In no particular order, collected across the migrations. Each of these cost at least a day; a couple cost a week.

The Arc<Mutex<Everything>> phase. The first draft of every port had one giant lock, because that's what the C++ had and it was the fastest way to make the borrow checker stop talking. It compiled, it passed the tests, and under load it serialised the whole service onto one core. The real design — owning tasks, channels, a couple of small locks — was the second pass. Budget for the second pass explicitly; the first one is not a milestone.

Half a frame on the wire. A select! with a shutdown branch dropped a write_all future mid-write. The peer got a truncated frame and desynchronised. Two days, because the symptom was on the other end of a TCP connection in a different team's service. The fix was to cancel at task granularity and let the write finish; the lesson was to read the cancellation-safety line of every method I put in a select!.

The guard across the .await that compiled. Ported under a current_thread runtime inside block_on, where futures don't need to be Send. A std::sync::MutexGuard held across an .await compiled fine and deadlocked the moment a second task contended. Switching to the multi-thread runtime for production produced about 200 compile errors in one go, every one of them this bug. Port against the runtime you'll ship on.

tokio::sync::Mutex everywhere. The reflex, after the above, was to switch every lock to the async mutex. It was measurably slower and fixed nothing; almost none of those locks needed to be held across an await. The async mutex is for the two places where you must; std::sync::Mutex with the guard dropped before the await is for everything else.

Shutdown ordering. In C++ the destructor order of members gave you an implicit, deterministic teardown: the flusher stopped before the buffer was freed because of where they were declared. In Rust, drop order is just as deterministic, but spawned tasks aren't members of anything, so dropping the runtime cancelled them in whatever order it liked. The buffer was gone before the flusher's last write. JoinSet so the parent owns its children, an explicit shutdown message, and Runtime::shutdown_timeout so a stuck task can't hold the process open forever.

Rust made the producer faster and exposed the missing backpressure. The C++ had an unbounded queue between the parser and the writer that "never" grew, because the parser was slower than the writer. The Rust parser was zero-copy and about 4× faster. The unbounded mpsc grew to a few gigabytes under a burst before anyone noticed. Bounded channels and a Semaphore, and the parser now stalls when the writer is behind, which is what should have happened all along.

FFI thread identity. C++ called into Rust from an ASIO thread; the Rust side did Handle::current() and panicked, and the panic crossed the FFI boundary and took the process down with a stack trace that pointed nowhere useful. Separately, a thread_local! buffer pool on the Rust side was being touched from whichever ASIO thread happened to call, which was fine right up until it wasn't. No thread-locals reachable from FFI, and a stored Handle for the runtime.

The Relaxed load that was only correct by accident. A flag read with Relaxed in the C++ was ordered, in practice, by a mutex in a caller three frames up. The Rust port replaced that caller with a task and the mutex disappeared, and so did the ordering. loom found it in about a second; nothing else would have for months.

recursive_mutex → self-deadlock on first run. A callback re-entered the object that fired it, and the C++ had hidden that with a recursive mutex. The literal port with std::sync::Mutex deadlocked against itself on the first test. The right fix was to pass the already-locked state into the callback rather than have it re-lock, and that fix was about three times bigger than I'd estimated because the re-entrancy went four levels deep.

It was slower at first. The first Rust build was 20–30 % slower than the C++ on the hot path. Not because of Rust: because the C++ had a pooled, reused buffer per connection and the port allocated a fresh Vec per message. bytes::BytesMut and a small pool closed the gap and then some. Don't assume the port is faster by default; the allocation strategy and the allocator matter at least as much as the language.

The C++ was already broken. TSan on the original, run before the port started, found four real data races in code that had been in production for years. Two of them explained incidents that had been closed as "could not reproduce". The migration got its budget approved on the strength of that report more than on any argument about Rust.

If you only do five things

The long version is above. The short version, in the order I'd do it:

• [ ] Run the audit. Threads, locks, condvars, io_context topology, lock-free structures, blocking calls. Write down the contracts the C++ enforced by convention. Run TSan on the C++ and keep the report.
• [ ] Decide the strategy in writing, with the audit attached. Incremental if the architecture is sound and the interface is wide; rewrite if the architecture is the problem and the contract is the wire. Pick a hybrid deliberately if that's what the audit says. Don't switch halfway.
• [ ] Translate invariants, not code. Every C++ mutex becomes a decision about which data it owns. Every strand becomes a task. Every shared_from_this becomes a question about ownership. Every hand-rolled queue that isn't on a hot path becomes a channel.
• [ ] Treat cancellation as a first-class design concern from day one. Cancel at task granularity, read the cancellation-safety docs for everything inside a select!, put cleanup in Drop, and write the drop-after-N-polls tests.
• [ ] Set up loom, Miri, deterministic time, and a soak test in week one, not after the first incident. The compiler removed one class of bug; the tooling is how you find the rest.

What the migration gets you, when it's done, isn't speed — though you'll probably get some, after the second pass. It's that the contracts from the audit are no longer a document. They're the types. The next engineer who inherits this system won't need to do the audit, because the compiler will do it for them every time they build. That's the thing I'd actually put in the proposal.

If you've been through one of these and your scars are different from mine, I'd like to hear about it in the comments — particularly if you went thread-per-core or io_uring on the Rust side, which is the part I've covered least here and think about most.