The Problem We Were Actually Solving
The original spec said: Events must be delivered in causal order, but strict global ordering is not required. That sentence was the root infection.
We shipped with a ring buffer per partition and a background reconcile task that ran every 50 ms. It used a B-tree to sort by event time, then emitted in order. The B-tree worked fine for 50 k events per second, but at 250 k the reconcile task would occasionally hold the write lock for 300 ms while it replayed two seconds of buffered events. The clients optimistic concurrency control would time out and retry. Each retry added a duplicate event; the B-tree would see a duplicate timestamp and break ties with the partition key hash, so the second event would appear first in the causal stream. The downstream fraud detection model exploded with false positives because it trusted event order.
What We Tried First (And Why It Failed)
We profiled with perf-rs on the reconcile path. The B-tree insertion was 42 % of CPU, but the lock contention metric from tokio-console showed the spin loop inside the ring buffer lock was 79 % of the wall time. Switching to a lock-free shard per partition dropped latency variance by 22 %, but we still leaked duplicates once every 2 000 000 events.
We tried disabling the reconcile task and shipping events as they arrived. Fraud analysts immediately complained that out-of-order deposits caused overdraft alerts to fire on the wrong account.
The Architecture Decision
At 4:47 a.m. I opened the Rust profiler in perfetto and toggled the timeline. The graph of veltrix::event::order::reconcile was a solid red bar from 3:17 to 3:21. I commented out the entire reconcile module and replaced it with a single atomic counter per partition incremented in sequence.
No B-tree, no background task, no lock.
The change took 18 minutes. We built with cargo rustc --release -C target-cpu=native and pushed to the canary. The p99 latency dropped to 9 ms. The duplicate event rate fell to zero.
The tradeoff: if two events actually arrived out of causal order, the downstream model would see them in arrival order. We documented this as a known limitation and scheduled a data quality run every hour to correct any visible anomalies. Fraud analysts grumbled, but overdraft alerts became deterministic again.
What The Numbers Said After
Latency percentiles after the atomic counter change (15 min rolling window, production, 280 k events/sec):
p50: 6 ms (was 12 ms)
p95: 12 ms (was 980 ms)
p99: 23 ms (was 1.8 s)
The reconcile task CPU usage dropped from 6.2 % to 0.03 %.
Heap allocations measured by dhat-rs fell from 4.7 MB per second to 120 KB per second.
The duplicate event detector, which had been triggering once every two minutes, reported zero incidents over the next 14 days.
What I Would Do Differently
I would reframe the original spec before writing a line of code. If the business really needs causal ordering, it should be enforced by the client or by idempotency keys, not by a background reconcile task that couples latency to write throughput. A sharded atomic counter is beautiful for throughput, but it is also a fire hose of monotonic events; any consumer that needs causality must re-derive it from application-level sequence numbers. We added a per-event metadata field called causal_sequence and made the clients responsible for filling it. The model now sorts on that field only when it matters. The reconcile task still exists, but it runs every hour instead of every 50 ms, and its now a read-only optimization instead of a correctness guard.
The lesson: when the runtime becomes the constraint, stop tuning the algorithm and ask whether the language feature you chose—the background task, the tree, the lock—is actually necessary for correctness. In our case, it wasnt.
Top comments (0)