Why Unbounded WAL Batching Kills Latency in High‑Throughput Services

we built a microservice that ingests ~200k events / sec and persists each to a PostgreSQL write‑ahead log (WAL) via libpq batch mode. to avoid a syscall per row we accumulated records in an unbounded in‑memory buffer and flushed only when the buffer hit 1 GB or the process terminated. under realistic load the 99th‑percentile latency jumped from ~2 ms to >200 ms, and tail latency kept growing as the service ran.

the bottleneck

unbounded buffering breaks three hard invariants:

1. memory pressure – the kernel’s dirty‑page throttling (vm.dirty_ratio) triggers writeback pauses once RAM fills, stalling the flush thread and inflating request latency.
2. fsync latency – a single fsync on a 1 GB buffer must wait for the entire dirty region to be flushed to disk, so any I/O hiccup (e.g. SSD GC) multiplies into a massive pause.
3. recovery cost – on crash the WAL must replay the entire buffered segment, extending downtime from seconds to minutes.

the result is not “just more batching”; it is a cascade of back‑pressure that overwhelms the page cache and the storage controller, turning a throughput optimization into a tail‑latency disaster.

a minimal rust example

use std::sync::mpsc::{self, Sender};
use std::thread;
use std::time::Duration;

const MAX_BATCH: usize = 64 * 1024; // 64 KB, safe for kernel writeback

fn start_writer(tx: Sender<Vec<u8>>) {
    thread::spawn(move || {
        let mut batch = Vec::with_capacity(MAX_BATCH);
        while let Ok(mut record) = tx.recv() {
            batch.append(&mut record);
            if batch.len() >= MAX_BATCH {
                // synchronous write + fsync ensures bounded latency
                let _ = std::fs::File::create("/tmp/wal").and_then(|mut f| f.write_all(&batch));
                let _ = std::process::Command::new("fsync").arg("/tmp/wal").status();
                batch.clear();
            }
        }
    });
}

fn main() {
    let (tx, _rx) = mpsc::channel();
    start_writer(tx.clone());
    // simulate high‑rate producer
    loop {
        tx.send(vec![0u8; 128]).unwrap();
        thread::sleep(Duration::from_nanos(500));
    }
}

the snippet caps each batch at 64 KB, a size that fits comfortably within typical dirty‑page limits and keeps fsync latency sub‑millisecond on modern SSDs.

trade‑offs & edge cases

• throughput vs latency – smaller batches increase syscalls but keep tail latency bounded; the sweet spot depends on CPU cost of encoding vs I/O bandwidth.
• burst handling – a token bucket or leaky‑bucket regulator can spill excess records to a secondary disk‑based queue, preventing OOM.
• durability guarantees – if you can tolerate loss of the last batch, drop the explicit fsync and rely on periodic checkpointing; otherwise every batch must be flushed synchronously.
• multi‑region replication – unbounded buffers amplify replay time on a follower, causing replica lag spikes.

conclusion

unbounded WAL batching trades a single‑syscall win for memory pressure, writeback stalls, and catastrophic recovery times. the practical answer is to enforce a hard batch size, couple it with back‑pressure to the producer, and measure end‑to‑end latency under realistic load.

how do you balance batch size and latency in your write path, and what safeguards have you added to prevent unbounded growth?