The Fast Path Read a Stale Frame Header

The production path can fail while every unit test stays green. I rebuilt a small frame reader that proves it. Would you catch a torn header without a second thread?

The Core Failure

I started from a quiet symptom, not a crash. Frames looked complete, then a length field jumped backward. That is a data race in disguise, not a parser bug.

ASan stayed silent on my first single-thread harness. TSan flagged the overlap on the second run. That gap is the lesson I keep relearning.

The Setup I Used

This walkthrough is a reconstructed teaching case on purpose. It is not a postmortem from a named employer. The types stay small so you can compile them.

I kept one reusable buffer for a cheap fast path. The generated sketch made that same reuse choice. Does that pattern look like your last review?

// Labeled example: reconstructed harness, not production code.
#include <atomic>
#include <cstdint>
#include <cstring>
#include <iostream>
#include <span>
#include <thread>
#include <vector>

struct Slot {
    std::vector<std::uint8_t> bytes;
    std::atomic<bool> ready{false};
};

static void write_frame(Slot& slot, std::uint32_t id, std::uint32_t n) {
    slot.ready.store(false, std::memory_order_release);
    slot.bytes.resize(8 + n);
    std::uint32_t header[2] = {id, n};
    std::memcpy(slot.bytes.data(), header, 8);
    std::memset(slot.bytes.data() + 8, static_cast<int>(id), n);
    slot.ready.store(true, std::memory_order_release);
}

static bool read_fast(Slot& slot, std::uint32_t expect_id) {
    if (!slot.ready.load(std::memory_order_acquire)) {
        return true;  // idle is not a failure in this sketch
    }
    // BUG: span aliases the recycled buffer. No snapshot.
    std::span<const std::uint8_t> view{slot.bytes};
    if (view.size() < 8) {
        return false;
    }
    std::uint32_t id = 0;
    std::uint32_t n = 0;
    std::memcpy(&id, view.data(), 4);
    std::memcpy(&n, view.data() + 4, 4);
    if (id != expect_id) {
        return false;
    }
    if (view.size() != 8 + n) {
        return false;
    }
    for (std::size_t i = 0; i < n; ++i) {
        if (view[8 + i] != static_cast<std::uint8_t>(expect_id)) {
            return false;  // payload from another frame
        }
    }
    return true;
}

Symptom First, Theory Later

I refused to start with a redesign. I wanted the lie on record. What did the reader actually observe?

1. Log the header id and the first payload byte together.
2. Keep the writer on a tight recycle loop.
3. Pin the reader to a std::span, not a copy.
4. Run one thread until you feel falsely safe.

The single-thread run stayed clean for thousands of frames. That is how the false confidence forms. Have you stopped at that green bar before?

# Labeled commands: local check, not a claimed benchmark.
c++ -std=c++20 -O1 -g -fsanitize=address -o frame_asan frame.cpp
./frame_asan

ASan reported nothing useful on that path. The bug is a schedule, not a use-after-free. Address sanitizers do not invent extra threads for you.

Blind Alleys I Walked

I blamed endian conversion first. The dump looked swapped on a bad frame. That was a torn four-byte mix, not a byte-order bug.

I blamed std::vector reallocation next. Resizing in place can still reuse storage. The span outlived the logical frame, not the allocation.

I blamed memory_order_relaxed after that. Stronger flags did not freeze the bytes. The reader still held a view into a mutating slot. Why would an atomic flag protect the payload?

Make the Overlap Inevitable

I added a writer that never rests. I added a reader that never copies. Then the failure stopped being rare.

// Labeled example: adversarial schedule for the teaching harness.
int main() {
    Slot slot;
    std::atomic<bool> stop{false};
    std::atomic<int> bad{0};

    std::thread writer([&] {
        std::uint32_t id = 1;
        while (!stop.load(std::memory_order_relaxed)) {
            write_frame(slot, id, 64);
            id = (id == 1) ? 2 : 1;
        }
    });

    std::thread reader([&] {
        while (!stop.load(std::memory_order_relaxed)) {
            bool ok_one = read_fast(slot, 1);
            bool ok_two = read_fast(slot, 2);
            if (!ok_one && !ok_two) {
                bad.fetch_add(1, std::memory_order_relaxed);
            }
        }
    });

    std::this_thread::sleep_for(std::chrono::milliseconds(200));
    stop.store(true, std::memory_order_relaxed);
    writer.join();
    reader.join();
    std::cout << "mismatches=" << bad.load() << "\n";
    return bad.load() ? 1 : 0;
}

c++ -std=c++20 -O1 -g -fsanitize=thread -o frame_tsan frame.cpp
./frame_tsan

TSan named the overlapping memcpy and the payload scan. That is the root cause with a file and line. Do you still want to argue with the sanitizer trace?

Root Cause in One Sentence

The fast path published a view, not a snapshot. The writer reused the same bytes under that view. Tests never created a second scheduler, so they could not fail.

The header id and length were not one atomic word. A reader could mix id from frame one with n from frame two. Payload bytes could change mid-scan. That is a torn frame, not a flaky parser.

The Fix I Actually Trust

I stopped returning std::span from a recycled slot. I copied the frame into a local vector first. The copy is the snapshot. The later parse only reads that snapshot.

// Labeled example: snapshot before parse.
static bool read_safe(Slot& slot, std::uint32_t expect_id) {
    if (!slot.ready.load(std::memory_order_acquire)) {
        return true;
    }
    std::vector<std::uint8_t> snap = slot.bytes;  // copy, then parse
    if (snap.size() < 8) {
        return false;
    }
    std::uint32_t id = 0;
    std::uint32_t n = 0;
    std::memcpy(&id, snap.data(), 4);
    std::memcpy(&n, snap.data() + 4, 4);
    if (id != expect_id) {
        return false;
    }
    if (snap.size() != 8 + n) {
        return false;
    }
    for (std::size_t i = 0; i < n; ++i) {
        if (snap[8 + i] != static_cast<std::uint8_t>(expect_id)) {
            return false;
        }
    }
    return true;
}

A copy is not free. It is cheaper than a silent mix of two frames. If you need zero-copy, you need generation counters and retired slots. Are you ready to maintain that protocol?

A Reusable Debug Sequence

Use this sequence on the next green, lying test.

1. Write down the impossible observation in one line.
2. Name every object the reader aliases, not owns.
3. Ask which thread may mutate those bytes now.
4. Add a second thread before adding assertions.
5. Build once with ASan and once with TSan.
6. If only TSan complains, stop hunting parser typos.
7. Snapshot first, then parse, then measure cost.

I keep a tiny decision table next to the harness. It stops me from swapping tools at random.

Observation	First tool	Do not start with
Wrong value, no crash	TSan + two threads	Extra log lines only
Crash on free	ASan + shrinking repro	Rewriting the algorithm
Fails at -O2 only	Lifetime of views	Microbenchmarks
Fails once per hour	Forced overlap loop	Longer CI timeouts

Where a Scratch Model Helped

I already knew the symptom. I needed meaner schedules, not prettier code. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode's free model access to list aliasing mistakes I still forget. I used the free server option as a scratch compile box for the harness. I discarded any suggestion that removed the second thread.

Generated tests tried to sleep_for and call that concurrency. Sleep is not a happens-before edge. I threw those tests away. The useful output was a checklist of views, not a patch I could merge.

Limitations

This snapshot fix does not make the slot lock-free. It does not prove wait-free progress. It does not replace a real frame protocol with sequence numbers.

TSan can miss bugs if you block intercepts. ASan can miss bugs if you never free. A green sanitizer run is evidence, not a theorem.

The teaching harness uses a single slot. Production queues add padding, caches, and batching. Those features create new aliases. Copy this pattern, not this file.

I did not measure throughput here on purpose. A number without a machine description becomes folklore. Run your own clock on your own hardware.

Who Should Not Use This Approach

Do not use a span-over-recycle design in safety-critical parsers. Do not ship it because an assistant called it a fast path. Do not treat free model access as a substitute for TSan.

Skip this workflow if you cannot run thread sanitizer. Skip it if your runtime forbids extra copies on the hot path. Skip it if you need a vendor SLA I did not claim.

Skip generated adversarial tests if you will not read them. A test that never starts a second thread is camouflage. It makes the next review harder, not easier.

What I Keep After the Fix

I keep the two-thread harness in the repo. I keep the decision table above the parser. I keep the rule that views do not outlive a recycle.

The interesting AI news this week is not a new slogan. It is how easy green tests become. Engineering still starts when a second thread can lie to you.

If you want a scratch box for this same harness, MonkeyCode's free server option is one place to park the repro and break the test on purpose.