DEV Community: Morgan Ma

The Window Treated Wraparound as a Stale Packet

Morgan Ma — Mon, 21 Sep 2026 13:22:32 +0000

A sixteen-bit sequence window can reject good packets. Mine did that after a quiet wrap. The unit tests never crossed 65535, so they stayed green.

I reconstructed the failure in a local harness. I did not invent production metrics. Do you trust a window that never wrapped in CI?

The core miss

Packets arrived in order on a ring counter. The decoder looked healthy in traces. The log still shouted stale_seq and dropped them.

Last accepted sequence sat near 65500. New sequences arrived as 20, 21, 22. Why would twenty look older than sixty-five thousand?

Because I compared them as plain integers. I did not compare them as a ring.

Where the draft came from

I wanted a tiny deduper for binary telemetry frames. I asked a coding model for a first helper. I used MonkeyCode for that draft only. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access and the free server option were enough for a stub. They were not enough for a wrap proof.

Labeled example. This is the broken check I compiled.

// Broken: treats wrap as a jump into the past.
bool is_fresh(uint16_t seq, uint16_t last) {
    return seq > last;
}

Looks honest, right? Tests used seq = 1, 2, 3, 4. Every assert passed. I never asked for a wrap fixture. That miss was mine.

Symptom to evidence

I did not start with protocol theory. I started with printed numbers.

Print seq and last as decimal and hex.
Print int16_t(seq - last) as a signed delta.
Dump the last sixteen accepted values.
Replay the same pair against a wrap fixture.
Fail the build if the ring delta disagrees.

The signed delta told the truth fast. seq - last was 20 minus 65500. That is a large negative. The ring delta is +56. See the gap?

A wrap fixture you can compile

Do not argue with a log line. Make the wrap fail in one file.

#include <cassert>
#include <cstdint>
#include <iostream>

bool is_fresh_broken(uint16_t seq, uint16_t last) {
    return seq > last;
}

// Ring check: forward distance must stay in the lower half.
bool is_fresh_ring(uint16_t seq, uint16_t last) {
    uint16_t delta = static_cast<uint16_t>(seq - last);
    return delta != 0 && delta < 32768u;
}

int main() {
    uint16_t last = 65500;
    uint16_t seq = 20;

    std::cout << "broken " << is_fresh_broken(seq, last) << "\n";
    std::cout << "ring   " << is_fresh_ring(seq, last) << "\n";
    std::cout << "delta  " << static_cast<uint16_t>(seq - last) << "\n";

    assert(!is_fresh_broken(seq, last));
    assert(is_fresh_ring(seq, last));

    last = 20;
    assert(!is_fresh_ring(20, last));  // duplicate
    assert(!is_fresh_ring(10, last));  // behind on the ring
    assert(is_fresh_ring(21, last));   // next value

    std::cout << "wrap fixture ok\n";
}

Keep the build boring. One translation unit. No extra flags.

g++ -std=c++17 -Wall -Wextra -Wconversion -o wrap_seq wrap_seq.cpp
./wrap_seq

Broken returns false. Ring returns true. That is the whole bug.

What the green tests hid

The original tests were not wrong on their inputs. They were incomplete on the type. uint16_t is a ring. A short increasing list is not a ring.

I added three cases after the first fail. Each case is one assert. That is the whole suite I needed.

Monotone increase far from the modulus.
Duplicate of last must drop.
Forward wrap, such as 65500 then 20.
Backward wrap, such as 20 then 10.

Did the model propose case three? Not in my draft. Would I have noticed without printing hex? Probably not.

Decision table for the next review

Use this table before you keep a comparison. Fill it with your real width.

Pair (`last` -> `seq`)	Integer `>`	Ring delta	Keep?
10 -> 11	true	1	yes
10 -> 10	false	0	no
10 -> 9	false	65535	no
65500 -> 20	false	56	yes
20 -> 65500	true	65480	no

The third and fourth rows are the trap. Integer order and ring order disagree there. If your tests never hit those rows, your tests are theatre.

Numbered debug workflow I reuse

This is the reusable part. The product is incidental.

Freeze one failing pair of numbers from logs.
Write those two numbers into a unit test first.
Print unsigned values, hex values, and the modular delta.
Classify the pair with the table above.
Only then change the comparison.
Re-run the wrap fixture under -Wconversion.

Why print hex? Because 65500 does not look like 0xffdc in your head. The hex makes the modulus obvious. Why freeze the pair first? Because a moving capture will gaslight you.

The fix I actually kept

I did not switch to int and pray. I did not widen the field on the wire. I subtracted in the unsigned domain and tested the half range.

bool is_fresh_ring(uint16_t seq, uint16_t last) {
    uint16_t delta = static_cast<uint16_t>(seq - last);
    return delta != 0 && delta < 32768u;
}

Half range is a choice, not a law. It assumes reordering stays below 32768. My stream is in-order telemetry. That assumption holds here. It will not hold for a lossy UDP mesh. Know your reorder bound.

I also stopped treating “model wrote a test” as coverage. A generated test that never wraps is a comment. Compile it. Then mutate the counter.

Command notes for the same harness

Run the fixture under sanitizers once. Do not skip that on a “pure logic” file.

g++ -std=c++17 -fsanitize=address,undefined -o wrap_seq wrap_seq.cpp
./wrap_seq

Then force the wrap in a tiny loop. Watch the first failure index.

uint16_t last = 0;
for (uint32_t i = 1; i < 70000; ++i) {
    uint16_t seq = static_cast<uint16_t>(i);
    if (!is_fresh_ring(seq, last) && seq != last) {
        std::cerr << "fail at i=" << i << "\n";
        break;
    }
    last = seq;
}

If is_fresh_broken sits in that loop, it dies at 65536. That number should have been in the first review. It was not.

Limitations

This write-up is a comparison postmortem. It is not a protocol design guide. Sixteen-bit sequence numbers are a constraint I inherited. They are not a recommendation.

The half-range test drops far-future jumps. That is intended for my in-order path. It is wrong if you must accept large gaps. It is also wrong if duplicates can arrive after a full half window.

I did not benchmark throughput. I did not claim a latency win. I only claimed the broken integer compare fails a wrap fixture. Sanitizers will not catch this by themselves. The values are defined unsigned arithmetic. You still need the fixture.

Free model drafts will happily emit seq > last. They pattern-match textbook loops. They do not owe you modular arithmetic. If you paste the helper and ship, you own the wrap.

Who should not use this approach

Do not copy this check into crypto windows. Do not copy it into authenticated replay defenses. Those need a spec, not a blog delta.

Skip this if your sequence field is already 32 or 64 bits and cannot wrap in the process lifetime. Skip this if a library already implements RFC-style serial number comparison and you can call it. Skip this if you cannot print the failing pair. Guessing without numbers is how I wasted the first hour.

Also skip the “generate tests and relax” loop. If the model never saw 65535, the suite cannot save you. You still write the wrap row.

What I keep from the mess

Short counters lie at the modulus. Green tests can lie with them. Print the modular delta before you rewrite the helper.

I still use a free model for boilerplate. I do not use it as evidence. The evidence is the fixture above. Steal that fixture if you maintain a similar window. The rest is optional.

The Empty Check Passed on a Full Ring

Morgan Ma — Sun, 20 Sep 2026 11:31:33 +0000

The ring reported empty after a full lap. Full and empty shared one modulus check. I lost every payload and still kept green tests.

Why did review miss this quiet bug? The happy path never filled the whole buffer.

The Symptom

I filled a four-slot ring inside a unit test. I pushed four times and never popped once. Then empty() returned true and dropped the batch.

There was no crash and no sanitizer line. CI never failed an assert on this path.

The alias is defined behavior, so UBSan stayed quiet. The only signal was a missing payload later.

Does that failure sound like a data race? It was not a race in this file.

I had one thread and one tight loop. I also had one copied empty predicate.

Build the Smallest Liar

Follow this sequence without skipping a dump. Do not start with a million slot ring.

Build a ring whose capacity is a tiny power of two.
Push exactly cap items with no intervening pops.
Call empty() before you pop a single item.
Watch empty() return true while live bytes remain.

I keep the liar in a file named ring_empty_lie.cpp. The core of the liar sits below this paragraph.

#include <cstddef>
#include <iostream>
#include <vector>

struct Ring {
    std::vector<int> buf;
    std::size_t cap;
    std::size_t r = 0;
    std::size_t w = 0;

    explicit Ring(std::size_t n) : buf(n, 0), cap(n) {}

    // Bug: modulus equality cannot tell full from empty.
    bool empty() const { return (w % cap) == (r % cap); }

    void push(int v) {
        buf[w % cap] = v;
        ++w;
    }

    bool pop(int& out) {
        if (empty()) return false;
        out = buf[r % cap];
        ++r;
        return true;
    }
};

int main() {
    Ring ring(4);
    for (int i = 1; i <= 4; ++i) ring.push(i);
    std::cout << std::boolalpha
              << "empty=" << ring.empty()
              << " w=" << ring.w
              << " r=" << ring.r << "\n";
    int x = 0;
    int n = 0;
    while (ring.pop(x)) ++n;
    std::cout << "popped=" << n << "\n";
    return n == 0 ? 1 : 0;
}

Compile the file with sanitizers and low optimization.

g++ -std=c++17 -O1 -g -Wall -Wextra -fsanitize=address,undefined \
    -o ring_empty_lie ring_empty_lie.cpp
./ring_empty_lie
echo $?

You should see empty=true and popped=0 on stdout. The process exits with status one here. That print shows the entire protocol bug.

Walk the Cursors

Dump cursor state at the failing assert. Do not guess those residues from memory.

Print raw w and r before any modulus.
Print w % cap and r % cap after that.
Print how many pushes you actually issued.
Compare occupied w - r against the cap.

After four pushes, w is four and r is zero. w % 4 is zero and r % 4 is zero.

The predicate says empty with four live slots. Occupied is four, so the check disagrees.

Who holds the truth in this dump? The occupied count holds the real truth. The modulus check is the real liar.

On larger rings the same lie waits. It only needs one exact capacity fill. Most of my tests stopped one below capacity.

Root Cause

I wanted one cheap empty check in the hot path. I copied the textbook head equals tail rule. I then stored unbounded cursors and reduced them with modulus only inside the check.

The head equals tail rule needs a gap. It fails when full wraps to the same residues.

I left no unused gap slot in the ring. I also skipped a separate occupancy count. Full and empty became the same picture.

Was this undefined behavior under the standard? No, it was only a wrong invariant.

Sanitizers do not know your ring protocol. They only know invalid memory use here. This bug never left the ring object.

Wrong Fixes I Almost Shipped

I almost added a boolean is_empty flag. That flag drifts as soon as two writers exist.

I almost used w minus r with 32-bit cursors. That form wraps too, just much later.

I almost kept modulus and reserved one slot. That design works if full is tested.

I still needed a test that fills to the new full. Without that test I would ship the next lie.

Debugging Sequence I Now Reuse

I run these steps on every silent drop. I refuse to start this hunt with threads.

Shrink capacity to four or eight slots.
Drive exact fills for cap-1, cap, and cap+1.
Record cursors on every push and every pop.
Assert w minus r against the visible occupancy.
Add threads only after the single-thread oracle is mean.

Why start with this brutal step order? Tiny rings lap inside a single test.

Huge rings hide the residue collision for days. Threads on a lying empty() only add noise.

Artifact: Oracle Asserts and a Decision Table

These are proposed tests, not a production incident dump.

#include <cassert>
#include <cstddef>
#include <vector>

struct Ring {
    std::vector<int> buf;
    std::size_t cap;
    std::size_t r = 0;
    std::size_t w = 0;

    explicit Ring(std::size_t n) : buf(n, 0), cap(n) {}

    std::size_t occupied() const { return w - r; }
    bool full() const { return occupied() == cap; }
    bool empty() const { return occupied() == 0; }

    bool push(int v) {
        if (full()) return false;
        buf[w % cap] = v;
        ++w;
        return true;
    }

    bool pop(int& out) {
        if (empty()) return false;
        out = buf[r % cap];
        ++r;
        return true;
    }
};

int main() {
    Ring ring(4);
    int x = 0;
    assert(ring.empty());
    assert(ring.push(1) && ring.push(2) && ring.push(3));
    assert(!ring.full());
    assert(ring.push(4));
    assert(ring.full());
    assert(!ring.empty());  // the old check failed here
    assert(ring.occupied() == 4);
    assert(!ring.push(5));  // must reject
    assert(ring.pop(x) && x == 1);
    assert(ring.push(5));
    assert(ring.occupied() == 4);
    return 0;
}

I pin this decision table next to the test.

Observation	Likely cause	Next probe
empty() after exactly cap pushes	residue collision	print w, r, w%cap, r%cap
missing payload, no crash	overwrite or false empty	shrink cap, replay
ASan silent, test green	protocol bug, not UB	add occupancy oracle
fails only on long soak	32-bit cursor wrap	64-bit cursors plus tests
fails only with threads	real race on r/w	TSan after the oracle exists

Keep the table beside the failing binary. It beats rereading the same cursor dump.

Commands I Actually Type

I want a failing binary before any theory. Then I want a log of both cursors.

g++ -std=c++17 -O1 -g -fsanitize=address,undefined -o ring_fix ring_fix.cpp
./ring_fix

g++ -std=c++17 -O1 -g -fsanitize=thread -o ring_tsan ring_fix.cpp
# only after the single-thread oracle is in place

GDB helps when the assert fires too late.

gdb -q ./ring_empty_lie
(gdb) break Ring::empty
(gdb) run
(gdb) print w
(gdb) print r
(gdb) print cap
(gdb) print (w % cap) == (r % cap)

Ask the debugger the same occupancy question. Do not ask your memory for residues.

Where a Scratch Model Fits

I still write the occupancy oracle by hand. I only want extra negative cases after that.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I draft fill-boundary tests with MonkeyCode free model access. I compile throwaway variants on the free server option.

I paste every candidate into the local sanitizer build. The model does not get to decide green.

Ask it for cap-1, cap, and cap+1 cases. Ask it for a wrap after a single pop.

Do not ask it to certify the ring. You own the occupancy assert in review.

Need a spare box for those extra cases? The free server option is one path. Keep the verdict on your own machine.

Limitations

This workflow does not catch a true data race. TSan comes after the sequential oracle exists.

Occupancy w minus r assumes r never outruns w. Thirty-two bit cursors still wrap under long soaks. Use sixty-four bit positions unless lifetime is proven.

Generated tests are only the cases you named. They omit the case you did not describe.

A free model will repeat head equals tail from blogs. That is how I got the first lying check.

The free server is not your release builder. Do not put secrets on a shared scratch box. Do not treat a remote compile as a sanitizer run.

Who Should Not Use This Approach

Skip it if you cannot shrink the ring. Skip it if you ship without a local ASan build.

Skip it if you need a proved wait-free queue. Skip it if packed cursors need a different model.

Do not use generated tests as the only gate. Do not skip the occupancy oracle after a clean model reply.

What I Keep

I keep three fills in every ring test. I keep occupied() next to empty() in the header.

I keep capacity tiny until the protocol is mean. I keep sanitizers on the laptop that signs off.

The modulus check was cheap in the hot path. It was also a liar under an exact fill. Cheap predicates still need an occupancy oracle.

The Fast Path Read a Stale Frame Header

Morgan Ma — Sat, 19 Sep 2026 09:33:58 +0000

The production path can fail while every unit test stays green. I rebuilt a small frame reader that proves it. Would you catch a torn header without a second thread?

The Core Failure

I started from a quiet symptom, not a crash. Frames looked complete, then a length field jumped backward. That is a data race in disguise, not a parser bug.

ASan stayed silent on my first single-thread harness. TSan flagged the overlap on the second run. That gap is the lesson I keep relearning.

The Setup I Used

This walkthrough is a reconstructed teaching case on purpose. It is not a postmortem from a named employer. The types stay small so you can compile them.

I kept one reusable buffer for a cheap fast path. The generated sketch made that same reuse choice. Does that pattern look like your last review?

// Labeled example: reconstructed harness, not production code.
#include <atomic>
#include <cstdint>
#include <cstring>
#include <iostream>
#include <span>
#include <thread>
#include <vector>

struct Slot {
    std::vector<std::uint8_t> bytes;
    std::atomic<bool> ready{false};
};

static void write_frame(Slot& slot, std::uint32_t id, std::uint32_t n) {
    slot.ready.store(false, std::memory_order_release);
    slot.bytes.resize(8 + n);
    std::uint32_t header[2] = {id, n};
    std::memcpy(slot.bytes.data(), header, 8);
    std::memset(slot.bytes.data() + 8, static_cast<int>(id), n);
    slot.ready.store(true, std::memory_order_release);
}

static bool read_fast(Slot& slot, std::uint32_t expect_id) {
    if (!slot.ready.load(std::memory_order_acquire)) {
        return true;  // idle is not a failure in this sketch
    }
    // BUG: span aliases the recycled buffer. No snapshot.
    std::span<const std::uint8_t> view{slot.bytes};
    if (view.size() < 8) {
        return false;
    }
    std::uint32_t id = 0;
    std::uint32_t n = 0;
    std::memcpy(&id, view.data(), 4);
    std::memcpy(&n, view.data() + 4, 4);
    if (id != expect_id) {
        return false;
    }
    if (view.size() != 8 + n) {
        return false;
    }
    for (std::size_t i = 0; i < n; ++i) {
        if (view[8 + i] != static_cast<std::uint8_t>(expect_id)) {
            return false;  // payload from another frame
        }
    }
    return true;
}

Symptom First, Theory Later

I refused to start with a redesign. I wanted the lie on record. What did the reader actually observe?

Log the header id and the first payload byte together.
Keep the writer on a tight recycle loop.
Pin the reader to a std::span, not a copy.
Run one thread until you feel falsely safe.

The single-thread run stayed clean for thousands of frames. That is how the false confidence forms. Have you stopped at that green bar before?

# Labeled commands: local check, not a claimed benchmark.
c++ -std=c++20 -O1 -g -fsanitize=address -o frame_asan frame.cpp
./frame_asan

ASan reported nothing useful on that path. The bug is a schedule, not a use-after-free. Address sanitizers do not invent extra threads for you.

Blind Alleys I Walked

I blamed endian conversion first. The dump looked swapped on a bad frame. That was a torn four-byte mix, not a byte-order bug.

I blamed std::vector reallocation next. Resizing in place can still reuse storage. The span outlived the logical frame, not the allocation.

I blamed memory_order_relaxed after that. Stronger flags did not freeze the bytes. The reader still held a view into a mutating slot. Why would an atomic flag protect the payload?

Make the Overlap Inevitable

I added a writer that never rests. I added a reader that never copies. Then the failure stopped being rare.

// Labeled example: adversarial schedule for the teaching harness.
int main() {
    Slot slot;
    std::atomic<bool> stop{false};
    std::atomic<int> bad{0};

    std::thread writer([&] {
        std::uint32_t id = 1;
        while (!stop.load(std::memory_order_relaxed)) {
            write_frame(slot, id, 64);
            id = (id == 1) ? 2 : 1;
        }
    });

    std::thread reader([&] {
        while (!stop.load(std::memory_order_relaxed)) {
            bool ok_one = read_fast(slot, 1);
            bool ok_two = read_fast(slot, 2);
            if (!ok_one && !ok_two) {
                bad.fetch_add(1, std::memory_order_relaxed);
            }
        }
    });

    std::this_thread::sleep_for(std::chrono::milliseconds(200));
    stop.store(true, std::memory_order_relaxed);
    writer.join();
    reader.join();
    std::cout << "mismatches=" << bad.load() << "\n";
    return bad.load() ? 1 : 0;
}

c++ -std=c++20 -O1 -g -fsanitize=thread -o frame_tsan frame.cpp
./frame_tsan

TSan named the overlapping memcpy and the payload scan. That is the root cause with a file and line. Do you still want to argue with the sanitizer trace?

Root Cause in One Sentence

The fast path published a view, not a snapshot. The writer reused the same bytes under that view. Tests never created a second scheduler, so they could not fail.

The header id and length were not one atomic word. A reader could mix id from frame one with n from frame two. Payload bytes could change mid-scan. That is a torn frame, not a flaky parser.

The Fix I Actually Trust

I stopped returning std::span from a recycled slot. I copied the frame into a local vector first. The copy is the snapshot. The later parse only reads that snapshot.

// Labeled example: snapshot before parse.
static bool read_safe(Slot& slot, std::uint32_t expect_id) {
    if (!slot.ready.load(std::memory_order_acquire)) {
        return true;
    }
    std::vector<std::uint8_t> snap = slot.bytes;  // copy, then parse
    if (snap.size() < 8) {
        return false;
    }
    std::uint32_t id = 0;
    std::uint32_t n = 0;
    std::memcpy(&id, snap.data(), 4);
    std::memcpy(&n, snap.data() + 4, 4);
    if (id != expect_id) {
        return false;
    }
    if (snap.size() != 8 + n) {
        return false;
    }
    for (std::size_t i = 0; i < n; ++i) {
        if (snap[8 + i] != static_cast<std::uint8_t>(expect_id)) {
            return false;
        }
    }
    return true;
}

A copy is not free. It is cheaper than a silent mix of two frames. If you need zero-copy, you need generation counters and retired slots. Are you ready to maintain that protocol?

A Reusable Debug Sequence

Use this sequence on the next green, lying test.

Write down the impossible observation in one line.
Name every object the reader aliases, not owns.
Ask which thread may mutate those bytes now.
Add a second thread before adding assertions.
Build once with ASan and once with TSan.
If only TSan complains, stop hunting parser typos.
Snapshot first, then parse, then measure cost.

I keep a tiny decision table next to the harness. It stops me from swapping tools at random.

Observation	First tool	Do not start with
Wrong value, no crash	TSan + two threads	Extra log lines only
Crash on free	ASan + shrinking repro	Rewriting the algorithm
Fails at -O2 only	Lifetime of views	Microbenchmarks
Fails once per hour	Forced overlap loop	Longer CI timeouts

Where a Scratch Model Helped

I already knew the symptom. I needed meaner schedules, not prettier code. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode's free model access to list aliasing mistakes I still forget. I used the free server option as a scratch compile box for the harness. I discarded any suggestion that removed the second thread.

Generated tests tried to sleep_for and call that concurrency. Sleep is not a happens-before edge. I threw those tests away. The useful output was a checklist of views, not a patch I could merge.

Limitations

This snapshot fix does not make the slot lock-free. It does not prove wait-free progress. It does not replace a real frame protocol with sequence numbers.

TSan can miss bugs if you block intercepts. ASan can miss bugs if you never free. A green sanitizer run is evidence, not a theorem.

The teaching harness uses a single slot. Production queues add padding, caches, and batching. Those features create new aliases. Copy this pattern, not this file.

I did not measure throughput here on purpose. A number without a machine description becomes folklore. Run your own clock on your own hardware.

Who Should Not Use This Approach

Do not use a span-over-recycle design in safety-critical parsers. Do not ship it because an assistant called it a fast path. Do not treat free model access as a substitute for TSan.

Skip this workflow if you cannot run thread sanitizer. Skip it if your runtime forbids extra copies on the hot path. Skip it if you need a vendor SLA I did not claim.

Skip generated adversarial tests if you will not read them. A test that never starts a second thread is camouflage. It makes the next review harder, not easier.

What I Keep After the Fix

I keep the two-thread harness in the repo. I keep the decision table above the parser. I keep the rule that views do not outlive a recycle.

The interesting AI news this week is not a new slogan. It is how easy green tests become. Engineering still starts when a second thread can lie to you.

If you want a scratch box for this same harness, MonkeyCode's free server option is one place to park the repro and break the test on purpose.

The Parse Succeeded on a Partial Token

Morgan Ma — Fri, 18 Sep 2026 07:54:59 +0000

The parser returned success on clearly junk input. The leftover suffix never reached a single test.

I burned a morning on a green suite. Why did every generated fixture stay so quiet?

This walkthrough is reconstructed from a common failure. Treat the function names as teaching examples. The reusable debugging method is the whole point.

The symptom that looked like a win

The config token was the literal 4096k. The job then allocated only 4096 bytes.

The log printed a completely happy parse line. No error branch ran at all.

Was the allocator wrong this time? Was the log lying to me?

Neither guess survived the first hex dump. The parser stopped at the first non-digit. It still returned success to the caller.

What I guessed first

I blamed errno before reading any input bytes. I blamed the C locale right after that.

The on-disk file was plain ASCII throughout. The process locale was already set to C. Those early guesses were pure debugging theater.

I needed the leftover pointer from from_chars. Did std::from_chars actually consume the trailing k?

Why the suite stayed green

The generated tests used only clean decimal tokens. They covered 0, 1, and 4096 only.

They never sent a trailing unit suffix. They also never sent a trailing space.

The tests quietly agreed with the implementation. They did not encode the config spec.

Does this pattern sound familiar to you yet? AI authored both sides of the same mistake.

The loose parse I started from

Label this block as example code. I did not lift it from a private tree.

// EXAMPLE: silent prefix parse
#include <charconv>
#include <cstdint>
#include <string_view>
#include <system_error>

bool parse_bytes_loose(std::string_view text, std::uint64_t& out) {
    if (text.empty()) {
        return false;
    }
    std::uint64_t value = 0;
    const char* first = text.data();
    const char* last = first + text.size();
    auto [ptr, ec] = std::from_chars(first, last, value);
    if (ec != std::errc{}) {
        return false;
    }
    out = value; // BUG: ptr may not equal last
    (void)ptr;
    return true;
}

See the unused ptr sitting there? That unused pointer is the whole bug.

std::from_chars may succeed on a prefix. The standard allows that leftover. Do you check ptr == last in review?

Most generated snippets skip that check. They copy the ec test and stop.

A fixture that hides the failure

// EXAMPLE: tautological tests
#include <cassert>
#include <cstdint>
#include <string_view>

bool parse_bytes_loose(std::string_view, std::uint64_t&);

void test_generated_fixtures() {
    std::uint64_t n = 0;
    assert(parse_bytes_loose("0", n) && n == 0);
    assert(parse_bytes_loose("1", n) && n == 1);
    assert(parse_bytes_loose("4096", n) && n == 4096);
    // Never: "4096k", "4096K", "4096 ", "0x10"
}

Every assert matches the happy decimal path. Production input is not in that list.

Would you ship this helper today? I almost shipped it. The suite was loud and green.

Numbered debugging steps

I used this sequence on the leftover. Steal it and keep it small.

1. Freeze the raw token

Print the exact bytes before any trim. Print the length beside the hex.

Do not trim the token first. Trimming hides the suffix you need.

# EXAMPLE command
python3 -c "print(list(b'4096k')); print(len(b'4096k'))"

You should see 53, 48, 57, 54, 107. The 107 is the letter k.

2. Split success from consumption

ec == std::errc{} is not enough. Ask how far ptr actually moved.

If ptr != last, you parsed a prefix. That prefix result is not success.

3. Add one hostile fixture

Do not add twenty new cases now. Add the token from production first.

4096k is enough for the proof. Then add 4096 and the empty string.

4. Fail the old helper on purpose

Keep the loose parser in the tree. Point the new test at that helper.

Watch it return true for 4096k. Watch out == 4096 in the debugger. That mismatch is your proof, not a theory.

5. Write the strict helper beside it

Do not “improve” the loose path in place. Side-by-side diffs teach the next review.

Then flip the single call site. Then delete the loose path after the flip.

The strict helper

// EXAMPLE: reject leftover characters
#include <charconv>
#include <cstdint>
#include <string_view>
#include <system_error>

struct ByteParse {
    bool ok;
    std::uint64_t value;
    std::size_t consumed;
    char leftover; // '\0' if none
};

ByteParse parse_bytes_strict(std::string_view text) {
    ByteParse r{false, 0, 0, '\0'};
    if (text.empty()) {
        return r;
    }
    const char* first = text.data();
    const char* last = first + text.size();
    auto [ptr, ec] = std::from_chars(first, last, r.value);
    r.consumed = static_cast<std::size_t>(ptr - first);
    r.leftover = (ptr == last) ? '\0' : *ptr;
    if (ec != std::errc{}) {
        return r;
    }
    if (ptr != last) {
        return r; // leftover suffix, still not ok
    }
    if (r.consumed == 0) {
        return r;
    }
    r.ok = true;
    return r;
}

The leftover char is only for logs. Do not guess units inside this helper.

Want k to mean kibibytes later? That is a second parser. Keep this one honest.

A tiny self-check main

// EXAMPLE: compile with
// g++ -std=c++17 -Wall -Wextra -Werror parse_check.cpp -o parse_check
#include <iostream>
#include <string_view>

ByteParse parse_bytes_strict(std::string_view);

int main() {
    const std::string_view cases[] = {
        "0", "4096", "4096k", "4096K", "4096 ",
        "", "k4096", "18446744073709551616", "+4096", "0x10"
    };
    int failures = 0;
    for (auto c : cases) {
        auto r = parse_bytes_strict(c);
        const bool expect_ok = (c == "0" || c == "4096");
        if (r.ok != expect_ok) {
            std::cerr << "mismatch on '" << c << "'\n";
            ++failures;
        }
        std::cout << "in='" << c
                  << "' ok=" << r.ok
                  << " value=" << r.value
                  << " consumed=" << r.consumed
                  << " leftover=";
        if (r.leftover) {
            std::cout << r.leftover;
        } else {
            std::cout << "NUL";
        }
        std::cout << "\n";
    }
    return failures == 0 ? 0 : 1;
}

Run it and read the leftover column. That column is the actual debugger.

+4096 fails on purpose in this harness. Integer from_chars does not treat + as optional padding.

Check your standard library before you assume a sign. Do not trust a comment from a chat log.

Decision table I keep above the function

Input	`ec` looks ok?	`ptr == last`?	Ship it?
`4096`	yes	yes	yes
`4096k`	yes	no	no
`4096`	yes	no	no
empty	no	n/a	no
`k4096`	no	no	no
`0x10`	yes on `0`	no	no
overflow digits	no	maybe	no

If a generated test only covers row one, reject the patch. Ask the patch one blunt question. Which row did you actually run?

Where a free compile loop actually helped

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode's free model access on the helper text. I used the free server option to compile this harness.

The model suggested the loose from_chars call. That suggestion was not a surprise. The server did not magically know the config grammar.

I still had to add 4096k myself. Do not treat a green remote build as verification. Treat it as a faster rebuild loop only.

I will not name models in this article. I will not quote quotas either. Those figures go stale by the next post.

What the generated patch got wrong

It copied a success check from common snippets. It ignored the leftover pointer completely.

It wrote tests from the function shape. It did not write tests from the config grammar.

That is not a model scandal by itself. That is a missing oracle in the repo.

Who owns that oracle on your team? You do. The grammar still lives in your head.

A second trap in the same file

std::from_chars does not skip whitespace. A trailing space fails the strict check.

Good. A leading space also fails here. That failure is also good.

Hex prefixes fail in a sneakier way. 0x10 consumes 0, then leftover is x.

Did your generated tests include 0x10? The example suite above did not at first.

Add it, then watch leftover print x. That single character ends the argument.

Overflow is a different failure

The huge digit string is not a prefix bug. from_chars should report result_out_of_range.

Do not fold overflow into leftover handling. Keep those two failures separate in logs.

I print ok, consumed, and leftover together. I still inspect ec in the debugger.

One flag cannot describe both bugs. Two flags keep the postmortem honest.

Limitations of this method

This harness does not parse size units. It does not parse decimal bytes.

It does not handle thousand separators. It does not handle locale numerals.

from_chars is base ten in this helper. Binary sizes need a later stage.

The self-check compares views to literals. That is fine for fixed cases. It is not a fuzz campaign.

It will miss mixed suffixes like 4096kb. I did not measure wall time here. I did not claim any speedup.

Who should not use this approach

Do not use a remote compile loop for secrets. Config tokens can be secrets too.

Do not paste production files into a hosted model. Redact the token stream first.

Do not use this parser for money fields. Rounding and overflow need a written spec.

Do not skip sanitizers after the parse fix. Prefix parse is not your only bug class.

If you cannot state the grammar today, stop generating tests. Write the grammar down first.

What I keep from the failure

Green tests can encode the original bug. Partial consumes look exactly like success.

ptr == last is the real success check. The leftover char is the breadcrumb you log.

Hostile fixtures beat more happy fixtures. One production token is usually enough.

Would I trust another generated parser tomorrow? Only with that table above the function.

If you already hold a failing token, a free compile server can shorten the rebuilds.

The Key View Survived the Buffer Reset

Morgan Ma — Thu, 17 Sep 2026 05:56:09 +0000

A string_view can outlive every byte it names. That failure compiles, links, and passes shallow tests. This walkthrough is a reconstructed lab case. The map stayed honest. The key was already dead.

Do not store a view into a reusable buffer. Copy the key first, or freeze the buffer. Sanitizers will not always save you here. Why? The storage remains allocated. Only the characters change underneath.

The symptom that looked like a race

The handler returned flags for the previous user. It happened after bursts of short names. Edge logs showed the right user string. The cache still answered for someone else.

Was it a race? That was my first story. ThreadSanitizer printed nothing useful at all. AddressSanitizer printed nothing useful either. Have you trusted a quiet sanitizer too early?

I printed the key pointer beside the key text. The pointer stayed stable across later requests. The text did not stay stable. That is aliasing, not a data race.

False lead one: the hash map

I blamed std::unordered_map for mixing tenants. I swapped in an ordered map. I dumped buckets and load factor. The structure was fine. The lookup key was a ghost of the last write.

False lead two: small-string optimization

Short keys failed more often than long keys. That smelled like small-string optimization. It was a hint, not the root. Short keys reused capacity without moving. The view address never jumped. Long keys sometimes reallocated and hid the bug.

Numbered walkthrough

Follow this sequence on any similar flake.

Capture one failing input pair, then replay it in-process.
Log key .data(), .size(), and a copied character snapshot.
Run under ASan and TSan anyway. Record that they were quiet.
Disable threads. If the flake remains, drop the race theory.
Find every clear, resize, and assign on the scratch buffer.
Insert a poison fill after each reset. Watch the held key change.
Replace the stored view with an owning std::string. Retest the matrix.

Step six is the cheap microscope. You do not need a new vendor tool. You need to mutate the bytes on purpose. Does the stored key change when you poison? Then you never owned it.

Reconstructed repro

The program below is a lab sample. Treat it as a reduced case. It is not a dump of production source.

#include <algorithm>
#include <cassert>
#include <iostream>
#include <string>
#include <string_view>
#include <unordered_map>

class ScratchKeyBuilder {
public:
    std::string_view make_key(std::string const& user, int id) {
        buf_.clear();
        buf_.append(user);
        buf_.push_back(':');
        buf_.append(std::to_string(id));
        return buf_;  // view into member storage
    }

    void recycle() {
        // Next request reuses the same bytes.
        std::fill(buf_.begin(), buf_.end(), '#');
        buf_.clear();
    }

private:
    std::string buf_;
};

int lookup_or_store(std::unordered_map<std::string, int>& cache,
                    std::string_view key,
                    int value) {
    auto it = cache.find(std::string{key});
    if (it != cache.end()) {
        return it->second;
    }
    cache.emplace(std::string{key}, value);
    return value;
}

int main() {
    ScratchKeyBuilder scratch;
    std::unordered_map<std::string, int> cache;

    std::string_view k1 = scratch.make_key("ada", 1);
    int v1 = lookup_or_store(cache, k1, 100);
    scratch.recycle();

    std::string_view k2 = scratch.make_key("bob", 2);
    int v2 = lookup_or_store(cache, k2, 200);

    std::string_view stale = scratch.make_key("ada", 1);
    scratch.recycle();
    std::string ghost{stale};  // may be empty, "###", or worse

    std::cout << "v1=" << v1 << " v2=" << v2
              << " ghost='" << ghost << "'\n";
}

Build the lab case with a pedantic sanitizer command.

clang++ -std=c++20 -O0 -g -Wall -Wextra \
  -fsanitize=address,undefined \
  -fno-omit-frame-pointer \
  stale_view.cpp -o stale_view
./stale_view

Keep those flags. They are still required. They will not always fire. That silence is the trap.

Why the sanitizers stayed quiet

AddressSanitizer hunts use-after-free and out-of-bounds access. This buffer was still alive. The std::string object still owned a valid region. Recycle overwrote that region in place. That is not a free.

ThreadSanitizer hunts concurrent conflicting memory accesses. My flake survived a single thread. No race existed in the schedule. The view was simply stale after clear.

Would a fuzzer have caught it on day one? Maybe, if the corpus reused the builder. A one-shot unit test would miss it completely. That is why the buffer must be recycled inside the test. Happy-path lookup is not an ownership proof.

Root cause

The helper returned std::string_view into member storage. A later request called clear and append on that storage. Callers still held the old view. Some callers copied late, after recycle. Those copies contained the next user, or poison, or nothing.

The zero-copy shape came from a generated patch. It compiled without a single new warning. It looked modern and allocation-free. It deleted a copy I could see. It introduced an alias I could not see. Sound familiar this month?

A green compile is not an ownership proof. Lifetime lives in the next call. Lifetime also lives in comments nobody asked the model to write. AI assistance did not remove that check. It made a missing check easier to ship.

The fix

Own the key before the buffer can move again.

std::string make_key(std::string const& user, int id) {
    std::string out;
    out.reserve(user.size() + 16);
    out.append(user);
    out.push_back(':');
    out.append(std::to_string(id));
    return out;
}

If the hot path cannot allocate, freeze the buffer. Do not recycle until every view is gone. A generation counter also works as a tripwire. Stamp the view with a generation. Reject a mismatch on use.

struct GenView {
    std::string_view bytes;
    std::uint64_t generation;
};

bool valid(GenView const& v, std::uint64_t current) {
    return v.generation == current;
}

I prefer the owning string unless a profiler says otherwise. Have you measured that copy on the real path? If not, keep the copy. Fancy views are cheaper only when they stay valid.

Decision table

Use this table before you return a view from a helper.

Handle	Safe to store?	Recycle while live?	Use when
Owning `std::string`	Yes	Yes	Default cache keys
`string_view` into a member buffer	No	No	Never across requests
`string_view` into a frozen arena	Yes	Not until freeze lifts	Documented window only
Interned pointer	Yes if intern outlives cache	Intern must not evict	Extra machinery

If your case is not in the yes column, copy. Do not negotiate with the buffer. The table is the whole API review.

Test plan you can paste

Do not trust a single happy-path test. Run this matrix on every scratch builder.

Two distinct users, same builder instance, sequential requests.
Same user twice, with a recycle between the lookups.
A short key, then a longer key that forces reallocation.
A longer key, then a short key that reuses capacity.
Poison fill after clear, then read any held view.
The same sequence under ASan, TSan, and a non-sanitized -O2 build.

Pass means no cross-user hits. Pass also means poison never appears in stored keys. Fail means you still hold a view across recycle. Put step five in CI. A human will skip it later.

Where free model access actually helps

I asked for extra negative tests after I knew the bug class. I did not ask a model to rewrite the cache first. That order matters. The first patch already looked elegant.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. Those two facts are the only product claims I am using here. I am not attaching model names, quotas, or timings. Those numbers go stale fast. Your own dashboard is the source of record.

Proposed workflow, labeled as a proposal, not a measured run:

Paste the reduced repro and the failing matrix.
Ask for more negative tests, not a new implementation.
Keep only tests that recycle storage or poison bytes.
Compile those tests on the free server with sanitizer flags.
Reject any patch that returns a view into member buffers.

The model is a generator. The sanitizer build is the judge. The recycle test is the oracle. Remove any one of those three and you are guessing again. If you try that matrix on a free server, keep the sanitizer flags.

Limitations

This approach will not catch every lifetime bug. It will not replace owning keys in a public cache API. It will not help if you cannot run a sanitizer build. Quiet ASan is not a proof of ownership. Quiet TSan is not a proof either.

Who should skip this pattern? Anyone shipping a view-based protocol without a freeze window. Anyone who cannot add a recycle test to CI. Anyone hoping a generated zero-copy helper is finished work. Anyone who cannot read the next caller of a returned view.

Do not use a reusable scratch buffer for keys you store. Do not keep string_view in containers that outlive the request. Do not treat a compiled helper as a lifetime review. The compiler will not object. Your next tenant might.

After the fix

I grep for string_view returns from member functions. I require a comment if the view is non-owning. I add one recycle test beside every scratch builder. That is cheap insurance. Another silent cache mix-up is not cheap.

If you run the repro, watch the ghost string first. Then make the key own its bytes. That is the whole lesson, and it still holds when the assistant writes the helper.

The Comparator Ranked a Tie as Less

Morgan Ma — Wed, 16 Sep 2026 04:09:56 +0000

The comparator treated a matching pair as less. Then std::sort walked into undefined behavior. Green tests on three unique keys will not save you.

I rebuilt the failure in a tiny harness. No vendor core dump. No secret production trace. Just a polite predicate that broke the named requirement.

Why did it look polite? It handled equality on purpose. That choice was the bug. Have you shipped that comment too?

The symptom I started from

I sorted a vector of records. Most permutations looked fine. One permutation dropped a name.

Another permutation duplicated a name. The size stayed constant. The multiset did not.

Have you seen that shape? Length matches. Contents lie. That is not a stability complaint. That is contract failure under std::sort.

I printed the vector before the call. I printed it after the call. Same size(). Different bags of values.

What I blamed first

I blamed the swap. I blamed a user-written move. I blamed a dangling string_view.

Wrong. Wrong. And wrong again. The record was a boring aggregate. Two fields. Trivial moves. No views.

So I read the comparator slowly. It even had a comment. "Handle ties by name." Cute. Deadly.

struct Rec {
    int score;
    std::string name;
};

bool by_score_then_name(const Rec& a, const Rec& b) {
    if (a.score != b.score) {
        return a.score > b.score;
    }
    return a.name <= b.name; // looks careful. it is not.
}

See the <=? Equal names return true. std::sort wants a strict weak ordering. comp(x, x) must be false.

Ask it out loud. Does comp(x, x) return false? If not, stop coding. Fix the predicate first.

Shrink the input until the law is visible

Do not debug this on ten thousand production rows. Three records can be enough. Four if you want a real tie.

Label this clearly. The listing below is a reconstructed example. It is not a client trace.

// sort_predicate_check.cpp
#include <algorithm>
#include <cassert>
#include <iostream>
#include <string>
#include <vector>

struct Rec {
    int score;
    std::string name;
};

bool bad_less(const Rec& a, const Rec& b) {
    if (a.score != b.score) return a.score > b.score;
    return a.name <= b.name;
}

bool good_less(const Rec& a, const Rec& b) {
    if (a.score != b.score) return a.score > b.score;
    return a.name < b.name;
}

template <class Cmp>
bool irreflexive(const std::vector<Rec>& v, Cmp cmp) {
    for (const auto& x : v) {
        if (cmp(x, x)) return false;
    }
    return true;
}

template <class Cmp>
bool both_ways_true(const std::vector<Rec>& v, Cmp cmp) {
    for (const auto& a : v) {
        for (const auto& b : v) {
            if (cmp(a, b) && cmp(b, a)) return false;
        }
    }
    return true;
}

int main() {
    std::vector<Rec> v{
        {10, "ada"},
        {10, "ada"},
        {7, "bek"},
        {10, "cy"}
    };

    std::cout << std::boolalpha;
    std::cout << "irreflexive bad: "
              << irreflexive(v, bad_less) << "\n";
    std::cout << "asymmetric bad: "
              << both_ways_true(v, bad_less) << "\n";
    std::cout << "irreflexive good: "
              << irreflexive(v, good_less) << "\n";
    std::cout << "asymmetric good: "
              << both_ways_true(v, good_less) << "\n";
}

Compile it with symbols. Read the booleans. Do not skip that print.

c++ -std=c++17 -O0 -g sort_predicate_check.cpp -o sort_predicate_check
./sort_predicate_check

You want this shape of output. Anything else means the law already failed.

irreflexive bad: false
asymmetric bad: false
irreflexive good: true
asymmetric good: true

The bad predicate fails before std::sort runs. That is the whole trick. Catch the contract. Do not wait for the crash.

Debug steps I now reuse

Freeze one permutation. Dump it as a brace list.
Ask comp(x, x) on every element. Demand false.
Ask both directions on every pair. Ban two trues.
Check transitivity on triples. Yes, it is tedious.
Only then call std::sort under sanitizers.

Step four is the one people skip. Why skip it? Because it is boring. Undefined behavior loves boring predicates.

c++ -std=c++17 -O1 -g -fsanitize=address,undefined \
    sort_predicate_check.cpp -o sort_san
./sort_san

Sanitizers do not always fire here. Treat a clean run as inconclusive. The pair checks are the test. The sanitizer is a spare net.

A debug standard library can help. It is not a portable promise. Keep the property checks anyway.

The patch that made it worse

I asked a coding model for a fix. It kept the tie comment. It wrote a still-illegal predicate.

bool still_bad(const Rec& a, const Rec& b) {
    if (a.score > b.score) return true;
    if (a.score < b.score) return false;
    if (a.name == b.name) return true; // "stable on equals"
    return a.name < b.name;
}

Stable on equals? std::sort is not std::stable_sort. Returning true for equals stays illegal. The draft optimized for a story. Not for [alg.sorting].

That is vibe-shaped C++. It compiles. It sounds kind. It is still undefined behavior. Would you merge it after a unique-key unit test? I would not.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode's free model access and free server option as a draft source. Not as an oracle. I pasted the ordering laws. I did not paste customer rows. Then every candidate hit the same irreflexive check.

Most drafts failed comp(x, x). Good. The harness did the engineering. The model only did the typing. A local compiler still owned the verdict.

The real fix

Use <. Never <=. Never return true for equal keys. Split fields with std::tie when the key grows.

#include <tuple>

bool good_less(const Rec& a, const Rec& b) {
    // descending score, then ascending name
    return std::tie(b.score, a.name) < std::tie(a.score, b.name);
}

Wait. Did I invert score by swapping operands? Yes. Write that once. Test it twice. A table beats a comment.

a	b	`good_less(a,b)`	`good_less(b,a)`	note
`{10, ada}`	`{10, ada}`	false	false	equal, both false
`{10, ada}`	`{10, cy}`	true	false	name order
`{10, ada}`	`{7, bek}`	true	false	higher score first
`{7, bek}`	`{10, cy}`	false	true	lower score later

If any row shows two trues, you failed. If an equal row shows a true, you failed. Print the row. Do not argue with it.

A cheap shuffle net

Four rows are not a proof. Shuffle. Repeat. Keep the asserts.

#include <numeric>
#include <random>

void fuzz_ordering(std::vector<Rec> v, unsigned seed) {
    std::mt19937 rng(seed);
    for (int i = 0; i < 200; ++i) {
        std::shuffle(v.begin(), v.end(), rng);
        assert(irreflexive(v, good_less));
        assert(both_ways_true(v, good_less));
        auto w = v;
        std::sort(w.begin(), w.end(), good_less);
        assert(w.size() == v.size());
        assert(std::is_sorted(w.begin(), w.end(), good_less));
    }
}

Two hundred shuffles. Not a benchmark. A cheap net. Illegal comparators often explode early. A quiet run still needs the pair checks.

Seed the generator. Log the seed. Replay the failing permutation. That replay is the debugging technique. The model output is not.

What I now refuse to merge

A comparator with <=. A comparator that returns true "for stability." A comparator that calls abs on the key. A comparator that reads a field the sort mutates.

Would I trust a green test that sorts three unique scores? No. Unique keys hide the tie bug. Ties are the actual test. Unique keys are theater.

I also refuse silent operator< on types I do not own. Wrapping those types is fine. Pretending they already order is not.

Limitations

This harness does not prove transitivity on infinite domains. Finite samples miss hostile keys. Floating-point keys are another swamp. NaN breaks < even when you were careful.

Debug iterators in libstdc++ or libc++ can catch some of this. They are build flags, not proofs. Do not drop property checks because a debug STL exists.

I did not measure model quality. I did not collect pass rates. One illegal <= is enough to reject a patch. Do not turn this into a leaderboard.

The free remote path is a draft source. It is not a verifier. If the code cannot leave your machine, do not paste it anywhere remote. Keep secrets local. Generate the harness locally.

Who should skip this approach

Skip it if you sort pointers and think addresses are keys. Skip it if your "tie" is a live timestamp. Skip it if you need a total order over NaN.

Do not point a remote free model at customer records. Do not use this loop as cover for missing operator< on foreign types. Do not call a unique-key sort a proof of ordering.

And do not skip step four because the sanitizer was quiet. Quiet is not evidence. Quiet is a missing assert.

The reusable bit

The technique is older than coding models. Write the ordering laws first. Irreflexive. Asymmetry. Transitivity. Then type the predicate. Then sort.

I still write two-field predicates by hand. I let a draft appear when the key has six nested fields. Then I run the table. Then I run the shuffle loop.

Did a draft type std::tie faster than I did? Sometimes. Did it catch <= alone? Not until the harness returned false.

That false is the engineering. Keep the false. Drop the story in the comment.

If you need a second draft of a synthetic comparator, MonkeyCode's free models and free server option are one place to run that loop. Keep the checks. Leave the vibes out of the merge.

One Run in Two Hundred: Debugging a Silent Fixture Fallback

Morgan Ma — Mon, 14 Sep 2026 21:45:30 +0000

One Run in Two Hundred: Debugging a Silent Fixture Fallback

The bug was not a race condition. It was a swallowed parse error.

One test read a fixture while another test rewrote the same temp file. The reader got half a JSON document. Then it fell back to empty defaults and passed.

The symptom: a green suite with a red job

Our ingest test failed about once every two hundred runs. Always on CI, never on my laptop. But why only on CI? More workers, same fixed temp path.

The failure moved around too. Different test names, different assertions. That pattern usually points at shared state or timing, so I looked at threads first. Wrong turn.

The log line that mattered was logged at debug level. Nobody reads debug logs on a green build.

DEBUG ingest.fixtures fixture unreadable, using defaults path=/tmp/fixture-shared.json

That line is not a warning. That line is the bug introducing itself.

Step 1: Make the silent path loud

Before hunting anything, count your silent fallbacks. This one command changed the whole investigation for me.

rg -n "except Exception" --type py src/ scripts/ | wc -l

If a parse failure returns a default, you do not have a bug. You have a guess. Failures must raise.

# before: a bug that hides itself
def load_fixture(path: str) -> dict:
    try:
        with open(path) as fh:
            return json.load(fh)
    except Exception:
        log.debug("fixture unreadable, using defaults")
        return {}

# after: loud, typed, and greppable
class FixtureError(RuntimeError):
    """Raised when a required fixture cannot be read."""

def load_fixture(path: str, *, required: bool = True) -> dict:
    try:
        with open(path) as fh:
            return json.load(fh)
    except (OSError, json.JSONDecodeError) as exc:
        if required:
            raise FixtureError(f"{path}: {exc!r}") from exc
        return {}

The failure rate went from 1 in 200 to 200 in 200. That is progress.

Step 2: Capture the artifact at failure time

Guessing from console text is slow. I wanted the exact bytes that broke it. So the harness keeps one directory per attempt.

# flake_hunt.py
import os, subprocess, sys

def run_once(i: int, outdir: str) -> str:
    env = dict(os.environ, PYTHONHASHSEED=str(i))
    attempt = os.path.join(outdir, f"attempt-{i:04d}")
    os.makedirs(attempt, exist_ok=True)
    log = os.path.join(attempt, "run.log")
    with open(log, "wb") as fh:
        proc = subprocess.run(
            [sys.executable, "-m", "pytest", "tests/test_ingest.py",
             "-x", "-q", f"--seed={i}"],
            stdout=fh, stderr=subprocess.STDOUT, env=env, timeout=120,
        )
    return attempt if proc.returncode != 0 else ""

if __name__ == "__main__":
    outdir = sys.argv[1]
    failures = [d for i in range(int(sys.argv[2])) if (d := run_once(i, outdir))]
    print(f"{len(failures)} failing runs kept under {outdir}")

Then let it grind in the background.

mkdir -p .artifacts/flake
python flake_hunt.py .artifacts/flake 200

Note the --seed flag. If your runner ignores it, read PYTHONHASHSEED inside the test instead. Same idea, fewer knobs.

Step 3: Reduce to the smallest input

Two hundred logs are too many to read. So sort them by size and take the cheap ones first. What is the cheapest artifact to read? The smallest one.

find .artifacts/flake -name run.log -printf "%s %p\n" | sort -n | head -5

The smallest failing log named the file directly: /tmp/fixture-shared.json. Fixed name, every worker, every run. The cause was plain after that.

Worker A opened that path for writing. Worker B opened it for reading. B parsed a partial document, and the silent handler turned that into empty defaults.

# before: one path shared by parallel workers
tmp_path = "/tmp/fixture-shared.json"

# after: one directory per worker, deleted on exit
from tempfile import TemporaryDirectory
with TemporaryDirectory(prefix="fixture-", dir=artifact_dir) as td:
    tmp_path = os.path.join(td, "fixture.json")

Parallel-safe names are not a style preference. They are a correctness requirement.

Step 4: Prove the fix with the same loop

A fix without a reproduction is just a hope. So I reran the identical 200 seeds.

python flake_hunt.py .artifacts/flake-after 200

Zero failing runs. The suite also got about one second slower per run. That is the honest price of isolation, and I paid it gladly.

Step 5: Where a free model tier actually helped

Reading 200 logs by hand is dull work. So I used AI assistance for triage only, never for the fix. I ran the long repetition loop on a free server option and pointed free model access at the failure logs.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The workflow that worked, in order:

Concatenate the five smallest failing logs into one file.
Ask for ranked hypotheses, each with the evidence it needs.
Test the cheapest hypothesis first, by hand.
Throw away any hypothesis you cannot test in five minutes.

The model's first hypothesis was a data race in the worker pool. The logs said no. Its third suggestion, "check for shared temp paths", was correct and specific.

So the free server mattered for one reason. A 200-run loop is a background job, not a laptop job. Token allowances and server availability are operator-supplied claims in this article. Quotas change, so check the project page instead of trusting a number in a blog post. That includes any number I could quote today.

A short decision table

Symptom	Cheapest next action	Stop when
Fails 1 in N runs, varies by test	Count silent `except` handlers	The count is zero
Fails only under parallelism	Log the resolved temp path	Paths differ per worker
Fails after a refactor	Diff the fallback defaults	Defaults are identical
Fails only on CI	Compare seeds and env vars	Seeds are pinned

Limitations and who should skip this

This approach costs wall-clock time and disk space. A 200-run loop is fine for a fast suite and painful for a slow one. Pick your run count from suite duration, not from enthusiasm.

Do not bother if your suite is already deterministic. Do not paste security-sensitive logs into any hosted tool, free or not. And do not ship an AI-suggested patch you cannot explain, because the next failure will be yours alone.

If your suite is flaky and you want a cheap place to run the loop, the free tier is a reasonable starting point. The project is open source, so its limits are readable before you commit.

The lesson is smaller and sharper than "flakiness is hard". A silent fallback converts a loud crash into a quiet lie. Fix the fallback first, and the bug will find you on the first run.

The Signed Length Passed the Maximum Check

Morgan Ma — Sun, 13 Sep 2026 20:20:04 +0000

A signed length is never a safe container size. I relearned that on a twenty line parser. The tests stayed green until one hostile frame arrived.

Did I need a cluster outage to see it? No. I needed four bytes and a printer. This write-up is that lab, step by step.

The symptom

The process died inside a call to std::vector::resize. AddressSanitizer stayed quiet on the happy path. Nothing overran a live buffer in those tests.

The allocator received a gigantic count instead. I printed the size before the crash returned. The number looked like a size_t with every bit set.

Does that sound like a heap leak to you? It was a type conversion in disguise. The copy never ran far enough to matter.

Why the happy tests lied

My fixtures used lengths of 0, 1, 16, and 1024. I also added one oversize value above kMax. Did any fixture set the high bit? Never.

So the branch len > kMax looked complete. It was not a complete range check. Negative lengths fail that upper bound test.

Then the cast into size_t made them huge. The remaining-bytes check used signed addition. That check also let -1 through.

The baseline reader

I kept the reader tiny on purpose. Four header bytes, then a payload. I assumed a little-endian lab host.

This is the broken core. Treat it as a labeled lab, not production code.

// lab_frame.cpp — baseline, intentionally broken
#include <cstdint>
#include <cstring>
#include <iostream>
#include <iterator>
#include <optional>
#include <vector>

constexpr int32_t kMax = 1 << 20;

std::optional<std::vector<uint8_t>> read_frame(
    const uint8_t* data, size_t n) {
  if (n < 4) {
    return std::nullopt;
  }

  int32_t len = 0;
  std::memcpy(&len, data, sizeof(len));

  if (len > kMax) {
    return std::nullopt;
  }

  // Signed add. Negative len can pass.
  if (4 + len > static_cast<int32_t>(n)) {
    return std::nullopt;
  }

  std::vector<uint8_t> payload;
  payload.resize(static_cast<size_t>(len));
  std::memcpy(payload.data(), data + 4, payload.size());
  return payload;
}

int main() {
  std::vector<uint8_t> buf(
      (std::istreambuf_iterator<char>(std::cin)),
      std::istreambuf_iterator<char>());
  auto out = read_frame(buf.data(), buf.size());
  if (!out) {
    std::cerr << "reject\n";
    return 1;
  }
  std::cout << "payload=" << out->size() << "\n";
  return 0;
}

Four header bytes sit in host order. The payload follows immediately. No version field exists in this lab.

Build the lab

Use a sanitizer build. Do not start with -O2.

g++ -std=c++17 -O0 -g -Wall -Wextra \
  -fsanitize=address,undefined \
  lab_frame.cpp -o lab_frame

Feed it a normal frame first. Four bytes of length one, then one byte.

python3 -c "import sys; sys.stdout.buffer.write(bytes([1,0,0,0, 65]))" \
  | ./lab_frame

That path prints payload=1 and I trusted it. The green output hid the missing lower bound. I had not yet sent a hostile header.

Now feed the hostile header. All 0xFF bytes. No payload body follows.

python3 -c "import sys; sys.stdout.buffer.write(bytes([255,255,255,255]))" \
  | ./lab_frame

What happens on your machine after that pipe? I got std::bad_alloc or a sanitizer abort. The happy path never produced that failure.

What I blamed first

I walked through the wrong rooms first. That wasted time is the actual lesson. Tools failed after the predicate failed.

I blamed endianness on a little-endian host. The fixture already matched the lab host layout. A swap bug would still show small positive lengths.
I blamed memcpy of int32_t as alignment UB. The buffer came from std::vector storage. That alignment worry was mostly noise here.
I blamed AddressSanitizer for missing a wild copy. There was no live copy yet. resize died before memcpy ran.
I blamed the test runner for swallowing aborts. The runner never saw the hostile frame at all. I had not added that row.

Notice the pattern in those four turns? I debugged the tools around the function. I did not debug the boolean predicates inside it.

The frame that told the truth

I printed len in decimal and hex. The hostile header decoded as -1 on this host. 0xFFFFFFFF as two's complement int32_t is -1.

Why did the guard miss that value? -1 > 1048576 is false. An upper bound is not a range.

Then came the second lie in the same function. 4 + len became 3. 3 > 4 is false, so the body check passed. The function then called resize.

static_cast<size_t>(-1) is not -1 in any useful sense. It is a mountain of bytes. The vector asked the allocator for that mountain.

Root cause

Three facts sat in one function. I treated them as one fact. They were not one fact.

The header type was signed.
The upper bound check had no lower bound.
The remaining-length check added signed values.

Any one of those can fail closed in isolation. Together they failed open on 0xFFFFFFFF. Then size_t hid the sign from resize.

This was not a clever exploit in any form. It was a missing row in a table. The compiler did exactly what I typed.

The debugging sequence I now reuse

I keep this order now. The sequence looks boring, and that is the point. Fancy stepping comes last.

Freeze the input bytes. Print them in hex. Do not pretty-print the struct first.
Print the decoded header fields with their real types. Print signed and unsigned views together.
Print each predicate with both operands. Force the types into the log line.
Only then step into resize, memcpy, or a parser loop. The crash site is often downstream.
Add the failing frame to a table. Do not patch by instinct after one abort.

Here is the log I wish I had first. It is not pretty. It is local.

std::cerr << "n=" << n
          << " len=" << len
          << " len_u=" << static_cast<uint32_t>(len)
          << " pred_max=" << (len > kMax)
          << " pred_need=" << (4 + len)
          << " n_i32=" << static_cast<int32_t>(n)
          << "\n";

The log is ugly and that is fine. It is fast enough for a parser lab. Compare len and len_u on the same line.

The bug stops being a story at that point. It becomes two numbers and a boolean. You can keep the debugger closed for a minute.

The matrix that would have caught it

I now run this table before I trust a frame reader. Each row is one process. Each process gets a fresh stdin.

name	header bytes	payload	expect
empty	none	none	reject
truncated header	`01 00`	none	reject
zero	`00 00 00 00`	none	accept 0
one	`01 00 00 00`	`41`	accept 1
max	`00 00 10 00`	`kMax` bytes	accept kMax
max plus	`01 00 10 00`	none	reject
all ff	`FF FF FF FF`	none	reject
high bit	`00 00 00 80`	none	reject
truncated body	`02 00 00 00`	`41`	reject

The last three rows are the whole point. I did not have them, and the first draft did not add them. Positive tutorials rarely emit a high bit.

How do you run a single row then? Same binary. Different stdin. Assert the exit code after the pipe.

python3 -c "import sys; sys.stdout.buffer.write(bytes([0,0,0,128]))" \
  | ./lab_frame ; echo exit:$?

0x80000000 is INT32_MIN on this little-endian layout. That INT32_MIN row is not optional here. Zero, one, and kMax never touch it.

The fix

Reject the sign before any arithmetic. Compare remaining bytes without signed add. Copy only after both checks pass.

std::optional<std::vector<uint8_t>> read_frame(
    const uint8_t* data, size_t n) {
  if (n < 4) {
    return std::nullopt;
  }

  int32_t len = 0;
  std::memcpy(&len, data, sizeof(len));

  if (len < 0 || len > kMax) {
    return std::nullopt;
  }

  const size_t need = static_cast<size_t>(len);
  if (n - 4 < need) {
    return std::nullopt;
  }

  std::vector<uint8_t> payload(need);
  if (need != 0) {
    std::memcpy(payload.data(), data + 4, need);
  }
  return payload;
}

Why n - 4 < need after the n < 4 guard? I already know n >= 4. Subtraction does not wrap on that path. Adding a huge need to 4 still can.

I also skip memcpy when need is zero. That avoids a null pointer discussion on some libraries. Empty payloads are legal in this lab.

Is the header still host-endian after the fix? Yes. A real protocol needs an explicit endian decode later. This lab isolates the signed-size mistake only.

What I asked a model to do

I wanted a second pair of eyes on the first draft. I did not want a story about agents. I wanted predicates I could print.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access and free server option as a scratch box for this lab. The model mirrored my green tests and restated len > kMax. It did not invent len < 0.

That miss is useful as a review lesson. Models copy the distribution you feed them. Hostile rows are rare in parser tutorials, so the predicate stays half-done.

The scratch box compiled what I pasted. It did not invent the table. I still had to type the 0xFF frame by hand.

A second miss I almost shipped

After the sign fix, I still had a sharp edge. int32_t addition of 4 + len is undefined when len is huge and signed. I removed that add instead of narrowing it.

I also almost compared len to n with mixed types. Usual arithmetic conversions would promote a negative len to a huge size_t. Then len > n could look true on poison input.

That rejection is not a specification I can trust. It is an accidental conversion on some paths. Write the checks in one type and cast on purpose.

Log the cast beside the predicate. If you cannot see the type, you cannot see the branch. Mixed comparisons belong in the same bucket as missing lower bounds.

Limitations

This lab assumes a little-endian host and a four-byte header. It does not prove a wire protocol. It does not fuzz a full parser with nested fields.

Sanitizers catch many wild copies. They do not catch every giant allocation. bad_alloc is not a green test, so treat it as a red row.

I did not measure throughput on this reader. I did not claim a quota, a model name, or a hardware size. Those details change, and I will not invent them here.

The matrix is not a fuzzer. A fuzzer would mutate the header and the body. Use one after the table is green, not before the predicates exist.

Who should not use this approach

Skip this pattern if you already parse with a checked slice type. Skip it if your header is unsigned and you still skip a max. An unsigned length still needs a cap.

Do not use a chat window as your only review. Do not deploy a frame reader with only positive fixtures. Do not silence bad_alloc and call that resilience.

If you cannot print the raw header bytes, stop and get a dump first. Hex before structs. Types before copies.

What I keep

Every length prefix is a trust boundary. A maximum check is not a range check. A green test suite can still be a signed-length suite.

I start with hex now. Then types. Then predicates. Then the copy into the vector.

Clone the matrix before you trust the parser. Then add one uglier frame than mine.

The Empty Input Wrapped the Loop Bound

Morgan Ma — Sat, 12 Sep 2026 18:23:16 +0000

The hang was not a deadlock. It was unsigned wrap in a size check.

An empty vector made size() - 1 explode. The loop then treated empty as huge. Have you audited every size() - 1 on a hot path?

I had not. I trusted English that only holds for nonempty ranges.

The symptom

The pairwise merge never returned on one fixture. CPU sat on a single core.

I expected a mutex in gdb. I found a for-loop instead. Why did empty input look busy?

No syscall. No futex. Just i climbing forever.

The reduced repro

I cut the helper out of the larger merge. The snippet compiled clean under -Wall.

This is a lab repro, not a production dump. I kept the types honest.

// lab repro: empty input, unsigned bound
#include <cstddef>
#include <cstdio>
#include <vector>

void pairwise_merge(std::vector<int>& items) {
    for (std::size_t i = 0; i < items.size() - 1; ++i) {
        items[i] += items[i + 1];
    }
}

int main() {
    std::vector<int> empty;
    std::fprintf(stderr, "size=%zu\n", empty.size());
    pairwise_merge(empty);
    std::fprintf(stderr, "returned\n");
}

Looks harmless, right? Two items merge. One item is a no-op.

Empty should also no-op. Empty does not no-op. Empty wraps.

Debugging steps

I treated this like any other hang. I did not start with a model.

I started with a frozen process and a printed bound.

1. Freeze the process, not the theory

Build without optimizations first. You want a readable bt.

g++ -std=c++17 -g -O0 -Wall -Wextra -o merge_hang merge_hang.cpp
./merge_hang

In another terminal, attach and ask where time went.

gdb -p "$(pgrep merge_hang)"
(gdb) bt
(gdb) info locals

bt showed the for-loop. i was already enormous. Still no lock.

2. Print the bound, not the body

I logged items.size() and the compared bound. Empty printed 0.

The bound printed a monster on 64-bit. That is the whole bug.

auto n = items.size();
auto bound = n - 1;
std::fprintf(stderr, "n=%zu bound=%zu\n", n, bound);

n=0. bound=18446744073709551615. That value is SIZE_MAX.

See the problem now? The < compare almost never fails.

3. Write a tripwire, not a timed test

A hang is a bad test oracle. I capped dummy steps instead.

#include <cassert>
#include <cstddef>
#include <vector>

int main() {
    std::vector<int> empty;
    std::size_t steps = 0;
    for (std::size_t i = 0; i < empty.size() - 1; ++i) {
        ++steps;
        if (steps > 8) {
            break;
        }
    }
    assert(steps == 0); // fires: unsigned compare never dies
}

The assert fired after eight dummy steps. Empty was never a no-op.

4. Ask which type did the minus

vector::size() returns size_t. size_t is unsigned.

0u - 1u wraps. The loop condition stays true for a long time.

Would a signed int have survived empty? Yes, until INT_MAX.

Different bomb. Same sloppy bound. Same missing empty case.

Root cause

The author wrote a pairwise loop. The author meant "last index".

That English only holds when size() >= 1. Empty breaks the English.

Humans emit this pattern. Models emit it too. -Wall stayed quiet.

The later items[i + 1] would be the crash. On empty you may never get there.

You just spin. Sanitizers do not always shout at the compare itself.

The fix

Guard the size before you subtract. Do not subtract from zero.

Prefer i + 1 < n over i < n - 1. Empty then does nothing.

void pairwise_merge(std::vector<int>& items) {
    if (items.size() < 2) {
        return;
    }
    for (std::size_t i = 0; i + 1 < items.size(); ++i) {
        items[i] += items[i + 1];
    }
}

Is i + 1 itself a wrap risk? Only if i is already SIZE_MAX.

A vector cannot hold that many elements. The loop condition dies first.

Decision table

I now keep this table next to unsigned loops. Empty is column one.

Check	Empty	One item	Many items	Wrap risk
`i < n - 1`	spin or OOB	no-op	OK	yes on empty
`i + 1 < n`	no-op	no-op	OK	no in practice
`n >= 2` then `i < n - 1`	no-op	no-op	OK	no
signed `int` and `n - 1`	no-op	no-op	OK until huge n	different overflow

I grep for size() - 1 in reviews. I read the empty path first.

rg -n "size\(\)\s*-\s*1" --type cpp
rg -n "\.count\(\)\s*-\s*1" --type cpp

If the hit sits in a loop header, I add three fixtures. Empty. Singleton. Two-plus.

Compile the tripwire under sanitizers

The compare can look defined. The later index may not.

Run AddressSanitizer and UndefinedBehaviorSanitizer on the real helper.

g++ -std=c++17 -O1 -g -fsanitize=address,undefined -o merge_ok merge_ok.cpp
./merge_ok

Empty must return. Singleton must return. Two items must merge once.

If any fixture hangs, the patch is wrong. Do not argue with the tripwire.

A second list of nearby bounds

I wanted sibling greps after the fix landed. I did not want a vibe rewrite.

I used MonkeyCode's free model access and free server option to draft nearby unsigned bounds. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The useful output was not a new helper. It was a list: end - 1, count - 1, len - 1 on size_t.

I kept the list. I threw away the prose. Then I compiled every candidate against the tripwire.

The model does not run your sanitizer. You do. Free access does not change that.

Iterator form when the index is the smell

I now skip unsigned index math when an iterator pair is enough. Empty becomes boring.

#include <iterator>
#include <vector>

void pairwise_merge(std::vector<int>& items) {
    if (items.size() < 2) {
        return;
    }
    for (auto it = items.begin(); std::next(it) != items.end(); ++it) {
        *it += *std::next(it);
    }
}

Longer. Clearer. The empty path does not invent SIZE_MAX.

Would I still use indexes for cache locality notes? Yes, with i + 1 < n.

What this method will not do

Do not use a model as the oracle for a hang. A hang needs a debugger.

A model needs a failing test. Comments do not wrap. Types do.

Skip this workflow if you cannot run ASan. Skip it if the loop is generated and untestable.

Skip it if you already iterate begin to end without a minus. Those loops usually die cleanly on empty.

Unsigned wrap is silent. -Wall stayed quiet on my first build.

So the test must cover empty, singleton, and two-plus. Three cases. Not one happy path.

Did the original loop include those cases? No. That is the whole story.

If you want a second grep list for size() - 1, free model access is enough for a draft. Compile the tripwire yourself.

The Tail Advanced Before the Bytes Did

Morgan Ma — Fri, 11 Sep 2026 15:55:46 +0000

Lock-free C++ can lie with green tests. I watched a ring buffer drop real payloads. The tail index moved before the bytes landed. One producer hid that race from every test. Do you still trust a single TSan run?

This is a lab retrospective, not a war story. I built a small SPSC ring. Then I let a model draft the atomics. The compile was clean. The unit tests were clean. Load was not.

What broke in the lab

The consumer printed garbage after a few million pushes. Sometimes it printed a stale struct. Sometimes it printed a half-updated id. Did the slot get overwritten? Or did the index leak early?

I had no production metrics to wave around. I had a repro and a failing assert. That was enough to start.

The code that looked finished

Here is the reduced producer path. Treat it as the broken lab copy.

// lab_repro: broken SPSC ring, C++20
#include <atomic>
#include <cstdint>
#include <vector>

struct Record {
  std::uint64_t id;
  std::uint64_t payload;
};

class BrokenRing {
 public:
  explicit BrokenRing(std::size_t cap)
      : cap_(cap), buf_(cap) {}

  bool push(Record r) {
    auto t = tail_.load(std::memory_order_relaxed);
    auto h = head_.load(std::memory_order_relaxed);
    if (t - h >= cap_) return false;
    buf_[t % cap_] = r;  // non-atomic payload write
    tail_.store(t + 1, std::memory_order_relaxed);  // leak
    return true;
  }

  bool pop(Record& out) {
    auto h = head_.load(std::memory_order_relaxed);
    auto t = tail_.load(std::memory_order_relaxed);
    if (h == t) return false;
    out = buf_[h % cap_];
    head_.store(h + 1, std::memory_order_relaxed);
    return true;
  }

 private:
  const std::size_t cap_;
  std::vector<Record> buf_;
  std::atomic<std::size_t> head_{0};
  std::atomic<std::size_t> tail_{0};
};

See the relaxed stores on both indexes? That is the whole trap. Why would a model pick relaxed? Because it compiles. Because x86 often forgives it.

Step 1: Freeze the symptom

I stopped changing the algorithm first. I only recorded failures. I logged id on push and id on pop. The sequences diverged under -O2.

Did debug mode still pass? Yes. Release did not. That split already screamed memory ordering. Have you seen that split before?

clang++ -std=c++20 -O0 -g ring_repro.cpp -o ring_o0
clang++ -std=c++20 -O2 -g ring_repro.cpp -o ring_o2
./ring_o0   # often quiet
./ring_o2   # garbage ids after load

Step 2: Shrink the surface

I removed logging from the hot path next. Logging itself can hide a race. I replaced it with a checksum in the record.

struct Record {
  std::uint64_t id;
  std::uint64_t payload;
  std::uint64_t checksum;
};

Record make(std::uint64_t id) {
  Record r{id, id ^ 0x9e3779b97f4a7c15ULL, 0};
  r.checksum = r.id ^ r.payload;
  return r;
}

bool valid(const Record& r) {
  return r.checksum == (r.id ^ r.payload);
}

The consumer then asserted valid(out). The assert died on the optimized binary. Was the slot reused too soon? Or was the write unpublished?

Step 3: Instrument the indexes

I sampled head and tail into a side log. I used a very large ring. Slot reuse became rare. The checksum still failed.

That killed the overwrite theory. The index was visible. The payload was not. Who published the tail too early?

// sampling only, never on the hot success path
if ((seen++ % 100000) == 0) {
  std::fprintf(stderr, "h=%zu t=%zu\n",
               head_.load(std::memory_order_relaxed),
               tail_.load(std::memory_order_relaxed));
}

Step 4: Ask for a patch, then distrust it

I wanted alternate memory-order patches quickly. I did not want another relaxed guess. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access on the free server option to draft candidate diffs. I kept every diff out of the binary until the harness failed or passed on my machine.

The first draft still used relaxed loads on tail in pop. The second draft mixed acq_rel on one index only. Useful? As a checklist. Authoritative? No.

Would you merge a lock-free patch from a chat window? I would not.

Root cause

The producer wrote buf_[slot] as a plain store. Then it published tail with relaxed. A consumer can observe the new tail first. Then it reads a slot that has not landed. On x86 this is rarer. On weaker orders it is not theoretical.

memory_order_relaxed does not create a happens-before edge. The payload store is not ordered against the index store. The tests used one producer and almost no delay. So the race starved itself.

Is TSan a full memory-model oracle? No. ThreadSanitizer catches many races. It does not prove your orders on every CPU. A green TSan run is a filter. It is not a proof.

Memory-order decision table

Use this table before you touch another ring.

Operation	Wrong order I had	Order I required	Why
Producer payload write	plain store, unpublished	happens-before the tail store	Consumer must see bytes
Producer `tail` store	`relaxed`	`release`	Publishes prior writes
Consumer `tail` load	`relaxed`	`acquire`	Pairs with that release
Consumer payload read	plain load after acquire	stays plain	Ordered by the acquire
Consumer `head` store	`relaxed`	`release`	Publishes free slots
Producer `head` load	`relaxed`	`acquire`	Sees freed slots

If any row stays relaxed, stop. Ask why. Then write a test that can fail.

The fix I actually compiled

bool push(Record r) {
  auto t = tail_.load(std::memory_order_relaxed);
  auto h = head_.load(std::memory_order_acquire);
  if (t - h >= cap_) return false;
  buf_[t % cap_] = r;
  tail_.store(t + 1, std::memory_order_release);
  return true;
}

bool pop(Record& out) {
  auto h = head_.load(std::memory_order_relaxed);
  auto t = tail_.load(std::memory_order_acquire);
  if (h == t) return false;
  out = buf_[h % cap_];
  head_.store(h + 1, std::memory_order_release);
  return true;
}

The payload write now happens before a release store of tail. The consumer acquire-loads tail before it reads the slot. That is the pairing. Miss the pair and you are guessing.

This is still SPSC only. Two producers still make it wrong. I did not “fix” that with more relaxed atomics. I refused the MPSC version until the SPSC harness stayed clean.

The harness that would have caught it

Do not call this a benchmark. It is a fail-fast loop.

#include <thread>
#include <cassert>

int main() {
  BrokenRing q(1 << 12);
  constexpr int N = 2'000'000;
  std::thread prod([&] {
    for (int i = 1; i <= N; ++i) {
      Record r = make(static_cast<std::uint64_t>(i));
      while (!q.push(r)) std::this_thread::yield();
    }
  });
  std::thread cons([&] {
    Record out{};
    int seen = 0;
    while (seen < N) {
      if (!q.pop(out)) {
        std::this_thread::yield();
        continue;
      }
      assert(valid(out));
      assert(out.id == static_cast<std::uint64_t>(seen + 1));
      ++seen;
    }
  });
  prod.join();
  cons.join();
}

Build it two ways. Always.

clang++ -std=c++20 -O2 -fsanitize=thread -g harness.cpp -o harness_tsan
clang++ -std=c++20 -O2 -g harness.cpp -o harness_opt
./harness_tsan
./harness_opt

Run the optimized binary on a quiet machine and a busy machine. Yield is not a lock. It only changes timing. If the assert is timing-shaped, you do not have a fix.

What I refuse to claim

I have no model names to recite here. I have no quota numbers that I can defend today. I have no speedup chart. Those would be invented. The only claims I will keep are the ones in the harness.

Free model access can draft a diff. A free server option can host that draft loop. Neither one proves release and acquire pairing. Your CPU does. Your assert does.

Limitations

This pairing does not make the ring MPSC. It does not make it MPMC. % cap_ is also a footgun if cap_ is not stable. vector reallocation would be another bug. I never resize this buffer.

x86 may hide a broken relaxed version for days. ARM may not. TSan may stay quiet if the race is “only” a missing barrier. A model may reintroduce relaxed on the next refactor. Why would it not? Relaxed looks symmetric.

Who should skip this approach

Skip it if you cannot run a stress harness. Skip it if you need MPSC today. Skip it if you will paste atomics from a chat and ship. Skip it if your records are not trivially copyable. Skip it if you need wait-free progress proofs.

Use a mutex queue first if the rate is human. Lock-free is not a personality trait. It is a constraint you can measure.

What I keep doing

I start from the failing assert now. I do not start from a prettier atomic. I treat every relaxed as guilty. I pair release with acquire in a table. I run TSan and an optimized harness. I ask a model for diffs only after the repro is small.

If you want a second draft of a patch, MonkeyCode’s free model access and free server option can host that loop. Keep the harness anyway. The tail can still outrun the bytes.

The Catch Block Never Joined the Worker Thread

Morgan Ma — Wed, 09 Sep 2026 16:11:20 +0000

The abort was not a data race at all. It was std::terminate from a joinable std::thread destructor. I rebuilt the failure as a labeled lab case, not a traffic report.

Happy-path tests joined every worker they started. The catch block never did that join. ThreadSanitizer stayed quiet the whole time. Why would a sanitizer miss a process that simply vanished?

This write-up is a debugging retrospective. The reusable part is the exit-path method. Treat the snippets as a reconstructed harness. I am not inventing production metrics or a customer outage.

The symptom that lied

The binary printed nothing useful on failure. It died with an abort, not a segfault. Several teammates said "race" before anyone opened a stack. Have you seen that reflex on a flaky shutdown test?

I reproduced it with one flag and one throw. Success always joined. Setup failure always aborted. That pattern is a clue. Races rarely line up with one Boolean.

g++ -std=c++17 -g -O0 -pthread joinable_abort.cpp -o joinable_abort
./joinable_abort            # happy path: prints ok
./joinable_abort --fail     # abort: no useful what()

What I captured first

I did not start by rewriting shutdown code. I captured the abort stack under gdb. The frame that mattered was not my lambda. It was std::thread::~thread.

#0  __GI_raise
#1  __GI_abort
#2  std::terminate
#3  std::thread::~thread
#4  run_batch(bool)

Ask one question at that frame. Is this thread still joinable? If yes, the destructor must call std::terminate. That is the contract in the C++ standard. It is not undefined behavior in the race sense. It is defined suicide.

Wrong hypotheses I wrote down

I listed three cheap guesses before the stack. Write them down anyway. Unwritten guesses come back as "fixes."

Data race on the task queue flag.
Exception escaping from the worker itself.
A sanitizer hole on ARM only.

Guess one failed a basic test. ThreadSanitizer reported nothing at -O1. Guess two failed the stack. The worker lambda had a try/catch. Guess three failed the laptop repro. The abort happened on one core. So the stack won. The guesses did not.

Minimal repro

Label this as lab code. It is the shape an agent often emits. Join lives on the success path only. The throw sits after std::thread construction. Destructor then runs during unwind.

// lab-only: joinable_abort.cpp
#include <iostream>
#include <stdexcept>
#include <string_view>
#include <thread>
#include <chrono>

static void worker_body() {
    std::this_thread::sleep_for(std::chrono::milliseconds(20));
}

static int run_batch(bool fail_setup) {
    std::thread worker(worker_body); // joinable from here
    if (fail_setup) {
        throw std::runtime_error("setup failed after spawn");
    }
    worker.join();
    return 0;
}

int main(int argc, char** argv) {
    const bool fail =
        argc > 1 && std::string_view(argv[1]) == "--fail";
    try {
        run_batch(fail);
        std::cout << "ok\n";
        return 0;
    } catch (const std::exception& ex) {
        std::cerr << "caught: " << ex.what() << "\n";
        return 1;
    }
}

Build it. Run both paths. The happy path looks professional. The fail path never prints caught:. Why not? Unwind starts. worker is still joinable. ~thread calls std::terminate. Your catch in main never runs. That is why logs looked empty.

Numbered debug workflow

Use this sequence the next time a process vanishes on errors only.

Confirm the signal. Is it SIGABRT, not SIGSEGV?
Run under gdb with catch throw and bt on abort.
If ~thread is on the stack, stop talking about races.
Print every std::thread local in the dying frame.
Draw every exit path: return, throw, and early co_return if any.
Add a test per exit path before you touch the fix.
Prefer join or std::jthread. Do not detach to silence abort.
Re-run the fail flag. Then re-run ThreadSanitizer for leftover races.

gdb -q ./joinable_abort
(gdb) catch throw
(gdb) run --fail
(gdb) bt
(gdb) info threads

Step five is the one people skip. Agents skip it too. They patch the happy path. They leave the catch block untouched. Your job is the matrix of exits, not a prettier lambda.

Where a free model actually helped

I did not ask a model to "make shutdown thread-safe." That prompt already produced this bug class once. After the stack named ~thread, I used MonkeyCode only as a checklist expander. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used its free model access and free server option to list exit paths I might have missed, then I compiled those paths myself.

The useful output was boring. It listed throw after spawn, return before join, constructor failure of a later object, and a catch that rethrows. I turned each item into a test flag. I discarded any rewrite that called detach(). A model can enumerate branches. It cannot see your destructor. You still own the stack.

The fix that keeps the abort honest

Join on every path, including unwind. The smallest honest fix is RAII. A join guard joins if joinable. It does not detach. Detach turns abort into a use-after-free festival.

struct JoinOnExit {
    std::thread& t;
    ~JoinOnExit() {
        if (t.joinable()) {
            t.join();
        }
    }
};

static int run_batch(bool fail_setup) {
    std::thread worker(worker_body);
    JoinOnExit guard{worker};
    if (fail_setup) {
        throw std::runtime_error("setup failed after spawn");
    }
    return 0; // guard joins here too
}

std::jthread is cleaner on C++20. Its destructor requests stop and joins. That is not magic either. You still must check the stop token inside the worker. Otherwise you join a thread that never looks up. Want a hung shutdown instead of an abort? That is how you get one.

static int run_batch_jthread(bool fail_setup) {
    std::jthread worker([](std::stop_token st) {
        while (!st.stop_requested()) {
            std::this_thread::sleep_for(std::chrono::milliseconds(5));
        }
    });
    if (fail_setup) {
        throw std::runtime_error("setup failed after spawn");
    }
    return 0;
}

Artifact: exit-path decision table

Keep this table next to any thread spawn. Fill the last column from a real run, not from a prompt.

Exit path	Joinable at unwind?	`detach` used?	Observed result
Success, explicit `join`	no	no	prints `ok`
`throw` after spawn	yes	no	`std::terminate`
`return` before `join`	yes	no	`std::terminate`
catch and rethrow, no join	yes	no	`std::terminate`
RAII `JoinOnExit`	no	no	catch in `main` runs
`detach` on error	no	yes	process lives, worker is a ghost
`std::jthread`, worker ignores stop	no	no	shutdown can hang
`std::jthread`, worker checks stop	no	no	fail path is catchable

The ghost row is the trap. Abort is ugly and honest. Detach is polite and false. Which failure would you rather debug at 2 a.m.?

Artifact: test plan you can paste

Do not call this coverage. Call it exit-path proof. Each command must observe a distinct row.

# A. happy path
./joinable_abort

# B. setup failure must be catchable after the fix
./joinable_abort --fail
# expect: caught: setup failed after spawn
# expect: exit code 1, not abort

# C. sanitizer is a second pass, not the first
g++ -std=c++17 -fsanitize=thread -g -O1 -pthread joinable_abort.cpp -o joinable_tsan
./joinable_tsan --fail

# D. prove we did not detach
# grep the diff: detach is a review reject
git grep -n "detach(" -- '*.cpp'

Add one more flag if your real code constructs a second object after the thread. Constructor throws are exit paths. Agents forget those too. Did your last generated "fix" add a logger after spawn? That logger can throw. Then you are back at ~thread.

Limitations

This method does not prove lock-freedom. It does not prove memory orders. It does not replace ThreadSanitizer for real sharing bugs. It only answers one question. Can this thread object die joinable?

Do not use a model as an oracle for the C++ abstract machine. Do not paste secrets into any web session. Do not treat free model access as a substitute for gdb. The free server option is fine for compiling this harness. It is not a formal verifier. I am not claiming quotas, model names, or hardware details here. Those numbers go stale. The destructor rule does not.

Who should not use this approach? Anyone about to detach to green a test. Anyone shipping a lock-free ring buffer from a chat transcript. Anyone who cannot run the fail flag locally. If you cannot observe SIGABRT yourself, you are not debugging. You are decorating.

What I will do next time

I will still let a model list branches after I have a stack. I will not let it own shutdown. I will keep one RAII guard per thread. I will keep the decision table in the review notes. Short rules survive better than slogans.

If you compile this harness and the fail path still aborts, look at ~thread before you look at queues. That single frame saves a week. The free model access and free server option are enough to generate extra exit-path tests once that frame is in hand. Keep the stack first. Keep the prompt second.

The Wait Returned in Zero Milliseconds

Morgan Ma — Tue, 08 Sep 2026 03:36:38 +0000

I lost events because a timeout rounded to zero. duration_cast truncated a leftover under one millisecond. wait_for stopped sleeping and started spinning. The queue looked idle. The core ran hot.

That is the whole failure. The rest is the autopsy. Want the short version first? Never convert a remaining budget with truncating casts.

The symptom I actually saw

Unit tests looked boring and green. A one-hour soak did not. One consumer core stuck at 100 percent. Logs screamed timeout thousands of times per second. Queue depth stayed near one. Packets were fresh. So why spin?

I did not start with chrono lore. I started with a lie I liked. I blamed the producer. Then I blamed notify_one. Both leads were wrong.

False leads I burned first

Was the producer stalled on a disk flush? No. The push log kept moving. Was the mutex convoy hiding a waiter? Unlikely here. Did I miss a wakeup? The traces said the waiter was awake. Too awake.

I needed numbers, not folklore. I needed the wait duration itself. Not the budget I intended.

The helper that looked adult

I reconstructed the consumer as a small program. The production code was larger. The bug fit in one function. Here is the shape I kept.

#include <chrono>
#include <condition_variable>
#include <cstdint>
#include <iostream>
#include <mutex>
#include <queue>
#include <thread>

using namespace std::chrono;

struct SliceQueue {
    std::mutex mu;
    std::condition_variable cv;
    std::queue<int> q;
    bool stop = false;
};

// Truncates toward zero. Sub-millisecond remainders become 0ms.
milliseconds remaining_ms(steady_clock::time_point deadline) {
    auto leftover = deadline - steady_clock::now();
    return duration_cast<milliseconds>(leftover);
}

bool wait_for_item(SliceQueue& s, steady_clock::time_point deadline, int& out) {
    std::unique_lock<std::mutex> lock(s.mu);
    for (;;) {
        if (!s.q.empty()) {
            out = s.q.front();
            s.q.pop();
            return true;
        }
        if (s.stop) {
            return false;
        }
        auto d = remaining_ms(deadline);
        if (d.count() <= 0) {
            return false;  // "timed out"
        }
        s.cv.wait_for(lock, d);
        // loop and re-check the predicate
    }
}

Looks tidy, right? Deadline in. Milliseconds out. Tests used 50ms slices. They never left 800 microseconds on the clock. Production used 200us slices. Different planet.

A repro you can compile

Label this as a reconstructed example. I am not citing a vendor incident. Build it locally. Watch the spin with your own eyes.

int main() {
    SliceQueue s;
    auto deadline = steady_clock::now() + microseconds{800};

    std::thread producer([&] {
        std::this_thread::sleep_for(microseconds{400});
        std::lock_guard<std::mutex> lock(s.mu);
        s.q.push(42);
        s.cv.notify_one();
    });

    int item = -1;
    auto t0 = steady_clock::now();
    bool ok = wait_for_item(s, deadline, item);
    auto spun = steady_clock::now() - t0;

    producer.join();

    std::cout << "ok=" << ok
              << " item=" << item
              << " spun_us="
              << duration_cast<microseconds>(spun).count()
              << "\n";
}

Suggested commands:

g++ -std=c++17 -O2 -Wall -Wextra -o wait_zero wait_zero.cpp
./wait_zero

# then the interesting one
g++ -std=c++17 -O0 -g -fsanitize=thread -o wait_zero_tsan wait_zero.cpp
./wait_zero_tsan

What should you see? ok=0 while the producer still pushed. Or ok=1 after a tight spin. Either way the wait never slept a full slice. The leftover was smaller than one millisecond. The cast ate it.

Debug steps I would repeat

Do not start inside the condition variable. Start at the duration you pass it.

Log leftover in nanoseconds before any cast. Print d.count() after the cast. If those disagree by a full unit, you found the knife.
Pin the clock. Use steady_clock for intervals. Do not mix in system_clock for remaining time. NTP steps will gaslight you.
Print deadline - now on every loop entry. A shrinking leftover that jumps to 0 is the smoking gun.
Sample CPU with top or perf stat. A waiter at 100 percent is not waiting. It is polling.
Compile a soak with -O0 first. Then -O2. Truncation is not an optimizer bug. Still check both.
Add a counter of immediate timeouts. If it explodes, your slice is smaller than your unit.

I asked myself one rude question. Did any test use a budget under 1ms? No. That is why review missed it.

Root cause, without poetry

duration_cast<milliseconds> truncates toward zero. It does not round. It does not ceil. 800us becomes 0ms. 1999us becomes 1ms. Your 200us deadline becomes a no-op wait.

wait_for(lock, 0ms) does not sleep. It is a try-wait. You re-check the queue. You cast again. You still have a leftover under 1ms. You spin until the deadline check returns false. You drop the item path even if a push is in flight.

Is the condition variable broken? No. Your unit is coarser than your budget. The API did what you typed.

A second cut hides nearby. People write if (d.count() <= 0) return false before waiting. That treats "almost now" as a hard timeout. A producer 100us away loses. You wanted a wait until the deadline. You coded a wait in whole milliseconds.

The fix I actually want

Stop converting remaining time into a fatter unit. Wait until the time point. Let the library do the math.

bool wait_for_item_until(SliceQueue& s,
                         steady_clock::time_point deadline,
                         int& out) {
    std::unique_lock<std::mutex> lock(s.mu);
    while (s.q.empty() && !s.stop) {
        if (s.cv.wait_until(lock, deadline) == std::cv_status::timeout) {
            break;
        }
    }
    if (!s.q.empty()) {
        out = s.q.front();
        s.q.pop();
        return true;
    }
    return false;
}

Need a millisecond log line anyway? Ceil, do not truncate. And keep the wait on the time point.

#include <cmath>

milliseconds ceil_ms(nanoseconds ns) {
    if (ns <= nanoseconds{0}) {
        return milliseconds{0};
    }
    auto ms = duration_cast<milliseconds>(ns);
    if (ms < ns) {
        return ms + milliseconds{1};
    }
    return ms;
}

I still prefer wait_until. Logging can ceil. Control flow should not. One more test belongs in the suite.

// Labeled test sketch. Run it under your harness.
void test_submillisecond_deadline_still_wakes() {
    SliceQueue s;
    auto deadline = steady_clock::now() + microseconds{800};
    std::thread p([&] {
        std::this_thread::sleep_for(microseconds{200});
        std::lock_guard<std::mutex> lock(s.mu);
        s.q.push(7);
        s.cv.notify_one();
    });
    int item = -1;
    bool ok = wait_for_item_until(s, deadline, item);
    p.join();
    // ok should be true when the push beats the deadline
}

If that test never existed, your green build was a costume.

Where a model shoved me into the hole

I asked a coding assistant to "normalize the timeout to milliseconds." It suggested duration_cast. It even added a comment about clean logs. It did not ask the slice size. It did not mention truncation. It did not add a sub-millisecond test. Why would it? I never said 200us.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access as a second reader on that helper, and the free server option to compile the same repro files. The model restated the cast. The compiler and the soak told the truth. I kept the human checklist above. The tool did not get a veto.

Ask the model a better question. "What happens when leftover is 800us?" Watch it hesitate. Then write the test yourself.

Limitations, said plainly

wait_until is not a shutdown story by itself. If you must poll a stop flag on a coarse interval, cap the wait. Use min(deadline, now + poll_slice). Still avoid truncating that slice to a bigger unit.

Spurious wakeups still happen. The while predicate stays mandatory. A correct clock does not replace that loop. Sanitizers will not flag truncation. This is value error, not a data race. TSAN can stay quiet while you burn a core.

Do not treat this writeup as a benchmark. I am not publishing rates, hardware, or quotas. Clocks differ across machines. Your 800us case may land differently under load. Measure leftover on the box you ship.

C++20 duration_cast still truncates. std::chrono::ceil exists. Use it for display. Prefer time points for waiting.

Who should not copy this path

Skip this if you do not own the wait loop. Fixing a framework timer from the outside will lie. Skip this if your budgets are already in whole seconds. The bug will not show. Skip this if you cannot compile a repro. Pasting the helper into chat is not proof.

Skip this if you need a hard real-time scheduler. User-space condition_variable is not that. Skip this if you refuse to log durations as integers. Feelings about "about 1ms" hid the zero.

What I keep on a sticky note

Casts drop information. Timeouts live in that dropped part. Tests that only use fat budgets will bless the drop. A waiter at 100 percent is a duration bug until proven otherwise.

Would I still log milliseconds? Yes, for humans. Would I wait in milliseconds? Not when the slice is smaller. I wait until the deadline. I test the awkward leftover. I do not let a tidy helper eat the last 800 microseconds.

If you rebuild the repro, keep the compiler as the referee. A second reader on the free model path is optional. The count() print is not.