48-Hour Field Notes: My Oldest Answer Won the Cache Because I Trusted Arrival Order

I spent last weekend convinced my summarizer was lying to me, and for once the model was innocent.

The worker sent eight requests for the same document, kept whichever response came back last, and wrote it to the cache, because that is what "last write wins" means when you never question what "last" refers to. By Saturday afternoon the cache held a summary of version 2 of a document that was already on version 9, and every downstream reader was quietly reading stale prose. So let me ask the obvious question I failed to ask on Friday: when you say the result arrived last, do you mean the request finished last, or that the work started last?

What I Actually Built, and What It Cost Me

I pointed a small worker at MonkeyCode's free model access, which the operator supplies as an available option, and ran the worker itself on the free server option so I would not have to babysit a laptop all weekend. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I did not benchmark either side, I did not record model-side latency numbers, and everything below is about my own worker's bookkeeping rather than about anyone's infrastructure.

The pipeline was boring on purpose: a queue of document revisions, one call per revision, one cache write per response, and a rewrite rule that said newer wins. I had already learned to hash inputs so I could tell whether a run had actually happened, and I had a replay cassette for repeatability, so I felt reasonably grown-up about the whole thing. Then I opened the cache on Saturday morning and read a summary that mentioned a paragraph I had deleted nine hours earlier.

The Assumption That Broke: Arrival Order Is Not Logical Order

Concurrency does not preserve your send order, and neither does any retry layer you bolt on top of it, which means "last response wins" is a rule about wall-clock arrival and not about freshness. The eighth revision can finish first because it was short, and the second revision can finish last because it hit a slow path and got retried twice, and your cache will faithfully record the older answer. Is that a model problem, a network problem, or a bookkeeping problem? It was the third one, and it was mine.

Here is the shape of the bug, stripped of my domain code:

• Revision 8 finishes at T+2s and writes summary = v8.
• Revision 2, still in flight, finishes at T+9s and writes summary = v2.
• Every reader after T+9s sees the older revision, and nothing errors, logs, or alerts.

Artifact 1: A Ticket Guard Instead of a Timestamp

Wall-clock timestamps did not save me, because two hosts disagreed by enough to invert the comparison, and because I was comparing the wrong moment anyway. What worked was stamping each job with a monotonically increasing ticket taken from the enqueue side, then refusing any commit whose ticket is not newer than the last committed one for that key.

# guard.py
from dataclasses import dataclass

@dataclass(frozen=True)
class Commit:
    ticket: int
    key: str
    value: str

class LastWriterGuard:
    """Rejects stale commits per key. Tickets must come from the enqueue side."""

    def __init__(self) -> None:
        self._seen: dict[str, int] = {}
        self.rejected = 0
        self.accepted = 0

    def commit(self, c: Commit) -> bool:
        previous = self._seen.get(c.key)
        if previous is not None and c.ticket <= previous:
            self.rejected += 1
            return False
        self._seen[c.key] = c.ticket
        self.accepted += 1
        # real code writes to Redis/SQLite/Postgres here
        return True

Fifteen lines, no dependencies, and the counter is the part I actually care about, because a rejection count above zero on a quiet afternoon tells me something is reordering. The same idea survives a restart if you put the comparison in the write itself:

UPDATE summaries
   SET value = :value, ticket = :ticket, updated_at = now()
 WHERE doc_key = :key
   AND ticket < :ticket;
-- zero rows affected means you lost the race on purpose, which is fine

Artifact 2: Reproduce the Reorder Without Touching a Network

The reason I trusted my old code for as long as I did is that the bug is invisible when calls are fast and uniform, so I wrote a test that fabricates the arrival order instead of waiting for it. The delays are synthetic, chosen to put the lowest ticket last, and the point is the assertion rather than the numbers.

# test_reorder.py
import asyncio
from guard import Commit, LastWriterGuard

TICKETS = [1, 2, 3, 4, 5, 6, 7, 8]
# ticket 1 finishes almost instantly, ticket 2 finishes last
DELAYS = [0.05, 0.90, 0.30, 0.20, 0.40, 0.10, 0.60, 0.15]

async def fake_call(ticket: int, delay: float, guard: LastWriterGuard, log: list):
    await asyncio.sleep(delay)  # stand-in for an HTTP round trip
    ok = guard.commit(Commit(ticket, "doc:42", f"v{ticket}"))
    log.append((ticket, ok))

async def run() -> tuple[list, LastWriterGuard]:
    guard, log = LastWriterGuard(), []
    await asyncio.gather(*(fake_call(t, d, guard, log) for t, d in zip(TICKETS, DELAYS)))
    return log, guard

def test_guard_blocks_stale_last_arrival():
    log, guard = asyncio.run(run())
    assert log[-1] == (2, False), "stale revision 2 arrived last and was rejected"
    assert guard.accepted == 4
    assert guard.rejected == 4

python -m pytest -q test_reorder.py
# 1 passed

Delete the guard and the last write becomes v2, which is exactly what my cache contained on Saturday morning. I ran this across a few hundred shuffled delay vectors with a random seed, and the guard never let a lower ticket overwrite a higher one, which is a weak property but a property I can state out loud.

The Second Failure: I Stamped the Ticket in the Wrong Place

My first version attached the ticket at send time, which meant a retry got a brand new ticket and politely re-committed older content as if it were fresh. The fix was to move the stamp to the enqueue step and carry it through every retry unchanged, so a retried ticket 2 stays ticket 2 forever. Did the guard hide work rather than save it? Partly, yes: rejected commits are wasted calls, and the rejection counter is now my cheapest signal that I am paying for answers I throw away.

Where This Guard Is the Wrong Tool

Situation	Use this	Why
One producer, one key, retries	Enqueue-side ticket guard	Cheap, monotonic, survives restarts
Several producers writing one key	Database sequence or a single writer service	Local tickets do not give a total order
Duplicate delivery, same payload	Idempotency key on the request	Guard rejects nothing useful
Strict ordering required	Serialize per key in a queue	No concurrency, no reordering

Limitations and Who Should Skip This

If your pipeline is already sequential per document, this guard buys you a branch and a dictionary lookup, and nothing else. If several machines enqueue jobs for the same key without a shared sequence, a per-process counter will happily accept the wrong order, so use a database sequence instead. And if downstream readers can observe a value mid-flight, the guard does not help, because the stale value was visible before the newer one ever landed.

I would repeat two things: stamping freshness at the enqueue step, and writing the reorder test before writing the fix. What I would not repeat is the assumption that finishing last means thinking last, because on a weekend with free model access and a free server option, the only thing I actually saved was the budget for one more bad assumption. If you keep your own field notes, the rejection counter is a good first line to add this week.