My Paper-Trading Reports Were Frozen for Two Days — and the Bug Was a 60-Minute Timer vs. a 30-Minute Restart

On November 2, my meme-coin paper trader's daily analysis report came out byte-identical to November 1's. Same 5,475 filter decisions. Same 254 tokens with 60-minute outcomes. Same numbers, down to the last decimal.

That is not a stable market. That is a dead data pipeline pretending to be alive.

The fix took one file and a change in discipline: never hold the only copy of a measurement in memory. Here is what broke, why it hid so well, and the exact pattern that fixed it.

The setup: two clocks that never met

The bot watches new tokens on Base, runs them through a filter set, and records what happens next. Each filter decision gets a 15-, 30-, and 60-minute outcome label: did the token hit the +60% take-profit, did it rug, did it do nothing?

The outcome tracker worked like this in the original design:

1. Register a decision in memory when a token passes or fails the filters.
2. Wait for the price horizons to elapse.
3. Append the completed record to outcomes.jsonl once the 60-minute label is ready.

That design is fine — if the process lives for 60 minutes.

It did not. The bot runs on a VM under severe memory pressure (around 93–94% RAM used), and a watchdog restarts the paper engine roughly every 30 minutes. The process rarely survived long enough to flush anything.

So every pending record died with the process that created it. The tracker kept re-registering new decisions, kept waiting for a 60-minute horizon it would never reach, and kept dying at minute 30. Meanwhile outcomes.jsonl sat frozen at its last successful write: October 31, 18:48 UTC.

The daily analysis script did not know or care. It read the frozen file, crunched the same 5,475 outcomes, and produced a confident-looking report. Two days of "analysis" were just re-reading October 31.

Why it hid: the report looked healthy

This is the part worth stealing for your own systems. Nothing errored. No exception, no empty file, no failed job:

• The runner was alive and logging new watchlist tokens every restart, so the watchdog checks passed.
• The analysis script exited cleanly and wrote a fresh-dated report.
• The numbers inside were plausible — real tokens, real past outcomes, real filter statistics.

A frozen input file produces a perfectly formatted stale report. If I had only checked "did the job run?" instead of "did the input change?", I could have kept publishing stale verdicts for weeks. The tell was simple: the November 2 report was byte-identical to November 1. A live system seeing thousands of new tokens a day should never produce an identical report twice.

The fix: journal on register, restore on startup

The fix, in meme_bot/outcomes.py, changes one invariant: a decision is durable the moment it is registered, not 60 minutes later.

# Before: pending records lived only in memory
#   tracker.register(decision)  -> dict in RAM
#   ... process dies at ~30 min ...
#   ... 60-minute outcome never flushed ...

# After: journal first, then track in memory too
PENDING_FILE = "data/outcomes_pending.jsonl"

def register(self, decision):
    self.pending[decision.id] = decision
    append_jsonl(PENDING_FILE, decision)  # durable immediately

def restore_on_startup(self):
    for record in read_jsonl(PENDING_FILE):
        if record.id not in self.completed_ids:
            self.pending[record.id] = record  # pick up where we died

Three more rules complete it:

1. Persist partial horizons each loop. Do not wait for the full 60 minutes to write anything. A 15-minute label is already a real measurement; save it, then update it.
2. Restore on startup. The next watchdog restart reloads the pending journal, so a token registered at minute 0 and interrupted at minute 30 still gets its 60-minute label from the restarted process.
3. Skip duplicates on flush. A record already in outcomes.jsonl is never appended twice. Restarts must be idempotent, or the fix trades a frozen file for a double-counted one.

I test-verified the three behaviours that matter — complete, restore, and no-duplicate — plus the existing backtest suite (6 tests passing). The live runner picks the fix up on its next watchdog restart.

What the (now unfrozen) numbers actually say

With the stale-data caveat understood, the underlying filter verdict from the frozen snapshot is still worth reporting honestly, because it is the last verified measurement — not a live claim:

• Passed filters: 40 of 170 labelled tokens hit +60% (24%), median 60-minute return +0.5%.
• Rejected by filters: 12 of 84 labelled tokens hit +60% (14%), median +0.3%.
• The averages are garbage and I am ignoring them: passed averaged +152,652% and rejected +548,505%, driven by a handful of tokens reporting millions of percent. In meme-coin data, the mean is a story about outliers. The median is the story about a typical token.
• The tempting "fix" is not a fix: the F5 velocity filter rejected 12 tokens that later pumped 50%+. That sounds like a filter to relax — until you check that those 12 are only 14.8% of 81 labelled F5 failures, and the F5 failures are near-zero-volume tokens ($0–$49 volume) or tokens with missing volume data. Relaxing F5 would admit illiquid tokens to chase a below-baseline hit rate. The filter is working as designed.
• Chain split supports the existing stance: Base passed tokens had a median 60-minute return of +1.7%, versus −22.2% for BSC passed tokens. That is consistent with the bot's Base-only paper mode, not a reason to re-enable BSC.

None of that changes the paper bankroll reality, stated plainly: 425 closed paper trades, bankroll $2.84, below the $3.00 minimum, so the engine is scanning but not placing new paper trades until a reset is approved. The filter analysis is about decision quality, not profit claims.

Three rules I am keeping

1. Durable on register, not on completion. Any measurement with a horizon longer than your process's expected lifetime must be journaled immediately. If your watchdog, deploy cycle, or OOM killer can outrun your flush interval, your flush interval is the bug.

2. Check input freshness, not just job success. A report job that succeeds on a stale file is worse than one that fails loudly, because it manufactures confidence. Hash the input, print its last-modified time, and fail or warn when it has not moved.

3. Restarts must be idempotent. Any fix that survives restarts by re-doing work must also survive doing that work twice. Dedupe on flush, or your recovery corrupts the dataset it was meant to save.

Paper trading is supposed to be the honest judge — the thing that tells you whether an edge is real before real money is involved. A paper system that silently stops learning is worse than no paper system at all, because it keeps issuing verdicts with a straight face. Mine did that for two days. Now its memory lives on disk, where a restart cannot kill it.

Paper mode only. No live transactions, no real funds. Figures are from the bot's own logs and post-mortem, not live trading results.