DEV Community: Cameron Meese

My bot logged hundreds of trades it never made — so I built something to check if it was lying

Cameron Meese — Sun, 31 May 2026 01:39:49 +0000

I have a rule for new strategies: observe before you bet. Before a single dollar (paper or otherwise) moves, the strategy runs in "would-have-traded" mode — every time it thinks it sees an edge, it writes a row to a log instead of placing an order. Decision, timestamp, the side it would have taken, and the edge it believed it had. You let that run, then you go back and check whether the bot was right.

This is the story of going back to check, and finding out the bot was lying to me in two different ways at once.

The setup

The strategy prices short-duration crypto "up or down" binary markets — will the price be higher at the top of the hour than it was at the start? It builds a fair-value probability from a volatility model and compares it to what the market is charging. When the gap clears fees, it logs a decision.

After a day of observing, the feed looked busy — lots of green, lots of "+5.2¢ edge" rows. And one number jumped out when I tallied it up: the bot was choosing "NO" over "YES" about 4 to 1.

I immediately had a story. My volatility estimate, sourced from one exchange's recent prints, probably runs a little hot — and an overestimate of volatility makes the unlikely side of a binary look underpriced. So the bot keeps "buying" the cheap tail. Made sense. I was about ten minutes from turning down the volatility input and calling it a fix.

That would have been a mistake. The 4:1 number was a hypothesis built on raw counts, and I hadn't checked a single one of those decisions against what actually happened.

The harness

So I built the thing I should have built first: a script that takes each logged decision, looks up the actual outcome of that market (did it close up or down?), and scores it. Win or loss. Then it aggregates — realized win rate vs. the win rate the model predicted, broken out by side and by confidence bucket.

The first run covered 35 resolved decisions. Here's what came back (all paper, all hypothetical — don't @ me about the size):

OVERALL   win 45.7% (16/35)   predicted 49.1%   net -$31.91

Net negative. The strategy I'd been admiring in the feed would have lost money. That alone was worth knowing before risking anything. But the two breakdowns underneath are where it got interesting.

Lie #1: the 4:1 skew was a measurement artifact

I split the decisions by side, deduped to one per opportunity:

buy_no    18
buy_yes   17

Even. Basically a coin flip.

So where did 4:1 come from? The bot re-evaluates every market on every scan, and in observe mode it was logging a decision each time a market still qualified — not once per opportunity. A market that sat in "NO looks cheap" territory for twenty minutes got logged dozens of times; a market that flickered into "YES" for one scan got logged once. The raw feed wasn't measuring my model's bias. It was measuring how long each opportunity lingered.

The "overestimated volatility → buy NO" story was a confident explanation for a number that was pure logging noise. Dedup first, then analyze. I'd skipped the first step and nearly tuned a real model parameter to chase a histogram artifact.

Lie #2: the losses were hiding in the longshots

The other breakdown bucketed every decision by the model's own predicted probability for the side it took:

predicted fair < 0.40   ->  0 wins out of 12
predicted fair 0.4-0.6  ->  64.7% win  (model said 50.4%)

There it is. Every single bet where the model itself rated the chosen side a longshot — taken purely because the asking price was even cheaper than that long shot — lost. Zero for twelve. Meanwhile the coin-flip-ish bets in the middle were actually fine, even good.

That's a different bug than "volatility too high everywhere." It's specifically: don't take a side your own model thinks will probably lose, just because it's on sale. The cheap-tail edge was an illusion of the pricing model on exactly the bets where the model is least trustworthy.

The fix (and the part where I don't trust my own fix)

The change wasn't a volatility knob. It was a floor: don't bet a side the model rates below 40% to win, no matter how cheap. Surgical — it removes the 0-for-12 segment and leaves the working middle alone.

Re-scored with the floor applied, the same data goes from −$31.91 to +$89.61, 69.6% win rate. Which sounds great, and which I am deliberately not celebrating, because that number is in-sample: I picked the 0.40 threshold by looking at this exact dataset. Of course it improves the dataset it was fit to. That's not evidence the floor works. It's evidence I can draw a line through points I already have.

The real test is fresh data the threshold has never seen. So the bot keeps observing — now with the floor live and the logging deduped — and in a few days I re-run the harness on decisions it couldn't have been tuned against. If it's still positive and balanced out of sample, the strategy earns a shot at paper execution. If not, back to the volatility model. Either way, I'll have measured it instead of guessed.

What I'd tell past me

Two things, and they're really the same thing.

A raw count is not a measurement. Before you explain a number, make sure the number is counting what you think it's counting. My "4:1 bias" was a logging cadence in a trench coat.

A result you fit your parameter to is not a result. In-sample improvement is the easiest thing in the world to manufacture and the easiest thing to fool yourself with. The only honest verdict comes from data the decision never touched.

The strategy might still be a dud. I genuinely don't know yet — and that "I don't know yet, here's how I'll find out" is the whole point. Observe before you bet. Then actually check the observations. Then check them again on data you can't have cheated on.

My bot ran for 48 hours and didn't do a thing

Cameron Meese — Sat, 23 May 2026 17:17:19 +0000

I'd been watching a paper-trading bot I've been building for two days. Just paper — no real money at stake — but the silence was getting loud. Zero trades. Not "no opportunities" zero — actively rejected every single one zero. The bot logged 1,262 entry attempts in 24 hours. Every one bounced.

This is the post-mortem.

The hook: why was nothing happening?

The bot's job is to spot setups across a handful of trading pairs and open positions when conditions line up. It had been working. Then I tightened the universe of symbols it watched — added some thinner, more volatile candidates I wanted to test against — and from that moment, nothing.

First instinct: market regime. Maybe nothing was qualifying. So I dumped the rejection log and bucketed by reason.

$ awk '/rejected/' state/decisions.jsonl | jq -r .reason | sort | uniq -c
   1015 stale_quote
    176 insufficient_inventory
     71 max_concurrent_reached

Stale quotes? On 14 actively-watched symbols, across three exchanges, in the middle of a normal trading day? That number didn't pass the sniff test.

The investigation: chasing a lying number

Two things were happening, and they were stacking.

First: the bot tracks "freshness" of price quotes per symbol — if the last quote from an exchange is older than ~60 seconds, you don't trust it for sizing. Reasonable rule.

But to get fresh quotes, the bot polls the exchange's orderbook (via the wonderful but occasionally-temperamental ccxt library). And those polls were timing out — silently, in batches. Five-minute window: 215 orderbook timeouts. Same five-minute window: zero successful quote refreshes.

OK, so the bot has bad quotes. Why doesn't it just… wait and retry?

It does. Sort of. Here's the part that had been working fine for weeks and quietly became the bomb:

# After 3 consecutive orderbook timeouts on (venue, symbol),
# stop scheduling that pair until the bot restarts.
if failure_count[(venue, symbol)] >= 3:
    quarantine.add((venue, symbol))

A reasonable defensive measure. If a (venue, symbol) is wedged, stop wasting cycles trying it. Restart-only recovery means a human is paying attention before it retries.

The bug isn't in the code. The bug is in the assumption the code encodes: "the only way this fails 3 times in a row is if something is permanently broken." That's true 99% of the time. The 1% is when an exchange has a 30-second session warmup on a cold start, and three consecutive 15-second timeouts trip every symbol you're trying to load.

9 of my 14 symbols got quarantined inside the first 65 seconds of boot. They stayed quarantined for the next 10 hours, until I noticed and restarted the bot.

The second bug, which lied to me about the first

While I was in there, I noticed something else weird. A lot of the rejections were tagged stale_quote, but they shouldn't have been — for some of those candidates, the bot didn't even have inventory available. The "do you have inventory?" check should have rejected first.

It was checking. In the wrong order. The freshness check ran before the inventory check, and a stale quote (which, we now know, was caused by the quarantine) was masking the real reason. So the rejection log was lying to me — over a thousand stale_quote entries were really insufficient_inventory events I couldn't see.

This is the part of debugging nobody writes about: you find one bug, and it was hiding two more. Reorder the gate stack, surface the truth, suddenly the histogram tells a different story.

The fix: auto-recovering quarantine

The real fix was conceptual. Permanent-until-restart is the wrong shape. What I wanted:

Sideline a flaky (venue, symbol) after N consecutive failures (keep this part)
After a cooldown, carefully retry (the new part)
If still broken, re-quarantine immediately — not after another N failures

That last invariant turned out to be the one that mattered. If you reset the failure counter on cooldown expiry, a permanently-broken resource costs N × cycles failures over the lifetime of your process. If you preserve the count, it costs exactly one failure per cycle.

Here's the whole thing, about 50 lines:

class AutoRecoveringQuarantine(Generic[K]):
    def __init__(self, *, threshold: int, recovery_seconds: float,
                 clock: Callable[[], float] = time.monotonic) -> None:
        self._threshold = threshold
        self._recovery_seconds = recovery_seconds
        self._clock = clock
        self._failure_count: dict[K, int] = {}
        self._skip_until: dict[K, float] = {}

    def is_quarantined(self, key: K) -> bool:
        until = self._skip_until.get(key)
        if until is None:
            return False
        if self._clock() < until:
            return True
        # Window expired — drop the deadline, KEEP the failure count.
        self._skip_until.pop(key, None)
        return False

    def record_success(self, key: K) -> None:
        self._failure_count.pop(key, None)
        self._skip_until.pop(key, None)

    def record_failure(self, key: K) -> int:
        count = self._failure_count.get(key, 0) + 1
        self._failure_count[key] = count
        if count >= self._threshold:
            self._skip_until[key] = self._clock() + self._recovery_seconds
        return count

The K is whatever hashable key identifies your "thing that flakes" — (venue, symbol), a tenant ID, a customer hash, whatever.

I pulled it out into its own repo: ccxt-auto-recovering-quarantine. Stdlib only. Drop it into any project where one flaky key shouldn't take out the rest.

The validation: the numbers don't lie this time

Before the fix, on the worst day of the cluster:

130 orderbook timeouts (one exchange)
1,286 stale_quote rejections (mostly lies, as we now know)
0 successful trades

After deploying the fix, over the next five days:

0 orderbook timeouts
stale_quote rejections collapsed: 464 → 56 → 0 → 0 → 0
First profitable trade closed: +$0.041 net of fees, held 15h 51min. (Paper money. Don't @ me about the size.)

Five days, four successful trades, 100% win rate on paper. The pattern is doing what it should, and the bot is no longer pretending to work while quietly doing nothing.

What I'd tell past me

If you're writing defensive code that says "after N failures, give up," ask one more question: what is the expected lifetime of "broken"? If it's "forever," your defense is correct. If it's "until something transient clears" — and most things are — you need a way back.

Permanent isn't always the right kind of safe.