Green all the way down: a trading bot that lied to me in four different languages

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

Here's a fun way to lose confidence in every dashboard you own.

I run a small fleet of automated trading bots. One of them, I'll call it the index bot because it trades index CFDs and gold rather than FX, sat one afternoon at exactly its starting balance, zero open positions, having placed no trades for a suspiciously long time.

Nothing was on fire. systemctl said the service was active. The bot's own risk circuit reported ok. The fix for the bug it was hitting had, according to git, been shipped weeks ago. The last deploy had said done.

Every single one of those was a lie. Here they are in the order I peeled them off.

Lie #1: active

$ systemctl is-active index-bot
active

Except it wasn't running. It was stuck in a crash loop, and had been for three days.

A stray systemd config override, left over from an unrelated experiment, had repointed the service at a minimal Python virtualenv that was missing two libraries the bot imports on startup (the Postgres driver and the market data client). So the process would launch, import its way a few lines in, hit the missing module, die, and get restarted by Restart=always. Launch, die, restart. Forever.

The trap: systemctl is-active doesn't answer "is this service working?" It answers "does a process exist right now?" A restart loop keeps a process existing. Poll it and you almost always land in the brief window between crashes, where the answer is, technically, active. Three days, no alert, because the one signal anyone was watching was structurally incapable of noticing.

Takeaway: a restart policy launders a dead process into one that looks healthy. Alert on NRestarts climbing, and make your health check prove the process did its work, wrote a heartbeat row or answered a ping, not merely that it drew breath.

I fixed the venv override. The bot came up and stayed up. Progress! Onto lie number two.

Lie #2: ok

Now that it was actually running, the journal filled with this, once a minute, forever:

poll: 1 new order intent
place_order INDEX500: INVALID_REQUEST "Relative stop loss has invalid precision"
router summary: { placed: 0, errors: 1, circuit_state: 'ok' }

Read that last line again. It placed zero orders. It logged one error. And it declared its circuit state ok.

The bot has a risk circuit breaker, and it's a good one. It trips on drawdown, on stale prices, on daily loss limits. But a broker rejecting 100% of your orders is not a drawdown event. No money is being lost because no orders exist. So the circuit, which only ever learned to watch for losing money, cheerfully reported green while the bot failed to trade at all.

Takeaway: a guard that watches one failure mode is blind to every other one. "Am I losing money?" and "am I actually able to place a trade?" are different questions, and a health signal that answers only the first will glow green through the second. A 100% rejection rate should page as loudly as a drawdown breach.

So why was every order rejected? That's the actual bug, and it's a beauty.

The bug all four green lights were hiding: the decoder and the encoder disagreed about what a price is

The bot talks to cTrader's Open API. For a market order, cTrader won't take an absolute stop loss price; it wants the stop as an integer distance expressed in a fixed wire scale. cTrader streams prices on a fixed scale of 100,000 integer units per price unit, and the adapter decoded incoming prices with exactly that constant:

_SPOT_PRICE_SCALE = 100_000.0          # cTrader's fixed wire scale

def decode_price(raw: int) -> float:
    return raw / _SPOT_PRICE_SCALE     # correct

But when it came time to encode the stop loss distance to send back, the adapter used the symbol's display digits instead of the wire scale:

# digits = the number of decimals the symbol quotes to
sl_units = int(stop_distance * (10 ** symbol.digits))   # this line is the bug

Look at what those two lines assume. The decoder says a price unit is worth 100_000. The encoder says it's worth 10 ** digits. Those are the same number only when digits == 5.

• A EUR/USD style FX pair quotes to 5 digits, so 10**5 == 100_000, encoder and decoder agree, and orders sail through.
• An index CFD or gold quotes to 2 digits, so 10**2 == 100, the stop is encoded 1000× too small, and cTrader rejects it as "invalid precision."

The bug had existed the whole time. It was invisible on every account that traded 5 digit FX, where the wrong answer happens to equal the right one. The lone account trading 2 digit instruments, an index CFD and gold, was the only place the two scales diverged, and it failed 100% of the time.

The fix is a single line: encode on the same scale you decode with.

def stop_distance_to_units(distance: float) -> int:
    return int(round(distance * _SPOT_PRICE_SCALE))

Takeaway: if you decode with one constant and encode with another, you've planted a bug that hides everywhere the two constants coincide and detonates the first time they don't. Encode and decode through the same function, and write the unit test for the case where they'd diverge. Here, a single assertion on a 2 digit symbol would have caught it years earlier.

Lies #3 and #4: shipped, and done

Here's the part that made me laugh, then wince.

That fix? It was already in master. Past me had already found this, written stop_distance_to_units, and left a comment right next to it warning not to use the display digits, because they encode 2 digit instruments far too small. Per git, the bug was solved and shipped.

The live box was running the old code anyway.

$ sha256sum  <live>/ctrader_adapter.py                       # c5bfc46f…
$ git show origin/master:…/ctrader_adapter.py | sha256sum    # f2948754…

Different bytes. The box had never received the fix. The most consistent explanation is grim: that file lives in a hardened, immutable set, locked with chattr +i so nothing can quietly tamper with the money path, and a plain scp over an immutable file fails. A blocked copy and a successful one look identical unless someone checks. And nobody was: whatever deployed this never compared the bytes on the box to the bytes in git, because if it had, this exact drift is what it would have caught. The deploy said done. The bytes said otherwise.

Takeaway: a deploy that doesn't verify its end state is a wish, not a deployment. Copy the file, then assert sha256(target) == sha256(source), or you will one day discover, as I did, that "deployed weeks ago" and "running in production" are unrelated facts.

The pattern: liveness is not health

Line them up:

The signal	What it claimed	What was true
systemctl is-active	active	stuck in a crash loop for 3 days
circuit breaker	circuit_state: ok	rejecting 100% of orders
git log on master	fix shipped	fix never on the box
the deploy	done	bytes never changed

Four independent "success" signals, four different lies, and each was technically correct about the narrow thing it measured. The unit did exist. The account wasn't drawing down. The commit was on master. The deploy command did run. None of them measured whether the system was doing its job, and the gap between "it's running" and "it's working" is where this whole afternoon lived.

Five things I'm taking into every system I touch after this:

1. Alert on restart counts, not just liveness. Restart=always turns a corpse into a green light.
2. Health checks must assert work happened, a heartbeat, a filled order, a written row, never just "a process is up."
3. Every guard is blind outside its one failure mode. Enumerate the failure modes; "can't lose money" is not the same as "can trade."
4. Encode and decode through the same constant, and test the input where two scales would disagree.
5. A deploy must verify its end state. Hash the target. scp onto an immutable file fails silently, and silence reads as success.

How this was actually caught

Full disclosure, because it's the most interesting part: I didn't find most of this by staring at logs. I pointed an AI agent (Claude) at the live journal and had it diff the running bytes against git, part of an audit pipeline I've been building whose entire premise is don't trust a green dashboard. A human who "knows the system is fine" skims right past circuit_state: 'ok'. A skeptical reader with no such prior stops on it and asks the dumb, correct question: ok according to whom?

That question, asked four times, at four layers, was the whole fix. The bugs were boring: a stray config file, a guard that watched too little, a units mismatch, an unverified copy. No clever algorithm, no exotic race. And that's exactly why they'd survived for weeks: nothing about them looked wrong, because everything that was supposed to tell me they were wrong was busy reporting green.

The least glamorous bugs are the ones that quietly cost the most. Smash them by distrusting your own status lights.

If your infra has ever said active while doing absolutely nothing, I'd love to hear your version in the comments.