A 16-second clock drift falsely tripped my trading bot's kill switch. A postmortem.

#python #debugging #api #postmortem

I run a small, deliberately boring trading bot: daily bars, spot only, no leverage, about ¥24,000 of my own money on a Japanese exchange (GMO Coin). It wakes up once a day at 06:05, looks at yesterday's candle, and usually does nothing.

On day six it halted itself and reported a 58% drawdown. My balance had not moved by a single yen.

(The free lite backtester for this system, public API only, no key: GMO Coin Trend Lab, source on GitHub.)

Nothing was lost. No order was sent. But the bug is a good one, because every link in the chain looked reasonable on its own.

The timeline

The PC had been powered off for three days. When it came back, Windows had not yet resynced the clock. It was running about 16 seconds fast.
The scheduled task fired. The bot signed its first private API call with the local timestamp.
The exchange rejected it: ERR-5009 Timestamp for this request is too fast.
My startup code treated "key validation failed" as "no usable key" and fell back to dry-run mode. I had written that fallback on day one so that pasting the wrong kind of key would not crash anything.
Dry-run mode has its own paper balance, ¥10,000, and it shared the same state file as live mode.
The bot computed equity as ¥10,000, compared it with the stored high-water mark of ¥23,900, got a 58% drawdown, and did exactly what it is supposed to do at 30%: flatten and halt.

Because it was in dry-run, "flatten" sent nothing. The halt flag, however, was written to the real state file. The bot would have sat there, halted, until a human noticed.

Why each decision looked fine in isolation

"Fall back to dry-run if the key does not validate." Friendly on day one, when the likely failure is a user pasting the FX key into the crypto slot. Dangerous on day six, when the key is fine and the failure is transient.

"One state file." Simple. Until two modes with different balances write the same high-water mark.

"Halt at 30% drawdown." Correct, and it worked. The kill switch is not the bug. The bug is that a number from one world was compared against a number from another.

Fix 1: sign with the server's clock, not mine

The exchange checks that your timestamp is close to its own. So ask it what time it is. Every HTTP response carries a Date header, and that is accurate enough for a tolerance measured in seconds.

import email.utils, time, requests

PUBLIC = "https://api.coin.z.com/public"

class GmoClient:
    def __init__(self):
        self.s = requests.Session()
        self._offset = None   # local clock minus server clock, in seconds

    def clock_offset(self, refresh=False):
        if self._offset is None or refresh:
            try:
                t0 = time.time()
                r = self.s.get(PUBLIC + "/v1/status", timeout=15)
                t1 = time.time()
                srv = email.utils.parsedate_to_datetime(r.headers["Date"]).timestamp()
                self._offset = (t0 + t1) / 2 - srv
            except Exception:
                self._offset = 0.0
        return self._offset

    def _timestamp_ms(self):
        # the Date header has one-second resolution, so lean half a second early:
        # "slightly slow" is tolerated, "too fast" is rejected
        return str(int((time.time() - self.clock_offset() - 0.5) * 1000))

Two details matter. Take the midpoint of the request so latency does not bias the estimate. And lean slightly early, because this API rejects timestamps from the future more strictly than ones from the recent past.

After this change the bot does not care whether the OS clock is right.

Fix 2: a failed validation must not touch live state

c = GmoClient()
if not c.dry_run:                      # keys are present, so we intend to be live
    try:
        c.assets()
    except GmoError as e:
        log(f"key validation failed, aborting this run without touching state "
            f"(clock offset {c.clock_offset():+.1f}s): {e}")
        return                          # try again tomorrow

if c.dry_run:
    STATE = ROOT / "state_dry.json"    # paper trading lives in its own file

The rule I took away: if the system intended to be live and cannot prove it is live, it should do nothing, loudly. Not "degrade gracefully" into a different mode that shares storage with the real one.

What I check now when writing a fallback

What does the fallback write, and where? If it shares storage with the normal path, it is not a fallback, it is a second writer.
Is the failure I am catching permanent (wrong key) or transient (clock, network, maintenance window)? Transient failures should abort and retry later, not change mode.
Would I notice? The halt was silent until I went looking. The bot now writes a dashboard every ten minutes with a red banner when it is halted.
Can the safety mechanism be fed garbage? A kill switch is only as good as the equity number going into it.

The boring numbers, since people ask

The strategy itself is a 20-day breakout with a 50-day filter and a 2×ATR stop, risking 1% per trade. My backtest on the exchange's own daily data from 2018 to 2026 gives about the same return as volatility-targeted buy-and-hold with roughly half the drawdown, and close to zero in ranging years. Walk-forward out-of-sample Sharpe is 0.85. Its first real trade happened on day eight: 0.0076 ETH, with about ¥240 at risk.

Operational holes scare me more than strategy losses. This one cost nothing, which is the best price to learn at.

If you want to poke at the backtest, the lite version is a free download (public API only, no key needed): https://wataflow1.gumroad.com/l/trend-lab-free

Not investment advice.

DEV Community