Getting a trading bot's detection and execution logic right is only half the problem — the other half is knowing, at any moment, whether it's actually alive, healthy, and behaving normally, without babysitting a chart yourself. This post covers the observability layer: heartbeat checks, anomaly alerting on trade frequency and size, log discipline that actually helps during an incident, and the specific failure modes that are silent by default unless you build something to surface them.
The failure mode nobody designs for on day one
Most write-ups about trading bots (including a couple of my own) focus on the interesting parts — signal detection, confluence scoring, execution logic. What they skip is the boring infrastructure question that actually determines whether you find out about a problem in five minutes or five days: how do you know, right now, whether your bot is working correctly?
This matters more for a trading bot than most automated systems, because the cost of "it silently stopped working three days ago and I didn't notice" isn't a stale dashboard — it's either missed opportunity cost or, worse, a bot that's still running but behaving abnormally with real capital attached.
Layer 1: Is it even alive?
The most basic check, and the one it's easy to assume you don't need until the day you do: a heartbeat.
import time
import requests
class HeartbeatMonitor:
def __init__(self, webhook_url, interval_seconds=300):
self.webhook_url = webhook_url
self.interval = interval_seconds
self.last_beat = time.time()
def beat(self):
self.last_beat = time.time()
def check_and_alert(self):
if time.time() - self.last_beat > self.interval * 2:
self._send_alert(
f"No heartbeat in {int(time.time() - self.last_beat)}s — bot may be down."
)
def _send_alert(self, message):
requests.post(self.webhook_url, json={"text": message})
This alone catches the crudest failure mode: the process crashed, the server rebooted and the service didn't restart, or a broker API outage hung a request indefinitely with no timeout. None of these are exotic scenarios — they're the ordinary failure modes of any long-running process, and a trading bot with no capital at risk while it's silently down is the good outcome. A trading bot that's silently malfunctioning while still running is worse, which is why heartbeat alone isn't enough.
Layer 2: Is it behaving normally, not just running?
A process can be technically alive while doing something wrong — stuck in a retry loop, placing far more trades than expected, or going unusually quiet during a session it should be active in. This requires baselining what "normal" looks like and alerting on deviation:
class AnomalyDetector:
def __init__(self, expected_trades_per_session=(1, 4)):
self.expected_range = expected_trades_per_session
self.session_trade_count = 0
def record_trade(self):
self.session_trade_count += 1
if self.session_trade_count > self.expected_range[1] * 2:
self._alert(
f"Trade count ({self.session_trade_count}) is well above normal "
f"range {self.expected_range} — possible duplicate execution or logic error."
)
def check_session_end(self):
if self.session_trade_count == 0:
self._alert("Zero trades this session — check signal detection and broker connectivity.")
self.session_trade_count = 0
def _alert(self, message):
print(f"[ANOMALY] {message}") # replace with real alerting channel
This is the layer that would have caught something like a webhook retry duplicating an order, or a confluence threshold silently misconfigured after an update — both of these are "the bot is technically running" failures, not "the bot crashed" failures, and a pure heartbeat check is blind to both.
Position size and exposure anomalies deserve their own check, separate from trade count, because this is the category where an undetected bug is most expensive:
def check_position_size_anomaly(current_position_size, expected_max_size, alert_fn):
if current_position_size > expected_max_size * 1.5:
alert_fn(
f"Position size {current_position_size} exceeds expected max "
f"{expected_max_size} by more than 50% — possible sizing bug."
)
Layer 3: Logs that actually help during an incident, not just after
The instinct is to log everything. The reality is that undifferentiated logs are close to useless at 3am when something's actually wrong and you need to find the relevant line among thousands of routine ones. Structured, leveled logging with consistent fields matters more than log volume:
import logging
import json
logger = logging.getLogger("goldmine_bot")
def log_trade_decision(signal, decision, reason):
logger.info(json.dumps({
"event": "trade_decision",
"signal_id": signal.get("id"),
"confidence": signal.get("confidence"),
"decision": decision, # "executed" | "skipped"
"reason": reason, # "below_threshold" | "risk_ceiling" | "executed"
"timestamp": signal.get("timestamp"),
}))
The specific discipline that pays off here: log every decision, not just executions. A gap in your logs where the bot should have evaluated a signal but didn't is often the first visible symptom of a real problem — and if you only log executed trades, that gap is invisible until you go looking for it, which usually means you're already troubleshooting a complaint rather than catching an issue proactively.
Layer 4: The dashboard question — what actually needs a human to see it live?
Not everything needs a real-time dashboard. Most of what matters can be handled by alerting on deviation (layers 1–2) plus reviewing structured logs after the fact (layer 3). The genuinely useful real-time view tends to be narrow: current open positions and their unrealized P&L, time since last signal evaluation, and time since last successful broker API call. Anything beyond that starts turning into a distraction — a dashboard designed to be stared at tends to encourage exactly the manual-override temptation an automated system was supposed to remove in the first place.
What this actually caught in practice
A broker API rate limit that started silently dropping order confirmations during a high-volatility news window — heartbeat stayed healthy (the process was fine), trade count looked plausible, but position-size reconciliation against the broker's actual account state caught a mismatch that wouldn't have surfaced any other way.
A logic change that quietly tightened the confluence threshold further than intended during an update — the bot stayed alive and logged normally, but the zero-trades-this-session alert fired for three consecutive Asian sessions before anyone noticed, which is exactly the kind of gradual, non-crashing failure that observability layers 1–2 are built to catch and a human staring at a chart occasionally would likely miss.
FAQ
Isn't this overkill for a single-strategy retail bot?
The heartbeat and zero-trade alerting are genuinely cheap to build and catch the most common failure modes — I'd consider those close to mandatory regardless of scale. The more granular anomaly detection can be added incrementally as you get a feel for which failure modes actually occur in your specific setup.
How do I set a reasonable "expected trades per session" baseline?
Start from your backtested or forward-tested signal frequency, add a reasonable margin, and adjust after a few weeks of real observed data — the goal is catching genuine anomalies, not generating so many false alerts that you start ignoring them.
What alerting channel is actually best for this?
Whatever you'll actually see promptly — a webhook to a messaging app you already check (Slack, Discord, Telegram) tends to work better in practice than email, which is easy to let pile up unread.
Should logs and alerts be built before or after the trading logic itself?
Build the heartbeat and basic decision logging alongside the trading logic from the start — retrofitting observability after a bot has been running blind for months means you have no historical baseline for what "normal" ever looked like.
Does more logging slow down execution-critical code paths?
Structured logging of decisions is cheap enough not to matter for typical trading frequencies, but if you're operating at very high frequency, asynchronous or buffered logging is worth considering so logging I/O doesn't sit in the critical execution path.
If you run any long-lived automated system — trading or otherwise — what's the failure mode that was invisible until you specifically built something to detect it? I have a suspicion "the process is alive but doing something subtly wrong" is a more universal blind spot than most of us design for on the first pass.
Top comments (0)