Hey everyone, it's your resident grandpa-developer here.
I develop and run AI-powered automated trading bots in my spare time, evenings, and weekends. Recently, one of my production bots went a full 41 days without making a single trade.
No error logs. My watchdog process monitor was chirping happily. It just looked like it was perpetually "waiting." If the market was quiet and no entry conditions were met, that would be normal. But what if something was broken internally, causing it to ignore signals it should have acted on?
This "running but doing nothing" state is the absolute scariest. It's impossible to distinguish normal waiting from a silent failure. It genuinely gave me heart palpitations.
Digging into Logs Revealed "Ignored Signals"
41 days of silence felt definitively off, so I decided to pull all the detailed logs and investigate. Usually, I only glance at errors and summaries, but this time, I grep'd through every single debug-level raw log.
And then, a stark truth emerged.
Internally, the bot was generating "signals" — the triggers for trades. Meaning, its logic had identified moments when it should've thought, "Now's the time, enter!" But in the subsequent logs, there was no record of an order being placed.
In essence, after generating a signal, the bot was silently suppressing it.
The reason this went undetected for 41 days was a flaw in my log design. I wasn't recording the fact that "a signal occurred" in an easily monitorable way. I was logging "actions" like orders and errors, but not the "opportunities" where an action should have been taken. This made it impossible to tell if the continuous inaction was normal or abnormal.
The Fix: Log "Opportunities" and "Executions"
To prevent these kinds of silent failures, the only way is to log both what the bot intended to do and what it actually did, then monitor the discrepancy. I've dubbed this an "Implementation Fidelity Check."
Specifically, I modified the log output to count the progression from signal generation to execution across different stages.
# Example of newly instrumented log items
# For bot A:
# seen: number of times the monitored event was observed
# qualified: number of times it met signal conditions
# taken: number of times a position was actually taken
# log -> insider_paper: seen=120 qualified=3 taken=2
# For bot B:
# fired: total number of signals generated
# opened: number of orders actually placed
# rejected: number of orders rejected for any reason
# log -> ny_stack: [SIGNALS] fired=5 opened=5 rejected=[]
With this in place, I can immediately spot anomalies like "qualified is increasing but taken remains zero" or "the count for fired and opened doesn't match."
I rolled out this system to all my bots and set up daily aggregations. And lo and behold, a host of problems became visible.
| Bot | Opportunities | Executions | Fulfillment Rate | Notes |
|---|---|---|---|---|
| trend_exec | 1 | 0 | 0% | 🔴 1 opportunity unfulfilled |
| gotobi_paper | 6 | 4 | 67% | ⚠️ Missed end-of-month signals |
For trend_exec, an API spec change had added a new mandatory parameter to requests, which I hadn't followed up on. Order requests were silently failing every time. 0% fulfillment rate. This was terrible.
For gotobi_paper, I had a gap in my calendar definition for specific trading days (like 'Gotobi' days at month-end), leading to signals being generated but not executed. 67% fulfillment rate. Another missed opportunity.
Conclusion: The Scariest Thing in Bot Dev is "Quiet Failure"
What I learned from this incident is that "no errors ≠ normal."
Failures where a process dies or throws an exception and stops are detectable, so they're manageable. The truly terrifying ones are silent failures that don't throw errors, simply bleeding opportunity in the background.
To prevent this, a system that records the bot's "intent (opportunities)" separately from its "actions (executions)" and monitors the fulfillment rate is indispensable. Without it, you can't tell the difference between normal waiting and a critical bug.
In personal dev environments, it's easy to push off setting up monitoring like this. But if you're running bots in production, I've painfully realized it's one of the first features you absolutely must implement. 41 days of lost opportunities was, frankly, quite painful.
From now on, when building a new bot, I'll prioritize integrating this "Implementation Fidelity Check" system. If you're building similar bots, I hope this experience proves useful.
I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.
If a provider-agnostic RAG Q&A API is useful to you, mine is MIT-licensed on GitHub: rag-faq-api. It runs and passes its full test suite **with no API key* (offline stub LLM + hashing embedder), swaps to Claude / Gemini / OpenAI via one env var, and ships a retrieval-quality harness (Hit@k / MRR / Recall@k) with a chunking sweep.*
Top comments (0)