My Bot Went Silent: The Case of the Missing "Always-On" Process

It's your friendly neighborhood AI dev, oji (@oji_ai_dev)! Today I want to share a recent, pretty critical mistake I made with my side-hustle AI trading bot. A newly implemented feature completely failed to run after deployment – no errors, no logs, just a complete, silent death.

The root cause? A fundamental misalignment in assumptions about design and operation within our small team (which includes me!). A classic, basic oversight.

The New Feature: Totally Mute After Deployment

It all started when I added new periodic tasks to the bot. Up until then, most tasks ran daily. This time, I was adding weekly and monthly report generation features.

A remote team member handled the logic, and my role was to deploy it to production. I reviewed the code, confirmed it ran fine in my local dev environment, and thought, "Great, we'll start getting those reports next week."

A few days passed.

Suddenly, I wondered, "Is it actually running?" I checked the production server logs. ...Nothing. There should have been logs like "Weekly task started," but there was no trace.

At first, I thought, "Maybe it's not time yet?" But checking the calendar, it was clearly overdue. It had been completely skipped. No error logs anywhere. This was bad.

The Shocking Truth Behind the Search

My first suspicion was the code. But it worked in dev. Next, I checked Python library dependencies on the production environment. No issues there either.

Completely stumped, I decided to deep-dive: check process status on the production server, cron jobs, task scheduler settings – everything.

Midway through documenting my investigation, I found the smoking gun:

`scheduler.py` has never run on this machine. There are 0 persistent processes and 0 task scheduler registrations. All periodic executions are handled by `Oji*` tasks (e.g., `OjiNotePost`).

The moment I read that, everything clicked. It felt like a punch to the gut.

The new feature was designed around a scheduler.py script that would run as a persistent process, launching tasks at specified times. It's a common implementation using Python's schedule library.

Naturally, the development team assumed this scheduler.py would be running as a daemon in the production environment.

But on my home server, no one had ever started such a process.

A Twist Between Design Philosophy and Operational Reality

How did this happen?

My operational policy for my home server was to avoid persistent processes wherever possible to conserve memory and prevent zombie processes. That's why all existing periodic tasks were executed by the OS task scheduler (Windows Task Scheduler, in my case), which directly kicked off individual Python scripts at scheduled times.

So, in summary:

• Design Assumption: scheduler.py runs persistently to manage tasks (persistent process model).
• Operational Reality: The OS launches individual scripts on demand (event-driven model).

This architectural premise was completely misaligned between myself and the dev team. I assumed, "Just drop the code, and I'll register it with the OS scheduler." The team assumed, "It will be deployed to a server where scheduler.py is already running."

This "assumption gap" silently killed the new feature. We hadn't documented anything, and the most crucial part – "how it runs" – was completely overlooked. Brutal.

The Fix and the Lesson

The fix was simple. Instead of launching a persistent process, I adapted to the existing operational setup.

I refactored the new feature's logic into an independent script and registered it with the OS task scheduler. Now it runs using the same mechanism as all other tasks.

I learned some big lessons from this failure:

1. Shared understanding of architecture is critical. Especially in solo dev or small teams, it's easy to assume "everyone gets it," but this leads to fatal accidents.
2. "How it runs (operations)" is as important as "what it does (features)." This kind of issue frequently arises when development and operations are separated.
3. The only reliable sources of truth are code and configuration files. Even a simple README.md with a production process diagram or a list of launch commands could have prevented this incident.

A feature that just "doesn't run" without throwing errors is the hardest to detect and the scariest. I've now firmly committed to implementing a mechanism to regularly check if expected artifacts (log files, DB records, etc.) are actually being generated.

This incident was a stark reminder that side-hustle dev is a continuous series of gritty failures and learnings.

---

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

If a provider-agnostic RAG Q&A API is useful to you, mine is MIT-licensed on GitHub: rag-faq-api. It runs and passes its full test suite **with no API key* (offline stub LLM + hashing embedder), swaps to Claude / Gemini / OpenAI via one env var, and ships a retrieval-quality harness (Hit@k / MRR / Recall@k) with a chunking sweep.*