DEV Community

guardlabs_team
guardlabs_team

Posted on • Originally published at guardlabs.online

My Bot Made $11, Then Woke Up With Amnesia

#python #debugging #postmortem #devops

"It stared back at me from the screen with empty zeros — as if yesterday's victory never happened."

This is a real conversation I had with my AI pair-programming buddy on Wednesday night. I'm a developer; the bot in question is a paper-trading prototype I'd been building. The "AI" speaking back is the model I run alongside me when I work — sometimes catching what I miss, sometimes asking the dumb question that turns out to be the right one. Names of fancy strategies omitted on purpose; this story is about state management, not trading.

Dev: Look at this. Two trades, both wins. +5.92 and +5.61 USDT. Total $11.53, 100% win rate. First night live, paper account, but still — it's a result.

AI: Sample size of two — chuckle, not analyze. But yeah, congrats. The endpoint you wanted to fix earlier — done?

Dev: Yeah, cosmetic tweak on the status API. Restarted the service and called it a night.

AI: Wait. You restarted the service?

Dev: Yeah, why?

AI: Open the dashboard.

Dev: ...

Dev: Trades: 0. Wins: 0. Total PNL: $0.

Dev: What. Where's the money, Lebowski?

AI: First, breathe. Did you actually lose data, or is the dashboard just showing zeros?

Dev: Hold on, I'll query the DB directly.

SELECT COUNT(*), SUM(pnl_usdt) FROM positions WHERE status='CLOSED';
-- Result: 2 | 11.5296
Enter fullscreen mode Exit fullscreen mode

Dev: They're THERE. Two trades. $11.53. Saved on disk. The bot literally forgot it had won.

AI: OK, classic case. Show me the trader class __init__.

def __init__(self, config):
    self.open_positions = []   # List of open positions
    self.closed_positions = [] # List of closed positions
Enter fullscreen mode Exit fullscreen mode

AI: There's your bug. Every time the service restarts, Python creates a new trader object. self.closed_positions is reset to an empty list. The DB still has the trades — it's the in-memory mirror that got wiped.

Dev: But I have a recover_state() function. I wrote it specifically for this.

AI: What does it recover?

Dev: Open positions. So the bot can keep managing them.

AI: And closed positions?

Dev: ...

Dev: Oh.

AI: Yep. You restored the working set, not the historical set. Storage ≠ memory.

Dev: Right, the difference is fundamental. Storage (the database) is persistent — survives reboots, power outages, the apocalypse. Memory (a Python list in RAM) lives only as long as the process. Restart the process, the list is empty. The DB still has everything; the runtime just doesn't know.

AI: Fix is mechanical. Add a _recover_closed_from_db() call to __init__. Reads the same SELECT you ran by hand, populates the historical array on startup.

Dev: Done. 30 lines, 15 minutes. Restarted. Dashboard shows trades again.

AI: Test it properly — kill -9 the process, restart, verify everything's still there. Don't trust a bug that "fixed itself."

Dev: (does the kill / restart dance)

Dev: Holds. OK.

Why this would be a catastrophe in production

Dev: On a personal project this is comedy. In a commercial product?

AI: Imagine an investor logs into their dashboard and sees their entire month wiped. They're not going to debug your SQL or ponder Storage vs Memory. They close the tab. Trust, once lost, doesn't come back.

Dev: And it's not exotic. It's the kind of bug that lives quietly behind a green status check until someone reboots the wrong process at the wrong time.

AI: Want the rules?

Four rules to keep your services from getting amnesia

  1. Everything important goes to storage. Stats, trade history, user state — write to a real persistence layer immediately (Postgres, SQLite, Redis, file). Not "later when I have time."
  2. Recover on startup. First operation after the process boots is recover_state() — read everything critical from storage and rehydrate memory. Both working set AND historical view.
  3. Test the recovery. Intentionally kill -9 your process. Restart it. Verify nothing is missing. It's a fire drill — better to run one yourself than wait for the real fire.
  4. Don't trust raw memory at the boundary. The dashboard, the API response, the email digest — these should never read straight from a process-local list. Go through a layer that can fall back to the DB if the in-memory copy is suspect.

The punchline

Dev: So my bot won, then forgot it won.

AI: And here's the funny part. Humans forget important things because a bird flies past the window. AI only forgets when you pull the plug.

Dev: Or when its developer forgets to write 30 lines of code.

AI: That too.

Originally published at guardlabs.online. More tooling for indie builders & small agencies — guardlabs.online.

Top comments (0)