It took two straight days of the same incident to find the real cause - local imports and Python's default-argument binding
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
A few days ago, one of my paper trading accounts stopped trading for the same reason two days in a row.
The first time, I concluded "some test or debug script must have written to the real database by mistake" and moved on. Only when the exact same thing happened the next day did I actually find the real cause.
What happened
The execution watchdog fired an alert. A paper trading account was hitting a safety gate — a check for booked holdings exceeding the broker's actual holdings — on every scheduled round, and blocking all trades because of it.
It wasn't a crash. The safety gate correctly detected that "the ledger shows more shares than the broker actually holds" and blocked trading exactly as designed.
The real question was why the ledger had more shares than the broker in the first place.
Opening the ledger, I found a handful of odd rows. A normal fill should leave a matching order-submission and order-fill journal entry, and these rows had neither.
The round identifier was empty, and the price was exactly 1/1000 of the real fill price. Something had called the ledger-write function directly.
What I thought at first
The timing lined up with recent work adding a scale guard — a safety check meant to catch exactly this kind of price-scale anomaly.
So I concluded some test or debug script running during that work must have accidentally written to the real database. I couldn't pin down which script exactly.
I backed up the odd rows, deleted them, and reconciled the ledger back to the broker's real holdings. I didn't set up any specific prevention — I closed it out as a one-off incident.
That conclusion turned out to be wrong, and it took a day to find out.
The real cause
The next day, the same pattern showed up again. This time I traced it all the way through.
The culprit was a recently added regression test file. It was meant to verify that the fill-price scale guard worked correctly, and it mocked out the broker and the journal directory with dummies.
But it missed one thing: the very last function on the success path — "once a fill is confirmed, write it to the ledger." The function that called it imported that ledger-write function locally, inside the function body, instead of at the top of the module.
A local import re-resolves the name every time it runs. So even though the test patched the "database path" constant on the higher-level module, this function never saw that patch — it kept writing to the real path every time.
On top of that, the function's database path default was already bound to the real path string at function-definition time (import time). Swapping the module constant afterward can't touch a default value that's already frozen — a classic Python default-argument trap.
As a result, every test case exercising this "successful booking" path left a fake fill row in the real production database, every single time the pytest suite ran.
Both the incident from two days earlier and this one had the same root cause. I just didn't know it the first time.
How I fixed it
The fix had two parts.
First, I explicitly added the missed function to the mock list. Since it was a local import, patching the module path directly worked — the function looks up the name again at call time anyway. I re-ran the regression suite and confirmed the database was left completely untouched.
Second, I registered this database file in the "files that must not change during a test run" watch list. This list checks whether specified files changed by the time the test suite finishes, and fails immediately the moment isolation leaks.
If that list had already included this file, both incidents would have been caught the instant they happened. The fact that it only got added after the incident is also a reminder that even the guard meant to catch isolation gaps can't cover everything from day one.
Generalizing
Mock where the name is looked up, not where it's defined. If a function locally imports a constant or function from another module every time it runs, patching that constant on the higher-level module does nothing. A local import means the lookup happens at call time, so the patch has to target that exact call-time namespace.
Python default arguments are evaluated once, when the function is defined — at module import time. Changing the module variable it originally referenced later doesn't change the already-frozen default. Combine these two behaviors and you get a very quiet way to end up with "I thought I patched this, but it didn't take."
When writing a regression test, enumerate every side effect the function under test can trigger internally. In this case, the broker call and the notification send were both mocked, but a third side effect — the database write — was missed. Without reading the code and listing out everything the function actually touches, gaps like this survive.
If you can't pin down the cause the first time, leave something behind that will connect it to the next occurrence. The first time, I closed it out as a one-off. Only the recurrence revealed that both incidents shared the same cause. Before deleting anomalous data, it's worth recording its shape — call pattern, timestamp spacing, a suspicious scale factor — so the next occurrence connects to it faster.
Success paths need as much isolation as failure paths. It's natural to ask "what happens if this function fails" when writing a test. It's easier to forget to ask "what does this function touch when it succeeds." That success path was exactly where this incident originated.
Top comments (0)