A research backfill triggered an external API block, and that block killed live trading processes during library import
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
During the day I was running a one-off research backfill script, pulling a large amount of historical data from an external market/disclosure API.
A few dozen minutes later, three separate live trading rounds crashed almost simultaneously. None of them were anywhere near the buy/sell decision logic.
What happened
Right before an intraday round, several independent execution units ("sleeves") crashed in a row with the same error. The pattern was Python's standard JSON parser throwing an exception that meant "expected a value but found none."
The stack trace was stranger than the error itself. The crash happened before any execution logic even ran — on the line that imports a third-party library used for the external API. A crash on a plain import statement meant the library was firing off a network request the moment the module loaded.
Luckily, buying itself was already blocked by an unrelated safety mechanism during that window, so no bad fills resulted. But the sell replanning for that round got pushed to the next one, and a parallel simulation round I ran for comparison lost that day's record entirely.
What I thought at first
My first suspicion was a library version issue or a transient network hiccup. The external API was already known to be occasionally slow or to time out, so I assumed today was just worse than usual and reached for a retry.
Before retrying, though, I counted how many requests my research backfill script had logged around the same time. It had tried logging in with the same account several times in a short window and fired off daily queries dozens to hundreds of times.
That's when my suspicion shifted. It wasn't "the library occasionally fails" — it was more likely "I overused the account/IP that library depends on."
The real cause
Looking closer, two separate problems had stacked on top of each other.
Layer one: shared resources. The research backfill and the live trading processes were both connecting to the same external service with the same IP and the same account. The two pieces of code had no idea the other existed, but to the external service, they looked like the same client.
Because the backfill repeated logins and queries in a short burst, the external service flagged the IP and started returning an HTML "access blocked" notice instead of a normal response for every request.
Layer two: an import-time side effect. The third-party library in question tried to log in the moment its module was imported, and parsed that response as JSON right at the top of the module.
Normally the login response was always JSON, so this never caused a problem. But while blocked, the server returned an HTML notice page instead, and the JSON parser threw on it immediately. That exception wasn't wrapped in any try/except, so the import statement itself failed — and every process that depended on this library died before it even started.
What made it worse: this login call wasn't needed by the execution logic at all. I was only using public, no-login endpoints, but the library was written to always attempt login the instant its module was loaded. A feature I wasn't even using held the entire process's availability hostage.
How I fixed it
Patching the library's source directly was off the table — forking and maintaining a third-party package is expensive, and every update would need the patch reapplied.
Instead, I wrapped just the one internal login function the library used. Before importing the library itself, I pre-loaded its internal submodule under the same name and swapped out the login function for a safe version.
The safe version called the original function but caught any exception — JSON parse failure, timeout, whatever — and downgraded it to a normal "login failed" return value. Since login was an unnecessary side effect to begin with, failing it and proceeding in a logged-out state was fine.
I added a before/after regression test: feed in a fake server response that mimics the blocked-access HTML page, and confirm the import no longer dies. The test faked only the response object, with no real network calls.
I also addressed the underlying resource-sharing problem separately. I wrote it into the operating rules that research scripts making bulk external calls should avoid time windows when production uses the same account/IP, and started explicitly stating "don't use production credentials" whenever handing off similar work to someone else, AI agent included.
The generalizable lesson
If a third-party library does network I/O at import time, that's a hidden single point of failure on its own. An import statement is normally assumed not to fail, and surrounding code skips error handling on that assumption. It's worth skimming a library's source once to check whether its top-level module code does network calls or file I/O.
Unused features can still affect availability. I wasn't using the login feature, yet a failure inside it killed the whole process. "I don't use this feature, so it doesn't matter" is only safe once you've confirmed the feature isn't wired into the import/init path.
When two independently written pieces of code share an external resource, one's behavior can break the other's availability. The research script and the live trading processes were completely separate in code, but the moment they shared the same IP and account, they stopped being independent. If multiple jobs hit the same external service, it's worth checking how your combined requests look from that service's side.
When the symptom (a JSON parse error) is far removed from the cause (an IP block), checking what else was running at the same time is usually the fastest path. It's tempting to blame a library bug or network flakiness from the error message alone, but lining up the timeline was the faster route here.
Since this incident, whenever I build a new job that makes bulk external API calls, I add "could this collide with credentials/IP used by another process?" to the checklist before running it.
Top comments (0)