Hey everyone, it's your friendly neighborhood old man dev here. I'm 38, working as an engineer during the week, and building AI algo trading bots on the weekends.
Sometimes, when I'm building personal projects like these bots, I'll notice something like, "Hmm, this thing hasn't been spitting out logs lately." That usually means it's silently halted. And that's exactly what happened this time.
The task was simple: regularly fetch data for about 250 tickers from a certain market API. It should have been straightforward. But the logs showed it was stopping midway every single time. No completion logs whatsoever.
My first thought was a momentary network interruption or a temporary API server glitch. But after multiple retries, it kept failing at the same spot. A deeper dive revealed a much simpler, more fundamental design flaw.
The Root Cause: A Marathon Designed to Fail
The reason was, quite simply, a timeout.
- Time to fetch data for 1 ticker: Average 2.8 seconds
- Number of tickers to fetch: 250
- Total theoretical time to fetch everything: 2.8 seconds × 250 = 700 seconds
Meanwhile, the execution environment where this bot runs has a timeout setting of 600 seconds (10 minutes).
I think you can see the problem now. I was running a process that requires 700 seconds to complete in an environment that forcibly terminates after 600 seconds. Of course, it would never finish. It was entirely my design fault. D'oh!
Why did I overlook such a basic thing? During development, I was testing with only about 10 tickers. 10 tickers would finish in just under 28 seconds. "Yep, it works, good to go!" I thought. Then, when I scaled it up to the full 250 tickers for production, I completely skipped calculating the total time it would take.
The Solution: Not Just a Cache, a "Self-Healing" Cache
When faced with this kind of problem, the first thing that comes to mind is, naturally, caching. Store the fetched data locally, and for subsequent runs, reuse the local data instead of hitting the API.
But in this case, that alone wasn't enough.
Why? Because the very first run to build the cache would still take 700 seconds and hit the timeout. When the process crashes due to a timeout, any partially fetched data in memory simply vanishes. So, the next time it starts, it would have to re-fetch all 250 tickers from scratch, leading to another timeout... an infinite loop where the cache never gets built.
That's where the idea of a 'self-healing cache' came in.
What I did was incredibly simple: "Save progress frequently at logical checkpoints."
Specifically, after fetching every 50 tickers, I would write the results obtained so far to a file in pickle format.
With this approach, even if the process is interrupted by the 600-second timeout, at least 50 × N items of progress remain on disk.
On the next startup, it first loads this cache file. Then, it only goes to the API to fetch the tickers that are not yet present in the file.
By repeating this, no matter how many timeouts occur, the cache for all 250 tickers will eventually be completed. Even if the task is interrupted, it can resume from where it left off.
The Code: Saving Progress Every 50 Items
Here's what the actual code looks like:
import pickle
import os
import time
CACHE_FILE = 'api_data_cache.pkl'
CACHE_TTL_SECONDS = 6 * 24 * 60 * 60 # Cache is valid for 6 days
def get_data_with_self_healing_cache(tickers_to_fetch):
"""
A caching mechanism that saves intermediate progress every 50 items
so the process can resume even if interrupted for a long time.
"""
cache = {}
# Load existing cache if it exists (and is within TTL)
if os.path.exists(CACHE_FILE):
if time.time() - os.path.getmtime(CACHE_FILE) < CACHE_TTL_SECONDS:
with open(CACHE_FILE, 'rb') as f:
cache = pickle.load(f)
# Create a list of tickers that truly need to be fetched via API this time
needed_tickers = [t for t in tickers_to_fetch if t not in cache]
print(f"Total: {len(tickers_to_fetch)}, Cached: {len(cache)}, To Fetch: {len(needed_tickers)}")
fetched_count = 0
for ticker in needed_tickers:
try:
# data = fetch_from_external_api(ticker) # This is the time-consuming API call
data = {'price': 1000, 'timestamp': time.time()} # Using dummy data for this example
cache[ticker] = data
fetched_count += 1
# ★★★ Core part of self-healing ★★★
# Save cache to disk every time 50 new items are fetched
if fetched_count > 0 and fetched_count % 50 == 0:
print(f"--- Saving intermediate cache progress ({fetched_count} new items) ---")
with open(CACHE_FILE, 'wb') as f:
pickle.dump(cache, f)
except Exception as e:
print(f"Error fetching {ticker}: {e}. Saving progress before exit.")
# Even if an error occurs, save progress before re-raising the exception
with open(CACHE_FILE, 'wb') as f:
pickle.dump(cache, f)
raise
# Finally, save all results
if fetched_count > 0:
print("--- Saving final cache ---")
with open(CACHE_FILE, 'wb') as f:
pickle.dump(cache, f)
# Return data for all requested tickers
return {t: cache.get(t) for t in tickers_to_fetch}
The key is the fetched_count % 50 == 0: part, where I regularly write progress to a file. It's also subtly important to include the save operation in the except clause. This ensures that even if an unexpected error occurs, all the hard work up to that point isn't lost.
Results and Lessons Learned
With this fix, the bot now runs stably.
As expected, the initial run timed out once. But looking at the logs, about 200 items of data were correctly saved to the cache file. On the second run, it fetched the remaining 50 items, and the cache for all tickers was successfully completed.
And the runs after that were wild.
With a 100% cache hit rate, there are zero API calls. The process that previously couldn't finish even in 600 seconds now completes in an average of 2.8 seconds. That's not just 10x faster – it's an order of magnitude improvement.
The lesson I learned from this is the importance of designing with the assumption that "tasks will be interrupted." Especially in personal development, cloud environments, or home servers, you never know when things might go down. When writing long-running processes, you absolutely must consider "resumption from interruption" as part of the design.
This "self-healing" concept can be applied in many situations beyond API fetching, like heavy data analysis batch jobs or processing tens of thousands of files.
If you're struggling with a "long-running task that inexplicably never finishes," I hope this helps you out!
Top comments (0)