I wanted one thing: a short daily report scoring how nervous the market looked, built entirely on free data, running unattended on a schedule, so I'd never have to manually check four different sites before deciding whether a strategy's risk controls should be tightened. Every individual piece of that was easy. Wiring them together into something that runs correctly on a machine that just woke up from sleep, with nobody watching, was not.
The shape of the report
The script pulls from three genuinely free sources -- no paid API keys required beyond a free-tier signup: the St. Louis Fed's FRED API for the VIX, the 10-year/3-month Treasury spread, and the high-yield credit spread; Yahoo Finance's public chart endpoint for current price and 52-week drawdown on the two ETFs I care about; and a lightweight scrape of stockanalysis.com for PE ratios as extra context. Each metric gets bucketed into a tier -- safe, caution, danger, extreme -- and the tiers sum into a single risk score out of eleven, logged to a dated JSON file every day a Windows Task Scheduler job fires.
The first surprise: the "free" API wasn't the one to trust
FRED has an unofficial, no-key-required CSV download endpoint (/graph/fredgraph.csv) that a lot of quick scripts use, and that's where I started. Running the report repeatedly during testing got that endpoint temporarily blocking my requests -- not documented anywhere I could find, just an observed behavior once I was hitting it more than a handful of times. The actual official FRED API, which requires a free registration and an API key, doesn't have that problem and returns clean JSON instead of a CSV to parse by hand. The unofficial shortcut was the exact kind of thing that works fine while you're building it casually and fails exactly when you start relying on it.
def fred_latest(series_id):
url = (
f"https://api.stlouisfed.org/fred/series/observations"
f"?series_id={series_id}&api_key={FRED_API_KEY}&file_type=json"
f"&sort_order=desc&limit=10"
)
raw = curl_text(url)
data = json.loads(raw)
for obs in data["observations"]:
if obs["value"] != ".":
return obs["date"], float(obs["value"])
raise RuntimeError(f"FRED {series_id}: no valid value found")The second surprise: PowerShell's own HTTP cmdlet was the wrong tool
Every other script in my automation setup uses plain PowerShell (Invoke-WebRequest) to make HTTP calls, and that's what I reached for here too. Running it manually, at the terminal, worked fine every time. Running it from an unattended Task Scheduler job produced intermittent multi-second delays before requests even started -- long enough to occasionally blow past timeouts on a script meant to run in a few seconds. The proximate cause, as best I could pin down, is that Invoke-WebRequest does its own proxy auto-detection on each call, and that detection step behaves differently -- slower -- in a non-interactive, freshly-woken session than it does in a terminal a human is actively sitting at. Switching every HTTP call in this script to shell out to curl.exe directly removed the delay entirely. Same underlying network call, different client, no more mystery stall.
def curl_text(url, timeout=20, retries=3, retry_delay=3):
for attempt in range(1, retries + 1):
try:
result = subprocess.run(
["curl.exe", "-s", "-m", str(timeout), "-A", UA, url],
capture_output=True, text=True, encoding="utf-8", errors="replace",
timeout=timeout + 10,
)
if result.returncode == 0 and result.stdout:
return result.stdout
except subprocess.TimeoutExpired:
pass
if attempt < retries:
time.sleep(retry_delay)
raise RuntimeError(f"curl failed after {retries} attempts: {url}")Why this rhymes with a bug I've already written about
This is the second time in this project that "works perfectly when I run it by hand, breaks specifically when Task Scheduler runs it unattended" has turned out to be the actual root cause of a failure, rather than anything about the API being called. The first time it was a PowerShell script file's text encoding silently corrupting itself depending on how Windows interpreted it outside an interactive session. This time it was a networking cmdlet behaving differently depending on session type. Neither failure mode announces itself as "this is an unattended-execution problem" -- both looked, at first glance, like the remote service being flaky. The actual lesson generalizes: when something only breaks unattended, the interactive/non-interactive distinction itself is a prime suspect, well before blaming the thing on the other end of the network call.
What the finished script actually does
Once both issues were sorted, the daily job is straightforward: pull the three macro indicators, pull price and drawdown data for the two ETFs I track, score everything into tiers, sum to a single number out of eleven, and write both a human-readable text summary and a dated JSON log. It runs once a day on a schedule, entirely on free data, and I never have to remember to go check four separate websites to answer "is the market doing anything unusual today."
Top comments (0)