DEV Community: mattleeee

Surviving Corporate HTTPS Interception: Life Behind Sangfor aTrust

mattleeee — Tue, 06 Oct 2026 04:54:16 +0000

Corporate laptops are strange machines. You have admin rights on paper, but every outbound TLS connection is terminated and re-originated by a local agent. Sangfor aTrust is one of the more common zero-trust clients in APAC enterprises, and it does exactly this: it installs a root CA into the OS trust store, proxies traffic through a local service, and re-signs certificates on the fly. Your browser shows a green padlock. pip install does not.

This post is a field guide to living with that setup without getting your laptop quarantined.

How the interception actually works

aTrust runs a local proxy (typically on 127.0.0.1 on some port) and pushes a PAC file or system proxy setting. When you connect to pypi.org, the agent completes the TLS handshake with the real server using its own credentials, then presents you a certificate signed by the corporate root CA. The OS trust store contains that CA, so anything using the system store is fine. Anything shipping its own bundle is not.

The tell is always in the chain. Run:

openssl s_client -connect pypi.org:443 -servername pypi.org -showcerts </dev/null 2>/dev/null \
  | openssl x509 -noout -issuer -subject

On a clean network you get something like issuer=C=US, O=Let's Encrypt, CN=R3. Behind aTrust you get issuer=C=CN, O=Sangfor Technologies, CN=Sangfor SSL Proxy CA or an internal enterprise CA name. That single line is the difference between a working environment and a day of debugging.

Distinguishing MITM from a normal corporate proxy

A plain forward proxy does not terminate TLS — it just tunnels bytes and the issuer stays untouched. MITM means the issuer changed. To confirm it's the agent and not a rogue AP:

Compare the issuer seen by openssl s_client against the issuer shown in the browser's certificate viewer for the same host. If they match and both are internal, it's the agent.
Check the local proxy port: netstat -ano | findstr LISTENING on Windows, then match the PID to the aTrust process.
Hit a host with a known public chain, e.g. openssl s_client -connect github.com:443. If GitHub is also re-signed, everything is being intercepted.

Once you know the CA's subject and fingerprint, you can decide per-tool what to do.

Which tools tolerate the corporate CA

The dividing line is where the trust anchor comes from.

Tolerates it (uses OS trust store):

Chrome, Edge, Firefox (with security.enterprise_roots.enabled=true)
Windows-native HTTP stacks, .NET, WinHTTP
curl on Windows when built against Schannel
macOS curl and NSURLSession-based tools
Python requests only if you point it at the system store or a bundle containing the corporate CA

Does not tolerate it (ships its own bundle or pins):

Python certifi (bundled, frozen at release time)
Node.js (bundles Mozilla's CA set)
Git for Windows (ships its own ca-bundle.crt)
Rust reqwest default roots
Anything doing certificate pinning: mobile SDKs, some CLI tools, go binaries with embedded roots

The fix is almost always the same: build a bundle that contains both the Mozilla roots and the corporate root, then point the tool at it.

Exporting the corporate root CA

On Windows:

# List candidate roots, then export the one you identified
Get-ChildItem Cert:\LocalMachine\Root | Where-Object { $_.Subject -like "*Sangfor*" } |
  ForEach-Object { Export-Certificate -Cert $_ -FilePath "$env:USERPROFILE\corp-root.cer" }

Then convert to PEM. If you have OpenSSL or Git's bundled one:

openssl x509 -inform DER -in corp-root.cer -out corp-root.pem

Keep this file somewhere stable, e.g. ~/.certs/corp-root.pem.

Fixing Python: certifi, requests, and pip

certifi is a vendored snapshot. The clean approach is to append the corporate root to a copy and set REQUESTS_CA_BUNDLE / SSL_CERT_FILE, not to mutate the installed certifi package (which breaks on every upgrade).

import os
import shutil
from pathlib import Path

import certifi

CORP_ROOT = Path.home() / ".certs" / "corp-root.pem"
BUNDLE = Path.home() / ".certs" / "combined-ca.pem"


def build_combined_bundle() -> Path:
    """Concatenate certifi's roots with the corporate root CA."""
    if not CORP_ROOT.exists():
        raise FileNotFoundError(f"missing corporate root: {CORP_ROOT}")

    with BUNDLE.open("wb") as out:
        out.write(Path(certifi.where()).read_bytes())
        out.write(b"\n")
        out.write(CORP_ROOT.read_bytes())

    return BUNDLE


if __name__ == "__main__":
    bundle = build_combined_bundle()
    os.environ["REQUESTS_CA_BUNDLE"] = str(bundle)
    os.environ["SSL_CERT_FILE"] = str(bundle)

    import requests

    r = requests.get("https://pypi.org/simple/", timeout=10)
    print(r.status_code, len(r.content))

Set the two environment variables globally (REQUESTS_CA_BUNDLE for requests, SSL_CERT_FILE for the stdlib ssl module and most other tools) and most Python breakage disappears. For pip specifically, either rely on SSL_CERT_FILE or set it explicitly:

pip config set global.cert ~/.certs/combined-ca.pem

If a package still fails, it's probably pinning. Check with:

import ssl, socket

host = "pypi.org"
ctx = ssl.create_default_context(cafile=str(BUNDLE))
with socket.create_connection((host, 443), timeout=10) as sock:
    with ctx.wrap_socket(sock, server_hostname=host) as ssock:
        print(ssock.getpeercert()["issuer"])

If that succeeds but the package fails, the package is doing its own pinning or bundling — you'll need its own override (e.g. NODE_EXTRA_CA_CERTS for Node, SSL_CERT_DIR for Go).

Fixing Git

Git for Windows ignores the Windows store by default. Two options: point it at the combined bundle, or tell it to use Schannel.

Option A — explicit bundle:

git config --global http.sslCAInfo ~/.certs/combined-ca.pem
git config --global http.sslBackend openssl

Option B — use the OS store (Windows only, often the cleanest):

git config --global http.sslBackend schannel

With Schannel, Git trusts whatever Windows trusts, including the aTrust root. This is my default on corporate Windows machines. On macOS, http.sslBackend doesn't exist; use the combined bundle approach there.

Verify:

GIT_CURL_VERBOSE=1 git ls-remote https://github.com/git/git HEAD 2>&1 | grep -i "issuer\|SSL certificate"

If you see the corporate issuer in the verbose output and the command succeeds, you're done.

SSH is usually untouched

aTrust typically proxies HTTP/HTTPS. SSH over port 22 or 443 is usually passed through or blocked outright, not MITM'd. If git clone git@github.com:... works, prefer it — SSH keys don't care about CA chains. If port 22 is blocked, GitHub's ssh.github.com:443 endpoint is the standard workaround:

Host github.com
  HostName ssh.github.com
  Port 443
  User git

Routing around blocked hosts, politely

Some hosts are blocked by policy, not by the proxy's technical limits. The wrong move is to spin up a personal VPN and tunnel everything — that's the fastest way to a security incident. The right move depends on the reason for the block.

If it's a category block (e.g. a package registry mirrors the security team hasn't approved): file a ticket. Internal mirrors of PyPI, npm, and Go modules are common in these environments, and using them is both faster and sanctioned. Ask for the internal index URL; it's usually already configured for some team.

If it's an allowlist model: the block is on the destination, and the proxy will happily pass traffic to approved hosts. Route around it by using an approved equivalent. For container images, that often means the company's Harbor or Artifactory instance rather than Docker Hub.

If you need to reach a host for legitimate work and it's genuinely missing from the allowlist: the polite ask is specific. "I need registry.npmjs.org for build reproducibility on project X; the internal mirror at npm.internal.corp is missing package Y as of version Z." Concrete, scoped, and easy for a security engineer to approve.

What you should not do: install a third-party VPN, use a public SOCKS proxy, or disable the aTrust agent. All three are detectable and all three turn a five-minute ticket into a meeting with your manager.

A note on certificate pinning

Some internal tools pin their own certs and will break the moment the corporate CA changes. If you maintain such a tool, don't pin the leaf; pin the corporate root's public key and make it configurable. A hardcoded SPKI hash is a time bomb.

import hashlib
import ssl

def spki_sha256(host: str, port: int = 443) -> str:
    """Return the base64 SHA-256 of the leaf's SubjectPublicKeyInfo."""
    der = ssl.get_server_certificate((host, port), ca_certs=str(BUNDLE)).encode()
    # In practice, parse with cryptography.x509 and hash the SPKI, not the whole cert.
    return hashlib.sha256(der).hexdigest()

The point isn't the exact hash — it's that pinning should be a config value, not a constant buried in source.

A short checklist

When something breaks behind aTrust:

Confirm the issuer with openssl s_client. If it's the corporate CA, you know the cause.
Build ~/.certs/combined-ca.pem from certifi plus the corporate root.
Export SSL_CERT_FILE and REQUESTS_CA_BUNDLE to that path.
For Git, set http.sslBackend=schannel on Windows or http.sslCAInfo everywhere.
Prefer SSH where possible; it sidesteps the whole problem.
For blocked hosts, use internal mirrors or file a scoped ticket — don't tunnel.

The interception isn't going away. Once you know where the trust anchors come from, most of the friction disappears.

More notes like this ship every week on this site.

Daily Picks

Long

Pair	Signal	Price	Take Profit	Stop Loss	R/R
SKYAI		$0.0416	$0.0433	$0.0406	1:1.6
RE		$0.4991	$0.5181	$0.4866	1:1.5

2 picks selected. Scanner runs every 15 minutes.

Health Checks for a Fleet of Scheduled Tasks

mattleeee — Tue, 06 Oct 2026 04:44:45 +0000

Nine unattended scheduled tasks is a fleet, whether you call it that or not. Once you pass three or four, checking taskschd.msc by hand stops being a habit and starts being a liability — you only look when something is visibly broken, which is exactly the failure mode you were trying to avoid by scheduling the work in the first place.

This post covers a small audit script that enumerates a set of tasks by name pattern, reads their state and last result, compares them against an expected schedule, and pushes an alert when something is wrong. It runs as a scheduled task itself, because a monitoring script that depends on you remembering to run it is not monitoring.

What the Task Scheduler actually exposes

The Task Scheduler COM API (Schedule.Service) is the authoritative source. The schtasks.exe text output is fine for humans and terrible for parsing — column widths shift, and localized Windows installs translate the field names. Go through COM.

import win32com.client

TASK_ENUM_HIDDEN = 1

def connect():
    service = win32com.client.Dispatch("Schedule.Service")
    service.Connect()
    return service

def iter_tasks(service, folder_path="\\"):
    folder = service.GetFolder(folder_path)
    for task in folder.GetTasks(TASK_ENUM_HIDDEN):
        yield task

Each IRegisteredTask gives you a handful of fields worth reading:

Field	Type	What it tells you
`Name`	str	Task name, without folder path
`Path`	str	Full path including folder, e.g. `\Quant\Quant_Factor`
`State`	int	`0` Unknown, `1` Disabled, `2` Queued, `3` Ready, `4` Running
`Enabled`	bool	Separate from state; a disabled task can still report `Ready` in some edge cases
`LastRunTime`	datetime	When the last run started
`LastTaskResult`	int	Exit code of the last completed run
`NextRunTime`	datetime	Next scheduled fire, `1899-12-30` if none

Two traps here. First, State and Enabled are not redundant. A task can be enabled but sitting in Disabled state if its trigger was disabled, or disabled at the task level while the state enum lags. Check both.

Second, LastTaskResult is only meaningful after a run completes. While a task is running, LastTaskResult still holds the previous run's value. If you alert on a non-zero result without checking State == 4, you will page yourself about a task that is mid-run and about to succeed.

The 1899-12-30 sentinel for "no next run" is a COM date artifact — it's the zero of the OLE automation date format. Treat anything before 1980 as "none".

Defining what "healthy" means

The script needs a table of expectations, because "task exists and ran once" is not a health check. Each task has a cadence, and a task that last ran three days ago when it should run hourly is broken even if its exit code was zero.

from dataclasses import dataclass
from datetime import timedelta

@dataclass(frozen=True)
class Expectation:
    pattern: str            # fnmatch pattern against task Path
    max_age: timedelta      # how stale LastRunTime may be
    allow_disabled: bool = False
    description: str = ""

EXPECTED = [
    Expectation("\\Quant\\Quant_*",        timedelta(hours=2),  description="quant pipeline"),
    Expectation("\\TeleAgent_*",           timedelta(minutes=30), description="telegram agent"),
    Expectation("\\Quant\\Quant_Nightly",  timedelta(hours=26), description="nightly batch"),
]

max_age should be generous enough to absorb a single missed cycle plus jitter, but not so generous that a task can silently skip twice. For an hourly task, two hours is a reasonable ceiling. For a nightly job, 26 hours covers a late start without hiding a full day of silence.

The allow_disabled flag matters for tasks you intentionally park. Without it, every disabled task shows up as a finding, and you learn to ignore the alerts — which defeats the point.

Classifying each task

The classification logic is where most naive audits go wrong. There are four distinct failure modes and they need different messages.

import fnmatch
from datetime import datetime, timezone

STATE_UNKNOWN, STATE_DISABLED, STATE_QUEUED, STATE_READY, STATE_RUNNING = range(5)

def classify(task, expectation, now):
    """Return (severity, reason) where severity is 'ok', 'warn' or 'fail'."""
    state = task.State
    enabled = task.Enabled
    last_result = task.LastTaskResult
    last_run = task.LastRunTime

    if not enabled or state == STATE_DISABLED:
        if expectation.allow_disabled:
            return "ok", "disabled by design"
        return "fail", "task is disabled"

    if state == STATE_RUNNING:
        # Don't judge LastTaskResult while a run is in flight.
        return "ok", "currently running"

    if last_result != 0:
        return "fail", f"last result {last_result:#010x}"

    if last_run is None or last_run.year < 1980:
        return "fail", "never ran"

    # COM returns naive local datetimes; compare in local time.
    age = now - last_run
    if age > expectation.max_age:
        hours = age.total_seconds() / 3600
        return "fail", f"stale, last run {hours:.1f}h ago"

    return "ok", f"ok, last run {age.total_seconds()/60:.0f}m ago"

The last_result non-zero check catches the common case: the task fired, the script inside exited with code 1, and nobody noticed because the scheduler doesn't care. 0x80070002 (file not found) and 0x80070005 (access denied) are the two you will see most often when someone moves a script or changes a service account.

Note the running-state short-circuit. A long-running task that takes 90 minutes will otherwise trip the staleness check on every audit pass, because LastRunTime only updates when a run starts. If your task runs longer than your max_age, you need to either raise max_age or check State == 4 before evaluating staleness — which is what the code above does.

Walking the folder tree

Tasks live in folders, and GetTasks only returns the tasks directly in the folder you asked for. If your tasks are organized under \Quant\ and \, you need to recurse.

def walk(service, folder_path="\\"):
    folder = service.GetFolder(folder_path)
    for task in folder.GetTasks(TASK_ENUM_HIDDEN):
        yield task
    for sub in folder.GetFolders(0):
        yield from walk(service, sub.Path)

GetFolders(0) returns immediate subfolders only; the recursion handles the rest. The 0 flag is a reserved bitmask that must be zero — passing anything else raises.

Matching tasks to expectations

Use fnmatch on the full Path, not the bare Name. Two folders can hold tasks with the same name, and matching on Name alone will silently pick the wrong one.

def audit(service, expectations, now):
    findings = []
    seen = set()

    for task in walk(service):
        for exp in expectations:
            if fnmatch.fnmatch(task.Path, exp.pattern):
                severity, reason = classify(task, exp, now)
                findings.append({
                    "path": task.Path,
                    "severity": severity,
                    "reason": reason,
                    "result": task.LastTaskResult,
                    "last_run": task.LastRunTime,
                })
                seen.add(exp.pattern)
                break

    # Expectations that matched nothing are themselves a finding.
    for exp in expectations:
        if exp.pattern not in seen:
            findings.append({
                "path": exp.pattern,
                "severity": "fail",
                "reason": "no matching task found",
                "result": None,
                "last_run": None,
            })

    return findings

The unmatched-expectation check is the one people skip, and it's the most valuable. A task that was deleted, renamed, or moved to a different folder disappears from the enumeration entirely — a naive audit reports "all clear" because there is nothing to complain about. The expectation table is what makes absence detectable.

The alert

PushPlus takes a token and a message body over HTTP POST and delivers to WeChat. Keep the alert short: a subject line that says how many tasks failed, and a body that lists only the failures. Nobody reads a wall of green checkmarks.

import requests

PUSHPLUS_TOKEN = "your-token-here"

def format_alert(findings):
    bad = [f for f in findings if f["severity"] != "ok"]
    if not bad:
        return None
    lines = [f"{len(bad)} task(s) need attention", ""]
    for f in bad:
        lines.append(f"{f['path']}")
        lines.append(f"  {f['reason']}")
    return "\n".join(lines)

def push(message):
    resp = requests.post(
        "https://www.pushplus.plus/send",
        json={
            "token": PUSHPLUS_TOKEN,
            "title": "Task fleet alert",
            "content": message,
            "template": "txt",
        },
        timeout=15,
    )
    resp.raise_for_status()

Send only when there is something to say. An alert channel that fires every hour with "all fine" trains you to swipe past it, and then the one real alert gets swiped too.

Wiring it together

def main():
    service = connect()
    now = datetime.now()
    findings = audit(service, EXPECTED, now)

    for f in findings:
        print(f"{f['severity']:4} {f['path']:40} {f['reason']}")

    message = format_alert(findings)
    if message:
        push(message)

if __name__ == "__main__":
    main()

Run it manually once, fix whatever it reports, then register it as a scheduled task. A 15-minute cadence is fine — the audit is cheap, and a shorter interval means you catch a failure closer to when it happened.

Why the audit must itself be scheduled

The obvious objection: "I'll just run this when I think about it." That is the same failure mode the audit exists to catch. If you remember to check, you don't need the check; if you forget, the check never runs. The only configuration where the audit has value is one where it runs without you.

That means the audit task is itself in the fleet, and it needs to be in the expectation table — a task that monitors nine tasks and is monitored by nobody is a single point of failure. Point an expectation at it with a max_age slightly larger than its own interval:

Expectation("\\Quant\\Audit_TaskFleet", timedelta(minutes=45), description="this audit"),

If the audit task dies, you stop receiving alerts — and the way you find out is that you stop receiving alerts, which is exactly the signal you would miss. There is no fully self-healing answer here, but you can reduce the blast radius: have the audit write a heartbeat row to a table or a file, and check that heartbeat from a second, independent mechanism (a cron job on a different host, a cloud function, anything that doesn't share the same scheduler). Two schedulers watching each other is more resilient than one scheduler watching itself.

What to watch over time

Three signals are worth tracking beyond pass/fail:

Run duration drift. A task that used to finish in 40 seconds and now takes 12 minutes is heading for a timeout. Log LastRunTime deltas and alert on a sustained increase, not a single spike.
Result code distribution. A task that alternates between 0 and 0x80070002 is flapping. Count non-zero results over a rolling window rather than checking only the most recent.
Stale-but-zero. The nastiest case: exit code 0, last run within tolerance, but the task is doing nothing because its input file is empty or its upstream dependency silently stopped producing. The scheduler can't see inside the process. If it matters, have the task itself emit a completion marker with a row count or a timestamp, and audit that marker instead of trusting the exit code.

The audit script is the first layer. It catches the failures the scheduler can see. The second layer — output validation — catches the ones it can't, and it belongs inside the task, not in the audit.

More notes like this ship every week on this site.

Daily Picks

Long

Pair	Signal	Price	Take Profit	Stop Loss	R/R
SKYAI		$0.0416	$0.0433	$0.0406	1:1.6
RE		$0.4991	$0.5181	$0.4866	1:1.5

2 picks selected. Scanner runs every 15 minutes.

Migrating a Python Automation Stack to a New Windows Server

mattleeee — Tue, 06 Oct 2026 04:44:07 +0000

Migrating a Python automation stack between Windows machines looks trivial until it isn't. Nine scheduled tasks, a blog pipeline, a trading system, and a few hundred state files moved from a Windows 11 dev box (UTF-8 default) to a fresh Windows Server 2016 instance (GBK default). The scripts were identical. The behavior was not. Here is the package approach that worked, including the encoding trap that cost the most time.

Why "just copy the folder" fails

The dev box had accumulated state that should not travel: logs, built site output, trade records, session cookies, and hundreds of JSON state files that referenced absolute paths from the old machine. Copying everything would have dragged stale data onto the server, and in the trading system's case, replayed old positions into a fresh account.

The other problem is that scheduled tasks store absolute paths. A task registered on C:\Users\dev\... will not resolve on a server where the user profile is C:\Users\Administrator\.... Re-registering tasks by hand across nine entries is a recipe for typos, so the migration needed to be scripted end to end.

The package layout

I built a single migration/ folder that could be copied to the new machine and run without editing:

migration/
  payload/            # the actual code, minus runtime data
  exclude.txt         # xcopy exclusion list
  install.bat         # one-click installer
  register_tasks.ps1  # re-registers all scheduled tasks
  verify.bat          # checks every task's action and last result
  requirements.txt

The payload/ directory is the source tree with runtime directories stripped out. Everything machine-specific is resolved at install time.

xcopy with an exclusion list

robocopy is the better tool for large trees, but xcopy with /EXCLUDE is simpler to reason about when you want a flat file list and predictable behavior on older Windows Server builds. The exclude file protects runtime data:

\logs\
\site\build\
\data\trades\
\data\cookies\
\state\*.tmp
\__pycache__\
*.pyc
.env

Then the copy is one line:

xcopy "%~dp0payload" "%USERPROFILE%\automation" /E /I /Y /EXCLUDE:%~dp0exclude.txt

/E copies empty directories, /I assumes the destination is a directory if it doesn't exist, /Y suppresses the overwrite prompt. The exclude file uses substring matching, so \logs\ catches any path segment named logs anywhere in the tree.

One caveat: xcopy's exclude matching is greedy and case-insensitive on Windows. If you have a source directory named Logs and a legitimate catalog directory, the pattern \logs\ will not match catalog because it requires the surrounding backslashes, but a bare logs pattern would. Keep the delimiters.

One-click installer that auto-detects paths

The installer resolves everything relative to %USERPROFILE% so the same batch file works on any account:

@echo off
setlocal

set "ROOT=%USERPROFILE%\automation"
set "PY=%LOCALAPPDATA%\Programs\Python\Python311\python.exe"

if not exist "%PY%" (
    echo Python not found at %PY%
    exit /b 1
)

echo Installing to %ROOT%
xcopy "%~dp0payload" "%ROOT%" /E /I /Y /EXCLUDE:%~dp0exclude.txt

if errorlevel 1 (
    echo Copy failed
    exit /b 1
)

"%PY%" -m pip install -r "%~dp0requirements.txt" --quiet
if errorlevel 1 (
    echo Dependency install failed
    exit /b 1
)

powershell -ExecutionPolicy Bypass -File "%~dp0register_tasks.ps1" -Root "%ROOT%"
if errorlevel 1 (
    echo Task registration failed
    exit /b 1
)

echo Install complete
endlocal

The %~dp0 prefix resolves to the directory containing the batch file, so the package can live anywhere. The installer fails fast at each stage instead of continuing with a broken environment.

Re-registering scheduled tasks

Nine tasks needed to move from the old machine to the new one. Rather than export XML and rewrite paths, I defined the tasks in a PowerShell array and used New-ScheduledTaskAction plus Register-ScheduledTask. This keeps the task definitions in source control and makes the paths parametric:

param(
    [Parameter(Mandatory=$true)]
    [string]$Root
)

$python = Join-Path $env:LOCALAPPDATA "Programs\Python\Python311\python.exe"

$tasks = @(
    @{ Name = "BlogPublish";   Script = "blog\publish.py";    Time = "06:00" },
    @{ Name = "BlogFetch";     Script = "blog\fetch.py";      Time = "06:30" },
    @{ Name = "TradeScan";     Script = "trading\scan.py";    Time = "09:00" },
    @{ Name = "TradeExecute";  Script = "trading\execute.py"; Time = "09:15" },
    @{ Name = "TradeReport";   Script = "trading\report.py";  Time = "16:00" },
    @{ Name = "StateBackup";   Script = "ops\backup.py";      Time = "23:00" },
    @{ Name = "HealthCheck";   Script = "ops\health.py";      Time = "00:00" },
    @{ Name = "LogRotate";     Script = "ops\rotate.py";      Time = "00:30" },
    @{ Name = "DailyDigest";   Script = "ops\digest.py";      Time = "07:00" }
)

foreach ($t in $tasks) {
    $scriptPath = Join-Path $Root $t.Script
    $action = New-ScheduledTaskAction `
        -Execute $python `
        -Argument "`"$scriptPath`"" `
        -WorkingDirectory (Split-Path $scriptPath)

    $trigger = New-ScheduledTaskTrigger -Daily -At $t.Time

    $settings = New-ScheduledTaskSettingsSet `
        -AllowStartIfOnBatteries `
        -DontStopIfGoingOnBatteries `
        -StartWhenAvailable `
        -ExecutionTimeLimit (New-TimeSpan -Hours 2)

    Register-ScheduledTask `
        -TaskName $t.Name `
        -Action $action `
        -Trigger $trigger `
        -Settings $settings `
        -User $env:USERNAME `
        -RunLevel Highest `
        -Force
}

Write-Host "Registered $($tasks.Count) tasks"

-StartWhenAvailable matters on a server that may be asleep or rebooting at the trigger time. -ExecutionTimeLimit prevents a hung script from blocking the next day's run. -Force makes the script idempotent, so re-running it after a code update just overwrites the existing task.

The encoding trap

This is where the migration nearly failed. The dev box ran Windows 11 with UTF-8 as the default code page. The server ran Windows Server 2016 with GBK (code page 936). The same Python scripts produced different output on each machine.

The symptoms were subtle. A blog post with a curly apostrophe rendered as â€™ on the server. A trading log line with a Chinese instrument name raised UnicodeEncodeError when redirected to a file. A JSON state file written on the dev box loaded fine on the server, but a state file written on the server and read back on the dev box returned mojibake.

The root cause: Python 3.7+ uses locale.getpreferredencoding() for open() when no encoding is specified, and sys.stdout on Windows uses the console code page unless redirected. On the dev box that was UTF-8. On the server it was GBK. Every unqualified open() and every print() was a latent bug.

The fix is to pin encoding everywhere. No exceptions.

Pinning file I/O

import json
from pathlib import Path

def read_json(path: Path) -> dict:
    with path.open("r", encoding="utf-8") as f:
        return json.load(f)

def write_json(path: Path, data: dict) -> None:
    tmp = path.with_suffix(path.suffix + ".tmp")
    with tmp.open("w", encoding="utf-8", newline="\n") as f:
        json.dump(data, f, ensure_ascii=False, indent=2)
    tmp.replace(path)

The newline="\n" matters when the same files are read by tools on other platforms. ensure_ascii=False keeps non-ASCII characters readable in the file instead of escaping them, which is fine as long as the encoding is pinned on both ends.

Pinning stdout

Redirected stdout on Windows inherits the console code page, which is why logging to a file behaved differently from logging to the terminal. Reconfigure the streams at process start:

import sys

def force_utf8_streams() -> None:
    for stream in (sys.stdout, sys.stderr):
        if hasattr(stream, "reconfigure"):
            stream.reconfigure(encoding="utf-8", errors="replace")

force_utf8_streams()

Call this before any imports that might log at module level. errors="replace" prevents a single bad byte from killing a long-running task; log the replacement character and move on.

Pinning subprocess output

Any subprocess.run that captures text inherits the same problem. Pass the encoding explicitly:

import subprocess

result = subprocess.run(
    ["git", "log", "-1", "--format=%s"],
    capture_output=True,
    text=True,
    encoding="utf-8",
    errors="replace",
    check=False,
)

Without encoding="utf-8", Python decodes the child process output using the locale code page. On the server that meant GBK decoding of UTF-8 bytes, which silently produced garbage in commit messages.

Pinning the environment

For belt and braces, set PYTHONUTF8=1 in the task environment. Python 3.7+ honors this and enables UTF-8 mode globally, which makes open() default to UTF-8 and sys.stdout use UTF-8 regardless of the console code page:

$env:PYTHONUTF8 = "1"

I set this in the scheduled task definition via the -Environment parameter on New-ScheduledTaskAction where supported, and as a system environment variable on the server otherwise. It does not replace explicit encoding arguments in code, but it catches anything I missed.

The verification bat

After install, verify.bat checks every task's action and last run result. This caught two tasks that had been registered with the wrong Python path and one that had never run because its trigger time had already passed on the install day:

@echo off
setlocal enabledelayedexpansion

set "FAIL=0"

for %%T in (
    BlogPublish BlogFetch TradeScan TradeExecute
    TradeReport StateBackup HealthCheck LogRotate DailyDigest
) do (
    for /f "delims=" %%A in ('powershell -NoProfile -Command ^
        "(Get-ScheduledTask -TaskName '%%T').Actions[0].Execute"') do (
        set "EXEC=%%A"
    )
    for /f "delims=" %%R in ('powershell -NoProfile -Command ^
        "(Get-ScheduledTaskInfo -TaskName '%%T').LastTaskResult"') do (
        set "RESULT=%%R"
    )

    echo %%T  exec=!EXEC!  last=!RESULT!

    if "!EXEC!"=="" (
        echo   MISSING ACTION
        set "FAIL=1"
    )
    if not "!RESULT!"=="0" if not "!RESULT!"=="267009" (
        echo   NONZERO RESULT
        set "FAIL=1"
    )
)

if "!FAIL!"=="1" (
    echo Verification failed
    exit /b 1
)
echo All tasks verified
endlocal

267009 is the "task is currently running" result code, which is not a failure. Any other nonzero LastTaskResult gets flagged. The action check catches the case where a task exists but points at a Python interpreter that was uninstalled or moved.

What I would do differently

Two things. First, set PYTHONUTF8=1 on both machines before writing any code, not after the first mojibake appears. Second, treat the exclude list as part of the source tree and review it whenever a new runtime directory is added; a stale exclude list is how you accidentally ship a trade record to a new machine.

The migration itself ran in about ten minutes once the package was built. Building the package took a day, most of it spent on the encoding audit. The scripts were never the problem. The environment was.

More notes like this ship every week on this site.

Daily Picks

Long

Pair	Signal	Price	Take Profit	Stop Loss	R/R
SKYAI		$0.0416	$0.0433	$0.0406	1:1.6
RE		$0.4991	$0.5181	$0.4866	1:1.5

2 picks selected. Scanner runs every 15 minutes.

Exit Code 267009: Debugging CMD Encoding Corruption

mattleeee — Tue, 06 Oct 2026 04:38:27 +0000

Scheduled tasks fail in the dark. There is no console, no visible stderr, and often no useful log — just a status code in Task Scheduler's "Last Run Result" column. When that code is 267009, the usual suspects (missing file, wrong working directory, permissions) all look wrong, because the task itself is fine. What is broken is the encoding of the launcher script.

This post walks through a real failure: a .bat file saved as UTF-8 containing Chinese comments and paths, executed by cmd.exe under a GBK system codepage. We'll reproduce it, decode the exit code, and lay out a checklist for batch files that need to survive being copied between machines.

The Symptom

A scheduled task ran a small batch launcher:

@echo off
REM 启动数据抓取流程
cd /d "C:\项目\抓取"
python fetch.py --output "C:\项目\数据\out.csv"

The task's "Last Run Result" showed 267009 (0x41301). The Python script ran fine when invoked manually from a terminal. Under the scheduler, it never produced output and never wrote a log line. Nothing in Event Viewer beyond the exit code.

That 0x41301 is the give-away. So the first move is not to touch the script — it's to learn what the code actually means.

Reading Opaque CMD Exit Codes

Windows exit codes are 32-bit values. Task Scheduler displays them in decimal and hex. When a code looks arbitrary, check the hex form against the documented HRESULT/Win32 ranges:

0x00000000–0x0000FFFF: normal Win32 error codes (ERROR_*).
0x80000000–0xFFFFFFFF: HRESULT-style values, usually from COM or the shell.
0x41301 specifically: SCHED_S_TASK_RUNNING — the task is still running.

That last one is the trap. 267009 is not an error at all. It is SCHED_S_TASK_RUNNING, a success-severity status meaning the task was still executing when the query was made. Task Scheduler reports it when:

The task genuinely takes a long time and the status snapshot was taken mid-run.
The launched process never exits — it hangs, usually waiting on input or blocked on a parse error.

In this case it was case 2: cmd.exe hit a malformed line, dropped into an interactive prompt, and sat there forever. The scheduled task "ran," produced no output, and never completed. 267009 was the only clue.

A quick way to normalize and classify codes before you go down a rabbit hole:

import ctypes
from ctypes import wintypes

def describe_exit_code(code: int) -> str:
    """Classify a Windows exit code into its documented family."""
    code &= 0xFFFFFFFF  # normalize to unsigned 32-bit

    if code == 0:
        return "success"
    if code == 0x41301:
        return "SCHED_S_TASK_RUNNING (task still executing or hung)"
    if code < 0x10000:
        msg = ctypes.FormatError(code).strip()
        return f"Win32 error {code} ({msg})" if msg else f"Win32 error {code}"
    if code >= 0x80000000:
        return f"HRESULT 0x{code:08X} (facility {code >> 16 & 0x1FFF})"
    return f"unknown 0x{code:08X}"

for c in (0, 267009, 2, 0x80070002, 1):
    print(f"{c:>10} 0x{c & 0xFFFFFFFF:08X}  {describe_exit_code(c)}")

Output:

         0 0x00000000  success
    267009 0x00041301  SCHED_S_TASK_RUNNING (task still executing or hung)
         2 0x00000002  Win32 error 2 (The system cannot find the file specified.)
  2147942402 0x80070002  HRESULT 0x80070002 (facility 7)
         1 0x00000001  Win32 error 1 (Incorrect function.)

Two rules of thumb fall out of this:

If you see 0x41301, the task is not failing — it is not finishing. Go look for a hang, not a crash.
0x8007xxxx is a Win32 error wrapped in an HRESULT; strip the 0x8007 prefix and you get the plain ERROR_* code. 0x80070002 is just "file not found."

Reproducing the Corruption

The hang was caused by encoding, not logic. Here is the minimal reproduction.

Take a .bat file with non-ASCII content — Chinese comments, a Chinese path, or both — and save it as UTF-8 without BOM. On a machine whose active code page is GBK (chcp returns 936), run it:

@echo off
REM 抓取任务
echo starting
cd /d "C:\项目\数据"
echo done

Under GBK, cmd.exe reads each byte of the UTF-8 sequence as a separate GBK character. UTF-8 encodes 项 as E9 A1 B9. GBK sees E9 A1 as one character and B9 as the start of another, consuming the next byte — which may be a quote, a newline, or part of echo. The parse decouples from the intended text.

The failure modes are consistent and annoying:

A REM comment line absorbs the following line, silently dropping a command.
A quoted path loses its closing quote, so cmd.exe keeps reading.
The script reaches end-of-file mid-command and cmd.exe falls back to reading from stdin — which, under a scheduled task, is an empty pipe. It blocks. Forever. Hence 267009.

You can reproduce this deterministically without the scheduler:

import subprocess

# A launcher with a UTF-8 Chinese comment, no BOM.
bat_utf8 = (
    "@echo off\r\n"
    "REM 抓取任务\r\n"
    "echo starting\r\n"
    "echo done\r\n"
)

with open("probe_utf8.bat", "wb") as f:
    f.write(bat_utf8.encode("utf-8"))

# Run it under a GBK console code page, capturing stdin from an empty pipe
# so a hang is observable as a timeout rather than a block.
try:
    proc = subprocess.run(
        ["cmd.exe", "/c", "chcp 936 >nul & probe_utf8.bat"],
        stdin=subprocess.DEVNULL,
        capture_output=True,
        timeout=5,
    )
    print("returncode:", proc.returncode)
    print("stdout:", proc.stdout.decode("gbk", "replace"))
except subprocess.TimeoutExpired:
    print("HANG: cmd.exe is waiting on stdin (this is the 267009 case)")

On a GBK system this prints HANG. The same content saved as GBK runs cleanly. That difference is the whole bug.

Why UTF-8 "Works" Until It Doesn't

Modern Windows 10/11 still defaults cmd.exe to the OEM code page — 936 (GBK) on Simplified Chinese installs, 437/850 on US installs — unless you enable the "Beta: Use Unicode UTF-8 for worldwide language support" option. Editors, by contrast, default to UTF-8. So the file is written in one encoding and read in another. It works on the developer's machine only if that machine happens to be configured the same way, which is exactly the kind of coincidence that breaks when you copy the file to a server or a VM.

The scheduled task made it worse: the interactive terminal had a code page that happened to tolerate the file, while the scheduler ran cmd.exe with no console and the system default. Same file, different reader.

The Fixes

Three changes, applied together, kill the class of bug. None of them require the "UTF-8 beta" checkbox.

1. Keep `.bat` files pure ASCII

The most robust launcher contains only ASCII bytes: no comments in Chinese, no non-ASCII paths inline. If you need a localized comment, put it in a .txt sidecar or in the Python script, not in the batch file. cmd.exe has no encoding problem with ASCII because ASCII is a subset of nearly every code page in use.

2. If non-ASCII is unavoidable, save as GBK and use CRLF

If a path genuinely must appear literally, encode the file in the system's OEM code page (GBK on Chinese Windows) rather than UTF-8, and terminate lines with \r\n. cmd.exe's parser treats a bare \n inconsistently across versions; CRLF is what it expects. Do not add a UTF-8 BOM — cmd.exe will try to execute the BOM bytes as part of the first command.

def write_bat(path: str, text: str, encoding: str = "gbk") -> None:
    """Write a .bat file cmd.exe can parse.

    - encoding: OEM code page of the target machine (gbk, cp437, ...)
    - line endings forced to CRLF
    - no BOM
    """
    normalized = text.replace("\r\n", "\n").replace("\r", "\n")
    data = normalized.replace("\n", "\r\n").encode(encoding, errors="strict")
    with open(path, "wb") as f:
        f.write(data)  # never a BOM

errors="strict" is deliberate: if a character cannot be represented in GBK, you want the write to fail loudly rather than silently replace it with ? and produce a path that doesn't exist.

3. Use 8.3 short paths for non-ASCII directories

Even with correct encoding, non-ASCII paths are fragile across tools, quoting layers, and remote sessions. Windows exposes an ASCII-only 8.3 alias for every path. Resolve it once and put the short form in the launcher:

import ctypes
from ctypes import wintypes

def short_path(long_path: str) -> str:
    """Return the 8.3 short form of a path (ASCII-only)."""
    GetShortPathNameW = ctypes.windll.kernel32.GetShortPathNameW
    GetShortPathNameW.argtypes = [wintypes.LPCWSTR, wintypes.LPWSTR, wintypes.DWORD]
    GetShortPathNameW.restype = wintypes.DWORD

    buf = ctypes.create_unicode_buffer(260)
    n = GetShortPathNameW(long_path, buf, len(buf))
    if n == 0:
        raise OSError(ctypes.get_last_error(), f"no short path for {long_path!r}")
    if n > len(buf):
        buf = ctypes.create_unicode_buffer(n)
        GetShortPathNameW(long_path, buf, n)
    return buf.value

print(short_path(r"C:\项目\数据"))  # e.g. C:\A1B2~1\A3C4~1

The generated launcher is now pure ASCII:

@echo off
cd /d "C:\A1B2~1\A3C4~1"
python fetch.py --output "C:\A1B2~1\A3C4~1\out.csv"

8.3 name generation can be disabled per-volume (fsutil 8dot3name query), so verify it is enabled on volumes where you rely on this. If it is off, fall back to fix 1 or 2.

A Checklist for Portable `.bat` Files

Before a batch file goes into a scheduled task or gets copied to another machine, run through this:

Bytes are ASCII-only. Verify with a script, not by eye — invisible non-ASCII is easy to miss.
Line endings are CRLF. No lone \n, no lone \r.
No BOM. The first three bytes must not be EF BB BF.
No non-ASCII paths inline. Use 8.3 short paths or pass paths as arguments from a caller that handles Unicode.
Encoding matches the target. If ASCII is impossible, the file is GBK (or the target's OEM code page), not UTF-8.
chcp is explicit if needed. If you must change the code page, do it as the first line and document why.
No interactive fallback. End with an explicit exit /b %ERRORLEVEL% so cmd.exe never drops into a prompt on parse failure.
Test under the scheduler's environment. A terminal session and a scheduled task are not the same reader.

A guard you can run in CI or as a pre-commit hook:

def audit_bat(path: str) -> list[str]:
    problems = []
    raw = open(path, "rb").read()

    if raw.startswith(b"\xef\xbb\xbf"):
        problems.append("has UTF-8 BOM")
    if b"\r\n" not in raw:
        problems.append("no CRLF line endings")
    if raw.replace(b"\r\n", b"").count(b"\n") or raw.replace(b"\r\n", b"").count(b"\r"):
        problems.append("mixed or bare line endings")
    try:
        raw.decode("ascii")
    except UnicodeDecodeError as e:
        problems.append(f"non-ASCII byte at offset {e.start}: {raw[e.start:e.start+1]!r}")
    return problems

for p in ("probe_utf8.bat", "launcher.bat"):
    issues = audit_bat(p)
    print(p, "OK" if not issues else issues)

Run it against the original failing launcher and it reports the non-ASCII byte immediately — the same byte cmd.exe choked on.

Takeaway

267009 is not an error code in the usual sense; it is SCHED_S_TASK_RUNNING, and it most often means cmd.exe is blocked on a parse failure that turned into a stdin read. The parse failure came from an encoding mismatch between the file (UTF-8) and the reader (GBK). Fix the file's bytes — ASCII where possible, GBK where not, CRLF always, no BOM — and use 8.3 short paths for anything non-ASCII. The exit code disappears because there is nothing left for cmd.exe to misread.

More notes like this ship every week on this site.

Daily Picks

Long

Pair	Signal	Price	Take Profit	Stop Loss	R/R
SKYAI		$0.0416	$0.0433	$0.0406	1:1.6
RE		$0.4991	$0.5181	$0.4866	1:1.5

2 picks selected. Scanner runs every 15 minutes.

Staggering Nine Scheduled Tasks to Eliminate Resource Contention

mattleeee — Tue, 06 Oct 2026 04:37:50 +0000

A market-signal job ran at HH:00. A data-refresh job ran at HH:00. Both touched the same JSON files and both wanted the CPU at the same instant. The result was predictable: PermissionError on Windows file locks, occasional half-written JSON that the signal job then parsed as empty, and a scheduler log full of overlapping runs.

The fix wasn't a faster machine or a rewrite. It was a timetable. This post walks through how I inventoried the triggers, decided on offsets, handled the shared-state file locks, and ended up with a single page that shows the whole fleet.

Why top-of-hour schedules collide

Cron-style schedulers are seductive because 0 * * * * reads like "do the thing." It doesn't say "do the thing when nothing else is doing the thing." Every job author independently picks round numbers, and round numbers cluster. If nine jobs all live on 0 * * * * or 0 4 * * *, they share a trigger boundary whether they need to or not.

Two failure modes show up immediately:

CPU contention. Two CPU-bound jobs start together, each takes roughly twice as long as it would alone, and the overlap window grows.
File-lock collisions. On Windows, opening a file for write while another process holds it raises PermissionError or silently truncates depending on the flags. Two writers to the same JSON file is a data-corruption bug waiting for a slow disk.

The signal job and the refresh job hit both. The signal job read signals.json and wrote positions.json. The refresh job wrote signals.json. When they ran together, the signal job sometimes read an empty file mid-write and produced an empty position list — which then propagated downstream.

Step 1: Inventory every trigger

You can't stagger what you haven't listed. I dumped every scheduled entry into one place. On Windows, the Task Scheduler is scriptable through schtasks or the COM interface. The COM path gives structured data without parsing text.

import win32com.client
from datetime import datetime

def list_scheduled_tasks(folder_path="\\"):
    """Walk a Task Scheduler folder and return trigger summaries."""
    scheduler = win32com.client.Dispatch("Schedule.Service")
    scheduler.Connect()
    root = scheduler.GetFolder(folder_path)
    tasks = root.GetTasks(1)  # 1 = include hidden

    rows = []
    for task in tasks:
        definition = task.Definition
        for trigger in definition.Triggers:
            rows.append({
                "name": task.Name,
                "enabled": task.Enabled,
                "type": trigger.Type,
                "start": getattr(trigger, "StartBoundary", None),
                "repetition": getattr(trigger, "Repetition", None),
            })
    return rows

for row in list_scheduled_tasks():
    print(f"{row['name']:<40} {row['start']}")

That gives you names and start boundaries. It doesn't tell you what the jobs do — which is the part that matters for spacing. So I added a second pass: for each task, record the action (the script path) and a manual tag for CPU-bound vs I/O-bound.

TAGS = {
    "MarketSignalHourly":   {"kind": "cpu", "reads": ["signals.json"],
                             "writes": ["positions.json"]},
    "DataRefreshHourly":    {"kind": "io",  "reads": [],
                             "writes": ["signals.json"]},
    "NightlyMaintenance":   {"kind": "cpu", "reads": ["positions.json"],
                             "writes": ["positions.json", "audit.log"]},
    "BlogPublish":          {"kind": "io",  "reads": ["posts/*.md"],
                             "writes": ["site/"]},
    "BlogPublishDE":        {"kind": "io",  "reads": ["posts/de/*.md"],
                             "writes": ["site/de/"]},
}

The reads/writes sets are the important column. Two jobs that share a write target must never overlap, regardless of how cheap they are.

Step 2: Rules of thumb for spacing

Once you have the inventory, spacing is mostly mechanical. Three rules cover almost everything:

CPU-bound jobs need a full estimated runtime of separation

If a job takes 12 minutes of CPU on a busy box, don't schedule the next CPU-bound job 5 minutes later. Measure the p95 runtime, not the median. The overlap you're trying to avoid is the tail, not the typical case.

I/O-bound jobs can share a slot if they touch different files

Two jobs that read disjoint file sets and write to disjoint outputs can run together. Disk throughput is usually fine for a handful of readers. The moment they write the same path, they can't share.

Never let a writer and a reader of the same file overlap

This is the hard rule. Even a fast reader can catch a partial write. Either serialize them or make the write atomic (see the next section).

Applying these to the fleet:

Job	Kind	Runtime (p95)	Constraint
MarketSignalHourly	cpu	14 min	writes `positions.json`
DataRefreshHourly	io	6 min	writes `signals.json`
NightlyMaintenance	cpu	40 min	writes `positions.json`, `audit.log`
BlogPublish	io	3 min	writes `site/`
BlogPublishDE	io	3 min	writes `site/de/`

The signal job and the refresh job both ran at HH:00. The signal job reads signals.json; the refresh job writes it. That's the writer/reader overlap, and it's the one that corrupted data. The fix: move the refresh job off the top of the hour so it finishes before the signal job starts, or at least so the write window doesn't intersect the read window.

I moved the refresh to HH:45 — 15 minutes before the signal job. The refresh takes 6 minutes at p95, so it's done by HH:51, leaving a 9-minute buffer. That buffer absorbs the tail.

Step 3: The offset schedule

Here's the timetable that emerged. The signal job stays at HH:00 because downstream consumers expect it there. Everything else moves around it.

HH:00  MarketSignalHourly      (cpu, 14 min)
HH:45  DataRefreshHourly       (io, 6 min)  -> completes before next HH:00
04:30  NightlyMaintenance      (cpu, 40 min) -> done by 05:10
09:00  BlogPublish             (io, 3 min)
10:30  BlogPublishDE           (io, 3 min)

The daily jobs sit in a quiet window. 04:30 is after the overnight batch and before the morning signal jobs ramp up. 09:00 and 10:30 are separated by 90 minutes, which is overkill for two 3-minute jobs, but the gap gives room for retries and for the English site build to finish before the German one starts — they share a site/ parent directory, so a partial build on one side can confuse a deploy that scans the whole tree.

The HH:45 offset is the one that matters most. It's not a round number, which is the point. Round numbers cluster; odd offsets don't.

Step 4: Handle the shared state file

Staggering reduces overlap but doesn't eliminate it. A job that overruns its window will still collide with the next one. The durable fix is atomic writes plus a lock file.

On Windows, os.replace is atomic on the same volume. Write to a temp file, then replace the target. Readers either see the old file or the new one, never a partial write.

import json
import os
import tempfile
from pathlib import Path

def atomic_write_json(path: Path, payload: dict) -> None:
    """Write JSON so readers never observe a partial file."""
    path = Path(path)
    fd, tmp_name = tempfile.mkstemp(
        dir=path.parent, prefix=path.name + ".", suffix=".tmp"
    )
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as fh:
            json.dump(payload, fh)
            fh.flush()
            os.fsync(fh.fileno())
        os.replace(tmp_name, path)  # atomic on same volume
    except BaseException:
        os.unlink(tmp_name)
        raise

That handles readers. For writers, add a lock so two processes can't both decide to write. A simple lock file with an exclusive create works on Windows and POSIX:

import time
from contextlib import contextmanager

@contextmanager
def file_lock(path: Path, timeout: float = 30.0):
    """Exclusive lock via atomic create. Windows-safe."""
    lock = Path(str(path) + ".lock")
    deadline = time.monotonic() + timeout
    while True:
        try:
            fd = os.open(lock, os.O_CREAT | os.O_EXCL | os.O_WRONLY)
            os.close(fd)
            break
        except FileExistsError:
            if time.monotonic() > deadline:
                raise TimeoutError(f"lock held: {lock}")
            time.sleep(0.5)
    try:
        yield
    finally:
        lock.unlink(missing_ok=True)

Then any job that writes signals.json wraps its write:

with file_lock(SIGNALS_PATH):
    data = load_json(SIGNALS_PATH)
    data.update(new_signals)
    atomic_write_json(SIGNALS_PATH, data)

The lock serializes writers; the atomic replace protects readers. Together they make the schedule forgiving of small overruns. Staggering is still the first line of defense — locks are the second.

One caveat: a stale lock file from a crashed process will block everything. Either add a PID check or a timestamp inside the lock file and expire it. For a small fleet, a timestamp is enough:

def lock_is_stale(lock: Path, max_age: float = 300.0) -> bool:
    try:
        age = time.time() - lock.stat().st_mtime
    except FileNotFoundError:
        return False
    return age > max_age

Check lock_is_stale before raising TimeoutError, and unlink if it's old.

Step 5: One page for the whole fleet

The inventory script and the tags table are useful, but they drift. The thing that actually kept the schedule sane was a single generated page — a timetable that lists every job, its trigger, its kind, and its file dependencies. I regenerate it from the source of truth (the task definitions plus the tags dict) and commit it.

def render_timetable(tasks, tags) -> str:
    lines = ["| Job | Trigger | Kind | Reads | Writes |", "|---|---|---|---|---|"]
    for task in sorted(tasks, key=lambda t: t["start"]):
        tag = tags.get(task["name"], {})
        lines.append(
            f"| {task['name']} | {task['start']} | {tag.get('kind', '?')} "
            f"| {', '.join(tag.get('reads', [])) or '-'} "
            f"| {', '.join(tag.get('writes', [])) or '-'} |"
        )
    return "\n".join(lines)

The value isn't the table itself — it's that adding a job forces you to fill in the reads/writes columns. That's the moment you notice two jobs want the same file at the same time. The timetable makes the collision visible before it happens, instead of after a corrupted JSON file shows up in the logs.

What changed

After the rebalance, the PermissionError entries stopped. The signal job's parse failures went to zero because signals.json is now written by a job that finishes 9 minutes before the signal job reads it, and even if it overruns, the atomic write means the signal job reads a complete file. The nightly maintenance window moved off the top of the hour, so it no longer competes with the first morning signal run.

None of this required new hardware. It required looking at the schedule as a system instead of a list of independent jobs, measuring the tail runtimes, and picking offsets that don't cluster on round numbers.

More notes like this ship every week on this site.

Daily Picks

Long

Pair	Signal	Price	Take Profit	Stop Loss	R/R
SKYAI		$0.0416	$0.0433	$0.0406	1:1.6
RE		$0.4991	$0.5181	$0.4866	1:1.5

2 picks selected. Scanner runs every 15 minutes.

When GitHub Is Blocked but You Still Need to Push

mattleeee — Tue, 06 Oct 2026 04:29:02 +0000

Corporate firewalls are rarely all-or-nothing. They are usually the result of a proxy policy, a TLS inspection rule, or a category filter that someone applied years ago and forgot about. That is why you can end up in a situation where github.com:443 is silently dropped, while api.github.com and raw.githubusercontent.com answer requests without complaint.

I ran into this on a Windows build box inside a restricted network. git push hung until timeout. git clone over HTTPS failed. But curl.exe https://api.github.com returned a JSON payload in under a second. The deployment pipeline needed to publish a handful of generated files to a repository, and the network had effectively amputated the git transport while leaving the REST API intact.

This post covers how to diagnose that split, how to push files through the GitHub Contents API instead of git, and why the HTTP-only path is worth keeping around even on networks where git works fine.

Probing the firewall with curl.exe

Before writing any Python, establish exactly which endpoints survive. On Windows 10 and 11, curl.exe ships with the OS, so you do not need to install anything or fight with the PowerShell curl alias (which is actually Invoke-WebRequest and behaves differently).

Test the three hosts that matter:

curl.exe -sS -o NUL -w "%{http_code} %{time_total}s\n" https://github.com
curl.exe -sS -o NUL -w "%{http_code} %{time_total}s\n" https://api.github.com
curl.exe -sS -o NUL -w "%{http_code} %{time_total}s\n" https://raw.githubusercontent.com

-o NUL discards the body, -w prints the status code and elapsed time. On the blocked network, the first command sat there for roughly 20 seconds and exited with code 28 (operation timed out). The other two returned 200 in a few hundred milliseconds.

A timeout is more informative than a 403. A 403 means something answered — a proxy, a filter appliance — and you can sometimes negotiate with it. A timeout means the connection was dropped or blackholed, which usually points to a routing or SNI-based block that you cannot talk your way out of.

Confirm the git transport specifically:

git ls-remote https://github.com/your-org/your-repo.git

If that hangs while curl.exe to api.github.com succeeds, you have confirmed the split: git's smart HTTP protocol talks to github.com, the API talks to api.github.com, and only one of them is reachable.

One caveat worth checking early: raw.githubusercontent.com serves file content but does not accept writes. It is useful for reads and for verifying that a push landed, but every write goes through api.github.com.

Authenticating without a browser

The Contents API needs a token. A classic personal access token with repo scope works, or a fine-grained token with Contents: Read and write on the target repository. Put it in an environment variable rather than the script:

$env:GITHUB_TOKEN = "ghp_..."

Then verify the token and check your rate limit budget in one call:

curl.exe -sS -H "Authorization: Bearer $env:GITHUB_TOKEN" `
  -H "Accept: application/vnd.github+json" `
  https://api.github.com/rate_limit

The response includes resources.core.limit, normally 5000, and remaining. That number is per hour, per authenticated token. Unauthenticated requests get 60 per hour from a shared IP pool, which is useless for any real deployment — always authenticate.

The Contents API, step by step

The Contents API models a repository file as a resource at:

/repos/{owner}/{repo}/contents/{path}

A write is a two-step dance, because the API is optimistic-concurrency aware. You cannot just PUT a file and hope; if the file already exists, you must supply the current blob SHA so the server knows you are updating the version you think you are updating.

Step 1: GET the current SHA

import base64
import os
import time

import requests

API = "https://api.github.com"
OWNER = "your-org"
REPO = "your-repo"
BRANCH = "main"
TOKEN = os.environ["GITHUB_TOKEN"]

HEADERS = {
    "Authorization": f"Bearer {TOKEN}",
    "Accept": "application/vnd.github+json",
    "X-GitHub-Api-Version": "2022-11-28",
}


def get_file_sha(path: str) -> str | None:
    url = f"{API}/repos/{OWNER}/{REPO}/contents/{path}"
    r = requests.get(url, headers=HEADERS, params={"ref": BRANCH}, timeout=30)
    if r.status_code == 404:
        return None
    r.raise_for_status()
    return r.json()["sha"]

404 means the file does not exist yet — that is a create, not an update. Anything else non-2xx should raise, because silently continuing on a 401 or 403 just produces confusing failures later.

Step 2: PUT the base64 content

def put_file(path: str, content: bytes, message: str, sha: str | None) -> dict:
    url = f"{API}/repos/{OWNER}/{REPO}/contents/{path}"
    payload = {
        "message": message,
        "content": base64.b64encode(content).decode("ascii"),
        "branch": BRANCH,
    }
    if sha is not None:
        payload["sha"] = sha

    r = requests.put(url, headers=HEADERS, json=payload, timeout=30)
    r.raise_for_status()
    return r.json()

The content field must be base64 with no line breaks. Python's base64.b64encode produces exactly that. If you are reading a text file, encode it to UTF-8 first — do not pass a str to b64encode, it will raise.

Step 3: One commit per file, with a sleep

The Contents API commits one file per request. There is no multi-file atomic commit through this endpoint; for that you would need the Git Data API (blobs, trees, commits, refs), which is more calls, not fewer. For a deployment that publishes a handful of generated artifacts, one commit per file is fine and much simpler to reason about.

def push_files(files: dict[str, bytes], message_prefix: str) -> None:
    for path, content in files.items():
        sha = get_file_sha(path)
        verb = "Update" if sha else "Add"
        result = put_file(path, content, f"{verb} {path}", sha)
        commit_sha = result["commit"]["sha"][:7]
        print(f"{verb.lower():>6} {path} -> {commit_sha}")
        time.sleep(0.4)

The 0.4 second sleep is not superstition. GitHub's secondary rate limits are separate from the 5000/hour primary budget and are triggered by bursty write patterns. A tight loop of PUTs to the same repository is exactly the shape that trips them. Four tenths of a second keeps you under the radar while still pushing a few hundred files in a couple of minutes. If you are pushing thousands of files, raise it or batch through the Git Data API instead.

A 409 response on the PUT means the SHA you supplied is stale — someone else committed between your GET and your PUT. The correct handling is to re-GET and retry once:

def put_file_with_retry(path, content, message, attempts=3):
    for i in range(attempts):
        sha = get_file_sha(path)
        try:
            return put_file(path, content, message, sha)
        except requests.HTTPError as e:
            if e.response.status_code == 409 and i < attempts - 1:
                time.sleep(1.0)
                continue
            raise

Do not loop forever on 409. If two processes are fighting over the same file, retrying just amplifies the conflict.

Verifying the push without git

After the PUTs, confirm the content actually landed. raw.githubusercontent.com is reachable on this network and serves the file at the branch tip:

def verify(path: str, expected: bytes) -> bool:
    url = f"https://raw.githubusercontent.com/{OWNER}/{REPO}/{BRANCH}/{path}"
    r = requests.get(url, timeout=30)
    r.raise_for_status()
    return r.content == expected

There is a caching layer in front of raw.githubusercontent.com with a short TTL — usually a minute or two. If verification fails immediately after a push, wait and retry before assuming the push failed. The commit itself is authoritative; check GET /repos/{owner}/{repo}/commits?path={path} if you need certainty faster than the CDN refreshes.

Why this path is worth keeping

The obvious use case is a network that blocks github.com but not the API. That is what prompted this work. But the HTTP-only push path has a second, less obvious value: it works when git itself is broken.

I have hit all of these on build machines:

A git binary too old to negotiate the current TLS or protocol version, with no admin rights to upgrade it.
A corrupted ~/.gitconfig or credential helper that makes every git invocation fail before it reaches the network.
A container image where the git install is stripped down and git push fails on missing helpers.
A CI runner with a read-only filesystem except for a scratch directory, where git's lock files cannot be written.

In all of those cases, a Python script using requests and a token pushes files just fine, because it depends on nothing but the standard library and an HTTP client. The API surface is stable and versioned via the X-GitHub-Api-Version header. There is no local state to corrupt.

The trade-off is real: one commit per file, no atomic multi-file changes, no branches created in a single call, and a hard 100 MB limit per file (the API rejects blobs over that size). For deployment artifacts — config files, generated reports, small data files, documentation — none of that matters. For a large repository with binary assets, stick with git and fix the network instead.

Operational notes

Keep the token out of the script and out of logs. requests will happily include it in an exception traceback if you print the request; log only the URL and status code.

Set an explicit timeout on every call. The default is no timeout, and on a flaky network that means a hung process rather than a clean failure.

If you are pushing to a protected branch, the token's identity must be allowed to bypass the protection or have the right to push. The API enforces branch protection the same way git does, and a 422 on the PUT often means a required status check or review is missing, not that your payload is malformed. Read the message field in the response body — it usually says which rule blocked you.

Finally, keep the probe script. Networks change. The day github.com comes back, git push is still the better tool, and you will want to know which path is live without re-deriving it from scratch.

More notes like this ship every week on this site.

Daily Picks

Long

Pair	Signal	Price	Take Profit	Stop Loss	R/R
SKYAI		$0.0416	$0.0433	$0.0406	1:1.6
RE		$0.4991	$0.5181	$0.4866	1:1.5

2 picks selected. Scanner runs every 15 minutes.

Taming BLAS Threads in Scheduled Python Jobs

mattleeee — Tue, 06 Oct 2026 04:27:49 +0000

Scheduled Python jobs on a shared box have a nasty habit of fighting each other. You write a script that runs in four seconds on your laptop, deploy it to a Windows Server with eight cores, and now it takes forty — but only when the nightly batch is also running. The culprit is usually not your code. It is the BLAS thread pool underneath numpy.

What Actually Spawns Threads

When you import numpy, the wheel you installed is linked against a BLAS implementation: OpenBLAS for the standard numpy wheels on PyPI, MKL for numpy built against Intel's math library (common with conda, or pip install mkl setups). Both ship multithreaded kernels for dgemm, sgemm, and friends. By default they read the machine's core count and spin up a pool of that size.

So import numpy on an 8-core server means eight OpenBLAS worker threads, plus the OpenMP runtime underneath, plus whatever your own concurrent.futures pool does. Matrix multiply is fast. Everything else pays for it.

The problem shows up when jobs overlap. Suppose Task Scheduler fires three jobs at 02:00:

Job A does a rolling regression over 200k rows.
Job B builds a covariance matrix.
Job C just reads a CSV with pandas and writes a parquet file.

Each one imports numpy, each one gets eight BLAS threads. That is 24 threads on 8 cores. The OS context-switches between them, cache lines bounce between cores, and the memory allocator thrashes. Job C, which does almost no math, is now slow because it is competing for scheduler time with two BLAS pools it never asked for.

The fix is boring and effective: pin every job to one BLAS thread and let the OS schedule your jobs in parallel instead of your math kernels.

The Three Variables

OPENBLAS_NUM_THREADS

Read by OpenBLAS at initialization. Controls the size of its internal thread pool. This is the one that matters for stock PyPI numpy wheels.

MKL_NUM_THREADS

The equivalent knob for Intel MKL. If your numpy is MKL-linked, this is the one that counts. MKL also honors MKL_DYNAMIC and MKL_THREADING_LAYER, but for our purposes MKL_NUM_THREADS=1 is enough.

OMP_NUM_THREADS

Read by the OpenMP runtime. Both OpenBLAS and MKL can use OpenMP as their threading backend, and so can other libraries (scikit-learn, some pandas paths, PyTorch). Setting this catches the cases where the BLAS-specific variable is ignored because the library was built with OpenMP threading.

Set all three. The cost of setting one that is unused is zero; the cost of missing the one that matters is a mystery slowdown you will spend an afternoon debugging.

There is also NUMEXPR_NUM_THREADS for numexpr, which pandas uses for some eval paths, and VECLIB_MAXIMUM_THREADS on macOS. On Windows servers the first three cover the realistic cases.

Where to Set Them

Two places, and they are not equivalent.

In the launcher

The safest place is the process environment before Python starts. In a .bat file:

@echo off
set OPENBLAS_NUM_THREADS=1
set MKL_NUM_THREADS=1
set OMP_NUM_THREADS=1
set NUMEXPR_NUM_THREADS=1
"C:\Python311\python.exe" "D:\jobs\build_features.py" >> "D:\logs\build_features.log" 2>&1

This is the version I use in Task Scheduler actions. The variables are set before python.exe is even loaded, so every library that reads them at import time gets the right value. No ordering concerns, no surprises.

If you drive jobs from PowerShell instead:

$env:OPENBLAS_NUM_THREADS = "1"
$env:MKL_NUM_THREADS      = "1"
$env:OMP_NUM_THREADS      = "1"
& "C:\Python311\python.exe" "D:\jobs\build_features.py"

Same effect. The environment block is inherited by the child process.

In Python, before importing numpy

Sometimes you cannot control the launcher — a CI runner, a wrapper someone else owns, a python -m invocation from a service. Then set the variables at the very top of your entry point:

# entry.py — must run before any numerical import
import os

os.environ.setdefault("OPENBLAS_NUM_THREADS", "1")
os.environ.setdefault("MKL_NUM_THREADS", "1")
os.environ.setdefault("OMP_NUM_THREADS", "1")
os.environ.setdefault("NUMEXPR_NUM_THREADS", "1")

# Now it is safe to import the numerical stack.
import numpy as np
import pandas as pd

Two things to notice.

First, setdefault rather than =. If the launcher already set a value, respect it. This lets you override per-job without editing the script.

Second, the ordering is load-bearing. OpenBLAS reads OPENBLAS_NUM_THREADS inside its constructor, which runs when the shared library is dlopened — i.e. during import numpy. If you set the variable after that import, you are too late. The pool is already sized.

This bites people who put the os.environ lines in a config.py that gets imported after numpy somewhere up the chain. If you cannot guarantee import order, use the launcher.

A guard for libraries that import numpy for you

If your entry point imports a helper module that imports pandas at module scope, you have already lost. A defensive pattern:

# sitecustomize.py or a bootstrap module imported first
import os

for var in (
    "OPENBLAS_NUM_THREADS",
    "MKL_NUM_THREADS",
    "OMP_NUM_THREADS",
    "NUMEXPR_NUM_THREADS",
):
    os.environ.setdefault(var, "1")

Drop this in a sitecustomize.py on PYTHONPATH and it runs before anything else in the interpreter, including numpy. It is heavy-handed for a laptop, but on a scheduled server it removes an entire class of ordering bugs.

When Single-Threading Wins, and When It Loses

Single-threading is not universally faster. It is faster when the machine is oversubscribed or the workload is memory-bound. It is slower when a single job has the box to itself and does real dense linear algebra.

Wins

Overlapping scheduled jobs. Three jobs each wanting eight threads on eight cores is worse than three jobs each wanting one.
Small matrices. A 200×200 dgemm does not have enough work to amortize thread wake-up. OpenBLAS's own heuristic often decides not to parallelize; forcing one thread removes the decision cost.
Memory-bound operations. Elementwise ops, np.where, pandas groupby aggregations — these are limited by RAM bandwidth, not FLOPs. Extra threads add cache-coherence traffic without adding throughput.
Predictable memory. Each BLAS thread gets its own scratch buffers. Eight threads means eight copies of the workspace. On a box with several jobs, that is the difference between fitting in RAM and swapping.
Container and CI environments. The container sees the host's core count but may be CPU-limited by cgroups. Threads get throttled and the pool spends time in futex waits.

Losses

A single large dense factorization. Cholesky on a 5000×5000 matrix, SVD on a big design matrix, a Monte Carlo sweep that is pure dgemm. Here BLAS threading is genuinely good, and forcing one thread can cost you 5–10×.
A job that owns the machine. If nothing else is scheduled and the box has 16 idle cores, pinning to one thread leaves them idle.

The practical rule: if jobs overlap, or if the workload is not dominated by large dense linear algebra, go single-threaded. If you have one job that does heavy dgemm and it runs alone, give it threads — but give it only that job, and set the thread count explicitly rather than letting it grab every core.

You can also split the difference per job. A feature builder that mostly does pandas joins gets OMP_NUM_THREADS=1. A nightly calibration that inverts a big matrix gets OPENBLAS_NUM_THREADS=4 and is scheduled in a window where nothing else runs.

A Benchmark Harness

Do not guess. Measure on the actual box with the actual workload. Here is a small harness that runs a representative operation at several thread counts and reports wall time and peak RSS.

"""bench_threads.py — compare BLAS thread counts on a real workload."""
import os
import subprocess
import sys
import time
import json

# The child process does the actual work so the environment is clean.
CHILD = r"""
import os, time, json, sys
import numpy as np

n = int(sys.argv[1])
reps = int(sys.argv[2])
rng = np.random.default_rng(0)

a = rng.standard_normal((n, n))
b = rng.standard_normal((n, n))

# warm up
_ = a @ b

t0 = time.perf_counter()
for _ in range(reps):
    c = a @ b
t1 = time.perf_counter()

# a memory-bound op for contrast
t2 = time.perf_counter()
for _ in range(reps):
    d = np.sqrt(np.abs(a))
t3 = time.perf_counter()

print(json.dumps({
    "matmul_s": (t1 - t0) / reps,
    "elementwise_s": (t3 - t2) / reps,
}))
"""


def run(n: int, reps: int, threads: int) -> dict:
    env = os.environ.copy()
    for var in ("OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS",
                "OMP_NUM_THREADS", "NUMEXPR_NUM_THREADS"):
        env[var] = str(threads)

    proc = subprocess.run(
        [sys.executable, "-c", CHILD, str(n), str(reps)],
        env=env, capture_output=True, text=True, check=True,
    )
    return json.loads(proc.stdout.strip().splitlines()[-1])


if __name__ == "__main__":
    n = 2000
    reps = 3
    for threads in (1, 2, 4, 8):
        r = run(n, reps, threads)
        print(f"threads={threads:>2}  "
              f"matmul={r['matmul_s']*1000:8.1f} ms  "
              f"elementwise={r['elementwise_s']*1000:8.1f} ms")

Run it on the target machine. On a typical 8-core VM with a 2000×2000 matmul you will see matmul drop steadily from 1 to 8 threads, while elementwise barely moves — it is memory-bound and the extra threads do nothing. That is the shape of the trade-off.

Now run it twice concurrently to simulate overlap:

import concurrent.futures as cf

with cf.ThreadPoolExecutor(max_workers=2) as pool:
    futures = [pool.submit(run, 2000, 3, t) for t in (1, 8)]
    for f in futures:
        print(f.result())

With threads=8 on both, the combined time is usually worse than running them sequentially. With threads=1 on both, they finish in roughly the time of one job each, in parallel. That is the whole argument for pinning to one thread in a scheduled environment.

Reading the results

Two numbers matter: wall time and peak RSS. Wall time tells you whether you gained throughput. Peak RSS tells you whether you gained predictability. In my experience the memory win is the one that stops the 3 a.m. pages — a job that used to balloon to 2 GB under thread contention now sits flat at 600 MB.

If you want peak RSS in the harness, wrap the child with psutil:

import psutil, subprocess

p = subprocess.Popen([sys.executable, "-c", CHILD, "2000", "3"], env=env)
proc = psutil.Process(p.pid)
peak = 0
while p.poll() is None:
    try:
        peak = max(peak, proc.memory_info().rss)
    except psutil.NoSuchProcess:
        break
    time.sleep(0.05)
print(f"peak RSS: {peak / 1e6:.1f} MB")

Checklist

For every scheduled numerical job on a shared Windows box:

Set OPENBLAS_NUM_THREADS, MKL_NUM_THREADS, OMP_NUM_THREADS, and NUMEXPR_NUM_THREADS to 1 in the launcher, before python.exe.
If you cannot control the launcher, set them at the top of the entry point with os.environ.setdefault, before any numerical import — or use a sitecustomize.py.
Benchmark on the actual hardware with the actual matrix sizes. Raise the thread count only for a job that owns the machine and does large dense linear algebra.
Watch peak RSS, not just wall time. Flat memory is the real prize.

The environment variables are not a micro-optimization. On a box running several jobs, they are the difference between a schedule that finishes and one that drifts into the morning.

More notes like this ship every week on this site.

Daily Picks

Long

Pair	Signal	Price	Take Profit	Stop Loss	R/R
SKYAI		$0.0416	$0.0433	$0.0406	1:1.6
RE		$0.4991	$0.5181	$0.4866	1:1.5

2 picks selected. Scanner runs every 15 minutes.

Fail-Fast Retry Design for Scheduled Scripts: Three Attempts with Backoff

mattleeee — Tue, 06 Oct 2026 04:23:43 +0000

Unattended scripts fail for the least interesting reasons. A DNS lookup times out at 02:00. An API returns 429 because a cron job on another box happened to fire at the same second. A TCP connection resets halfway through a response. None of these are bugs in your logic, and all of them will page you if the job is important enough.

The instinct is to wrap everything in a try/except and loop until it works. That instinct produces a worse failure mode: jobs that hang forever, hide real errors, and leave the scheduler unable to tell whether the previous run finished. This post describes a retry wrapper that stays small, fails fast, and gives the Windows Task Scheduler layer enough information to act.

The failure modes worth designing for

Before writing the decorator, it helps to be explicit about what counts as retryable. In practice, the set is narrow:

Network-level errors: connection reset, timeout, DNS failure.
HTTP 5xx responses.
HTTP 429 with a Retry-After header or a known rate limit.
Transient file locks on a shared drive (common on Windows when another process holds a handle).

Everything else should fail immediately. A 401 means your token expired — retrying three times just wastes thirty seconds before the same error. A 400 means your payload is malformed — retrying will never fix it. A KeyError in your parsing code means the upstream schema changed, and no amount of backoff will help.

The practical rule: retry on exceptions that represent transport problems, not logic problems. In Python, that usually means requests.RequestException and its subclasses, plus a small allowlist of OS-level errors.

A decorator that does one thing

Here is the core implementation. It retries up to three times with linear backoff — 10, 20, then 30 seconds — and logs each attempt with its failure reason.

import functools
import logging
import time

import requests

logger = logging.getLogger(__name__)

RETRYABLE = (requests.RequestException,)


def retry(max_attempts: int = 3, base_delay: int = 10):
    """Retry a function on transport errors with linear backoff.

    Delays are base_delay * attempt: 10s, 20s, 30s for the default.
    KeyboardInterrupt and SystemExit propagate untouched.
    """
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            last_exc = None
            for attempt in range(1, max_attempts + 1):
                try:
                    return func(*args, **kwargs)
                except RETRYABLE as exc:
                    last_exc = exc
                    logger.warning(
                        "%s attempt %d/%d failed: %s: %s",
                        func.__name__, attempt, max_attempts,
                        type(exc).__name__, exc,
                    )
                    if attempt < max_attempts:
                        delay = base_delay * attempt
                        logger.info("Sleeping %ds before retry", delay)
                        time.sleep(delay)
            logger.error(
                "%s exhausted %d attempts, last error: %s",
                func.__name__, max_attempts, last_exc,
            )
            raise last_exc
        return wrapper
    return decorator

Three details matter here.

except RETRYABLE, not except Exception. Catching Exception would swallow KeyboardInterrupt? No — KeyboardInterrupt inherits from BaseException, not Exception, so a bare except Exception would actually let it through. But except Exception would catch SystemExit? Also no. The real danger is catching AttributeError, KeyError, ValueError — logic errors that retrying cannot fix. Those should crash the job immediately so you see them in the log at 02:01, not at 02:06 after three pointless sleeps.

Explicit re-raise after the loop. If you let the loop fall through, the function returns None, and the caller has no idea anything went wrong. Re-raising last_exc preserves the original traceback context and lets the exit-code layer above decide what to do.

Logging on every attempt, not just the final failure. When you're debugging a flaky 3 AM job, the pattern matters. Three timeouts in a row points at the network. A 429 followed by two successes points at rate limiting. A 500 followed by a 200 followed by a 500 points at a load balancer routing to a bad backend. You only see these patterns if you log each attempt.

Backoff shape: linear vs exponential

Linear backoff (10, 20, 30) is the right default for scheduled jobs that run on a fixed cadence. If your job runs every five minutes, the total retry window of 60 seconds fits comfortably inside the interval. Exponential backoff (10, 20, 40, 80…) is better for interactive or high-frequency callers where you want to shed load quickly, but for a job that fires every N minutes, linear keeps the worst-case runtime predictable.

If you need to honor a server-provided delay, extract it before sleeping:

def _retry_after(exc: requests.RequestException) -> int | None:
    resp = getattr(exc, "response", None)
    if resp is None or resp.status_code != 429:
        return None
    header = resp.headers.get("Retry-After")
    if header and header.isdigit():
        return int(header)
    return None

Then in the loop, prefer _retry_after(exc) over the computed delay when it's present. This is the one case where the server knows better than you do.

Exit codes: the contract with the scheduler

Windows Task Scheduler records the exit code of every run, and you can filter on it in the history view. More importantly, a wrapper script can branch on it. The useful convention is small and unambiguous:

Code	Meaning
0	Success — work was done
1	Failed after retries — investigate
2	Nothing to do — no work this cycle
3	Configuration or setup error — fix before next run

The distinction between 1 and 2 is the one people skip, and it's the one that saves the most time. A job that polls an API for new records will legitimately find nothing most cycles. If "nothing to do" returns 1, your monitoring is useless — every quiet night looks like an outage. If it returns 0, you can't tell from the logs whether the job actually ran.

Here's how the wrapper looks:

import sys

EXIT_OK = 0
EXIT_FAILED = 1
EXIT_NOTHING = 2
EXIT_CONFIG = 3


def main() -> int:
    try:
        records = fetch_pending()
    except requests.RequestException:
        return EXIT_FAILED
    except ConfigError:
        return EXIT_CONFIG

    if not records:
        logger.info("No pending work")
        return EXIT_NOTHING

    process(records)
    return EXIT_OK


if __name__ == "__main__":
    sys.exit(main())

The fetch_pending function is decorated with @retry(). When it exhausts its attempts, it re-raises, main catches the specific exception type, and returns EXIT_FAILED. The scheduler sees 1, your alert rule fires, and the log already contains three timestamped failure reasons.

Why retry state must live on disk

This is the part that bites people who test their retry logic interactively and then deploy it. Consider a job that processes a queue of items:

@retry()
def process_batch():
    items = load_queue()  # reads from disk
    for item in items:
        call_api(item)
        mark_done(item)

The retry state here is the queue on disk. If attempt 1 processes items 1–5 and then fails on item 6, attempt 2 reloads the queue and sees items 6 onward. That's correct. The progress is durable because it lives in a file, not in a variable.

Now consider the in-memory version:

@retry()
def process_batch():
    items = load_queue()
    for item in items:
        call_api(item)
        mark_done(item)
    return items  # cached in memory

If you cache items at module level to avoid re-reading, a retry after a partial failure will re-process items 1–5. Depending on the API, that's either wasted work or a duplicate side effect. Worse, if the process is killed between attempts — a reboot, a taskkill, a power event — any in-memory state is gone entirely, and the next scheduled run has no idea what happened.

The rule: the retry loop should re-read its input from the durable source on every attempt. The decorator doesn't know or care what the function does; it just calls it again. It's the function's responsibility to start from a consistent, on-disk state each time.

This has a second benefit. If the job crashes hard — no retry, just a blue screen — the next scheduled run picks up from the same disk state. The retry decorator handles the fast failures; the disk state handles the slow ones.

Wiring it into Task Scheduler

The scheduler itself has retry settings, and they interact with yours. Under the task's Settings tab, "If the task fails, restart every" is a separate mechanism from your in-process retry. Keep them distinct:

In-process retry handles transient errors within a single run. Fast, cheap, no scheduler involvement.
Scheduler restart handles process-level failures: a crash, a hung process killed by a timeout, a machine that rebooted mid-run.

A reasonable pairing is three in-process attempts with linear backoff, and one scheduler restart after a short delay. That gives you four total chances spread over a few minutes without turning a broken job into an infinite loop.

One Windows-specific gotcha: if your task uses "Stop the task if it runs longer than", set that duration longer than your worst-case retry window. Three attempts with 10/20/30 second backoff plus request timeouts can easily exceed two minutes. If the scheduler kills the process at 60 seconds, you never see the third attempt, and the exit code is a generic timeout rather than your EXIT_FAILED.

What this design deliberately doesn't do

It doesn't retry on HTTP 4xx (except 429). It doesn't use jitter, because scheduled jobs don't have the thundering-herd problem that interactive clients do. It doesn't persist retry counts across runs — if the job fails three times and the scheduler restarts it, that's a fresh three attempts, and the log will show eight failures total, which is a signal in itself.

It also doesn't try to be clever about partial success. If a batch of 100 items processes 60 and then fails, the retry starts from item 61 because that's what the disk says. If your API doesn't support idempotent writes, you need a different design — one that records the exact operation before calling out, not after.

The whole wrapper is about forty lines. Most of the value is in the parts that don't retry: the narrow exception tuple, the explicit re-raise, the exit code that separates "failed" from "empty." Fail fast, log everything, and let the disk be the source of truth.

More notes like this ship every week on this site.

Daily Picks

Long

Pair	Signal	Price	Take Profit	Stop Loss	R/R
SKYAI		$0.0416	$0.0433	$0.0406	1:1.6
RE		$0.4991	$0.5181	$0.4866	1:1.5

2 picks selected. Scanner runs every 15 minutes.

Running Scheduled Tasks Silently: The VBS Wrapper Pattern

mattleeee — Tue, 06 Oct 2026 04:22:45 +0000

The Problem: Black Windows Every Hour

A monitoring server had nine scheduled tasks, each firing hourly. The machine stays logged in permanently—it runs a dashboard that needs an interactive session. That detail matters, because it meant every task launched a console window on the desktop. Nine tasks, hourly, each flashing a black CMD rectangle over whatever was on screen. Unacceptable for a machine someone actually looks at.

The tasks did real work: pulling metrics, rotating logs, syncing a couple of directories. They were created over time by different tools—some by hand with schtasks, some by a third-party installer, one by an old deployment script. That mix turned out to be the source of the second problem.

Diagnosis: Why `schtasks /change` Failed

The obvious fix is to flip each task to run whether the user is logged on or not, which suppresses the window:

schtasks /change /tn "\Metrics\HourlyPull" /ru "SYSTEM"

That failed with a parameter error:

ERROR: The parameter is incorrect.

No further detail. The task existed, the name was right, the quoting was right. I checked the XML with schtasks /query /tn "\Metrics\HourlyPull" /xml and the task definition looked normal. The failure was not in my command.

The real cause is in how schtasks.exe is built. It is a thin command-line parser sitting on top of the Task Scheduler COM API (the ITaskService interface family). When you run /change, it reads the existing task, mutates a few properties, and writes it back. Tasks created by other tools—installers, deployment frameworks, anything that talks to the COM API directly—often carry property combinations that the schtasks parser does not expect. In this case, the offending task had a principal block with a logon type the CLI did not know how to round-trip. The COM call inside returned E_INVALIDARG, and schtasks.exe surfaced it as the generic "parameter is incorrect."

The lesson: when a task was not created by schtasks, do not try to fix it with schtasks. Go to the same API layer the creating tool used.

PowerShell Reaches the Same API Without the Parser

Set-ScheduledTask and Get-ScheduledTask are PowerShell cmdlets that wrap the identical COM interface. They do not re-parse a command line, so they do not choke on properties the CLI does not understand. The same mutation that failed above worked immediately:

$t = Get-ScheduledTask -TaskName "HourlyPull" -TaskPath "\Metrics\"
$t.Principal.LogonType = "S4U"
$t.Principal.RunLevel = "Highest"
Set-ScheduledTask -InputObject $t

S4U (service-for-user) runs the task without storing a password and without an interactive session, which is exactly the behavior needed to suppress the console window. Highest avoids the elevation prompt path.

But this only solved part of the problem. Switching every task to S4U changes the security context, and a couple of the tasks depended on the interactive user's mapped drives and environment. Rewriting all nine tasks was more invasive than necessary. There is a lighter, more surgical pattern.

The VBS Wrapper Pattern

The window appears because the task action is cmd.exe or powershell.exe, and those are console applications. When Task Scheduler launches a console application in an interactive session, Windows creates a console window. You can suppress that at the launch site rather than by changing the task principal.

The launch site is wscript.exe. It is a GUI-subsystem host: it has no console of its own. When it starts a child process, it can ask the shell to run that child with a hidden window. That is the entire trick.

The Two-Line Runner

Create silent_runner.vbs:

Set sh = CreateObject("WScript.Shell")
sh.Run "cmd /c """ & WScript.Arguments(0) & """", 0, False

Line by line:

CreateObject("WScript.Shell") gives us the shell automation object.
sh.Run takes three arguments: the command line, the window style, and whether to wait.
Window style 0 means hidden. This is the parameter that kills the black rectangle.
False means do not wait for the child to exit. The task returns immediately; the actual work runs detached.

The command line is cmd /c "target.bat". The double quotes inside the VBS string are escaped by doubling them, so the final command is cmd /c "<path to bat>". The cmd /c wrapper is there so the batch file runs in a proper command interpreter with its own environment, and exits when done.

Two lines, no dependencies, works on every Windows version since XP.

Wiring It Into Task Scheduler

Each task action changes from:

Program:   C:\Scripts\metrics_pull.bat
Arguments: (none)

to:

Program:   wscript.exe
Arguments: "C:\Scripts\silent_runner.vbs" "C:\Scripts\metrics_pull.bat"

That is the whole change. The task still runs as the same user, in the same session, with the same environment. Only the window is gone.

The PowerShell Loop

Nine tasks, done in one pass. This script rewrites the action of every task under a given path:

$runner = "C:\Scripts\silent_runner.vbs"
$path   = "\Metrics\"

Get-ScheduledTask -TaskPath $path | ForEach-Object {
    $task = $_
    foreach ($action in $task.Actions) {
        # Only rewrite actions that launch a console host directly.
        if ($action.Execute -match '^(cmd|powershell|pwsh)(\.exe)?$') {
            $target = $action.Arguments
            if ($target -match '^/c\s+(.+)$') { $target = $matches[1].Trim('"') }

            $action.Execute   = "wscript.exe"
            $action.Arguments = "`"$runner`" `"$target`""
        }
    }
    Set-ScheduledTask -InputObject $task | Out-Null
    Write-Host "Rewrote $($task.TaskName)"
}

A few things worth noting:

Get-ScheduledTask -TaskPath returns the task objects, and Set-ScheduledTask -InputObject writes them back through the COM API. No CLI parsing, no E_INVALIDARG.
The Execute check is a guard. If a task already runs a non-console program, leave it alone.
The regex ^/c\s+(.+)$ strips a leading /c from the arguments, which is how schtasks stores a cmd /c invocation. If the arguments are empty, $target stays empty and the rewrite is skipped by the guard—handle that case if your tasks vary.
Out-Null keeps the pipeline quiet; Set-ScheduledTask returns the updated task object otherwise.

If you want to be more conservative, add a -WhatIf style dry run first by printing $action.Execute and $action.Arguments before mutating.

Verification

Two checks, both fast.

First, confirm the task definition changed:

(Get-ScheduledTask -TaskName "HourlyPull" -TaskPath "\Metrics\").Actions |
    Format-List Execute, Arguments

Expected output:

Execute   : wscript.exe
Arguments : "C:\Scripts\silent_runner.vbs" "C:\Scripts\metrics_pull.bat"

Second, confirm the behavior. Run the task on demand and watch the desktop:

Start-ScheduledTask -TaskName "HourlyPull" -TaskPath "\Metrics\"

No window should appear. Then check that the work actually happened—the log file, the output directory, whatever the batch file produces. A hidden window that fails silently is worse than a visible one, so verify the side effects, not just the absence of the window.

If the task still flashes a window, the usual cause is that wscript.exe is not the actual launcher. Some tasks are configured with an "action" that is a script host already, or a task created by a tool that wraps the action in its own launcher. Query the raw XML to see what is really stored:

Export-ScheduledTask -TaskName "HourlyPull" -TaskPath "\Metrics\"

Look at the <Exec> block. If <Command> is not wscript.exe, the rewrite did not take.

When This Pattern Is the Right Choice

The VBS wrapper is not the only way to hide a task window. The alternatives and their trade-offs:

Run as S4U or SYSTEM: suppresses the window because there is no interactive session, but changes the security context. Breaks tasks that depend on mapped drives, user environment variables, or interactive credentials.
Use pythonw.exe instead of python.exe: works if your task is a Python script, but not if it is a batch file or a PowerShell script. Also changes how stdout and stderr behave.
Set the task to "Run whether user is logged on or not": same security-context change as S4U.
The VBS wrapper: keeps the same user, same session, same environment, same stdout/stderr behavior. Only the window is hidden. This is the least invasive option, which is why it was the right one here.

The pattern generalizes. Any time you need to launch a console process without a console window—from Task Scheduler, from a startup script, from another automation tool—wscript.exe plus a two-line VBS is the smallest possible shim. It has no runtime dependency, no installation, and it survives Windows updates because it uses an API that has been stable for two decades.

Adapting It to Python

If you prefer to keep everything in Python, you can generate the VBS and the task definitions from a script. This is useful when you have many tasks to convert or when the task list changes:

import subprocess
from pathlib import Path

RUNNER = Path(r"C:\Scripts\silent_runner.vbs")
RUNNER.write_text(
    'Set sh = CreateObject("WScript.Shell")\n'
    'sh.Run "cmd /c """ & WScript.Arguments(0) & """", 0, False\n',
    encoding="ascii",
)

def rewrite_task(task_name: str, task_path: str, target_bat: str) -> None:
    """Point an existing task at the silent runner."""
    ps = (
        f"$a = (Get-ScheduledTask -TaskName '{task_name}' "
        f"-TaskPath '{task_path}').Actions[0]; "
        f"$a.Execute = 'wscript.exe'; "
        f"$a.Arguments = '\"{RUNNER}\" \"{target_bat}\"'; "
        f"Set-ScheduledTask -TaskName '{task_name}' -TaskPath '{task_path}' "
        f"-Action $a | Out-Null"
    )
    subprocess.run(["powershell", "-NoProfile", "-Command", ps], check=True)

for name in ("HourlyPull", "HourlyRotate", "HourlySync"):
    rewrite_task(name, r"\Metrics\\", rf"C:\Scripts\{name}.bat")

The Python layer is just orchestration. The actual suppression still happens in the VBS, because that is the only place the window style can be set.

Summary

The failure mode was specific: schtasks /change returned a generic parameter error on tasks created by other tools, because the CLI's parser could not round-trip properties the COM API produced. The fix was to stop using the CLI and use PowerShell's Set-ScheduledTask, which reaches the same API without the parsing layer. The window suppression was a separate concern, solved by routing each task action through wscript.exe and a two-line VBS that runs the real command with window style 0. Nine tasks, one loop, no more black rectangles.

More notes like this ship every week on this site.

Daily Picks

Long

Pair	Signal	Price	Take Profit	Stop Loss	R/R
SKYAI		$0.0416	$0.0433	$0.0406	1:1.6
RE		$0.4991	$0.5181	$0.4866	1:1.5

2 picks selected. Scanner runs every 15 minutes.

Why I Replaced Daemon Processes with Windows Task Scheduler

mattleeee — Tue, 06 Oct 2026 04:15:40 +0000

Corporate Windows machines are hostile territory for long-running processes. You write a daemon, it runs fine on your dev box, then you deploy it to a managed laptop and it dies at some random point in the night with no traceback and no Windows Event Log entry. I spent a few weeks chasing this before giving up on the daemon pattern entirely and moving everything to Windows Task Scheduler. Here's what happened and how the migration looks in practice.

The Symptom: A Daemon That Dies Quietly

The first version of my job runner was a classic Python daemon: a parent process that spawned child workers, a watchdog thread that restarted workers if they crashed, and a log file that recorded every restart. On my own machine it ran for weeks.

On the corporate laptop, the log looked like this:

2024-03-11 09:14:02 INFO  worker started (pid 18432)
2024-03-11 09:14:02 INFO  worker pid 18432 exited with code 0
2024-03-11 09:14:02 INFO  restarting worker...
2024-03-11 09:14:02 INFO  worker started (pid 19004)
2024-03-11 09:14:02 INFO  worker pid 19004 exited with code 0
2024-03-11 09:14:02 INFO  restarting worker...

A tight restart loop with no error. The child processes were being created and immediately killed. No Python exception, because the kill happened at the OS level before the interpreter could raise anything.

The culprit was Sangfor aTrust, the endpoint security suite that ships with a lot of enterprise environments. It hooks process creation and, depending on policy, blocks or silently terminates child processes that don't match an allowlist. A daemon that spawns workers looks exactly like the kind of behavior a security product is built to stop. The watchdog made it worse: every restart attempt was another blocked CreateProcess call, which is why the loop was so tight.

You can confirm this pattern quickly. Run a trivial spawner and see whether the child ever gets to execute:

import subprocess
import sys

# If this prints nothing, the child never ran.
# That is a security-policy block, not a Python bug.
proc = subprocess.Popen(
    [sys.executable, "-c", "print('child alive')"],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
)
out, err = proc.communicate(timeout=10)
print("returncode:", proc.returncode)
print("stdout:", out)
print("stderr:", err)

If returncode is non-zero and both streams are empty, the child was killed before it could write anything. That's the signature.

Why Task Scheduler Survives Where Daemons Don't

The key insight is that Task Scheduler is not a user process. It's a Windows service (Schedule) running in session 0, and the processes it launches are created by the service host, not by your Python interpreter. Sangfor aTrust and similar suites generally allow this path because:

The parent is a trusted system service. Process-creation hooks are usually scoped by parent identity. svchost.exe running the Schedule service is on the allowlist; python.exe spawning python.exe is not.
Task Scheduler is a documented management surface. Enterprise security products are designed not to break the OS's own scheduling. Blocking it would break Windows Update, backup agents, and half the management stack. So it stays open.
No persistent parent process. There's nothing long-running to kill. Each task run is a short-lived process, launched, executed, and reaped by the service. The "daemon" is the scheduler itself, which you don't have to keep alive.

That last point is the real architectural win. Instead of building a process that stays up and supervises itself, you hand the supervision to the OS. The OS is very good at it and the security suite leaves it alone.

Mapping Jobs to Tasks

Once I accepted the model, the migration was mechanical. Each recurring job becomes one scheduled task with four things defined: a trigger, a working directory, a timeout, and a result convention.

Trigger

Triggers replace your while True: sleep(...) loop. For a job that runs every five minutes, use a daily trigger with a repetition interval. For a job that runs once at 06:00 on weekdays, use a weekly trigger with the day set. The scheduler handles the clock; you don't need time.sleep at all.

Working Directory

This is the most common source of "it works in my terminal but not in the task." Task Scheduler defaults the working directory to C:\Windows\System32. Any relative path in your script breaks. Always set an explicit working directory, and make your script resolve its own paths rather than relying on os.getcwd():

from pathlib import Path

# Anchor everything to the script location, not the process cwd.
BASE_DIR = Path(__file__).resolve().parent
LOG_DIR = BASE_DIR / "logs"
STATE_DIR = BASE_DIR / "state"
LOG_DIR.mkdir(exist_ok=True)
STATE_DIR.mkdir(exist_ok=True)

Timeout

Task Scheduler has a per-task execution time limit. Set it. A hung job that holds a database connection or a lock file forever is worse than a failed job. For a job expected to run in under two minutes, set the limit to ten. The scheduler terminates the process and records a failure, which you can see.

Result Convention

The scheduler records the process exit code. Use it. A script that always exits 0 is invisible to the scheduler's history. Exit non-zero on failure so the last-run result is meaningful:

import sys
import logging

logging.basicConfig(
    filename=str(LOG_DIR / "job.log"),
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
)

def main() -> int:
    try:
        run_job()
    except Exception:
        logging.exception("job failed")
        return 1
    return 0

if __name__ == "__main__":
    sys.exit(main())

Creating Tasks from Python

You can register tasks by hand in the GUI, but that doesn't scale and it isn't reviewable. I keep a single setup_tasks.py that registers every job idempotently. It shells out to schtasks.exe, which is present on every Windows install and needs no extra dependency.

import subprocess
from pathlib import Path

PYTHON = r"C:\Python311\python.exe"
BASE = Path(r"C:\jobs")

TASKS = [
    {
        "name": "QuantJob_Fetch",
        "script": BASE / "fetch.py",
        "schedule": "MINUTE /MO 5",
        "timeout_min": 10,
    },
    {
        "name": "QuantJob_Report",
        "script": BASE / "report.py",
        "schedule": "DAILY /ST 06:00",
        "timeout_min": 30,
    },
]

def register(task: dict) -> None:
    cmd = [
        "schtasks", "/Create",
        "/TN", task["name"],
        "/TR", f'"{PYTHON}" "{task["script"]}"',
        "/SC", *task["schedule"].split(),
        "/RL", "LIMITED",          # run with normal user rights
        "/F",                       # overwrite if it exists
    ]
    subprocess.run(cmd, check=True, capture_output=True, text=True)

    # Set working directory and execution time limit separately.
    # schtasks has no direct flag for cwd, so use XML for full control.
    set_task_xml(task)

def set_task_xml(task: dict) -> None:
    # Export, patch, re-import. This is the reliable way to set
    # WorkingDirectory and ExecutionTimeLimit.
    name = task["name"]
    xml = subprocess.run(
        ["schtasks", "/Query", "/TN", name, "/XML"],
        check=True, capture_output=True, text=True,
    ).stdout

    xml = xml.replace(
        "<WorkingDirectory></WorkingDirectory>",
        f"<WorkingDirectory>{BASE}</WorkingDirectory>",
    )
    xml = xml.replace(
        "<ExecutionTimeLimit>PT72H</ExecutionTimeLimit>",
        f"<ExecutionTimeLimit>PT{task['timeout_min']}M</ExecutionTimeLimit>",
    )
    # StartWhenAvailable: if the machine was off at trigger time,
    # run the task as soon as it comes back.
    if "<StartWhenAvailable>false</StartWhenAvailable>" in xml:
        xml = xml.replace(
            "<StartWhenAvailable>false</StartWhenAvailable>",
            "<StartWhenAvailable>true</StartWhenAvailable>",
        )

    subprocess.run(
        ["schtasks", "/Create", "/TN", name, "/XML", "-", "/F"],
        input=xml, check=True, capture_output=True, text=True,
    )

if __name__ == "__main__":
    for t in TASKS:
        register(t)
        print(f"registered {t['name']}")

The XML round-trip is the part people skip and then wonder why their working directory is System32. schtasks doesn't expose /WorkingDirectory or /ExecutionTimeLimit as command-line flags, so you export the task, edit the XML, and re-import. It's ugly but it's stable and it's scriptable.

Operational Wins You Get for Free

The migration wasn't just about surviving the security suite. Three things got better immediately.

Reboot survival

A daemon dies on reboot and you need a service wrapper, a startup shortcut, or a login script to bring it back. A scheduled task just runs on its trigger. If the trigger is a repetition interval, the task resumes on the next interval after boot. If you need it to catch up, <StartWhenAvailable>true</StartWhenAvailable> makes the scheduler run a missed task as soon as the machine is back. That single flag replaced an entire startup-recovery code path in my old daemon.

Built-in retry

Task Scheduler has a native retry policy. If a task fails, you can configure it to retry every N minutes for M attempts. That's the watchdog pattern, implemented by the OS, and it doesn't spawn child processes from your interpreter so the security suite has nothing to object to. You set it in the task's Settings tab, or in XML:

<RestartOnFailure>
  <Interval>PT5M</Interval>
  <Count>3</Count>
</RestartOnFailure>

Last-run result codes

Every task records its last run time and result code. You can query them without any custom instrumentation:

import subprocess
import csv
from io import StringIO

def last_results() -> list[dict]:
    out = subprocess.run(
        ["schtasks", "/Query", "/FO", "CSV", "/V"],
        check=True, capture_output=True, text=True,
    ).stdout
    rows = list(csv.DictReader(StringIO(out)))
    return [
        {
            "name": r["TaskName"],
            "status": r["Status"],
            "last_run": r["Last Run Time"],
            "result": r["Last Result"],
        }
        for r in rows
        if r["TaskName"].startswith("\\QuantJob_")
    ]

for row in last_results():
    print(row)

Last Result is the process exit code. 0 means success, anything else means the job failed and you can look at your log file. That's a free health dashboard for every job on the machine. I pipe this into a small status page and I haven't written a single line of process-monitoring code since.

What I'd Do Differently

Two things. First, I'd set explicit working directories from day one instead of debugging FileNotFoundError for an afternoon. Second, I'd design every job to be idempotent and short-lived. The scheduler model rewards jobs that start, do one thing, and exit. If your job needs to hold state between runs, write it to a file or a database rather than keeping it in memory. The process is gone when it's done; that's the point.

The daemon pattern isn't wrong in general. It's wrong on a locked-down corporate machine where you don't control the process-creation policy. Task Scheduler is the platform's own answer to "run this on a schedule," and in that environment, using the platform's answer is the shortest path to something that actually stays running.

More notes like this ship every week on this site.

Daily Picks

Long

Pair	Signal	Price	Take Profit	Stop Loss	R/R
SKYAI		$0.0416	$0.0433	$0.0406	1:1.6
RE		$0.4991	$0.5181	$0.4866	1:1.5

2 picks selected. Scanner runs every 15 minutes.

DEV Community: mattleeee

Surviving Corporate HTTPS Interception: Life Behind Sangfor aTrust

How the interception actually works

Distinguishing MITM from a normal corporate proxy

Which tools tolerate the corporate CA

Exporting the corporate root CA

Fixing Python: certifi, requests, and pip

Fixing Git

SSH is usually untouched

Routing around blocked hosts, politely

A note on certificate pinning

A short checklist

Daily Picks

Long

Health Checks for a Fleet of Scheduled Tasks

What the Task Scheduler actually exposes

Defining what "healthy" means

Classifying each task

Walking the folder tree

Matching tasks to expectations

The alert

Wiring it together

Why the audit must itself be scheduled

What to watch over time

Daily Picks

Long

Migrating a Python Automation Stack to a New Windows Server

Why "just copy the folder" fails

The package layout

xcopy with an exclusion list

One-click installer that auto-detects paths

Re-registering scheduled tasks

The encoding trap

Pinning file I/O

Pinning stdout

Pinning subprocess output

Pinning the environment

The verification bat

What I would do differently

Daily Picks

Long

Exit Code 267009: Debugging CMD Encoding Corruption

The Symptom

Reading Opaque CMD Exit Codes

Reproducing the Corruption

Why UTF-8 "Works" Until It Doesn't

The Fixes

1. Keep .bat files pure ASCII

2. If non-ASCII is unavoidable, save as GBK and use CRLF

3. Use 8.3 short paths for non-ASCII directories

A Checklist for Portable .bat Files

Takeaway

Daily Picks

Long

Staggering Nine Scheduled Tasks to Eliminate Resource Contention

Why top-of-hour schedules collide

Step 1: Inventory every trigger

Step 2: Rules of thumb for spacing

CPU-bound jobs need a full estimated runtime of separation

I/O-bound jobs can share a slot if they touch different files

Never let a writer and a reader of the same file overlap

Step 3: The offset schedule

Step 4: Handle the shared state file

Step 5: One page for the whole fleet

What changed

Daily Picks

Long

When GitHub Is Blocked but You Still Need to Push

Probing the firewall with curl.exe

Authenticating without a browser

The Contents API, step by step

Step 1: GET the current SHA

Step 2: PUT the base64 content

Step 3: One commit per file, with a sleep

Verifying the push without git

Why this path is worth keeping

Operational notes

Daily Picks

Long

Taming BLAS Threads in Scheduled Python Jobs

1. Keep `.bat` files pure ASCII

A Checklist for Portable `.bat` Files

Diagnosis: Why `schtasks /change` Failed