DEV Community: Sam Hartley

The Test Wasn't Wrong. It Was Answering an Older Question.

Sam Hartley — Mon, 19 Oct 2026 00:23:17 +0000

The Test Wasn't Wrong. It Was Answering an Older Question.

I shipped a change I had reviewed, simulated, and deployed. An hour later my acceptance suite was red.

The obvious reading is that the change broke something. I've made that reading correct plenty of times, so I started where anyone starts: at the failure. And the failure was, in a narrow sense, true — an assertion had stopped holding. It just wasn't an assertion about the code as it existed that afternoon. It was an assertion about the code as it existed three weeks earlier, frozen into a file nobody had touched since.

This post is about that gap, because red is not one thing. In my suite — and in most suites I've seen — a red run can mean at least three very different things, and only one of them is "you broke it."

Three kinds of red

Here's the taxonomy I now use before I touch anything:

A missing artefact. The suite opens a fixture by name and the file isn't there. The stack trace ends in FileNotFoundError naming the fixture, not a line of product code. This is an environment problem wearing a test failure's clothes.
A stale expectation. The test asserts a concrete value or a concrete shape that the product used to produce. The product changed on purpose; the assertion didn't. The test is answering a question about a version of the code that no longer exists.
A real regression. Behaviour that should still hold, doesn't.

All three show up as a red X with a line number. The difference lives in what the assertion says and where it dies. So the first thing I do now is not "find the bug" — it's classify. Which of the three am I looking at?

Classifying wrong costs the most when it costs the least to check. A FileNotFoundError takes thirty seconds to confirm. A stale expectation takes reading one file. A real regression takes an afternoon. But if you skip the first two checks because you "know" your change was fine, you'll eventually skip them on the day the third one is true.

The oracle that pinned a number

My suite has a behaviour oracle: a small module that calls the shipped functions and asserts specific outputs. It exists so that a refactor that changes behaviour without changing structure still fails. (The numbers below are stand-ins, not my actual defaults.)

That's a good idea with a sharp edge. The oracle held a literal from the previous version — a default parameter value that fed into a scoring function. When I changed that default deliberately (the old one was wrong for the data I now have), the oracle's assertion failed on correct output.

The failure message was honest: expected 0.20, got 0.35. Nothing in that message says "the expectation is three weeks old." The number just looks wrong, so the reflex is to look at the code that produced the new number.

What makes this class insidious is that the oracle is supposed to be rigid. I don't want a test that shrugs and says "close enough." I want it to pin behaviour so drift is visible. The cost of that rigidity is that every intentional behaviour change requires an explicit rebaseline of the expectation — and if you forget, you get red on green code.

The check that requires exactly one thing to change

The second red flag was structural. One of my integrity checks doesn't assert a value; it asserts a diff. Specifically: exactly one function may differ from a reference snapshot, and a short list of protected functions must be byte-identical.

That is a genuinely good guard against drive-by edits — the thing it catches is real. But it has an implicit assumption baked in: that history stopped at the snapshot. It doesn't know that two documented follow-up changes landed afterwards. Each of those changed additional functions, so the diff is now "one function plus the ones I meant to change," and the check says red.

changed = ['open_trade', 'score_signal', 'load_config']
expected = ['open_trade']
=> FAIL: protected_same=False

Reading that output, "the change broke something" is a perfectly reasonable first hypothesis. It's also wrong. The message is not lying; it's comparing against a museum piece and reporting the difference.

The tell: the failure lists functions you know you changed on purpose, and a protected one that moved as a direct consequence. If every entry in the diff has a commit message, you're not looking at a regression. You're looking at a baseline that stopped being maintained.

The driver that called the old signature

The third one is my favourite, because it points at the wrong file.

The suite's own driver called a function and then indexed the result:

trade = scanner.open_trade(signal, size)
confidence = trade["confidence"]

Then a gate I'd added made that function return None when the signal didn't qualify. Which is the correct behaviour — a gated signal shouldn't produce a trade. But now the driver does None["confidence"], and the traceback says:

TypeError: 'NoneType' object is not subscriptable
  at scanner.py line ...

The traceback names the product file, so that's where people look. The actual defect is in the test: it assumes a call always returns an object. The test was written before the gate existed, and the gate made an impossible path possible.

The general shape: a test that encodes an old contract fails when the contract changes, and blames the callee. If the caller is the test, the fix belongs in the test. If the caller is production code, the fix belongs in production code. The traceback never tells you which — only the call site does.

The dangerous fix

Every one of these three failures has the same tempting remedy: make it green.

For the first two, "make it green" means editing the expectation. Sometimes that's right. But the version of it I've caught myself doing is worse: bending the product back into the shape the test expects. Reverting a correct gate because a caller crashes. Putting the old default back because an oracle asserted it. That's how a suite stops being a safety net and becomes a museum with an admission policy.

A test suite is a model of the behaviour you want. When the model and the code disagree, one of them is stale. Deciding which one is the entire job, and it's a judgement call that shouldn't be made by whoever is closest to the red X.

Rebaselining is a decision, not an edit

When all three flags trace to stale expectations, the correct action is a rebaseline. It's also the action that quietly makes your suite weaker, so I don't do it silently anymore.

The concrete loss in my case: the side-effect audit only inspects functions that appear in the diff. Fold a function into a new baseline and it drops out of the audit — including, if you're careless, a brand-new function that opens files or calls things it shouldn't. The suite goes green by forgetting to look. That's a real trade, and it belongs to the person who owns the risk, not to the agent who wants a clean run.

What I do instead:

Say what's being weakened. Not "rebaselined, all green" — but "these four assertions now describe the new behaviour, and this function is no longer audited."
Simulate the whole rebaseline on a copy. A partial simulation is worse than none: it can pass the cheap checks and miss the one that reads the live directory.
Move every anchor in one edit. A manifest here, a hardcoded digest there, a driver call somewhere else — fix one and the others still fail, and you'll spend an hour convinced the fix didn't work.
Stop the background writer. My suite has a hermetic check that requires certain files to be byte-identical before and after. A running service writes to them every few seconds, so the check reads zero — which looks exactly like a defect. Pause the writer for the run, then confirm it's back.
Re-run against the deployed artefact. Green against my working copy proves my copy is self-consistent. Green against the file that actually runs proves something about the machine.

What I check now, in order

Read where it died, not just that it died. A file-not-found, a value mismatch, and a TypeError are three different investigations.
Ask what the assertion is about. Does it describe behaviour I still want? If yes, the code is suspect. If it describes last month's behaviour, the test is.
Check whether the test's caller is the test. Tracebacks blame the callee; contracts live at the call site.
Never edit the oracle to get green without saying so. If I change an expectation, I write down what guarantee I just gave up.

Four things I'd tell my past self

Red is a category, not a verdict. Missing fixture, stale expectation, real regression. Classify before you debug.
A baseline is a snapshot of an argument, not of the truth. It records what you believed when you froze it. If you keep changing the product and never the snapshot, the snapshot becomes an opinion about history.
Structural tests have a hidden expiry date. "Exactly one function changed" is true exactly once.
The cheapest wrong fix is bending the product to satisfy the test. It works immediately, and it removes the thing that was protecting you.

The failure that taught me this cost an afternoon and produced no bug fix at all — the code was right the whole time. Which is its own kind of lesson: sometimes the most valuable outcome of a red suite is the sentence "nothing is broken, my model is out of date," said out loud, on the record, before anyone touches the code.

Curious how other people handle this. Do you treat a stale assertion as a bug in the test, a bug in the process, or just debt? And has anyone found a rebaseline workflow that doesn't quietly shrink what the suite covers? Drop it in the comments.

Part of my Building in Public series — previously: the pipeline that went silent for five hours, the scheduled job that died with 'Operation not permitted', and the exit rule that counted the wrong thing.

My Pipeline Went Silent for Five Hours — Every Job Failed the Same Quiet Way

Sam Hartley — Sun, 11 Oct 2026 08:31:10 +0000

My Pipeline Went Silent for Five Hours — Every Job Failed the Same Quiet Way

I run a small scanner pipeline on a Mac in a closet. A handful of Python scripts on schedules, all reading and writing to one project folder, a couple of them talking to an exchange API. It's the kind of setup that hums along for months and quietly convinces you it has no moving parts.

Then one morning it did nothing for five hours — and the first I heard about it was me, sitting down at my desk, noticing that a normally chatty system had posted nothing since before dawn.

No error. No alert. No red anything. Just five hours of a pipeline that was completely, quietly dead.

This is the postmortem of a single point of failure I'd built without noticing, and the two changes that mean it can't fail this silently again.

The clue was an absence

The honest version of how I noticed: I didn't. Nothing paged me. I only realized because the volume of output I expect by mid-morning wasn't there. When your monitoring is "I check it and it looks busy," a dead system and a calm system are the same picture.

So I went looking for errors and found none — which is exactly the problem. The jobs weren't crashing. They were failing in the one way that never generates a complaint: they never got to the point of running.

The thing everything depended on

Every scheduled job had one thing in common: its working directory lived on a network share mounted at /Volumes/share.

mount | grep share
# //user@nas/… on /Volumes/share (smbfs, nodev, nosuid, …)

The code was fine. The schedules were fine. But the entire pipeline's ability to do anything at all rested on that mount being present. And last night it wasn't.

The timestamp that started everything:

02:35:43  [TRANSITION] OK -> UNMOUNTED

From 02:35 on, a mount point that every script treated as a normal directory was simply not there.

Anatomy of a silent failure

Here's what "the share is gone" does to a set of scripts that assume it's there — and why none of it screams:

Scheduled jobs fail at startup. A cron or launchd job whose script lives on the missing share can't even begin. If its output goes nowhere anyone reads, the failure is invisible.
Cron mail is off. The classic line MAILTO="" in the crontab — I set it years ago to stop noise — means a job that dies produces no message at all. A silent crontab plus a missing dependency equals nothing in your inbox, forever.
Processes that do start hang on I/O. My tracker process was alive but wedged:

  TimeoutError: [Errno 60] Operation timed out
    ... /Volumes/share/shadow_data/correlations.jsonl

A process that's stuck isn't a process that's dead — to any "is the PID running?" check, it looks healthy.

The paper trail was just a hole in a log. My fill-watch log's last line before the outage was at 02:34. The next line was at 07:34, after I fixed it by hand. Five hours with no entries, and nothing that treated an empty log as an event.

The bug that made it worse

If the mount had simply failed and retried, this might have been a ten-minute blip. But my auto-mount helper turned a blip into a five-hour outage.

I had a launchd job that ran every five minutes and tried to remount the share. It worked like this:

# auto-mount, every 5 min
mount_share() {
  mkdir -p "$MOUNT_DIR"
  osascript <<'EOF'
    mount volume "smb://user@nas/share"
  EOF
}

That osascript call has no timeout. Last night it hung — and stayed hung for five hours. Worse, the script held a lock directory while it waited, so the next run five minutes later found the lock held and bailed out. The self-healing loop had blocked the very recovery it existed to perform.

02:39:48  mount attempt starts  → hangs
…          lock held, every later attempt exits early
07:xx     I kill it by hand, clear the lock, mount manually

This is the part that stung: the thing I'd built to fix the outage was the reason it lasted all morning.

The fix, part one: every network call gets a deadline

The remount logic now wraps the blocking call in a hard timeout:

run_with_timeout() {          # default 25s, override via SMB_MOUNT_TIMEOUT_SECONDS
  local secs="${SMB_MOUNT_TIMEOUT_SECONDS:-25}"
  ( "$@" ) & local pid=$!
  ( sleep "$secs"; kill -9 $pid 2>/dev/null ) & local watcher=$!
  wait $pid; local rc=$?
  kill $watcher 2>/dev/null
  return $rc
}

mount_share() {
  mkdir -p "$MOUNT_DIR"
  run_with_timeout osascript -e "mount volume \"smb://user@nas/share\""
}

I tested it against a deliberately hanging mount:

hanging osascript → rc=124 after 4s   (previously: hung for 5 hours)

A hung mount now dies in seconds, releases the lock, and the next cycle can try again. The rule I should have applied years ago: any call that touches the network gets a deadline. Retries without timeouts aren't resilience — they're a new lock waiting to happen.

The fix, part two: stop depending on the share

The deeper fix was to remove the single point of failure rather than babysit it. Two changes:

Jobs run from local disk. The working directory moved off the network share entirely. The mount can now be missing and the scripts still run — they just can't reach their data, which is a state they're written to detect and report.
Logging is local-first, and shipped to the NAS as a delta. Writers always append to a local file, so the network can never block them. A small shipper process pushes only the new bytes since its last offset:

   local log → offset state → append delta to NAS copy

If the NAS is down, the shipper buffers locally and catches up when it returns — no lost lines, no duplicates. The canonical copy still lives on the NAS; the local copy is the buffer that keeps the pipeline from caring whether the NAS is up.

Now a dead NAS degrades the pipeline instead of killing it — and, crucially, the pipeline keeps producing the evidence I need to see that something's wrong.

The fix, part three: a watchdog that alerts on absence

Last piece: something that watches for the absence of a healthy share, not just the presence of errors.

A tiny watcher runs every five minutes and only speaks on a state change:

Share down for more than 15 minutes → one alert. Not every five minutes, once.
Share comes back → a recovery message, including how many bytes the shipper had buffered while it was away.

It's written fail-safe — it swallows its own exceptions so a broken watchdog can't take down what it's watching — and it stays quiet under the threshold, because an alert you learn to ignore is worse than no alert.

What I'd tell myself before the next outage

"Nothing is broken" and "nothing ran" look identical. If your monitoring only fires on errors, a dependency that's simply gone produces total silence — and silence reads as success. Monitor for the absence of expected effects, not just the presence of failures.
A self-healing loop without a timeout is worse than none. Mine held a lock for five hours and blocked its own recovery. Every retry, mount, and network call needs a hard deadline, or your fix becomes the outage.
If every job shares one dependency, its failure mode is "everything looks fine." And the monitor must never live behind the thing it monitors — mine did, on the same share, which is why the outage stayed invisible.

The cost of this one was an annoying morning. The lesson is cheaper than the next one: on a shared dependency, silence isn't health. It's just the shape of a failure you haven't built a question for yet.

Has anyone else been burned by a "self-healing" script that needed healing itself — or by monitoring that couldn't see an absence? I'd genuinely like to know how people detect "the scheduled thing didn't run" without inventing a whole observability stack for three Python scripts. Drop your approach in the comments.

Part of my Building in Public series — previously: the scheduled job that died with 'Operation not permitted', the exit rule that counted the wrong thing, and the drawdown breaker watching a number that never moved.

My Exit Rule Said '2 Periods' — My Code Was Counting Something Else Entirely

Sam Hartley — Mon, 05 Oct 2026 08:31:02 +0000

My Exit Rule Said '2 Periods' — My Code Was Counting Something Else Entirely

My funding-rate bot has an exit rule that reads like plain English:

exit_below_threshold_periods = 2
# "exit if the funding rate stays below threshold for 2 periods"

Two periods. Clear. The problem was the code, which counted this:

if rate < threshold:
    state["below_threshold_count"] += 1   # ...once per scan
if state["below_threshold_count"] >= EXIT_PERIODS:
    exit_position()

Once per scan. Not once per period. The bot scans about once an hour, so the counter was measuring scans (~1h each) while the config was talking about funding periods — which on some contracts are 8 hours.

Two scans is about two hours. Two funding periods is sixteen hours. So on 8-hour contracts my "2 period" rule was firing roughly 8× too early, on a counter I'd named below_threshold_count and therefore never questioned.

Why it survived so long

Here's the part that stung. The rule was correct for my hourly contracts. An hourly contract has a 1-hour period, the scan loop runs about hourly, so "2 scans" really is "2 periods." The bug was invisible on exactly the contract I looked at when I wrote it, and wrong by 8× on the contracts I added later.

That's the general shape of this class of bug: the unit of the name and the unit of the increment drift apart, and something in your test data quietly agrees with the wrong one. A counter named periods that increments per loop is only correct while loop-interval happens to equal period-length. Nothing in the code enforces that. The name just suggests it.

I've written before about a breaker that read a number which never moved. This is the same family: the label was right, the thing being measured was not what the label said.

Fixing it by deriving, not counting

The fix is to stop incrementing and start deriving. Instead of "how many times did the loop run," ask "how much time has passed, and how many periods is that?"

def _acc_periods_since(since, now, period_h):
    if not since:
        return 0
    elapsed_h = (now - since).total_seconds() / 3600.0
    return int(elapsed_h / period_h)

# on the exit site:
if rate < threshold:
    state.setdefault("below_threshold_since", now_iso())
else:
    state["below_threshold_since"] = None

if _acc_periods_since(state.get("below_threshold_since"),
                      now, _funding_period_hours(contract)) >= EXIT_PERIODS:
    exit_position()

Two changes matter here beyond the arithmetic:

The period length comes from the contract, not from a constant. _funding_period_hours(contract) returns 1 for hourly, 8 for 8-hour. The counter now scales with the thing it's named after.
There's a fallback that reuses the same period_h. If the module that knows the interval is missing, the helper still computes with the real period rather than silently reverting to a hardcoded 8. A fallback that guesses a different unit is worse than no fallback — it's how you get the original bug back, wearing a safety hat.

After the swap: an 8-hour contract's "2 periods" is 16h, and an hourly contract's is 2h. Each one correct for its own interval. The rule finally means what the config says.

The bug the fix uncovered

I shipped the change, then sent it to two external reviewers — a Grok pass and a Hermes pass — before rolling it to the live path. Both came back "accept with conditions," and both independently flagged the same medium-severity issue, which is the kind of agreement that makes you sit up.

The problem was in the reset. My new code only cleared below_threshold_since in the else-branch — the "rate is healthy" path. But there are early returns before that point: a flip-cooldown guard and a minimum-hold guard can both return early, skipping the healthy-path reset entirely. So the sequence was:

Scan A: rate healthy (+0.20%), but the position is in a cooldown → early return, below_threshold_since never cleared.
Scan B: rate dips below threshold (−0.03%) → the stale since from an earlier dip is still there, so the elapsed time already exceeds 2 periods → premature exit.

I reproduced it at the code level, not just in theory: a long position, one healthy scan during cooldown, then one weak scan, and it exited. The fix was to move the reset before the early returns, in both the paper and live traders.

This is the second time in a month that a "small counter refactor" turned out to have a control-flow seam in it. Counters look like data. They're actually state, and state has to be maintained on every path, including the ones that bail out early.

How I now test a counter

The checklist is short and I've started applying it to anything that counts anything:

Write the unit next to the name. below_threshold_count is a lie; below_threshold_periods or below_threshold_since is not. If the name doesn't say what unit it's in, assume a future reader will get it wrong — that reader was me.
Derive counts from timestamps, never from loop iterations. Loop iterations measure your scheduler, not the world.
Prove the rule with two intervals. The original bug only existed for 8h contracts while being invisible for 1h. If your test suite only has one contract type, you have a blind spot shaped exactly like the bug. My harness now has an explicit case — "2 scans on an 8-hour contract must not exit."
Reset the clock on every path, including early returns. A continue/return that skips your state maintenance is a time bomb.
Mutate the fix and confirm the test goes red. I removed the fix block on purpose; the new test failed. That's the only proof that the test is testing anything.

The honest summary: the arithmetic fix was maybe ten lines. The finding — that a counter named periods was counting scans, and that its reset was skipped on early returns — is the part worth keeping. The code got smaller and more literal. The process got a rule:

a number should be derived from the thing it claims to measure, and the name has to match the measurement.

I'd genuinely like to know how common this is: has anyone else shipped a counter or a threshold whose unit was fine in the case you tested and wrong in the case you didn't? Config values that say "periods," "minutes," or "count" but are really "loop iterations" feel like they're everywhere. Drop your version in the comments.

Part of my Building in Public series — previously: my drawdown breaker replayed to zero fires because it watched a number that never moved, my margin guard was reading zero, and the assertion that catches runs that do nothing.

My Drawdown Breaker Replayed to Zero Fires — Because It Was Watching a Number That Never Moved

Sam Hartley — Sat, 03 Oct 2026 08:30:39 +0000

My Drawdown Breaker Replayed to Zero Fires — Because It Was Watching a Number That Never Moves

I built a rolling drawdown breaker for my trading bot a few weeks ago. The idea is boring and correct: if equity drops more than X percent from its recent peak, stop opening positions. I wired it in, ran it in paper, and then did the responsible thing — I replayed the last month of real trades through it to see how it would have behaved.

Zero fires.

I sat with that for a while, and the conclusion I almost wrote down was: the breaker is useless, the strategy never drew down enough to need it. That conclusion would have been wrong, and it would have been wrong for a reason I've now added to a checklist.

"Zero" is an ambiguous result

There are two reasons a guard can have never fired:

Nothing ever crossed the threshold — the guard is fine, the market was kind.
The input the guard watches can't move — the guard is dead and you've been testing a corpse.

They look identical in the output. Both print 0. And the second one is much worse, because it doesn't just mean "no protection" — it means every replay, every backtest, and every paper run you did while believing in that guard was measuring nothing.

So the question I should have asked first isn't "did it fire?" It's "could this number have triggered it at all?"

Auditing the input

The breaker reads an equity series. Mine came out of the trader state and was synced from the exchange's account endpoint. On my exchange's futures API there are two different totals, and I had grabbed the wrong one:

total — the account's margin balance. It includes realized funding and PnL.
cross_margin_balance — the balance marked to market, i.e. total plus the open position's unrealized PnL.

I was feeding the breaker total. That number moves when a trade closes, when funding settles, when you deposit. It does not move when your open position bleeds in real time. From the breaker's point of view, a position could halve in price and the equity curve would sit perfectly flat until the moment of exit — and a rolling drawdown that only observes realized points will, by construction, almost never cross a threshold.

I checked the correlation directly, because that's a claim I could verify rather than reason about:

import numpy as np

# delta of state equity (what the breaker watched)
# delta of the same-moment unrealized PnL (what actually happened)
corr = np.corrcoef(d_eq, d_unrealised)[0, 1]
print(corr)   # -> 0.015

0.015. The two series are statistically unrelated. My equity curve had a 6-figure open loss sitting in it and recorded a gentle flat line. The breaker was reading a number that, at the scale and frequency of the drawdowns I cared about, essentially cannot move.

Re-running against a live input

Then I re-ran the exact same replay, same trades, same thresholds, but fed it mark-to-market equity (equity + unrealized):

input: state.equity (as shipped)     -> 0 fires
input: equity + unrealised (MTM)     -> 17 fires

Seventeen. And not scattered noise — clustered on one position, on one afternoon:

With a 5% threshold, the breaker fires repeatedly on 30.09 between 15:00 and 17:00. That is exactly the window where a single open position went from a normal drawdown to a multi-hundred-dollar loss.

So the breaker wasn't useless. It was correct. It just had no eyes.

The trap underneath the trap

And here's the part that still annoys me. Once the input was fixed, my first instinct was to ask "so would it have helped?" I ran the counterfactual — what would the account look like if the breaker had force-closed at the moment it fired?

Exit at 30.09 17:00: −801.82
Actually held to now: −576.87

Holding won by about +225. Which made me want to write "glad I didn't have a working breaker!" — and that would have been a second mistake stacked on the first. Two reasons:

One: survivorship in a sample of one. I have exactly one live position that went deep red. A sweep I ran over stop widths 5–30% showed no stop beating no-stop on the window I measured — but the moment you include that one still-open loss in the PnL, the sign flips:

closed trades only:     +144.01
closed trades + open:   -498.93

Same trades. Opposite verdict. A backtest that excludes an open drawdown isn't measuring a strategy, it's measuring the part of the strategy that already ended.

Two — the real point — the live account has no brake at all. The rolling breaker only exists in the paper path. The live path has a separate check_drawdown() that exits on the first trigger and is even stricter. So "live didn't stop the loss" is not evidence that holding was the right call. It's evidence that the mechanism was never wired. I was about to draw a lesson from an absence.

I also, while auditing, found my stop-width sweep had an inverted label bug — it was reporting "stop helped" as "stop hurt." One line. It had been quietly feeding me the wrong conclusion for a week.

How I now test a guard's input

The fix took a few lines. The finding is the checklist. For anything that enforces a threshold, before I trust a "0 events" result I now do this:

Prove the input can move. Feed the guard a synthetic worst-case through the real code path and assert it fires. If I can't make it trigger in a test, I have no evidence it can trigger in production.
Name what the number means, not where it came from. total and cross_margin_balance are both "the account balance." Only one is marked to market. equity is not a definition.
Correlate input against the thing it's supposed to track. corr(Δequity, Δunrealised) = 0.015 took ten seconds and would have saved a week. A guard's input should be strongly correlated with the phenomenon it guards against; if it isn't, the guard is decorative.
Never conclude from a sample that excludes the live case. Open positions are exactly the data most likely to carry the signal you're looking for. A sweep that drops them is a sweep that drops the finding.
Re-check the input after every "fix." My first fix swapped a missing value for a remote field that reports zero in cross-margin mode. The number was now structurally zero instead of accidentally zero. Same dead sensor, better provenance.

The assertion I keep coming back to, across all of this:

# a guard that has never fired is not "safe" — it's unproven.
# make it fire on purpose, or you've tested the test.
triggered = run_breaker(replay_with_synthetic_drawdown())
assert triggered, "breaker did not fire on a guaranteed drawdown -> input is dead"

The boring takeaway

I don't trust a guard because it's running. I trust it because I've watched it fire on an input I constructed to break it. "It returned 0" and "it returned 0 because it couldn't return anything else" are the same string in a log line, and only one of them means you're protected.

The breaker code got a two-line change. My process got a rule: audit the sensor before you celebrate the silence.

Curious if this is common: has anyone else shipped a guard that was technically correct and completely blind? I'm especially interested in cases where the input looked right — a balance, a rate, a percentage — but meant something different than you assumed. Drop it in the comments.

Part of my Building in Public series — previously: the margin guard that read zero, a single int() that disabled my risk system, the assertion that catches runs that do nothing, and the circuit breaker that caught three outages.

My Margin Guard Was Reading Zero. Four Separate Bugs Kept It That Way.

Sam Hartley — Tue, 29 Sep 2026 08:32:22 +0000

My Margin Guard Was Reading Zero. Four Separate Bugs Kept It That Way.

There's a number I keep in my head from this build. Five positions open, $933.89 of my account tied up in margin, against a rule that says never more than 50%. The rule existed. The rule was running every hour. The rule read zero.

I want to write this one down carefully, because the fix is about four lines and the finding took a week — and because "the guard read zero" turned out to be four independent bugs that only worked as a team.

The guard

A scheduled scan wakes up every hour, looks at funding rates across a few hundred contracts, and can open up to three positions. Each position takes a fixed slice of equity as margin, and there's a cap: total margin in use should never cross 50% of the account.

The check is the shape of every risk guard I've ever written:

margin_cap = equity * 0.50
if used >= margin_cap - 0.01:
    return  # account is full; don't open anything else

A number, a threshold, a comparison. Simple enough that I stopped reading it. Which is exactly the profile of the bugs that hurt.

Bug one: `None` became `0.0`, and `0.0` means "you're fine"

used came out of the saved state:

used = state.get("margin_used") or 0.0

On a state where that key was missing — and it was missing from most of them — or 0.0 turned it into zero. That line reads like defensive programming. It's actually a safety failure with good manners. The guard's whole job is to compare a number; when the number is absent, the honest answer is I don't know, don't open. The code said zero, go ahead. A fail-open default in the one place where fail-closed is the entire point.

Bug two: the number, when it existed, was in the wrong unit

Behind the None bug sat a worse one. When margin_used was populated, it wasn't always margin — some code paths wrote the position's notional value (size × leverage) into it and then compared that against a cap expressed in account percentage. Two units, one comparison.

This is why the bug survived so long: a unit error doesn't crash. It just makes the guard either fire early (annoying, you notice) or fire never (quiet, you don't).

Bug three: the exchange told me the truth, and the truth was zero

At some point I "fixed" bug one by syncing margin_used from the exchange instead of trusting local state. That felt rigorous. It wasn't.

My account runs in cross-margin mode. In cross mode the collateral sits at the account level, not on each position — and the API returns position_margin = 0 and order_margin = 0 for every open position. I checked it read-only: three positions open, both fields zero, real cross initial margin $460.97. My new source of truth was structurally zero.

I had outsourced a safety decision to a field whose semantics I never verified in my own account mode. The field wasn't lying. It was answering a different question than the one my guard was asking.

Bug four: the guard ran once, the loop opened five

This is the one I'm least proud of, because it's a bug I've written about before in a different costume.

The guard was evaluated once per scan — before the entry loop, from a single sync — and then the loop went to work opening positions. Nothing re-checked. Whether it's a rate limiter, a budget, or a margin cap: if you read your limit before the loop and mutate your usage inside the loop, your limit is a snapshot, not a limit.

The demonstration

So I stopped guessing and ran it end-to-end against the real code: five identical entry signals in one scan, everything stubbed except the guard.

Old: the guard saw $0.00 on every iteration. Five positions opened. Margin in use: $933.89 — 83.5% of the account, against a 50% cap.

New: the guard saw 0 → 186.78 → 373.56 → 560.34. It blocked the fourth entry. Three positions, $560.34 — 50.1%.

The fix

One idea: derive the number from the positions I actually hold, in the unit the cap is actually in, and read it inside the loop where it matters.

def margin_in_use(positions):
    """Sum of the margin each open position actually ties up."""
    return sum(
        float(p.get("capital_at_entry", 0)) / max(float(p.get("leverage", 1)), 1.0)
        for p in positions
    )

for candidate in candidates:
    # re-read every iteration, from the live position list
    used = margin_in_use(get_active_positions(state))
    if used >= margin_cap - 0.01:
        break
    execute_entry(candidate)

The exchange numbers stay in the code, but only as diagnostics — logged, printed, never used to make the decision. The rule I took away: don't outsource a safety decision to a remote field whose semantics you haven't verified for your own account configuration.

The test that actually proved it

This is the part I'd hand to anyone building a guard like this.

My first test rebuilt the guard formula inside the test file and compared its output against itself. It passed. It proved nothing — a tautology with a green checkmark. A test that re-implements the logic under test is only measuring its own author.

So I rewrote it to pull the guard block verbatim out of the source file, run the same scenarios through both the unpatched and the patched version, and — this is the important bit — assert that the two disagree. A control probe. If patched and unpatched produce identical output, the test isn't sensitive to the change, and any pass is meaningless.

scenario: margin_used missing
  unpatched: stops after 2 positions   (buggy — should be blocked elsewhere)
  patched:   opens 3, then blocks      (correct)
  control:   patched != unpatched ✅

That control line has caught more bad tests for me than any assertion since. It's the same instinct as checking that a scheduled job actually did something: make the test fail when nothing changes, if you want it to mean something when it passes.

What's still imperfect

I'd rather write the remaining gap down than pretend it's closed. The guard checks used, not used + next_position. So the worst-case overshoot is one full slot: with 16.7% slots, three of them is 50.1% against a 50% cap — 0.1pp over, arithmetic, and deliberate. With 30% slots it would be +10pp, and that's a decision I'd revisit before I ever changed the slot size.

What I'd copy

If you have anything that enforces a limit — margin, spend, rate, quota — the pattern is boring, and it took me far too long to land on it:

Absence is not zero. A missing value in a safety check should stop the action, never permit it.
Check units, not just numbers. A notional value and a margin value can both be "the right number" and still be incomparable.
Don't trust a remote field you haven't verified in your own mode. Cross margin reports zero at the position level. That's not a bug in their API; it's a question I wasn't asking.
Re-read the limit inside the loop. A limit evaluated against a snapshot is a snapshot.
Make your test prove it can fail. Extract the logic under test, and assert that the old and new versions disagree.

The cap was correct on paper the entire time. What it needed was a number that meant what I thought it meant — and the humility to stop treating "no value" as good news.

Same question I keep asking on these: has anyone else been bitten by a guard whose default was the unsafe path? I keep going back and forth on whether a missing state value should halt the job entirely or degrade gracefully — drop a comment with what's worked for you.

Part of my Building in Public series — previously: budget caps that held at 2am, the assertion that catches runs that do nothing, timeout means no on approval gates, the guardrail stack before going live, and a circuit breaker that caught three outages.

A Single int() Silently Disabled My Entire Risk System

Sam Hartley — Sun, 27 Sep 2026 08:30:29 +0000

A Single int() Silently Disabled My Entire Risk System

I didn't find this bug by debugging. I found it by answering a question.

Someone asked me what leverage my live trading bot actually runs at. I said "one and a half times, sometimes two, depends on the funding spread" — because that's what the ladder in my code says, and I'd read that code recently. Then I decided to prove it instead of trusting my memory, and the answer was 1. Not 1.5. Not "up to 2". One. Every position, every time, for months.

The fix took four characters. Finding it took longer, because the code was correct — under a config value I'd changed two months earlier.

The ladder

My position sizing has a small leverage ladder. Wide funding spread plus a stable rate, size up; thin spread, stay at 1x.

LEVERAGE_LADDER = [1, 1.25, 1.5, 2]
MAX_LEVERAGE = 1.5

def calc_smart_leverage(rate, stability):
    idx = 0
    if rate > 0.05: idx += 1
    if stability > 0.8: idx += 1
    if rate > 0.15: idx += 2
    leverage = LEVERAGE_LADDER[min(idx, len(LEVERAGE_LADDER) - 1)]
    return int(leverage)   # <- here

The ladder returns floats. int() truncates them. At MAX_LEVERAGE = 2.5 — the value this code was written against — the truncation was invisible:

The ladder max is 2.0, the cap is 2.5, so int(2.0) == 2. Nothing got clipped.
The only value that could have been clipped was 2.5 itself, and the ladder never returned it.

Then, in July, I lowered the cap to 1.5, because the 2.5x era had produced a drawdown I didn't like. That one-line config change turned a dormant int() into an active bug: int(1.5) == 1. Every rung above 1x collapsed to the bottom of the ladder. The bot was permanently, silently at minimum leverage.

The part that made me sit down

Leverage being 1x instead of 2x wasn't the real problem. The real problem was a guard that read leverage to decide whether the risk logic should run at all:

def check_drawdown(state):
    has_leveraged = any(p["leverage"] > 1 for p in state["positions"])
    if not has_leveraged:
        return "normal"     # no leverage, no risk to compute
    ...  # the actual drawdown math

any(leverage > 1) was never true, because leverage was always 1. So check_drawdown() returned "normal" on the first line, every time, without computing anything. My drawdown breaker was not "not triggering". It was not running. In the live trader and the paper trader, for months.

That's the shape I want to write down: the guard didn't fail. It didn't raise. It returned the reassuring answer, which is the worst possible failure mode for a safety check.

Why my tests never caught it

I had tests. They passed.

def test_leverage_respects_cap():
    lev = calc_smart_leverage(rate=0.3, stability=0.95)
    assert lev <= MAX_LEVERAGE   # 1 <= 1.5, green

This asserts the wrong thing. It checks that leverage is bounded above, which was always true and always irrelevant. It never asserted that the levered path is reachable. When I lowered the cap, the assertion stayed green — correctly! — while the feature it was supposed to protect went to zero.

The invariant I actually cared about wasn't "leverage ≤ cap". It was "a high-conviction signal produces leverage > 1". Those are different claims, and only one of them fails when the code is broken.

How I found the boundary

I didn't want to reason about it. I wanted a number. So I ran both implementations against the same inputs with a stubbed API client — no orders, nothing live — and printed the leverage for a spread that should have maxed the ladder.

Paper: 1. Live: 1.

Then I grepped the trade history for the field. leverage: 2 appeared exactly 6 times out of roughly 440 trades, all of them at the very start, before the cap change. Every trade after that: 1. The history had been telling me for two months. I just hadn't asked it the right question.

What I changed

Three things, none of them clever:

The sizing function returns a float. No truncation. If the ladder says 1.5, the position gets 1.5.
The "should I run risk logic" decision is computed once, explicitly, from the ladder's actual output — not inferred from a value that a truncation bug, a rounding change, or a config edit can quietly flatten.
The tests assert reachability, not bounds: given a maximal signal, assert the levered branch executes. If someone lowers the cap below the lowest rung again, that test goes red.

The general version

Two things I'd offer anyone running scheduled or automated systems:

A guard clause that returns the safe answer is a silent kill switch. if not has_leveraged: return "normal" is the same class of bug as if not errors: return True. It looks defensive. It's the thing that hides the failure.

Config changes can change code semantics. The int() was written when the cap was 2.5. It was correct then. Nothing in the diff that lowered the cap to 1.5 touched that function, and the tests stayed green — because they tested the bound, and the bound still held. When you change a constant that other code reasons about, the code that reasons about it is part of the change, whether the diff says so or not.

I spent a weekend on a four-character bug. The four characters were int(. The weekend was for the two months I didn't know my risk system was off.

If you have a similar one — a "harmless" coercion or rounding that quietly took a whole branch with it — I'd genuinely like to hear it. Drop it in the comments.

My Agent Spent $23 Overnight. Here's Why My Limits Held and My Alerts Didn't.

Sam Hartley — Fri, 25 Sep 2026 08:30:50 +0000

My Agent Spent $23 Overnight. Here's Why My Limits Held and My Alerts Didn't.

I woke up to a notification that said my nightly agent had failed. That's the good version of a bad morning — something went wrong, and I knew about it.

Then I opened the bill. Twenty-three dollars on a cloud model, spent between roughly 01:40 and 04:10, on a job that normally costs about forty cents. So the thing I actually want to write about isn't the twenty-three dollars. It's the timeline, because the interesting part is what happened around 01:40.

The cap fired. The job stopped. The log has the entry. And I found out five hours later, from a different job that failed for an entirely separate reason.

What the job does, briefly

Every night at 01:30, a scheduled job reads my notes, pulls a few feeds, and writes a draft summary to a local file. Ninety-five percent of the work runs on a local model. A cloud call is there for one narrow escalation: if the local model reports low confidence on a section, the job re-runs just that section on a bigger model via API.

That escalation is supposed to fire a handful of times a night. On the night in question it fired continuously for two and a half hours, because of a retry loop I wrote, tested in the happy path, and never once tested with a model that kept saying "low confidence."

Each retry was cheap. The loop wasn't. That's the whole shape of the bug: nothing in it is expensive, and the product of it is.

The part I got right, mostly by accident

The spend cap held. I put a daily budget ceiling in the executor months ago, after I wrote up the guardrail stack I use before anything touches real money. Same three-number pattern: a per-run cap, a per-day cap, and a freeze when the day's cap trips.

The reason it held is a design choice I'd make again: the cap is mechanical, not statistical. It doesn't ask a model whether this call is reasonable. It doesn't consult a moving average. It checks a counter, and when the counter crosses a line, calls stop being made.

class Budget:
    """Spend caps. No model, no heuristics. A counter and a lock."""

    def __init__(self, per_run_usd, per_day_usd, store):
        self.per_run = per_run_usd
        self.per_day = per_day_usd
        self.store = store  # survives restarts; this is the whole trick

    def authorize(self, estimated_usd):
        spent_today = self.store.get("spent_today")   # from the ledger, not memory
        run_spent = self.store.get("run_spent")

        if run_spent + estimated_usd > self.per_run:
            raise Denied("per-run budget exceeded")
        if spent_today + estimated_usd > self.per_day:
            raise Denied("daily budget exceeded — freezing")

        self.store.incr("spent_today", estimated_usd)
        self.store.incr("run_spent", estimated_usd)

    def settle(self, actual_usd, estimated_usd):
        self.store.incr("spent_today", actual_usd - estimated_usd)
        self.store.incr("run_spent", actual_usd - estimated_usd)

Two details in there did all the work:

The counter lives in a store, not in a process variable. The first version of my cap kept spent_today in memory. Any restart — and my agent restarts a lot — reset the counter to zero. A cap that resets when the thing it's capping restarts is a cap made of wishes. Moving it to a small on-disk ledger was a four-line change and it's the difference between the cap working and the cap being theater.

Authorize before the call, settle after it. I reserve against the estimate, then correct to the actual. If the actual comes back bigger than the estimate, the ledger absorbs the difference — which means the next call is judged against reality, not against my optimism. My estimates were reliably low. That's not a bug I fixed, it's a bias I compensated for.

The cap is not the alert

Here's the mistake, and it's the reason this post exists.

When the daily cap tripped, my code did the correct thing: it raised Denied("daily budget exceeded — freezing"), wrote an audit entry, and returned. Clean, fail-closed, no drama.

And it logged that at INFO.

My alerting rules only looked at ERROR and above, plus a couple of specific failure classes I'd enumerated. So from the alerting system's point of view, the night was perfect. No error. Nothing to see. The one job that did notify me was the morning job, and it failed because the draft file it expected never got written — a downstream symptom that arrived five hours after the causal event.

The lesson generalizes past my stack: a fail-closed cap is a refusal, and refusals are not exceptions. From inside the system, refusing is the system working. From outside, refusing is the system not working. Only a human can tell you which — and the human needs to be told at the moment it happens.

The fix was one line of category, plus one line of routing:

# this was the bug, dressed as good hygiene
log.info("daily budget exceeded — freezing")

# what it is now: a refusal is an incident, not a status
log.warning("BUDGET_FROZEN", extra={"spent": spent_today, "cap": self.per_day})

And BUDGET_FROZEN is now on the alert list — not the page-me-now list, but the tell-me-immediately list. There's a real distinction there. A budget freeze at 01:40 doesn't mean wake the human; it means the pipeline is dead until morning, and the human gets to know before the morning job fails, with the number attached. Twenty-three dollars is a shrug. Twenty-three dollars discovered at 04:15 by a confused downstream job is a five-hour blind spot.

The second bug: the loop had no shape

The retry loop is a separate story and it's more embarrassing, so here it is quickly.

My escalation logic was: if confidence is low, retry on the bigger model. I tested it by mocking a model that returns "low confidence" once and then confidently. So in my head, the loop was a single hop.

In reality, the bigger model also said low confidence, because the section genuinely was ambiguous, and the section never got less ambiguous, because I never changed anything between attempts. It retried the same query against the same input with the same context and hoped for a different answer. That is, definitionally, the thing I have a circuit breaker for — and the breaker didn't catch it, because every individual call succeeded. A 200 is a success. The breaker watches for failure, and my loop produced nothing but successes until the money ran out.

So the retry loop got two additions:

Attempt cap. Three tries, hard stop. Not "until it works."
Escalation must change the input. Each retry has to add something — more context, a decomposition step, a different prompt. A retry that repeats the previous attempt byte-for-byte isn't a retry, it's a subscription.

And the effect-count check I now put on every job — the assertion that a run must prove it produced something — is exactly what would have caught this at 01:45 instead of 04:10. The job was producing effects, so it would have passed. But the loop had no cost assertion, and cost is an effect too if you're paying for it by the token.

Two metrics that actually predict this

Post-incident, I stopped watching "spend total" as my primary number and started watching two ratios:

Spend per unit of work. Dollars per draft, per filing, per summary. This is the number that would have been screaming. My absolute spend looked fine for the first hour — forty cents an hour is normal-ish. It was spend per draft that went vertical, because the job was producing zero drafts while burning tokens. Same shape as the flatline I wrote about in the silent-failure post: the aggregate looked normal, the ratio was on fire.

Retry ratio: retries divided by first attempts. Mine sits near 0.05 on a normal night. Anything above ~0.3 means a loop or a degraded model, and it's visible in minutes rather than dollars. Cheap to compute, no new infrastructure, and it's the only number that moved before the bill did.

One more, and it's the uncomfortable one: the alert you're not watching is the alert you've muted. I'd been getting budget-adjacent notices for weeks and had stopped reading them, because most were routine. That's not an alerting failure, it's a calibration failure — and it's the same terminal state I wrote about with approval gates that ask too often. Any channel that asks more than a few times a month stops being oversight. If you're drowning in routine notices, the fix is fewer notices, not more attention.

What I'd tell you to copy

If you run anything that spends money on your behalf — cloud LLM calls, API credits, compute — here's the short version:

Caps in the executor, not the model. A counter and a line. No model gets a vote on whether to spend.
Counters survive restarts. If it lives in memory, it isn't a cap.
Reserve, then settle. Judge the next call against the real number, not your estimate.
A refusal is an incident. Log it at a level your alerts can see, with the number in the message.
Cap your retries, and make each retry change something. A repeated call isn't a retry.
Watch the ratio, not the total. Spend per unit of work is the early-warning number; the total is the receipt.

The cap worked exactly as designed. The $23 was the cap doing its job — the difference between a runaway bill and a bounded one. The failure was that I didn't hear it close. A guardrail you don't get alerted about is a guardrail you're trusting on faith, and I'd rather have the boring version: it stops, and it tells me it stopped.

Curious how other people handle this — do you alert on budget refusals immediately, or roll them into a digest so you don't train yourself to ignore them? I keep going back and forth. Drop a comment with what's worked.

Part of my Building in Public series — previously: the assertion that catches runs that do nothing, timeout means no on approval gates, the guardrail stack before going live, and a circuit breaker that caught three outages.

My Scheduled Agent Ran 40 Times and Did Nothing — Here's the Assertion That Fixed It

Sam Hartley — Wed, 23 Sep 2026 08:30:57 +0000

My Scheduled Agent Ran 40 Times and Did Nothing — Here's the Assertion That Fixed It

I found this one by accident, which is the only way you find it.

Every night at 02:30, a job pulls new filings from an API, normalizes them, and drops them in a queue that feeds my morning briefing. It had been running for about six weeks. Every run: exit code 0. No errors in the log. The scheduler dashboard showed forty consecutive green runs. If you'd asked me, I'd have said that job was my most reliable piece of automation.

Then I noticed my briefing had gotten thin. Not empty — thin. Some days two items, some days none, and I'd been assuming "slow news." I finally got suspicious, went to the log, and realized I had absolutely nothing to be suspicious with: the log said fetched items, wrote to queue on every run, and that's it. No counts. No IDs. Just a sentence I'd written myself, once, and never verified.

So I added one line — a count — and reran it manually.

Zero. Every night. Forty times.

The HTTP call returned 200 with an empty array, because a parameter name in the query had been renamed on their side (since → from_date) and the API's response to an unknown parameter was to cheerfully ignore it. My for loop iterated over an empty list and completed successfully, which is exactly what a for loop over an empty list does. Nothing raised. Nothing retried. Nothing was logged as a problem, because as far as every layer of my stack was concerned, nothing was a problem.

That's the failure mode I want to write about, because I'd covered the other ones: I have a circuit breaker for hard failures, a health monitor for agents that are dying, and dedup for jobs that fire twice. All three of those watch for something happening. None of them watch for something not happening. And "not happening" doesn't throw.

Errors page you. Silence doesn't.

The reason this took six weeks is structural, not lazy. An exception has a moment: it fires, a handler catches it, an alert goes out. A silent no-op has no moment. It has a shape over time, and only if you're graphing the right thing.

My monitoring was graphing the wrong thing. I was tracking run status (success/failed) and run duration. Both were perfect. What I wasn't tracking was the only number that mattered: effects produced. Runs completed. Effects were zero. I had a dashboard that was green because it was measuring the system's opinion of itself.

The fix isn't more logging. It's a different question: not "did it run?" but "did it change anything?"

Rule one: assert the effect, not the attempt

Every job I write now has to declare what a successful run looks like in terms of side effects — rows written, messages sent, files created, IDs emitted. Then the job asserts it before it can report success.

def assert_effect(name, produced, expected=">=1"):
    if expected == ">=1" and produced < 1:
        raise JobProducedNothingError(
            f"{name}: ran fine, produced {produced} effects"
        )
    log.info("%s: effect ok (%s)", name, produced)

The point isn't the helper. The point is that a job can now fail for the crime of doing nothing. Before, "did nothing" and "did everything" were the same outcome — success — which means my success signal was measuring the wrong universe.

Two things I had to get right, and got wrong first:

An assertion that can't fail isn't an assertion. My first pass was assert len(rows) >= 0. That's a tautology wearing a safety vest. It passed on the empty list and I felt protected for another two days. The assertion has to be able to actually fail on the thing you're worried about — which sounds obvious until you write one at 1 AM.

Assert on the effect, not the attempt. resp.status_code == 200 is an attempt check. len(written_ids) > 0 is an effect check. The whole incident was a case where every attempt check passed and the effect was zero. If your assertion can be satisfied by a system that is doing nothing, it's not an assertion, it's decoration.

Rule two: expected-empty is not unexpected-empty

Here's where I nearly overcorrected, and it's the subtlest part.

Some of my jobs legitimately do nothing most of the time. A watcher that looks for new items in a feed should find zero items on a quiet day. If I make "zero effects" a hard failure everywhere, those jobs scream every night and I'm back to a muted alert channel — which I already learned is worse than no alerts.

So the assertion can't be "produced > 0". It has to be "the source was successfully queried and the emptiness is trustworthy." That's a different check: the request succeeded, returned the expected shape, and the emptiness was an explicit, well-formed empty rather than a default. In my case the giveaway was available the whole time: the response had no pagination envelope. A real empty result would have come back with a cursor and a total. A silently-ignored-parameter empty came back as a bare [].

Practical form: for jobs that may legitimately produce nothing, assert on the envelope, not the count.

data = api.get("/filings", params={"from_date": since})
if not isinstance(data, dict) or "items" not in data:
    raise UnexpectedShapeError(f"got {type(data)}: {str(data)[:80]}")
items = data["items"]  # zero here is fine — we proved the query was honored

Now "quiet day" and "the API stopped honoring my request" look different, which is the entire job.

Rule three: three states, not two

This is what finally made it usable for both kinds of jobs. Every run now ends in one of three states, and they are not the same thing:

succeeded_with_effect — it did the thing, with a count attached.
no_work_needed — it did nothing and proved that nothing was the correct answer (envelope check above passed).
failed — including JobProducedNothingError, i.e. it ran, and the absence of effect was not explained.

Only the third one alerts. The second one is a normal, boring, expected outcome that shows up in the briefing as a single quiet line. Before this split, "did nothing" was indistinguishable from "worked" — and the alerting I did have was tuned to a world where those were the same.

The part that actually caught things

Beyond the original bug, the effect counters caught two more silent failures in the next month: a token that expired and got silently replaced with an anonymous, lower-privilege session (200s, empty results), and a path change that made a downstream job read an empty directory and write a valid, empty file. Neither threw. Both were invisible to status-based monitoring. Both showed up instantly as a flatline in effects.

The dashboard line I now look at first isn't uptime or error rate. It's effects per run, per job. A job that runs and produces nothing, forever, is not a healthy job — it's a green light with the bulb unscrewed.

The transferable bit

If you take one thing: "no error" is not "worked." Your scheduler, your orchestrator, and your CI will happily report success for a process that queried nothing, wrote nothing, and sent nothing, because that is exactly what they were asked to measure.

Whatever your stack is — a nightly ETL, a webhook consumer, a CI step, an agent that posts things — pick one number per unit of work that would be zero if the work silently stopped. Assert on it. Track it over time. And make sure that some of your alerts can fire on a flatline, not just a spike. Spikes are loud and self-reporting. Flatlines need someone to be watching for them on purpose.

Curious how other people handle this: do you assert on effect counts in production jobs, or do you rely on downstream freshness checks (data arriving late → alert)? Both seem to have holes. Drop a comment with what's caught silent failures for you.

Part of my Building in Public series — previously: the guardrail stack I built before going live, timeout means no on approval gates, a circuit breaker that caught three outages, and a health monitor that tells me when my agents are dying.

Timeout Means No — The One Rule That Makes AI Agent Approval Gates Actually Work

Sam Hartley — Mon, 21 Sep 2026 08:33:08 +0000

My trading bot went live recently, and I wrote up the guardrail stack I built first: hard budget caps, idempotency keys, a dead-man switch, and an approval gate for anything unusual. One of those has quietly become my favorite, and it's the least technical of the bunch.

The approval gate. Not the asking-for-permission part — the part everyone, including me, gets wrong on the first try. The rule that makes the whole thing work:

If I don't answer, the answer is no.

Sounds obvious now. It wasn't. I shipped two versions of that gate before this one, both politely broken in ways that only show up when the human is asleep. Since the pattern generalizes to any agent that touches the real world — posts, payments, deletes — it's worth writing down how I got here.

The 2 AM Test

Back when the bot was still in paper mode, it flagged a setup as unusual — Tier 3 in my system, "pause and wait for a button tap." The request went out at 2:14 AM. I was asleep, as humans are.

Version one of the gate treated "no answer" as "still pending." The intent sat there. Polite. Patient. And because my executor is single-threaded by design — one intent at a time, in order — everything queued behind it sat there too. I woke up to a stopped pipeline and a missing morning briefing, because the briefing job was stuck behind the stuck job.

Here's the thing I had to admit to myself: a blocked agent isn't a safe agent. It's a stopped one. "Doing nothing" feels like the conservative failure mode until you realize the bot also stopped managing the positions it already had open. Safe would have been: reject, move on, keep running.

So version two overcorrected in the opposite direction: no answer → ask again, every 15 minutes, until answered. Reminders, right? Humans like reminders.

I woke up to 28 notifications about the same trade. I didn't read notification four, let alone twenty-eight. I swiped the fourth away half-awake and then spent a genuinely dangerous moment trying to remember whether what I'd swiped was approve or dismiss. That's when it clicked: a notification channel that nags gets muted, and a muted approval channel is worse than none — because it still feels like oversight.

Version three is what runs now:

Ask once, with the full intent, the reasoning, and a deadline in plain text: "Reply by 03:14 or this is rejected."
One reminder at the halfway mark. Then silence.
Deadline passes with no reply → the intent is rejected, written to the audit log as rejected_timeout, and listed in the next morning briefing.
The queue never blocks on an approval. The next job runs immediately.

That night, the bot asked, I slept, the trade didn't happen, and the briefing told me about it over coffee with the reasoning attached. I read it, decided the bot was right to want the trade, and entered it manually an hour later when the setup was still valid. Nothing was lost except the illusion that 2 AM me makes good decisions.

The Code, Which Is Embarrassingly Short

The whole gate is maybe thirty lines. This is the core of it:

class ApprovalGate:
    WAIT         = 3600   # total patience, in seconds
    REMIND_AFTER = 1800   # one nudge, then silence

    def request(self, intent):
        deadline = now() + self.WAIT
        self.send(intent,
                  f"Reply by {fmt(deadline)} or this is rejected. "
                  f"No reply = no trade.")
        state = self.await_reply(deadline, remind_after=self.REMIND_AFTER)

        if state is UNANSWERED:
            self.audit.log(intent, outcome="rejected_timeout")
            return Denied("timeout means no")

        if state is REJECTED:
            self.audit.log(intent, outcome="rejected_by_human")
            return Denied("human said no")

        return Approved(state.edits)  # edits may shrink the intent, never grow it

Two details in there matter more than the mechanics:

The deadline is in the message. "No reply = no trade" is written down, every time. The system's behavior when I'm absent isn't a surprise — it's a stated contract inside the request itself.

Approvals can only shrink. When I reply, I can approve, reject, or approve at a smaller size. There is no path where my reply makes the action bigger. The gate edits down or blocks; it never amplifies. That asymmetry closes a whole class of "I replied with a typo and the bot did something heroic" incidents.

"Timeout Means Yes" Is Always One Boolean Away

Nobody writes if timeout: go_ahead() on purpose. But the pressure is real, and it never announces itself as recklessness. It sounds like: "the pipeline keeps stalling overnight, let's add a default," or "if he doesn't respond in an hour, he probably would have said yes." One boolean, one config value, and your fail-closed gate is fail-open — and nothing about the system looks different until the night it matters.

The reframe that fixed my thinking: an unanswered request is information. The human is asleep, or busy, or deliberately ignoring you. All three of those are "no." Not "maybe." Not "retry." No.

What the Logs Taught Me

Two numbers tell you whether a gate like this is healthy.

How often it asks. Tier 3 requests should be rare — mine currently fire about once a week. If they fire daily, the problem isn't the human, it's upstream: your thresholds and caps are miscalibrated and the gate is absorbing the overflow. A gate that asks constantly gets rubber-stamped, and reflex-tapping approve is the terminal failure state of every approval system ever built. Gate volume is a health metric for the whole stack.

How often I disagree with the rejects. The rejection log is a free dataset on where the bot's world model and reality diverge. In the first two weeks of logging timeouts, I noticed the bot kept flagging perfectly ordinary setups as unusual. Turned out a volatility threshold was stale and everything looked scary to it. The gate didn't just prevent bad trades — the rejects pointed me at a config bug I'd been living with. Rejections aren't noise in the system. They're the system talking.

One more, and I'm embarrassed by this one: I originally logged timeout-rejections as warnings, so my health monitor kept paging me about them. Wrong category. A timeout-rejection is the system working exactly as designed — it belongs in the routine morning briefing, not in alerting. The alert should fire only when the gate can't reach me at all. If your monitoring treats correct behavior as an incident, you'll mute the monitoring, and then the real incidents arrive as surprises.

It's a Decision Record, Not a Security Boundary

One nuance worth being precise about, because approval flows have a way of quietly becoming the only thing between an agent and disaster: the gate is not what keeps me safe. The executor's hard caps are. Even with a forged or fat-fingered "yes," the position-size cap, the exposure cap, and the daily-loss cap still bound what can happen. The gate decides whether; the caps decide how much. Two systems, one boring outcome.

It's the same reason the gate plays nicely with the dead-man switch: if the bot's heartbeat drops, pending approvals auto-reject along with everything else — a half-alive bot's outstanding questions can't be trusted either.

The Pattern, Without the Trading

You don't need a bot that moves money for this to apply. Anything an agent does that's hard to undo — publishing a post, sending an email on your behalf, deleting records, touching production — fits the same three-tier table:

Reversible → act alone, log everything
Bounded and routine → act alone, but send the receipt first
Unusual or irreversible → ask once, remind once, silence means no

And the one line worth stealing, in any stack: timeout means no, and "no" is a first-class outcome, not an error.

Have you wired approval flows into your agents? I'm curious how other people handle the "human is asleep" case — bounded waits, escalation to a second human, or just letting it queue until morning? Drop a comment, genuinely curious.

Part of my Building in Public series — previously: the guardrail stack I built before going live, a postmortem of a double-fired cron job, a circuit breaker for agent outages, and a health monitor that tells me when my agents are dying.

My Paper Trading Bot Went Live This Week — Here's the Guardrail Stack I Built First

Sam Hartley — Sat, 19 Sep 2026 08:36:16 +0000

My Paper Trading Bot Went Live This Week — Here's the Guardrail Stack I Built First

Confession time: my trading bot has been "almost ready for real money" for about four months.

Classic side-project arc. I built a scanner, it found opportunities, it paper-traded them beautifully. The equity curve went up and to the right. I showed a friend the chart. He asked the awkward question: "So when are you turning it on for real?"

The honest answer was: not until it bores me.

Paper trading tells you whether your logic works. It tells you nothing about what happens when your logic is wrong at 3 AM with real money on the line. So before flipping the switch, I spent a week building the boring parts — the guardrails. This is that stack, roughly ordered by how paranoid each one is.

What Paper Trading Actually Tests (Spoiler: Not the Scary Parts)

Paper mode is a simulator with the serial numbers filed off. Mine was optimistic in four specific ways:

No slippage. Paper fills happen at the displayed price. Real fills happen at whatever the order book feels like.
No fees. Small strategies live and die on fees. Paper mode is a world where the house doesn't take a cut.
Infinite liquidity. Paper mode never gets a partial fill or a rejected order.
No consequences. The big one. In paper mode, a bug is a curiosity. In live mode, a bug is an incident.

I've already lived the last one in miniature: my scheduler once double-fired and placed the same order twice. In paper, that's a weird chart. Live, that's exposure you never chose to take.

So before going live, I built six guardrails. None of them are clever. All of them are boring. That's the point.

Guardrail 0: The Model Doesn't Decide What It's Allowed to Do

Quick architecture note, because everything below depends on it.

My setup separates the brain from the hands. The "brain" — an LLM doing analysis — never talks to the exchange. It writes an intent: "open position in X, size S, stop at level L." A separate, dumb executor picks intents up. The executor is ~200 lines of Python with no model in it. The model can hallucinate whatever it wants; the executor only knows what's on the allowlist.

If your agent stack has the model calling money-moving APIs directly, this is the first thing I'd change. The part that decides should be as far away as possible from the part that does.

Guardrail 1: Hard Budget Caps, Enforced in the Executor

The executor has three numbers burned into it: a max position size, a max total exposure, and a max daily loss. Any intent that would breach one is rejected and logged — no model consultation, no retry, no "are you sure?"

class Limits:
    MAX_POSITION  = 50    # % of allocated capital, single position
    MAX_EXPOSURE  = 200   # % total, including leverage
    MAX_DAILY_LOSS = 3    # % — breach freezes everything

def check_intent(intent, state):
    if intent.size_pct > Limits.MAX_POSITION:
        return Reject("position too big")
    if state.exposure_pct + intent.size_pct > Limits.MAX_EXPOSURE:
        return Reject("exposure cap")
    if state.day_pnl_pct <= -Limits.MAX_DAILY_LOSS:
        return Reject("daily loss cap tripped — frozen until reset")
    return OK

The daily loss cap is the one I care about most. Strategies don't fail slowly — they fail on one bad day when everything correlates. The cap turns "my bot blew up" into "my bot stopped."

Guardrail 2: Rate Limits and Cooldown Windows

Autonomous agents in a loop are fast. Humans are not. So the executor enforces pace:

Max N new positions per hour. Mine is a single-digit number, on purpose.
Cooldown after any closed loss. No instant re-entry — that's how revenge trades happen, and bots are not immune to the sunk-cost loop. They just express it in code.
Consecutive-loss circuit break. After M losses in a day, the bot goes read-only until I flip it back on.

That last rule wasn't in v1. It is now, because I once watched my paper bot "scale into" a losing position five times in a row. It was following its rules perfectly. The rules were the problem.

Guardrail 3: Idempotency Keys on Every Action

Every intent gets a unique key before execution, and the executor checks it against a ledger of keys it has already acted on:

def execute(intent):
    key = f"{intent.day}:{intent.asset}:{intent.action}"
    if ledger.seen(key):
        return Skipped("duplicate intent")
    ledger.record(key)            # written BEFORE acting
    return broker.place(intent)

Two details matter. First, the key is derived from the decision (day, asset, action), not from a random ID — so a replayed or double-fired job produces the same key and gets dropped, which is exactly what saved me after the double-fire incident. Second, the key is recorded before the action, not after. If the process dies between the write and the fill, the retry gets skipped — I'd rather reconcile one orphan intent than double-spend.

Guardrail 4: Action Tiers With an Approval Gate

Not all actions are equally scary, so they get sorted into tiers:

Tier 1 — reversible: scans, reads, paper actions. The bot does these freely.
Tier 2 — bounded: small real positions, inside the caps. The bot does these alone, but I get a message with the full intent before execution.
Tier 3 — unusual or irreversible: anything at the caps, anything outside normal parameters, anything the bot itself flags as low-confidence. This tier pauses and waits for a button tap on my phone.

The approval message shows the full intent, the reasoning, and — this is the important part — what happens if I don't answer. The default is nothing. Timeout equals rejection. An unanswered request must never silently become a yes.

Tier 3 has caught exactly zero dramatic events so far. That's the correct number. The gate exists for the day something almost makes sense.

Guardrail 5: The Dead-Man Switch

The bot must check in on a heartbeat. Missed check-ins, and it freezes itself:

HEARTBEAT_EVERY = 300   # seconds
FREEZE_AFTER   = 3      # missed beats

def on_tick():
    if missed_beats() >= FREEZE_AFTER:
        freeze(reason="dead-man switch")
    if hung_jobs() > 0:
        freeze(reason="job stuck — state is suspect")
    beat()

The insight I stole from running a health monitor: a hung bot is more dangerous than a dead one. A dead bot does nothing — annoying, but safe. A half-alive bot might fire one leg of a two-step plan and never execute the other half.

So the freeze triggers on ambiguity too, not just silence. A stuck job means state can't be trusted. Freezing means: no new opens. Existing positions still get managed — stops always work — but nothing new starts. And there's a kill command that freezes everything from my phone, tested weekly, because an untested kill switch is a decorative kill switch.

Guardrail 6: An Audit Log Written Before Anything Happens

Every intent lands in a local log — timestamp, full payload, the reasoning — before execution, then gets updated with the result. Every morning I get a briefing message with a reconciliation: what the log says vs. what the account says.

If those two ever disagree, the bot assumes its own state is wrong, freezes, and waits for me. The account is the source of truth. The log is the source of intent. My job is reading both.

Week One, Honestly

So how's the first week of real money? Deeply educational, in the way a small electric shock is educational:

Fees and slippage turned "up and to the right" into "up and slightly to the right." The strategy still works, just with worse numbers. Better to learn that at small size than large.
The dead-man switch fired once. A job wedged itself after a network blip, the bot froze, and I spent an evening untangling it. Annoying — and exactly the failure mode it was built for. I came home to a frozen bot, not a runaway one.
The approval gate caught nothing dramatic. The correct amount of drama is zero.

The strangest part is psychological. In paper mode, I checked the bot obsessively, because nothing was watching it but me. Now I check it less — because the guardrails already asked the questions I would have asked.

What's Next: Shadow Mode

One more thing I'm adding for the next month: every live intent also gets mirrored into paper mode with the same timestamps. At the end, I diff them — paper fill vs. real fill, paper P&L vs. real P&L. That gap is the true cost of going live, measured instead of guessed. If it shrinks, my execution is tight. If it grows, the bot is paying a hidden tax I can hunt down.

If you're running agents that touch the real world — orders, payments, posts, deletions — the caps, the tiers, and the dead-man switch are the pieces I'd build first. All of it is a few hundred lines of deeply boring Python, and it converts "I hope it behaves" into "when it misbehaves, it stops."

Where do you draw the line — full autonomy, approval gates, or still in paper mode? Drop a comment, I'm genuinely curious.

Part of my series on running AI on my own machines — previously: a circuit breaker for agent outages, a postmortem of a double-fired job, and a health monitor for my agents.

I Added a Circuit Breaker to My AI Agent Setup — It Caught Three Outages This Month

Sam Hartley — Tue, 15 Sep 2026 08:34:53 +0000

I have a confession: for the longest time, my local AI setup had a very embarrassing failure mode.

One of my machines goes down — a reboot, a driver update, someone trips over a power cable — and the agents that depend on it just keep firing requests at it. Every single request times out. Each one burns 30-60 seconds waiting for a connection that will never succeed. Meanwhile the task queue piles up, retries stack on retries, and by the time the machine is back, I have a swamp of failed jobs to clean up.

The fix took an afternoon: a circuit breaker pattern. If you've never heard the term, it comes from electrical engineering — when too much current flows, the breaker trips and the circuit opens. No drama, no fire. You flip it back when things are safe.

The software version does the same thing for network calls. After X consecutive failures, stop calling. Fail fast, fall back to something that works, and check periodically whether the dead service is back instead of hammering it with every request.

Since I added it, it's caught three real outages this month. Here's what I learned building it.

My Setup (The Short Version)

For context: I run a small AI lab — a Mac as the orchestrator, a WinPC with a GPU for heavy inference, and an old UbuntuPC for background services. The orchestrator has agents that route tasks to the other machines: code generation goes to the GPU box, quick chat stays local, image parsing goes to the UbuntuPC, and so on.

The routing itself is a dumb little Python function with a model registry and keyword matching. It's the piece that decides where each request goes. What it didn't have until recently was any concept of whether the destination is actually alive.

The Problem, In Numbers

Here's what a typical outage looked like before:

The GPU machine reboots for a Windows update at 14:02
Agent jobs keep hitting it: image generation, code refactors, embeddings
Each request has a 30-second timeout
The KI Bot retries failed requests up to 3 times
So one failed job = up to 2 minutes of pure waiting, then a cascade of retry jobs

On a bad day, a 15-minute outage would produce 40+ queued tasks that all tried the dead endpoint, failed, and retried — burning a couple of hours of queue time on nothing. And since some of those jobs had follow-up jobs ("after generating the images, post them to X"), the failures propagated downstream. I'd come back to find half my evening pipeline in a error state because Windows decided 14:02 was a great time to update.

The worst part: the requests that should have just gone somewhere else — the UbuntuPC runs the same models, just slower on CPU — sat in the same queue behind the doomed ones.

The Circuit Breaker, Minimal Version

The pattern has three states. I stole this directly from the classic description (it's all over the resilience literature):

Closed — normal operation. Requests pass through. Failures are counted.
Open — tripped. Requests fail immediately, no network call at all. This is the "stop hammering the dead thing" state.
Half-open — after a cooldown, let one request through as a probe. If it succeeds, close the breaker. If it fails, stay open.

Here's roughly what I run per endpoint (simplified, but functionally what's in production):

import time

class CircuitBreaker:
    def __init__(self, name, threshold=3, cooldown=60, timeout=5):
        self.name = name
        self.threshold = threshold        # consecutive failures before tripping
        self.cooldown = cooldown          # seconds before probing again
        self.timeout = timeout
        self.failures = 0
        self.state = "closed"            # closed | open | half-open
        self.opened_at = None

    def allow_request(self):
        if self.state == "closed":
            return True
        if self.state == "open":
            if time.monotonic() - self.opened_at >= self.cooldown:
                self.state = "half-open"  # probe time
                return True
            return False                  # fail fast, no network call
        return False                      # a probe is already in flight

    def record_success(self):
        self.failures = 0
        self.state = "closed"

    def record_failure(self):
        self.failures += 1
        if self.state == "half-open" or self.failures >= self.threshold:
            self._trip()

    def _trip(self):
        self.state = "open"
        self.opened_at = time.monotonic()
        self.failures = 0

And in the request wrapper (again, simplified):

def call_endpoint(breaker, url, payload):
    if not breaker.allow_request():
        raise CircuitOpen(f"{breaker.name} is tripped, failing fast")

    try:
        resp = requests.post(url, json=payload, timeout=breaker.timeout)
        resp.raise_for_status()
        breaker.record_success()
        return resp.json()
    except Exception:
        breaker.record_failure()
        raise

One breaker instance per endpoint, keyed in a dict. The router that decides which machine handles which task now consults the breaker before doing anything else.

Note the aggressive timeout — 5 seconds, not 60. When a machine is genuinely there, a health probe responds in well under a second. Long timeouts are for slow responses, not dead machines. Distinguishing those two cases is half the value.

The Part That Made It Actually Useful: Fallbacks

A circuit breaker that just fails fast is... okay. Your errors arrive sooner. Great. Congratulations.

The real win comes from pairing the breaker with your fallback chain. My router already knew fallbacks existed — big model down? use the small one. GPU box rented out? CPU box takes over. What it lacked was the signal to switch. The breaker provides exactly that signal, in constant time, without a single wasted network call.

So the flow became:

Router picks the destination for a task
Is the breaker for that destination open? If yes, use the fallback destination immediately
Otherwise, call with a short timeout; failures count toward the threshold
If the breaker trips mid-flight, the next request reroutes — not to a retry queue, but to the fallback

A 30B code model being unavailable now means requests degrade to the 4B model on the Mac. Code quality drops slightly. The pipeline keeps flowing. Image jobs meant for the GPU get routed to the UbuntuPC and take 3 minutes instead of 20 seconds. Still flowing.

Three Outages, Zero Cascade Failures

I've had this running for a bit over a month now. Three times the breaker earned its keep:

1. The Windows Update (14:02, of course). GPU box gone for 17 minutes. Before the breaker: ~40 doomed tasks, cascading failures, an evening of cleanup. With the breaker: it tripped after 3 failures, code and image tasks rerouted to the fallback chain, and when the box came back the half-open probe re-closed it within a minute. I didn't even notice until I read the logs.

2. The flaky PCIe riser. One of my GPU cards had a flaky riser connection and the machine hard-froze. When a box is hung (not off), connections don't get refused — they just hang there. The 5-second timeout is what saved this one: three hangs, breaker trips, traffic reroutes. Without the short timeout, every request would have sat there for 60 seconds apiece, pretending to work.

3. The UbuntuPC's full disk. A background service filled the disk with logs, and the service started returning errors while the machine itself stayed up. The breaker tripped on HTTP errors just like it does on connection failures — the pattern doesn't care why the endpoint is failing, only that it is. Tasks that needed the vision service waited for the half-open probe instead of piling up garbage.

Three outages. Zero cascade failures. Zero evenings spent cleaning up retry swamps.

What I Got Wrong First Try

I counted total failures instead of consecutive ones. My first version tripped the breaker after N failures ever, which meant a busy day with a handful of one-off timeouts could trip it even though the endpoint was healthy. Consecutive-failure counting with reset-on-success is the fix. One-off failures happen; the breaker should ignore them.

My cooldown was too short. I started with 10 seconds, which meant a dead endpoint got probed every 10 seconds — fine — but a slowly recovering endpoint (say, a machine that just booted and is still loading a 30B model into VRAM) would get marked healthy by a TCP-level probe, then fail on the actual request and re-trip. 60 seconds worked better. If the first real request after recovery is heavy, give it room to finish loading.

I forgot to reset breakers on deploy. The breaker state lives in memory. When I shipped a new version of the router, all breakers came up closed while the actual machine was still down — instant trip, three failures wasted. The state should be persisted (or at least seeded from a health check at boot). It's on my list.

Fail-fast errors need to be distinguishable. When a task fails because "the breaker is open," that's a different failure than "the endpoint returned garbage." My first version raised the same generic exception for both, which made my alerting noisy and my logs confusing. One extra exception class fixed it. Now "breaker open" is not even an alert — it's the expected state during a known outage. The alert fires when a breaker trips, not on every request it rejects.

The Dashboard Bit

Because I was already logging breaker trips, I added one more thing: a daily message to my Telegram with any breakers that tripped in the last 24h and why.

That message has quietly become one of the most useful signals in my whole setup. Not because outages are frequent — they're not — but because it tells me which machine is the flaky one this month. September it was the riser. Last month it was power settings putting a box to sleep mid-job. The pattern in the trips is a pattern in my hardware.

Should You Build One?

If you have agents calling services and you have no failure isolation between them — yes. Honestly, it's the kind of thing that sounds like Enterprise Java Architecture Astronautics until the first time it saves your evening, and then it's just plumbing.

You don't need a library. The class above is ~40 lines and covers 90% of the value. If you want batteries included, there are solid packages out there — but read their retry/timeout defaults carefully, because a circuit breaker that waits 60 seconds per failure isn't protecting you from much.

The mental model that made it click for me: retries ask "will it work this time?" A circuit breaker asks "is it worth asking?" During an outage, the answer is no, and knowing that up front is worth more than any amount of persistence.

Have you wired failure isolation into your agents, or are you still in the retry-swamp phase? I'm curious what patterns people are using for multi-machine setups — drop a comment.

This is part of my series on running AI entirely on my own machines — a router for picking the right model per task, getting real work out of small local models, and evaluating local models before trusting them.

My Cron Job Double-Fired and Placed the Same Order Twice — Here's My Postmortem

Sam Hartley — Sun, 13 Sep 2026 08:20:58 +0000

I keep a Mac in a closet running a handful of small scheduled Python jobs. One of them scans market data on a schedule and, when its rules fire, places tiny orders through an exchange API. Positions are around $12 each. It ran for months without drama.

Then one morning the exchange said I held 16 contracts, and my local state file said 8.

Not a rounding error. Not a stale cache. The same order, placed twice, ten seconds apart — by two instances of the same script that were both alive at the same time, each keeping its own private idea of reality.

What follows is the postmortem I wrote for myself, cleaned up. The punchline: the scheduler was fine. The API was fine. Both root causes were me.

The morning I noticed two of everything

Context first, because the boring design matters later:

A Python script runs on a schedule (and occasionally on demand)
It fetches about a thousand tickers, applies some rules, sometimes opens or closes a small position
Every decision lands in a JSON state file; all my reports read from that file

That morning the fee log showed two order fees for the same position, timestamps ten seconds apart. The exchange held 16 contracts. My state file tracked 8. The state file wasn't lying about what it knew — it had only ever been told about one of the two orders.

I eventually found two distinct ways this had happened. Both were my fault, in different ways.

Incident 1: I restarted a job that was never dead

The first one is embarrassing in the most ordinary way possible.

The scan isn't instant — a thousand tickers plus math takes a bit. One evening I triggered a manual run, and the terminal just sat there. No output. No prompt back. "Hung," I thought — and ran it again.

Here's what actually happens when a chatty Python script runs with stdout connected to a pipe or a background handle instead of a real terminal: output is block-buffered. The job was alive and working the whole time. Its output was sitting in a buffer, waiting to be flushed. I read the silence as "dead," and my restart created a second live instance.

Those two order fees, ten seconds apart? One order from the run I thought had died, one from the resurrection.

The lesson that stings: silence is not a signal. "It's not printing" and "it's not running" are unrelated statements. Before you rerun anything: check the process table, check the log file's modification time, check whether it's burning CPU. Poll the thing. Re-running a silent-but-alive job is how you manufacture concurrency you never designed for.

Incident 2: The scheduler and I raced each other

The second one took an evening of forensics.

A few days later: same symptom, but one position was triplicated. The order timestamps lined up with the top of the hour — and my scheduler fires at :02. In my shell history, right there: a manual run of the same script at :02. I'd been poking around and wanted a fresh scan "right now," and it simply never occurred to me that the clock was about to do the same thing.

Two entry points, same script, same minute, zero coordination. A scheduler is just a clock — it doesn't know what you're doing in a terminal, and I wasn't thinking about the clock.

The rule I added: manual runs never land on a scheduled minute. If the job runs at :02, I run manual checks at :15. It feels comically simple. It has prevented at least two repeats that I know of.

Why this wasn't a disaster (this time)

I want to be honest here, because the honest version is more useful than the heroic one. Three things saved me, and none of them was foresight:

My exits are close-all. The exit logic closes the entire position on the exchange — not "the size I think I have." A duplicated position resolves itself at the next exit. The bug was self-healing by accident.
The money was small. $12 doubling to $24 is a shrug. The same code with position sizes that scale with an account would not be a shrug.
The exchange is the source of truth. My reports were wrong and my history file had gaps, but the actual money was always consistent with reality. The drift lived in my shadow copy — the state file — not in the world.

That third point rewired how I think about every state file I keep. They're caches. The API on the other side is the truth. Once that lands, "reconcile before acting" stops being an enterprise buzzphrase and becomes just reading the real number before making a decision.

The fixes

A lockfile — the boring kind

The core fix is about fifteen lines at the top of main():

import fcntl, os, sys

LOCK = os.path.join(os.path.dirname(os.path.abspath(__file__)), ".job.lock")
lock_fd = open(LOCK, "w")
try:
    fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
    print(f"[lock] another instance holds {LOCK}; exiting", flush=True)
    sys.exit(0)
lock_fd.write(str(os.getpid()))
lock_fd.flush()

A second instance starts, tries to grab the lock, fails, exits with a message. On Linux and macOS that's fcntl; on Windows it's msvcrt.locking — same idea, different syscall. And if you'd rather not touch Python for it, the shell version is one line:

flock -n /tmp/job.lock python3 job.py || echo "already running"

Announce yourself in the logs

Every run now logs start and end with its PID:

import logging

logging.basicConfig(
    filename="job.log",
    format="%(asctime)s pid=%(process)d %(levelname)s %(message)s",
)
logging.info("scan start")

When the lockfile blocks a second instance, the log says so in plain words. And looking back at the incident logs now, the duplicate jumps out in ten seconds — two different pid= numbers in the same minute — instead of needing an evening of timestamp archaeology.

Flush like you mean it

flush=True on prints — or just run with python3 -u. This one directly kills Incident 1: the terminal would have shown live output, and "no output" would have actually meant something.

Reconcile before acting

Before the job opens anything, it now pulls actual positions from the exchange API and treats the local state file as a hint, not as truth. Anything the file claims that the API doesn't confirm gets flagged for review instead of acted on.

The part I'd been getting wrong for years

The reframe that made all of this click:

I used to think of idempotency as an API-design topic — something server people worry about. But the moment your cron job calls an API that mutates state, you're running a distributed system. Multiple processes, lying transports (buffered stdout counts as a lying transport — it lied straight to my eyes), shared mutable state on the other side of a network. All the classic distributed-systems failure modes are now your failure modes. Just at hobby scale.

A lockfile doesn't make a distributed system safe. What it does is collapse the most common failure — scheduler plus human, or human plus impatient human — into a no-op. That's most of the practical risk, for fifteen lines of code.

Wrapping up

If you take away two things:

A silent job is not a dead job. Poll, don't rerun.
If a scheduled job of yours talks to anything that mutates state, add the lockfile today — not after your own double-fire. Mine cost an evening of forensics and $12 of unplanned exposure. The same bug with real sizing would have cost a weekend and my confidence in the whole setup.

I'm curious how others handle this, especially anyone running scheduled jobs against mutating APIs on a homelab box. Lockfiles, per-action idempotency keys, a real job queue instead of cron? Drop what's actually held up for you in the comments — I'm always stealing ideas for my checklist.

DEV Community: Sam Hartley

The Test Wasn't Wrong. It Was Answering an Older Question.

The Test Wasn't Wrong. It Was Answering an Older Question.

Three kinds of red

The oracle that pinned a number

The check that requires exactly one thing to change

The driver that called the old signature

The dangerous fix

Rebaselining is a decision, not an edit

What I check now, in order

Four things I'd tell my past self

My Pipeline Went Silent for Five Hours — Every Job Failed the Same Quiet Way

My Pipeline Went Silent for Five Hours — Every Job Failed the Same Quiet Way

The clue was an absence

The thing everything depended on

Anatomy of a silent failure

The bug that made it worse

The fix, part one: every network call gets a deadline

The fix, part two: stop depending on the share

The fix, part three: a watchdog that alerts on absence

What I'd tell myself before the next outage

My Exit Rule Said '2 Periods' — My Code Was Counting Something Else Entirely

My Exit Rule Said '2 Periods' — My Code Was Counting Something Else Entirely

Why it survived so long

Fixing it by deriving, not counting

The bug the fix uncovered

How I now test a counter

My Drawdown Breaker Replayed to Zero Fires — Because It Was Watching a Number That Never Moved

My Drawdown Breaker Replayed to Zero Fires — Because It Was Watching a Number That Never Moves

"Zero" is an ambiguous result

Auditing the input

Re-running against a live input

The trap underneath the trap

How I now test a guard's input

The boring takeaway

My Margin Guard Was Reading Zero. Four Separate Bugs Kept It That Way.

My Margin Guard Was Reading Zero. Four Separate Bugs Kept It That Way.

The guard

Bug one: None became 0.0, and 0.0 means "you're fine"

Bug two: the number, when it existed, was in the wrong unit

Bug three: the exchange told me the truth, and the truth was zero

Bug four: the guard ran once, the loop opened five

The demonstration

The fix

The test that actually proved it

What's still imperfect

What I'd copy

A Single int() Silently Disabled My Entire Risk System

A Single int() Silently Disabled My Entire Risk System

The ladder

The part that made me sit down

Why my tests never caught it

How I found the boundary

What I changed

The general version

My Agent Spent $23 Overnight. Here's Why My Limits Held and My Alerts Didn't.

My Agent Spent $23 Overnight. Here's Why My Limits Held and My Alerts Didn't.

What the job does, briefly

The part I got right, mostly by accident

The cap is not the alert

The second bug: the loop had no shape

Two metrics that actually predict this

What I'd tell you to copy

My Scheduled Agent Ran 40 Times and Did Nothing — Here's the Assertion That Fixed It

My Scheduled Agent Ran 40 Times and Did Nothing — Here's the Assertion That Fixed It

Errors page you. Silence doesn't.

Rule one: assert the effect, not the attempt

Rule two: expected-empty is not unexpected-empty

Rule three: three states, not two

The part that actually caught things

The transferable bit

Timeout Means No — The One Rule That Makes AI Agent Approval Gates Actually Work

The 2 AM Test

The Code, Which Is Embarrassingly Short

"Timeout Means Yes" Is Always One Boolean Away

What the Logs Taught Me

It's a Decision Record, Not a Security Boundary

The Pattern, Without the Trading

My Paper Trading Bot Went Live This Week — Here's the Guardrail Stack I Built First

My Paper Trading Bot Went Live This Week — Here's the Guardrail Stack I Built First

Bug one: `None` became `0.0`, and `0.0` means "you're fine"