DEV Community: Sam Hartley

My Margin Guard Was Reading Zero. Four Separate Bugs Kept It That Way.

Sam Hartley — Tue, 29 Sep 2026 08:32:22 +0000

My Margin Guard Was Reading Zero. Four Separate Bugs Kept It That Way.

There's a number I keep in my head from this build. Five positions open, $933.89 of my account tied up in margin, against a rule that says never more than 50%. The rule existed. The rule was running every hour. The rule read zero.

I want to write this one down carefully, because the fix is about four lines and the finding took a week — and because "the guard read zero" turned out to be four independent bugs that only worked as a team.

The guard

A scheduled scan wakes up every hour, looks at funding rates across a few hundred contracts, and can open up to three positions. Each position takes a fixed slice of equity as margin, and there's a cap: total margin in use should never cross 50% of the account.

The check is the shape of every risk guard I've ever written:

margin_cap = equity * 0.50
if used >= margin_cap - 0.01:
    return  # account is full; don't open anything else

A number, a threshold, a comparison. Simple enough that I stopped reading it. Which is exactly the profile of the bugs that hurt.

Bug one: `None` became `0.0`, and `0.0` means "you're fine"

used came out of the saved state:

used = state.get("margin_used") or 0.0

On a state where that key was missing — and it was missing from most of them — or 0.0 turned it into zero. That line reads like defensive programming. It's actually a safety failure with good manners. The guard's whole job is to compare a number; when the number is absent, the honest answer is I don't know, don't open. The code said zero, go ahead. A fail-open default in the one place where fail-closed is the entire point.

Bug two: the number, when it existed, was in the wrong unit

Behind the None bug sat a worse one. When margin_used was populated, it wasn't always margin — some code paths wrote the position's notional value (size × leverage) into it and then compared that against a cap expressed in account percentage. Two units, one comparison.

This is why the bug survived so long: a unit error doesn't crash. It just makes the guard either fire early (annoying, you notice) or fire never (quiet, you don't).

Bug three: the exchange told me the truth, and the truth was zero

At some point I "fixed" bug one by syncing margin_used from the exchange instead of trusting local state. That felt rigorous. It wasn't.

My account runs in cross-margin mode. In cross mode the collateral sits at the account level, not on each position — and the API returns position_margin = 0 and order_margin = 0 for every open position. I checked it read-only: three positions open, both fields zero, real cross initial margin $460.97. My new source of truth was structurally zero.

I had outsourced a safety decision to a field whose semantics I never verified in my own account mode. The field wasn't lying. It was answering a different question than the one my guard was asking.

Bug four: the guard ran once, the loop opened five

This is the one I'm least proud of, because it's a bug I've written about before in a different costume.

The guard was evaluated once per scan — before the entry loop, from a single sync — and then the loop went to work opening positions. Nothing re-checked. Whether it's a rate limiter, a budget, or a margin cap: if you read your limit before the loop and mutate your usage inside the loop, your limit is a snapshot, not a limit.

The demonstration

So I stopped guessing and ran it end-to-end against the real code: five identical entry signals in one scan, everything stubbed except the guard.

Old: the guard saw $0.00 on every iteration. Five positions opened. Margin in use: $933.89 — 83.5% of the account, against a 50% cap.

New: the guard saw 0 → 186.78 → 373.56 → 560.34. It blocked the fourth entry. Three positions, $560.34 — 50.1%.

The fix

One idea: derive the number from the positions I actually hold, in the unit the cap is actually in, and read it inside the loop where it matters.

def margin_in_use(positions):
    """Sum of the margin each open position actually ties up."""
    return sum(
        float(p.get("capital_at_entry", 0)) / max(float(p.get("leverage", 1)), 1.0)
        for p in positions
    )

for candidate in candidates:
    # re-read every iteration, from the live position list
    used = margin_in_use(get_active_positions(state))
    if used >= margin_cap - 0.01:
        break
    execute_entry(candidate)

The exchange numbers stay in the code, but only as diagnostics — logged, printed, never used to make the decision. The rule I took away: don't outsource a safety decision to a remote field whose semantics you haven't verified for your own account configuration.

The test that actually proved it

This is the part I'd hand to anyone building a guard like this.

My first test rebuilt the guard formula inside the test file and compared its output against itself. It passed. It proved nothing — a tautology with a green checkmark. A test that re-implements the logic under test is only measuring its own author.

So I rewrote it to pull the guard block verbatim out of the source file, run the same scenarios through both the unpatched and the patched version, and — this is the important bit — assert that the two disagree. A control probe. If patched and unpatched produce identical output, the test isn't sensitive to the change, and any pass is meaningless.

scenario: margin_used missing
  unpatched: stops after 2 positions   (buggy — should be blocked elsewhere)
  patched:   opens 3, then blocks      (correct)
  control:   patched != unpatched ✅

That control line has caught more bad tests for me than any assertion since. It's the same instinct as checking that a scheduled job actually did something: make the test fail when nothing changes, if you want it to mean something when it passes.

What's still imperfect

I'd rather write the remaining gap down than pretend it's closed. The guard checks used, not used + next_position. So the worst-case overshoot is one full slot: with 16.7% slots, three of them is 50.1% against a 50% cap — 0.1pp over, arithmetic, and deliberate. With 30% slots it would be +10pp, and that's a decision I'd revisit before I ever changed the slot size.

What I'd copy

If you have anything that enforces a limit — margin, spend, rate, quota — the pattern is boring, and it took me far too long to land on it:

Absence is not zero. A missing value in a safety check should stop the action, never permit it.
Check units, not just numbers. A notional value and a margin value can both be "the right number" and still be incomparable.
Don't trust a remote field you haven't verified in your own mode. Cross margin reports zero at the position level. That's not a bug in their API; it's a question I wasn't asking.
Re-read the limit inside the loop. A limit evaluated against a snapshot is a snapshot.
Make your test prove it can fail. Extract the logic under test, and assert that the old and new versions disagree.

The cap was correct on paper the entire time. What it needed was a number that meant what I thought it meant — and the humility to stop treating "no value" as good news.

Same question I keep asking on these: has anyone else been bitten by a guard whose default was the unsafe path? I keep going back and forth on whether a missing state value should halt the job entirely or degrade gracefully — drop a comment with what's worked for you.

Part of my Building in Public series — previously: budget caps that held at 2am, the assertion that catches runs that do nothing, timeout means no on approval gates, the guardrail stack before going live, and a circuit breaker that caught three outages.

A Single int() Silently Disabled My Entire Risk System

Sam Hartley — Sun, 27 Sep 2026 08:30:29 +0000

A Single int() Silently Disabled My Entire Risk System

I didn't find this bug by debugging. I found it by answering a question.

Someone asked me what leverage my live trading bot actually runs at. I said "one and a half times, sometimes two, depends on the funding spread" — because that's what the ladder in my code says, and I'd read that code recently. Then I decided to prove it instead of trusting my memory, and the answer was 1. Not 1.5. Not "up to 2". One. Every position, every time, for months.

The fix took four characters. Finding it took longer, because the code was correct — under a config value I'd changed two months earlier.

The ladder

My position sizing has a small leverage ladder. Wide funding spread plus a stable rate, size up; thin spread, stay at 1x.

LEVERAGE_LADDER = [1, 1.25, 1.5, 2]
MAX_LEVERAGE = 1.5

def calc_smart_leverage(rate, stability):
    idx = 0
    if rate > 0.05: idx += 1
    if stability > 0.8: idx += 1
    if rate > 0.15: idx += 2
    leverage = LEVERAGE_LADDER[min(idx, len(LEVERAGE_LADDER) - 1)]
    return int(leverage)   # <- here

The ladder returns floats. int() truncates them. At MAX_LEVERAGE = 2.5 — the value this code was written against — the truncation was invisible:

The ladder max is 2.0, the cap is 2.5, so int(2.0) == 2. Nothing got clipped.
The only value that could have been clipped was 2.5 itself, and the ladder never returned it.

Then, in July, I lowered the cap to 1.5, because the 2.5x era had produced a drawdown I didn't like. That one-line config change turned a dormant int() into an active bug: int(1.5) == 1. Every rung above 1x collapsed to the bottom of the ladder. The bot was permanently, silently at minimum leverage.

The part that made me sit down

Leverage being 1x instead of 2x wasn't the real problem. The real problem was a guard that read leverage to decide whether the risk logic should run at all:

def check_drawdown(state):
    has_leveraged = any(p["leverage"] > 1 for p in state["positions"])
    if not has_leveraged:
        return "normal"     # no leverage, no risk to compute
    ...  # the actual drawdown math

any(leverage > 1) was never true, because leverage was always 1. So check_drawdown() returned "normal" on the first line, every time, without computing anything. My drawdown breaker was not "not triggering". It was not running. In the live trader and the paper trader, for months.

That's the shape I want to write down: the guard didn't fail. It didn't raise. It returned the reassuring answer, which is the worst possible failure mode for a safety check.

Why my tests never caught it

I had tests. They passed.

def test_leverage_respects_cap():
    lev = calc_smart_leverage(rate=0.3, stability=0.95)
    assert lev <= MAX_LEVERAGE   # 1 <= 1.5, green

This asserts the wrong thing. It checks that leverage is bounded above, which was always true and always irrelevant. It never asserted that the levered path is reachable. When I lowered the cap, the assertion stayed green — correctly! — while the feature it was supposed to protect went to zero.

The invariant I actually cared about wasn't "leverage ≤ cap". It was "a high-conviction signal produces leverage > 1". Those are different claims, and only one of them fails when the code is broken.

How I found the boundary

I didn't want to reason about it. I wanted a number. So I ran both implementations against the same inputs with a stubbed API client — no orders, nothing live — and printed the leverage for a spread that should have maxed the ladder.

Paper: 1. Live: 1.

Then I grepped the trade history for the field. leverage: 2 appeared exactly 6 times out of roughly 440 trades, all of them at the very start, before the cap change. Every trade after that: 1. The history had been telling me for two months. I just hadn't asked it the right question.

What I changed

Three things, none of them clever:

The sizing function returns a float. No truncation. If the ladder says 1.5, the position gets 1.5.
The "should I run risk logic" decision is computed once, explicitly, from the ladder's actual output — not inferred from a value that a truncation bug, a rounding change, or a config edit can quietly flatten.
The tests assert reachability, not bounds: given a maximal signal, assert the levered branch executes. If someone lowers the cap below the lowest rung again, that test goes red.

The general version

Two things I'd offer anyone running scheduled or automated systems:

A guard clause that returns the safe answer is a silent kill switch. if not has_leveraged: return "normal" is the same class of bug as if not errors: return True. It looks defensive. It's the thing that hides the failure.

Config changes can change code semantics. The int() was written when the cap was 2.5. It was correct then. Nothing in the diff that lowered the cap to 1.5 touched that function, and the tests stayed green — because they tested the bound, and the bound still held. When you change a constant that other code reasons about, the code that reasons about it is part of the change, whether the diff says so or not.

I spent a weekend on a four-character bug. The four characters were int(. The weekend was for the two months I didn't know my risk system was off.

If you have a similar one — a "harmless" coercion or rounding that quietly took a whole branch with it — I'd genuinely like to hear it. Drop it in the comments.

My Agent Spent $23 Overnight. Here's Why My Limits Held and My Alerts Didn't.

Sam Hartley — Fri, 25 Sep 2026 08:30:50 +0000

My Agent Spent $23 Overnight. Here's Why My Limits Held and My Alerts Didn't.

I woke up to a notification that said my nightly agent had failed. That's the good version of a bad morning — something went wrong, and I knew about it.

Then I opened the bill. Twenty-three dollars on a cloud model, spent between roughly 01:40 and 04:10, on a job that normally costs about forty cents. So the thing I actually want to write about isn't the twenty-three dollars. It's the timeline, because the interesting part is what happened around 01:40.

The cap fired. The job stopped. The log has the entry. And I found out five hours later, from a different job that failed for an entirely separate reason.

What the job does, briefly

Every night at 01:30, a scheduled job reads my notes, pulls a few feeds, and writes a draft summary to a local file. Ninety-five percent of the work runs on a local model. A cloud call is there for one narrow escalation: if the local model reports low confidence on a section, the job re-runs just that section on a bigger model via API.

That escalation is supposed to fire a handful of times a night. On the night in question it fired continuously for two and a half hours, because of a retry loop I wrote, tested in the happy path, and never once tested with a model that kept saying "low confidence."

Each retry was cheap. The loop wasn't. That's the whole shape of the bug: nothing in it is expensive, and the product of it is.

The part I got right, mostly by accident

The spend cap held. I put a daily budget ceiling in the executor months ago, after I wrote up the guardrail stack I use before anything touches real money. Same three-number pattern: a per-run cap, a per-day cap, and a freeze when the day's cap trips.

The reason it held is a design choice I'd make again: the cap is mechanical, not statistical. It doesn't ask a model whether this call is reasonable. It doesn't consult a moving average. It checks a counter, and when the counter crosses a line, calls stop being made.

class Budget:
    """Spend caps. No model, no heuristics. A counter and a lock."""

    def __init__(self, per_run_usd, per_day_usd, store):
        self.per_run = per_run_usd
        self.per_day = per_day_usd
        self.store = store  # survives restarts; this is the whole trick

    def authorize(self, estimated_usd):
        spent_today = self.store.get("spent_today")   # from the ledger, not memory
        run_spent = self.store.get("run_spent")

        if run_spent + estimated_usd > self.per_run:
            raise Denied("per-run budget exceeded")
        if spent_today + estimated_usd > self.per_day:
            raise Denied("daily budget exceeded — freezing")

        self.store.incr("spent_today", estimated_usd)
        self.store.incr("run_spent", estimated_usd)

    def settle(self, actual_usd, estimated_usd):
        self.store.incr("spent_today", actual_usd - estimated_usd)
        self.store.incr("run_spent", actual_usd - estimated_usd)

Two details in there did all the work:

The counter lives in a store, not in a process variable. The first version of my cap kept spent_today in memory. Any restart — and my agent restarts a lot — reset the counter to zero. A cap that resets when the thing it's capping restarts is a cap made of wishes. Moving it to a small on-disk ledger was a four-line change and it's the difference between the cap working and the cap being theater.

Authorize before the call, settle after it. I reserve against the estimate, then correct to the actual. If the actual comes back bigger than the estimate, the ledger absorbs the difference — which means the next call is judged against reality, not against my optimism. My estimates were reliably low. That's not a bug I fixed, it's a bias I compensated for.

The cap is not the alert

Here's the mistake, and it's the reason this post exists.

When the daily cap tripped, my code did the correct thing: it raised Denied("daily budget exceeded — freezing"), wrote an audit entry, and returned. Clean, fail-closed, no drama.

And it logged that at INFO.

My alerting rules only looked at ERROR and above, plus a couple of specific failure classes I'd enumerated. So from the alerting system's point of view, the night was perfect. No error. Nothing to see. The one job that did notify me was the morning job, and it failed because the draft file it expected never got written — a downstream symptom that arrived five hours after the causal event.

The lesson generalizes past my stack: a fail-closed cap is a refusal, and refusals are not exceptions. From inside the system, refusing is the system working. From outside, refusing is the system not working. Only a human can tell you which — and the human needs to be told at the moment it happens.

The fix was one line of category, plus one line of routing:

# this was the bug, dressed as good hygiene
log.info("daily budget exceeded — freezing")

# what it is now: a refusal is an incident, not a status
log.warning("BUDGET_FROZEN", extra={"spent": spent_today, "cap": self.per_day})

And BUDGET_FROZEN is now on the alert list — not the page-me-now list, but the tell-me-immediately list. There's a real distinction there. A budget freeze at 01:40 doesn't mean wake the human; it means the pipeline is dead until morning, and the human gets to know before the morning job fails, with the number attached. Twenty-three dollars is a shrug. Twenty-three dollars discovered at 04:15 by a confused downstream job is a five-hour blind spot.

The second bug: the loop had no shape

The retry loop is a separate story and it's more embarrassing, so here it is quickly.

My escalation logic was: if confidence is low, retry on the bigger model. I tested it by mocking a model that returns "low confidence" once and then confidently. So in my head, the loop was a single hop.

In reality, the bigger model also said low confidence, because the section genuinely was ambiguous, and the section never got less ambiguous, because I never changed anything between attempts. It retried the same query against the same input with the same context and hoped for a different answer. That is, definitionally, the thing I have a circuit breaker for — and the breaker didn't catch it, because every individual call succeeded. A 200 is a success. The breaker watches for failure, and my loop produced nothing but successes until the money ran out.

So the retry loop got two additions:

Attempt cap. Three tries, hard stop. Not "until it works."
Escalation must change the input. Each retry has to add something — more context, a decomposition step, a different prompt. A retry that repeats the previous attempt byte-for-byte isn't a retry, it's a subscription.

And the effect-count check I now put on every job — the assertion that a run must prove it produced something — is exactly what would have caught this at 01:45 instead of 04:10. The job was producing effects, so it would have passed. But the loop had no cost assertion, and cost is an effect too if you're paying for it by the token.

Two metrics that actually predict this

Post-incident, I stopped watching "spend total" as my primary number and started watching two ratios:

Spend per unit of work. Dollars per draft, per filing, per summary. This is the number that would have been screaming. My absolute spend looked fine for the first hour — forty cents an hour is normal-ish. It was spend per draft that went vertical, because the job was producing zero drafts while burning tokens. Same shape as the flatline I wrote about in the silent-failure post: the aggregate looked normal, the ratio was on fire.

Retry ratio: retries divided by first attempts. Mine sits near 0.05 on a normal night. Anything above ~0.3 means a loop or a degraded model, and it's visible in minutes rather than dollars. Cheap to compute, no new infrastructure, and it's the only number that moved before the bill did.

One more, and it's the uncomfortable one: the alert you're not watching is the alert you've muted. I'd been getting budget-adjacent notices for weeks and had stopped reading them, because most were routine. That's not an alerting failure, it's a calibration failure — and it's the same terminal state I wrote about with approval gates that ask too often. Any channel that asks more than a few times a month stops being oversight. If you're drowning in routine notices, the fix is fewer notices, not more attention.

What I'd tell you to copy

If you run anything that spends money on your behalf — cloud LLM calls, API credits, compute — here's the short version:

Caps in the executor, not the model. A counter and a line. No model gets a vote on whether to spend.
Counters survive restarts. If it lives in memory, it isn't a cap.
Reserve, then settle. Judge the next call against the real number, not your estimate.
A refusal is an incident. Log it at a level your alerts can see, with the number in the message.
Cap your retries, and make each retry change something. A repeated call isn't a retry.
Watch the ratio, not the total. Spend per unit of work is the early-warning number; the total is the receipt.

The cap worked exactly as designed. The $23 was the cap doing its job — the difference between a runaway bill and a bounded one. The failure was that I didn't hear it close. A guardrail you don't get alerted about is a guardrail you're trusting on faith, and I'd rather have the boring version: it stops, and it tells me it stopped.

Curious how other people handle this — do you alert on budget refusals immediately, or roll them into a digest so you don't train yourself to ignore them? I keep going back and forth. Drop a comment with what's worked.

Part of my Building in Public series — previously: the assertion that catches runs that do nothing, timeout means no on approval gates, the guardrail stack before going live, and a circuit breaker that caught three outages.

My Scheduled Agent Ran 40 Times and Did Nothing — Here's the Assertion That Fixed It

Sam Hartley — Wed, 23 Sep 2026 08:30:57 +0000

My Scheduled Agent Ran 40 Times and Did Nothing — Here's the Assertion That Fixed It

I found this one by accident, which is the only way you find it.

Every night at 02:30, a job pulls new filings from an API, normalizes them, and drops them in a queue that feeds my morning briefing. It had been running for about six weeks. Every run: exit code 0. No errors in the log. The scheduler dashboard showed forty consecutive green runs. If you'd asked me, I'd have said that job was my most reliable piece of automation.

Then I noticed my briefing had gotten thin. Not empty — thin. Some days two items, some days none, and I'd been assuming "slow news." I finally got suspicious, went to the log, and realized I had absolutely nothing to be suspicious with: the log said fetched items, wrote to queue on every run, and that's it. No counts. No IDs. Just a sentence I'd written myself, once, and never verified.

So I added one line — a count — and reran it manually.

Zero. Every night. Forty times.

The HTTP call returned 200 with an empty array, because a parameter name in the query had been renamed on their side (since → from_date) and the API's response to an unknown parameter was to cheerfully ignore it. My for loop iterated over an empty list and completed successfully, which is exactly what a for loop over an empty list does. Nothing raised. Nothing retried. Nothing was logged as a problem, because as far as every layer of my stack was concerned, nothing was a problem.

That's the failure mode I want to write about, because I'd covered the other ones: I have a circuit breaker for hard failures, a health monitor for agents that are dying, and dedup for jobs that fire twice. All three of those watch for something happening. None of them watch for something not happening. And "not happening" doesn't throw.

Errors page you. Silence doesn't.

The reason this took six weeks is structural, not lazy. An exception has a moment: it fires, a handler catches it, an alert goes out. A silent no-op has no moment. It has a shape over time, and only if you're graphing the right thing.

My monitoring was graphing the wrong thing. I was tracking run status (success/failed) and run duration. Both were perfect. What I wasn't tracking was the only number that mattered: effects produced. Runs completed. Effects were zero. I had a dashboard that was green because it was measuring the system's opinion of itself.

The fix isn't more logging. It's a different question: not "did it run?" but "did it change anything?"

Rule one: assert the effect, not the attempt

Every job I write now has to declare what a successful run looks like in terms of side effects — rows written, messages sent, files created, IDs emitted. Then the job asserts it before it can report success.

def assert_effect(name, produced, expected=">=1"):
    if expected == ">=1" and produced < 1:
        raise JobProducedNothingError(
            f"{name}: ran fine, produced {produced} effects"
        )
    log.info("%s: effect ok (%s)", name, produced)

The point isn't the helper. The point is that a job can now fail for the crime of doing nothing. Before, "did nothing" and "did everything" were the same outcome — success — which means my success signal was measuring the wrong universe.

Two things I had to get right, and got wrong first:

An assertion that can't fail isn't an assertion. My first pass was assert len(rows) >= 0. That's a tautology wearing a safety vest. It passed on the empty list and I felt protected for another two days. The assertion has to be able to actually fail on the thing you're worried about — which sounds obvious until you write one at 1 AM.

Assert on the effect, not the attempt. resp.status_code == 200 is an attempt check. len(written_ids) > 0 is an effect check. The whole incident was a case where every attempt check passed and the effect was zero. If your assertion can be satisfied by a system that is doing nothing, it's not an assertion, it's decoration.

Rule two: expected-empty is not unexpected-empty

Here's where I nearly overcorrected, and it's the subtlest part.

Some of my jobs legitimately do nothing most of the time. A watcher that looks for new items in a feed should find zero items on a quiet day. If I make "zero effects" a hard failure everywhere, those jobs scream every night and I'm back to a muted alert channel — which I already learned is worse than no alerts.

So the assertion can't be "produced > 0". It has to be "the source was successfully queried and the emptiness is trustworthy." That's a different check: the request succeeded, returned the expected shape, and the emptiness was an explicit, well-formed empty rather than a default. In my case the giveaway was available the whole time: the response had no pagination envelope. A real empty result would have come back with a cursor and a total. A silently-ignored-parameter empty came back as a bare [].

Practical form: for jobs that may legitimately produce nothing, assert on the envelope, not the count.

data = api.get("/filings", params={"from_date": since})
if not isinstance(data, dict) or "items" not in data:
    raise UnexpectedShapeError(f"got {type(data)}: {str(data)[:80]}")
items = data["items"]  # zero here is fine — we proved the query was honored

Now "quiet day" and "the API stopped honoring my request" look different, which is the entire job.

Rule three: three states, not two

This is what finally made it usable for both kinds of jobs. Every run now ends in one of three states, and they are not the same thing:

succeeded_with_effect — it did the thing, with a count attached.
no_work_needed — it did nothing and proved that nothing was the correct answer (envelope check above passed).
failed — including JobProducedNothingError, i.e. it ran, and the absence of effect was not explained.

Only the third one alerts. The second one is a normal, boring, expected outcome that shows up in the briefing as a single quiet line. Before this split, "did nothing" was indistinguishable from "worked" — and the alerting I did have was tuned to a world where those were the same.

The part that actually caught things

Beyond the original bug, the effect counters caught two more silent failures in the next month: a token that expired and got silently replaced with an anonymous, lower-privilege session (200s, empty results), and a path change that made a downstream job read an empty directory and write a valid, empty file. Neither threw. Both were invisible to status-based monitoring. Both showed up instantly as a flatline in effects.

The dashboard line I now look at first isn't uptime or error rate. It's effects per run, per job. A job that runs and produces nothing, forever, is not a healthy job — it's a green light with the bulb unscrewed.

The transferable bit

If you take one thing: "no error" is not "worked." Your scheduler, your orchestrator, and your CI will happily report success for a process that queried nothing, wrote nothing, and sent nothing, because that is exactly what they were asked to measure.

Whatever your stack is — a nightly ETL, a webhook consumer, a CI step, an agent that posts things — pick one number per unit of work that would be zero if the work silently stopped. Assert on it. Track it over time. And make sure that some of your alerts can fire on a flatline, not just a spike. Spikes are loud and self-reporting. Flatlines need someone to be watching for them on purpose.

Curious how other people handle this: do you assert on effect counts in production jobs, or do you rely on downstream freshness checks (data arriving late → alert)? Both seem to have holes. Drop a comment with what's caught silent failures for you.

Part of my Building in Public series — previously: the guardrail stack I built before going live, timeout means no on approval gates, a circuit breaker that caught three outages, and a health monitor that tells me when my agents are dying.

Timeout Means No — The One Rule That Makes AI Agent Approval Gates Actually Work

Sam Hartley — Mon, 21 Sep 2026 08:33:08 +0000

My trading bot went live recently, and I wrote up the guardrail stack I built first: hard budget caps, idempotency keys, a dead-man switch, and an approval gate for anything unusual. One of those has quietly become my favorite, and it's the least technical of the bunch.

The approval gate. Not the asking-for-permission part — the part everyone, including me, gets wrong on the first try. The rule that makes the whole thing work:

If I don't answer, the answer is no.

Sounds obvious now. It wasn't. I shipped two versions of that gate before this one, both politely broken in ways that only show up when the human is asleep. Since the pattern generalizes to any agent that touches the real world — posts, payments, deletes — it's worth writing down how I got here.

The 2 AM Test

Back when the bot was still in paper mode, it flagged a setup as unusual — Tier 3 in my system, "pause and wait for a button tap." The request went out at 2:14 AM. I was asleep, as humans are.

Version one of the gate treated "no answer" as "still pending." The intent sat there. Polite. Patient. And because my executor is single-threaded by design — one intent at a time, in order — everything queued behind it sat there too. I woke up to a stopped pipeline and a missing morning briefing, because the briefing job was stuck behind the stuck job.

Here's the thing I had to admit to myself: a blocked agent isn't a safe agent. It's a stopped one. "Doing nothing" feels like the conservative failure mode until you realize the bot also stopped managing the positions it already had open. Safe would have been: reject, move on, keep running.

So version two overcorrected in the opposite direction: no answer → ask again, every 15 minutes, until answered. Reminders, right? Humans like reminders.

I woke up to 28 notifications about the same trade. I didn't read notification four, let alone twenty-eight. I swiped the fourth away half-awake and then spent a genuinely dangerous moment trying to remember whether what I'd swiped was approve or dismiss. That's when it clicked: a notification channel that nags gets muted, and a muted approval channel is worse than none — because it still feels like oversight.

Version three is what runs now:

Ask once, with the full intent, the reasoning, and a deadline in plain text: "Reply by 03:14 or this is rejected."
One reminder at the halfway mark. Then silence.
Deadline passes with no reply → the intent is rejected, written to the audit log as rejected_timeout, and listed in the next morning briefing.
The queue never blocks on an approval. The next job runs immediately.

That night, the bot asked, I slept, the trade didn't happen, and the briefing told me about it over coffee with the reasoning attached. I read it, decided the bot was right to want the trade, and entered it manually an hour later when the setup was still valid. Nothing was lost except the illusion that 2 AM me makes good decisions.

The Code, Which Is Embarrassingly Short

The whole gate is maybe thirty lines. This is the core of it:

class ApprovalGate:
    WAIT         = 3600   # total patience, in seconds
    REMIND_AFTER = 1800   # one nudge, then silence

    def request(self, intent):
        deadline = now() + self.WAIT
        self.send(intent,
                  f"Reply by {fmt(deadline)} or this is rejected. "
                  f"No reply = no trade.")
        state = self.await_reply(deadline, remind_after=self.REMIND_AFTER)

        if state is UNANSWERED:
            self.audit.log(intent, outcome="rejected_timeout")
            return Denied("timeout means no")

        if state is REJECTED:
            self.audit.log(intent, outcome="rejected_by_human")
            return Denied("human said no")

        return Approved(state.edits)  # edits may shrink the intent, never grow it

Two details in there matter more than the mechanics:

The deadline is in the message. "No reply = no trade" is written down, every time. The system's behavior when I'm absent isn't a surprise — it's a stated contract inside the request itself.

Approvals can only shrink. When I reply, I can approve, reject, or approve at a smaller size. There is no path where my reply makes the action bigger. The gate edits down or blocks; it never amplifies. That asymmetry closes a whole class of "I replied with a typo and the bot did something heroic" incidents.

"Timeout Means Yes" Is Always One Boolean Away

Nobody writes if timeout: go_ahead() on purpose. But the pressure is real, and it never announces itself as recklessness. It sounds like: "the pipeline keeps stalling overnight, let's add a default," or "if he doesn't respond in an hour, he probably would have said yes." One boolean, one config value, and your fail-closed gate is fail-open — and nothing about the system looks different until the night it matters.

The reframe that fixed my thinking: an unanswered request is information. The human is asleep, or busy, or deliberately ignoring you. All three of those are "no." Not "maybe." Not "retry." No.

What the Logs Taught Me

Two numbers tell you whether a gate like this is healthy.

How often it asks. Tier 3 requests should be rare — mine currently fire about once a week. If they fire daily, the problem isn't the human, it's upstream: your thresholds and caps are miscalibrated and the gate is absorbing the overflow. A gate that asks constantly gets rubber-stamped, and reflex-tapping approve is the terminal failure state of every approval system ever built. Gate volume is a health metric for the whole stack.

How often I disagree with the rejects. The rejection log is a free dataset on where the bot's world model and reality diverge. In the first two weeks of logging timeouts, I noticed the bot kept flagging perfectly ordinary setups as unusual. Turned out a volatility threshold was stale and everything looked scary to it. The gate didn't just prevent bad trades — the rejects pointed me at a config bug I'd been living with. Rejections aren't noise in the system. They're the system talking.

One more, and I'm embarrassed by this one: I originally logged timeout-rejections as warnings, so my health monitor kept paging me about them. Wrong category. A timeout-rejection is the system working exactly as designed — it belongs in the routine morning briefing, not in alerting. The alert should fire only when the gate can't reach me at all. If your monitoring treats correct behavior as an incident, you'll mute the monitoring, and then the real incidents arrive as surprises.

It's a Decision Record, Not a Security Boundary

One nuance worth being precise about, because approval flows have a way of quietly becoming the only thing between an agent and disaster: the gate is not what keeps me safe. The executor's hard caps are. Even with a forged or fat-fingered "yes," the position-size cap, the exposure cap, and the daily-loss cap still bound what can happen. The gate decides whether; the caps decide how much. Two systems, one boring outcome.

It's the same reason the gate plays nicely with the dead-man switch: if the bot's heartbeat drops, pending approvals auto-reject along with everything else — a half-alive bot's outstanding questions can't be trusted either.

The Pattern, Without the Trading

You don't need a bot that moves money for this to apply. Anything an agent does that's hard to undo — publishing a post, sending an email on your behalf, deleting records, touching production — fits the same three-tier table:

Reversible → act alone, log everything
Bounded and routine → act alone, but send the receipt first
Unusual or irreversible → ask once, remind once, silence means no

And the one line worth stealing, in any stack: timeout means no, and "no" is a first-class outcome, not an error.

Have you wired approval flows into your agents? I'm curious how other people handle the "human is asleep" case — bounded waits, escalation to a second human, or just letting it queue until morning? Drop a comment, genuinely curious.

Part of my Building in Public series — previously: the guardrail stack I built before going live, a postmortem of a double-fired cron job, a circuit breaker for agent outages, and a health monitor that tells me when my agents are dying.

My Paper Trading Bot Went Live This Week — Here's the Guardrail Stack I Built First

Sam Hartley — Sat, 19 Sep 2026 08:36:16 +0000

My Paper Trading Bot Went Live This Week — Here's the Guardrail Stack I Built First

Confession time: my trading bot has been "almost ready for real money" for about four months.

Classic side-project arc. I built a scanner, it found opportunities, it paper-traded them beautifully. The equity curve went up and to the right. I showed a friend the chart. He asked the awkward question: "So when are you turning it on for real?"

The honest answer was: not until it bores me.

Paper trading tells you whether your logic works. It tells you nothing about what happens when your logic is wrong at 3 AM with real money on the line. So before flipping the switch, I spent a week building the boring parts — the guardrails. This is that stack, roughly ordered by how paranoid each one is.

What Paper Trading Actually Tests (Spoiler: Not the Scary Parts)

Paper mode is a simulator with the serial numbers filed off. Mine was optimistic in four specific ways:

No slippage. Paper fills happen at the displayed price. Real fills happen at whatever the order book feels like.
No fees. Small strategies live and die on fees. Paper mode is a world where the house doesn't take a cut.
Infinite liquidity. Paper mode never gets a partial fill or a rejected order.
No consequences. The big one. In paper mode, a bug is a curiosity. In live mode, a bug is an incident.

I've already lived the last one in miniature: my scheduler once double-fired and placed the same order twice. In paper, that's a weird chart. Live, that's exposure you never chose to take.

So before going live, I built six guardrails. None of them are clever. All of them are boring. That's the point.

Guardrail 0: The Model Doesn't Decide What It's Allowed to Do

Quick architecture note, because everything below depends on it.

My setup separates the brain from the hands. The "brain" — an LLM doing analysis — never talks to the exchange. It writes an intent: "open position in X, size S, stop at level L." A separate, dumb executor picks intents up. The executor is ~200 lines of Python with no model in it. The model can hallucinate whatever it wants; the executor only knows what's on the allowlist.

If your agent stack has the model calling money-moving APIs directly, this is the first thing I'd change. The part that decides should be as far away as possible from the part that does.

Guardrail 1: Hard Budget Caps, Enforced in the Executor

The executor has three numbers burned into it: a max position size, a max total exposure, and a max daily loss. Any intent that would breach one is rejected and logged — no model consultation, no retry, no "are you sure?"

class Limits:
    MAX_POSITION  = 50    # % of allocated capital, single position
    MAX_EXPOSURE  = 200   # % total, including leverage
    MAX_DAILY_LOSS = 3    # % — breach freezes everything

def check_intent(intent, state):
    if intent.size_pct > Limits.MAX_POSITION:
        return Reject("position too big")
    if state.exposure_pct + intent.size_pct > Limits.MAX_EXPOSURE:
        return Reject("exposure cap")
    if state.day_pnl_pct <= -Limits.MAX_DAILY_LOSS:
        return Reject("daily loss cap tripped — frozen until reset")
    return OK

The daily loss cap is the one I care about most. Strategies don't fail slowly — they fail on one bad day when everything correlates. The cap turns "my bot blew up" into "my bot stopped."

Guardrail 2: Rate Limits and Cooldown Windows

Autonomous agents in a loop are fast. Humans are not. So the executor enforces pace:

Max N new positions per hour. Mine is a single-digit number, on purpose.
Cooldown after any closed loss. No instant re-entry — that's how revenge trades happen, and bots are not immune to the sunk-cost loop. They just express it in code.
Consecutive-loss circuit break. After M losses in a day, the bot goes read-only until I flip it back on.

That last rule wasn't in v1. It is now, because I once watched my paper bot "scale into" a losing position five times in a row. It was following its rules perfectly. The rules were the problem.

Guardrail 3: Idempotency Keys on Every Action

Every intent gets a unique key before execution, and the executor checks it against a ledger of keys it has already acted on:

def execute(intent):
    key = f"{intent.day}:{intent.asset}:{intent.action}"
    if ledger.seen(key):
        return Skipped("duplicate intent")
    ledger.record(key)            # written BEFORE acting
    return broker.place(intent)

Two details matter. First, the key is derived from the decision (day, asset, action), not from a random ID — so a replayed or double-fired job produces the same key and gets dropped, which is exactly what saved me after the double-fire incident. Second, the key is recorded before the action, not after. If the process dies between the write and the fill, the retry gets skipped — I'd rather reconcile one orphan intent than double-spend.

Guardrail 4: Action Tiers With an Approval Gate

Not all actions are equally scary, so they get sorted into tiers:

Tier 1 — reversible: scans, reads, paper actions. The bot does these freely.
Tier 2 — bounded: small real positions, inside the caps. The bot does these alone, but I get a message with the full intent before execution.
Tier 3 — unusual or irreversible: anything at the caps, anything outside normal parameters, anything the bot itself flags as low-confidence. This tier pauses and waits for a button tap on my phone.

The approval message shows the full intent, the reasoning, and — this is the important part — what happens if I don't answer. The default is nothing. Timeout equals rejection. An unanswered request must never silently become a yes.

Tier 3 has caught exactly zero dramatic events so far. That's the correct number. The gate exists for the day something almost makes sense.

Guardrail 5: The Dead-Man Switch

The bot must check in on a heartbeat. Missed check-ins, and it freezes itself:

HEARTBEAT_EVERY = 300   # seconds
FREEZE_AFTER   = 3      # missed beats

def on_tick():
    if missed_beats() >= FREEZE_AFTER:
        freeze(reason="dead-man switch")
    if hung_jobs() > 0:
        freeze(reason="job stuck — state is suspect")
    beat()

The insight I stole from running a health monitor: a hung bot is more dangerous than a dead one. A dead bot does nothing — annoying, but safe. A half-alive bot might fire one leg of a two-step plan and never execute the other half.

So the freeze triggers on ambiguity too, not just silence. A stuck job means state can't be trusted. Freezing means: no new opens. Existing positions still get managed — stops always work — but nothing new starts. And there's a kill command that freezes everything from my phone, tested weekly, because an untested kill switch is a decorative kill switch.

Guardrail 6: An Audit Log Written Before Anything Happens

Every intent lands in a local log — timestamp, full payload, the reasoning — before execution, then gets updated with the result. Every morning I get a briefing message with a reconciliation: what the log says vs. what the account says.

If those two ever disagree, the bot assumes its own state is wrong, freezes, and waits for me. The account is the source of truth. The log is the source of intent. My job is reading both.

Week One, Honestly

So how's the first week of real money? Deeply educational, in the way a small electric shock is educational:

Fees and slippage turned "up and to the right" into "up and slightly to the right." The strategy still works, just with worse numbers. Better to learn that at small size than large.
The dead-man switch fired once. A job wedged itself after a network blip, the bot froze, and I spent an evening untangling it. Annoying — and exactly the failure mode it was built for. I came home to a frozen bot, not a runaway one.
The approval gate caught nothing dramatic. The correct amount of drama is zero.

The strangest part is psychological. In paper mode, I checked the bot obsessively, because nothing was watching it but me. Now I check it less — because the guardrails already asked the questions I would have asked.

What's Next: Shadow Mode

One more thing I'm adding for the next month: every live intent also gets mirrored into paper mode with the same timestamps. At the end, I diff them — paper fill vs. real fill, paper P&L vs. real P&L. That gap is the true cost of going live, measured instead of guessed. If it shrinks, my execution is tight. If it grows, the bot is paying a hidden tax I can hunt down.

If you're running agents that touch the real world — orders, payments, posts, deletions — the caps, the tiers, and the dead-man switch are the pieces I'd build first. All of it is a few hundred lines of deeply boring Python, and it converts "I hope it behaves" into "when it misbehaves, it stops."

Where do you draw the line — full autonomy, approval gates, or still in paper mode? Drop a comment, I'm genuinely curious.

Part of my series on running AI on my own machines — previously: a circuit breaker for agent outages, a postmortem of a double-fired job, and a health monitor for my agents.

I Added a Circuit Breaker to My AI Agent Setup — It Caught Three Outages This Month

Sam Hartley — Tue, 15 Sep 2026 08:34:53 +0000

I have a confession: for the longest time, my local AI setup had a very embarrassing failure mode.

One of my machines goes down — a reboot, a driver update, someone trips over a power cable — and the agents that depend on it just keep firing requests at it. Every single request times out. Each one burns 30-60 seconds waiting for a connection that will never succeed. Meanwhile the task queue piles up, retries stack on retries, and by the time the machine is back, I have a swamp of failed jobs to clean up.

The fix took an afternoon: a circuit breaker pattern. If you've never heard the term, it comes from electrical engineering — when too much current flows, the breaker trips and the circuit opens. No drama, no fire. You flip it back when things are safe.

The software version does the same thing for network calls. After X consecutive failures, stop calling. Fail fast, fall back to something that works, and check periodically whether the dead service is back instead of hammering it with every request.

Since I added it, it's caught three real outages this month. Here's what I learned building it.

My Setup (The Short Version)

For context: I run a small AI lab — a Mac mini as the orchestrator, a Windows PC with a GPU for heavy inference, and an old Ubuntu box for background services. The orchestrator has agents that route tasks to the other machines: code generation goes to the GPU box, quick chat stays local, image parsing goes to the Ubuntu box, and so on.

The routing itself is a dumb little Python function with a model registry and keyword matching. It's the piece that decides where each request goes. What it didn't have until recently was any concept of whether the destination is actually alive.

The Problem, In Numbers

Here's what a typical outage looked like before:

The GPU machine reboots for a Windows update at 14:02
Agent jobs keep hitting it: image generation, code refactors, embeddings
Each request has a 30-second timeout
The agent framework retries failed requests up to 3 times
So one failed job = up to 2 minutes of pure waiting, then a cascade of retry jobs

On a bad day, a 15-minute outage would produce 40+ queued tasks that all tried the dead endpoint, failed, and retried — burning a couple of hours of queue time on nothing. And since some of those jobs had follow-up jobs ("after generating the images, post them to X"), the failures propagated downstream. I'd come back to find half my evening pipeline in a error state because Windows decided 14:02 was a great time to update.

The worst part: the requests that should have just gone somewhere else — the Ubuntu box runs the same models, just slower on CPU — sat in the same queue behind the doomed ones.

The Circuit Breaker, Minimal Version

The pattern has three states. I stole this directly from the classic description (it's all over the resilience literature):

Closed — normal operation. Requests pass through. Failures are counted.
Open — tripped. Requests fail immediately, no network call at all. This is the "stop hammering the dead thing" state.
Half-open — after a cooldown, let one request through as a probe. If it succeeds, close the breaker. If it fails, stay open.

Here's roughly what I run per endpoint (simplified, but functionally what's in production):

import time

class CircuitBreaker:
    def __init__(self, name, threshold=3, cooldown=60, timeout=5):
        self.name = name
        self.threshold = threshold        # consecutive failures before tripping
        self.cooldown = cooldown          # seconds before probing again
        self.timeout = timeout
        self.failures = 0
        self.state = "closed"            # closed | open | half-open
        self.opened_at = None

    def allow_request(self):
        if self.state == "closed":
            return True
        if self.state == "open":
            if time.monotonic() - self.opened_at >= self.cooldown:
                self.state = "half-open"  # probe time
                return True
            return False                  # fail fast, no network call
        return False                      # a probe is already in flight

    def record_success(self):
        self.failures = 0
        self.state = "closed"

    def record_failure(self):
        self.failures += 1
        if self.state == "half-open" or self.failures >= self.threshold:
            self._trip()

    def _trip(self):
        self.state = "open"
        self.opened_at = time.monotonic()
        self.failures = 0

And in the request wrapper (again, simplified):

def call_endpoint(breaker, url, payload):
    if not breaker.allow_request():
        raise CircuitOpen(f"{breaker.name} is tripped, failing fast")

    try:
        resp = requests.post(url, json=payload, timeout=breaker.timeout)
        resp.raise_for_status()
        breaker.record_success()
        return resp.json()
    except Exception:
        breaker.record_failure()
        raise

One breaker instance per endpoint, keyed in a dict. The router that decides which machine handles which task now consults the breaker before doing anything else.

Note the aggressive timeout — 5 seconds, not 60. When a machine is genuinely there, a health probe responds in well under a second. Long timeouts are for slow responses, not dead machines. Distinguishing those two cases is half the value.

The Part That Made It Actually Useful: Fallbacks

A circuit breaker that just fails fast is... okay. Your errors arrive sooner. Great. Congratulations.

The real win comes from pairing the breaker with your fallback chain. My router already knew fallbacks existed — big model down? use the small one. GPU box rented out? CPU box takes over. What it lacked was the signal to switch. The breaker provides exactly that signal, in constant time, without a single wasted network call.

So the flow became:

Router picks the destination for a task
Is the breaker for that destination open? If yes, use the fallback destination immediately
Otherwise, call with a short timeout; failures count toward the threshold
If the breaker trips mid-flight, the next request reroutes — not to a retry queue, but to the fallback

A 30B code model being unavailable now means requests degrade to the 4B model on the Mac. Code quality drops slightly. The pipeline keeps flowing. Image jobs meant for the GPU get routed to the Ubuntu box and take 3 minutes instead of 20 seconds. Still flowing.

Three Outages, Zero Cascade Failures

I've had this running for a bit over a month now. Three times the breaker earned its keep:

1. The Windows Update (14:02, of course). GPU box gone for 17 minutes. Before the breaker: ~40 doomed tasks, cascading failures, an evening of cleanup. With the breaker: it tripped after 3 failures, code and image tasks rerouted to the fallback chain, and when the box came back the half-open probe re-closed it within a minute. I didn't even notice until I read the logs.

2. The flaky PCIe riser. One of my GPU cards had a flaky riser connection and the machine hard-froze. When a box is hung (not off), connections don't get refused — they just hang there. The 5-second timeout is what saved this one: three hangs, breaker trips, traffic reroutes. Without the short timeout, every request would have sat there for 60 seconds apiece, pretending to work.

3. The Ubuntu box's full disk. A background service filled the disk with logs, and the service started returning errors while the machine itself stayed up. The breaker tripped on HTTP errors just like it does on connection failures — the pattern doesn't care why the endpoint is failing, only that it is. Tasks that needed the vision service waited for the half-open probe instead of piling up garbage.

Three outages. Zero cascade failures. Zero evenings spent cleaning up retry swamps.

What I Got Wrong First Try

I counted total failures instead of consecutive ones. My first version tripped the breaker after N failures ever, which meant a busy day with a handful of one-off timeouts could trip it even though the endpoint was healthy. Consecutive-failure counting with reset-on-success is the fix. One-off failures happen; the breaker should ignore them.

My cooldown was too short. I started with 10 seconds, which meant a dead endpoint got probed every 10 seconds — fine — but a slowly recovering endpoint (say, a machine that just booted and is still loading a 30B model into VRAM) would get marked healthy by a TCP-level probe, then fail on the actual request and re-trip. 60 seconds worked better. If the first real request after recovery is heavy, give it room to finish loading.

I forgot to reset breakers on deploy. The breaker state lives in memory. When I shipped a new version of the router, all breakers came up closed while the actual machine was still down — instant trip, three failures wasted. The state should be persisted (or at least seeded from a health check at boot). It's on my list.

Fail-fast errors need to be distinguishable. When a task fails because "the breaker is open," that's a different failure than "the endpoint returned garbage." My first version raised the same generic exception for both, which made my alerting noisy and my logs confusing. One extra exception class fixed it. Now "breaker open" is not even an alert — it's the expected state during a known outage. The alert fires when a breaker trips, not on every request it rejects.

The Dashboard Bit

Because I was already logging breaker trips, I added one more thing: a daily message to my Telegram with any breakers that tripped in the last 24h and why.

That message has quietly become one of the most useful signals in my whole setup. Not because outages are frequent — they're not — but because it tells me which machine is the flaky one this month. September it was the riser. Last month it was power settings putting a box to sleep mid-job. The pattern in the trips is a pattern in my hardware.

Should You Build One?

If you have agents calling services and you have no failure isolation between them — yes. Honestly, it's the kind of thing that sounds like Enterprise Java Architecture Astronautics until the first time it saves your evening, and then it's just plumbing.

You don't need a library. The class above is ~40 lines and covers 90% of the value. If you want batteries included, there are solid packages out there — but read their retry/timeout defaults carefully, because a circuit breaker that waits 60 seconds per failure isn't protecting you from much.

The mental model that made it click for me: retries ask "will it work this time?" A circuit breaker asks "is it worth asking?" During an outage, the answer is no, and knowing that up front is worth more than any amount of persistence.

Have you wired failure isolation into your agents, or are you still in the retry-swamp phase? I'm curious what patterns people are using for multi-machine setups — drop a comment.

This is part of my series on running AI entirely on my own machines — a router for picking the right model per task, getting real work out of small local models, and evaluating local models before trusting them.

My Cron Job Double-Fired and Placed the Same Order Twice — Here's My Postmortem

Sam Hartley — Sun, 13 Sep 2026 08:20:58 +0000

I keep a Mac Mini in a closet running a handful of small scheduled Python jobs. One of them scans market data on a schedule and, when its rules fire, places tiny orders through an exchange API. Positions are around $12 each. It ran for months without drama.

Then one morning the exchange said I held 16 contracts, and my local state file said 8.

Not a rounding error. Not a stale cache. The same order, placed twice, ten seconds apart — by two instances of the same script that were both alive at the same time, each keeping its own private idea of reality.

What follows is the postmortem I wrote for myself, cleaned up. The punchline: the scheduler was fine. The API was fine. Both root causes were me.

The morning I noticed two of everything

Context first, because the boring design matters later:

A Python script runs on a schedule (and occasionally on demand)
It fetches about a thousand tickers, applies some rules, sometimes opens or closes a small position
Every decision lands in a JSON state file; all my reports read from that file

That morning the fee log showed two order fees for the same position, timestamps ten seconds apart. The exchange held 16 contracts. My state file tracked 8. The state file wasn't lying about what it knew — it had only ever been told about one of the two orders.

I eventually found two distinct ways this had happened. Both were my fault, in different ways.

Incident 1: I restarted a job that was never dead

The first one is embarrassing in the most ordinary way possible.

The scan isn't instant — a thousand tickers plus math takes a bit. One evening I triggered a manual run, and the terminal just sat there. No output. No prompt back. "Hung," I thought — and ran it again.

Here's what actually happens when a chatty Python script runs with stdout connected to a pipe or a background handle instead of a real terminal: output is block-buffered. The job was alive and working the whole time. Its output was sitting in a buffer, waiting to be flushed. I read the silence as "dead," and my restart created a second live instance.

Those two order fees, ten seconds apart? One order from the run I thought had died, one from the resurrection.

The lesson that stings: silence is not a signal. "It's not printing" and "it's not running" are unrelated statements. Before you rerun anything: check the process table, check the log file's modification time, check whether it's burning CPU. Poll the thing. Re-running a silent-but-alive job is how you manufacture concurrency you never designed for.

Incident 2: The scheduler and I raced each other

The second one took an evening of forensics.

A few days later: same symptom, but one position was triplicated. The order timestamps lined up with the top of the hour — and my scheduler fires at :02. In my shell history, right there: a manual run of the same script at :02. I'd been poking around and wanted a fresh scan "right now," and it simply never occurred to me that the clock was about to do the same thing.

Two entry points, same script, same minute, zero coordination. A scheduler is just a clock — it doesn't know what you're doing in a terminal, and I wasn't thinking about the clock.

The rule I added: manual runs never land on a scheduled minute. If the job runs at :02, I run manual checks at :15. It feels comically simple. It has prevented at least two repeats that I know of.

Why this wasn't a disaster (this time)

I want to be honest here, because the honest version is more useful than the heroic one. Three things saved me, and none of them was foresight:

My exits are close-all. The exit logic closes the entire position on the exchange — not "the size I think I have." A duplicated position resolves itself at the next exit. The bug was self-healing by accident.
The money was small. $12 doubling to $24 is a shrug. The same code with position sizes that scale with an account would not be a shrug.
The exchange is the source of truth. My reports were wrong and my history file had gaps, but the actual money was always consistent with reality. The drift lived in my shadow copy — the state file — not in the world.

That third point rewired how I think about every state file I keep. They're caches. The API on the other side is the truth. Once that lands, "reconcile before acting" stops being an enterprise buzzphrase and becomes just reading the real number before making a decision.

The fixes

A lockfile — the boring kind

The core fix is about fifteen lines at the top of main():

import fcntl, os, sys

LOCK = os.path.join(os.path.dirname(os.path.abspath(__file__)), ".job.lock")
lock_fd = open(LOCK, "w")
try:
    fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
    print(f"[lock] another instance holds {LOCK}; exiting", flush=True)
    sys.exit(0)
lock_fd.write(str(os.getpid()))
lock_fd.flush()

A second instance starts, tries to grab the lock, fails, exits with a message. On Linux and macOS that's fcntl; on Windows it's msvcrt.locking — same idea, different syscall. And if you'd rather not touch Python for it, the shell version is one line:

flock -n /tmp/job.lock python3 job.py || echo "already running"

Announce yourself in the logs

Every run now logs start and end with its PID:

import logging

logging.basicConfig(
    filename="job.log",
    format="%(asctime)s pid=%(process)d %(levelname)s %(message)s",
)
logging.info("scan start")

When the lockfile blocks a second instance, the log says so in plain words. And looking back at the incident logs now, the duplicate jumps out in ten seconds — two different pid= numbers in the same minute — instead of needing an evening of timestamp archaeology.

Flush like you mean it

flush=True on prints — or just run with python3 -u. This one directly kills Incident 1: the terminal would have shown live output, and "no output" would have actually meant something.

Reconcile before acting

Before the job opens anything, it now pulls actual positions from the exchange API and treats the local state file as a hint, not as truth. Anything the file claims that the API doesn't confirm gets flagged for review instead of acted on.

The part I'd been getting wrong for years

The reframe that made all of this click:

I used to think of idempotency as an API-design topic — something server people worry about. But the moment your cron job calls an API that mutates state, you're running a distributed system. Multiple processes, lying transports (buffered stdout counts as a lying transport — it lied straight to my eyes), shared mutable state on the other side of a network. All the classic distributed-systems failure modes are now your failure modes. Just at hobby scale.

A lockfile doesn't make a distributed system safe. What it does is collapse the most common failure — scheduler plus human, or human plus impatient human — into a no-op. That's most of the practical risk, for fifteen lines of code.

Wrapping up

If you take away two things:

A silent job is not a dead job. Poll, don't rerun.
If a scheduled job of yours talks to anything that mutates state, add the lockfile today — not after your own double-fire. Mine cost an evening of forensics and $12 of unplanned exposure. The same bug with real sizing would have cost a weekend and my confidence in the whole setup.

I'm curious how others handle this, especially anyone running scheduled jobs against mutating APIs on a homelab box. Lockfiles, per-action idempotency keys, a real job queue instead of cron? Drop what's actually held up for you in the comments — I'm always stealing ideas for my checklist.

I Don't Vibe-Check New Local Models Anymore — I Run This 30-Minute Eval Instead

Sam Hartley — Fri, 11 Sep 2026 08:03:28 +0000

I Don't Vibe-Check New Local Models Anymore — I Run This 30-Minute Eval Instead

A new "best open model ever" drops roughly every week now. And for most of this year, I handled every release the same dumb way: download it, ask it three questions in a terminal, nod at the answers, think "yeah, this one's smart" — then spend the next two weeks finding out where it actually breaks, in my real work, one task at a time.

That's a vibe-check. It feels like evidence. It isn't. A model that nails a clever one-liner and then mangles JSON in my log pipeline is useless to me, and no amount of terminal flirting reveals that until something breaks at 2 AM.

For context: my daily drivers are a 9B model on my Mac Mini and a 30B coder model on the RTX 3060 in my PC. Swapping either of them isn't free — every model has quirks baked into my prompts by now. So I stopped vibe-checking. Now every model goes through a small eval suite before it's allowed anywhere near my stack. Thirty minutes to run, built in an afternoon, and it has saved me from three bad migrations this year.

Why the public benchmarks don't help me

I read benchmark posts. I just don't make decisions with them, because they measure the average internet's tasks, not mine.

My actual workload on a normal day:

Extracting structured JSON out of messy log lines and API responses
Editing Monkey C for my watch face projects (a language most models barely know)
Writing small glue Python scripts against local APIs
Summarizing long docs where specific facts have to survive, not just the vibe

A model can top every leaderboard and still be the wrong model for me if it can't emit valid JSON without a chaperone. The reverse is true too — the "boring" model that always follows the output format is worth more to my setup than the genius that occasionally freestyles.

What's in the suite

One folder, 40 cases, all lifted from real work:

12 extraction tasks — real (scrubbed) log lines; the model must return valid JSON with specific keys
10 code edit tasks — "here's a function, add retry with backoff, keep it under 30 lines" style
8 summarization tasks — checked for must-mention facts, not tone
6 format-adherence tasks — strict output shapes, no prose allowed
4 trap tasks — ones that look easy but contain a subtle gotcha. Models that bluff instead of flagging the problem fail here, which is exactly what I want to catch

The rule for growing it: every time a model messes something up in daily use and I have to fix it by hand, that failure becomes a new case. The suite grows from my own scars. It's the most valuable folder on my disk and it's maybe 60 KB of text.

The harness

60-ish lines of Python. No framework, no LLM-as-judge (a judge model just adds its own preferences and noise). Each case runs three times at temperature 0, and the score is the average pass rate:

import json, re, sys, requests

OLLAMA = "http://localhost:11434/api/chat"

def run_case(model, case):
    r = requests.post(OLLAMA, json={
        "model": model,
        "messages": [{"role": "user", "content": case["prompt"]}],
        "stream": False,
        "options": {"temperature": 0.0},
    })
    return r.json()["message"]["content"]

def check(case, output):
    kind = case["checker"]
    if kind == "json":
        try:
            data = json.loads(output)
            return all(k in data for k in case["expect_keys"])
        except ValueError:
            return False
    if kind == "regex":
        return bool(re.search(case["pattern"], output, re.S))
    if kind == "contains":
        return all(s.lower() in output.lower() for s in case["must_contain"])
    return True

def score(model, cases, runs=3):
    total = 0
    for case in cases:
        passed = sum(check(case, run_case(model, case)) for _ in range(runs))
        total += passed / runs
    return round(100 * total / len(cases), 1)

if __name__ == "__main__":
    cases = [json.loads(l) for l in open("cases.jsonl")]
    print(sys.argv[1], "->", score(sys.argv[1], cases))

A few decisions baked in:

Temperature 0, always. Evaluating at temperature 0.7 is judging a model on moods.
Three runs per case catches flaky formatting. One model I tested passed a JSON case on run 1 and failed runs 2 and 3. That's not a model I trust with nightly jobs.
One number out. If a model scores 87.5, it goes into a plain text file next to the previous scores. That file is now my real benchmark history.

What running it taught me

Newer isn't automatically better. The suite caught a hyped release that scored 9 points below my incumbent, even though everyone on my timeline was calling it the new default. On my tasks, it wasn't.

Quantization damage shows up in format, not intelligence. Q4 quants pass the logic checks fine, then blow output formatting at weird, inconsistent rates. If I'd only vibe-checked, I'd never have seen it — the prose looks equally smart either way.

Wins are task-shaped, not model-shaped. One challenger beat my incumbent by 15 points on extraction and lost by 8 on code edits. I didn't switch — I routed. That result is literally why extraction jobs in my setup now go to a different model.

My prompts drift too. I run the suite after big prompt changes, not just model changes. Twice this year a "clever prompt improvement" quietly regressed three cases. A vibe-check would never have caught that.

The switch rule

A challenger replaces my current model only if it wins by 10+ points overall AND doesn't drop more than 5 points in any single category. Below that, the migration isn't worth it — rewriting prompts, re-learning quirks, updating the docs I keep for myself.

The one time I ignored my own rule was that hyped release I mentioned. The timeline got me. I spent a weekend migrating prompts, ran the suite afterward out of curiosity, and the old model won by 9. I crawled back and re-deployed the old setup Sunday night. The suite forgave me. My Saturday didn't.

Wrapping up

If you run local models for anything that matters, don't vibe-check. Build a suite of YOUR tasks — 20 cases is enough to start — run new models against it at temperature 0, and let a number make the decision. It's 30 minutes of compute versus two weekends of regret.

If you've built something similar, I'm curious what made it into your cases. Drop your weirdest eval case in the comments — always looking to steal ideas for mine.

How I Get Frontier-Quality Output from Local Models That Are 10x Smaller

Sam Hartley — Thu, 10 Sep 2026 09:13:31 +0000

How I Get Frontier-Quality Output from Local Models That Are 10x Smaller

I run Qwen 3.5 9B on my Mac Mini and Qwen 3 Coder 30B on an RTX 3060. That's the hardware reality — I'm not fitting a 200B+ parameter model locally. Not even close.

But here's the thing: most of my daily output from these models is indistinguishable from what I used to get from Claude or GPT-4o. Not because the models are that good. Because I learned how to talk to them.

This isn't another "here are 10 generic prompt tips" post. This is the specific, weird stuff I do to squeeze frontier-quality output from models that are 10-20x smaller than the cloud alternatives. Stuff that actually changed my day-to-day results.

Why Prompt Engineering Matters More for Small Models

Big models are forgiving. You can throw a vague prompt at GPT-4o and still get something usable. The model has enough parameters to infer what you probably meant.

Small models don't have that luxury. A 9B model takes your prompt literally. If you're vague, the output is vague. If you're specific, the output is specific. The quality gap between a badly-prompted small model and a well-prompted small model is massive — far bigger than the gap between a well-prompted small model and a well-prompted big model.

In other words: the same 9B model can produce garbage or gold depending entirely on how you ask.

Trick 1: The "Role + Format + Constraint" System Prompt

Most system prompts I see are one-liners: "You are a helpful assistant." That's wasted space on a small model.

My system prompts always have three sections:

You are a senior Python developer who writes clean, minimal code.

FORMAT: Respond with code only. No explanations unless asked. Use type hints.

CONSTRAINTS:
- Never use global variables
- Always include error handling for network calls
- Prefer standard library over third-party packages
- Maximum 50 lines per function

The role tells the model who to be. The format tells it what shape the answer should take. The constraints tell it what not to do.

Small models are terrible at deciding what to leave out. The constraints section forces the model to make fewer choices, which means fewer wrong choices.

Before this pattern: My 30B coder model would return 200-line functions with inline comments explaining every line. Useful, but I spent more time deleting than coding.

After: 50-line functions, type hints, no fluff. I went from 40% of the output being noise to maybe 5%.

Trick 2: One-Shot Examples Over Zero-Shot

Small models are pattern matchers. Give them a pattern and they follow it. Don't give them a pattern and they guess — badly.

Instead of:

Summarize this article in 3 bullet points.

I do:

Summarize articles as 3 bullet points.

ARTICLE: "Apple announced..."
SUMMARY:
- Apple released new MacBook Pro with M4 chip
- Starting at $1,599, shipping next week
- Performance claims 2x faster than previous generation

ARTICLE: "The Federal Reserve..."
SUMMARY:

That example does more work than any instruction. The model sees the exact format, tone, and level of detail I want. It mirrors that pattern for the next article.

This is the single highest-impact change I made. One-shot examples turned my 9B model from "sometimes okay" to "reliably good" for structured tasks like summarization, extraction, and formatting.

Trick 3: Decompose, Don't Compose

I used to ask small models for big things:

Build me a REST API with authentication, CRUD operations, rate limiting, and tests.

And I'd get 500 lines of interconnected code that looked reasonable but had subtle bugs in the auth middleware, the rate limiter didn't actually work, and the tests were testing the wrong things.

Now I break it into steps, running each one separately:

Step 1: "Create a FastAPI app skeleton with health check endpoint"
→ I review, test, fix

Step 2: "Add JWT authentication to this app. Here's the current code: [paste]"
→ I review, test, fix

Step 3: "Add CRUD endpoints for users with SQLAlchemy. Current app: [paste]"
→ I review, test, fix

Each step produces 20-40 lines. Small models are excellent at 20-40 lines. They're mediocre at 200 lines. They're terrible at 500 lines.

This is more work for me — I'm running 4 prompts instead of 1. But the output quality is dramatically better, and I spend less time debugging because each piece is correct before I build on it.

Trick 4: Temperature Isn't What You Think

I used to set temperature to 0.0 for "factual" tasks and 0.7 for "creative" tasks. That's what every tutorial says.

For small models, I've found the opposite works better:

Code and structured output: 0.0 — no surprise, this is standard
Summarization and extraction: 0.0 — small models hallucinate more at higher temperatures. Keep it deterministic.
Creative writing: 0.3-0.4 — not 0.7. At 0.7, a 9B model produces wild tangents. At 0.3, it's slightly varied but stays on track.
Brainstorming: 0.5-0.6 — this is the only time I go above 0.5 on a small model.

The key insight: small models have less "knowledge density" per parameter. Higher temperatures let them drift further from their training distribution, which means more hallucination. Keep the temperature low and guide the creativity through your prompt instead.

Trick 5: The "Bad Example" Anti-Pattern

Small models learn from examples, and they learn from what not to do just as much as what to do.

I include explicit bad examples in my prompts:

Write a technical explanation. Do NOT write like this:

"Machine learning is a fascinating field that has revolutionized countless industries.
In today's rapidly evolving technological landscape, leveraging artificial intelligence
has become paramount for organizations seeking to maintain a competitive edge."

Instead, write like this:

"ML models map inputs to outputs. You train them on data, then they predict on new data.
The three types: supervised (labeled data), unsupervised (no labels), reinforcement (reward signals)."

Your turn. Explain distributed systems:

The bad example shows exactly the kind of verbose, corporate, filler-heavy output I don't want. The good example shows the concise, specific, jargon-free style I do want.

This is especially effective for small models because it gives them two reference points instead of one. They can triangulate the style more accurately.

Trick 6: Post-Processing Pipeline (The Unsexy Secret)

Here's the thing nobody wants to hear: my local model output goes through a second pass.

Not another LLM call. A simple Python script that:

Removes filler phrases — "It's important to note that", "In today's world", "As we can see"
Enforces sentence length limits — any sentence over 30 words gets flagged for manual review
Checks for repetition — if two paragraphs say essentially the same thing, it removes one
Validates code blocks — runs python -c "compile(code, ...)" to check for syntax errors

This script catches maybe 15% of the model's output. But that 15% is the difference between "obviously AI-generated" and "reads like I wrote it."

FILLER_PHRASES = [
    "it's important to note",
    "in today's world",
    "as we can see",
    "it's worth mentioning",
    "needless to say",
    "at the end of the day",
    "in this rapidly evolving",
    "leverage",  # as a verb
    "delve into",
    "tapestry",
]

def clean_output(text: str) -> str:
    for phrase in FILLER_PHRASES:
        text = text.replace(phrase.capitalize(), "")
        text = text.replace(phrase.lower(), "")
    return text.strip()

Simple. Dumb. Effective. The model doesn't know I'm doing this, and it doesn't need to.

Trick 7: Context Recycling for Multi-Turn Work

Small models have small context windows. My 9B model handles about 8K tokens of context before it starts losing the thread.

When I'm working on a multi-turn coding session, I don't keep the full conversation history. After every 3-4 exchanges, I summarize the key decisions into a "state" block and start fresh:

[STATE]
Project: Task queue manager in Python
Stack: FastAPI + Redis + SQLite
Decisions so far:
- Using Redis for job queue, SQLite for persistence
- Async workers with asyncio.gather
- Auth via JWT with 5-minute expiry
Current file: worker.py (in progress)
Known issues: Need to handle worker crashes mid-task

Continue implementing the worker crash recovery:

This keeps the context lean and focused. The model gets exactly what it needs — no conversation history padding, no stale context from turns 1-3 that's no longer relevant.

Without context recycling: After 5-6 turns, the model starts contradicting decisions from turn 2. It forgets what it decided and re-decides differently.

With context recycling: I can work for 20+ turns and the model stays consistent because it only sees the current state, not the full history.

Where These Tricks DON'T Help

Let me be honest about the gaps:

Complex reasoning. A 9B model can't trace a 6-step logical argument no matter how you prompt it. I still route complex reasoning to the 30B model or to cloud APIs. Prompt engineering closes the gap for structured, well-defined tasks. It does not create intelligence that isn't there.

Long-context understanding. If I need to analyze a 20-page document, no amount of prompting will make a 9B model with 8K context do it well. I chunk the document, summarize each chunk, then feed the summaries to the model. It works, but it's a workaround, not a solution.

Factual accuracy about recent events. Local models have training cutoffs. My Qwen 3.5 doesn't know what happened in 2026. No prompt trick fixes stale training data. For current information, I use cloud APIs or RAG with real-time data.

Nuanced code architecture. The 30B coder model can write a single module well. It cannot design a multi-service system where auth, data, and queue layers interact correctly. I still design the architecture myself and have the model implement each piece.

The Numbers (Before and After Prompt Engineering)

I tracked output quality for a month before and after I started using these tricks consistently:

Task	Before (raw prompts)	After (engineered prompts)	Cloud (GPT-4o)
Code generation (< 50 lines)	6/10	9/10	9.5/10
Code generation (> 200 lines)	3/10	5/10	8/10
Summarization	5/10	9/10	9/10
Data extraction	6/10	8.5/10	9/10
Creative writing	4/10	7/10	8.5/10
Debugging	5/10	7/10	9/10

For tasks under ~50 lines of code, the gap between well-prompted local and cloud is almost gone. For longer tasks, local models still lag, but prompt engineering narrows it from "embarrassing" to "acceptable."

The Quick Reference

If you take one thing from this article, make it this checklist:

System prompt = Role + Format + Constraints — never just "you are helpful"
One-shot examples — always show the model exactly what you want
Decompose big tasks — 4 small prompts beat 1 big prompt
Low temperature — 0.0 for code, 0.3-0.4 for creative, never above 0.6 on small models
Bad examples — show what NOT to write
Post-processing — a 20-line Python script catches 15% of AI-isms
Context recycling — summarize state, don't keep full history

These aren't theoretical. I use every single one daily on my Mac Mini and RTX 3060 setup. They took my local models from "good enough for rough drafts" to "good enough for production output."

The model didn't change. My prompts did.

Why I Stopped Chasing the Newest LLM (And What I Run Instead)

Sam Hartley — Sat, 29 Aug 2026 08:02:10 +0000

Why I Stopped Chasing the Newest LLM (And What I Run Instead)

There was a stretch last year where I downloaded every new model within hours of it dropping. Llama 3.1? Done. Mistral Nemo? Running. Qwen 2.5? Let me benchmark it at 2 AM on a Tuesday.

I was spending more time downloading, configuring, and benchmarking than actually building things. And the worst part? Most of the "upgrades" didn't change my daily output at all.

So I stopped. I picked a stack and stuck with it. And somehow, I'm getting more done now than when I had 40 models installed.

The Upgrade Treadmill

Here's what the cycle looked like:

New model drops on HuggingFace
Someone posts benchmarks showing it's 3% better at X
I spend an evening downloading 20GB, tweaking parameters, running my test suite
It's marginally better at something and marginally worse at something else
Two days later, another model drops
Repeat

In one month I downloaded 12 models. I actually used 3 of them regularly. The other 9 sat on my drive eating 180GB of SSD space.

The math was brutal: roughly 15 hours of my time that month on model management. That's almost two full workdays. For what? A 3% improvement on a benchmark I don't even care about?

My Stable Stack (Same Models for 4+ Months)

I haven't changed my primary models since April 2026. Here's what I run and why:

Machine	Model	Role	Why This One
Mac Mini M4	Qwen 3.5 9B	Orchestration, routing, quick chat	Fast, solid at instruction-following, 9B is the sweet spot for 16GB RAM
Mac Mini M4	Granite 3.2 8B	Writing, editing, summaries	Better prose than Qwen at this size. Less hallucination in creative tasks.
Windows PC (RTX 3060)	Qwen 3 Coder 30B	Code generation, refactoring	30B is the smallest that reliably handles multi-file projects. Fits in 12GB VRAM.
Windows PC (RTX 3060)	DeepSeek R1 8B	Reasoning, debugging, analysis	The chain-of-thought on this model is genuinely useful for tracing bugs.
Ubuntu box	DeepSeek R1 (larger)	Heavy reasoning tasks	When the 8B isn't enough and I don't want to pay for cloud.
Ubuntu box	MinCPM-V	Vision tasks	Only vision model that runs reasonably on consumer hardware.

That's it. Six models across three machines. I use maybe 4 of them daily.

Why These Specific Models

Qwen 3.5 9B — The Workhorse

This model handles 80% of what I throw at it. Routing decisions, summarizing articles, writing first drafts, parsing structured data. It's fast (under 2 seconds for most responses on the Mac Mini) and it doesn't hallucinate much at this size.

I tried replacing it with Llama 4 Scout when it came out. Same parameter class, theoretically better benchmarks. In practice? The Qwen model followed my system prompts more precisely and produced more consistent output formats. Benchmarks don't measure "does it actually follow instructions" — real usage does.

Qwen 3 Coder 30B — The Specialist

30B parameters is the minimum I've found for reliable multi-file code generation. Anything smaller starts forgetting what function it defined two files ago.

The RTX 3060 with 12GB VRAM runs this model in 4-bit quantization at about 12-15 tokens/second. Not blazing, but fast enough for code review and refactoring where I'm not sitting there waiting.

Could I get better results from a 70B model? Probably. But a 70B model doesn't fit on a 3060, and I'm not buying another GPU just for marginally better code suggestions.

DeepSeek R1 8B — The Thinker

This one surprised me. The chain-of-thought reasoning in R1 is genuinely useful, even at 8B. When I have a tricky bug or a design decision, I run it through R1 first. It often surfaces edge cases I missed.

The 8B version is the minimum viable reasoner. For really complex stuff, I bump up to the larger version on the Ubuntu box. But 8B handles 90% of my reasoning needs.

Granite 3.2 8B — The Writer

Most people don't know this model well. IBM's Granite series doesn't get the hype of Llama or Qwen. But for writing tasks — summarizing, editing, rephrasing — it's noticeably better than Qwen at the same size.

It produces cleaner prose with fewer awkward constructions. It also hallucinates less on factual summaries. I use it for all article drafts and content work.

What I'm Missing By Not Upgrading

Let me be honest about the trade-offs:

I'm probably 5-10% behind the frontier on code generation. The latest Claude or GPT-4o will write better code than my 30B Qwen. But I'm not paying $200/month for 5-10% better code that I still have to review anyway.

I'm definitely behind on very long context tasks. If I need to analyze a 50-page document, local models with 8-32K context windows can't compete with Gemini's 1M+ window. I still use cloud for those specific tasks — about 15% of my workload.

I'm missing out on multimodal advances. The vision models I run are functional but not impressive. GPT-4o Vision and Gemini Pro Vision are significantly better. I use cloud for complex vision tasks.

But here's the thing: I know these gaps. They're predictable. And I've built workflows that route around them without thinking about it.

The Routing System That Makes It Work

The reason I can run a "stale" stack is that I built a routing layer that automatically sends tasks to the right model:

Incoming task
  → Quick chat / formatting / extraction?
    → Qwen 3.5 9B (local, instant)
  → Code generation or refactoring?
    → Qwen 3 Coder 30B (local, ~12s)
  → Reasoning or debugging?
    → DeepSeek R1 8B (local, ~8s)
  → Long document or complex vision?
    → Cloud API (routed, paid)

This routing is 15 lines of Python. It checks the task type and sends it to the appropriate model. No AI needed to route AI. A simple classifier works fine.

The result: I use cloud APIs for maybe 15% of my tasks, and those are the ones where local genuinely can't compete. Everything else stays local, stays free, and stays fast.

The Money Math (After 14 Months)

Item	Cost
RTX 3060 12GB (used, eBay)	$150 one-time
Mac Mini M4 (already owned)	$0
Ubuntu box (repurposed)	$0
Electricity (all 3 machines, 24/7)	~$12/month
Cloud API usage (hybrid)	~$25/month
Total monthly cost	~$37/month

My previous cloud-only spend was $187/month. That's $1,800/year saved. Minus the $150 GPU (paid for in under a month) and $37/month ongoing, I'm saving over $1,400/year.

And I'm not even counting the intangible benefits: no rate limits, no vendor lock-in, full privacy, works offline.

When I Will Upgrade

I'm not against upgrades. I'm against pointless upgrades. Here's when I'll swap a model:

A new model is meaningfully better at a task I do daily. Not 3% on a benchmark — a real, noticeable improvement in my actual workflow.
A model drops that fits my hardware and replaces two models I currently run. Consolidation is a valid reason to upgrade.
Security or stability improvements. Bug fixes count. Feature glitter doesn't.

I test new models in a sandbox before they ever touch my production stack. The bar is: "Does this meaningfully change my daily output?" If the answer is no, I don't switch.

The Lesson (For Myself As Much As Anyone)

The model treadmill is real. FOMO is real. That HuggingFace notification that a new model just dropped is designed to make you feel like you're falling behind.

You're not.

The best model is the one you actually use to ship things. Not the one with the highest benchmark score. Not the one released yesterday. The one that's loaded, configured, and producing output for your project right now.

I spent months chasing the cutting edge. Now I spend that time building. Turns out, building is more fun than benchmarking.

Sam Hartley runs a three-machine local AI setup and writes about making it actually work. He hasn't downloaded a new model in 4 months and he's fine with it.

→ Custom automation setups on Fiverr
→ Follow CelebiBots on Telegram

ai #llm #selfhosted #ollama #buildinginpublic

I Replaced All My Cloud AI With Local Models — Here's What Actually Broke

Sam Hartley — Thu, 27 Aug 2026 08:02:20 +0000

I Replaced All My Cloud AI With Local Models — Here's What Actually Broke

I write a lot about local AI. How it saves money. How it's private. How a $150 GPU can replace $200/month in API subscriptions. And all of that is true.

But here's what I don't talk about enough: some things still don't work. Not "kind of work" — actually fail. Like, I-open-ChatGPT-and-do-it-manually fail.

After 14 months of running a local-first AI setup (Mac Mini M4 + Windows PC with RTX 3060 + Ubuntu fallback), I want to be honest about the dealbreakers. The tasks where local models still lose. Because nobody's building the "my local LLM produced garbage and I wasted 40 minutes" blog post, and someone should.

My Setup (Context for the Numbers)

Before I list failures, here's what I'm working with:

Machine	Hardware	Models	Role
Mac Mini M4	10-core CPU, 16GB RAM	Qwen 3.5 9B, Granite 3.2 8B	Orchestration, routing, writing
Windows PC	RTX 3060 12GB	Qwen 3 Coder 30B, DeepSeek R1 8B	Code generation, reasoning
Ubuntu box	Older GPUs, 128GB RAM	DeepSeek R1 (larger), MinCPM-V	Heavy tasks, vision

Electricity cost: ~$12/month. API costs: $0 for most tasks.

It's good. It handles maybe 85% of what I throw at it. That other 15%? That's where the pain lives.

Dealbreaker #1: Complex Multi-Step Reasoning

This is the big one. The one everyone talks about and nobody wants to admit.

I had a bug in a distributed task queue — the kind where Worker A processes a job, Worker B picks up the result, but under load, B sometimes gets a stale version. Classic distributed systems problem.

I fed the entire codebase (~2000 lines across 6 files) to DeepSeek R1 8B on the RTX 3060. It gave me a confident, detailed answer about adding a version field to the message format.

The problem? The version field was already there. The bug was in the retry logic — when Worker B got a 409 conflict, it was retrying with the OLD payload instead of re-fetching. The model didn't catch this because it couldn't actually trace the execution flow through the code. It pattern-matched "distributed systems bug" and gave me the textbook answer.

I pasted the same code into Claude Sonnet. It found the retry bug in 30 seconds.

Time wasted on local: 40 minutes (trying the "fix," testing, realizing it was wrong, trying again)
Time with cloud: 2 minutes
Cost difference: $0 vs ~$0.03

That $0.03 would have saved me 38 minutes. Local models are bad at multi-step reasoning where you need to trace causation through code. They pattern-match instead of actually following the logic.

Dealbreaker #2: Anything Requiring Current Information

My local models don't have internet access. I could hook them up to RAG pipelines and web search tools, and I have for specific projects. But for general "what's the latest X?" questions, local models are blind.

Examples where I had to switch to cloud:

"What's the current version of Ollama?" — Local told me 0.5.7. It was 0.6.2.
"Does Qwen 3.5 support tool calling?" — Local said no. It does (since training data cutoff was before that release).
"What's the best GPU under $300 in 2026?" — Local recommended the RTX 3060 (which I already own). But the RX 7700 XT dropped to $280 and has 12GB VRAM too — local didn't know.

If your task depends on current information, local models are a non-starter unless you invest in a RAG pipeline that fetches real-time data. And at that point, you're building infrastructure, not just running a model.

Dealbreaker #3: Long-Context Understanding

Here's a scenario: I have a 15-page API contract from a client. I need to extract the authentication flow, the error handling requirements, and the rate limits, then generate a Python SDK that matches all of it.

My 30B model can ingest the text. It produces something that looks like a Python SDK. But it misses things:

The contract specifies rate limits per endpoint (not global) — the model made them global
Error codes are hierarchical (4xx for client, 5xx for server, but also custom codes like 4201, 4302) — the model treated them as flat
Authentication uses a rotating token with a 5-minute expiry — the model implemented static tokens

Each of these is a subtle but critical failure. The model "understood" the document in the sense that it could summarize it. But it didn't actually follow the specifications precisely.

For documents under ~3 pages, local models are fine. Beyond that, especially with technical specifications, you start losing details that matter.

Dealbreaker #4: Vision Tasks (On Consumer Hardware)

I run Granite 3.2 Vision 2B and MinCPM-V for image understanding. They're okay. They can tell you what's in a photo. They cannot:

Read a complex diagram and extract the architecture it describes
Understand a UI mockup and generate matching CSS
Accurately transcribe handwritten notes with mixed languages
Distinguish between visually similar but functionally different components in a circuit diagram

The 2B vision model runs fast (4 seconds on the Windows GPU). But it's 2B parameters. It sees a diagram and says "this is a flowchart." Great. I can see that too.

The larger vision model (MinCPM-V on Ubuntu) does better but takes 3 minutes per image. For my automated pipeline that processes 20+ images per article, that's an hour of GPU time. Cloud vision APIs do the same in seconds.

Where local vision works: "What's in this photo?" "Is this a cat or a dog?" "Extract the text from this screenshot."
Where it fails: Anything requiring nuanced visual understanding.

Dealbreaker #5: Generating Consistent Long-Form Content

I know, I know — I wrote an article about how my AI content pipeline works. And it does. For 1000-1500 word articles with a clear structure.

But give a local model a 3000-word target with a specific voice, specific technical examples, and a specific argument structure, and it starts to drift. Paragraph 3 sounds like me. Paragraph 7 sounds like a Wikipedia article. Paragraph 12 is repeating Paragraph 4 in different words.

This is why my content pipeline uses short articles. Not because I don't have enough to say — because the model's coherence window is about 1500 words before it starts repeating itself or losing the thread.

For this article, I'm writing it myself. The local model can handle the structure and individual sections. But threading a consistent argument through 2500 words? That's still a human job.

What I Actually Do Now (The Hybrid Setup)

I've stopped pretending local can do everything. Here's my actual workflow:

Task	Where It Goes	Why
Quick questions, summaries	Local (Qwen 3.5 9B)	Fast, free, good enough
Code generation (< 200 lines)	Local (Qwen 3 Coder 30B)	Solid for most tasks
Code generation (complex, multi-file)	Cloud (Claude Sonnet)	Better at tracing logic
Short-form writing	Local (Granite 3.2 8B)	Good structure, decent tone
Long-form writing (> 1500 words)	Myself, with local for structure	Coherence window is the limit
Vision (simple)	Local (Granite Vision 2B)	"What's in this photo" — fine
Vision (complex)	Cloud (GPT-4o Vision)	Architecture diagrams, UI mocks
Current information queries	Cloud (Perplexity/GPT-4o)	Local doesn't know today's news
Data extraction	Local	Surprisingly good, even at 7B

Monthly cloud spend now: ~$25 (down from $187)
Monthly local spend: ~$12 (electricity)
Total: ~$37/month for 85% local, 15% cloud

The Real Takeaway

Local AI is incredible for what it does well. I saved over $2,000 in a year. I run my entire content pipeline, my code review, my daily summaries, and my monitoring — all on hardware I own.

But if you're starting out and someone tells you "you don't need cloud AI at all," they're either lying or they don't do the kind of work that requires deep reasoning, current data, or complex visual understanding.

The honest answer is hybrid. Local for the 80-85% of tasks where it's good enough (and cheaper and faster and more private). Cloud for the rest. And being smart about which is which.

The biggest trap isn't the technology. It's the denial. When you've invested time and money into a local setup, you want it to work for everything. You start accepting lower quality outputs because "it's free." But free output that's wrong isn't free — it costs your time to fix it.

I still use local for the vast majority of my work. I just know when to stop.

Sam Hartley runs a multi-machine AI home lab and writes about the infrastructure that makes local AI actually usable. Yes, he still has a ChatGPT subscription. No, he's not proud of it.

→ Custom automation setups on Fiverr
→ Follow CelebiBots on Telegram

ai #llm #selfhosted #ollama #honest #localai #buildinpublic

DEV Community: Sam Hartley

My Margin Guard Was Reading Zero. Four Separate Bugs Kept It That Way.

My Margin Guard Was Reading Zero. Four Separate Bugs Kept It That Way.

The guard

Bug one: None became 0.0, and 0.0 means "you're fine"

Bug two: the number, when it existed, was in the wrong unit

Bug three: the exchange told me the truth, and the truth was zero

Bug four: the guard ran once, the loop opened five

The demonstration

The fix

The test that actually proved it

What's still imperfect

What I'd copy

A Single int() Silently Disabled My Entire Risk System

A Single int() Silently Disabled My Entire Risk System

The ladder

The part that made me sit down

Why my tests never caught it

How I found the boundary

What I changed

The general version

My Agent Spent $23 Overnight. Here's Why My Limits Held and My Alerts Didn't.

My Agent Spent $23 Overnight. Here's Why My Limits Held and My Alerts Didn't.

What the job does, briefly

The part I got right, mostly by accident

The cap is not the alert

The second bug: the loop had no shape

Two metrics that actually predict this

What I'd tell you to copy

My Scheduled Agent Ran 40 Times and Did Nothing — Here's the Assertion That Fixed It

My Scheduled Agent Ran 40 Times and Did Nothing — Here's the Assertion That Fixed It

Errors page you. Silence doesn't.

Rule one: assert the effect, not the attempt

Rule two: expected-empty is not unexpected-empty

Rule three: three states, not two

The part that actually caught things

The transferable bit

Timeout Means No — The One Rule That Makes AI Agent Approval Gates Actually Work

The 2 AM Test

The Code, Which Is Embarrassingly Short

"Timeout Means Yes" Is Always One Boolean Away

What the Logs Taught Me

It's a Decision Record, Not a Security Boundary

The Pattern, Without the Trading

My Paper Trading Bot Went Live This Week — Here's the Guardrail Stack I Built First

My Paper Trading Bot Went Live This Week — Here's the Guardrail Stack I Built First

What Paper Trading Actually Tests (Spoiler: Not the Scary Parts)

Guardrail 0: The Model Doesn't Decide What It's Allowed to Do

Guardrail 1: Hard Budget Caps, Enforced in the Executor

Guardrail 2: Rate Limits and Cooldown Windows

Guardrail 3: Idempotency Keys on Every Action

Guardrail 4: Action Tiers With an Approval Gate

Guardrail 5: The Dead-Man Switch

Guardrail 6: An Audit Log Written Before Anything Happens

Week One, Honestly

What's Next: Shadow Mode

I Added a Circuit Breaker to My AI Agent Setup — It Caught Three Outages This Month

My Setup (The Short Version)

The Problem, In Numbers

The Circuit Breaker, Minimal Version

The Part That Made It Actually Useful: Fallbacks

Three Outages, Zero Cascade Failures

What I Got Wrong First Try

The Dashboard Bit

Should You Build One?

My Cron Job Double-Fired and Placed the Same Order Twice — Here's My Postmortem

The morning I noticed two of everything

Incident 1: I restarted a job that was never dead

Incident 2: The scheduler and I raced each other

Why this wasn't a disaster (this time)

The fixes

A lockfile — the boring kind

Announce yourself in the logs

Flush like you mean it

Reconcile before acting

The part I'd been getting wrong for years

Wrapping up

I Don't Vibe-Check New Local Models Anymore — I Run This 30-Minute Eval Instead

I Don't Vibe-Check New Local Models Anymore — I Run This 30-Minute Eval Instead

Why the public benchmarks don't help me

Bug one: `None` became `0.0`, and `0.0` means "you're fine"