DEV Community: Jackson Ly

Refreshing a token I did not own logged our account out

Jackson Ly — Tue, 11 Aug 2026 17:34:25 +0000

We run a monitoring job that checks a Bluesky account for replies. Last week I fixed a real bug in it, and the fix took the account offline for two days. The bug was worth fixing. The fix was the wrong shape, for a reason that is easy to miss when you are integrating with an app you did not write.

The bug I was fixing

The job read the session out of the browser's local storage and called the notification endpoint with it. Once the token expired, every call came back as:

{"error":"ExpiredToken","message":"Token has expired"}

The job never checked. It read notifications off the response, got undefined, counted zero, and reported no new inbound.

That is the failure mode worth caring about, because it does not look like a failure. An empty result from a quiet account and an empty result from a dead credential are byte-identical in the log. I only caught it because the number was too round: every prior run had read 26 notifications with a stable 14 reply / 9 like / 3 follow split, and then the counts came back as an empty object. The zero was the anomaly.

So the check now asserts that the call succeeded instead of trusting the shape of the result:

const res = await fetch(url, { headers })
if (!res.ok) throw new Error(`sweep failed: ${res.status}`)
const { notifications } = await res.json()
if (!Array.isArray(notifications)) throw new Error('sweep returned no array')

That part was right and I would write it again.

Where I went wrong

Having found an expired token, I made the job refresh it. One call to com.atproto.server.refreshSession, take the new access token, retry the sweep. It worked on the first run.

Two runs later the account was signed out. Not expired, signed out, sitting on the create-account screen, with refreshSession now returning:

{"error":"ExpiredToken","message":"Token has been revoked"}

Revoked is a different word from expired, and I had caused it.

Why one successful refresh breaks the next one

AT Protocol refresh tokens are single-use and rotating. Presenting one gets you a new pair and destroys the one you presented. The SDK docs put it plainly: "On the token refresh, the old refresh token will be revoked instantly", and "You must use only the fresh pair of tokens (access + refresh). It's not possible to reuse the same refresh token multiple times." Access tokens last about two hours, refresh tokens about two months.

It is worth being precise about where that is documented, because I got it wrong in my own notes first. The XRPC spec describes the refresh endpoint and the access/refresh split, but it never says the tokens are single-use. That behaviour is in the SDK documentation, and in the maintainer discussion the position is that clients should expect rotation and expiry at any time, since the policy is left to the auth server. Reasoning from the protocol spec alone will not get you there.

Now add the part about ownership. The browser app was holding that session and refreshing it on its own schedule. My job reached into its storage, spent its refresh token, and got back a replacement pair that I deliberately did not write back, because the app owns the shape of that storage object and writing into it is its own kind of bug.

The app was left holding a token that had already been consumed. It could still read for a couple of hours on the unexpired access token, which is why the first run looked fine. The moment it tried to refresh, its token was gone and the session dropped.

The breakage is delayed by exactly one access-token lifetime, which is what makes it hard to catch. The harmful call and the visible symptom are hours apart, so by the time anything looks wrong you have stopped suspecting the thing you changed.

The rule

Never spend a single-use credential that another program owns. Read its tokens if you have to, but do not rotate them.

Rotation is a write, even though it looks like a read. Nothing in the call signature suggests you are mutating shared state, and the request that consumes the token looks identical to one that just fetches. If two programs share a session and both can refresh, the one that refreshes without persisting the result quietly destroys the other's ability to continue. None of this is specific to atproto. It is how rotating refresh tokens work wherever they are implemented, which is why the SDK ships an on_session_change hook whose entire job is to make you persist the rotated pair.

Once you accept that, the recovery is to make the owner do the refresh. Reload the app, give it a few seconds to refresh on its own schedule, read the fresh token back out, retry once. If that still fails, it is a genuine block and a human needs to sign in. That is cheaper than the alternative and it cannot corrupt anything.

The part I did not expect

While the account was signed out I went looking for how much of the job could still run, assuming the answer was none of it.

Most of it, as it turned out. The reads I actually needed were public:

GET /xrpc/com.atproto.repo.listRecords?repo=<did>&collection=app.bsky.feed.post
GET /xrpc/app.bsky.feed.getPostThread?uri=<at-uri>
GET /xrpc/app.bsky.feed.getPosts?uris=<at-uri>

No token on any of them. That was enough to pull 77 of our own posts, walk the threads under the six that had replies, and work out which replies were still unanswered by diffing against the parent URIs in our own outbox. It gave the same answer the authenticated notification sweep would have, from endpoints that cannot log anybody out. The engagement counts I use for measurement came back the same way.

Only the notification list itself needs auth, and notifications were never the thing I wanted. They were the thing I reached for because they were the obvious surface, and reaching for them is what put a credential in the loop to begin with.

So before you build refresh handling, check whether the data is public. An unauthenticated read has no session to lose, and for anything you published yourself there is a decent chance it is already sitting behind a public endpoint.

A 200 response is not a page, and your policy check is grepping an empty shell

Jackson Ly — Sun, 09 Aug 2026 11:27:31 +0000

I run automated checks over terms of service and publisher agreements before we use a platform. Fetch the page, strip the tags, grep for the clauses that matter. It is not sophisticated and it has worked for months.

This week it lied to me three times.

The numbers

Same script, same extraction, three legal pages:

peerlist.io/terms returned 60 characters of text. daily.dev/terms returned 70. substack.com/pa, which is a full publisher agreement, returned 1,345.

For comparison, the same script on a server-rendered legal page returns between 20,000 and 35,000 characters. Substack's terms of use, fetched the same way, gives 22,045.

All three responses were HTTP 200. Nothing errored. Nothing retried. The script reported no matching clauses and moved on.

Why this is worse than a 404

Those three pages are client rendered. The HTML that arrives is a shell, and the text I care about gets assembled by JavaScript after load. Old news, and normally a visible problem, because you print the body, you see nothing, and you reach for a browser.

What let it past me was the shape of the check rather than the shape of the page.

My check is a negative assertion. The pass condition is "this grep found nothing." An empty document satisfies every negative grep ever written, so a page that failed to load produces the same verdict as a page that loaded fine and genuinely has no such clause. The failure is silent, and it resolves toward clean, which is the expensive direction.

The Substack case is the one that would have cost me. That publisher agreement does contain a clause I needed to know about. My script pulled 1,345 characters of shell, found no match, and would have told me the document was clean if I had trusted it.

The rule

Any check whose success condition is absence has to prove the input was present first.

That reads as obvious written down. It is not obvious while you are writing the thing, because the presence check is the part nobody writes. You write the grep, the grep works on the first page you try, and the invariant that the page actually loaded stays implicit forever.

What I do now

Three assertions before the grep runs, and a verdict of unresolved rather than clean when any of them fail.

MIN_LEN = 3000          # a real legal page is 20k+, a shell is under 1.5k
SENTINELS = ("terms", "agreement", "policy", "last updated")

def check_policy(url, patterns):
    text = extract_text(fetch(url))

    if len(text) < MIN_LEN:
        return "UNRESOLVED", f"only {len(text)} chars, probably a client-rendered shell"
    if not any(s in text.lower() for s in SENTINELS):
        return "UNRESOLVED", "no legal-document markers, wrong page or a redirect"

    hits = [p for p in patterns if re.search(p, text, re.I)]
    return ("HITS", hits) if hits else ("CLEAN", [])

The length floor does most of the work. Pick it from your own corpus rather than from a guess: fetch ten pages you know are fine, take the smallest, halve it.

The sentinel check catches the second thing that bit me. You can pull 30,000 characters of perfectly good text off entirely the wrong document, because the URL redirected to a marketing page, or because a 404 rendered as a styled landing page instead of a status code.

Unresolved is a real state and it needs somewhere to go. Mine escalates to a headless browser, which costs about a second and settles it. The point is that unresolved never quietly collapses into clean.

The second trap: the URL you were given may not be the URL

substack.com/publisher-agreement returns 404. The document exists and lives at substack.com/pa. I only found it because the footer link on the terms page pointed there.

So stop guessing slugs. Fetch a page you know exists, read its links, and follow the one whose anchor text matches what you want:

def resolve_doc(index_url, label):
    soup = BeautifulSoup(fetch(index_url), "html.parser")
    for a in soup.find_all("a", href=True):
        if re.search(label, a.get_text(), re.I):
            return urljoin(index_url, a["href"])
    return None   # not found is not the same as not there, so escalate

Three lines of real work, and it survives the vendor reorganising their legal pages, which they do more often than you would expect.

Where else this shape turns up

Status page checks that pass because the incident list never rendered. Sitemap validators reporting zero broken links because the sitemap came back empty. Index freshness checks reporting no stale documents because the query errored and returned an empty set. Anything that counts problems and alerts on a nonzero count will report perfect health the moment its input pipeline dies.

The tell is the same every time: the healthy state and the broken state produce identical output. If your monitoring cannot separate "nothing wrong" from "nothing checked", it is not monitoring that thing.

The short version

Feed your check an empty string once and read what it says. If it says everything is fine, you have found the bug.

Written from a week of policy checks that returned 200 and told me nothing. Drafted with AI assistance, verified against the live pages described, and edited by hand.

Your agent's inbox check is measuring read state, not answered state

Jackson Ly — Sat, 08 Aug 2026 21:28:02 +0000

If you run an agent that answers inbound, it needs exactly one boolean per item: have we replied to this yet. Every platform hands you something that looks like that boolean, and none of them actually is.

I spent this week finding three different ways that gap swallows real messages. All three came out of the same agent, on three different platforms, and the fix turned out to be the same shape every time.

1. Read state gets cleared by looking at it

On Reddit, GET /message/unread.json returns your unread items. Fetching it does not mark them read, which sounds convenient right up until you notice the other half: answering does not mark them read either.

So the counter only ever drifts. You answer someone, the item stays unread, and inbox_count keeps reporting work that no longer exists.

That is harmless if the counter is decoration. Ours was not. The scheduler treats inbox_count > 0 as a reason to skip a cheap no-op check and run the full sweep instead, so a single answered-but-unread item made every subsequent run do the expensive thing for nothing.

The fix is to close the loop yourself, after you have actually handled the item:

POST https://old.reddit.com/api/read_message
body:    id=t1_<comment_id>&uh=<modhash>
header:  X-Modhash: <modhash>      # from /api/me.json -> data.modhash

The ordering matters more than the call. Mark read after you have confirmed the thing is dealt with, because marking read is precisely what hides it from the next run.

2. The badge counts things you will never act on

Bluesky gives you an unread badge. It counts likes, follows, reposts and replies the same way.

One morning the badge showed a single unread. That unread was a like. Sitting underneath it, already flagged read and therefore invisible to anything that triages on the badge, was a four round technical thread whose last message had been mine to answer for five days.

Nothing was broken. The badge answered the question it was designed to answer, which is "is there anything new", and I was asking it "is there anything owed".

const notifs = await listNotifications({ limit: 25 });
const conversational = notifs.filter(n =>
  ['reply', 'mention', 'quote'].includes(n.reason)
);

Filter by reason, and ignore the read flag entirely. A reply you have never answered goes read the moment anything fetches the list, which is to say the moment you look without acting.

3. The thread view disagrees with itself

This is the one that would have made me reply to the same person twice.

Having fixed the badge problem, I wrote what looks like the obvious check: fetch the thread for their message and see whether any reply on it is mine.

const thread = await getPostThread({ uri: theirPost, depth: 1 });
const answered = thread.replies.some(r => r.post.author.handle === me);

On a thread I had answered two hours earlier, that returned false. Their post reported replyCount: 1. The replies array came back empty. My reply existed, with record.reply.parent.uri pointing at exactly that post.

I do not know why the appview rendered it that way, and for this purpose it does not matter. What matters is that the check had a failure mode I had not designed for, and the failure pointed the wrong way.

Read your own outbox instead

The fix that generalizes: stop asking whether they were answered, and start asking whether you answered.

Your own sent items are authoritative. They do not depend on their read state, their badge semantics, or how their renderer decided to assemble a thread today.

// Every parent we have ever replied to, from our own repo.
const repliedTo = new Set();
let cursor;
do {
  const page = await listRecords({
    repo: myDid,
    collection: 'app.bsky.feed.post',
    limit: 100,
    cursor,
  });
  for (const rec of page.records) {
    const parent = rec.value?.reply?.parent?.uri;
    if (parent) repliedTo.add(parent);
  }
  cursor = page.cursor;
} while (cursor);

const open = conversational.filter(n => !repliedTo.has(n.uri));

The same shape works anywhere. On Reddit it is your own comments and their parent_id. On a mail API it is your sent folder and the In-Reply-To header. The lookup is cheap, and it is derived from something you control.

The asymmetry that should drive the design

These two failures are not equally bad, and that should decide how you build the check.

A false "already answered" means you miss a reply. That is embarrassing and recoverable, and a human would forgive it.

A false "still open" means you reply to the same person twice. On an account that posts automatically, that is what spam looks like from the outside, and it is not recoverable, because the second message is already sent.

So when the two signals disagree, believe the one that says you already answered.

Written up from a week of debugging an agent that answers its own inbound across four platforms. Drafted with AI assistance, verified against the live APIs described, and edited by hand.

Durable approval is not the same as valid approval

Jackson Ly — Fri, 07 Aug 2026 19:31:52 +0000

A few weeks ago we shipped a general command tool for a local agent, with a deny-list in front of it and a human approval gate behind it. I wrote at the time that the deny-list is the part that looks like engineering and does not hold, because you cannot enumerate the dangerous set. The gate is the part that holds.

Someone pushed back with a good refinement. The gate only holds if approval is real state with a lifecycle: requested, then approved or rejected, then consumed, then expired. A confirmation step living inside the agent loop cannot survive the process dying while it waits, so it is not a control, it is a prompt.

He is right, and most implementations skip it. What I want to add is that making approval durable does not finish the job. It moves the failure somewhere quieter.

An approval is granted against a world, not a string

When someone clicks approve, they are approving the specific thing they were shown: this file, with this content, going to these two people. What you persist is usually a record that a decision happened, plus an identifier. That is not a record of what was decided.

Between approve and consume, the world keeps moving. The file gets renamed, or a symlink under it now points somewhere else, or the recipient list picked up one more address while the process was down. Your token survives all of it, because it was never bound to any of it. The agent resumes after the crash exactly as designed and executes something the human never saw.

The old name for this

Time-of-check to time-of-use. You verify a condition, then act on it, and something changes in the gap. The textbook examples are filesystem races measured in microseconds.

Human approval stretches that gap to minutes or hours, because a person gets the prompt, goes to lunch, comes back and clicks yes. Durability stretches it further on purpose, since surviving a restart is the entire feature. We took a race condition and made it comfortable enough to design around.

The fix nobody enjoys building

Bind the approval to a fingerprint of what was actually rendered to the human. Not the intent, the literal resolved parameters. Recompute it at consume time, compare, and fail closed on any drift. Then expire aggressively, since expiry is the only real bound on how stale a decision can get.

Fail closed is the part I keep seeing people soften, and I understand why. The drift is almost always innocent, a timestamp or a reordered field, and failing on it feels pedantic. It is pedantic. It is also the one behaviour that makes the gate mean anything on the day the drift is not innocent.

Why this is worse than losing the approval

If approval evaporates when the process dies, you get a bad experience and a safe failure. The agent comes back, finds nothing, and asks again. The human sees the current state and can say no.

If approval survives unbound, you get a good experience and a silent failure. The agent finds a valid-looking token and proceeds against a world that moved. Nobody is asked anything. Nothing looks wrong in the logs, because from the system's side everything worked exactly as specified.

So durability on its own is not a safety property. It is a convenience that only becomes a safety property once the persisted thing is specific enough to check.

What I would do differently

Bind before you persist. Ship the approval carrying a hash of its resolved parameters from day one, even if the storage is a flat file and the lifecycle has two states. I would rather have a crude gate that revalidates than a carefully modeled lifecycle that trusts its own token, and I say that having built the second one first.

The lifecycle framing is right. It is just the easy half.

Written with AI assistance.

Running local and cloud models in the same coding agent: what actually ships in 2026

Jackson Ly — Fri, 07 Aug 2026 03:09:57 +0000

Someone on r/LocalLLaMA asked a question this week that turned out to be harder to answer than it looks: which open-source coding harness lets a local model and a cloud model work on the same task, together, in one loop?

The answers he got were a proxy and a shrug. He rejected both, correctly. So here is the actual answer, with the config knobs, and an honest account of the part that does not exist yet.

Why a gateway is not the answer

The first suggestion for this is always LiteLLM, or Bifrost, or whatever OpenAI-compatible proxy is current. Those are good at what they do. They are not what the question is asking for.

A proxy routes on what it can see in the request: the model name you asked for, weights you configured, health checks, budgets. It sits below the agent and has no idea whether the call it is forwarding is a throwaway commit message or the one architectural decision in the run. Point an agent at a gateway and you get failover and cost control. You do not get a local model and a cloud model dividing labour on a task, because nothing in the request says what the task needed.

The signal you want lives one layer up.

What actually ships, by declared role

Three tools do this today. None of them frame it as routing, which is why searching for "routing" misses them. They all frame it as roles.

Cline lets you set one model for Plan mode and a different one for Act mode. In settings, enable "Use different models for Plan and Act", then pick a model for each. Switching modes switches the model automatically, so a single task can be planned by one model and executed by another without you touching anything mid-run. The documented pairing is a stronger reasoning model for planning and a faster one for implementation, and nothing stops you pointing one of those at a local endpoint.

Aider takes three models in one session. --model is the main one that does the editing. --editor-model handles the edit-application step. --weak-model picks up the cheap work like commit messages and history summarization. You set them independently, so a large cloud model can do the reasoning while a small local model absorbs the chatter that would otherwise burn tokens.

Continue assigns models per role, and this is the closest thing to the original question. Autocomplete is a separate role from chat. Their docs cover running the autocomplete model locally through Ollama, and chat can sit on a cloud provider at the same time. That is genuinely a local model and a cloud model both live in one workflow, serving different jobs, all day.

So the honest answer to "does anything blend local and cloud in one loop" is yes, three things do, and they have shipped it for a while. It just is not called routing.

The pattern, and the part nobody has

Look at what those three have in common. In every case, you declare the split once, and the harness honours it for the rest of the run. Plan gets this model. Autocomplete gets that one. Commit messages get the cheap one.

What none of them do is decide the split themselves. No shipping harness looks at a subtask, works out that this one needs vision or a 400k context window or tool calling that survives twenty turns, and picks accordingly.

That gap is not an oversight, and it is worth understanding why, because it tells you what to build if you are going to build it. The requirement is known inside the planner, at the moment a subtask is created. By the time a model is selected, that context is gone: the selection layer sees a prompt and a model name. Nobody has plumbed the requirement from where it is known to where it is needed. Until someone does, autonomous model selection has nothing to select on.

If you are wiring this yourself

Two rules that survive contact with real work.

Route on declared capability, never on estimated difficulty. Capability is checkable: this subtask has an image in it, this one exceeds the small model's context, this one needs a tool call. Those facts stay true tomorrow. Difficulty is a guess, it is not measurable at the point you need it, and every heuristic you write for it goes stale on the next model release. A rule that rots silently is worse than no rule.

Put the cheap model where the work is bounded and verifiable. Commit messages, summarization, autocomplete, structured extraction with a schema you validate. In each case, a bad output is obvious immediately and costs one retry. That is why the shipping tools all made the same choice: the roles they hand to the weak model are exactly the roles where being wrong is cheap and detectable.

The interesting version of this problem is not building a smarter router. It is getting the planner to say what each piece of work actually needs, in a form something downstream can act on. Everything else follows from that.

Sources: Cline's Plan and Act docs, Aider's advanced model settings, and Continue's autocomplete deep dive, all checked while writing this.

Your interval-triggered agent pays full price for empty runs

Jackson Ly — Wed, 05 Aug 2026 17:31:39 +0000

We run an agent on an interval. It wakes about a dozen times a day, checks whether anything needs doing, and usually the answer is no. That "usually no" turned out to be the whole cost problem, and it took us longer than it should have to see it.

An interval trigger has no idea whether there is work. It fires exactly as often on a dead Tuesday as during a busy hour. So if the first thing your wake path does is the full sweep, you pay full price for every empty tick. We counted six fires in about two hours and twenty minutes, and half produced nothing but a "nothing changed" log line. Each of those still ran the complete orientation: read state, check five external services, recompute derived metrics, refresh performance snapshots. Minutes of work and real API spend to conclude the world was exactly as we left it.

Put a cheap guard in front of the expensive path

Before doing anything real, answer one question as cheaply as you can: has anything plausibly changed since last time?

For us that is a single indexed query against our own activity log.

SELECT run_session, max(ts)
FROM action_log
GROUP BY run_session
ORDER BY max(ts) DESC
LIMIT 1;

If the last run was under twenty minutes ago, we make exactly one external check, the cheapest signal we have that new work exists. If it comes back empty, we write two rows to the log and exit. No scouting, no recomputation, no metrics refresh.

last_run = db.last_activity_ts()
if now - last_run < timedelta(minutes=20):
    if cheapest_external_signal() == 0:      # one HTTP call
        log("monitor", "no-op")
        log("skip", "fast-exit, nothing changed")
        return
# ... otherwise fall through to the full pass

The guard reads local state first, because local state is free and the network is not. And the tiebreaker should be the single cheapest external call you have rather than a representative sample of your integrations. The goal is a yes or no answer for close to zero cost, not an accurate picture of the world.

Choose the threshold from how fast your input actually accrues

Twenty minutes is not a magic number. It came from watching how our inbound behaves: replies arrive over hours, so two wakes fifteen minutes apart genuinely cannot differ. If your agent watches a build queue that changes every thirty seconds, twenty minutes is absurd. Match the window to the thing you are watching, and write down why you picked it, because the next person to read the code will assume it was arbitrary.

Do not let the guard trust a cached answer it never verified

This one is subtle enough that we shipped it without noticing.

Our agent keeps a human-readable file of things that are blocked: expired sessions, missing permissions, failing integrations. That part is fine. The bug was that the agent started reading that file to decide what to skip. One entry said an API key lacked write scope, which was true on the day it was written. We skipped that integration on every run afterwards on the strength of the note.

When we finally tested it live, the write succeeded on the first attempt. The key had been fine for who knows how long. Nothing had failed, so nothing had corrected the note, so the agent went on believing it.

The rule we settled on is that a block is a fact with a timestamp rather than a permanent state. If a guard is going to skip work, it has to confirm the reason still holds right now, and that confirmation should be cheap enough that there is no excuse to skip it. A note tells humans what happened. It should never be an input to control flow.

# wrong: the note decides
if "devto" in blocked_notes:
    skip("devto")

# right: the note informs, the live check decides
if not devto.can_write():        # one API call
    note_block("devto", "write scope missing")
    skip("devto")

Make state durable per run, not per tick

The other thing that bites scheduled agents is interruption. A cron-triggered process that dies halfway leaves no trace of how far it got, so the next tick starts from zero and redoes work that already succeeded. If any of that work had side effects, they happen twice.

Persist progress at the step level, keyed by run, and have the wake path check for an unfinished run before starting a new one. Resuming a half-finished pass is almost always cheaper than repeating it, and it turns a crash from a correctness problem into a latency problem.

This is also what separates an agent that can ask a human for approval from one that cannot. If approval lives in memory inside the process, the process dying takes the pending decision with it. If approval is durable state the run can be resumed into, someone can answer twenty minutes later from their phone and the work carries on.

Where we landed

Guard the expensive path with the cheapest check that can answer no. Read local state before touching the network. Confirm a block is still true instead of inheriting it from a note. Keep per-run state durable so an interrupted pass resumes.

The agent is no smarter for any of this. The idle case just costs almost nothing now, and for us the idle case is most of them.

Human in the loop is not a switch: classify the commands

Jackson Ly — Sun, 02 Aug 2026 23:43:21 +0000

Human in the loop is usually implemented as a switch: approvals on, or approvals off. Both settings are wrong for an agent that runs shell commands, and you find that out the same way everyone does, which is by watching a long task die on its third ls.

The stall looks like this. Your agent is working through something multi-step, it needs to look at a directory, the harness raises an approval, and the run stops dead waiting for a human who has gone to make coffee. Multiply that by forty tool calls and the agent is no longer autonomous, it is a very expensive interactive shell.

So people reach for one of two fixes, and both make it worse.

The two fixes that are not fixes

Turning the gate off. Now rm -rf runs unattended too. The gate was not the problem, and you have removed the only thing standing between a plausible-looking command and an irreversible action.

Wrapping the tool in something else. This is the one I see most: the agent stalls on the built-in shell tool, so someone routes it through a different tool layer and the stall goes away. It goes away because the new layer does not ask. You have not removed the approval, you have moved who is responsible for asking, and usually the answer is now nobody.

The interrupt is not a bug. It is the gate doing its job. The bug is that the gate cannot tell ls from rm.

Classify by reversibility, not by tool

The useful axis is not which tool is being called, it is whether the call can be undone. Reads are recoverable by definition, since the worst case is that you read something and discard it. Writes, deletes, and anything that leaves the machine are not.

So the gate gets a classifier, and the classifier decides who has to wake up:

READ_ONLY = {
    "ls", "cat", "head", "tail", "wc", "stat", "file", "find", "grep", "rg",
    "git status", "git log", "git diff", "git show", "git branch",
}

RISKY_PREFIXES = ("rm", "mv", "dd", "chmod", "chown", "kill", "curl", "wget",
                  "git push", "git reset", "npm publish", "docker rm")

# Anything that lets one approved command smuggle in another.
SHELL_OPERATORS = ("&&", "||", "|", ";", "`", "$(", ">", ">>", "\n")


def needs_human(command: str) -> bool:
    cmd = command.strip()

    # A command containing shell operators is not one command, it is several.
    # Do not try to reason about the pieces. Ask.
    if any(op in cmd for op in SHELL_OPERATORS):
        return True

    if any(cmd == allowed or cmd.startswith(allowed + " ") for allowed in READ_ONLY):
        return False

    # Everything not explicitly known-safe requires a human, including
    # commands that merely look harmless.
    return True

Two things about that function matter more than the word lists.

It is an allowlist, not a blocklist. A blocklist is a bet that you thought of every dangerous command, and you did not. RISKY_PREFIXES above is there for logging and for showing the user why something is being escalated; it is not what makes the decision. The decision is made by the final return True, which is the deny-by-default line. Delete that line and the whole thing is decoration.

It refuses to parse compound commands. ls && rm -rf build starts with an allowlisted token. If your check is startswith, you just approved a delete. You can go and write a real shell parser, or you can treat the presence of an operator as an automatic escalation, which is a two-line rule that does not have parser bugs.

The asymmetry is what makes this safe

Every classifier is wrong sometimes, so the question is what its errors cost.

If the classifier calls a safe command risky, the user gets one unnecessary approval prompt. Annoying, bounded, visible.

If it calls a risky command safe, something irreversible happens with nobody watching. Not bounded, and often not visible until later.

Those two are not comparable, which is why the default branch has to be "ask" rather than "allow". You are not trying to build a classifier that is right. You are trying to build one whose mistakes all land on the cheap side, and then curating the allowlist until the cheap mistakes are rare enough to live with.

This is also the reason the allowlist should be boring and explicit rather than clever. Every entry you add is a small, deliberate decision that this exact thing can run while you sleep.

What this looks like in practice

We build recal, a local-first assistant on macOS, and its command tool works this way: read-only commands execute unattended, everything else raises an approval that carries the actual command text so the user is approving a thing rather than a category. Long tasks stopped stalling, and the approvals that remain are the ones worth reading.

The honest catch is that the allowlist is maintenance. New tooling shows up, someone's workflow needs jq or kubectl get, and the list has to grow. There is no version of this where you write the classifier once. What you get instead is a system where the cost of being wrong is a prompt instead of a restore from backup, and that trade has been worth it every time.

The other honest catch: this does nothing about a command that is individually safe and collectively terrible. Forty approved writes in the wrong directory is still forty approved writes. Reversibility classification bounds the blast radius of a single call, not of a plan.

If you are building one of these

Start with deny-by-default and an allowlist of four or five read commands, then add entries when a real task trips on one. Log every escalation with the command text, because that log is what tells you which entries to add next. And resist the urge to make the classifier smart. The value here is not intelligence, it is that the failure mode is a question rather than a deletion.

Written with AI assistance and edited by a human. The allowlist-over-blocklist and compound-command guidance follows the security notes in Anthropic's bash tool documentation; the rest is what we shipped and what it cost us.

Your cache_read_input_tokens is zero. Here is what silently did it.

Jackson Ly — Thu, 30 Jul 2026 05:32:37 +0000

Your cache_read_input_tokens is zero. Here is the list of things that silently did it.

You added cache_control, the docs said up to 90% cheaper, and the counter still reads zero on every request. Nothing errored. There is no warning. The cache just never gets read.

This is almost always one of a small number of specific mistakes, and they are all instances of a single rule.

The one rule

Prompt caching is a prefix match. The cache key is the exact bytes of the rendered prompt up to each breakpoint. Any byte that changes anywhere in the prefix invalidates the cache for every breakpoint at or after that position.

Render order is:

tools  ->  system  ->  messages

That ordering is the whole game. tools render at position zero, so a change there invalidates everything. A breakpoint on the last system block caches tools and system together.

Once you internalise "prefix match, tools first", every item below is predictable rather than surprising.

The audit list

Grep your prompt-assembly path for these. In my experience the first one accounts for most zero-hit reports.

In the prefix	Why the cache never hits
`datetime.now()` / `Date.now()`	Prefix differs on every request
`uuid4()` / a request ID	Same, and unfixable by retrying
`json.dumps(d)` without `sort_keys=True`	Key order is not guaranteed, so the bytes move
Iterating a `set` to build content	Iteration order is not stable
Session or user ID in the system prompt	Per-user prefix, zero sharing across users
`if flag: system += ...`	Every flag combination is a separate prefix
`tools=build_tools(user)`	Tools render first, so a per-user tool set caches for nobody

The fix in every case is the same shape: make it deterministic, move it after the last breakpoint, or delete it if it is not load bearing.

A "current date" line in the system prompt is the classic. It feels harmless because it is one short line. It sits in front of everything.

What does not invalidate everything

This is the part people over-correct on. There are three cache tiers, and a change only invalidates its own tier and below:

Change	Tools	System	Messages
Tool definitions added, removed, reordered	gone	gone	gone
Model switch	gone	gone	gone
Web search or citations toggle	kept	gone	gone
System prompt content	kept	gone	gone
`tool_choice`, images, thinking on/off	kept	kept	gone
Message content	kept	kept	gone

So you can flip tool_choice per request, or toggle thinking, without losing the tools and system cache. Only tool-definition changes and model switches force a full rebuild.

Two of those rows have an escape hatch, and they are not gated together:

System prompt content. Instead of editing top-level system mid-conversation, append a {"role": "system", "content": "..."} message to messages. It lands after the cached history, so the prefix survives. Available today on Claude Opus 5, Opus 4.8, Fable 5 and Mythos 5, with no beta header. Unsupported models return a 400, so catch it and fall back.
Tool definitions. On Claude Opus 5 there is a beta (mid-conversation-tool-changes-2026-07-01) that adds and removes tools between turns via tool_addition / tool_removal blocks, without invalidating the prefix. Tools you plan to add must already be declared with "defer_loading": true.

Model switching has no escape hatch. Caches are model scoped. If you want a cheaper model for a sub-task, spawn a subagent rather than switching the main loop.

The minimum is not monotonic

There is a minimum cacheable prefix. Below it, nothing caches, you get no error, and cache_creation_input_tokens is just zero.

The trap is that newer does not mean lower:

Model	Minimum
Opus 5, Fable 5, Mythos 5	512
Opus 4.8, Sonnet 5, Sonnet 4.6, Sonnet 4.5	1024
Opus 4.7	2048
Opus 4.6, Opus 4.5, Haiku 4.5	4096

A 3K-token prompt caches on Opus 5 and Opus 4.8 and silently does not on Opus 4.6 or Haiku 4.5. If you wrote a prompt off as "too short to cache" on an older model, it may be worth re-checking.

Two timing gotchas

The 20-block lookback. Each breakpoint walks backward at most 20 content blocks looking for a prior cache entry. Agentic loops blow through that easily, because one turn can add a dozen tool_use and tool_result pairs. If a single turn adds more than 20 blocks, the next request's breakpoint does not find the previous cache and silently misses. Fix: place an intermediate breakpoint every ~15 blocks in long turns.

Concurrent requests all miss. A cache entry becomes readable only once the first response begins streaming. Fire N identical-prefix requests in parallel and all N pay full price, because none of them can read what the others are still writing. For fan-out, send one, await the first streamed token, then fire the rest.

Reading the counters honestly

Three fields, and the third one misleads people:

cache_creation_input_tokens: written to cache this request, billed at a premium
cache_read_input_tokens: served from cache, billed at roughly a tenth
input_tokens: the uncached remainder only

Total prompt size is the sum of all three. If your agent ran for an hour and input_tokens shows 4K, the rest was served from cache. Check the sum, not the single field, before concluding anything about prompt size.

Is it even worth it

Reads cost about 0.1x. Writes cost 1.25x on the 5 minute TTL and 2x on the 1 hour TTL.

So break-even depends on which TTL you picked. With 5 minutes, two requests pay for it (1.25 + 0.1 against 2 uncached). With 1 hour you need at least three (2 + 0.2 against 3). The 1 hour TTL keeps entries alive across gaps in bursty traffic, but the doubled write means it needs more reads to come out ahead.

And if the first 1K tokens of your prompt genuinely differ every request, do not cache. You will pay the write premium for zero reads. Leaving it off is the correct answer.

Pre-warming, and when not to

You can pay the cache write up front with a max_tokens: 0 request. Prefill runs, the cache is written, and you get back content: [] with stop_reason: "max_tokens" and zero output tokens billed.

Put the breakpoint on the last block shared with the real request, usually the system prompt or the tool definitions. Not on the placeholder user message, and not via top-level automatic caching, which would key the cache to the placeholder.

It is worth doing when first-request latency is user visible, the shared prefix is big enough that a cold write is noticeably slow, and there is a moment before traffic to fire it: app startup, worker boot, post-deploy.

It is not worth doing when traffic is continuous. If real requests arrive more often than the TTL, they keep the cache warm on their own and a separate warm call is a pure extra write. Scheduled re-warms only make sense when your traffic has gaps longer than the TTL.

Note that max_tokens: 0 is rejected alongside stream: true, thinking.type: "enabled", output_config.format, a forced tool_choice, or inside a Batches request.

The debugging loop

If the counter is zero, do this rather than adding more markers:

Capture the fully rendered request body on two consecutive calls.
Diff them.
The first differing byte is your answer.

That diff is faster than reasoning about it, and it turns "caching is not working" into a one-line fix nearly every time.

Written with AI assistance and human review. Behaviour described reflects the Claude API as documented in July 2026; caching internals move, so verify against current docs before you rely on a specific number.

Your baseline scored 0.000. That's a broken harness, not a result.

Jackson Ly — Sat, 25 Jul 2026 03:34:51 +0000

Your baseline scored 0.000? Before you publish the win, here is the checklist I now run, because a zero from a baseline is almost never a result. It is usually your harness.

This week I watched a builder on r/Rag do something rare: he benchmarked his own memory library against plain RAG expecting a win, got three null results, and published the confounds instead of the victory. Two of his bugs are so common, and so quiet, that I think every RAG or agent-memory benchmark should gate against them by construction. This post is the checklist that came out of that thread.

Bug 1: your arms have different context budgets

His first run compared a memory arm retrieving k=20 sentence-level hits against a BM25 arm returning whole sessions. Same "top-k" on paper. In characters, one arm got 1.3k of context and the other got 11.9k. BM25 looked dramatically better.

Once he matched the budget, accuracy went 0.28 to 0.59 for the memory arms and the ranking flipped. The original result was a budget difference wearing a granularity costume.

The fix is embarrassingly simple: print the context size, in characters or tokens, next to every accuracy number, per arm. If a comparison does not state budget parity, the number does not mean anything. Chunked-vs-whole-document comparisons are especially prone to this because "top-k" hides a 10x budget difference in plain sight.

Bug 2: the baseline that scored 0.000 with clean logs

He ran a competing memory tool as a baseline. It scored 0.000. He re-ran it with a stronger extraction model. 0.000 again, logs clean. Very tempting to publish "competitor discards memories under load."

It was his bug, twice. His harness truncated each session to 6,000 characters before ingestion, cutting off the injected evidence. And he was passing limit= to an API whose parameter is top_k=, so his setting was silently ignored.

That second one deserves its own paragraph, because it is everywhere in this ecosystem: APIs that accept unknown kwargs without raising. They make a broken configuration look like a clean loss. In a benchmark, treat any client that swallows unknown parameters as hostile. Do not trust that a parameter did something. Assert it: if you move top_k from 5 to 20, the retrieved count has to move. If it does not, your harness is configuring nothing.

The three gates that make results trustworthy

Out of that exchange came a harness structure I now consider the minimum for memory and RAG benchmarks. Three separate liveness gates, each with its own abort message, all pre-registered before the run.

1. A positive control per arm. Every arm gets a probe it cannot fail, built from the corpus's own ground truth. If any arm scores below a threshold you wrote down before the run, the whole benchmark aborts as "dead harness," not "clean loss." This is the gate that catches truncation bugs, auth failures, and empty stores. A zero is never evidence until this gate passes.

2. An evidence-in-context ceiling. For each probe, check whether the gold evidence is actually present in the retrieved context. Aggregated, this is the recall ceiling you report next to accuracy. In his broken run it was 3.5 percent. You cannot out-rank evidence that was never retrieved, and no reranker or prompt change will save a run whose ceiling is on the floor. Cheap to compute, needs no LLM calls.

3. A parameter-efficacy assertion. For each knob the benchmark claims to vary, assert that observable behavior changes when the knob moves. This is the gate that catches the swallowed-kwargs class of bug, and it is the one nobody runs.

Why three gates and not one

The temptation is to fold these into a single sanity check. Do not. The whole value of a gate is that its failure names the broken layer. Evidence-in-context failing means retrieval never delivered. The positive control failing means the store or the answerer is dead. The efficacy assertion failing means your configuration is fiction. One merged gate tells you "something is wrong" and leaves the first hour of debugging still ahead of you.

There is a general lesson under all of this. A benchmark harness is software that lies by default, because every bug it has produces a plausible-looking number instead of a crash. The discipline that fixes it is the same one that fixes agent evaluation generally: never trust a claim the system makes about itself unless something the system cannot touch verifies it. A test that has never been proven to fail will happily pass forever.

Credit where it is due: the null-result writeup that prompted this is worth reading in full, including the part where he catches himself half-writing a false finding about a competitor before his own positive control saved him. Publishing that takes more spine than publishing a win.

I build a local-first assistant for the Mac, which is why I spend my days in retrieval evals. No product pitch here; the checklist stands on its own.

Our Rust file watcher ate 23.6 GB of RAM, and our ignore rules never had a chance

Jackson Ly — Mon, 13 Jul 2026 13:40:17 +0000

We build a local-first indexer for macOS. It watches the user's home folder so that when a file changes, we reindex just that file. Watching one project directory is a solved problem. Watching a real human's $HOME is a different sport, and it broke us in a way that took a heap profiler to see.

The symptom

The app sat at roughly 25 GB of resident memory. Not a slow leak either. It climbed there shortly after launch and stayed. One core also pinned itself for about ten minutes at every boot.

The obvious suspects were wrong. It was not the embedding model, and it was not our index. We had a careful ignore list (caches, node_modules, Library, build output), so surely we were not touching millions of files.

We were not. But something else was.

The finding

We use the excellent notify crate, specifically notify-debouncer-full, which is what you reach for when you want raw filesystem events collapsed into something sane. The setup you copy from the README uses RecommendedCache.

RecommendedCache resolves to FileIdMap. From the docs, the debouncer "can optionally keep track of the file system IDs all files and stitches rename events together". That is a genuinely useful feature: on macOS FSEvents and on Windows, a rename shows up as two unrelated events, and pairing them requires knowing that the file at the old path and the file at the new path are the same inode.

To do that, it has to know the file ID of every file under the watch root. So it walks the entire watch root and caches a (PathBuf, FileId) for every entry it finds.

Here is the part that mattered: that walk does not know about your ignore rules. Our ignore list filters events. The cache is built underneath that, from the watch root down. We told the debouncer to watch $HOME, so it faithfully walked all of $HOME, including everything we thought we had excluded, following symlinks as it went.

Under malloc_history, the damage was specific:

23.6 GB of live heap in the cache
4.16 GB of that in the hashbrown table itself
19.4 GB spread across 54.2 million path allocations
roughly 6.4 million files stat-ed at boot, which is the pinned core and the ten minutes

54 million allocations to remember the inode of files we had explicitly said we did not care about.

The fix, and what it costs

We switched the cache to NoCache.

That is not free, and it is worth being precise about the tradeoff rather than pretending we outsmarted the library. FileIdMap exists for a reason: with NoCache, the debouncer can no longer stitch rename events together. A rename stops arriving as "this moved from A to B" and degrades into an unrelated delete at A and a create at B.

We could absorb that, for two reasons that are specific to our design:

Our indexer is idempotent. Indexing the same file twice converges to the same row, so a spurious create is cheap and harmless.
We already run a rescan and reconcile pass that removes index entries whose files no longer exist. Deletions get collected there regardless of whether the watcher paired them correctly.

So a rename becomes delete-plus-create, and the system settles into the right state on its own. If your indexer is not idempotent, or you have no reconcile pass, this trade is not available to you and you should fix the scope instead.

Result: 24.8 GB down to 1.45 GB.

The second bug we found on the way

While measuring, we caught something else. When FSEvents drops events (it does this under load, by design, and tells you so), the debouncer emits a rescan. Our code fanned that out as one rescan per configured root.

Our roots were nested. We watched $HOME, and we also watched Documents, Downloads, and Desktop because they were configured separately. So a single dropped event triggered four concurrent, roughly $HOME-scale re-walks, and every file under the nested roots got embedded twice. You could see it plainly in the logs: two EmbeddingsGenerated lines per document.

The fix was to de-nest the roots into independent subtrees before emitting any rescan, so overlapping configuration collapses into one walk.

The lesson worth stealing

Library defaults are calibrated for the common case, and the common case is a project folder with a few thousand files. Nothing in the API warns you, because nothing is wrong with the API. FileIdMap is correct. It is correct at 5,000 files and it is a memory bomb at 6.4 million, and the difference is entirely in what you point it at.

So, two things:

Know what your watcher caches before you widen the scope. Filtering events is not the same as filtering what the library indexes internally. Ours sat below our filter, where we could not see it.

Measure the heap, do not reason about it. We would never have found 54.2 million path allocations by reading code. malloc_history found it in one pass. Every hour we spent theorizing about the embedding model was an hour we did not spend attaching a profiler.

If you are watching a user's whole home directory, assume every default in your stack was written by someone who was not.

Point any app at a local LLM on your Mac (OpenAI-compatible endpoints)

Jackson Ly — Fri, 10 Jul 2026 00:17:10 +0000

Most apps that grew an "AI" feature in the last two years talk to one of a handful of cloud APIs, and almost all of them speak the same dialect: the OpenAI Chat Completions format. That one detail is the reason you can pull the cloud out and run the whole thing locally on a Mac without the app ever noticing.

Here is the trick, why it works, and the gotchas that bite.

The one interface everything agrees on

OpenAI's /v1/chat/completions endpoint became the de facto standard. So when an app lets you "use your own key" or "set a custom base URL," it is almost always going to POST to {base_url}/chat/completions with a JSON body of messages and read back the same shape. It does not care what is on the other end, only that the response matches.

Local runners leaned into this. Both popular Mac ones expose exactly that endpoint:

Ollama serves an OpenAI-compatible API at http://localhost:11434/v1 (its native API lives on /api, but the /v1 path speaks the OpenAI dialect).
LM Studio has a built-in server you switch on from the Developer tab, serving on http://localhost:1234/v1.

So "make this app local" usually reduces to: point its base URL at one of those, put any non-empty string where it wants an API key, and pick a model you have pulled.

The 60-second version

Ollama:

brew install ollama        # or the .dmg from ollama.com
ollama serve &             # server on :11434
ollama pull llama3.1:8b    # pull a model once

Confirm it speaks OpenAI:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1:8b",
    "messages": [{"role": "user", "content": "say hi in 3 words"}]
  }'

If that returns a choices[0].message.content, any OpenAI-compatible client can use it. In the app, set:

Base URL: http://localhost:11434/v1
API key: ollama (or literally anything; it is ignored)
Model: llama3.1:8b

LM Studio is the same idea with a GUI: load a model, toggle the server on, and use base URL http://localhost:1234/v1.

Pointing real tools at it

The pattern shows up everywhere once you look for it. The official OpenAI SDKs are the clearest example: change one field.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
r = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "summarize this in one line: ..."}],
)
print(r.choices[0].message.content)

Same code, cloud or local. Only the base_url changed. In JavaScript it is the baseURL option; in a lot of CLI and editor tools it is an OPENAI_BASE_URL environment variable. Some end-user apps expose it too: DEVONthink 4, for instance, lets its Chat point at Ollama or LM Studio directly, so search and summarize run over your own documents with nothing leaving the machine.

The honest gotchas

It is not always drop-in. The places it breaks, roughly in order of how often they catch people:

Context window. A local model often defaults to a small context (Ollama defaults to 2048 tokens unless you raise num_ctx). If an app sends a big document and the model only reads the last 2k tokens, it looks like the model "ignored" your content. Raise the context in the model config, or send smaller chunks.
Missing features. Some apps assume function calling, JSON mode, or vision. Not every local model or runner supports all of them, and the failure is often silent. Check the runner's compatibility notes for the specific feature you depend on.
Streaming quirks. Most local servers stream fine, but a few clients expect particular SSE framing. If a response hangs, turn streaming off in the client as a test to isolate it.
Model quality. A 7-8B model is not GPT-class. For classification, summarizing, tagging, and search over your own material it is often plenty; for long chains of reasoning it will disappoint. Match the job to the model rather than expecting parity.
RAM. Everything is bounded by unified memory on a Mac. 16GB comfortably runs a capable 7-8B model; below that, stay in the 3B range and keep the context modest.

Why bother

Two reasons that actually matter. It is free after the download, so there is no per-token meter running while you iterate on a prompt fifty times. And nothing leaves the machine, which for anything sensitive (client data, personal notes, a private codebase) is the difference between "I can use AI on this" and "I am not allowed to."

The best part is that because it is one interface, you do not have to commit. Keep a cloud base URL for the genuinely hard reasoning and a local one for the bulk, private, high-volume work, and switch between them by changing a single string.

Written with AI assistance and edited by a human. Endpoint details reflect public docs as of July 2026 and move quickly, so check each project's current docs for exact paths and defaults.

MLX vs GGUF on Apple Silicon: which local LLM format should you actually use?

Jackson Ly — Tue, 07 Jul 2026 01:16:53 +0000

If you run local models on a Mac, you eventually hit the same fork in the road: the same model is available as a GGUF file and as an MLX version, and something has to tell you which one to download. The short answer is that MLX is faster on Apple Silicon and GGUF goes everywhere. The useful answer is knowing when that trade actually matters, because for a lot of setups it does not.

Here is the practical version, with the reasons.

What the two formats actually are

GGUF is a single self-contained file. Weights, tokenizer, metadata, and quantization parameters are all bundled into one portable blob that llama.cpp (and everything built on it) can load anywhere: Mac, Linux, Windows, CPU, CUDA, Metal. That portability is the whole point of the format.

MLX is Apple's array framework, not a file. An MLX model is a directory of safetensors files plus a config that the MLX runtime reads directly. It is built to run on Apple Silicon and quantize natively against the unified memory pool. It does not leave Apple Silicon, full stop.

So this is not really "two file formats." It is one portable format and one Apple-native runtime that happen to ship the same model weights.

The performance gap is real but bounded

On the same Mac, at the same quantization level, MLX runs roughly 15 to 40 percent faster than GGUF and uses about 10 percent less memory. The speedup comes from MLX being compiled for Apple Silicon and operating directly on the unified memory pool, where the CPU and GPU share one block of RAM and every gigabyte is usable as model memory.

There is one quality nuance worth knowing. At 4-bit, GGUF's Q4_K_M uses mixed precision inside each layer, so it can hold quality slightly better than a naive 4-bit MLX quant. It is a small difference, but if you are quantizing aggressively on a memory-tight machine, it is the kind of thing that shows up in outputs.

	GGUF	MLX
What it is	Portable single file	Apple framework, safetensors dir
Runs on	Mac, Linux, Windows, CPU, CUDA, Metal	Apple Silicon only
Speed on M-series	Baseline	~15 to 40% faster
Memory	Baseline	~10% less
4-bit quality	Slightly better (Q4_K_M mixed precision)	Fine, marginally behind at 4-bit
Best for	Portability, ecosystem, cross-platform	Raw throughput on Apple Silicon

Tool support is the part people get wrong

The format you should pick is partly decided by the tool you already use.

LM Studio has had an MLX backend since late 2024 and also runs GGUF through its bundled llama.cpp. So on LM Studio you genuinely get to choose per model, and picking the MLX build on an M-series chip is usually free speed.

Ollama ran GGUF through llama.cpp's Metal path for a long time. Its 0.19 preview added an optional MLX backend that reports roughly double the old speed, but it targets Macs with 32GB or more of unified memory. On a 16GB machine you stay on the GGUF/Metal path, so the MLX question does not even come up for you yet.

That 32GB line matters more than the raw benchmark. A lot of "MLX is 2x faster" claims quietly assume a machine most people do not have.

The portability catch that bites later

MLX being Apple-only is fine until it is not. If you are building something that might need a CUDA fallback, a Linux server, or a Windows user, and you shipped MLX as your only build, you are re-quantizing under pressure the day that requirement lands. GGUF would have just run.

This is the real decision axis, and it is not about speed. It is about how long your setup has to live and where it has to run. A weekend project that only ever runs on your own MacBook has no reason not to take the MLX speedup. Infrastructure that will outlive any single runtime should default to GGUF, or ship both.

The decision, compressed

Personal use on an M-series Mac with 32GB or more: take MLX. It is faster and lighter and you lose nothing you will miss.
16GB Mac, or you just want the simplest path: GGUF. You are likely on the GGUF path anyway, and the portability is free insurance.
Cross-platform, a server, or anything that might need CUDA later: GGUF, or build both. Do not paint yourself into an Apple-only corner for a 20 percent speedup.
You are not sure: GGUF. It runs everywhere, and you can always add the MLX build later when you know you want it.

The honest summary is that both are good, and on a modern Mac you can usually try each in a couple of minutes with the same tool. Speed is the easy axis to measure and the wrong one to optimize first. Portability is the one you only notice when it is gone.

Written with AI assistance and edited by a human. Format details reflect public information as of July 2026 and move quickly (Ollama's MLX backend in particular is young), so check each tool's current release notes before you rely on a specific number.