DEV Community: Niv Dvir

What broke when my Claude Code projects started talking to each other

Niv Dvir — Tue, 01 Sep 2026 22:59:26 +0000

I run about a dozen Claude Code projects on one laptop. Each owns its own domain, and they open sessions against each other to ask questions they aren't allowed to answer themselves. The rules governing that are a separate piece. This one is the plumbing: six things that broke while making those rules actually run, and what each one cost to diagnose.

Everything below was observed on Claude Code builds 2.1.234 through 2.1.241 in August 2026. Two items have since been re-verified on 2.1.252 and are marked; the rest have not been re-tested, so read them as dated observations rather than permanent properties. Where a fix is a workaround rather than a cure, I say so.

Two different stalls that look identical

A session that tries to touch anything outside its own working tree raises a permission dialog. Unattended, nobody answers it. The session hangs, and a hung session is visually indistinguishable from one that is thinking hard.

Separately, shell commands the permission system cannot statically verify (a loop with variable expansion, say) prompt every time, with the same silent hang.

These look like one failure and need different cures. Pre-approving read-only tools at launch and forbidding shell use in the request itself, sub-agents included, cures the shell class. It does not touch the cross-tree class: pre-approving the Read tool authorises the tool, not paths outside the session's tree. A session launched under that same recipe still blocked forever when asked to read one file in another project.

Only confinement cures the second one. Give the receiving session nothing that requires leaving its own project, and there is no dialog to hang on.

One consolation, and it is a real one: a stalled session is recoverable. Attach to it from a terminal and the pending dialog renders. Either answer resolves it, and the session then drains every message that queued while it was blocked. Nothing is lost.

A trust prompt that fires before your message arrives

A project whose MCP servers nobody has approved yet blocks at startup, before it receives anything at all. Your message never lands, and nothing tells you that. From the sending side it looks exactly like a session that is ignoring you.

A human at a terminal can clear it one session at a time. The only fix that scales is a standing trust decision for that server.

The receiving model refusing on safety grounds

One request came back refused because several terms in it read as security-related in combination. The receiving agent classified the request and declined to answer it.

Re-sending with that section removed worked. That is the one cure I measured; rephrasing in place seems plausible and I did not try it.

Worth naming because it is a structural property rather than a bug: when the receiver is a model, the content of a request can prevent its delivery. No transport-layer guarantee survives that.

The silent stall that looks like working

A background session with no usage credits left accepts your message, prints it, and dies. It goes idle without replying, which is visually identical to still thinking.

The cause is almost never a real budget problem. A background session does not inherit the model of the session that launched it. It takes the default from user settings. An interactive session running one model therefore spawns background sessions on a different one, and those die on the second model's quota while the first sits untouched.

Passing the model explicitly at launch fixes it. Before that, read the session's log rather than inferring from its silence: idle-with-no-reply and still-working are the same picture from outside, and only the log distinguishes them.

An addressing bug, and how I isolated it

Background sessions print an id at launch. After processing their first task they rename themselves from that task's subject, and the launch id then stops resolving for messaging, reporting the session as unreachable, while every other surface still accepts it: logs, stop, and the id field in the JSON agent listing.

I isolated it with a negative control. A session that blocked before producing any output never renamed itself, and messaging its launch id worked. I then unblocked that same session, watched it rename, and messaged the identical address again: same session, same process, same build. Rejected. The rename was the only variable.

That is filed as anthropics/claude-code#89946. An issue's status is live state owned by someone else, so check the link rather than trusting a sentence I wrote weeks ago.

The workaround is small: re-discover a session by its current name before any follow-up message.

One sign the addressing layer is moving. On a newer build, a bad address now returns a disambiguating error naming the candidate sessions instead of a bare "not reachable." That is a different failure from the rename bug, but it shows the surface is being worked on.

Re-verified on 2.1.252: cross-session delivery and typed message exchange still work as described.

The one that isn't a bug: a read-only transport can carry a ruling, but not work

The recipe above deliberately strips the receiving session down to read-only tools. That confinement is exactly what makes unattended delivery safe: nothing can prompt, so nothing can hang.

The cost surfaced on the first real cross-tree job: a project rename whose final stage required edits a confined background session had no way to make. The job finished only because a human instructed a different session to run shell commands inside the other tree. That was outside the protocol, on explicit instruction, and recorded in our own log with the sentence "it worked, and it is not a precedent."

A fair summary of the field state: the mechanics reliably move questions and answers. They have not yet executed a real cross-tree modification end to end.

The rule that came out of this responds to it rather than solving it. Constrain what a session may touch, not which tools it holds. Launching a head stripped of the tools it needs and then asking it a question only those tools could answer is the mistake underneath the whole class. Pre-approve what the work requires, bound the scope, and let it refuse if it still cannot answer.

Whether a full cross-tree modification runs cleanly under that rule is untested.

Re-verified on 2.1.252: a session rooted in a directory that contains other projects' trees still reads all of them with no prompt at all.

What the list is for

Two of these six are bugs. Two are consequences of running models rather than services on the receiving end. Two are properties of the permission model that no configuration removes. Only the first pair will be fixed by someone else.

That distribution is the useful part. When you wire independent agent projects together, the failures you should budget for are not the ones in the transport layer. They are the ones where a session's silence has three possible meanings and the interface shows you the same thing for all three. Most of my diagnosis time went to telling those apart, not to fixing them once told.

A protocol write-up with no failures in it is a design document wearing field-report clothes.

The rules these sessions were carrying, and who answers to whom, are set out in the companion piece.

A Garbage-Collector Layer Between Claude Code Projects

Niv Dvir — Sat, 29 Aug 2026 01:29:12 +0000

Different species, the same marks. Separate trees, one soil, and the birds carrying what crosses between them.

A request you have made a dozen times, one that used to run at once, now sits and thinks. Tokens burn before a file changes. The forums blame the model, and often they are right. One of the reasons is quieter: the project is working from a picture of your code that stopped being true.

A Claude Code project keeps a managing layer above its content: instruction files, notes, who owns what. You write that layer once. The content moves every day, and nothing tells the layer. A missing file fails loudly. A stale one fails in silence. The project acts on an old map, learns of changes late, and cannot measure how far it has drifted. That drift is the garbage, and nothing collects it.

It cannot collect its own. A stale picture cannot know it is stale. Only the owner of the live state can. Change something another project owns and you leave work you cannot see. Asking its owner puts the only one who can spot it in the loop while you are still there, not after it has cost you a week.

A boss does not fix it, because the worlds differ. One project builds a product, one writes about it, one runs the outreach. The product project knows nothing about publishing. The article project cannot tell which version shipped this morning. Put either in charge and it rules on what it does not understand. Each needs the opposite: what its peers know, taken whole, to be the best in its own field.

That is what a working collector gives you, and the collector is a rule. No project acts on another's content because it looked right from outside. It asks the owner, who holds what the files do not say, names the work the change drags with it, and knows when someone touched it. Work happens by consent.

No runtime does this. The six laws that follow are the collector, and the owner of each project does the collecting.

The shape: a forest of projects

The unit here is the Claude Code project (CC-project from here on). An agent is one worker inside it. Inside a project, orchestrators already handle hierarchy: one spawns workers, all under one owner. Between projects there is no default. A tree in a forest is not higher than the tree beside it. Hierarchy exists only inside each trunk. Between trunks there is law.

A directory is a CC-project if and only if it carries its own CLAUDE.md. That file gives it a head, the one owner that answers for everything under it. Everything else is content of the nearest CC-project above. A subproject passes the same test and is a whole unit, the way a subsidiary is a whole company.

Two kinds of edge exist. Inside a trunk, head to sub-head: commands down, answers owed up. Between trunks, head to head: asks across, with no power to command.

Containment shows ownership but does not prove it. A file can sit in one tree and belong to another's domain. A scheduled job belongs to the domain it serves. Where the layout and the record disagree, the record decides. A contested entry stays contested until the dispatcher or the human rules, and a project that settles its own contested entry has ruled in its own favour.

The permission system can see the solid line. The domain is the dashed one.

One entity sits above the trees. The dispatcher owns the working method and no content, serves every project, commands none, and insists on correct conduct. Gaps between projects end there: a contradiction that will not dissolve, an owner nobody can reach, a remit nobody recorded. A ruling inside a tree does not. With three CC-projects you hold that role yourself.

Two kinds of edge, and only two: commands down inside a trunk, asks across between them.

The six laws

Each law covers one dimension, so you can read or apply any of them alone. Only the person these projects serve may amend one. Together they are the collector: the first tells you whose a subject is, the second stamps what comes back with its kind, the third carries the request and the answer, the fourth says what a request may not carry, the fifth gates what it may cause, and the sixth keeps two paths honest with each other. The walk-through after them shows it running.

1. Ownership

Every domain has one owner: a head.

Ownership is three inseparable things: the authority to rule on the domain, the responsibility for its records, and the capability to refresh them. Capability exercised through a human’s gated hands (a login, a credential, a click they must perform) still counts; it is mediated, not missing. If a leg is genuinely missing, the ownership itself is broken: fix that, never the symptom. Tree containment is evidence of ownership, never proof: an ownership recorded as contested in the registry stands contested until the dispatcher or the human rules, whatever tree holds the files. OS-level artifacts (a LaunchAgent, a cron entry) belong to the domain they serve, not the directory that contains them. The shared configuration directory is neutral ground, stewarded by the dispatcher. The registry is ownership’s map: read it before any cross-project work; keep your own row true; never write another’s; two heads whose subjects touch must recognize each other in both directions. (Standing rulings, 2026-08: the subject of a piece, never the platform, decides its ownership; every head carries its own publication capability, and no head manages publication for others.) Nobody else touches, verifies, or rules on a domain. They ask.

The mechanism is a shared registry, the thing FIPA calls a Directory Facilitator and describes as a yellow pages service.

Read it before any cross-project work.
Keep your own row true.
Never write another head's row.
Two heads whose subjects touch must recognise each other in both directions.

Who wrote an entry matters. One written by the head it describes is authoritative. You may use anyone else's entry about a head, and it carries no authority. An owner who cannot refresh a class of fact says so and stops. A caveat does not stand in for an answer. What this removes is the confident wrong answer.

2. Records

Every record is one of five kinds, and naming the kind is part of writing it:

a fact, true about the world, refreshable from a source; a positioning choice, a deliberate presentation (a CV emphasis, an outbound stance), which may differ from fact without contradiction, set and recorded by whoever owns the presentation, the human’s own presentation only by the human, and never invented by a head; a ruling, a decision by the human or an owner, which exists only once written into the file that governs the behaviour it changes, dated and sourced; a snapshot, a dated copy taken for one task, expiring with the task, never authority, and a snapshot of another head’s live state is taken with that owner’s knowledge, stamped with source, and discarded when the task ends; a draft, what was true when written, an archive, never a queue. A record’s kind is fixed when it is written; reclassifying your own record under challenge is resolving the dispute in your own favour: the defect, not a defence. Authority has exactly one home per kind: conduct here, ownership in the registry, procedure in the mechanics file, each domain’s rules in its owner’s governing files. Everything else anywhere, memories, boxes, histories, seeded context, is a pointer or a snapshot. A fix that has not landed in its single home has not happened. When two records disagree, check their kinds first: most contradictions are a positioning choice read as a fact, and dissolve. A genuine contradiction between owners, including a dispute about a record’s kind, is handled one way: tell the peer first, record both readings openly, neither side acts on either, until it is ruled. The dispatcher rules, except where the question is about the person all these projects serve, whose identity, history, or intent only they answer. Holding it open is correct behaviour; quietly resolving it is the defect.

Hold a pointer, not a copy. For any fact another head owns and can change, store the address and ask when you need it. When you must copy, stamp it with date and source and let it expire with the task.

Going stale is the small problem. The large one is the owner hearing their own record quoted back at them. And a ruling agreed in conversation has not happened, because the next session reads the governing file and not the conversation.

The figure shows the shape of it: two projects disagreed about where a third one lived, both held snapshots, and naming the kinds settled it. What the laws do not yet do is notify. Nothing told those projects the path had moved, and a rule that dissolves the contradiction after the fact leaves room for one that announces the change before it.

One is a snapshot, the other the owner's live fact.

3. Communication

You reach a domain only through its owner.

Enter at the head; the request descends one level at a time, and every node on the path is a participant, free to amend, answer, or refuse, never a relay. The answer returns up the same path, hop by hop. Name the subject, never another tree’s internals. One session per coherent request, and the test is: would the answer to one change the answer to the other? Yes, one session; no, separate sessions. A request is one of three kinds, named explicitly: a ruling (is this true, is this yours), evidence (give me your material, and the holder decides what leaves its scope), or capability (operate your tool for me, tiered by what the tool does, never by who asks). Asking is how facts cross boundaries. Never verify another’s domain yourself, and never impose a constraint that forbids an owner from refreshing its own facts. Constraints bind scope, not tools. An owner who cannot refresh a fact class says so and stops; a caveat is not a substitute for an answer. An owner who cannot be reached at all is a gap: log it to the dispatcher’s queue and stop, because self-service is not a fallback, and urgency that cannot wait goes to the human.

Skipping the middle costs. A node that holds two commitments, each visible to only one of its children, is the only one placed to see that they conflict.

Every node on the path may amend or refuse the request. The answer climbs back annotated.

You never need another tree's internals. Name the subject, address the head, and let the chain place it. If you find yourself naming a sub-subproject, you have left the protocol.

4. Authority

A request carries the requester’s need, never its permission.

Laundering is the circumvention of a refusal, actual or reasonably anticipated: if you were denied something, or never asked because you expected denial, a peer doing it for you is the confused deputy, and it goes to the human, however large the efficiency argument. Providing a capability you legitimately own to a peer who was never refused is service, exceptional, not routine, and it is gated like any other action, by its consequence (Law 5). The complement of service: a deliverable’s substance is produced inside the domain responsible for it, and a peer supplies evidence and capability, never your work. If you cannot tell whether you are looking at service or at a refusal being routed around, you are looking at the second. Ask.

This is the confused deputy, arriving as a polite message from a peer. The law makes the deputy check the authority instead of inheriting it. The harder half in practice is the complement of service: the substance of a deliverable comes from the domain that answers for it, and a peer supplies evidence and capability. Claude Code now ships the first half of this rule itself. A message from another session does not count as consent, and a session must not ask a peer for an action its own permissions denied.

5. Consequence

The check precedes the act, always: a ruling that arrives after delivery is damage control.

Then gate every action by its consequence. Reversible, inside your own domain: proceed (GREEN / AMBER). Reversible, inside another’s domain: its owner decides. Irreversible, outbound, money, credentials, deletion, in whoever’s domain: the human, never automatic (RED). A recorded ruling outranks this ladder wherever it is stricter, and a standing gate binds even inside your own domain. And where an intervention risks real harm to a domain and rational weighing gives no sharp answer, the question returns to the human. Ordinary cross-project gaps are not that: they go to the dispatcher’s queue, not to the human’s desk. Operational tests: reversible means the same actor can restore the prior state, at the same tier, with no residue. Outbound means it reaches any party other than the human. Real harm means severe enough that the human would have preferred the act avoided or undone. GREEN, AMBER and RED are the official names of these rungs; satellite files that speak tier language are speaking this law.

Every action lands on one rung. A recorded ruling outranks the ladder wherever it is stricter.

Write the operational tests next to the gates, because a gate is only as good as the predicate under it.

6. Convergence

Two lanes serving one goal are not independent, however unrelated their subjects look.

Each must know the other’s live state, in both directions, or neither can price its own urgency. Say so explicitly the moment you discover a convergence, and keep the state flowing.

They are alternative routes to one outcome, so neither can price its urgency alone: progress on one changes what the other should spend. Detecting a convergence is the hard part. Two projects with different subjects, owners and vocabularies have no natural moment at which either notices. It is the law used least and found late most often.

One request, end to end

Project A needs a fact that project B owns.

A reads the registry, learns the subject is B's, and sends an evidence request into a session inside B. It addresses B's head, not the node A suspects holds the file, and names the subject in A's own terms. B's head decides the request is B's and places it one hop down. Each node answers or passes it on, until the owner of the content decides what leaves its scope. On the way back up, the middle node adds the one thing only it can see: a commitment its sibling made last week bounds the answer. B's head consolidates and replies once. A records the reply as a snapshot with source and date, keeps the pointer to B, and asks again next time.

When A wants B to do something, it sends a capability request naming the tool's effect, and B applies the gates. What A may not do is take a refusal from B and send the same action to project C because C has the same tool and no history of saying no. A records the refusal, stops, and takes it to the human.

When A and B contradict each other, they check the kinds first, because most of these are a snapshot read as a fact and dissolve on sight. If the contradiction survives, neither side resolves it. A tells B, both readings go on record with their authors, neither acts, and the dispatcher rules. Holding it open is the correct state until then.

What it costs, and what it does not do yet

This is a set of files and behaviours, and nothing in it intercepts a call. Only behaviour enforces it. The files exist to make correct behaviour cheap to look up. Renumbering a live standard has a price too: every note that cites a bare number then cites nothing, so write the concordance before you delete the old numbering.

Three limits stand today.

The mechanics move questions and answers between heads and have not yet carried a full cross-tree modification from request to landed change.
One lane asserted convergence with another, and nobody has confirmed it from the other side.
The laws are new, checked on paper and not by months of use.

What broke in the tooling is a separate field-notes piece. A protocol write-up with no failures in it is a design document in field-report clothing.

Test your rules like code

Rules a system runs on are code, and "it reads better now" is not a test. You wrote the sentences, so you are the worst judge of whether a stranger can follow them. Three checks catch what the others miss.

Blind adjudication. Take a dozen decisions your projects have already made and strip every trace of how they came out. Hand them, with the rule text and nothing else, to two or three agents that have never seen your system, and let each rule alone. A divergence from your verdict is either a real disagreement about what the rule should say or a sentence that reads two ways. You want both before the rules go live.

An inventory of undefined predicates. List every term the rules lean on without defining, then attach a test to each or delete it. Gates fail on undefined predicates, because everyone applying one is sure they know what the word means.

An adversarial read. Set an agent on the draft with no brief except to break it, and watch for a tidier definition that decides an open question while claiming to reorganise. A clean-up pass produces that failure more than any other.

The bar is not that every situation has a precomputed answer. It is that no situation lacks a lawful next move: following the rules always gives you something you may do, even when that is to hold the question open, tell the other side, and act on neither reading. Holding open is lawful. Resolving in private is the defect.

Where the ideas come from

Almost every piece of this has a published name already.

A tree, one head plus everything under it, is a bounded context in Evans and a holon in Koestler. The forest of independent trees interacting as peers is Koestler's holarchy, and distributed systems call it a federated architecture. The shared roster of who owns what is FIPA's Directory Facilitator, a yellow pages service, and Evans's context map. The dispatcher is a platform team in Skelton and Pais. Never modifying another tree's content, only asking, is Evans's anticorruption layer. A request that carries no authority of its own is Hardy's confused deputy.

Koestler describes hierarchies whose branches interlock with other hierarchies to form horizontal networks, which is the forest, half a century early. Agent communication languages typed requests by act long before this, with KQML's ask-if and FIPA's query-if and request. Entering at the head is Evans's aggregate root. Gating by reversibility is Amazon's one-way and two-way doors.

Two pieces are in no source I reached. The intermediate node as an active participant, free to amend a request in transit, where the closest specified behaviour requires a faithful relay. And marking a registry entry by who wrote it, so that anyone may use an entry written about a head, and it carries no authority, where FIPA's registry disclaims responsibility for what anyone registers and restricts nobody from registering anything.

Since this was written, part of it has shipped. Claude Code's cross-session messaging gives independent sessions a channel and a refusal switch, and its docs carry the fourth law: a message from another session does not count as consent, and a session must not ask a peer for an action its own permissions denied. The docs carry no notion of ownership, jurisdiction, or a request typed by what it asks for. Single-agent collectors exist too. Self-GC prunes a long-running agent's context, and context-diet audits persistent context for staleness. Each collects inside one agent. None collects between projects.

What transfers

If you run one agent, none of this applies. If you run several with real capability over shared resources, four things do.

Only behaviour enforces ownership. No configuration makes the permission guard and the ownership boundary coincide. "It did not prompt me" is not "it was mine to do."

Ask, don't verify. When another head owns a fact, ask for a per-claim verdict of confirmed, wrong, or not stated, and close with "what did I get backwards?"

Hold a pointer, not a copy. Store the address of anything that can change, and stamp any copy you take with its date and source.

Test your rules the way you test code. Blind adjudication is an afternoon, and it is the only way to learn whether a stranger reads your rules the way you meant them.

The literature is in domain-driven design, Team Topologies, Koestler, and a 1988 note about a compiler tricked into writing a file it should not have touched. Read those before you write your eleventh clause.

Reading, most useful first

Team Topologies (Skelton and Pais, 2019). Settles the platform-team role, and is the only source built for this problem shape rather than adapted to it.
Domain-Driven Design (Evans, 2003). Settles domain ownership and the seams between domains. The condensed DDD Reference is enough if time is short.
Koestler on the holon. Settles the forest shape. Two texts get confused: "Some General Properties of Self-Regulating Open Hierarchic Order" is Appendix I of The Ghost in the Machine (1967), and "Beyond Atomism and Holism: The Concept of the Holon" is the 1969 Alpbach paper. Check which you want before citing.
Hardy, "The Confused Deputy" (1988). Four pages, and the failure mode the anti-laundering rule exists to prevent.

FIPA's specifications are worth reading for the Directory Facilitator definition. The live fipa.org domain no longer serves them, so use the Internet Archive copies.

Originally published at https://nivdvir.github.io on September 8, 2026.

max_pixels is a token budget in disguise — and the right cap depends on the size of what you're looking for

Niv Dvir — Thu, 11 Jun 2026 07:38:21 +0000

Run the same image through the same Qwen2.5-VL model on different runtimes, and it can cost anywhere from 8 to 16,384 visual tokens — a 2,000× spread — depending on which inference stack you picked. Nobody changed the model. They just disagree about one config value: max_pixels.

This post is three things: a one-line identity that makes max_pixels legible, a measured set of accuracy-vs-budget curves showing that the optimal cap depends on the size of the thing you're trying to find, and a survey of what every major runtime defaults to.

TL;DR — max_pixels ÷ 28² is your image-token budget. The vendor recommends ≤ 1,280 tokens; most runtimes silently ship 16,384. Measured on real 4K screenshots: big targets (panels) peak at 1,280 — more tokens makes them worse. Tiny targets (icons) never stop gaining. Set the cap to the size of what you're hunting.

Decoding max_pixels: one line of math

Qwen-VL models turn pixels into tokens in two stages. The vision transformer slices the image into non-overlapping 14×14 px patches; then a small MLP merges every 2×2 block of neighboring patches into one token. Compose the two and each LLM token owns a contiguous 28×28 px square of the image:

factor = patch_size × merge_size          # 28 for Qwen2/2.5-VL, 32 for Qwen3-VL
tokens = (W × H) / factor²                # area ÷ area-per-token

Which means the max_pixels value every config ships is just a token cap wearing a big number:

max_pixels = max_tokens × factor²

Decode the Qwen2.5-VL checkpoint default with it: max_pixels = 12,845,056 = 16,384 × 28². That's a 16,384-token image budget. The model card's own recommended range is 256–1,280 tokens. The shipped default is 12.8× the recommended ceiling — and because smart_resize only downscales to the cap, every sufficiently large image silently runs at a resolution the model wasn't tuned for.

(The recommended token budget is the same across the whole Qwen-VL line; only factor changes per model — 28 vs 32. So Qwen3-VL's recommended pixel cap is 1280 × 32² = 1,310,720, not 1,003,520. The token number is the portable one; the pixel number is not.)

So what's the right cap? Wrong question — right for what target?

An earlier article established that this cap is the load-bearing preprocessing variable for grounding (capping at 1,280 alone lifted mean IoU from 0.54 to 0.74 on high-res screenshots — Bug #5 there). This post asks the next question: is 1,280 right for everything?

I first ran the obvious follow-up — sweep the budget, find the knee — and got a clean answer that fell apart the moment I controlled for one variable: how big the target is relative to the frame. So here is the experiment that holds up.

Setup. Qwen2.5-VL-7B-4bit on Apple Silicon (mlx-vlm), grounding on real ~4K screenshots from ScreenSpot-Pro (26 professional apps). 200 target elements spanning the size spectrum — from sub-0.1% icons (ScreenSpot-Pro's own human-annotated targets) up to >5% panels and canvases. Budgets 256 / 1,280 / 4,096 / 16,384 tokens; metric is IoU against the reference box.

Where the large-target references come from (ScreenSpot-Pro has essentially no large-target annotations — its biggest is 4.7% of frame):

The model proposes named regions it recognizes on each screen.
Each proposal is re-localized by an independent zoom-in pass at native resolution.
Every element passes an ambiguity screen: if its description also matches another region on the same screen, or the box fails crop-level verification, it's out.

That screen removed 64% of candidates.

The screen isn't pedantry — it's load-bearing. An ambiguous description doesn't just add noise; it systematically fakes a high-budget penalty: at a low budget the model picks the one salient candidate, at a high budget it can suddenly resolve the other match and box that instead. On the contaminated subset the high-budget drop measures −30%. On the cleaned set, −18%. The cleaned number is the claim.

The result — peaks march right as targets shrink:

target size	n	GT source	256	1,280	4,096	16,384	optimum
large (>5% of frame)	36	audited	0.343	0.643	0.574	0.526	1,280
medium (1–5%)	8	audited	0.221	0.295	0.279	0.277	1,280, then flat
small (0.1–1%)	9	human	0.029	0.111	0.179	0.154	~4,096
tiny (<0.1%)	43	human	0.003	0.040	0.056	0.079	still climbing at 16,384

Two failure directions, and they belong to different size regimes:

Large targets degrade above ~1,280 tokens (−18% IoU by 16k, audited — and 9.7s → 62s per query on these 4K images, a 6× latency tax). The failure mode is specific: the model still finds the panel (click-through-center holds at 0.86 → 0.75) but the predicted box drifts past the true boundary. More tokens, worse geometry.
Tiny targets never stop gaining. Their curve is still rising at the model's architectural ceiling. An icon a few dozen pixels wide on a 4K frame dissolves at any budget the large-target regime would call sane.

Why would more detail hurt? The budget isn't a compute limit — it's a distribution limit: 16,384 tokens is 12× past the range the model was tuned on, and long-range box geometry is what goes first when position statistics drift out of that range.

So the model card's 1,280 isn't a universal sweet spot — it's the large-target optimum, and the crossover point where the two regimes trade places. If your targets are panels, 1,280 is exactly right. If they're icons on 4K screens, no flat cap is right, which is why the current research wave (more below) zooms instead of raising resolution.

Honest scope notes: one model family, one task family (UI grounding), and the large-target references are model-anchored (validated, but not third-party human labels — those don't exist for large UI regions; the small/tiny rows are pure human GT). The medium/small bins are thin (n=8/9). And one finding about the method itself: asked to propose elements, the model essentially cannot name tiny ones uniquely — only 2 of the 148 elements it proposed were tiny — so the tiny bin rides on ScreenSpot-Pro's human annotations.

This is an old law in new units

If "optimal scale depends on target size" sounds familiar, it should — other fields measured it years ago. Object detection: SNIP (CVPR 2018) showed large objects "become too big to be correctly classified" at high resolution, and HRDNet measured large-category AP dropping 7.6 points when input resolution doubled. Aerial imagery: SAHI tiles images for small objects but has to add full-frame inference back specifically to recover large ones. The principle is scale-space theory's "characteristic scale" (Lindeberg, IJCV 1998): every structure has one scale at which it's best detected.

What hasn't existed — as far as I can find — is this curve in VLM token-budget units, binned by target size, on screens. The pieces are published separately: Phi-Ground measured token-budget ablations (plateau past ~2k, no size bins); AdaZoom-GUI measured unconditional zoom hurting easy targets; Qwen's own report (Table 7) saw the off-distribution mechanism from the upscaling side; I reported the shipped-default discrepancy in mlx-vlm #1175. This post adds the size-binned curve that connects them. Connecting dots, not discovering them.

It also explains why the 2025–26 GUI-grounding literature (ZoomClick, UI-Zoomer, MEGA-GUI, AdaZoom-GUI) converged on coarse-pass-then-conditional-zoom: a first pass at a moderate cap is the large-target optimum and localizes coarsely; the zoom pass gives small targets high effective resolution inside a crop. The curve above is, in effect, the operating-point table those policies have been picking by ad-hoc sweep.

What every runtime actually defaults to

Every major stack implements the same resize algorithm (HF's smart_resize, sometimes imported verbatim) — and then they all pick different budgets:

runtime	effective default budget	notes
transformers	16,384 tokens	the checkpoint config wins over the class default of 1,280
vLLM	16,384	inherits the checkpoint; profiling reserves 16,384/image
SGLang	16,384	env default `SGLANG_IMAGE_MAX_PIXELS = 16384·28²`
TensorRT-LLM / NeMo	16,384	reuse the HF processor as-is
llama.cpp	max 4,096, min 8 (!)	so low it logs a warning telling you to pass `--image-min-tokens 1024` for grounding
Ollama	~1,280	hardcoded ~1 MP cap — closest to the recommended value
mlx-swift-lm	1,280	defaults to the card's recommendation as of PR #243, override per request

Same model, same algorithm, defaults spanning three orders of magnitude. If your Qwen-VL grounding accuracy differs across serving stacks, check this value before suspecting the weights. The inflated default is dangerous precisely because it only bites on large inputs: test on 1080p screenshots and everything looks fine; feed a Retina capture hunting for a panel and the boxes quietly bloat.

What to actually do

Know your size regime, then set the cap from the table. Panels, windows, layout regions: max_pixels = 1280 × factor² — the vendor number, now with a measured reason. Buttons and fields on ordinary screenshots: headroom to ~2–4k tokens. Icons on 4K: no flat cap saves you — budget what latency allows, or use a coarse-then-zoom pass like the papers above.
Never let large images run uncapped into the ≥5k zone when your targets are large — the curve says you pay 6× latency to lose accuracy.
Per runtime: transformers/vLLM/SGLang — pass max_pixels explicitly (e.g. mm_processor_kwargs) rather than trusting the checkpoint config. llama.cpp — raise the floor (--image-min-tokens 1024) exactly as its warning says. Ollama and current mlx-swift-lm — you're already at the large-target optimum; raise deliberately for small-target workloads.
Don't copy pixel numbers across models — Qwen3's factor is 32, so the same token budget is a different max_pixels.

Provenance

This grew out of an earlier project getting Qwen2.5-VL grounding working natively in Swift on macOS, where the pixel cap turned out to be the load-bearing preprocessing variable. The default-to-recommended-budget change is merged upstream in mlx-swift-lm #243. The sweep harness, the element set with its audit verdicts, and the raw per-test results are reproducible; the budget math is checkable from any model's preprocessor_config.json in two lines.

If you maintain a runtime that serves Qwen-VL models and ships the 16,384 default — the fix is one line, and your users' large-target grounding gets better and 6× faster.

Your parity gate must enforce the number you publish: a testing methodology for porting ML models across runtimes

Niv Dvir — Fri, 08 May 2026 16:28:08 +0000

You ship an ML model port from Python to another runtime, say Swift, C++ or ONNX. The build succeeds. The model loads. You feed it the canonical test image and the output looks right: valid JSON, sensible structure, plausible coordinates.

Then you measure pixel-for-pixel against the Python reference. One bbox edge is off by 9 px. Or 200 px. Deterministically. On every test image. It has been that way for weeks, because your test gate ran at a 30 px tolerance and never flagged it.

Silent drift like this hits any cross-runtime ML port that doesn't make a strict parity gate a first-class part of the build. Here is the four-component setup that catches it, generalized for your own port.

Your parity gate has to enforce the number you publish. A gate looser than the claim is not a gate. It's a decoration that gives you false confidence.

The 30 px gate that hid a 9 px bug

The published claim was "≤ 2 px parity on all 8 edges of both panels" against the Python mlx-vlm reference. The gate enforcing that claim ran at TOLERANCE=30 px, and had run at 30 px since early prototyping, when anything sub-50-px was a win.

Two of three canonical test images were drifting by ~9 / ~8 / ~5 px on outer edges. Nobody noticed, because the gate was 15× looser than the published number it was supposed to enforce.

Preparing to upstream the patch forced the forensic re-measurement, which put the actual measurements next to the published prose and exposed the gate as a decorative artifact rather than a check.

The fix wasn't a single bug, though there was a bug, in chat-template content ordering. The deeper fix was recognizing that the test infrastructure no longer matched the claim it was supposed to enforce. Once a strict gate was wired to the actual published number, the bug surfaced on the first run.

Identical weights do not mean identical output

A port keeps the weights and the architecture, and that does not guarantee the same output. The places where sub-pixel error accumulates:

Image preprocessing. Every framework's "Lanczos" is a different Lanczos. PIL, Core Image, OpenCV, and TensorFlow's tf.image.resize all produce subtly different output for the same input. A 1- or 2-level difference at 8-bit precision in a few hundred edge pixels propagates through attention.
Numerical precision boundaries. When the model mixes FP16/BF16 in some places and FP32 in others, the exact ordering of cast → multiply → sum can move a least-significant bit. Across a few thousand multiplies, the LSBs add up.
Tokenizer / chat-template subtleties. If your runtime emits image-token positions one off from what the reference does, attention is one position off everywhere. The output looks plausible because the model degrades gracefully. It just degrades wrong.
Initialization timing. Anything that persists between calls: caches, position-ID buffers, RoPE state. A buffer that resets in Python but persists in your port produces different output on the second image than the first.

Each alone produces small drift; stacked, they produce wildly wrong output. None of them throws an error: they all pass shape checks and they all return something.

You only catch them if you compare your runtime's output against a known-correct reference, at the strictness you care about, on every edge of every output, every time the gate runs.

Four things have to exist together

1. A reference that pins three things

The reference is the source of truth. Every other piece compares against it.

canonical_baselines.json (the name in this project; call it whatever) holds the Python reference output for every canonical test image, generated at temperature=0 with a pinned model snapshot:

{
  "model": "mlx-community/Qwen2.5-VL-7B-Instruct-4bit",
  "snapshot_hash": "fdcc572e8b05ba9daeaf71be8c9e4267c826ff9b",
  "mlx_vlm_version": "0.4.3",
  "max_edge_delta_allowed": 2,
  "prompt": "Detect these two UI panels...",
  "images": [
    {
      "name": "leetcode_test",
      "path": "_zero_px_test/leetcode_test.png",
      "size_px": [3078, 2114],
      "model_resize": [1260, 868],
      "panels": {
        "question": [1, 146, 421, 626],
        "editor":   [421, 146, 881, 626]
      }
    }
  ]
}

This file pins three things: the model snapshot hash, the reference framework's version, and the input dimensions / preprocessing parameters that produced the output. All three are required. Change one and the saved output stops being a valid reference, so the gate must be re-baselined deliberately rather than passing trivially.

The published claim, "≤ 2 px on every edge", sits in that file as max_edge_delta_allowed: 2. The number now lives in code, not in prose: publishing a different number means editing this file, and the gate adjusts with it.

2. Compare every edge, not every box

strict_2px_gate.py runs the freshly-built non-reference binary on every canonical image, parses the model's output, compares each edge of each output element against the saved reference, and exits non-zero if any edge exceeds the allowed delta.

for img in spec["images"]:
    out = run_runtime(args.binary, ipath, rw, rh)
    panels = parse_bboxes(out)
    for label, ref_bbox in img["panels"].items():
        actual = panels.get(label)
        if not actual:
            failed.append((img["name"], label, "missing"))
            continue
        for i, edge_name in enumerate(["x1", "y1", "x2", "y2"]):
            delta = abs(actual[i] - ref_bbox[i])
            if delta > max_allowed:
                failed.append((img["name"], label, edge_name, delta))

Two non-obvious things this gate gets right:

Per-edge, not per-bbox. A bbox can average an IoU of 0.95 against the reference and still be wrong by 12 px on one specific edge, which makes IoU too forgiving as the headline metric. Per-edge maxes catch the asymmetric drift that the most subtle preprocessing bugs produce.
Missing outputs are failures. A bbox for "question" but none for "editor" is not "no measurement." It is a failure, and the gate flags it loudly.

3. Prove the patch lines landed

setup_and_verify.sh is the only sanctioned way to set up a parity-test clone. It fresh-clones the upstream library at the pinned commit, applies your patch, builds the reproducer with the production-equivalent toolchain, and runs the gate, aborting loudly on any failure.

# After git apply, sanity-check the patch landed the canonical fix line:
if ! grep -q 'message.images.map { _ in ["type": "image"] }' \
        Libraries/MLXVLM/Models/Qwen2VL.swift; then
    echo "Patch applied but Qwen2VL.swift is missing the image-first chat-template fix."
    echo "The patch file is incomplete — re-derive from working tree."
    exit 1
fi

The grep-the-patched-checkout trick catches the most insidious version of this problem: a patch that applies cleanly but is missing a fix line, because the patch file itself was generated incompletely. The patched build looks fine and runs to completion, and it produces output that's wrong by 9 px. Your gate would have to be tight enough to catch that.

The grep makes the gate stricter by proving that the patch landed the specific lines that fix the bug you claim to have fixed.

4. Cross-check adapters at zero tolerance

If your project has more than one consumer (an SDK, an MCP server, a dynamic-library entry point, an alternative runtime path), wire them all to one cross-adapter gate:

# scripts/parity/cross_adapter_gate.sh — runs the same image through every adapter
A_BBOX=$(./run_adapter_a "$IMAGE")
B_BBOX=$(./run_adapter_b "$IMAGE")
C_BBOX=$(./run_adapter_c "$IMAGE")
diff <(echo "$A_BBOX") <(echo "$B_BBOX") || fail "A vs B"
diff <(echo "$A_BBOX") <(echo "$C_BBOX") || fail "A vs C"

This catches integration drift: one adapter holding stale state, taking a different image preprocessing path, or drifting from the canonical implementation. The expected result here is 0 px tolerance. All your adapters call the same model with the same preprocessing, so any difference is a bug.

In this project it runs at zero tolerance across every adapter that ships, catching anyone who breaks adapter parity before the field does.

There is no fuzzy pass

The gate runs in this order on every parity check:

Fresh-clone the reference library at the pinned commit. (No reusing dirty trees.)
Apply your patch. Fail if it doesn't apply cleanly.
Grep the patched tree for the literal lines your patch was supposed to introduce. Fail if any are missing.
Build with the production-equivalent toolchain (in this project's case xcodebuild, not swift build, which takes different code paths).
Run the binary on every canonical image.
Compare every edge of every output element to the saved reference.
Run the cross-adapter gate.
Pass = no edge of any output exceeds max_edge_delta_allowed AND every adapter agrees at zero tolerance.

Any step that fails aborts the whole gate with a specific error. There is no "warning"; there is no "fuzzy pass." There is "the published number holds" or "the published number does not hold." Binary.

Build the gate before the bug-hunt

Build the gate first, before the bug-hunt, because the bugs you find in your port are the ones the gate flags. A gate at the right tolerance turns "this is mysteriously wrong" into "this specific edge of this specific output exceeds the threshold by N px on this specific image." That's debuggable. A gate at the wrong tolerance produces no information at all.

The specific scripts above are tuned for a Swift port of Qwen2.5-VL on Apple Silicon with mlx-swift-lm as the runtime, and they replaced the 30 px gate. The methodology is platform-agnostic.

If you're porting an ML model across runtimes, the minimum-viable parity setup looks like this:

One reference output file, pinned to a specific reference-framework version and model snapshot, with the number you'd publish embedded as a hard limit.
One gate script that runs the non-reference binary on every test case and compares per-output-element, per-edge (or per-token, per-pixel, whatever the units of your model output are) against the reference, with a strict tolerance.
One bootstrap script that builds the binary under test from a clean checkout with the canonical patch fully applied, and aborts if any patch line is missing.
One cross-adapter sanity check if you have multiple consumers.

Four components: a few short scripts plus a JSON file. It runs in a few minutes after the first build, which dominates wall time anyway.

The parity gate must enforce the number you publish. A gate that's looser than the claim is not a gate. Build it first.

Provenance

The bug-hunt war story (failed model approaches, the bug list, what eventually worked) is on dev.to as On-Device Document Grounding on macOS: Getting Qwen2.5-VL to Actually Work in Swift. This post is the testing-methodology companion.

The scripts I described (canonical_baselines.json, strict_2px_gate.py, setup_and_verify.sh, cross_adapter_gate.sh) live in the project's scripts/parity/ directory if you want to adapt them.

On-Device Document Grounding on macOS: Getting Qwen2.5-VL to Actually Work in Swift

Niv Dvir — Sat, 18 Apr 2026 04:05:36 +0000

The models that failed, the bugs that took weeks, and the architecture that survived.

◆ What I Wanted to Achieve

I wanted to read what was on my own screen — a long Wikipedia article, an arXiv PDF, a release note — and render an overlay on top with content annotations (summary bullets, section anchors, corner-to-corner perspective lines). Everything local on Apple Silicon. No cloud, no audio capture, no hidden assistance — just "here's a document on screen, understand it in-device, draw something useful on top."

On paper this is a well-defined pipeline: detect content regions in a screenshot, OCR them, accumulate text across scroll positions, render guide markers over the panels. In practice every step broke in a way I hadn't expected, and the working combination took months to find.

◆ What I Tried (and Why It Didn't Work)

This section might be the most useful part of this article. Each of these approaches cost days to weeks of effort. If you are building anything similar on macOS, you can skip all of them.

Florence-2

Microsoft's Florence-2 was the first vision model I tried. It supports grounding tasks out of the box -- you give it an image and ask "where is the text panel?" and it returns bounding box coordinates. On paper, perfect for UI panel detection.

In practice, Florence-2 cannot run on macOS with Apple Silicon. The model uses a custom architecture that requires trust_remote_code=True, depends on flash-attention (a CUDA-only library), and cannot be converted to CoreML. There is no MLX port. I spent two days trying different conversion paths before accepting that this model simply does not exist on Apple's platform.

If you are searching for a grounding-capable vision model on macOS, remove Florence-2 from your list immediately.

Ferret-UI

Apple's own UI understanding model seemed like the obvious choice for an Apple Silicon project. Ferret-UI was specifically designed to understand user interfaces -- element detection, widget classification, spatial reasoning about UI layouts.

It was a dead end. Ferret-UI requires CUDA flash-attention, which means it needs an NVIDIA GPU. Apple's own UI understanding model does not run on Apple's own hardware without significant porting effort. Beyond the runtime issue, the model's grounding output was not usable for my task -- I needed precise pixel-coordinate bounding boxes, and the model's output format did not map cleanly to that.

The irony of Apple publishing a UI model that cannot run on macOS was not lost on me.

Qwen2.5-VL-3B (the Small One)

After the first two dead ends, I found that Qwen2.5-VL had an MLX port via the mlx-vlm library. The 3B parameter variant (4-bit quantized) was only 2.9GB, loaded in 1.9 seconds, and ran inference in 3-7 seconds. Fast and light.

But too weak. The 3B model could identify that UI elements existed in an image -- it would say "there is a text panel on the left" -- but the bounding box coordinates it returned were hallucinated. Boxes would be off by hundreds of pixels, overlap incorrectly, or enclose regions that contained nothing. For panel detection where you need to know "the question text lives between pixels (0, 120) and (900, 800)," a model that hallucinates coordinates is worse than no model at all.

The 7B variant turned out to be the sweet spot. More on that in the next section.

Pixel-Edge Detection (No ML)

Before committing to a VLM, I tried the traditional computer vision approach. Each UI panel has a uniform background color. The question panel might be rgb(53, 67, 83), the editor panel rgb(22, 43, 54). In theory, you can find panel boundaries by detecting where the background color changes.

The algorithm worked on test screenshots. Then I tested it on a page where both panels used similar background colors. The panel border was a thin 1-pixel line that blended into the surrounding regions. Same-color-background UIs -- which are increasingly common with modern design trends -- broke the approach entirely.

Pixel-edge detection is fragile because it depends on an assumption (panels have visually distinct backgrounds) that is not guaranteed. A VLM can detect panel boundaries semantically -- it understands "this is a question panel" regardless of what color it is.

Accessibility API (AX API)

macOS has a built-in accessibility API that lets you programmatically read UI elements. For a screen reader, this sounds ideal.

The problem is that the Accessibility API cannot see inside web content rendered in Chrome. The browser exposes high-level structural elements -- the window, the tab bar, the content area -- but not individual text lines, panel layouts, or the DOM structure within the page. You get a single "web area" element that says "this is a web view" with no ability to drill into it.

If your target is a native macOS application, the AX API might work. For reading web-based UIs through the browser, it is a dead end.

Spawning a New Python VLM Process Per Inference

My initial integration spawned a new Python process for each VLM inference call. The Python script imported mlx-vlm, loaded the Qwen2.5-VL-7B model (5.3GB of weights), ran inference on one image, printed the result, and exited. The next cycle, 15 seconds later, spawned a new process that loaded the 5.3GB model again.

After three or four cycles, the Mac froze. Each process was loading the full model into unified memory, and the previous processes had not fully released their allocations before the next one started. OOM within minutes.

The fix was a persistent server architecture: load once, serve many.

◆ The Architecture That Worked

Here is the system that survived. Each component earned its place by being the last option standing after everything else failed.

Panel Detection: Qwen2.5-VL-7B via MLX

The 7B parameter Qwen2.5-VL model, 4-bit quantized, is the sweet spot for UI panel detection on Apple Silicon. The 3B model hallucinates bounding boxes. Larger models (14B+) are too slow for interactive use. The 7B variant reliably returns accurate panel coordinates when prompted correctly.

Why MLX matters. Apple Silicon's unified memory architecture means the CPU and GPU share the same physical RAM. MLX exploits this -- the model weights live in unified memory once and are accessed by both the CPU (for attention computations) and the GPU (for matrix multiplications) without copying. The 4-bit quantized model shows ~238MB resident memory in Activity Monitor, not the full weight file size, because MLX memory-maps the weights and pages them in on demand.

The prompt that works. After testing dozens of prompt variations, this format reliably produces usable output:

Detect the following UI panels in this screenshot and output their
bounding box coordinates in JSON format:
1. The "question" panel (problem description text area)
2. The "editor" panel (code editor area)

Return JSON with format: [{"label": "question", "bbox_2d": [x1,y1,x2,y2]},
{"label": "editor", "bbox_2d": [x1,y1,x2,y2]}]

Key details: ask for each panel by name on a numbered list (the model sometimes merges panels into one bbox if you describe them in a single sentence), and specify the exact JSON format you want (the model follows format instructions well).

The initial architecture: a persistent Python server. The model takes ~12 seconds to load. Rather than paying that cost every cycle, I built a Python server process that loads the model at startup and accepts requests over a simple stdin/stdout protocol:

# Server: load model once, serve forever
import sys, json
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

model, processor = load("mlx-community/Qwen2.5-VL-7B-Instruct-4bit")
config = load_config("mlx-community/Qwen2.5-VL-7B-Instruct-4bit")

for line in sys.stdin:
    request = json.loads(line)
    prompt = apply_chat_template(processor, config,
                                  request["prompt"],
                                  num_images=1)
    result = generate(model, processor, prompt,
                      image=request["image_path"],
                      max_tokens=512, verbose=False)
    print(json.dumps({"result": result}), flush=True)

The Swift host process spawns this server once, sends JSON requests on stdin, and reads JSON responses from stdout. No HTTP server, no sockets, no serialization framework -- just newline-delimited JSON over pipes.

Coordinate conversion. The model returns bounding boxes in the coordinate space of the resized image (max 1280px on the longest side, rounded to multiples of 28). To get screen pixels:

screen_x = model_x * (original_width / resized_width) / retina_scale
screen_y = model_y * (original_height / resized_height) / retina_scale

On a Retina display, retina_scale is 2.0. Forgetting this division is a common source of bounding boxes that are exactly 2x too large.

From Python Server to Native Swift

The Python persistent server worked. But it had friction: a Python subprocess to manage, a PIL resize helper, stdin/stdout JSON marshaling, and ~50ms of overhead per inference just from process communication. For an interactive-latency pipeline I wanted everything native.

The mlx-swift-lm library promised exactly this -- a Swift implementation of the MLX model runtime, including Qwen2.5-VL. Load the model in Swift, run inference in Swift, no Python anywhere. In theory, a single-binary solution.

In practice, the Swift implementation had 10 bugs (the 10th surfaced after this article was published — see the postscript). Finding and fixing them took weeks. But the result was worth it: a fully native Swift binary that runs Qwen2.5-VL-7B with zero Python dependencies, matching the Python reference's grounding accuracy on canonical test images — and exceeding it on high-resolution screenshots once the 1280-pixel cap from Bug #5 is in place.

The 9 Bugs in mlx-swift-lm's Qwen2.5-VL (plus a 10th found after publication)

This section documents those weeks. The bugs collectively made the model produce wrong bounding boxes. Fixing them was the difference between "the model hallucinates" and "the model returns grounding boxes within bench tolerance of the Python reference on every panel of every canonical test image."

1. MROPE section selection (split-select vs slice-replace). Multi-Resolution Rotary Position Embedding (MROPE) assigns different frequency bands to temporal, height, and width dimensions. The Swift implementation split the frequency tensor into three parts using modulo indexing (i % 3), which interleaves the frequencies. Python's implementation starts with temporal frequencies and overwrites height/width slices in-place: [T_0-15, H_16-39, W_40-63]. The layouts are completely different, and the wrong layout produces subtly wrong attention patterns.

2. Chat template ordering. The Swift message generator placed text before the image token in the content array. The Python implementation puts the image first: <|vision_start|><|image_pad|><|vision_end|>PROMPT. This ordering matters because the model's attention patterns are position-dependent -- putting text before the image means the text tokens attend to positions where image features have not yet been injected.

3. invFreq registered as a Module weight. The invFreq tensor was declared as a property on an Attention class that inherits from Module. MLX's weight-loading mechanism scans all Module properties and tries to load matching weights from the checkpoint. Since invFreq is a computed constant (not a learned weight), the loader either threw keyNotFound errors or silently overwrote it with garbage. The fix was wrapping it in a non-Module class to hide it from reflection.

4. rope_deltas unused during autoregressive generation. After the prefill pass, the code cleared the cached position IDs but never applied rope_deltas during subsequent token generation. The correct computation is positionIds = cache_offset + rope_deltas + arange(seqLen). Without the deltas, position embeddings drifted with each generated token, degrading output quality progressively.

5. Image resize using 1800px max instead of 1280px. The Swift code resized input images to a maximum of 1800 pixels on the longest side, producing 2688 visual tokens. A follow-up sweep showed 1280 isn't an arbitrary number — it's the empirical accuracy peak for this model on high-resolution UI screenshots. Cap-sweep on five 3000+-px canonical test images: 1280 → 0.736 mean IoU vs 0.542 at the model's default unconstrained smart_resize (the same ranking holds on UI-Vision N=20: 0.435 → 0.407). Log-probability also peaks at 1280, cross-validating the IoU signal. The training resolution and the accuracy peak coincide, which is presumably no accident — feeding the model 1800-px images put visual token positions outside its training distribution and dropped accuracy substantially.

6. Prompt format for single bbox output. Using a single sentence asking for both panels caused the model to sometimes return one combined bounding box. Switching to a numbered list with explicit labels ("1. question panel" / "2. editor panel") reliably produced two separate bboxes.

7. maxTokens not set. Without an explicit max_tokens parameter, the model generated tokens until hitting an internal limit or running out of memory. For a task that should return ~100 tokens of JSON, this caused multi-second waits and occasionally produced thousands of tokens of hallucinated output.

8. MROPE state not reset between successive images. The cached position IDs and rope deltas from one image persisted into the next inference call. When processing a new screenshot, the model's position embeddings started from where the previous image left off instead of resetting. This caused progressively worse results on the second, third, and subsequent images.

9. Vision attention mask ignored -- the ROOT CAUSE. This was the single bug most responsible for bounding box inaccuracy. The vision encoder's self-attention uses a mask to implement windowed attention (the model processes the image in patches, and each patch should only attend to patches within its window). The Swift code passed mask: .none to the scaled dot-product attention call instead of mask: .array(floatMask). Without the mask, every patch attended to every other patch globally, destroying the spatial locality that the model relies on for precise coordinate prediction.

// WRONG -- ignores the attention mask entirely
let attnOutput = scaledDotProductAttention(
    queries: q, keys: k, values: v, scale: scale, mask: .none
)

// CORRECT -- applies the windowed attention mask
let attnOutput = scaledDotProductAttention(
    queries: q, keys: k, values: v, scale: scale,
    mask: .array(floatMask)
)

After fixing all 9 bugs (and a 10th, surfaced after publication), the Swift implementation produces grounding output that matches the Python reference's accuracy on the canonical test set — and exceeds it on high-resolution images once the 1280-pixel cap from Bug #5 is in place (mean IoU 0.736 vs 0.542 at unconstrained smart_resize). The model was not hallucinating; the implementation was broken. Canonical preprocessing now lives in PR #243; the rest of the fixes are in #222 and its splits #238/#239/#242.

Upstream status. The 8 bugs that live inside mlx-swift-lm (all of the above except #6 prompt format and #7 maxTokens, which belong in consumer code) are submitted upstream as the omnibus #222 plus four isolated splits per upstream review feedback: #238 (vision attention mask), #239 (MROPE + rope_deltas + invFreq + state-reset), #242 (chat-template image-first), and #243 (preprocessing: 1280-pixel cap with CIImage Bicubic resize).

These patterns aren't specific to Qwen2.5-VL. The same mask: .none attention bug appears in Qwen2VL.swift and GlmOcr.swift; the MROPE plumbing (invFreq, rope_deltas, section selection) is shared across all MROPE-based VLMs in the library (Qwen2VL, Qwen25VL, Qwen3VL, Qwen35, GlmOcr). PR #222 is the flagship fix with model-by-model follow-ups coming as each is validated against the Python reference.

Cross-architecture validation. To sanity-check that the combined patch really does something model-independent, I ran the same A/B on mlx-community/UI-TARS-1.5-7B-4bit — ByteDance's click-prediction model, which shares Qwen2_5_VLForConditionalGeneration architecture, same hidden size, same special tokens. Same deterministic input image, same model, only the source of Qwen25VL.swift differs.

With the PR applied: 200 tokens generated, 9+ distinct coordinates tracking actual content positions. Without the PR: 52 tokens generated, output collapses to two entries at identical coordinates (141, 141) — a clean signature of broken position encoding. Two independent Qwen2.5-VL-family models, same failure mode when the fixes aren't present. The A/B reproducer is attached to PR #222.

◆ Combining Everything Into a Working Pipeline

OCR: Apple Vision Framework

Apple's Vision framework provides on-device OCR that runs on the Neural Engine at ~300ms per frame.

Recognition levels are confusing. The API has two recognition levels: level 0 and level 1. Intuitively, you might assume level 0 is the baseline (fast) and level 1 is the premium (accurate). It is the opposite. Level 0 is accurate (slower, higher quality). Level 1 is fast (lower quality). I ran with level 1 for weeks thinking I was getting the best results, then discovered I was using the fast path the entire time.

RecognizeDocumentsRequest vs VNRecognizeTextRequest. Apple's Vision framework has two OCR APIs, and they behave very differently on code content. RecognizeDocumentsRequest (the newer, WWDC25 API) is optimized for documents -- prose, forms, receipts. It silently drops lines that look like code: indented lines with brackets, semicolons, and unusual formatting. For a code editor panel, it would capture 15 out of 20 visible lines, silently losing the rest.

VNRecognizeTextRequest (the older API) captures everything -- every line, regardless of formatting. For reading code from screen, use VNRecognizeTextRequest. I discovered this after weeks of mysterious "missing lines" that turned out to be the newer API being too clever about what constitutes document text.

Bounded OCR. Rather than scanning the entire screen (which picks up menu bars, dock icons, and other noise), the OCR is bounded to the panel regions detected by the VLM. This reduces both processing time and false positives -- you only extract text from the panel you care about.

Scroll Accumulator

Most non-trivial content does not fit in a single viewport. A problem description might be 40 lines long, but only 15 are visible at once. The scroll accumulator solves this by scrolling through the content in steps, OCR-ing each viewport, and stitching the results into a complete transcript.

The stitching problem. Adjacent viewports overlap. When you scroll down by 100 pixels, the bottom 80% of the previous viewport is still visible. Naive concatenation produces massive duplication. The accumulator uses Levenshtein distance to fuzzy-match each incoming OCR line against all accumulated lines.

Threshold tuning. A line is classified as "already seen" if its Levenshtein similarity to any accumulated line exceeds 60%. I tested thresholds from 40% to 80%:

40%: too permissive -- novel lines were classified as duplicates and dropped
80%: too strict -- lines with minor OCR variations were classified as novel and added twice
60%: best F1 score for the duplicate-vs-novel classification task

Metal GPU Overlay Rendering

The overlay renders detected text and annotations as a transparent window on top of the target application.

let window = NSWindow(
    contentRect: screenFrame,
    styleMask: .borderless,
    backing: .buffered,
    defer: false
)
window.level = NSWindow.Level(rawValue: 25)
window.isOpaque = false
window.backgroundColor = .clear
window.ignoresMouseEvents = true
window.hasShadow = false

Self-exclusion from screen capture. This is critical: the overlay must not appear in its own screenshots. If it does, the next VLM inference cycle sees the overlay text, interprets it as UI content, and the system enters a feedback loop where it reads its own annotations. The fix is captureScreenExcluding(windowID:), which tells ScreenCaptureKit to exclude the overlay window from the captured frame.

◆ The Final Result

A Taste of How It Looks

Here's the system running in reader mode — detecting the main content region, reading text across scroll positions, and rendering a summary overlay with perspective lines anchored to the source:

Performance Numbers

Component	Latency	Resource
VLM model loading	~3s	Unified memory (one-time)
VLM panel detection	~18s per inference	GPU (MLX unified memory)
OCR per frame	~300ms	Neural Engine
Overlay render	<16ms (60fps)	Metal GPU
Full scroll accumulation	~40s (20 steps)	Combined
Model resident memory	~5.5GB peak	Unified memory

The VLM inference is the bottleneck at ~18 seconds, but it only needs to run when the panel layout changes (e.g., navigating to a new page). During normal operation, the OCR and overlay run continuously at ~300ms per cycle while the VLM-detected panel bounds remain cached. On an M1 Pro with 16GB, the system runs comfortably alongside Chrome and other applications.

A note on "native Swift vs Python"

Worth being honest about what "native Swift" does and doesn't buy you here. The VLM forward pass is the same Metal kernels in either language, so a one-shot mlx-vlm Python benchmark on the same image finishes within a few percent of the Swift equivalent. Swift doesn't make the model run faster.

What Swift changes is everything around the model:

Cold start ~3 s vs ~15 s — no interpreter, no PyTorch import, just mmap the weights.
Zero-IPC pipeline — CGWindowListCreateImage → VLM → VNRecognizeTextRequest → Metal overlay all run in one process with shared memory. A Python pipeline has to serialize each screenshot across the subprocess boundary, adding 30–100 ms per cycle on top of inference.
Real-time frame work becomes viable — a per-frame 16 ms budget has room for actual OCR and overlay redraw; it doesn't fit a round-trip to a Python worker.
Bounded memory over long sessions — autoreleasepool around CGImage ops keeps a 100-minute session at ~5.5 GB peak. The earlier Python-subprocess version leaked ~900 MB over the same duration through PyObjC bridging.

The headline 18-second number is the same either way. The difference is whether you can wrap that around a responsive app — startable in 3 seconds, no IPC between stages, 60 fps overlay — rather than a command-line script.

How to Reproduce This

Requirements:

macOS 14+ on Apple Silicon (M1/M2/M3/M4)
Xcode 16+
~16GB unified memory (8GB minimum, 16GB comfortable)

Model:

mlx-community/Qwen2.5-VL-7B-Instruct-4bit from Hugging Face (~5.3GB)

Key dependencies:

mlx-swift-lm (Swift package, for native VLM inference)
Apple Vision framework (built into macOS)
Metal (built into macOS)
ScreenCaptureKit (built into macOS)

Source Code

GroundingKit is the open-source macOS app extracted from this project. Clone it, build it, and try panel detection on your own screen:

github.com/NivDvir/screen-overlay-toolkit

git clone https://github.com/NivDvir/screen-overlay-toolkit.git
cd screen-overlay-toolkit
bash build-app.sh

The build script uses xcodebuild (swift build doesn't link MLX's Metal kernel library properly — the binary builds but the model returns broken output). Produces a .app bundle you can open from Finder. Menu bar app, runs entirely local on Apple Silicon.

◆ Postscript

2026-04-25 — 10th bug + reframing

After publication, while preparing PR splits per a maintainer's review request, forensic re-measurement found the Swift output drifting +9 px from the Python reference on a canonical LeetCode test image — despite this article's original "0 px on all 8 edges" claim. Two things came out of the follow-up work:

A 10th bug was hiding. Qwen2VLMessageGenerator in Qwen2VL.swift ordered chat-template content as [text, image], but HuggingFace's template for Qwen2.5-VL emits <|vision_start|><|image_pad|><|vision_end|>{text} — image content first. Swapping the order removes a deterministic +9 / +8 / +5 px bbox shift. Fix filed upstream as mlx-swift-lm #242.
Bug #5 turned out to be the load-bearing fix — not the choice of resampler. The original preprocessing PR (#243) carried a custom PIL-byte-exact Lanczos resampler so that the Swift output could pass a strict ≤2 px parity check against the Python reference. A 2×2 ablation (resampler × cap) on the canonical test set + UI-Vision N=20 then showed the resampler choice (Bicubic vs Lanczos) is no-op for IoU within bench noise — the 1280-pixel cap is the entire driver of accuracy (0.542 → 0.736 mean IoU on five high-resolution canonical images; 0.407 → 0.435 on UI-Vision N=20; both vs HF defaults). PR #243 was simplified accordingly: dropped _pilLanczosCore (~157 lines), switched to upstream's existing CIImage Bicubic chain. The preprocessing change is now +15/-5 in a single file.

The patch file (patches/mlx-swift-lm-mrope-fixes.patch) covers both Qwen25VL.swift and Qwen2VL.swift. The omnibus PR #222 has four isolated companion PRs (#238, #239, #242, #243) per the maintainer's split-friendly review preference.

2026-05-25 — upstream `max_pixels` fix landed in mlx-vlm

A separate finding: the same bug, but upstream. The mlx-community/Qwen2.5-VL-7B-Instruct-4bit snapshot's preprocessor_config.json shipped max_pixels: 12,845,056 — 12× the model's ~1.04 M training resolution. On a 5-image high-resolution canonical set this drove grounding from 0.736 (with a 1280-pixel external cap) down to 0.542 (HF defaults). I filed Blaizzy/mlx-vlm #1175 on 2026-05-12; Blaizzy landed a fix in PR #1213 on 2026-05-23.

The article's working claim is unchanged: pixel cap is the load-bearing variable. What's new is that Python mlx-vlm users now get correct behavior out of the box. Swift / mlx-swift-lm still needs the cap applied externally because its image path resamples before any HF-config check.

◆ Closing

Building this system produced more failure than success. Six major approaches failed before the working architecture emerged, and even the working approach required fixing 10 implementation bugs in a third-party library before it produced correct output (the 10th surfaced after publication). The total development time from "I want to read a panel from the screen" to "this reliably works" was measured in weeks, not days.

The experience of building this on-device overlay led directly to a testing methodology I call CCSV (Cross-Channel Spatiotemporal Verification) -- the idea that you can verify a UI by reading it through two completely independent channels (DOM and pixels) and comparing what they see. That methodology is described in a companion article.

If you are building something similar -- local VLMs on Apple Silicon, on-device document grounding, overlay rendering -- I would like to hear what you have tried and what worked. The failure modes are not well documented anywhere, and the community benefits from sharing them.

Niv Dvir is a software developer who builds tools at the intersection of computer vision and UI automation. You can find him on GitHub.

How I Built a Cochlear Spiral Spectrogram That Visualizes Music Like the Inner Ear

Niv Dvir — Sun, 22 Mar 2026 12:23:13 +0000

What if you could see music the way your inner ear hears it?

I built a visualization system that maps audio frequencies onto a Fermat spiral — the same geometric curve that describes how the human cochlea arranges its frequency-sensitive hair cells. The result reveals the hidden geometry of harmony: you can literally see the difference between a major and minor chord.

The Core Idea

Traditional spectrograms show frequency vs. time as a rectangular heatmap. They're useful but clinical — they don't capture the feeling of music.

The cochlea (your inner ear) isn't rectangular. It's a spiral. Low frequencies resonate at the outer end, high frequencies at the inner end — logarithmically spaced, just like musical octaves.

So I asked: what if we visualize frequencies on an actual spiral?

How It Works

1. Audio Analysis (scipy FFT)

381 logarithmically-spaced frequency bins (20 Hz — 8 kHz)
ISO 226 equal-loudness contours for perceptual accuracy
60 FPS frame-by-frame analysis

# Simplified core: FFT → cochlear frequency mapping
import numpy as np
from scipy.fft import rfft, rfftfreq

def analyze_frame(samples, sample_rate=44100, n_bins=381):
    spectrum = np.abs(rfft(samples))
    freqs = rfftfreq(len(samples), 1/sample_rate)

    # Logarithmic bins: 20 Hz to 8 kHz (cochlear range)
    bin_edges = np.logspace(np.log10(20), np.log10(8000), n_bins + 1)
    amplitudes = np.zeros(n_bins)

    for i in range(n_bins):
        mask = (freqs >= bin_edges[i]) & (freqs < bin_edges[i+1])
        if mask.any():
            amplitudes[i] = spectrum[mask].mean()

    return amplitudes

2. Spiral Mapping (Fermat Spiral)

Each frequency bin gets a position on a Fermat spiral: r = sqrt(θ)

Low frequencies sit at the outer edge (like the cochlea's apex), high frequencies spiral inward.

# Map frequency bins to spiral coordinates
theta = np.linspace(0, 8 * np.pi, n_bins)
r = np.sqrt(theta)
x = r * np.cos(theta)
y = r * np.sin(theta)

3. Chromesthesia Color Mapping

Colors follow a chromesthesia mapping — the neurological phenomenon where people "see" sounds as colors:

Low frequencies (bass) → warm reds/oranges
Mid frequencies (voice, guitar) → greens/yellows
High frequencies (cymbals, harmonics) → cool blues/cyans

4. Temporal Features (The Secret Sauce)

Static spectrograms miss the movement of music. I added 5 temporal features, each validated across 1,704 audio samples:

Feature	What it does	Optimal parameter
Melodic trails	Short glowing trails following melody	10 frames, 0.70 decay
Rhythm pulses	Radial pulse on beat hits	0.50 intensity, 0.25 decay
Harmonic auras	Sustained glow for held chords	4.0s blend time
Atmospheric context	Background mood from 60s window	0.35 influence
Harmonic connections	Lines between harmonically related notes	Octave + fifth detection

Why Harmony Looks Beautiful

This is the magical part. When notes are harmonically related (octaves, fifths, thirds), they land at symmetric positions on the spiral. A major chord creates a visually balanced, symmetric pattern. Dissonance creates asymmetric, chaotic (but still beautiful) patterns.

Different musical traditions create remarkably different visual signatures:

Classical harmony → orderly radial symmetry
Arabic maqam → quarter-tone asymmetry with unique geometric beauty
EDM/electronic → explosive, pulsing energy patterns

Try It: The Wellspring

I also built a crowdsourcing platform called The Wellspring where people can rate how well these visualizations capture the music. The goal: build an open dataset for AI-powered audio visualization evaluation.

Tech Stack

Audio analysis: scipy (FFT), librosa
Rendering: PIL (2D), PyVista (3D optional)
Video encoding: FFmpeg (H.264, CRF 18, 60 FPS)
Web platform: React 18 + TypeScript, Node/Express, PostgreSQL

What's Next

I'm working on browser-based creation tools so anyone can create their own audio-visual harmony — no installation needed. The vision: a global community of creators exploring the intersection of sound and moving image.

The ancient dance between rhythm and movement, renewed with modern tools.

Channel: youtube.com/@NivDvir-ND
The Wellspring: synesthesia-labeler.onrender.com

I'd love to hear your thoughts — especially from anyone working on audio visualization, creative coding, or signal processing!

DEV Community: Niv Dvir

What broke when my Claude Code projects started talking to each other

Two different stalls that look identical

A trust prompt that fires before your message arrives

The receiving model refusing on safety grounds

The silent stall that looks like working

An addressing bug, and how I isolated it

The one that isn't a bug: a read-only transport can carry a ruling, but not work

What the list is for

A Garbage-Collector Layer Between Claude Code Projects

The shape: a forest of projects

The six laws

1. Ownership

2. Records

3. Communication

4. Authority

5. Consequence

6. Convergence

One request, end to end

What it costs, and what it does not do yet

Test your rules like code

Where the ideas come from

What transfers

Reading, most useful first

max_pixels is a token budget in disguise — and the right cap depends on the size of what you're looking for

Decoding max_pixels: one line of math

So what's the right cap? Wrong question — right for what target?

This is an old law in new units

What every runtime actually defaults to

What to actually do

Provenance

Your parity gate must enforce the number you publish: a testing methodology for porting ML models across runtimes

The 30 px gate that hid a 9 px bug

Identical weights do not mean identical output

Four things have to exist together

1. A reference that pins three things

2. Compare every edge, not every box

3. Prove the patch lines landed

4. Cross-check adapters at zero tolerance

There is no fuzzy pass

Build the gate before the bug-hunt

Provenance

On-Device Document Grounding on macOS: Getting Qwen2.5-VL to Actually Work in Swift

◆ What I Wanted to Achieve

◆ What I Tried (and Why It Didn't Work)

Florence-2

Ferret-UI

Qwen2.5-VL-3B (the Small One)

Pixel-Edge Detection (No ML)

Accessibility API (AX API)

Spawning a New Python VLM Process Per Inference

◆ The Architecture That Worked

Panel Detection: Qwen2.5-VL-7B via MLX

From Python Server to Native Swift

The 9 Bugs in mlx-swift-lm's Qwen2.5-VL (plus a 10th found after publication)

◆ Combining Everything Into a Working Pipeline

OCR: Apple Vision Framework

Scroll Accumulator

Metal GPU Overlay Rendering

◆ The Final Result

A Taste of How It Looks

Performance Numbers

A note on "native Swift vs Python"

How to Reproduce This

Source Code

◆ Postscript

2026-04-25 — 10th bug + reframing

2026-05-25 — upstream max_pixels fix landed in mlx-vlm

◆ Closing

How I Built a Cochlear Spiral Spectrogram That Visualizes Music Like the Inner Ear

The Core Idea

How It Works

1. Audio Analysis (scipy FFT)

2. Spiral Mapping (Fermat Spiral)

3. Chromesthesia Color Mapping

4. Temporal Features (The Secret Sauce)

Why Harmony Looks Beautiful

Try It: The Wellspring

Tech Stack

What's Next

2026-05-25 — upstream `max_pixels` fix landed in mlx-vlm