DEV Community: praveenlavu

Agent Routing Caches: A Competence Ratchet from SOAR Chunking

praveenlavu — Wed, 08 Jul 2026 14:37:06 +0000

Agent Routing Caches: A Competence Ratchet from SOAR Chunking

I was watching my own routing agent send the same task to the same sub-agent for the forty-seventh time. "Summarize this PDF." Same shape, same answer, every single time. And on attempt forty-eight, it stopped, thought hard, burned the tokens, and arrived at the exact route it had already arrived at forty-seven times before.

It bothered me more than it should have. The agent was not learning anything. It had no memory of competence. It knew nothing about what had worked a hundred times before, only what the model currently predicted was most likely to work given the prompt in front of it. Every dispatch was the first dispatch. The planner tax was being paid forty-seven times for a route that never changed.

I knew the obvious fix, and I knew it was wrong, which is the worst kind of knowing. Embed the incoming task, find the nearest cached task by cosine distance, reuse the route if the similarity clears a threshold. It demos beautifully. It falls apart in production, and I had three reasons sitting in my head for exactly why.

Embedding similarity is not the same thing as task identity. "Summarize the Q3 earnings report" and "summarize this PDF about competitor pricing" sit close together in vector space. They are not the same task. The things that actually decide which agent handles the job are structural: document type, domain, the output format the caller needs. Those are discrete questions with crisp answers. A similarity score smears them into one continuous number that approximates meaning, not identity.

A similarity cache also has no concept of confidence. A route cached from one lucky run carries the same weight as one confirmed fifty times. And when a sub-agent's capabilities shift underneath it, a new tool, a swapped model, a deprecated endpoint, the cache has no way to express that it should trust the old entry less. The stale route just sits there at the same threshold it always had. Worse, the thing grows forever. Every unique-enough task adds another entry, and the longer it runs the more it accumulates embeddings from workflows that no longer exist. There is no signal telling it what to forget.

So I had a problem I understood and a solution I did not trust. The routing problem was structural, and I wanted a structural answer. I just could not see one.

Then I remembered SOAR.

I had read about it years ago, the cognitive architecture from a 1987 paper, the kind of thing you file away as intellectual furniture and never expect to use. SOAR runs a loop: take a state, pick an operator, apply it, update the state, repeat until the goal is reached. When it hits a situation where no operator applies, an impasse, it opens a subgoal, works the problem out in that smaller space, and resolves it. The expensive part is that same impasses recur across different tasks, and without help the architecture solves each one from scratch every time. Replaying the same moves. Deliberation without memory.

That last phrase was the moment it clicked. Deliberation without memory was exactly what I was staring at. My routing agent was a machine for re-resolving the same impasse forty-seven times.

SOAR closes that loop with a mechanism called chunking. When a subgoal resolves successfully, the architecture traces the conditions that led to the impasse and the operators that resolved it, and compiles them into a single rule: given this context, fire this directly. Next time the same context shows up, the rule fires immediately and the whole deliberation is skipped. Recall replaces reasoning.

Two properties make it precise, and they are the two properties I had been missing. Chunking only happens on successful resolution; failed attempts compile nothing, so the cache is built exclusively from evidence of competence. And it fires on context equivalence, not similarity. The match is structural, a pattern match, not a nearest-neighbor guess. The payoff is what the SOAR authors call a competence ratchet: performance over time only holds steady or improves. The architecture cannot get slower at a problem it has already solved.

That was the abstraction I needed, sitting in a paper older than most of the people building agents today.

The mapping turned out to be almost embarrassingly direct. A SOAR problem context becomes a task fingerprint: a deterministic hash of the structural attributes that decide the route, task type, input modality, required output format, domain flags. Not an embedding. A fingerprint. Two tasks with the same fingerprint should get the same route, and if they would not, the fingerprinting schema is wrong, not the threshold. The operator trace becomes a routing trajectory, the ordered list of agents and tools a successful run actually used, captured only after the outcome is confirmed good. A SOAR production rule becomes a cached route keyed by fingerprint. An impasse resolution is the planner call. A chunk firing is a direct dispatch with the planner bypassed entirely.

And the ratchet, in this setting, is one clean rule: after K confirmed successful dispatches for the same fingerprint, stop deliberating and dispatch the cached route. The planner does not run. The route is known.

I went with K equal to three. One success could be luck. Two is a signal. Three is enough confirmation to skip the deliberation while still being small enough to adapt fast when something changes. I want to be honest that this is not a deeply principled number. It is a reasonable prior that should be configurable for wherever it runs. I would rather flag that than dress it up.

The honesty extends to a second mechanism I borrowed from biology rather than cognitive science. A chunk that has not been used in a long time is a liability, not an asset, because its route may point at an agent that no longer exists. So chunks die. In cell biology, apoptosis is programmed cell death, the body clearing cells that have stopped being useful. A chunk untouched past its age window gets swept out the same way. Ninety days is my conservative default; for a fast-moving deployment with frequent model swaps, thirty is more honest. The invariant I care about: no chunk survives long enough to become a trap after the ground has shifted under it.

I also kept a list of places where I refuse to chunk at all, because a ratchet that bypasses thought is dangerous in the wrong context. High-stakes routing, where a misroute means data written to the wrong system or a destructive operation on the wrong resource, the planner is cheap insurance and I leave it in. Drift-detected contexts, where a distribution-shift signal says the world has changed, the chunks go on probation. Genuinely new task shapes, where there is no fingerprint match, the planner runs and that is correct, and I resist every temptation to add fuzzy matching, because structural precision is the entire point. And tasks where the route selection is itself the valuable reasoning, A/B comparisons, exploring route diversity, caching would eliminate the exploration I actually wanted. This is an optimization for settled decisions, not for routing research.

The moment I want you to feel is the fourth run. The first three dispatches for a fingerprint pay the full planner cost and accumulate their successes, and the third one is what pushes the count over the threshold. From the fourth run on, the lookup returns the cached route, the planner is skipped, and the latency just collapses. In a toy loop the per-run cost drops from around a hundred milliseconds to around twenty, the planner taken clean off the critical path. In a real deployment, where the planner is an actual model API call with a network round-trip, the delta is bigger, four hundred to twelve hundred milliseconds depending on the model and the infrastructure. The agent stops re-deciding what it already knows.

That is the whole story, and it is also the principle. The agents we build re-deliberate settled decisions because we never gave them a way to remember being right. A thirty-nine-year-old paper on general intelligence had already solved that, and the only new work was recognizing my problem in its shape. The competence ratchet is not a clever trick I came up with. It is an old idea I was lucky enough to remember at the right moment, on attempt forty-eight, when I finally got tired of paying the same tax twice.

If you are building routing agents, you have probably paid it too. You just might not have noticed yet.

Citation

Laird, J. E., Newell, A., & Rosenbloom, P. S. (1987). SOAR: An architecture for general intelligence. Artificial Intelligence, 33(1), 1-64. https://doi.org/10.1016/0004-3702(87)90050-6

Semantic Loop Detection: Catching Stuck AI Agents

praveenlavu — Tue, 07 Jul 2026 14:37:05 +0000

Semantic Loop Detection: Catching Stuck AI Agents

It is 2am. The agent has burned 40k tokens and reverted the same file four times, and from where I am sitting it looks like it is working hard. That is the part that fooled me. It was busy. Every loop produced a new patch, a new diff, a new paragraph of reasoning about why this time would be different. The log scrolled. Things were happening. The agent just was not getting anywhere.

The task was a bug fix. Generate a patch, run the tests, watch them fail. Read the error, generate another patch with different variable names and different line numbers and the exact same underlying logic. Tests fail again. Third attempt, it wraps the fix in a try/except. Still fails. Same root cause, untouched. By attempt seven it had written and reverted the same file four times and was no closer than it had been on attempt one. I was watching a machine spend my tokens to stand perfectly still.

The thing I could not get past was that my loop detector said everything was fine.

The detector that lied to me

I had a guard for exactly this. A set of seen actions, each action hashed to a string, and the rule was simple: if the same action shows up twice, you are looping, halt. It had caught dumb loops before. So I trusted it.

It reported no loop. Not once across seven attempts. And it was technically correct, which is the worst kind of correct. Each action string really was different. "Change the timeout from 30 to 60." "Set the timeout to 60 seconds where it was 30." "Apply the same value at the call site instead." Three different strings, three different SHA-256 hashes, three entries the set saw as three distinct actions. The agent was not repeating itself, not in the only sense my detector understood. It was absolutely, structurally stuck, and my guard waved it through every single time.

That is the moment that got under my skin. The hash changes when the text changes. That is the entire failure mode. My detector was measuring whether the words moved, when the only thing I cared about was whether the situation moved. Those are not the same question, and I had been treating them as if they were.

I sat with the three timeout edits for a while. Look at what actually happened underneath them. The first two are the same edit with the prose rephrased. The third shifts where the edit lands but leaves the real bug, a hardcoded default upstream, completely untouched. The next test run would throw the identical error. Three "different" actions, one unchanged reality. The string was new. The progress was zero. My guard could only see the first one.

The thing I had been measuring wrong

Once I named it, I could not unsee it. There are two different things going on every time an agent acts, and I had collapsed them into one. There is syntactic novelty, which is just "the string changed." And there is semantic progress, which is "the underlying situation changed." Loop detection has to live at the second level. Hashing the action string only ever sees the first.

So the question stopped being "how do I make a better hash of the action" and became "what is the smallest honest description of what this agent is actually doing." Not how it described the work. What the work was. That reframing is the whole turn. Everything after it is bookkeeping.

This is where I stopped reinventing and went looking, and found that KARIMO (github.com/opensesh/KARIMO) had already published the shape of the answer: a four-dimensional fingerprint. Instead of hashing the action string, you describe the action along four axes and hash that. Each axis catches a different way an agent goes nowhere.

The first axis is a coarse action class. Not the full action string, just the category: a file edit, a test run, a shell command, a search. This is the one that would have saved me at 2am. My three timeout edits, with all their rephrased prose, collapse into the same category the instant you stop caring about the words. The phrasing difference simply disappears.

The second axis is where the work is landing, a normalized hash of the files involved. An agent that keeps touching the same two files on every attempt is structurally looping no matter what story it tells about each pass.

The third axis is the state of the world, supplied by the caller. A hash of whatever counts as the agent's environment: test results, file checksums, the context it is working against. This is the brutal one. If the state hash has not moved between two actions, then by definition nothing the agent did had any effect. Two identical state hashes across two different action strings is not a hint. It is proof.

The fourth axis is the one I wish I had had first, and it is the key to the bug-fix stall specifically. You take the error output and strip the parts that wander without meaning anything: line numbers collapse to a placeholder, memory addresses collapse, absolute paths reduce to bare filenames, timestamps collapse. What survives is the error type and the message template. So AssertionError at line 47 and AssertionError at line 52, after a refactor nudged the lines, normalize to the same thing. The agent is still hitting the same wall even when the traceback shifts under it. My old detector treated that shift as progress. It was noise.

Combine the four into one tuple, hash it, and you have a fingerprint that holds still under rephrasing and only moves when something real moves. That is the whole insight, and it is almost embarrassing how much calmer the problem felt once I had it.

Watching it catch the thing that fooled me

The first time I ran the new detector against the same stuck pattern, it was quietly satisfying in a way debugging rarely is.

Picture the five attempts. The same config file, the same timeout assertion failing, the environment state hash never moving an inch. The agent narrates a different story on every pass: change the timeout from 30 to 60, set it to 60 seconds, bump the request timeout parameter, update it in config, fix it again because the last fix was incomplete. Five confident, distinct sentences. But each one feeds the detector the same action class, the same file, the same state, the same normalized error. Five times.

The old seen-actions set looks at those five strings, counts five unique entries, and stays silent. The fingerprint looks at the same five and sees one identical tuple, over and over. And because detection without a response is just logging, the fingerprint is paired with a count. KARIMO's pattern is a two-step ladder. At three matches, escalate: the current approach is dead, but a stronger model or a different prompt might break the wall, so hand it up. At five, stop and surface to a human, because the stall has now survived escalation and is just burning money.

So the streak climbs. The first two attempts pass as routine. The third trips the escalate threshold and the detector says try something stronger. The fourth stays escalated. The fifth hits the halt and kicks it to me. Five syntactically distinct actions, one semantic fingerprint, and for the first time the guard actually saw what I had seen at 2am.

The two-step design is not arbitrary either, and I came around to why. The costs are asymmetric. A false alarm at three costs one unnecessary escalation, slightly more expensive, but the agent keeps going. A false alarm at five interrupts a run that might have finished, while a missed stall lets an agent burn resources with no path forward at all. That asymmetry is what makes five a defensible place to halt for most tasks and three a cheap early warning you can afford to be wrong about.

Where it still gets it wrong

I want to be straight about the limits, because this is a heuristic, not a truth oracle, and pretending otherwise is how you ship a guard that fights your own agent.

Intentional retries will trip it. Exponential backoff on a flaky API, or a polling loop waiting on a job, will look exactly like a stall if the error and the state stay stable across attempts. The fix is to loosen the threshold for those paths, reset the detector on known retry patterns, or hand them a different action class so they cannot pile onto the same streak.

Planned iteration is the friendlier case. A write-test, run, fix, run cycle produces alternating fingerprints and will not trip the streak, which is correct. But if the fix step itself stalls while the test and run steps keep cycling around it, the alternation hides the problem, and you may need a second detector scoped to just the fix-class actions.

And the thresholds are task-specific. An agent doing exploratory code search might legitimately re-examine the same files three times from three different angles. Three is far too tight there; seven or eight is more honest. Wire the thresholds through config and tune them per task type rather than baking in numbers that were only ever right for one kind of work.

None of that undoes the core move. The fingerprint pattern I leaned on here, the combination of action class, files touched, state, and normalized error, is published by KARIMO at github.com/opensesh/KARIMO; check the current license at the repo before you reuse it. What I took from it was not code. It was the reframing.

That is the thing I keep coming back to. The bug was never in the agent. The agent was doing what stuck things do, churning. The bug was in me, in what I had chosen to measure. I had built a detector that asked whether the words were changing, when the only question that ever mattered was whether anything real was. If your agent can spend 40k tokens looking productive while standing still, your loop detector is measuring the wrong thing, and it will keep lying to you politely until you teach it to look underneath the words.

A Prior-Art Discipline for IP-Sensitive Builders: Reading Competitors'' Code

praveenlavu — Mon, 06 Jul 2026 14:37:05 +0000

A Prior-Art Discipline for IP-Sensitive Builders: Reading Competitors' Code Safely

Picture the worst version of a deposition. You are three years past the build, sitting in a conference room you do not own, and opposing counsel slides two printouts across the table. One is your git history. The other is your browser history. They are not accusing you of copying a single line of code. They have something quieter and worse. A highlighted row on each page, and a date that lines up between them: your commit adding "the new mechanism" landed twenty-two days after you opened that competitor's GitHub repo. They let the silence sit. Then they ask, very politely, to walk through what you were thinking on the day you say you conceived it.

That scene has never happened to me. Nothing here is a war story I lived. It is the scene I was imagining the night I decided I could no longer read other people's code the way I had been reading it. I build things that might be patentable, and I live in open source, and for a long time I treated those two facts as if they had nothing to say to each other. The imagined deposition is what made me stop and build a workflow instead.

Because here is the thing I had wrong. I thought the danger was copying. It is not, or not mainly. The danger is that I could read a competitor's implementation honestly, internalize a pattern, build my own version six weeks later from scratch, file on it, and then, three years out, have no contemporaneous record of what I understood and concluded on the days in between. The merits might be entirely on my side. It would not matter much. I would be litigating my own mental state with nothing in hand but memory, and memory loses to a timestamp every time.

The two ways this goes wrong

There are two failure modes, and they pull in opposite directions, which is what makes this hard.

The first is ignorance. You build something genuinely interesting, you file, and during prosecution the examiner surfaces a repo from 2022 that implements your core mechanism under an MIT license. Now you are spending money arguing around prior art you could have accounted for, or your claims get rejected outright. Filing fee, attorney time, the whole prosecution arc: wasted, or worse, wasted and a little embarrassing.

The second is the deposition I opened with. Contamination. You read the competitor's code while researching, you build later, and the dates line up badly enough that someone can tell a story where you adopted a pattern from prior art and filed on it as if it were yours.

For a while I thought the lesson was "read less." Stay clean by staying ignorant. That instinct is exactly backwards, and it took me a while to see why.

Why reading the code is the protective move

Novelty in patent law is defined against the prior-art landscape. If you do not know that landscape, you do not know where your novelty actually lives. You will write claims that are broader than the art allows, not because you copied anyone, but because you never looked. Those claims get rejected, you narrow them under pressure, and you walk away with weaker protection than if you had done the homework first.

The deeper point is the one that flipped me. Reading prior art with discipline gives you contemporaneous documentation of your own novelty analysis. A dated line that says, in effect, on this day I read this approach and here is exactly how mine differs. That entry, stamped before your filing, is evidence. It is the difference between being ambushed by prior art during prosecution and walking in already holding your distinction, written down, dated, never reconstructed under time pressure.

Avoiding the art does not protect you. It just moves the collision somewhere more expensive and strips you of the one document that would have helped. The builders who run clean IP read everything relevant. They just record what they read and what they concluded.

The license gate goes first, always

Before I read any open-source code as part of IP-sensitive research, I check the license. Two minutes. Skipping it can manufacture a problem you cannot talk your way out of later.

The rule is blunt: GPL, AGPL, and LGPL carry patent-related risk that MIT and Apache-2.0 do not. The GPL family ships explicit patent-retaliation clauses, where bringing a patent claim related to the licensed software can terminate your license to use it, and some readings extend that to work merely interoperating with GPL code. AGPL adds network-use provisions that can spread copyleft obligations in ways that interact badly with a patent strategy. MIT and Apache-2.0 are the safe-to-read tier, and Apache-2.0 even carries an express patent-license grant that cuts in your favor when you are reading it as documented prior art.

So the first field in any ledger entry is the license. If it comes up GPL or AGPL, I stop and get explicit legal sign-off before reading. That is not a call I make on my own. The interaction between copyleft and patent strategy is fact-specific and needs professional advice. What I can do mechanically, without a lawyer, is recognize the gate and not walk through it without checking. It applies to small repos, to "just the README," to archived projects. The check is always first.

What a ledger entry actually is

It is not a notes file. It is a structured, dated record with one job: contemporaneous evidence of what I read, what I pulled from it, and where I drew my novelty boundary at that point in time.

Each entry pins a sequential id so I can cross-reference it from an invention disclosure later. It records the date read, which is the evidence anchor, the thing that proves the reading predates the filing. It names the source precisely, full URL, commit hash, DOI, patent number, because vague references are weak evidence. It logs the license and an explicit note that the license was checked, an affirmative record that I ran the gate. Then three substantive fields in my own words: a short description of what the system does, the abstract patterns I extracted, and the field that carries the whole weight, my novelty-boundary analysis. Something like: the individual mechanisms here are prior art as of this date; my contribution is the composition and the priority-resolution logic that binds them. A status field tracks where the entry sits, documented when written, disclosed once it has been formally handed to a patent attorney for an Information Disclosure Statement.

Store the file in version control. The commit timestamps are part of the evidence. Never backdate an entry. The moment you backdate one, the whole ledger stops being a record and becomes a liability.

What I extract, and what I refuse to

This is the line that keeps the ledger clean and the position defensible.

I extract ideas. Architectural patterns, algorithmic approaches, interface contracts, problem framings, the conceptual mechanism underneath a system's behavior. The abstract things a skilled engineer would recognize from reading. I do not extract implementation. No copy-paste, no line-by-line translation into another language, no reproducing the specific sequence of operations at the code level.

That maps onto how the law splits ideas from expression. An algorithm is an idea. A specific implementation is an expression of it. Prior-art analysis lives at the idea level. The real question is never "did you copy this code." It is "did you arrive at this approach independently, or did reading it contaminate your conception." So if I read a fingerprint-based loop-detection approach and later build my own from first principles, working from what the mechanism achieves rather than the code in front of me, that is independent development of a concept that happened to already exist. The ledger documents exactly that: read it, understood it, recorded it as prior art, noted it in the novelty boundary before writing a line. What I refuse to do is read code and then write code that walks the same structural sequence. That is the thing that looks bad in hindsight no matter what my intent was.

A worked example (fictional, on purpose)

To keep this concrete without exposing anything real, here is a fictional invention: a routing system using preference-weighted ensemble voting for multi-model task assignment, with dynamic weight decay based on outcome recency. It is not a real product. The point is the paper trail it leaves.

Before writing code, I run prior-art searches. First, multi-armed bandit LLM routing, which surfaces the ADWIN drift-detection work, window-based statistical change detection used in some routing systems (Bifet and Gavaldà, SIAM SDM 2007). Academic publication, no code-license issue. Entry PA-001 documents the mechanism and draws the boundary: ADWIN handles distribution shift via threshold-crossing window detection and reselection; my weight decay is continuous and recency-weighted and needs no threshold event. Same problem, different mechanism. Second search, ensemble voting across models, surfaces llm-council, an MIT-licensed library that routes queries to several models and aggregates via peer-review voting. Entry PA-002: llm-council addresses post-response aggregation; mine addresses pre-dispatch routing assignment. Different layer. Third search, preference learning for model selection, surfaces two RLHF-based preference-modeling papers. Entries PA-003 and PA-004.

Four dated entries before I write a line of implementation. The boundary is explicit: the sub-mechanisms exist in prior art; the composition and the specific weighting approach are the claim. Three months later, when the patent attorney asks what the prior art is, I hand over four dated entries with analysis. Claims get drafted to the composition. Prosecution goes faster. The novelty argument is pre-built instead of improvised.

Why this strengthens prosecution

An examiner's job is to find art that anticipates or renders your claims obvious. If you have already found the relevant art, dated your analysis, and disclosed it, you have done a real piece of that work and shown your claims were written to account for it. It flips your posture from reactive, the examiner found X and now we argue around it, to proactive, we disclosed X and here is the distinction. Examiners respond differently to those two situations.

There is also a duty-of-disclosure dimension. Applicants are required to disclose material prior art they are aware of, and a maintained ledger helps you meet that systematically rather than from memory months later. Discuss the specifics of your IDS obligations with your patent attorney. That part is not optional. But the engineering workflow that produces the documentation is entirely within your control.

The asymmetry, and the honest caveat

The value is lopsided in your favor. If you never file, the ledger cost you a couple of hours per significant review session and left you a useful research artifact. If you do file, the dated record of your novelty analysis is the kind of thing that changes prosecution outcomes and makes any future litigation posture dramatically cleaner. Start the ledger before you think you will need it. The date on the first entry is the evidence.

And then there is the deposition I opened with, the one that never happened to me. The difference between that being a bad afternoon and being a catastrophe is whether, when counsel lays your git history next to your browser history, you have a dated entry that already says what you read, when, and exactly why your work is distinct. I would much rather hand them that than try to reconstruct what I was thinking twenty-two days before a commit, under oath, from memory.

This post describes an engineering workflow, not legal advice. Patent prosecution, IDS obligations, license risk analysis, and inequitable conduct exposure are fact-specific legal questions that require professional counsel. Nothing here should be read as a substitute for working with a registered patent attorney on any specific filing or IP strategy decision.

A Field Guide to Multi-Agent Orchestration in Late 2025: ruflo, KARIMO, llm-council

praveenlavu — Sun, 05 Jul 2026 14:37:06 +0000

A Field Guide to Multi-Agent Orchestration in Late 2025: ruflo, KARIMO, llm-council

I read three orchestration repos so you do not have to. It started because I was sick of the pattern. Every few months something announces that multi-agent orchestration is figured out, and inside its own demo, it is. Then you hand it a real workload and it dies in the seams I actually live in. So one week I stopped running these things and started reading them, several files deep at the kind of hour where you forget you have not eaten, trying to understand what each one bets its design on.

What I was hunting for is the stuff the demos never show. Whether a plan can quietly drift once it is running. Whether a loop detector can tell a real loop from an intentional retry. Whether the router learns from anything, or just reads a config file. Whether a panel of models can grade each other without playing favorites. None of that shows up in a fixed-horizon, single-model, no-concurrency benchmark. All of it shows up at 2am in something you shipped. So this is a field report, not a leaderboard. Three honest repos, each a different bet on the seams. Here is what each one actually does, what I would push on, and the one gap I noticed running through all three.

Before any of it: I read the repos, I did not run a controlled comparison. The mechanism claims below are what the projects document. The worries are mine, framed as worries, not measurements.

The three bets

ruflo (github.com/ruvnet/ruflo, MIT) bets on behavioral adaptation across a federation of agents. The conviction underneath it is one I share: which peer you trust should come from what the system has actually observed, not from a static config you tune once and forget. ruflo makes this concrete with a documented federation trust formula, a weighted blend of success rate, uptime, threat signal, and integrity, that continuously evaluates peers and downgrades untrusted ones with no human in the loop. It is model-agnostic, routing across several providers with failover. The bet is that trust is a measured property, not a declared one.

KARIMO (github.com/opensesh/KARIMO, Apache-2.0) bets on structural isolation and disciplined context. It is built on a commercial agent SDK as a coding-assistant plug-in, and it runs each agent in its own git worktree with branch identity verification, so agents cannot trample each other's working state. Execution is wave-ordered: tasks inside a wave run in parallel, waves run in sequence, across three loops it calls Foundation, Decomposition, and Orchestration. It layers context for token efficiency rather than dumping everything into every prompt, and it advertises semantic loop detection as a capability beyond the base SDK. The bet is that isolation and tiered context buy you reliability at scale.

llm-council (github.com/karpathy/llm-council, MIT) bets on adversarial peer review done honestly. A single model grading its own output is self-referential and you cannot trust it. llm-council runs three stages: every model answers the query, then each model reviews the others and ranks them on accuracy and insight, then a designated Chairman compiles a final answer. The detail that makes it work is in its own words, the model identities are anonymized so a model cannot play favorites when judging outputs. It is small, deliberately scoped, and by Karpathy. If you only read one of the three, read this one.

How they compare, in plain terms

I am not going to show you code, and the honest comparison lives at the level of design choices anyway, so line them up. Isolation runs from federation-level trust gating in ruflo, to hard per-agent git worktree isolation in KARIMO, to none in llm-council's single-pass review. Execution is a trust-routed federation, a wave-ordered three-loop pipeline, and a one-shot three-stage panel. Trust scoring is ruflo's whole thesis, expressed as a documented weighted formula; KARIMO does not center it; llm-council replaces it with anonymized rank-order review. On models, ruflo is explicitly multi-provider with failover, while KARIMO is built on a single vendor's SDK and routes by complexity within it. Loop detection is something KARIMO names as a feature and the other two do not foreground. Persistence runs from a vector memory store in ruflo, to git history as the durable artifact in KARIMO, to effectively none in llm-council's single exchange. That is the map. Now the parts I would push on.

ruflo, up close

Behavioral trust as a measured quantity is the strongest idea in the repo, and reading it gave me that small jolt of recognizing a good instinct. A trust score that blends success, uptime, threat, and integrity, and that downgrades a peer the moment the numbers say so, is the right shape for a system that has to keep working while individual agents go bad. The conviction that trust should be earned from observed behavior rather than declared in config is exactly the conviction I would build on.

The worries I would carry into production are about what a learning router does on a bad day, and these are my worries, not defects I measured. A score that moves with observed success can thrash when the pool is small and the feedback is noisy. And any system that routes toward what has worked has to answer the cold-start question: if a downgraded agent gets fewer tasks, it gets fewer chances to prove it recovered, and you can slide into rich-get-richer unless something deliberately explores. I did not see that explicitly addressed, so I would want to know how the trust loop avoids starving an agent that had one bad stretch. That is the general failure mode for any router that learns from outcomes: it will get quietly betrayed by its own feedback unless you design against it. The flip side is the part I would trust: when a peer genuinely goes bad, instant no-human downgrade is the behavior you want.

KARIMO, up close

The git-worktree-per-agent isolation is the cleanest reliability idea across all three codebases, and I knew it the moment I understood it. Most frameworks let agents share working state and rely on prompt instructions to keep them in their lane, which fails the instant two agents reach for the same file. Giving each agent its own worktree with branch identity verification removes a whole class of shared-state race structurally, not by asking nicely. That did something to me as a principle. Structure holds where instructions do not.

The context layering is the part I would read more carefully before trusting at scale. KARIMO tiers context for token efficiency, a level-of-detail approach that loads compact summaries first and full content only when needed, which is a genuinely good token-conservation strategy and not a small one. What it is not, and the repo is honest about this, is a security boundary; it is a scanning discipline, not a wall an agent physically cannot climb. So my worry is the ordinary one for any retrieval-by-relevance scheme: when the right context is filed under words that do not match the query, the efficient path can skip it, and you find out at runtime. On loop detection, the repo names semantic loop detection as a capability but does not, in what I read, document the internals, so I will not describe a mechanism it does not state. The honest open question I would ask the maintainers is whether it can tell an intentional retry, an agent re-running a step after new information arrives, from a true loop. That distinction is hard, and I could not verify how they handle it.

llm-council, up close

Identity anonymization is the sharpest single idea I read all week. The repo says it plainly: the model identities are anonymized so a model cannot play favorites when judging outputs. That one move targets the bias that makes self-grading worthless, a model preferring work from its own family, and it targets it structurally, because a reviewer that does not know whose answer it is reading cannot favor a name. Ranking on accuracy and insight rather than handing out absolute scores is the right instinct alongside it, since models are calibrated differently and a rank order sidesteps the worst of that. The README does not state the exact aggregation math, so I will not name an algorithm it never claims; the principle stands on the anonymization and the ranking, and that is enough.

The gaps are scope gaps, not bugs, and the repo does not pretend otherwise. It is a peer-review primitive, not an orchestration framework. The flow is single-pass: every model answers, the panel ranks, the Chairman compiles, and that is the run. There is no reconciliation round where reviewers see each other's verdicts and revise, and no state carried across evaluations, which are exactly the things you would add if you wanted this to be a standing judge rather than a one-shot panel. None of that is a knock. It is a small tool that does one thing well, and the one thing is the right thing.

The thing none of them claims to fix

Each project is genuinely good at one bet. ruflo at measured trust across a federation, KARIMO at structural isolation and token-disciplined context, llm-council at unbiased multi-model review. And reading all three back to back, the same hole stayed open in every one, which is the part that has stuck with me since.

None of the three, in what I read, treats the meaning of a failure as a first-class thing that gets routed on. A failure can be the model getting it wrong, or the context being stale, or the plan asking for something impossible, or a downstream tool throwing a transient error. Those want completely different responses, re-route, re-plan, escalate, retry, and the natural default in a system like this is to collapse them into one undifferentiated failure signal and react to all of it the same way. The piece I keep wanting is a structured failure taxonomy with routing by cause, a layer that reads the failure mode out of an agent's output and dispatches to a handler built for that class, which means every agent has to conform to a typed failure schema. I did not see one defined in any of the three.

I want to be careful here, because this is an observation about a gap, not a claim that I originated the fix. I did not, and typing your failures and routing on them is not new, it is just not something these three foreground. But it is the thing I cannot stop thinking about, because I have lived the adjacent version of it. The failure that does not announce itself as a failure is the one that costs you. In my own pipeline a fresh planner once read a stale plan and confidently re-derived work that had already been done, because nothing told it the difference between an open step and one already closed, and the cleanest defense I found was to stop trusting the carried-over note and treat the durable record as the only source of truth. That is the same shape: a signal that looks fine until you ask what it actually means. Until failure semantics are structured and routable, orchestration frameworks will keep recovering gracefully from the easy failures and badly from the hard ones. That is the part the demos will never show you, and the only part that ever kept me up.

Citations

ruflo source code: https://github.com/ruvnet/ruflo (MIT)
KARIMO source code: https://github.com/opensesh/KARIMO (Apache-2.0)
llm-council source code: https://github.com/karpathy/llm-council (MIT)
Crandall, J. W., & Goodrich, M. A. (2005). Learning to compete, compromise, and cooperate in repeated general-sum games. ICML 2005. (Background on multi-agent trust and behavioral adaptation.)
Stiennon, N. et al. (2020). Learning to summarize from human feedback. NeurIPS 2020. (Documents self-preference bias in model evaluation, motivating anonymization approaches.)

Tiered Context Loading: Fit a Huge Agent Registry in Your Context Window

praveenlavu — Sat, 04 Jul 2026 14:44:57 +0000

Tiered Context Loading: Fit a Huge Agent Registry in Your Context Window

Pattern source: KARIMO (Apache-2.0)

It is late, the kind of late where you are doing back-of-the-envelope math on a system you were sure was fine, and the envelope keeps telling you it is not. The dispatcher had been happily picking the right agent for each incoming task. The fleet kept growing, more capabilities every week, and that was supposed to be a good thing. Then I actually multiplied two numbers I had never bothered to multiply, and the floor quietly dropped out. The router I had shipped could not, on paper, do the thing I was watching it do. It was only still standing because the registry had stayed small enough to hide the truth.

I had built it the way you build everything the first time, which is to say without imagining it ever getting big. Every time a task came in, I loaded every capability's full spec into context, let the model read all of it, and asked it to choose. Description, parameters, examples, edge cases, error modes, the whole document for every tool the system owned. It worked beautifully when I had a dozen capabilities. It worked fine at fifty. The trouble is that nothing in the code complains as you climb. There is no alarm at the moment it stops being affordable. It just keeps working right up until the arithmetic says it cannot, and you only find that wall if you go looking for it.

So I did the math I should have done before I ever shipped it, and the number was genuinely funny in the way that only an own goal can be. Take a realistic multi-agent registry. I will use round engineering estimates here, not a census of the real fleet, because the shape of the problem is what matters and the shape holds at any scale: call it somewhere between two hundred and four hundred capabilities, each full spec running maybe four to eight kilobytes once you count all the parameters and examples and edge cases. Put four hundred capabilities at a ballpark six kilobytes each and you are holding around 2.4 megabytes of plain text. GPT-4o gives you a 128K-token window. At roughly four bytes a token, that is about 512 kilobytes of usable room. You can fully load on the order of eighty-five capabilities into that. The estimate said four hundred.

Loading all of them was not a little over budget. Four hundred specs at eight thousand tokens each is 3.2 million tokens against a 128K window, before a single word of conversation history. Not twenty-five percent over the window. Roughly twenty-five times over it. I had not built a routing system with a performance problem. On the numbers, I had built one that could not run, and had been quietly getting away with it only because the live registry stayed small enough to keep the bill from coming due. The exact figures are illustrative; the order of magnitude is the point, and the order of magnitude does not forgive you.

The three bad doors

My first move was the obvious one, and the obvious one was a trap. Just load less. Load none of the specs upfront and fetch them on demand. But fetch them how? The honest options were all doors that opened onto the same drop. Load all the specs: impossible, that was the wall the arithmetic had just drawn. Load none and route blind: useless, the model cannot pick a tool it knows nothing about. Load by keyword match: brittle in exactly the ways that bite you in production, where synonyms miss, categories overlap, and the router confidently sends a task to the wrong arm because two descriptions happened to share a word. These were not three alternatives to a solution. They were three different ways to fail, and I spent a frustrating evening discovering that each one personally.

What dragged me out of it was a reframe I should have started with. I had been treating "what tools exist" and "how do I use this tool" as the same question, loaded at the same time, paid for on every single call. They are not the same question. To pick a capability, the router barely needs to know it is there and roughly what it does. It does not need the parameter types, the examples, the retry semantics, any of it. All of that detail only matters at the instant of dispatch, for the one tool actually being called. I had been paying the full price of knowing how to use four hundred tools in order to make one decision about which one to reach for.

The turn

I did not invent the way out of this. I found it. KARIMO, an Apache-2.0 project, had already codified exactly the reframe I was groping toward, and given it a clean shape: three tiers, L0, L1, L2, each holding a different depth of knowledge, each loaded at a different moment.

L0 is the always-present index. For every capability you keep almost nothing: the name, a one-line description, the primary task type. No parameters, no examples. At roughly a hundred tokens per capability, four hundred of them cost on the order of 40K tokens, about a third of the window. That is the rent you pay unconditionally, on every call, and in exchange the router always knows the full menu of what exists. It can scan all four hundred names in one pass and shortlist the candidates without ever fetching a retrieval system or risking a recall miss. It just does not yet know how to use any of them.

L1 is the category layer, and it loads only when the router gets stuck. When the shortlist comes back ambiguous, two or three candidates in the same category whose one-liners are too close to separate, the router pulls the full parameter summaries for that one category and nothing else. One category, on demand, a couple thousand tokens. When one candidate clearly wins, this step never happens at all.

L2 is the full spec, the complete document I used to load for everything: every parameter with its types and constraints, the examples, the edge cases, the error modes. It loads for exactly one capability, the one being dispatched, at the moment of dispatch, and never a beat earlier. The invariant underneath all of it is almost insultingly simple. At any point in a routing decision, the total context you are carrying is L0 for everything, plus L1 for at most one category, plus L2 for at most one capability. That is the whole ceiling.

What the math becomes

Run the same arithmetic that had been mocking me, now against the tiered shape. L0 across all four hundred capabilities is around 40K tokens. One category of L1 is about 2K. One capability of L2 is about 8K. The worst case the system can ever be in is roughly 50K tokens, which leaves about 78K of a 128K window free for conversation history, retrieved documents, and the actual output.

Fifty thousand tokens, holding steady no matter how the registry grows, versus the millions the naive version demanded. Same capabilities, same window, same model. The difference is not a clever compression trick. The router never needed all that detail at once; I had just never given it permission to look things up only when it had to. The wall the numbers drew was not a limit of the model. It was a consequence of asking the wrong question on every call.

The cleanup that keeps it honest

There is one more piece that makes this hold up over a long session rather than just on paper, and it is the part I find quietly elegant. When context does eventually press against the ceiling, the eviction order is a hard rule, not a guess. The L2 spec for an in-flight dispatch is untouchable, because truncating it mid-call means the router generates against incomplete parameters and emits a malformed request. A category's L1 is atomic, all of it or none, because half a category overview is worse than no overview. So truncation always falls on L0 first, and never at random: the least-routed capabilities drop out before the most-routed ones.

The effect is that, over a long-running session, the always-present index slowly sheds the tools nobody is calling and keeps the ones the work actually leans on. The system concentrates its remaining headroom on its real working set. That is not decay. It is the design doing its job.

What I would tell you

If you are routing across a registry that can grow, build the tiered split before it grows, not after the wall. The whole lesson is one I had to learn the embarrassing way: knowing that a tool exists and knowing how to use it are different questions, and a router that pays for the second one on every decision is buying something it does not need yet. Separate them, load each at the moment it actually matters, and a registry that could never fit suddenly costs a flat, predictable slice of your window forever.

None of the core idea is mine. The L0/L1/L2 discipline is KARIMO's, published openly. What I brought was the bad evening, the arithmetic I should have run first, and the relief of recognizing my own problem in someone else's clean answer. It needs no embeddings, no retrieval infrastructure, no dependency beyond a registry you author with three depths of detail instead of one. The only cost is writing those three layers when you register a capability, and that cost pays for itself on the first routing decision the old way could not have survived.

Repository

Pattern: github.com/opensesh/KARIMO, Apache-2.0

The GPL v3 patent trap nobody checks until a lawyer walks your requirements.txt

praveenlavu — Fri, 03 Jul 2026 14:37:05 +0000

The GPL v3 patent trap nobody checks until a lawyer walks your requirements.txt

Disclaimer: I am not a lawyer, and nothing here is legal advice. What follows is my reading of what these license texts actually say in plain language, and the patent implications that seem to follow from that text. For any real IP decision, take it to a qualified patent attorney. On a public page a wrong legal claim is worse than a cautious one, so I am going to stay close to the words on the page.

Picture the moment a patent attorney goes through your requirements.txt line by line, asking which licenses are in your dependency tree. Not "can we ship this." A different question, one almost nobody asks before it matters: does using this code constrain my ability to assert a patent later?

I was not in a courtroom when this clicked for me. I was building, the way I always am, late, with a dependency list I had grown the lazy way, one pip install at a time, each one chosen by an engineer who was thinking about getting the thing to run, not about what happens in 2027 when a competitor ships a similar feature. And then the question landed and I realized I was about to do archaeology on my own decisions.

Here is the trap in one line. Most engineers treat open-source licenses as a shipping question. Can I include this in a commercial product, can I keep my changes private. The GPL-versus-proprietary boundary gets all the attention. The patent question is different, and it is the one that quietly waits.

Plaintiff one day, infringer the next

The stakes are concrete. GPL v3 code in your dependency tree means that asserting a patent against someone using the same GPL v3-covered functionality could, by the plain text of the license, terminate your own license to that code. You go from plaintiff to infringer in a single filing.

The mechanism is a patent retaliation clause, text that is deliberately written to make patent assertion legally expensive for anyone who incorporated the protected code. It is not an accident or a side effect. The license was drafted to do exactly this. And it sounds abstract right up until someone with a law degree is reading your lockfile out loud.

So let me walk the actual sections, because the whole thing lives in the text, not in the vibe.

The core mechanism: sections 10, 11, and 8 in combination

GPL v3's retaliation works through three sections acting together. None of them does the job alone.

Section 10 imposes a condition on every licensee. In its own words, you may not initiate litigation, including a cross-claim or counterclaim in a lawsuit, alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it.

Section 11 is the grant. Each contributor hands every downstream user a license to that contributor's essential patent claims. That is the thing of value you are holding.

Section 8 is the trigger. Violate the license and your rights terminate automatically, and that termination reaches the patent grants made under Section 11.

Now stack them. The sequence the text creates runs like this. You incorporate GPL v3 code into your system, a library, a component, even a small utility you copied and adapted. A competitor ships something you believe infringes a patent you hold. You file suit, alleging infringement of the program or a portion of it. That filing is the Section 10 violation. Section 8 then terminates your rights, including the Section 11 patent grants, as of the date you filed. And now you are distributing GPL v3 code without a license, which is its own infringement.

The uncomfortable step is the one in the middle. "Any patent claim infringed by making, using, selling the Program or any portion of it" is not a precisely bounded phrase. Patent claims are written broadly on purpose. The GPL code's contribution to your system might be peripheral to your core invention, and that does not automatically insulate you. What matters, as I read it, is whether the patent you are asserting can be characterized as one infringed by making or using the GPL v3 work. If opposing counsel can draw that line, the text says the sequence can fire. Whether a court agrees in your specific case is exactly the kind of question I am not qualified to answer, which is the point of the disclaimer at the top.

AGPL widens it, GPL v2 does not have it

Two siblings are worth naming because people assume they all behave the same, and they do not.

AGPL v3 carries the same Section 10 and Section 11 language, plus the extra reach of Section 13. Providing the software as a network service, without ever shipping a binary, triggers the same source-disclosure obligations as distribution. For patent strategy that means the full retaliation exposure travels with the network-use surface. If you are building an AI inference service on an AGPL-licensed library, you are inside the AGPL's reach whether or not anyone ever downloads a binary from you. That one catches people who think "I never distribute, so I am fine."

GPL v2 is the other direction. It has no equivalent patent retaliation mechanism. Its Section 7 prohibits adding further restrictions to GPL'd code, which can complicate certain downstream patent assertion strategies, but it does not contain the automatic license-termination trigger that v3 introduced. v2 and v3 are not equivalent on patent risk, and treating them as one thing is its own quiet error.

What "using" GPL code does to your position

The danger is not confined to wholesale copying. I think of it as a gradient, and the middle of the gradient is where the real trouble lives.

At one end, you use a GPL v3 utility that parses files, and your patent covers an optimization on the parsed data. The GPL code touches none of the claimed functionality. As I read the clause, it is looking for an allegation that the GPL work itself constitutes infringement. If you are not making that allegation, the text gives it nothing to grab.

At the other end, you copied GPL v3 code in, modified it, and ship it as part of a system your patent claims cover. You have incorporated the work, and your claims arguably allege that the combined system, which is the work, does the patented thing. That one is almost certainly a problem.

The middle is the one that should scare you, and it is the reason I am writing this. You use a GPL v3 library that does something adjacent, say an approximation method, and your patent claims a system that uses approximation methods to get a result. You assert against a competitor. Their counsel argues your claimed functionality is intertwined with the GPL library's behavior. Whether that argument wins is a legal question, but you are then paying attorneys to litigate it either way. "Adjacent" is decided by how your claims are written and how good the other side's lawyer is. The risk is fuzzy and front-loaded. You cannot fully know at filing time whether a court will find the connection, and by then the GPL code has been in your tree for eighteen months.

LGPL v3 belongs in this conversation too. It is "lesser" only in the copyleft sense, you can link against it without your application becoming GPL'd, but it incorporates the GPL v3 patent framework by reference. It is not lesser on retaliation. Same Section 10, 11, and 8 story.

The safe operating rule I landed on

After sitting with the text, the rule I actually run is simple, and deliberately conservative. No GPL v3, AGPL v3, or LGPL v3 code in any codebase where I intend to file or assert patents covering the same functional area. And "same functional area" gets read broadly, because patent claims are written broadly.

The good news, and it is genuine good news, is that this almost never costs you anything in modern AI work. The permissive ecosystem covers nearly everything you would actually reach for. The scientific Python stack is BSD-licensed. The major deep-learning frameworks are BSD or Apache-2.0. The big transformer libraries are Apache-2.0. The GPL libraries you bump into are usually legacy data tools or specialized components that predate the Apache-and-MIT-everything norm, and they have permissive alternatives in almost every case. The work of finding the alternative is small next to the work of defending the decision not to.

If you discover a GPL v3 dependency late, it is not a death sentence, just deliberate work. Re-implement the functionality clean-room, from the interface contract and not the source, and document that you did. Or wrap it as a separately deployed service you call over a network instead of linking it in, with the loud caveat that AGPL v3's Section 13 specifically addresses that pattern, so confirm with counsel before leaning on it. Or, if the component is peripheral, just drop it and carry the small technical debt instead of the legal debt. Every one of those paths is cleaner than explaining the dependency to an attorney three years from now.

The principle

None of this is original to me. The retaliation mechanism is right there in the license text, drafted by people who understood patents far better than I do, and IP counsel have been reasoning about it for years. The engineering distinctive, the part that is actually mine, is small and boring and saves you anyway: treat the license as a property of a dependency you check before adoption, not a footnote you reconstruct under adversarial scrutiny after you file.

The check costs about thirty seconds per dependency. Reconstructing a clean development history while opposing counsel reads your commit log costs a great deal more. The license is sitting in the repository the whole time. Read it before the import statement is in your tree, not before the filing is on the docket.

I am still not a lawyer. If any of this maps onto a real decision you are about to make, that is precisely the moment to put it in front of one.

A green test suite proves less than you think

praveenlavu — Thu, 18 Jun 2026 22:20:26 +0000

A green test suite proves less than you think

The test that scared me was the one that passed.

I had an integration test for a routing agent, the kind that takes a task and picks a capability to handle it. The test registered a new capability at runtime and then checked that the router would eventually route to it. Green run after run. Solid. I trusted it.

Then I read it properly. It reused the same task string on every iteration of the loop. My scorer was deterministic by design, it hashed the task and indexed into the capability list, so a fixed string mapped to a fixed slot, and the newly registered capability lived at a different slot that the fixed string could never reach. The test asserted that the new capability got selected. The new capability was structurally unreachable. And the assertion passed anyway, because the loop happened to land on something registered every time, which was all the weak version of the check actually demanded.

The test was not testing what it said it was testing. It was green for a reason that had nothing to do with the thing I cared about. The fix was almost insulting in its smallness, vary the task strings so the hash spreads across every slot including the new one, and suddenly the test could fail when the feature was broken, which is the entire point of a test. One line. I had been shipping false confidence behind a checkmark.

That is the moment this whole piece is about. Not the bug. The checkmark.

The number that lies

Here is the setup that produces this every time. You build an agent. You write unit tests. You watch line coverage climb to ninety-something percent. CI turns green. You deploy. And within a week the thing is making nonsensical decisions under load, falling over on inputs you never imagined a user would send, and getting stuck in loops your loop detector cannot see because two threads stepped on each other's state at the same instant.

The unit tests were not lying to you. The functions genuinely worked in isolation. That is the trap. Line coverage measures whether your tests executed a line, not whether they cornered it. You can run every line in a file and assert nothing that matters about any of them, exactly like my integration test ran its loop and asserted the wrong thing. A green suite built on coverage tells you your tests touched the code. It tells you almost nothing about whether the code survives contact with production.

And autonomous systems, agents that route, retry, fall back, remember, do not fail in isolated functions. They fail in the seams between functions. They fail where two modules meet and disagree about a type. They fail on the input the author never pictured. They fail when two requests arrive at once. They fail when a dependency dies and the system panics instead of limping. They fail on the edge case nobody wrote down. Coverage walks straight past all five, because every one of those failures lives in territory a unit test is structurally built to avoid.

Five seams, five suites

The shift that changed how I test agents was to stop asking "did my tests run the code" and start asking "what are the distinct ways this system actually breaks, and do I have a suite aimed at each one." Five answers came back, and they are genuinely distinct failure classes, not five flavors of the same check. None of these dimensions is mine to claim as an invention, they are long-standing testing practice, integration testing, adversarial and fuzz testing, concurrency testing, fault injection, and property-based testing each have decades of prior art behind them. The engineering distinctive is narrower and more honest, it is recognizing that an autonomous agent needs all five aimed at it at once, because it can fail in all five ways in a single week, and that a coverage number cannot stand in for any of them.

The first seam is integration, where modules compose. The most common bug in a multi-module system is not "function X has wrong logic," it is "X works fine but Y expected a different type," or "A only works if B was set up first." Mocks paper over exactly this, they return what you told them to and never enforce the real interface, which is how my same-string test slept through a real defect.

The second is adversarial input, the gap between the task you imagined and the task a real user sends, the hundred-thousand-character string, the embedded newline carrying a fake directive, the injection attempt, the empty string, the wall of emoji. The contract is not that nothing weird arrives. It is that weird input gets a safe answer or an honest error, never a crash and never a leak.

The third is concurrency, the races that only appear when many requests hit shared state at once. A history list, a registry, a loop detector, anything two threads can write without a lock, will silently corrupt under load in a way no single-threaded test will ever reproduce.

The fourth is failure cascade, what happens when the pieces an agent depends on, the registry, the scorer, the loop detector, start dying. A naive build lets any one failure crash the whole call. A real one degrades, and the failure you actually have to test is all of them dying at once, because real outages are correlated and take down several things together.

The fifth is property-based testing, where instead of writing examples you state an invariant and let a generator hunt thousands of inputs for the one that breaks it. The invariants that look obvious, "routing always returns a real capability or a clean error, nothing in between," are exactly the ones a generated single-character task or a Unicode combining sequence quietly violates.

What the checkmark should mean

No single one of these dimensions catches everything, and that is the whole argument. Integration finds the type-contract and setup-order bugs and is blind to races. Adversarial finds the injection and the boundary crash and never sees a component failure. Concurrency finds the race and ignores the malformed input. Failure-cascade finds the un-graceful crash and says nothing about invariants. Property-based finds the violated invariant and the boundary input and cannot see a thread race. The value is not in any one suite. It is in the combination, five independent nets under the trapeze, each catching what the others structurally drop.

Each suite is its own short read, linked below, with the one specific failure it exists to catch and the cheapest test that catches it.

I am not going to pretend this makes a system bulletproof. It does not. There are failures none of these five see, and there will be a sixth seam I learn about the hard way at 2am some night that is already coming. But the difference between the green checkmark before and the green checkmark after is the difference between a number and a sentence. Before, green meant "the tests ran the code." After, green means "this composes correctly, handles hostile input without leaking, holds together under concurrent load, degrades instead of crashing when its dependencies die, and keeps its invariants across inputs I never thought to write down."

That is a checkmark worth trusting. The first one was just a string that happened to match.

This is the hub for a five-part series, one short read per dimension:

LLM Self-Preference Bias: How Anonymized Peer Review Fixes It

praveenlavu — Thu, 18 Jun 2026 22:20:23 +0000

LLM Self-Preference Bias: How Anonymized Peer Review Fixes It

The panel had been agreeing with itself for a week before I noticed, and the worst part is that the logs looked healthy the whole time.

I had built what felt like a clean idea. Several frontier models, different families, each one judging a pool of candidate outputs and ranking them best to worst. A jury of machines. I would generate a handful of answers, let the panel vote, take the winner, and trust that five independent opinions beat one. That was the whole pitch I had sold myself at 1am, and for a few days it ran without complaint. The rankings came in. A winner emerged every round. The dashboard was green.

Then I started actually reading what won.

The outputs the panel kept crowning were not the sharpest. They were the ones that sounded a particular way. Numbered lists where the content did not need numbering. A certain rhythm to the sentences. A house style. I stared at it for a while before the shape of it landed, and when it did it was a little sickening: my panel was not selecting for quality. It was selecting for resemblance. The judges were rewarding the candidates that wrote the way the judges write. I had built a popularity contest and dressed it up as an evaluation.

The thing nobody tells you you assumed

The premise underneath every multi-model panel is that the judges are neutral. You assume a model reading an unlabeled answer scores it on merit. It does not. Panickssery and colleagues measured this directly in 2024, in a NeurIPS paper with the unambiguous title "LLM Evaluators Recognize and Favor Their Own Generations." They found GPT-4 preferred its own output at a pairwise win rate above 0.90 on summarization tasks. Over ninety percent of head-to-head comparisons, the model picked the answer it had written. Not because it was better. Because it was its.

The effect is directional across families. Prose in one model family's house style reads better to a judge from that same family. A more hedged, more structured answer reads better to a judge that writes that way. So when I assembled a panel and let it vote on a pool that included its own members' outputs, what I actually measured was which style happened to be most common among my evaluators. The highest-scoring answer was the one whose fingerprint matched the room. I had spent the planning at 1am congratulating myself on independence, and built the opposite.

And it is not only the obvious bias. Once I went looking, there were three of them stacked on top of each other. Self-preference was the loud one. Underneath it sat verbosity bias, where models score longer answers higher because length reads as effort and authority, even when the extra words say nothing. So my selection criterion was quietly drifting toward "writes the most" rather than "answers best." And under that sat position bias, where the first answer in an ordered list anchors the judgment, the same anchoring documented in human juries, so whichever candidate happened to appear first carried a structural head start that had nothing to do with being right.

Three biases, one panel, all of them invisible in a green dashboard.

The wrong fix I reached for first

My first instinct was to out-engineer it. Add a rubric. Tell every judge, in the prompt, to ignore style and length and score only on correctness. Lecture the jury about fairness before it deliberates.

It did almost nothing, and in hindsight it could not have. You cannot instruct a model out of a preference it does not know it has. The recognition is happening below the level the prompt can reach. The judge is not consciously thinking "this is mine, I shall reward it." It is reading prose that matches its own training distribution and finding it more fluent, more correct-feeling, more right. Asking it to be fair is asking it to notice a bias it cannot see. I was trying to argue a model out of its own reflection.

The real problem was not that the judges were biased. It was that the judges could tell whose work they were reading. The bias needed information to operate, and I was handing that information over for free.

The turn

The fix was not mine, and I want to be clear about that, because the elegant part was already sitting in public when I got there. Andrej Karpathy had published a small project called llm-council that solves exactly this, and the mechanism is almost insultingly simple: do not let the judges know whose output they are reading.

That is the entire idea. Before the panel votes, you strip every identity off the candidates. The first answer becomes "response A," the second "response B," and so on. No model name. No provider. No tell. The server keeps a private mapping of which label belongs to which model, a clean one-to-one assignment in both directions, so that after the votes are in you can reverse it and reconstruct exactly who scored what. The judges see only neutral labels and the text. The information the bias needs to operate is simply absent during the vote.

It works because you cannot favor what you cannot identify. Self-preference dies the moment the judge does not know which answer is its own. Hiding the names also strips the most obvious recognition signal, which dents style bias too, though not all the way, because if a model writes in an unmistakable rhythm its identity is still legible in the prose itself. Anonymization breaks the label, not the fingerprint. But the label was doing most of the damage, and removing it changed the room.

The first time I rewired my panel to run blind and watched the rankings come back, the winners were different. The house-style answers stopped sweeping. The thing that had been quietly rigging my evaluation for a week was just gone, because I had taken away the one piece of information it ran on. That is a strange and specific kind of satisfaction, watching a bias evaporate not because you argued with it but because you starved it.

Counting the votes honestly

Hiding the names fixes who wins a comparison. There is a second question underneath it: how you turn a panel of rankings into a single decision. Karpathy's project keeps that part as plain as the anonymization. Each judge ranks the anonymized pool, and the project aggregates those rankings by average rank position. You take every judge's placement for a given candidate, average them, and the answer with the best average ranking across the panel wins. That is it. No weighting, no points table, just the mean of where each judge put each answer.

What I like about averaging the rank is what it captures and what it ignores. It does not care how many head-to-head matchups an answer technically won, which is the trap of naive majority vote. Majority vote can crown an answer that one judge adored and the rest found mediocre, because a thin win still counts as a win. Average rank position cannot do that. A candidate that four of five judges place second and one judge places first lands at a strong average, and the panel correctly reads it as broadly acceptable rather than narrowly adored. Broad acceptability is exactly the signal you want when you are picking the single best output from a pool, and the mean of the rankings is what surfaces it.

If I were extending the project I would probably reach for something like a Borda-style scoring on top, turning each placement into points and summing them so a near-miss second place carries explicit weight rather than just nudging an average. That is my own refinement, not what the repo ships. What llm-council actually does is the simpler and honestly sufficient thing: anonymize, rank, average the positions, take the winner. The discipline is in the order of operations, not in any clever counting.

Where this is not enough

I want to be honest about what anonymization does not fix, because I shipped it feeling like I had solved the panel, and I had solved one third of it.

Self-preference is gone. Two biases are still in the room.

Verbosity bias is completely untouched. A longer answer is longer regardless of what label it wears. Anonymization operates on identity, not length, so if your task rewards thoroughness the panel will keep favoring the candidate that simply wrote more. The only real mitigations are a rubric that explicitly penalizes length without substance, or normalizing every candidate to the same length before the vote. Neither comes for free.

Position bias is only half-addressed. Randomizing which model draws which label between rounds helps, so no single model always sits in the anchor slot. But within any one judge's view, response A still appears before response B, and on the marginal calls, which is most of the interesting ones, first-listed still wins a little more often. The honest fix is an independent random ordering per judge, not just per round.

And there is a quieter trap I walked into while feeling clever about diversity. A five-judge panel built from five models in the same family is not five opinions. Shared training lineage means shared preferences, so in the limit a fully correlated panel of five is one judge counted five times wearing different name tags. Anonymization cannot save you from that, because the bias is in the composition, not the labels. The fix is upstream: compose the panel from genuinely different architectures, or measure how often your judges disagree and weight accordingly. A panel that always agrees is not a panel. It is an echo with a quorum.

The principle

The mechanism is the right foundation even with those three caveats, and the reason is structural. You do not coax a biased judge into fairness with a better prompt. You remove the information the bias needs to operate, so it cannot operate, and then you treat the residue as second-order cleanup. Structure the problem so the failure mode is impossible rather than asking the model nicely not to fail.

That is the part I keep coming back to. I lost a week to a panel that looked healthy while it voted for its own reflection, and the fix was not a clever model or a longer rubric. It was taking away a name tag. Karpathy had already shipped the idea, plainly, and the only work left for me was recognizing my own problem in it and admitting the version I had built was a popularity contest. If you are wiring models to judge models, run the panel blind before you trust a single ranking it gives you. Mine looked fine for a week. It was quietly rigged the whole time.

References

llm-council, Andrej Karpathy, 2024. The label-anonymization design that this piece leans on, which aggregates the anonymized rankings by average rank position. Source: github.com/karpathy/llm-council

LLM Evaluators Recognize and Favor Their Own Generations, Arjun Panickssery, Samuel R. Bowman, Shi Feng. NeurIPS 2024. The source of the GPT-4 self-preference win rate above 0.90. arXiv: 2404.13076

Drift Detection for LLM Routing: Catching Silent Model Degradation

praveenlavu — Thu, 18 Jun 2026 22:14:32 +0000

Drift Detection for LLM Routing: Catching Silent Model Degradation

It's 2am and I am staring at a routing layer I spent weeks tuning, running a thought experiment that will not let me sleep. The router is doing exactly what I built it to do. Nothing in my code would change, nothing in my config would change, and yet I can see, plain as day, the night this system goes confidently, repeatedly wrong while every line of it stays correct. The failure is already baked in. I just have not been bitten by it yet.

The setup is simple. I route incoming tasks across four capabilities: a fast cheap model, a slow expensive one, a retrieval tool, and a code-execution agent. Each task goes to one of them, and I watch a single binary signal, did the output pass the quality gate or not. Run that for a few thousand calls and the policy converges, the weights stabilize, the dispatcher learns which arm wins. For a while, life is good. And that good stretch is exactly the trap.

Here is the scenario that keeps me up. The fast cheap model gets silently updated by its vendor, and its accuracy on my tasks quietly collapses. My router has no idea. It is carrying a high historical success estimate for that arm, earned honestly over weeks of good performance, and it keeps routing there because three weeks ago that was the right call. The dispatcher would not be broken. It would be right about a world that no longer existed. It would be wrong because it remembered too well.

The assumption nobody tells you you made

Multi-armed bandit routing, from Thompson Sampling to UCB to plain epsilon-greedy, rests on one quiet premise: each arm's true success rate is fixed. Pick the arm with the best estimate, keep nudging it as outcomes arrive, converge on the best option. Clean, and it works right up until the ground moves. In production LLM routing the ground moves constantly. Models get updated, prompts age, the external API you lean on degrades. The question was never whether drift would hit me. It was whether my routing layer would notice before my users did. By default, it wouldn't.

The cruel detail is the inertia. With a slow learning rate the running estimate remembers roughly the last twenty observations, which feels fast enough until you picture an arm that has served two thousand calls at 85 percent. A moving average with that much history behind it does not flinch when the truth drops to 30 percent. It takes dozens of fresh failures just to halve the gap, and far longer to close it, and every one of those failures is a task routed straight into the hole.

My first instinct was the obvious one, and it was wrong: turn up the learning rate, make the estimate forget faster. A faster learning rate makes every estimate jittery all the time, even on arms where nothing has changed. I would be trading one silent failure for a router that twitches at noise. That is not a fix, it is a different bug. What I actually needed was narrower. Not a shorter memory everywhere, but a way to forget on purpose, surgically, only on the one arm that had genuinely shifted, and only when a real shift had occurred. A tripwire per arm.

The turn

The tool already existed, and it had existed since 2007. ADWIN, for adaptive windowing, published by Bifet and Gavaldà at SIAM SDM 2007, does exactly the surgical thing I was reaching for. It watches a single stream of binary outcomes and keeps a window over them. After every new result it asks one question: is there a point inside this window where the older stretch and the recent stretch look like two different distributions, too far apart to be the same thing wearing noise?

If no, the window just grows. Stable periods accumulate evidence, and a bigger window makes the test harder to fool, so it does not trip on ordinary variance. If yes, ADWIN declares drift, throws away everything before the split, keeps only the recent post-shift stretch, and tells you, so you reset the arm's estimate using only what survived. That collapse is the whole idea. A fixed window can only notice a shift once it has been present for about half its length, so you are always looking backward at a horizon you had to guess in advance. ADWIN's window grows without bound while things are calm, building the power to resist false alarms, then collapses hard the instant a real shift lands. The window size is an answer the data gives you, not a knob you set and pray over. For a bandit with several arms, I run one independent ADWIN per reward stream; they share nothing, because arms do not generally degrade at the same moment, and each one watching its own stream in isolation is not a simplification, it is the correct model.

A single sensitivity knob governs how eager the test is to fire, really the false-positive rate you will tolerate on a stable stream. Under the hood its tolerance band tightens when the split is balanced and is nearly impossible to trip when only one or two observations sit on one side, so a single fresh result never declares drift on its own; it carries a gentle penalty for checking every possible split point; and it tightens further on a low-variance arm that almost always succeeds, so real degradation on a near-perfect arm gets caught sooner rather than hiding in slack. The river library uses a variance-aware form of the bound that runs tighter for large windows and high-quality arms than the simpler version in the original 2007 paper; the two agree closely when the window is very small. For routing I settled on a sensitivity of 0.002. With a window around a thousand observations that keeps spurious firings well under one per five hundred evaluations per arm, which across four arms at a hundred routing decisions an hour is a false drift event roughly once every thirty hours, low enough not to pollute the policy, high enough that I am not waiting weeks to catch the real thing.

The original authors prove two guarantees, and both held up. On a stationary stream the odds of a false alarm stay bounded by the sensitivity setting. And once a real shift of a given size lands, ADWIN catches it within a number of observations that scales inversely with the square of the shift magnitude, so a big drop is caught fast and a subtle one takes proportionally longer. Both bounds are tight.

Watching it work

I did not want to trust any of this on faith, so I built a small synthetic run to validate the design before it ever touched a real reward path. Four arms, five hundred steps, a fixed seed so it reproduces. Three arms hold steady at roughly 0.70, 0.65, and 0.60. The fourth, capability A, starts strong at 0.85 and then, at step 300, drops off a cliff to 0.25, standing in for exactly the silent vendor update I had been losing sleep over.

The first time I saw the log line appear, the feeling was disproportionate to a synthetic test. Around step 318, eighteen steps after the true shift, ADWIN fired on capability A. Its estimate dropped from about 0.85 to about 0.28 as the window collapsed from over three hundred observations down to roughly a dozen, dumping the stale high-accuracy history in one motion. A second, smaller event near step 412 was just the window settling onto the new regime. By the end, capability A's routing weight had fallen from near 0.40 before the drift to around 0.11, its honest post-drift share, while the three stable arms held near 0.25 to 0.29 each. The eighteen-step lag lines up with the guarantee: the shift here is 0.60, which puts the theoretical floor at only a handful of observations, and constants and warm-up account for the rest.

One design choice mattered more than the rest. The arm's estimate has to be decoupled from ADWIN's internal window: when drift fires, the monitor reads the fresh collapsed window's mean and adopts it as the new estimate, and that is the fast policy refresh the whole exercise exists to produce. The other thing I refused to give up was that no arm is ever zeroed out. A tiny floor on every weight keeps a sliver of exploration alive even for a failing arm, seeded at an uninformed 0.5 when it is new, so the system keeps probing it and can notice the day it recovers. A degraded arm is not a dead arm. Sometimes the vendor ships a fix and you want to find out.

Where it does not belong

A clean synthetic result does not make ADWIN a hammer for every problem, and pretending otherwise is how you get burned somewhere else.

If an arm sees fewer than thirty observations an hour, the tolerance band is enormous, only a total collapse trips it, and the post-drift estimate is too noisy to trust anyway. Aggregate at a coarser granularity or reach for a Bayesian change-point detector with an informative prior.

If the drift is gradual, ADWIN is the wrong tool by design. It is built for abrupt shifts, a model update, an endpoint degrading, a sudden change in the prompt mix. For an arm whose success rate decays a percent a week as its world knowledge ages, the window grows slowly, the gap between old and recent stays narrow, and detection lags by months. A scheduled two-sample test on rolling seven-day buckets is what that job wants.

And if your non-stationarity is structural, if reward correlates with time of day or task type or session by design, ADWIN will fire constantly and churn your estimates into noise. That is not a routing system with drift detection. That is a contextual bandit in denial, and the fix is to model the context explicitly rather than detect it as drift.

The principle

You don't have to implement any of this yourself. ADWIN ships maintained in the river online-machine-learning library; the work is wiring one detector per arm into the reward path and letting a drift signal reset that arm's estimate to its post-collapse window. I kept the companion implementation small, one monitor, one synthetic run, an optional plot, and a test suite above 80 percent coverage, with river as the only real dependency, so the whole thing reproduces from a single command.

But the wiring is not the lesson. The lesson is the one that kept me up at 2am: a router that learns is also a router that can be confidently, durably wrong the moment the world it learned stops being true. Convergence is not the finish line. A converged policy is a strong opinion about a fixed reality, and production has no fixed reality. A dispatcher that was right three weeks ago and wrong tonight is not malfunctioning. It is believing its own history a little too hard, and nothing in the loop is watching for the day that history stops being true.

So now something is. One tripwire per arm, quietly asking after every outcome whether the past still predicts the present, ready to forget on purpose the moment it doesn't. If you route anything across capabilities that can change underneath you, and in this field everything can, you want that tripwire in the loop before the page arrives, not after.

References

Bifet, A., & Gavaldà, R. (2007). Learning from time-changing data with adaptive windowing. In Proceedings of the 2007 SIAM International Conference on Data Mining (SDM 2007), pp. 443-448. Society for Industrial and Applied Mathematics. doi:10.1137/1.9781611972771.42

river Python library. Online machine learning in Python. BSD-3-Clause license. Source: github.com/online-ml/river. ADWIN implementation: river.drift.ADWIN.

Two queues for local-LLM fleets

praveenlavu — Thu, 18 Jun 2026 22:13:18 +0000

Two queues for local-LLM fleets

Two ollama pulls, plus an LM Studio Llama 70B load, plus two subagents hitting a cloud LLM provider's API, plus seven daemons running scheduled scans. All at once. 2026-05-13, 10:58 UTC. Kernel panic.

I'd triggered all of them myself, carelessly, inside ten minutes. The ollama pulls were fetching the latest quantized weights for two different oracle models. LM Studio was loading a 70B parameter model into resident memory for a council review. Two subagents were dispatched via my orchestration layer, each making concurrent calls to a frontier model API. Seven launchd daemons fired on schedule because it was the top of the hour. The machine had 96GB of unified memory. It wasn't enough.

The postmortem produced a rule I now follow religiously: local-heavy tasks run serially, one at a time; remote-API fleet tasks run with bounded concurrency; never cross-mix the two. This is the two-queue discipline.

A quick vocabulary stop before we go further. By fleet I mean a set of cloud-LLM agent calls running in parallel against remote APIs. Each agent is a thin process; the work happens server-side. By oracle I mean a heavy local model I load into resident memory for high-stakes reasoning that has to happen on the machine itself, usually for IP or latency reasons. Both are part of the same agent-orchestration pattern, but they have completely different resource profiles. That difference is exactly why mixing them blows up.

The two task classes

When you're running local LLMs alongside cloud APIs, the work splits into two classes based on where the saturation happens.

Local-heavy tasks saturate your machine's resources directly. Model downloads via ollama pull. Loading a 30-70GB quantized model into LM Studio or ollama with keep_alive set to hold it resident. Running a full pytest sweep across your entire repo. Oracle council dispatch where you're loading massive models into RAM for inference. These compete for unified memory bandwidth, disk I/O, CPU cores for dequantization.

Remote-API fleet tasks saturate neither your memory nor your CPU. They saturate network connections and your cloud provider's rate limits. Subagent dispatch via an orchestration layer where each agent calls a frontier model via API. Parallel web scrapes. Batch processing jobs that fan out to external services. These tasks are I/O-bound on network latency and remote throughput, not local compute.

The distinction matters: saturation failure modes are different. Local-heavy tasks cause memory pressure, thermal throttling, and in the worst case, kernel panics when the memory allocator can't satisfy a request. Remote-API tasks cause connection pool exhaustion, rate limit errors, and cascading timeouts when you overwhelm either your network stack or the remote service.

Why mixing them saturates faster than you'd expect

Here's the non-obvious part. On Apple Silicon, the memory is unified: the same physical RAM serves CPU, GPU, and the Neural Engine. When ollama loads a 67GB quantized model with keep_alive > 0, it's not just occupying 67GB. It's pinning that allocation in a way that fragments the address space for everything else. The OS can't trivially reclaim it because the model needs to stay resident for fast inference.

Now add two ollama pulls in parallel. Each one is streaming gigabytes of weights from the network and writing them to disk while simultaneously validating checksums and decompressing blobs. That's sustained disk I/O and memory allocation churn. The system is juggling resident model memory, in-flight download buffers, filesystem cache pressure, and whatever else is running.

Then add an LM Studio load: another 30-70GB allocation request. LM Studio is trying to mmap the model file into a contiguous region. If the address space is already fragmented by the ollama allocations, the kernel has to work harder to find a suitable range. On a system with 96GB total and a 67GB model already loaded, the remaining headroom is only ~29GB. That's before fragmentation overhead, filesystem cache, and kernel allocations eat into it. A 30-70GB request collides with that headroom immediately. At the low end, a smaller model squeezes in with zero margin for spikes; at the high end, the request fails outright. Either way the kernel starts swapping, which on a machine built for low-latency inference is effectively a soft hang.

Now add two subagents making concurrent API calls. They're not heavy on memory, but they are allocating connection state, buffers for HTTP responses, and JSON parsing overhead. Not huge individually, but enough to tip the balance when the system is already under memory pressure from the local-heavy work.

Add seven daemons firing on schedule. Maybe they're just doing lightweight scans, but each one spawns a process, allocates a stack, opens file descriptors, and touches the filesystem. Again, not huge individually. But in aggregate, on a system already saturated, it's the cumulative load with no single smoking gun. Each spawn adds overhead the kernel can't reclaim fast enough.

The kernel panic isn't random. It's deterministic. You've exceeded the practical working set the unified memory architecture can sustain under concurrent pressure. The math isn't "96GB total, models fit if they sum to less than 96GB." The math is "96GB total minus fragmentation overhead minus in-flight buffers minus filesystem cache minus kernel allocations minus margin for allocation spikes." You hit the limit well before the total RAM. Once you account for everything else the system is doing, the practical working set is significantly smaller.

The two-queue rule

Don't mix the classes. Run them in separate queues with separate concurrency limits.

Local-heavy tasks: serial only. One at a time. No exceptions. If you're pulling a model via ollama, nothing else heavy runs until it's done. If you're loading an oracle model into LM Studio, no other model loads or downloads run concurrently. If you're running a full test sweep, no oracle dispatch, no model pulls. Serial.

Remote-API fleet tasks: bounded concurrency, ≤5 concurrent by default. Five subagents hitting cloud APIs in parallel is fine. They're network-bound, not memory-bound. You can saturate your rate limit before you saturate your machine. But don't go unbounded. Connection pool exhaustion is real, and most cloud providers have per-account concurrency limits anyway.

Never cross-mix. If a local-heavy task is in flight, the remote-API queue is paused. If the remote-API fleet is running, no local-heavy tasks start. This is the hard rule. It eliminates the saturation-interaction failure mode entirely.

The pre-flight gate

Before starting any heavy task, I check three things.

Load average. uptime shows the 1-minute load average. If it's above 4.0 on my 12-core machine, something is already saturated. Wait or kill. Don't pile on.

Free disk space. df ~ shows available space on the home volume. I need at least 30GB free before pulling a large model. Ollama writes to a temp location before moving the final file, so you need roughly 2× the model size in transient headroom.

In-flight heavy task check. I explicitly verify no other local-heavy task is running. ps aux | grep -E 'ollama.*pull|lmstudio|pytest|oracle_dispatch' as a sanity check. It's manual, it's primitive, but it works. Automation can come later. Discipline comes first.

If any gate fails, I don't proceed. I reschedule or I kill the conflicting task. The pre-flight check takes ten seconds. Recovery from a kernel panic takes ten minutes and loses whatever state was in flight.

The forbidden combinations

Some combinations are always wrong:

Local-heavy + remote-API fleet active: memory pressure + connection churn → saturation
Fleet active + oracle dispatch starting: oracle loads a 30-70GB model while fleet holds HTTP state → OOM
Oracle dispatch + ollama pull: two large allocations competing for the same unified memory → kernel panic
Two oracle models loaded simultaneously: 67GB + 30GB > practical working set → swap death spiral
More than 5 concurrent fleet dispatches without override: connection pool exhaustion, cascading timeouts

Each of these surfaced as a real failure at some point. The rule isn't theoretical. It's scar tissue.

The discipline

Postmortems for solo founders means writing rules your future self will hate following, until the day they save you.

What comes next

The two-queue rule is the floor, not the ceiling. Once the discipline is in place, four trajectories open up.

Auto-scheduling. The pre-flight gate is currently ten seconds of manual checks. With the gate logic codified, you can wrap it in a scheduler that decides automatically: when an ollama pull finishes, fire the next queued heavy task. When the fleet rate-limit ceiling approaches, throttle. When load average rises, defer. Manual discipline becomes automatic policy. The hard part is not the scheduler. The hard part is encoding "what counts as heavy" precisely enough that the scheduler doesn't have to ask.

Cross-machine fleet coordination. The same logic extends to a multi-node setup. One machine handles oracle work, another handles fleet, a third runs daemons. The queues become network-coordinated and the rule generalizes: never let a node accept a task that would push it past its per-class concurrency cap. The interesting design question is where the queue state lives. A Redis sorted set works for two nodes. Past five nodes you start wanting a real durable queue, and now you have a different operations problem.

Predictive saturation modeling. The kernel-panic math at the end of section 3 is a working-set predictor in disguise. Given a tuple of (free memory, in-flight model sizes, filesystem cache pressure, kernel allocation headroom), you can compute whether a candidate task will fit before dispatching. The math is there. What's missing is the wrapper that runs it on every dispatch and refuses the unsafe ones. That refusal is more valuable than the scheduler, because it stops you from making the mistake in the first place.

Observability that matches the discipline. A single surface showing what's queued vs running per queue, real-time load average, projected memory headroom after the next dispatch. Not a separate observability project. Just the gate logic exposed as state, with a small UI on top. The reason this is worth building: when the rule fails it fails silently for several minutes before the kernel intervenes. A live view of "you're about to violate the rule" beats a postmortem of "you violated the rule" every time.

Each of these builds on the two-queue rule without abandoning it. The rule stays the same: serial on local, bounded on remote, never cross-mix. What changes is how much of the discipline you have to hold in your head, and how much the system holds for you.

AI Disclosure

This artifact was prepared with assistance from generative AI tools, in
accordance with COPE+STM "AI in scholarly publishing" guidance and the
target venue's AI-use policy:

Drafting: Author-Enthusiast agent via a frontier large language model
Voice + IP firewall: Author-Human agent via a frontier large language model
AI-tell removal + readability: Author-Humanizer agent via a frontier large language model
Final responsibility: The named human author has read and approved the final content. No generative AI is listed as a co-author. Substantive intellectual contribution remains with the human author.