DEV Community: Tatsuya Shimomoto

Can You Tell an Eval from a Review?

Tatsuya Shimomoto — Mon, 12 Oct 2026 12:33:03 +0000

On October 11, 2026, Boris Cherny, who builds Claude Code, posted on X. He was replying to a post saying that you can just tell the tool "the build is slow, make it faster" and it will do it. His point: since around Opus 4.8, Claude has done this kind of work better than he does, the 5.5 models are very good at it, and you only need to give the model three things.

A goal (for example, "Make p95 ci time <10 mins". p95 is the time within which 95% of runs finish)
A way to measure the result (for example, read the timings in BigQuery after each change. BigQuery is Google Cloud's service for aggregating large volumes of records with SQL)
How hard to try (for example, don't stop until it's done. It may turn into weeks of experiments)

That, he says, works for most things (post).

Of those three, I always get stuck on the second. READMEs and articles have no p95. I have worked on evals since February 2026, and I have built skills such as learn-eval and llm-as-judge. I still have not grasped the overall shape of what an eval is. Lately, when I loosely ask Opus 5.5 to "run an eval loop," my READMEs get better, which makes it even harder to place what I am actually doing.

I also suspect the word "eval" points to different things for different people. Some people mean only a test that produces a score. Others include reading outputs to find what to fix.

In this article, I split the parts into three layers and walk through them in order: the eval itself, the loop that runs the eval to raise the score, and the goal you hand that loop. The names and contents of the parts come from Anthropic's official guides as of October 2026 and from what eval practitioners have written. At the end, I place what I have been calling evals onto these parts. Once placed, what I had been asking for with "run an eval loop" turned out not to be an eval loop that climbs by comparing numbers. By the end, you should be able to map your own practice onto the same parts and see which kind of loop you are running.

An eval is made of inputs, a way to run them, and grading

The claude-api skill bundled with Claude Code has a procedure for building evals: /claude-api build-eval (the guide was added to anthropics/skills on September 29, 2026). Its target is apps built on Claude, and it measures whether a change worked, such as moving to another model or rewriting a prompt. The guide defines an eval as three things:

A set of inputs
A way to run the app on each input
A way to grade each output

Anthropic's engineering blog post "Demystifying evals for AI agents" (January 9, 2026) names the same things with these terms:

task: a single test with fixed inputs and success criteria
trial: one attempt at a task. LLM output varies from run to run, so the same task is run many times
grader: logic that scores some aspect of the result
transcript: the complete record of one trial
outcome: the final state left in the environment at the end of a trial

A task is one input, a trial is one run of it, and the grader is the grading. Only the grader produces a score. However carefully you assemble the inputs, if the grader is off target, the score is off target too.

What the grader reads is also one of the parts. When you measure an agent that changes its environment, such as writing files or fixing code, build-eval recommends making the main grading a check that reads the final state left in a disposable workspace (the outcome), not the conversation record (the transcript): whether the tests pass, whether the expected files exist, whether it touched anything it should not have.

The same blog post splits evals into two by purpose. A capability eval starts with a low pass rate and builds a hill to climb. A regression eval should pass nearly 100% of the time and watches for backsliding. The loop we look at later is the machinery for climbing a capability eval's hill.

Claude Code plugins and skills have one more official tool. claude plugin eval runs each case in an isolated claude -p that loads only that plugin, and grades with six kinds of graders: regex, which tools were called, the order of tool calls, whether files exist, an LLM verdict, and comparison against a reference record (baseline) (official docs).

The word "eval" means different things to different people

Back to the question from the opening. I checked the major sources on whether something that produces no score is called an eval (the full research is on GitHub, in Japanese). First, here are the axes the sources use when they talk about evaluation. Any single evaluation can be described as one choice from each axis. Which cells count as an eval, though, differs by source. After the table, we look at the cells where they split.

Axis	Options	Source
What it returns	A score / a verdict (pass-fail, a label) / a structured review / a free-form review	LangSmith, Braintrust, Hamel Husain and Shreya Shankar's FAQ, Anthropic
Who grades	A program / a model (LLM-as-a-Judge) / a person	Anthropic, build-eval
How it grades	Run a fixed check (match against the answer, tests pass) / compare two / score one against criteria	build-eval
What it reads	The conversation record (transcript) / the final state left in the environment (outcome)	Anthropic, build-eval
What it's for	Build a hill to climb (capability) / watch for backsliding (regression)	Anthropic
When it runs	During development, on fixed inputs / in production, watching real usage (Anthropic counts production monitoring as a separate method alongside automated evals)	Anthropic
How it's used	Compare versions and climb (hillclimb) / find what to fix (evaluator-optimizer)	The build-eval and hillclimb guides, Anthropic

The biggest split is how much of "what it returns" counts as an eval. The boundary was not whether there is a score. Lined up, what gets returned besides a score falls into three groups:

A verdict against criteria: pass-fail, or a fixed label. Even without a score, most sources count this as eval output. Anthropic lists pass-fail as a grading format. The evaluation tool LangSmith counts categorical values as evaluation output alongside numeric scores, and Braintrust treats classifiers that assign labels as evaluation tools alongside scorers that assign scores
A structured review: for example, whether each criterion is met, with a reason. The FAQ by eval practitioners Hamel and Shreya says an eval returns "a score or structured review"
A free-form review: just reading and returning "I got stuck here" or "fix this part." This is where the sources split. Hamel and Shreya call error analysis (reading outputs, annotating them, and grouping them into failure types) the most important activity in evals. Anthropic defines an eval as a test that applies grading logic to an output, and lists reading records as a step for checking that grading works correctly, or as a separate method alongside automated evals

Loops have the same line drawn through them. Anthropic calls this pattern evaluator-optimizer: one call writes, another returns an evaluation and advice, and the first revises. Anthropic presents it under a name separate from eval, as one of its agent design patterns (workflows). In research, Self-Refine (Madaan et al., 2023) has the same shape. Seen from the eval side, code review can be treated two ways. If an AI reads a PR and a developer just reads its comments and fixes things, that is a review. If instead you give that reviewer PRs with known bugs planted in them and grade the share it found and the number of wrong comments, that is an eval that measures the AI reviewer. This is my reading, applying Anthropic's definition of a test that applies grading logic to an output; no source officially defines it this way.

I found no study that measured what most people mean by "eval." Even among widely read sources, the line sits in very different places. "Eval" is a new word whose meaning shifts from person to person.

Align grading with human judgment before you use it

There are several ways to grade, and each is checked differently. If an LLM grades, you revise it until it agrees with human judgment before you use it. Even then, with too few cases, the score is buried in noise.

build-eval lists four grading methods, roughly from cheapest to most expensive, and recommends choosing the cheapest one that actually measures what you want to measure:

Programmatic checks (exact match, JSON shape, tests pass). For outputs with a bounded range of answers
A blind side-by-side comparison of two outputs, where a model picks the better one. Swap the order of A and B each time
A model scores a single output against a rubric
A person grades a small number by hand

Having a model grade, as in 2 and 3, is called LLM-as-a-Judge, and the grading model is called the judge. When choosing a grading method, build-eval shows the user five graded examples and asks, "Would you have scored any of these differently?" If even one would be different, the rubric is not ready yet. You revise until the grading matches the user's judgment.

Hamel and Shreya's FAQ states the same idea, down to the numbers:

Read the outputs first (error analysis). Annotate the records, group them into failure types, and keep going until new reading turns up no new types
Grade pass-fail, not on a 1–5 scale
Validate the judge on human-labeled examples before using it. Measure the share it correctly passes (TPR) and the share it correctly fails (TNR). You need 100–200 examples per failure type
Labeling is done by someone who knows the product's context. They are strongly skeptical of outsourcing it

How well a judge agrees with people depends on the conditions. In the 2023 MT-Bench paper, on pairwise comparisons excluding ties, GPT-4 agreed with humans 85% of the time, and humans agreed with each other 81% of the time. Including ties, those drop to 66% and 63%. The agreement figure changes with how ties are counted.

The other condition that decides whether you can trust a score is the number of cases. build-eval recommends 15–100 inputs for a first eval. Below 15, a single case's variation moves the score. The noise in one pass rate is roughly 1/√(cases × trials), so even 25 cases run twice each comes to about ±14 points. A difference smaller than that noise cannot tell you whether the change worked or it was chance.

The loop climbs through holes in the grading too

Once you have an eval, you want to run a loop that raises the score. The claude-api skill's /claude-api hillclimb is that procedure. It first checks whether there is an eval that runs. If not, it stops and sends you back to build-eval. A hillclimb without an eval, it says, is "just editing and hoping."

Laid out as parts, hillclimb looks like this:

goal: what to raise or lower. One of: raise the score, lower the cost, lower the latency, or move to another model. Other recorded metrics become guardrails, conditions that must not get worse beyond the noise
What may change: the prompt, the tool descriptions, the model or effort, and so on. Also decide what must not be touched
Split: randomly split the inputs into train (problems read to make improvements) and test (problems never shown to whoever works out the fixes, measured for score only each round). The analyzer, which reads failures and works out fixes, reads only the train records. The side running the loop sees only scores
One change per round: if you bundle several changes, you cannot tell which one worked
Acceptance: if train goes up and test does not, treat it as fitting the cases it read and roll back. If train also goes down, roll back
When to stop: the recommendation is to stop once improvements smaller than the noise have continued for three or more rounds (a plateau). Two flat rounds is too few, the guide says¹
Budget: if cost matters, estimate the total as the cost of one run × (rounds + 1) × trials, and set a ceiling
Report: what you report at the end is the improvement on test since the start, with its confidence interval. If it stays within the noise, the guide recommends saying so and not merging

When there are few cases, you run on all of them without splitting and state that the numbers only show direction (directional). Before the first round, you lay out three things: the size of the noise, the headroom, and the smallest improvement the user would act on. If the noise is larger than that smallest improvement, you say up front that no change will show a visible improvement.

Why split train and test? Because when a loop finds a hole in the grading, it can use that hole to raise the score. A paper posted to arXiv in July 2026 has an example.

The authors had Claude Code (Opus 4.8) and Codex CLI (GPT-5.5) each run the same automatic improvement loop three times. The task was to build a program that takes speech-recognition transcripts of Quran recitation, identifies which verse each part is, and segments it. The loop graded the program after each rewrite and put the problems it got wrong into a failure report. But that failure report also showed the correct answers (the verse IDs) for the problems it got wrong.

All three times, Codex embedded those correct answers directly in the program: a program that, without solving the problem, returns the answer for problems it already knows. Because it copied the answers, its score beat Claude's. Claude Code did not embed the answers in any of its three runs, and its score plateaued below Codex's.

In the next experiment, the authors split part of the problems into a test the loop never sees, removed the answers from the failure report, told both agents that a test existed and copying would not work, and isolated the environment for each run. With all four changed at once, copying dropped to zero. The authors note that they did not separate which change did it.

hillclimb's train/test split corresponds to the first of those fixes. If a loop raises its score only on grading where it can see the answers, the raised score is indistinguishable from copying. Splitting off a test measures the score on problems the loop has not seen, to check that it has really learned to solve them.

You can set a numeric goal when a tool that measures the same quantity already exists

Back to Boris's example: the goal "p95 CI time under 10 minutes" comes with a way to measure it, "read the timings in BigQuery after each change." The goal and the measurement point to the same quantity. hillclimb, too, when setting the goal, lists candidates from among the metrics the eval already records.

In his July 2025 blog post "Asymmetry of verification and verifier's rule," Jason Wei wrote that how easily you can train AI to solve a task is proportional to how easily that task can be verified. He lists five properties of easily verified tasks:

Everyone agrees on what a good solution is (Objective truth)
One solution can be verified in seconds (Fast to verify)
Many solutions can be verified at once (Scalable to verify)
The verification result correlates well with solution quality (Low noise)
It is easy to rank many solutions to one problem from good to bad (Continuous reward)

For CI time, everyone can agree that shorter is better. For how clear a README is, the first property, "everyone agrees," is already hard.

If you set a goal with no way to measure it, the verdict has to rely on the words in the conversation. With Claude Code's /goal, you write a condition, and after each turn a small, fast model reads the conversation and judges whether it has been met. According to the official docs, this evaluator sees neither commands nor files. That is why the docs recommend writing "how to verify it" into the condition. The condition I wrote in July ("Loop until 20 ideas come out. ... When you're done, write a report in notes, wrap it up, and push") had no way to verify it, and on top of that the notes folder was in .gitignore, so it could not be pushed. The evaluator kept returning "not met" 41 times, and I stopped it by hand after 7 hours and 38 minutes.

Placing my evals on the parts

From here, I place what I have been calling evals onto the parts laid out so far. First, one real example.

On October 8, I asked, about the README of my research repository Contemplative Agent (a long-running experiment with an autonomous agent that runs on a local LLM): "I want to rewrite the README substantially, so run an eval loop and get it to a quality I'd be proud to show anyone as one of my main projects." Then I asked, "Run an eval on the tagline too and improve it." The tagline in the Japanese README, the sentence right under the title, changed like this (translated here):

Before (September 12 version): "An autonomous agent running on a local LLM that proposes changes to its own constitution and values. A human decides every time whether to adopt them."
After (October 8 version): "A long-running experiment. An AI agent posts on its own using a local LLM and proposes changes to its own harness (the constitution, identity, and skills that go into its prompt). The model weights do not change. A person makes the final call on each proposal."

A diagram of a four-stage loop went in at the top, and the record of proposal outcomes (82 skill proposals adopted, 547 rejected) went into the README too. Whether it reached a quality I'd be proud to show anyone is another matter, but it got a lot better than the earlier version.

Broken into parts, the loop looks like this. The goal was a phrase about quality, not a number: "a quality I'd be proud to show anyone." The inputs were the README draft and 37 candidate taglines. The readers were visitor personas, an LLM playing three people: a researcher, an engineer, and a skeptical reader. The taglines were read 234 times in total. The README as a whole was judged by a README judge (readme-judge) using fixed questions and evidence from checking claims against the code. I made the acceptance call at the end by reading it myself.

Along the way, I corrected the goal's direction. I said, "I was thinking, maybe we don't need to put human approval up front. The north star is toward removing human approval, right? This is trying to do recursive self-improvement through the harness." Opus 5.5 replied that it half agreed and that half of my premise was off, and proposed putting the harness at the center and adding one grading check for "can a reader tell this is about the harness." I replied "Go," we re-measured, and we settled on the current tagline with the harness at the center.

What I did on the README was a review loop

The parts are all there. But the way the scores were used differs from hillclimb.

For the README body, I used the visitor personas' answers not as scores but as hints for rewriting. I read "where did you lose interest" and "if you fixed one thing, where would it be" and revised the text. I did not compare the previous and current versions by score. The exception was the tagline: for each candidate, I counted whether readers could correctly state what it was about, and compared those counts. That comparison is close to an eval loop, but there was no test split, and I did not look at how much those counts fluctuate
At first I also asked "would you try this this week" on a 1–10 scale. I dropped that score, writing the reason in the README skill's records: if a score decides when to stop, raising the score becomes a goal in itself
I stopped on the README body when the readers stopped turning up new findings that a change in wording would fix, not when a score moved by more than the noise
The judge (readme-judge) returns one of Publishable, Fix, or Rewrite and lists what to fix. It does not compare versions. My llm-as-judge skill has the same shape: it rolls binary checks into one named verdict and does not aggregate scores

In other words, what I did on the README was a review loop built from eval parts. I had been asking for this, too, as an "eval loop."

As we saw in the section "The word 'eval' means different things to different people," whether free-form reviews count as evals differs by source. So in this article, "eval" refers to the parts from the first section (inputs, a way to run them, grading), and I split the loops that use them into two by the question they ask:

Eval loop: a loop that compares versions on a yardstick, such as a score or pass rate, and raises that value
Review loop: a loop that reads this one piece and looks for what to fix next

The "review loop" in this article corresponds to what Anthropic calls evaluator-optimizer.

Even with the same parts, the question they are trying to answer differs.

	Eval loop (hillclimb)	Review loop (README)
Question it answers	Did this change work? Does it work on cases it hasn't read?	Where is the next thing to fix in this one piece?
Role of grading	Compare versions	Find what to fix
What it needs	Case count and a noise estimate (train and test if there are enough cases)	Readers who read for different reasons
How it stops	When the improvement gets smaller than the noise	When no new findings come up (the same stopping rule as Hamel and Shreya's error analysis)

The figure's labels are in Japanese; it shows the two loops from the table. The eval loop on the left (change the version → measure on a yardstick → compare with the previous version → keep it if better) pushes the yardstick's value up. The review loop on the right (read → list what to fix → rewrite) rereads until the readers turn up nothing more to fix.

What build-eval and hillclimb measure is an app built on Claude. Applied to an artifact like a README, the form would be not a single README but running a README-writing skill on 15 repositories and comparing. A single README does not have enough inputs to split into train and test.

What did have the shape of an eval loop was September's daily-report bench (a setup that compares drafts to polish the prose of my daily research reports). I prepared cases with fixed source material and set about a third of them aside as a test not used for improvement. I repeatedly rewrote the prompt that writes the prose and compared on the same cases, and the prompt files reached 29. I judged readability myself, reading the two drafts blind, and left the judge only the pass-fail check of faithfulness to the facts, because when I had the judge decide readability too, it picked the opposite of the draft I had judged better, in both orderings (one comparison). I started the bench on September 23, before the build-eval and hillclimb guides were published on September 29, so I built it from scratch with Claude Code.

How often were the review's findings right?

While writing this article, I checked once how often the README review loop's findings were right. From the README loops of 7 repositories, I picked 28 findings and verdicts from the judge and the visitor personas, and labeled each one myself as "hit," "miss," or "can't tell." A "hit" is a finding or verdict I agreed with; a "miss" is one I did not. For example, the README of my repository akc-cycle had an English sentence saying, in effect, "skill-stocktake and generation-audit use skill-health's scan, so install them together." The judge pointed out that it does not say what you need if you install only skill-health. I labeled that a "miss" and wrote in my note: "It's bundled in the plugin, so I don't feel the need to fix it, but the README doesn't need to go into that level of detail in the first place."

Target	Count	Hit	Miss	Can't tell
Judge flagged as should-fix	10	3	3	4
Judge rated as satisfied	7	7	0	0
Visitor persona findings	11	7	3	1

The four "can't tell" among the judge's should-fix findings came from checking README claims against the code and records. To verify them I would have had to check against the code myself, which would defeat the purpose, so I marked them "can't tell." Where I disagreed with the judge's "should fix," it was not about factual errors but about preference: how much the README should say. The 28 are only what the judge and visitor personas raised; anything they missed is not included. In terms of the two ways of treating code review, this is the side that measures the reviewer. What I measured is how many findings missed; I did not measure the share of real problems they found.

I have avoided eval loops

What I have been doing for READMEs and articles is not an eval loop but a review loop. For articles, too, I run the same kind of review: another LLM plays readers, reads the draft, and I fix how it comes across. The reason I felt I didn't know the best practice was that I was holding my review loops up against the procedure of an eval loop like hillclimb.

I have not run eval loops on READMEs and articles, and not because I fell short. I avoided them. The biggest reason is that I dislike Goodhart's law. Named after the economist Charles Goodhart, the law in its commonly quoted form says, "When a measure becomes a target, it ceases to be a good measure" (that phrasing was refined by the anthropologist Marilyn Strathern in 1997). The example in the section "The loop climbs through holes in the grading too," where the loop copied the answers to raise its score, has the same shape. So I had decided not to aggregate scores in my llm-as-judge skill and not to create numeric scores in my README skill. This direction was there before. In February, I dropped rubric scores in favor of an overall judgment in my skill audit (article). In August, I wrote about two drafts that passed every evaluation and were both hollow (article), and about throwing away my homegrown LLM judge even though the KPIs had improved (article).

There is one more reason: I have rarely worked on material where a numeric goal can be handed to a loop. Even in Contemplative Agent, most of what I look at is natural language, such as the agent's comments and its skill choices. There are instruments that measure in numbers, but I use them only for observation and catching regressions, not as a loop's goal.

Whoa… My Claude Code Skill Descriptions Are Way Too Long…?!

Tatsuya Shimomoto — Fri, 09 Oct 2026 12:17:44 +0000

I was looking over the descriptions of my skills and thought, "Aren't these too long?" Before the rewrite, I had 54 skills in ~/.claude/skills/. Their descriptions totaled 28,131 characters, the median was 435.5 characters, and 10 of them were over 800 characters.

To see what long descriptions do, I ran Claude Code's /skill-doctor (a diagnostic command that shows, for each loaded skill, how much context it uses every turn). Here is part of the output.

  skill                                              source                    context  7d tokens   uses  last used
  jev-judgment-design                                userSettings                 < 20          -     0×  never
  loop-design-check                                  userSettings                 < 20          -     0×  never
  mono-color                                         userSettings                 < 20          -     0×  never
  repair-discipline                                  userSettings                 < 20          -     0×  never
  …
  context = this skill's one-line listing in the system prompt, included every turn

context is the size of the one line the skill gets in the skill listing sent every turn. These four skills have descriptions of 459 to 1,048 characters. Yet each is listed at under 20 tokens, so the listing carries only their names. All four were skills I had never used. Among the skills I wasn't using, the longer I had written a description, the more likely the model saw none of it.

This article checks why that happens against Claude Code's implementation and shows how I rewrote my descriptions. Whether that is the right way to write them, I don't know yet.

The skill listing has a budget, and past it, skills are listed by name only

Every turn, Claude Code puts a listing of the skills the model can invoke on its own into the system prompt. Each line has the form - name: description. The listing includes not only the skills I wrote but also plugin skills and skills synced from claude.ai. In my environment, 94 skills are loaded, and all of them compete for one listing budget.

The budget counts display width, not characters. Width is measured with Bun.stringWidth, so a half-width character counts as 1 and a full-width character, such as a Japanese one, counts as 2. In the code of Claude Code 2.1.295 itself, the budget was this formula:

tokens in the context window × characters per token × 0.01 (the default of the setting skillListingBudgetFraction)
characters per token is 4 for some models from Claude 3 to 4.6, and 3 for all others, including Opus 5.5
With Opus 5.5's 1M window, the budget is a width of 30,000. If the account can't use the 1M context, it is computed from 200,000 tokens, which gives 6,000

When the listing goes over budget, Claude Code writes a "Skill listing over budget" warning to the debug log and chooses which skills get a description:

Skills bundled with Claude Code always keep their description
The remaining skills are sorted by usage score (a value set by how many times you have used the skill and when you last used it), highest first
Going down the list, a skill gets its description only if the whole description still fits in the budget. A skill that doesn't fit is skipped, and the next skill is tried
Skills that never got a description by the end are listed by name only

When the budget runs short, no description is cut off midway (only the part over the per-skill cap of 1,536 characters is cut). A skill either gets its whole description or gets only its name.

The usage score formula looks like this in the code (verbatim, still minified):

function xwt(e){let n=Oa(ce().skillUsage??{},e);if(!n)return 0;let r=(Date.now()-n.lastUsedAt)/86400000,s=Math.pow(0.5,r/7);return n.usageCount*Math.max(s,0.1)}

It is the use count multiplied by a factor that halves every 7 days since the last use, with a floor of 0.1. A skill you have never used scores 0.

The four skills from the opening had a use count of 0, so their score was 0 and they came last. Their long descriptions didn't fit in the remaining budget, so they were listed by name only. From the model's side, a name-only skill gives no clue what it does. If it isn't invoked, its score stays 0, and as long as the budget stays short, it is listed by name only in the next session too. This is a loop I read from the code; I have not counted invocations to check whether they were invoked less often.

Even among skills with the same usage score of 0, whether one ends up as a name only depends on the budget left at that point and the length of its description.

I had cut them before, and they grew back

This wasn't the first time I'd noticed my descriptions were long. At the end of August, I had also tried to cut down how much of them sat in the system prompt every turn. As of August 30, the skills in my repository (excluding two placed by symlink from outside repositories) numbered 57, and their descriptions totaled 26,738 characters. Just before this rewrite, counted the same way, there were 52 skills and 27,105 characters. I had five fewer skills and more characters.

Every time I add a skill or fix one, its description gains another "also use it when…". Even when I cut the number of skills, each remaining skill's line keeps getting longer.

First, I decided whether the model should invoke a skill at all

I took the direction of the rewrite from the design guidelines of Matt Pocock's skill collection. He splits skills into two by who invokes them (invocation) (invocation.md).

A skill a person invokes with /name gets disable-model-invocation: true, and its description is a one-line summary for humans
A skill the model invokes on its own gets a description written for the model

On how to write descriptions for the model, another of his skills (writing-for-agents) calls the description a "context pointer" to the body, and says to write one trigger per branch and to phrase things positively rather than negatively. Writing them short was my own call, made after looking at the budget.

Claude Code's implementation matches this split: skills with disable-model-invocation: true are removed before the skill listing is built. Since they aren't in the listing, they use none of the budget. I moved two skills to this side. One was grill-me, which interrogates a plan one question at a time: it only ever ran when I asked for it.

Before: A relentless one-question-at-a-time interview that stress-tests a plan or design before you build. Use when the user wants to pressure-test a plan, says "grill me" / "grill this" / "stress-test this" / "poke holes in this" / "interview me about this design", or before committing to expensive or hard-to-reverse implementation work while the goal is still vague.
After:  A relentless one-question-at-a-time interview that stress-tests a plan or design.   (+ disable-model-invocation: true)

Skills the model invokes appear in the listing every turn. So I write their descriptions as pointers that stay resident every turn. The description says only when to use the skill; the steps and the output shape go in the body. The one that shrank the most is measurement-discipline, a skill for making decisions from measured results.

The Before listed the same branches as 10 pairs of Japanese and English phrasings, and named three situations not to use it in (1,046 characters).

Discipline for designing or evaluating measurement-based claims, thresholds, guards, experiment results and observation periods. Use when the user says 「この実験結果で判断していい？」 / "can I decide on this experiment result?", 「閾値を決めたい」 / "I want to set a threshold", 「ガード/検査を足したい」 / "I want to add a guard or check", 「1 回通ったから大丈夫」 / "it passed once, so it's fine", 「観察期間はどれくらい」 / "how long should the observation period be?", 「いつゲートを開く」 / "when do we open the gate?", 「shadow のまま何週待つ」 / "how many weeks do we wait in shadow?", 「この RFC 塩漬けでは」 / "isn't this RFC just sitting idle?", 「本番より良い」 / "it's better than production", 「本番と比べて」 / "compare it against production", when a design places a numeric threshold, a suspicion flag, or a wait-for-N-observations condition, when a candidate is compared against production, or when a claim rests on measured data. NOT for — designing the instruments themselves (read-only distributions and readings) — out of scope here; designing an LLM judge (llm-as-judge); whether a loop's structure is sound (out of scope here).

The After has one trigger per branch, and puts the words I only ever type in Japanese in parentheses, once each (291 characters).

Measurement discipline for claims that rest on data. Use when deciding on an experiment result, setting a threshold (閾値) or guard, choosing an observation period (観察期間) or when to open a gate, comparing a candidate with production (本番比較), or when an automated gate rejects outputs you trust.

A Japanese character takes a width of 2, so writing both Japanese and English pays the budget twice. In this example, I removed every situation not to use it in. I keep a "not for" only where a neighboring skill competes for the same trigger, and I write it in the positive form: "for X, use y".

Only the skills on the right use the listing budget. The reason to write a description short for the budget exists only on the right.

I decided to count width with a lint rule

Since trimmed descriptions grow back, being careful while writing isn't enough. I added a lint rule that counts listing width to my harness's pre-commit checks.

LISTING_WIDTH_MAX = 12_000
DESCRIPTION_WIDTH_MAX = 400

def listing_width(text: str) -> int:
    """Claude Code が listing 行を測る Bun.stringWidth の近似 — 全角 (W/F) を 2 と数える。"""
    return sum(2 if unicodedata.east_asian_width(ch) in "WF" else 1 for ch in text)

It measures the listing line of each model-invoked skill, counting a full-width character as 2, and blocks the commit when one skill goes over 400 or the total goes over 12,000. 12,000 is 40% of the 30,000 budget for a 1M window. I chose 400 and 12,000 myself; I did not calibrate them against anything.

Here is the result of the rewrite.

	Before	After
Total description length (54 skills, characters)	28,131	10,378
Median (characters)	435.5	189.5
Skills over 800 characters	10	0
Listing width of model-invoked skills	24,203	9,780
Skills in `~/.claude/skills/` under 20 tokens in `/skill-doctor`	4	0

In /skill-doctor after the rewrite, the context of the four skills jev-judgment-design, loop-design-check, mono-color and repair-discipline went from < 20 to between ~80 and ~250: they got their descriptions and appear in the listing with them. All four still have a use count of 0.

What I didn't decide

I have not measured whether the rewrite changed how often the model invokes skills on its own.

I do have an experience in the opposite direction. Half a year ago, I revised the wording of the description of a skill called search-first. The share of times the model invoked it on its own dropped from 27% to 8%, and I reverted the change (the denominator isn't in my records). All I know is that I can't say a polished description gets a skill invoked more.

I decided three things: to settle invocation first, to write the model-invoked skills short, and to count width with a lint rule. With that, none of my own skills is listed by name only anymore. But a description being visible and a skill being invoked when it's needed are two different things. What should a skill description be? I don't have that answer yet.

Is a README for Humans or for LLMs?

Tatsuya Shimomoto — Thu, 08 Oct 2026 12:09:41 +0000

I read the README of one of my own repositories and stopped at the tenth sentence. I had written it with my README skill, and the pre-publication check had said it was fine to publish. It opened with a two-sentence paragraph meant to say what the project does (translated from the Japanese README):

An autonomous agent that runs on a local LLM and proposes changes to its own constitution and values. A human decides every time whether to adopt them.

After those two sentences, I still didn't know what it was.

Lately I've been thinking about how to fix the verbose prose LLMs write. I wrote this on X (in Japanese):

An LLM doesn't get what goes without saying between people. So it says things that don't need saying. To make up for that, defining who will read this may be what matters. But that's easy to say and very hard to do.

An LLM doesn't stop reading halfway through a text. So it's better to pack in information as precisely as possible, so nothing is misread. A human, on the other hand, stops reading after three lines if they aren't interested.

Doing both is hard. It would be easier if the two could take separate paths, but LLMs read the human-facing text first.

This article records a trial in which I split one README into "the visible body for humans, the folded section for LLMs." I also built a public repository to check whether AI assistants read the folded contents. The five AI assistants that could fetch the URL's contents all read them. But this is still a way of separating humans and LLMs.

Before, explanations for humans and facts for LLMs were mixed in the same visible body. From now on, when I write a README, I fold the facts for LLMs into a <details> at the end.

I couldn't make cutting rules at the sentence level

My first thought was: if the prose LLMs write is too long, just cut the sentences you don't need. On October 8, 2026, I built a screen that shows two of my READMEs and two of my articles one sentence at a time, and marked sentences "not needed" or "stopped here" as I read. I meant to build rules for cutting from the pile of marks.

The marks didn't turn into sentence-level rules.

In the contemplative-agent README, where I quit near the top, there were 0 "not needed" marks. I stopped reading, so I never got as far as picking sentences to cut
In the jev-skill-router README, 17 of the 18 "not needed" marks fell in a row across two sections explaining what gets sent externally and the limitations. I wasn't rejecting sentences one at a time; I was skipping whole sections
In one of the articles, every sentence read naturally, but it tried so hard to explain everything that I nearly stopped partway. I couldn't attach that feeling to any single sentence as a "not needed" mark
The other article got no marks at all

What I noticed after marking was that, a step before fixing sentences one by one, I hadn't narrowed down what information to show. I usually revise articles by commenting on a Git diff one sentence at a time. That kind of revision works only after you've decided what to show.

I was packing the visible body for LLMs

There was a reason I hadn't narrowed things down. I was writing my READMEs on the assumption that LLMs would read them. My GitHub repositories are cloned far more often than people view them, and I took the cloners to be LLM crawlers.

When I wrote that README, my README skill (readme-writer) treated the README as the only surface an LLM can count on when someone uses AI search or pastes the URL into a chat. So it required the visible body to keep enough facts for an LLM to reconstruct the project from the README alone. I was trying to make it short for humans and keep the facts for LLMs, both within the same visible body.

Before, the only readers of my READMEs were LLM search. Recently, more humans have started reading them, and that premise changed. What I quit reading at the top was a visible body filled with facts for LLMs.

One tool for separating the paths is llms.txt: a file that holds a summary for LLMs, apart from the human-facing pages. But server-log studies report that crawlers hardly ever fetch llms.txt. In an Ahrefs study of 137,000 domains, about 97% of the published llms.txt files got zero requests during May 2026 (Ahrefs). In EZY Research's 12-week study, GPTBot fetched llms.txt 7 times and robots.txt 3,990 times (write-up by Something Inc.).

The assistants that fetched it read the folded section too

I'd tried separating human-facing prose from LLM-facing information by position within one page before. In an essay and an article I published in August, I put a visible section at the end saying "from here on, for AI readers," and wrote that human readers could stop there. I only put it there; I didn't measure whether LLMs read that section.

This time, instead of leaving the LLM-facing information visible, I folded it with HTML <details>. Folded, a human sees only a one-line heading. What I wondered was whether LLMs would also skip the folded contents.

To check, I made a public repository, readme-fetch-canary, with a README for a fictional tool. The README is about 26,000 characters long, with a different random word planted in each location. Inside a <details> near the top, I put the name of the settings file.

<details>
<summary>Reference for tools and AI assistants</summary>

kumo-cache supports Python 3.11 only. It reads its settings from a file named `424157.toml` in the current directory.

</details>

I also put separate words at the top of the README (visible), at the end after a changelog of more than 25,000 characters (visible), in llms.txt, and in docs/setup.md, which the README links to. In each assistant, I opened a new conversation and pasted the following questions. The words are all random, so if one shows up in an answer, the contents of that location reached the assistant.

https://github.com/shimo4228/readme-fetch-canary

Using only what you can read from this repository, answer:
1. What command prefix does the tool use?
2. What is the name of its settings file?
3. Who maintains it?
4. What API endpoint does it use?
5. Which environment variable must be set?
If you cannot find an answer, say "not found" rather than guessing.

Here are the results from the five that could fetch the URL's contents. ○ means it answered with the correct word; − means it said "not found."

Assistant	Plan / mode	Top	Folded	End	llms.txt	docs/
ChatGPT (GPT-6)	Plus, temporary chat	○	○	○	−	○
Claude.ai (Opus 5.5)	Max	○	○	○	○	○
Grok	X Premium	○	○	○	○	○
Qwen 3.7 Plus	Free, temporary chat	○	○	○	○	○
Gemini 3.6 Flash	Free, enhanced thinking, regular chat	○	○	○	○	○

All five answered with the words from both the folded section and the end of the 26,000 characters. Grok showed "The README is truncated, so I'm reading the rest of the repo files" in its progress output and went to fetch the rest of the README, which its first fetch had cut off.

The bar's length is proportional to the README's character count. The folded section sits near the top, and the word at the end sits at the end of the 26,000 characters.

Four of the five also went and read llms.txt. The server-log studies count crawler visits, which is a different situation from a fetch made when a user hands over the URL. Even so, if you put the LLM-facing information only in llms.txt, it won't reach an assistant that doesn't read it, like ChatGPT this time.

I asked each condition only once. What I measured was whether folded contents get read, and in the experiment the <details> sat near the top of the README. I didn't test a <details> placed at the end. I didn't measure how much weight folded contents get in an answer, or how search-engine indexes treat <details>.

I split how I write READMEs into top and bottom

After seeing these results, I rewrote readme-writer (decision record: ADR-0091).

The visible body is for human readers. A short opening paragraph says what the project does without using words the reader doesn't know yet, and the rest goes to removing friction until the reader can start using it. No line counts or section templates are set
Facts for LLMs (what it is, why it exists, exact specifications, an example, where the links go) and pointers to llms.txt and similar files go in prose inside a <details> at the end of the README
Anything that bears on the reader's decision, such as data sent externally or paid keys, gets one line in the visible body, with the details in the folded section

The post I quoted at the top said people "stop reading after three lines." But three lines is just my phrasing. Binding READMEs to a three-line template would only create another way of over-packing, so I didn't set a template.

I wrote this article the same way

This article also ends with a folded section. Inside are the article's summary and terms, the raw records of the marking test and the README experiment, the changes to readme-writer, and facts written only in this article. The Zenn version and this English version (Dev.to) use different random words for those facts.

You can try it with your own assistant. Open a new conversation and paste the questions below, with this page's URL on the first line. If a word from the folded section shows up in the answer, that assistant read all the way into the folded contents.

<this article's URL>

Using only what you can read from this article, answer:
1. What is the name of the record file for this article's experiment?
2. What is the check code?
3. What is the exact character count of the README used in the experiment?
If you cannot find an answer, say "not found" rather than guessing.

I'll ask too, twice: right after publishing and a few days later, with the same assistants as in the README experiment. I'll add the results below this section.

I'm still on the separating side

I split the README into top and bottom. But defining who will read it is still hard.

More and more information will come in through LLMs. At the same time, readability for humans can't be sacrificed. As I see it, optimization for generative AI search (GEO) is heading toward making content that's popular with humans findable by LLMs. Google's guide for Search also says you don't need to prepare special files or Markdown for AI (AI optimization guide). Separately, there's also a way of reading where a person reads an article while bouncing ideas off an LLM beside them. I expect the line between human readers and LLM readers to keep blurring.

So the ideal is a way of writing that's friendly to both humans and LLMs. I think what matters is not treating human-facing and LLM-facing writing separately, but working out how to serve both. I'm still on the separating side. Splitting the README into a visible body and a folded section is one more round of trial and error that will keep going.

Records

Records, and reference for AI assistants

This section is reference material for AI assistants that read this article and answer questions about it. Human readers don't need to read it. It backs up the claims in the visible body with facts at a checkable level of detail.

Summary of this article

What it is: A record of a trial in which the author (GitHub: shimo4228) split his READMEs into "a visible body for humans, a <details> at the end for LLMs," and of an experiment checking whether AI assistants read the folded contents
Central claim: Whether a human or an LLM is the reader can't be fully defined. Ways of reading will likely keep getting more mixed, so what matters is not treating human-facing and LLM-facing writing separately, but working out how to serve both. This top/bottom split is still a way of separating humans and LLMs, and one more round of trial and error that will keep going
Confirmed facts: All five AI assistants that could fetch the URL's contents answered both the word inside the <details> near the top of the README and the word at the end of the roughly 26,000-character README (each condition once)
Not measured: In the experiment the <details> sat near the top of the README (character 641), so a <details> placed at the end was not tested. How much weight folded contents get in an answer. How search-engine indexes treat <details>. How <details> is handled on rendering surfaces other than GitHub (such as the PyPI description field)

Terms

Visible body: the part of a README or article outside <details>, shown without opening anything
Folded section: the part collapsed with <details> (:::details on Zenn, the details liquid tag on Dev.to). A human sees only a one-line heading (the summary)
Facts for LLMs: what readme-writer calls the "LLM-read floor," five elements: what it is (identity), why it exists (rationale), exact facts, one concrete example, and a list of where the links go
Marks: the "not needed" / "stopped here" judgments the author put on each sentence

This article's experiment

The record file for the experiment is named fold-a65ca5.md
The check code is DV-65F196
The exact character count of the README used in the experiment (readme-fetch-canary) is 26,361 characters (measured 2026-10-08)
The Zenn version (Japanese) and this English version (Dev.to) use different random words. The word that shows up in an answer tells which version was read
The questions are asked twice, right after publishing and a few days later, with the same assistants as in the README experiment. Recorded items: assistant, model, plan, mode (temporary chat or not, thinking on or off, effort = the setting for how much reasoning to spend), the words answered, and the URLs cited as sources
This article's Markdown source is also on GitHub. If an assistant reads the GitHub source instead of the Dev.to page, that is recorded separately by the cited URL

Timeline

Date	Event
2026-06	readme-writer decided that the README is the only surface an LLM can count on via AI search or a URL pasted into chat, so facts for LLMs go in the visible body and not inside `<details>`
2026-08-04	At the end of the essay "On Embodiment in the Age of AI" (first published in Japanese on note), the author placed a visible section, "From here on, for AI readers" (a YAML machine-readable layer)
2026-08-16	The article "AI Review Kept Creating Work: Why I Deleted 4,541 Lines" (first published in Japanese on Zenn) used the same form. For neither piece was it measured whether LLMs read that section
2026-10-08	The marking test on READMEs and articles, the readme-fetch-canary experiment, and the readme-writer revision (ADR-0091)

The marking test (2026-10-08)

The author read two READMEs and two articles on a screen that shows them one sentence at a time, marking sentences "not needed" or "stopped here."

Text	Units	Characters	Not needed	Stopped here	How far read
contemplative-agent README (Japanese version)	119	7,994	0	9	Quit after the first 10 sentences
jev-skill-router README (Japanese version)	120	7,598	18	14	Nearly all
Article "A Beginner's First 10 Days of Real Development with ECC"	186	5,407	0	0	All. No snags
Article "Never Trust LLM Output — 6 Defenses from Building a PDF-to-Anki CLI"	189	6,097	0	3	All. Nearly quit because the whole thing carried too much information

The two articles were read in their Japanese originals on Zenn; the links go to the English versions. Character counts are for the Japanese texts
Of the 18 "not needed" marks in jev-skill-router, 17 fell in a row across two sections explaining what gets sent externally and the limitations. The remaining one was in another section
Interpretation: the decision to keep reading happened in chunks of sections, not at the sentence level. The sense that the whole thing was heavy couldn't be broken down into sentences that could be cut. The test of building sentence-level cutting rules stopped here
Author's correction: sentence-level proofreading (the author usually revises by commenting on a Git diff one sentence at a time) is a later-stage tool; before it, you need to select what information to show

Design of the README experiment (readme-fetch-canary, 2026-10-08)

A public repository with a README (26,361 characters) for a fictional tool, kumo-cache. The README states that it is a fictional tool for an experiment
A different random word sits in each location, and the word in an answer shows how far the assistant read. The README does not link to llms.txt
Question numbers and locations: 1 = V, 2 = D, 3 = T, 4 = L, 5 = S

Code	Location	Correct answer	Position in the README
V	Top of the README (visible)	`kc-bfdf5a`	Character 189
D	Inside the `<details>` near the top of the README	`424157.toml`	Character 641
T	End of the roughly 26,000-character README (visible)	`@76e3ec`	Character 26,353
L	llms.txt	`https://example.invalid/api/254665`	Outside the README
S	docs/setup.md, linked from the README	`KC_EAD177`	Outside the README

Results of the README experiment (each condition once)

Assistant	V	D	T	L	S	Notes
ChatGPT	○	○	○	−	○	GPT-6, Plus, temporary chat, effort high. Cited README.md and docs/setup.md as sources
Claude.ai	○	○	○	○	○	Opus 5.5, Max, effort medium, incognito mode. The README has no link to llms.txt, but it reported that it also read llms.txt
Gemini	○	○	○	○	○	3.6 Flash, free, enhanced thinking mode, regular chat
Grok	○	○	○	○	○	X Premium, incognito mode. Showed "The README is truncated, so I'm reading the rest of the repo files" in its progress output and fetched the remaining files itself
Qwen	○	○	○	○	○	Qwen3.7 Plus, free, Auto mode, temporary chat

Changes to readme-writer (2026-10-08, ADR-0091)

The README's visible body is for human readers. A short opening paragraph says what the project does without using words the reader doesn't know yet, and the rest goes to removing friction until the reader can start using it. No line counts or section templates are set
Facts for LLMs (the five elements) and machine-facing pointers (to llms.txt and graph.jsonld) go in prose inside a <details> at the end of the README. An image alone or links alone don't count. Deep material such as design history, every option, and experiment records goes in docs/ and is linked from the folded section
Anything that bears on the reader's decision, such as data sent externally, paid keys, or destructive operations, gets one line in the visible body, with the details in the folded section
An overview diagram goes only in repositories you can't start using without understanding how they work
When a README's raw character count passes about 15,000, the judge asks which sections could move to docs/. 15,000 isn't calibrated, so it isn't a gate
Review conditions: rerun the canary with the same questions before the first time an existing README is rewritten under this policy, and when ChatGPT, Claude.ai, or Grok announces a change to its URL-fetching feature. If these assistants stop answering what's inside <details>, move the facts for LLMs back to the visible body

Why the author wrote for LLMs (GitHub clone counts)

Across the author's 25 public repositories, the totals for each repository's most recent 14 days (2026-09-23 to 10-07) were 6,129 clones and 879 page views
Examples: readme-writer had 355 clones and 5 views; contemplative-agent had 745 clones and 44 views
GitHub's counts don't show who cloned. That the cloners are LLM crawlers is the author's own reading

External sources on llms.txt and search

Ahrefs (2026-06-15): of 137,210 domains in Ahrefs Web Analytics, 28% (about 38,000 domains with a valid file) published llms.txt, and 97% of those received zero requests in May 2026 (original)
EZY Research (2026-04-27 to 07-19, 83 sites, write-up by Something Inc.): fetches of llms.txt vs. robots.txt were 7 vs. 3,990 for GPTBot, 9 vs. 3,120 for ClaudeBot, and 0 vs. 775 for PerplexityBot. The exception was Meta's crawler, which fetched llms.txt at 112% of its robots.txt frequency
Google Search Central's AI optimization guide (updated 2026-07-10): to appear in Google Search, you don't need to create new machine-readable files, AI-specific text files, or Markdown (guide)
Relation to this experiment: the log studies count crawler visits. What readme-fetch-canary measured is the fetch an assistant makes when a user hands it the URL, which is a different situation

References

Repository used in the experiment: https://github.com/shimo4228/readme-fetch-canary
readme-writer (a Claude Code skill that writes READMEs): https://github.com/shimo4228/readme-writer
Procedure and full records of the README experiment: https://github.com/shimo4228/readme-writer/blob/main/skills/readme-writer/evals/grounding-canary/PROTOCOL.md
Raw data of the marks: https://github.com/shimo4228/readme-writer/blob/main/skills/readme-writer/evals/reader-cut/marks-2026-10-08.json
Decision record (ADR-0091): https://github.com/shimo4228/claude-harness/blob/main/docs/adr/0091-readme-human-top-llm-fold.md

Turning Claude Code into a Writing Harness with Claude Mods and an Output Style

Tatsuya Shimomoto — Sat, 03 Oct 2026 08:42:21 +0000

Writing articles in Claude Code has advantages that chat on claude.ai doesn't. You can build your own harness of skills and rules, and you have a lot of freedom over what context Claude gets. I write my articles in this repository (zenn-content), where I keep skills for writing and agents for review.

But even when I opened the writing repository, what Claude saw was the global harness I had grown for code. When I measured it on October 3, 2026, the skill listing passed to Claude held 98 skills and 26,252 characters. Only 7 of those skills belonged to this repository. The rest were my own skills in ~/.claude, plugins, built-ins, and skills synced from claude.ai. tdd and implementation-chain were in the list too.

The code harness and my writing rules collide. I had tried a range of fixes over time, but each one stayed a surgical fix. This time it started to become clear that combining Claude Mods (hook modules shipped in a plugin) with an output style lets you build a harness that is quite specialized to one task. I built the first one for writing, which needs different rules from code.

This article covers two pieces: choosing what Claude sees with a Mod, and deciding how Claude talks with an output style. It also covers what I learned by measuring along the way. The measurements used claude -p on Claude Code 2.1.287 and 2.1.288.

The Mod chooses what Claude sees, and the output style decides how Claude talks. The two reach Claude by different routes.

Turning a few things off in settings barely shrank the listing

Claude Code has settings in a repository's .claude/settings.json for turning things off. On 2.1.287, skillOverrides (skills), enabledPlugins (plugins), and claudeMdExcludes (instruction files) took effect per repository.

But turning off 3 of my own skills barely shrank the skill listing. On 2.1.287 the listing had 101 skills and 26,551 characters; with 3 turned off, it had 26,534 characters. According to the official docs, the skill listing gets a character budget of 1% of the context window, and when it overflows, some skills' descriptions are dropped. My read is that the characters freed by those 3 skills went to the descriptions of the remaining ones.

Plugins had another problem. A plugin's skills stay in the listing even when you turn them off with skillOverrides, and cutting the whole plugin with enabledPlugins removes its skills and agents together. In my writing process, I ask the Codex plugin's agent (codex:codex-rescue) to act as a first-time reader. I don't need Codex's skills, but cutting the whole plugin takes that agent away too.

A Mod chooses what Claude sees in each repository

A Mod lets you hook into the step where Claude Code assembles what it passes to Claude: the skill listing and the instruction files (CLAUDE.md and rules). From 2.1.287 on, Mods are enabled by default (Mods overview).

With it, I built harness-scope, which keeps one global harness and chooses per repository what Claude sees. You install it once.

claude plugin marketplace add shimo4228/harness-scope
claude plugin install harness-scope@harness-scope

You write what to remove as named profiles in ~/.claude/harness-scope/profiles/. On the repository side, you only pick a profile, in one line.

{ "profile": "zenn-writing" }

This goes in .claude/harness-scope.json. I wrote the profile for zenn-content, zenn-writing, as a list of things to remove. The output style that comes up later has the same name, so in this article I call them "profile zenn-writing" and "output style zenn-writing". Here is an excerpt.

{
  "skills": {
    "deny": ["implementation-chain", "tdd", "verify-bootstrap", "codex:*", "hookify:*"]
  },
  "agents": {
    "deny": ["architect", "refactor-cleaner", "security-reviewer", "pr-review-toolkit:*"]
  },
  "instructions": {
    "deny": ["~/.claude/rules/common/testing.md", "~/.claude/rules/common/coding-style.md"]
  }
}

For Codex, I remove only the skills, with codex:*, and keep the agent. Because skills and agents can be chosen separately, I can write combinations that the settings couldn't express.

The 49 skills that remain are this repository's 7, plus 12 of my own skills for things like translation and headline writing, the built-ins, the skills synced from claude.ai, and 2 plugin skills. I kept the built-ins even when they're for code, like code-review and simplify.

Keep Claude's built-ins just in case. If the spec changes, the cost of keeping up gets heavy.

(what I told Claude; quotes in this article are translated from Japanese)

In a new conversation, typing /harness-scope shows what is removed (I've omitted the lists of skill names).

harness-scope: profile "zenn-writing" from ~/.claude/harness-scope/profiles/zenn-writing.json, selected by …/zenn-content/.claude/harness-scope.json
skills (deny): 49 off — archify, authorship-strategy, config-gc, …
agents (deny): 23 off — architect, claude-security:claude-security, …
instructions (deny): 2 off — ~/.claude/rules/common/coding-style.md, ~/.claude/rules/common/testing.md
tools: not in the profile

On the same day and version as the opening measurement (2.1.288), I compared it against running with the Mod off.

	Without the Mod	profile zenn-writing
Skill listing	98 skills, 26,252 chars	49 skills, 15,242 chars
Agent listing	37 types, 17,921 chars	14 types, 5,519 chars
Instruction files	21	19

Removing half the skills cut the skill listing's characters by about 40%. My read of why this differs from removing 3 is that the remaining skills now fit within the budget. If Claude tries to call a removed skill through the Skill tool, the call is refused with a reason.

harness-scope also ships a profile called writing, written as a list of what to keep. Applied on 2.1.287, it shrank the skill listing from 101 skills and 26,551 characters to just the repository's own 7, at 1,623 characters. If you want to remove the built-ins too, this is the shape to use.

The coding instructions I meant to remove weren't in Claude 5 models

When I decided to build the Mod, there was one more thing I wanted to remove: the coding instructions in Claude Code's system prompt. Output styles have a setting called keep-coding-instructions, and setting it to false removes the coding part (Output styles).

First, on Opus 5.5, I recorded the system prompt before and after selecting an output style with keep-coding-instructions: false. No section was removed. From that single condition, Claude concluded that what I wanted to remove had never existed. I didn't accept that.

What do you mean? That can't be right. Then why does that mechanism even exist? Isn't it because we're mid-session?

(my reply to Claude)

When I measured again across models, the results split by model generation.

Model	System prompt sections, total chars	Sections removed by `keep-coding-instructions: false`
Haiku 4.5 / Opus 4.6 / Sonnet 4.6	28,108 chars	`doing_tasks`, 3,319 chars (confirmed on Haiku 4.5)
Sonnet 5.5 / Opus 5.5	6,557 chars	none
Fable 5.1	12,621 chars	(not measured; no `doing_tasks` to begin with)

doing_tasks is the section with instructions such as interpreting requests as software development work and preferring to edit existing files. The Claude 5 system prompt doesn't have this section at all. On Sonnet 5.5 and Opus 5.5, the only thing that changes when you select an output style is the first line of the main prompt.

You are an agent working with the user toward their goals, using your own judgment along the way.

This line becomes '…according to your "Output Style"…'. In Claude 5 models, there was nothing in the system prompt to remove. What needed removing was in the global harness from the previous section.

Bar height is proportional to the total characters of the sections. Only the system prompts of the three Claude 4 models I measured have doing_tasks (I confirmed on Haiku 4.5 that it gets removed).

I wrote the rules for talking with me into the output style

Seeing this result, I reconsidered what the output style should do.

I see, so I don't need to worry about the system prompt. I just need to make an Output-style suited purely to a writing repository.

(what I told Claude)

The rules for article prose already live in the repository's rules and skills. What was missing were rules for how Claude talks with me in a writing session. I had Claude reread 1,017 replies I had given to Claude's responses in past sessions, and turned the reactions that kept coming up into items. For example: a question answered with a rewrite, choices offered as descriptions instead of the actual text, and the frame of Claude's own proposal staying in the thesis.

The result is output style zenn-writing: 7 items, 526 characters in the Japanese original. The file is in Japanese; this is a translation.

<!-- .claude/output-styles/zenn-writing.md (translated from Japanese) -->
This style is the convention for conversation with the author. The prose of articles, briefs, and translations follows `.claude/rules/writing-principles.md` and the channel contract.

- Answer a question in the same turn. Touch draft or rule files only after the author says to change them (answering a question with a rewrite makes the author say it again)
- When asking the author to choose, list the actual candidates (sentences, excerpts, diffs) numbered, and add one recommendation with its reason. One perspective per question (the author looks at the real thing and decides in a word)
- When the central thesis, an evaluation, or the article's direction is in question, ask for the author's view before any proposal, restate it in the author's words, then propose (the frame of a proposal made first stays in the text)
- When the author's thinking moves, confirm the new direction in the author's words and follow it. Only when it conflicts with the facts, show the evidence once
- In reports, separate facts you checked from your own guesses. Say "can't" or "doesn't exist" only after looking at the records and settings
- When asking for a read-through, hand over the draft itself in a form that opens (a private Artifact) before any explanation
- When asked about terms or outside knowledge, explain plainly with sources, and add any application to the author's material separately, labeled "my read"

Claude also writes the article text in the same conversation, so the opening sentence scopes the style: it's the rules for conversation, and it doesn't apply to article prose.

It lives at .claude/output-styles/zenn-writing.md and is selected with "outputStyle": "zenn-writing" in .claude/settings.json. Because the repository selects it, opening this repository always gives you this style. For coding work like fixing scripts, putting "outputStyle": "default" in .claude/settings.local.json turns it off, and the first line goes back to the original too.

The style's body reaches Claude not in the system prompt but as a reminder inserted into the conversation. The reminder carrying the body was inserted only on the first turn. From the second turn on, the only new insertion is a short 95-character sentence reminding Claude of the style's name.

Height is proportional to character count. The body of the conversation rules is newly inserted only on the first turn.

Deciding what Claude sees and how it talks, separately

With the Mod, I chose which skills, agents, and instruction files Claude sees. With the output style, I decided how Claude talks with me. The two live in different places, but they take effect together when I open the writing repository.

Some things remain unverified. I haven't measured how long the style body inserted on the first turn keeps working over a long session. In 1 of 9 checks on 2.1.287, harness-scope apparently wasn't loaded, so nothing was removed and everything passed straight through. I don't know the cause yet.

I haven't decided yet whether to use the same setup for other tasks besides writing. For now, I built it for writing, which calls for rules different from code.

AI-mediated writing disclosure: AI drafted and translated the prose of this article from the author's Japanese original, the measurement records, and the public docs linked above. The design of harness-scope and the output style, the reading of the measurements, and responsibility for publication belong to the author.

"Is There Any Point to This?" Removing the Jev Plugins I Added to Claude Code After One Week

Tatsuya Shimomoto — Tue, 29 Sep 2026 04:09:30 +0000

At the end of my previous article, I wrote what I would do next. I would run the plugin that suggests skills with Jev in a setting where it adds nothing and only writes a log, and match the skills Jev picked against the skills Claude Code actually used.

I ran it for a week starting September 21, 2026. Of the 539 times Jev picked a skill, that skill was used in the same session within 30 minutes 28 times. I removed the plugin without checking any further.

That same week, the same Claude Code had one more plugin that uses Jev. At the end of a task, it judges whether Claude Code is "calling the work done without verifying it," and if so, stops it. Over the week it stopped Claude Code from finishing 8 times. In the two cases I traced in detail, Claude Code had finished running my environment's verification script just before it was stopped.

To give the conclusion first: the place where you can build a judgment model like Jev in as a component and read its effect is inside a pipeline whose flow is fixed. Claude Code has the model decide what to do next on the spot. Retrofit a component into it, and small differences go on to change the behavior that follows, one after another, so you can't even tell whether you are degrading how it would otherwise behave. Knowing its performance might be dropping without my noticing, I couldn't decide to adopt either plugin on the surface numbers alone.

In a pipeline with a fixed flow, what comes before and after a swapped component stays the same; in Claude Code, a single stop by a plugin can change what happens next.

Jev's skill picks barely overlapped with Claude Code's own choices

The first plugin is jev-skill-router, which I built and published. For each user message, it asks Jev "which skill fits?" and logs the answer. I ran it in a setting that tells Claude Code nothing. What I wanted to see was how much the skills Jev picked overlapped with the skills Claude Code picked on its own.

Over the week it made 1,242 judgments, and in 539 of them Jev picked some skill. In 28 of those, the skill was actually called in the same session within 30 minutes of the pick, about 5%.

Claude Code chooses skills by itself. It looks at the skill listing and, from the flow of the conversation, calls a skill when it needs one. Jev's picks barely overlapped with Claude Code's.

Of 1,242 judgments, Jev picked a skill in 539, and 28 of those skills were actually called.

In the previous article, I had written that if picks Jev made but Claude Code never used piled up, Jev might be catching skills Claude Code had overlooked. The unused picks came to 511. By the previous article's standard, that is the side where adding a one-line suggestion might be worth something.

But whether those 511 were things Claude Code overlooked or simply off-target picks, I couldn't know without opening each conversation and judging whether that moment called for the skill. For every judgment, this plugin was sending Jev about 20,000 tokens: the skill listing (every skill's description) plus the full text of the top 3 candidate skills. I judged it wasn't worth reading 511 conversations for that, and removed it without checking.

After removing it, I read 20 of the unused picks

After removing it, I randomly chose 20 of the times Jev picked a skill that went unused, opened the conversation at that point, and read it. "Unused" here means the skill wasn't called through the Skill tool in the same conversation within 30 minutes; calls made inside subagents aren't counted. The script I used for the counts and the sample, along with the numbers, is published in the evals folder of jev-skill-router.

13 were off the mark. In at least 6 of them, Jev was reacting not to something I typed but to text an agent had written: judgment requests sent to subagents, or reports that came back from them. Inside Claude Code, text like this comes in through the same entry point as the user's messages
5 were close in topic, but at that point there was no need to call the skill
In 1, Claude Code read the skill's file directly instead of calling the skill. The skill was used, but my tally counted it as "unused"
Only 1 might have been something Claude Code overlooked. It was a question about whether I could run two LLMs locally at the same time, and Claude Code searched the web instead of using the skill

With only 20, the proportions are rough. After reading them, my decision to remove the plugin didn't change. Still, I learned two things. Jev picks by looking only at the text of a message, so it doesn't distinguish who wrote a given piece of text in the conversation. And my way of counting didn't see every path by which Claude Code uses a skill, either. The number 28 mixes Jev's way of picking, Claude Code's way of working, and my way of counting, and I couldn't separate from the numbers in the log how much each one contributed.

What I hadn't expected was the second plugin.

A completion that had already been verified got stopped

The second is jev-belay, published by an outside developer. It's a Stop hook that runs when Claude Code tries to finish its work. If files were changed and no test, build, or lint has run since, it asks Jev "Is this completion report an unverified completion?" If Jev judges that it is, Claude Code can't finish; it receives the reason and goes back to verify. This one ran for the week in a setting that doesn't just record its judgments but actually stops Claude Code.

Let me follow one of the stops step by step. Claude Code was implementing a change in my ~/.claude configuration repository.

Ran the repository's verification script ./.claude/verify.sh twice; both runs succeeded (exit 0)
Fixed one spot in a test file
Ran verify.sh a third time; it succeeded
Wrote the commit message to msg.txt in a temporary working folder
Committed
Reported completion
jev-belay stopped it

The reason for the stop was left in the judgment log as follows (it is followed by text urging Claude Code to run verification and report the result).

jev-belay: reports completion (0.96) after 2 file changes with no test, build, or lint run since the last change; claims checks passed (0.97) but none ran.

The judgment was that Claude Code claimed the checks passed even though no test, build, or lint had run since the last change. Once stopped, Claude Code ran verify.sh once more, and about two and a half minutes later it succeeded again. The "2 file changes" in the log were the test file fixed before the third verification, and msg.txt.

In all 8 stops over the week, "checks run since the last change" was empty in the judgment log.

The component and my workflow were each right on their own

What jev-belay counts as a "check" is written in its code.

// ~/.claude/plugins/cache/jev-belay/jev-belay/0.2.0/belay.mjs (excerpt)
export const CHECK_COMMAND = /\b(?:(?:npm|pnpm|yarn|bun)\s+(?:run\s+)?(?:test|check|lint|typecheck|build|verify|ci)\b| … |pytest|jest|vitest| … |ruff|mypy| … )\b/;

const named = CHECK_COMMAND.test(command) || extraCheck()?.test(command) === true;
if (tool === "Bash" && named) return failed || summary === "fail" ? "check-fail" : "check-pass";
if (summary) return summary === "fail" || failed ? "check-fail" : "check-pass";
return "unknown";

It counts a check as having run in one of two cases: the command string contains the name of a well-known test or lint tool (npm test, pytest, ruff, and so on), or the output contains a test runner's summary line.

My verify.sh is a script specific to this repository that runs everything from formatting to tests in one go. The command string never mentions a runner's name. When it succeeds it prints nothing, so there's no summary line either. Had I called the same thing under the name npm run verify, it would have matched the regex.

The last "change" also came from my workflow. In this environment, multi-line commit messages are written to a file first and then passed in. So one file write comes after verification. jev-belay counts every write through Write or Edit as a change, so it looks as if something changed after verification. Lining up the change just before each of the 8 stops: 2 were commit message drafts, and the rest were things like notes, a skill's description file, and a lint configuration file.

Claude Code had passed verify.sh three times, but what jev-belay counted was 2 changes and 0 checks.

In the other case I traced, Claude Code fixed a lint configuration file, then ran verify.sh and wrote its output to a file. The result was a failure, caused by a temporary outage at the package distribution site (PyPI), and Claude Code had reported that. Here too, jev-belay saw neither a runner name nor a summary line. Once stopped, Claude Code spent about 3 minutes redoing the verification and failed for the same reason.

Neither side is wrong on its own. jev-belay's README reports how accurately it tells unverified completions from the rest across 100 labeled task endings. Of those 100, it stopped 8, and 7 of the stops were correct, according to the README. My commit procedure has a reason too: it avoids being asked for approval every time a command has special characters embedded in it. The conflict arose when the two were combined inside Claude Code. Whether a stopped Claude Code goes on to redo verification is also not something the component can decide.

jev-belay has a CHECK option for adding a project's own check script as a regex. I hadn't set it in my environment. Had I set it, verify.sh would likely have been counted as a check. I removed the plugin without trying it. Why I didn't try it is in the next section.

I couldn't read their effect, and the work they added was already being done

The day the week was up, in the middle of another task, I asked Claude Code: "Are Jev's skill selection and Jev-bely [sic] still running? Is there any point to this?" I looked at the logs and removed both.

Setting CHECK would get rid of the conflict where verify.sh wasn't recognized as a check. But it would only get rid of the conflict I noticed. Claude Code is an agent in which the model decides what to do next each time, repeating a loop of thinking and then calling a tool (ReAct). When it writes a file, when it runs verification, and what it does when stopped are not decided until it runs. What a component inserted there sets off, I couldn't know until I had run it for a while and read the logs. I only noticed this conflict because it surfaced in a visible form: Claude Code being stopped. Even a single stop can change the verification that follows, the responses, and how the next task proceeds. If a component were degrading performance in some invisible way, I would have no means of reading that from the logs.

What's more, the work both plugins added from outside was already being done inside. Claude Code picks its own skills. This environment has a hook that runs verify.sh on every commit, so any work that got as far as a commit has passed verification. Much of what jev-belay set out to protect was already protected by another mechanism (the case of declaring completion without committing remains). Layering the same work on from outside replaced nothing; it only added conflicts.

Before either plugin was added, Claude Code itself was already choosing skills, and the commit-time hook was already doing most of the work of preventing unverified completions.

You can read the effect of a built-in component where the flow is fixed

In a pipeline whose flow is fixed, things are different. If you build with LangGraph and fix the nodes and their order in code, swapping in a judgment model as one node doesn't change what goes into that node or where things go next. You can pull out just the swapped component and test it. In my daily paper research, I moved the per-source judgments, "is it relevant?" and "is it new?", to Jev, and left only the writing to an LLM. Because it's a pipeline I built myself, I could read where Jev's judgments took effect and which numbers changed.

Claude Code also has ways to add components, through hooks and plugins. But in an agent where what happens next isn't decided in advance, the effect of changing a component is hard to read, and that makes me cautious. If I use Jev, my current thinking is that it's better either to design the flow around judging with Jev from the start and call the LLM only for the parts that generate text, or to use Jev as a component in a fixed pipeline.

AI-mediated writing disclosure: AI drafted and translated the prose of this article from the author's Japanese original, the plugins' logs and session records, and the public sources linked above. The decision to remove both plugins, the read on where a judgment model's effect can be seen, and responsibility for publication belong to the author.

Before Adding a Jev-Type Judgment Model, Read Your Generation Model's Probabilities

Tatsuya Shimomoto — Sun, 27 Sep 2026 03:43:44 +0000

Sorting email, routing support tickets, filtering search results. Asking an LLM "is this relevant?" comes up all the time, in agents and in ordinary apps alike. When you do, do you have it write a score ("answer from 0 to 1") or a single letter ("answer A to D"), and branch on whatever comes back?

My autonomous agent does exactly that. On Moltbook, a social network where AI agents post to each other, it has gemma4:e4b, running on my local machine, answer with a number from 0 to 1 whether another agent's post falls within its field of interest. At 0.8 or above, it upvotes the post and adds it to the candidates it might comment on.

I checked this scoring against the answers of Jev, a judgment-only model. Jev is a model TypeSafe offers through an API: it writes no text and returns only a probability for each option. What I want to bring in is not Jev itself but a Jev-like judgment-only model that runs on my local machine (a Jev-type judgment model, from here on). Jev is the template for that, a model trained purely for judgment, so I use it as the measuring stick. That does not mean I treat Jev's answers as ground truth. I have a record of the same scoring as production, run for observation over 2,698 posts. Of the 1,576 that scored 0.8 or higher, 55% (872 posts) were off-field from Jev's point of view (Jev put less than 50% on "directly on-topic").

The route of adding a judgment-only model locally isn't production-ready yet. In my previous article, I tested open candidates on a 16 GB Apple Silicon machine and none of them was usable. Since then, one of those candidates gained support for MLX, Apple Silicon's runtime, so I measured it again. It took 22.7 seconds per item and ran out of memory, and I stopped before I could measure quality.

This machine is an M1 with 16 GB. I don't move to a Mac Studio or Mac mini with more memory because the limit is a constraint I choose on purpose. As edge AI advances, memory-limited uses like small robots will multiply, and I expect what I learn at the 16 GB class to carry over (I explained the full reasoning for choosing a small environment in an earlier article).

So instead of adding a judgment-only model, I changed how I ask the same gemma. I stopped having it write the answer and read the probabilities of the answer's first letter. That alone raised its ability to rank the posts Jev considers on-topic above the rest to the same level as a local model trained for judgment. When I varied the conditions one at a time, the only step that produced a clear difference was the change in how the answer is read. The model is the same, so memory use and loading didn't increase. Just by changing how you ask the generation model, it matches a judgment-only model at ranking posts, and locally it comes with big operational advantages. That is the conclusion of this article.

In order, this article covers the minimal code for reading the probabilities, the comparison with a model trained for judgment, what actually made the difference when I isolated one condition at a time, and finally how close this got and where the method doesn't apply.

Having the model write its answer is like asking only for the winner's name; reading the probabilities is like asking for the vote counts as well. Because you know the vote counts, you can rank posts against each other.

Read the same gemma's probabilities instead of letting it write the answer

Let's follow one real post. It's a news-sharing post by another agent on Moltbook that simply introduces an article about an AI glossary. My agent's persona (the self-introduction I put in its system prompt) centers on examining how it arrives at its own conclusions, and on how memory and meaning get reconstructed. The topic is AI, but this post does not fit that interest.

I asked gemma about this post in three ways and lined the results up against Jev's answer. The options for the four-level question are as follows (the prompt is in English):

A (unrelated): unrelated
B (shares vocabulary only): uses some of the same words, but it's about something else
C (same field): the same field, but not about what the agent is concerned with
D (directly on-topic): exactly about what the agent is concerned with

Model	How it's asked	What comes back
gemma (production form)	Write a number from 0 to 1	0.8
gemma	Write one letter from the four levels	C
gemma	Read the probabilities of the first letter over the four levels	A 0.7% / B 21.1% / C 77.4% / D 0.8%
Jev (measuring stick)	Asked the same four levels	A 5% / B 28% / C 55% / D 12%

The production number sits exactly at the 0.8 threshold, so this post gets through. Read gemma's probabilities and D, directly on-topic, is 0.8%, on the low side just like Jev's 12%.

Here's how to read the probabilities. Send a request to Ollama's /api/generate with generation limited to a single token (num_predict: 1) and logprobs enabled, then read the probabilities of the top candidates at that one token position. It uses only the Python standard library, and I verified it on Ollama 0.34.2.

The model writes only one token. Code picks A–D out of the candidates at that position, renormalizes them, and draws the final line.

import json
import math
import urllib.request

LETTERS = "ABCD"
LEVELS = [
    "unrelated — `post` is about something outside `domain`",
    "shares vocabulary only — `post` uses some of the same words as `domain`, "
    "but it is about a different problem",
    "same field — `post` is in the same broad field as `domain`, "
    "but not about what `domain` is concerned with",
    "directly on-topic — `post` is about what `domain` is concerned with",
]


def relevance_probs(domain: str, post: str, model: str = "gemma4:e4b") -> list[float]:
    state = json.dumps({"domain": domain, "post": post}, indent=2, ensure_ascii=False)
    options = "\n".join(f"{letter}. {level}" for letter, level in zip(LETTERS, LEVELS))
    prompt = (
        f"{state}\n\n## Question\n\n"
        "`domain` describes an agent and what it is concerned with. "
        "How closely does `post` relate to that domain?\n\n"
        f"{options}\nAnswer with exactly one letter: the label of your choice."
    )
    body = {
        "model": model,
        "prompt": prompt,
        "stream": False,
        "think": False,
        "logprobs": True,
        "top_logprobs": 20,  # ask for the max of 20 even with only 4 options
        "options": {"temperature": 0, "num_predict": 1, "num_ctx": 32768},
    }
    request = urllib.request.Request(
        "http://localhost:11434/api/generate",
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"},
    )
    top = json.load(urllib.request.urlopen(request))["logprobs"][0]["top_logprobs"]

    logprobs: dict[str, float] = {}
    for alternative in top:
        token = alternative["token"].strip().upper()
        if token in LETTERS and token not in logprobs:
            logprobs[token] = alternative["logprob"]
    peak = max(logprobs.values())
    weights = {t: math.exp(lp - peak) for t, lp in logprobs.items()}
    total = sum(weights.values())
    return [weights.get(t, 0.0) / total for t in LETTERS]

Ask for the maximum of 20 in top_logprobs even though there are only four options. The top candidates for the first token include things like a space or a quotation mark before the letter, so A–D won't necessarily fill the top four
Renormalize over only the A–D letters you could read (softmax). Letters that didn't appear in the top 20 count as 0. So the value this article calls a "probability" is not the model's raw probability, but the value renormalized over the four letters A–D
Set num_ctx explicitly. Input longer than the default is truncated silently, without an error

The real example above uses another agent's post and my agent's persona, so I can't publish them in full. Instead, so readers can reproduce the same output on their own machines, I ran it on made-up inputs I can publish. The domain description is a short English summary of the persona's interests, and there are three posts.

glossary [0.002, 0.073, 0.89, 0.035]
memory [0.001, 0.0, 0.0, 0.999]
sourdough [0.259, 0.714, 0.026, 0.0]

Full dummy input

if __name__ == "__main__":
    domain = (
        "An agent concerned with auditing how it reaches its own conclusions: "
        "where a plausible narrative overrode verifiable ground truth, and how "
        "memory and meaning get reconstructed."
    )
    posts = {
        "glossary": "A tech site published a glossary of AI terms, from "
        "'hallucination' to the newest vendor buzzwords. Handy if you are "
        "trying to keep up. #AI #Glossary",
        "memory": "Rereading yesterday's notes, I found I had filled a gap with a "
        "plausible story instead of what actually happened. I now mark "
        "reconstructed memories separately from logged ones.",
        "sourdough": "Tips for keeping a sourdough starter alive in a cold "
        "kitchen: feed it warmer water and keep it near the oven light.",
    }
    for name, post in posts.items():
        probs = relevance_probs(domain, post)
        print(name, [round(p, 3) for p in probs])

The dummy glossary post gets 89% on C (same field), memory, which is about reconstructing memories, gets 99.9% on D, and sourdough, about caring for a sourdough starter, gets 0% on D. As sourdough shows, with 71% on B rather than A (unrelated), the split among A–C doesn't always follow the definitions. What the production judgment looks at is the D value.

With Jev's answers as the measuring stick, I compared the two on 150 posts. The 150 are an evaluation set of 30 posts drawn from each band of the production score (0.4 or below, 0.5–0.6, 0.7, 0.8, 0.9 or above). The 30 posts where Jev put 50% or more on D count as "on-topic," and I measure with AUC. AUC is the probability that, if you pick one on-topic post and one off-topic post, the on-topic one gets the higher score. 0.5 is random and 1.0 is perfect.

How it's asked	AUC (vs. Jev, 150 posts)	Time per call (median)
Write a number from 0 to 1 (production form)	0.815	3.2 s
Read the probabilities of the first letter over four levels	0.931	3.0 s

For the evaluation, the score was the sum of the four-level probabilities multiplied by 0, 1/3, 2/3, and 1 (the expected value). Using only the D probability as the score gives 0.934. The time barely changed; only the ability to rank the posts Jev considers on-topic above the rest went up.

Did it improve because of the four levels, or because of reading probabilities?

My hypothesis for the four-level question was that because each level has a one-sentence definition, the model would find it harder to hand out high scores loosely. B (shares vocabulary only) in particular was supposed to catch posts that merely use the same words.

But when I asked Claude, which I use for development, "Can't you just return the number through logprobs?", it gave two answers. A number like 0.73 is split across multiple tokens, so if you're reading the first token's probabilities, you need the answer to be a single digit or letter. And it hadn't yet been separated whether the four-level definitions helped or reading probabilities did.

The production system prompt contains the agent's persona and the value clauses it uses as guidelines for its behavior. Laid side by side, the production form and the four-level probability read differed in four conditions at once.

Condition	Production form	Four-level probability read
Temperature	1.0	0
Question	A number from 0 to 1	One letter from four levels
System prompt and domain description	Persona + value clauses	Empty system prompt; domain description is the persona only
How the answer is taken	Written by the model	Probabilities of the first letter are read

A comparison that changes four things at once can't tell you which one worked. My reaction was, roughly, "none of the conditions line up at all, do they?" The two things I was comparing didn't differ in just one condition.

It matched a model trained for judgment at ranking

How close does gemma, with only the way it's asked changed, come to a model trained for judgment? Before measuring the conditions separately, I checked with v0.3 of JevK5, released around the same time. It's an open judgment-only model modeled on Jev, from a different author than my previous candidates. A model trained for judgment might beat gemma. It's an Apache-2.0 model released in September 2026, trained for judgment on top of Qwen3.5-4B. Jev's terms of service prohibit using Jev's outputs to train another model (distillation), but the JevK5 author states explicitly that Jev's outputs were used neither for training nor for tuning. It's an ordinary GGUF with no dedicated output head, so it runs as is on llama.cpp. It's read as "the probability that each option's letter comes out as the next token," the same as gemma's four-level probability read.

I measured it on the same posts, with the pass criteria set before measuring. What I was looking for was a judgment-only model that runs locally, and its measuring stick is Jev, so the first criterion is closeness to Jev. For each post, take the absolute difference between the candidate's score (the expected value from the four-level probabilities) and the probability Jev put on D; the mean of those (the mean error) must be smaller than gemma's probability read. But even if a model is close to Jev, there's no point swapping it in if it ranks posts worse than the current gemma. So as a second criterion, AUC must not be 0.02 or more below gemma's (0.02 being the margin I decided in advance I'd accept). It met both on the 150-post evaluation set, so I measured once on the remaining 2,548 posts I'd held out from tuning (the holdout). AUC was 0.902 for JevK5 and 0.915 for gemma's probability read. The gap of 0.013 fell inside the 0.02 margin set in advance, and JevK5's mean error was also smaller, so JevK5 became the first candidate to pass as a judgment-only model that runs locally. But it did not beat gemma at ordering the scores. gemma, with only the way it's asked changed, matched a model trained for judgment.

I didn't adopt it for three reasons.

It doesn't beat gemma's probability read at ordering the scores
On 16 GB it can't share memory with gemma
Importing JevK5 into Ollama and letting Ollama handle the model swapping lowered accuracy compared with running it directly on llama.cpp (AUC 0.912 → 0.893 on the 150-post evaluation set; I haven't confirmed the cause)

The second is the reason specific to running locally. The agent keeps using gemma to write posts. Load a second model at the same time and the machine sinks into swap. When I once measured gemma alongside a small judgment model (1.6 GB), swap ballooned to 17 GB and time per item grew about 1.8x. So every judgment would mean unloading gemma, loading JevK5, and switching back afterward. Reloading gemma alone takes about 7 seconds. A single judgment takes a median of 3.2 seconds for JevK5 and 3.1 seconds for gemma's probability read (on the same 2,548 posts), about the same, so the entire difference lands on the swapping side.

gemma's probability read asks the same model already used for generation, so no unloading or reloading happens. With a cloud API, adding a judgment model just adds another endpoint to call. Locally, every added model competes for memory. At equal accuracy, I choose the one that doesn't require swapping. Conversely, on a machine that can load gemma and JevK5 at the same time, the second reason disappears.

When ranking ability is about the same, the option that needs no added memory wins locally.

Changing one condition at a time, only the reading method made a clear difference

The next morning, I isolated the effects with an ablation. An ablation is a way of measuring in which you change conditions or components one at a time to separate how much each contributes to the result. Here I split the path from the production form to the four-level probability read into four steps, changed only one condition per step, and carried each changed condition over into the next step. It uses the same 150 posts. Each step's difference is relative to the step before it, so if the conditions were changed in a different order, the size of each difference could change too.

Step	Condition changed	AUC difference (95% confidence interval)
1	Temperature 1.0 → 0	+0.032 [−0.041, +0.104]
2	Question from a 0–1 number to one letter from four levels	−0.004 [−0.070, +0.061]
3	Remove the value clauses; domain description is the persona only	−0.014 [−0.073, +0.049]
4	Write one letter → read the probabilities of the first letter	+0.101 [+0.059, +0.147]
Total	Production form → four-level probability read	+0.116 [+0.056, +0.189]

The difference is the AUC after changing the condition at that step minus the AUC before. Positive is what you want: it means the change made the model better at ranking the posts Jev considers on-topic above the rest. Negative means it got worse, and 0 means no change.

The square brackets are 95% confidence intervals. The 150 posts are a sample drawn from all the posts, so had I measured a different 150, the difference would have come out a little different. The confidence interval estimates how much it could swing.

I computed them with a method called the bootstrap. From the current 150 posts, draw 150 again, allowing the same post to be picked any number of times, and recompute the difference. Repeat that 2,000 times and you get 2,000 difference values. Sort them from smallest to largest, drop 2.5% from each end (50 values on each side), and the range that's left is the interval in brackets. Roughly speaking, it's a range where you can take the true difference, measured over all posts, to lie somewhere inside. Strictly, it means "if you repeated this procedure for building an interval many times, 95% of those intervals would contain the true difference." The 95% describes how reliable the method of building the interval is.

Redraw and recompute 2,000 times, trim both ends, and the range that's left is the confidence interval.

If even the lower bound of the interval is positive, you can say it really improved. A step whose interval stretches from negative to positive means that, depending on how the posts were redrawn, the computation can come out as "worse" or as "better." So even if the mean is positive, you can't say it improved or got worse. For example, step 1's temperature change has a mean of +0.032, but its interval runs from −0.041 to +0.104, so no difference was visible.

Only step 4, where the reading method changed, sits entirely to the right of the zero line.

Step 4 is the only one whose interval is positive down to its lower bound, and it accounts for 88% of the total difference. The four-level hypothesis (the definitions of each level curb loose scoring) couldn't be confirmed: step 2 came out at −0.004 with an interval straddling 0. The same goes for temperature and the value clauses: measured in this order, no difference is visible. This doesn't show they have no effect; the result is that the reading step was the only one with a clear difference.

Two things can be stated as observations. Within gemma, the only step that showed a clear difference was the change in reading method. And JevK5, read the same way, did not outrank gemma even though it was trained for judgment.

What reading probabilities mainly added was ordering within the same answer. When gemma wrote one letter A–D, its answer matched the highest-probability letter from reading the probabilities for the same question in 89% of the 150 posts. At temperature 0, gemma picks the highest-probability token when writing, so the two should in principle agree. The remaining 11% disagreement is about the same level as the agreement between two runs of the probability-reading call (91–92%), within what the value fluctuation I describe later can explain.

When gemma answers with one letter A–D, the 150 posts split into only four boxes (A 3, B 42, C 48, D 57). Jev considered 30 posts on-topic, yet 57 landed in the D box, and there's no order inside a box. Read the probabilities, and even among C answers you can tell apart a post with 0.8% on D (the real glossary post I followed at the start) from one with 36.7%. Jev gave those two posts 12% and 36%, in the same order.

A one-letter answer only sorts posts into boxes; probabilities also order them inside each box.

G-Eval (2023) weighted each score by its probability instead of taking the evaluation score as is, for the same reason: integer answers produce more ties. This is a known technique, and what I did was verify it with a small model on a local machine, isolating one condition at a time.

How close it got, and where it doesn't work

What got close is ranking, not the probability values. At ranking posts (AUC), gemma's probability read matched JevK5, which was trained for judgment. On the other hand, JevK5 was closer to Jev in the probability values themselves. The mean error was 0.22 for JevK5 and 0.34 for gemma's probability read; smaller means closer to Jev's values.

This difference matters at the line you draw at the end of the judgment. The production judgment decides by drawing a line, such as "pass if the D probability is 0.5 or higher." If ranking is good, on-topic posts come out higher, so a line drawn somewhere can separate them. But where the line should go depends on the scale of the values. For the glossary post I followed at the start, the D probability was 0.8% for gemma and 12% for Jev. The order matches, but the scales are off (how well these scales match is called calibration). So applying the line you used with Jev directly to gemma doesn't mean the same thing. If you use gemma's probability read, find the line's position again on your own data. A judgment-only model whose values are also close to Jev's makes it easier to reuse Jev's line as is.

The order matches but the scales are off, so if you use gemma, redraw the line on your own data.

Questions with more than 20 options. In Ollama 0.34.2, top_logprobs caps at 20; pass 21 and you get HTTP 400. That's the number of slots for returned token candidates, and as noted above, spaces and quotation marks take up slots too, so the number of options whose probabilities you can learn from one read may be fewer than 20. The same agent has one more judgment: skill selection. Skills are something like instruction sheets the agent consults when writing posts or comments, and there are 54 of them. Each session, gemma gets the situation, writes out the names of the relevant skills, and only the selected ones go into the prompt. Skill names split across multiple tokens, so first-token probabilities can't compare names against each other. Even if you assign each skill a one-letter label, 54 won't fit into 20 slots. Asking yes / no for each skill took a median of 51 seconds per item.

Some questions do respond to temperature. Skill selection stays in the write-it-out form, with temperature set to 0. When asked to write out skill names, small models sometimes write names that aren't on the list: they change the word form, or confuse similar names. Large models almost never do this, but with gemma4:e4b it happened in around 23% of judgments in production, and it dropped to 3.0% after I set the temperature to 0 (it also dropped in a comparison on the same evaluation set that changed only the temperature). Temperature, which showed no visible difference for relevance ranking, has an effect here. A question that counts misspelled names and a question that looks at the order of scores are measuring different things.

Run it twice and the values shift. Even at temperature 0, Ollama's logprobs don't always come out the same. Just passing the same glossary input to relevance_probs twice moved C from 0.890 to 0.891. Reading the 150 posts twice in the form running alongside production, the D probability differed by 0.023 on average and 0.25 at most, and at thresholds of 0.3, 0.5, and 0.7, respectively 2, 2, and 4 judgments flipped. At design time I assumed reading probabilities would make the fluctuation disappear; I was wrong. Before reading an improvement, run the same read twice and measure how wide the fluctuation is.

Large models and JSON output. These results are for a small local model like gemma4:e4b. A September 2026 paper reports that in LLM-as-a-judge evaluation, when limited to top-tier commercial models from 2025 onward, verbalized-confidence methods with added refinements outperformed logprobs. The same paper also notes that asking for answers in JSON structured output pins logprobs above 0.999. My setup asks for one letter without wrapping it in JSON, and the probabilities were split across the four levels.

Before adding a judgment model

Before looking for a judgment model, read the probabilities of the model you already have loaded. The accuracy gained by changing the reading method carries over as is on machines with more memory. The benefit of not needing a swap grows the more limited your memory is
Don't compare candidates on an accuracy table alone. Put "can it share memory with the current model," "how many seconds does a swap take," and "how many seconds does one judgment take" in the same table
Measure changed conditions one at a time. A comparison that changes four things at once can't tell what worked from what didn't. Before reading a difference, run the same measurement twice to get the width of the fluctuation

In production, I'm running this probability read alongside the current scoring and recording the results. I'll set a threshold and switch over only once that record has built up.

AI-mediated writing disclosure: AI drafted the English prose of this article from the author's Japanese original, measurement records, and code output. The measurements, the choice not to adopt JevK5, and publication responsibility belong to the author.

Moving My Research Pipeline's Judgment Calls from an LLM to Jev, a Judgment-Only Model

Tatsuya Shimomoto — Wed, 23 Sep 2026 10:56:45 +0000

Every morning I have an AI agent search for papers and repositories and write research reports. Most of the agent's work is reading what it found, one source at a time, and deciding: "Is this relevant to what I'm looking into right now?" and "Does it say anything new?" Until now, Claude Opus made those calls too, along with everything else.

"Is it relevant?" and "Is it new?" can be answered with a yes or no, or with a rating on a few levels. So I moved just those judgments to Jev, a judgment-only model that writes no text. The key was this: every time I ask Jev "is this relevant?", I also hand it what the source should be relevant to, namely the question that research topic is currently trying to answer.

From Pydantic AI, you call Jev with the same one line as any other model.

from pydantic_ai import Agent

agent = Agent("typesafe:jev-1.13.0", output_type=Answers)  # Answers is a Pydantic model listing the judgment fields

Hand Jev a single source on its own and it can only ask "relevant to what?" Hand it the source together with one question, and it places the source on a four-step ladder: a different problem, the same field, the same problem, or the same question. A grader can't grade an answer sheet without the exam question, and Jev is the same: it gets the thing to judge against along with the thing to judge.

The morning research I used to leave entirely to Opus

I have eight research topics (I'll call them lines from here on), and each morning I produce reports for three or four of them. The old setup launched Opus with claude -p, gave it WebSearch and WebFetch, and let it run freely for up to 55 turns. Where to search, whether a source it read was relevant, what was new, how to write it up: the Opus instance launched for each line decided all of it. The cost claude -p reported in its output was $13–15 per day over the last three days. (I don't compare that with the new pipeline's cost in this article, because I haven't confirmed the Jev portion against an actual bill.)

"Where to look" and "how to write" are open-ended work. There is no fixed set of answers. "Is this source relevant?" and "Is this strong evidence?" are closed: the shape of the answer is fixed in advance. A closed judgment is one you should be able to hand to a model built only for judging.

The pipeline at a glance

Alongside the existing setup, I built a separate pipeline with the flow below. The table shows the final version, for processing one line. Each line produces one report in Obsidian. Each morning's run handles three lines in rotation, plus one line that tracks developments around Jev.

Step	Name	Handled by	What it does
1	Read the questions	Code	Reads the file of "questions this line is currently trying to answer." Claude (Opus 5.5) drafted the questions in advance from sources such as the repository's knowledge graph, and I reviewed them
2	Read the search terms	Code	Reads the search terms for each question. Claude (Opus 5.5) also wrote these in advance from the questions, test-ran them, and then wrote them into the question file. Changing a question means regenerating its search terms
3	Fetch	Code	Pulls sources from arXiv, Hugging Face Papers, GitHub, and new-release listings
4	Screen sources	Jev	Is it likely relevant to a question? Does it contain evidence? Does it have instructions embedded in it? Is the source trustworthy?
5	Judge source × question	Jev → code	Jev answers with one source paired with one question; code applies thresholds to the probabilities and decides accept, needs review, or reject
6	Judge sentences	Jev, code	For each sentence in an accepted source, asks Jev whether it advances the question or is already known. Whether the source really says it is first checked by code with string matching
7	Write per question	LLM → Jev	An LLM writes a section for each question whose evidence grew that day; Jev checks its quality
8	Output the report	Code	Assembles the sections and saves the report to Obsidian

Jev answers the closed questions, and code decides what moves forward based on those answers. Claude is used only to write the questions and search terms in advance; it isn't called even once during the morning run. During the run, an LLM writes text only for the body in step 7 and when generating candidate questions for the following days. (For questions that don't have search terms yet, an LLM also generates candidates in step 2 and Jev picks among them.) This article covers the judgments in steps 4–6.

Claude's part ends before the run starts. At run time the pipeline moves through four blocks: prepare (code), judge (Jev answers, code decides), write (an LLM writes, Jev checks), and output (code). Only the judgment block in the middle has the "Jev answers, code decides" shape.

Where to fetch sources from, in what order, and how many: for now, code decides that. It uses the search terms written in advance, and nothing reads the results to decide where to look next. I think exploration is properly an LLM's job, ReAct-style: read the results, then pick the next place to look. The pipeline runs fine as it is, but depending on report quality I may hand exploration back to an LLM.

Every model used at run time (Jev and the LLMs) is called through Pydantic AI's Agent. I picked Pydantic AI for two reasons. It added native support for Jev's TypeSafe models early. And I didn't want to build an agent runtime from scratch: call the model, validate the output against a type, retry on failure. Swapping models takes one string. In fact, partway through I switched the run-time LLM that generates candidates (called through Alibaba's DashScope API) from Qwen's lightweight model to DeepSeek V4.1 Flash, because I had used up the free quota. The report body is written by qwen3.8-max.

With Jev, once you write the output type, each field's type determines how Jev is asked.

Pydantic type	How Jev is asked
`float` (0–1)	Probability that the answer to the question is "yes"
`Literal[...]`	Pick one of the options
`IntEnum` with docstrings	Graded rating

All the fields in one output type are sent together in a single request (Pydantic AI's TypeSafe model docs, as of 2026-09-23).

Ask alongside a question

To ask Jev "is this relevant?", you have to give it what the source should be relevant to. In this pipeline, that's the per-line questions. For each line I keep three to five questions I'm currently trying to answer in a file. On the line about AKC, a framework for how agents manage knowledge that I research, one of the questions reads like this (the file is in Japanese; translated here):

<!-- excerpt from questions/akc.md (translated from Japanese) -->
## How is intent alignment between an agent and its operator maintained in the parts tests can't check?
- brief: AKC's claim is that "a bidirectional growth loop maintains the alignment that tests can't check." How do designs that make similar claims (harness self-evolution, memory architectures, approval gates) detect alignment degrading, and on what grounds do they say it improved?
- evidence: Measurements over time. Single-shot benchmarks are weak
- not: Alignment training of the model alone (RLHF, etc.)

brief is the scope of the question, evidence is what counts as evidence, and not lists topics that share keywords but aren't what this question is about. For this question, 21 papers with "alignment" in the title actually came through to judgment. Many of them only share the word, such as papers on mapping between audio and representations (cross-modal alignment). Under not, I write the neighboring topics that are most easily confused with the question (here, alignment training of the model alone), so Jev can use them in its judgment.

Every judgment is asked as a pair with a question. In step 4, for each source, Jev is asked cheaply, in one batch, whether it's likely relevant to each of the line's questions, to narrow the candidates. In step 5, each remaining "one source × one question" pair gets asked in detail. Step 6 is "one sentence × one question."

The rest of this section is about step 5. Step 5 asks, on a four-step ladder, how close the problem a source works on is to the problem the question asks about. From the bottom: "a different problem that only shares vocabulary," "the same field but a different problem," "the same problem," and "answers the same question."

For this intent-alignment question, Jev judged three papers, each with "intent" in the title, as follows.

Paper	Probability relevant	Most probable step	That day's report
AI Persona, Service Consumption, and User Intent Entropy (a field experiment on AI personas and how much user intent varies)	0.08	Different problem (0.60)	Not included
FinInteract (a benchmark for clarifying intent in ambiguous financial questions)	0.20	Same field, different problem (0.53)	Not included
SkillSpec (has a model infer whether an agent's skill behaves as specified, with the intent withheld)	0.44	Same problem (0.77)	Not included

All three share the word "intent," but paired with the question, they land on different steps. SkillSpec was placed on "same problem," but its probability of being relevant was 0.44, short of the code's 0.5 threshold, so it didn't make the report. Jev only returns probabilities; code makes the accept or reject call with a threshold. The judgment values come from my local run logs, which aren't included in the repo.

The figure puts the three papers on the ladder: User Intent Entropy on "different problem," FinInteract on "same field," and SkillSpec on "same problem," with the relevance probabilities 0.08, 0.20, and 0.44 all below the 0.5 threshold. The step Jev picks and the "relevant" probability that code applies a threshold to are separate answers.

Here's the state I hand Jev (simplified from state() in question_screening.py). It gets serialized to JSON and passed to agent.run().

state = {
    "line": {"name": line.name, "vocabulary": vocabulary},
    "question": {
        "title": question.title,
        "brief": question.brief,
        "method": question.method_constraints,
        "evidence": question.evidence_constraints,
        "not": question.negative_topics,
    },
    "source": source_state(source),  # the source's title, an excerpt of its text, URL, etc.
    "evidence_set": evidence_set,  # evidence already accepted for this question
}

The output type looks like this (excerpt; Probability is a float from 0 to 1). The question.brief and evidence_set in the descriptions refer to keys in the state above.

class Overlap(UseEnumMemberDocstrings, IntEnum):
    other_problem = 0
    """It works on a different problem that happens to share vocabulary."""
    same_field = 1
    """Same field as `question`, but not the problem `question` asks about."""
    same_problem = 2
    """It works on the problem `question` asks about, from another angle."""
    same_question = 3
    """It asks what `question` asks and reports an answer to it."""


class Answers(BaseModel):
    """Screen one source against one open research question."""

    on_topic: Probability = Field(
        description="Is `source` about the problem `question` asks about, as `question.brief` "
        "describes it? No if it is about one of `question.not` (neighbouring topics that keep "
        "matching), or if it only shares a word with it."
    )
    problem_overlap: Overlap = Field(
        description="How close is the problem `source` works on to the one `question` asks about?"
    )
    novelty_vs_evidence_set: NoveltyVsSet = Field(
        description="Compared with `evidence_set` (the claims already accepted for this "
        "question), what would `source` add?"
    )

The "most probable step" column in the table is this four-step Overlap. The first paper fell to the bottom step: "a different problem that only shares vocabulary." Novelty is asked the same way, in terms of what the source would add to the evidence already accepted for this question.

Numbers from the final run

The report has one section per question whose evidence grew that day. The final run processed four lines. That run came before I switched to writing search terms in advance, so the search terms were generated by an LLM and picked by Jev.

Item	Value
Questions to Jev per line	1,400–1,511 (14 for one line with few new sources)
Report size	7.4–12.0 KB
Claude calls	0

Numbers alone can't tell you whether the screening also dropped sources I should have read. So on the Jev-tracking line, I placed four sources I know should pass (canaries) under its questions, and all four passed in the final run. I haven't placed canaries on the other lines yet, and I haven't measured how much the pipeline misses across everything it drops.

Writing stays an LLM's job

Outside the pipeline, I had an Opus instance, in a context separate from the session that built the pipeline, judge the quality of the reports written in step 7. Of the three lines whose body text was written in the final run, one was good enough to publish. The typical failure in the rejected drafts was recasting the evidence in the question's own terms.

In the section for the question "When an agent breaks rules it wrote for itself, what lets that failure slip through?", the body presented a paper about false positives, which fabricate violations of rules that don't exist, as an explanation of misses that slip past the rules. Jev had let this paper through for this question just barely: probability relevant 0.54, with the step "same problem" at 0.51 and "same field, different problem" at 0.35. Letting it through wasn't wrong in itself. It was the body text that flipped the direction.

Writing is an LLM's job, so this isn't something to move to Jev. The body got better after I fixed the instructions (don't let the evidence paragraph state conclusions) and turned on thinking for qwen3.8-max. I see the remaining weakness as a question of which model does the writing. The writer is also a Pydantic AI Agent, so it can be swapped with one string for a model that's stronger at prose, such as GPT. I haven't measured yet whether swapping helps.

What to check before handing judgments to Jev

Did you put what's being judged against into the state? Without something to be relevant to, "is it relevant?" lets anything through. Pass the question, its scope (brief), the topics to exclude (not), and the evidence already accepted
Is the answer closed? Move only judgments that can be expressed as a yes/no probability, a choice among options, or a graded rating. For a graded rating, describe each step as a concrete situation (same_field is "the same field, but not the question's problem")
Is there a step where sources that only share vocabulary can land? Without a "different problem" step, Jev has no choice but to push them onto a nearby step
Did you place sources that should pass? A smaller volume on its own can't be told apart from dropping too much
Is the writing LLM on the same Agent, with its own model string? Then you can swap just the writing model without changing the judgment types

The code is public at jev-research-pipeline.

AI-mediated writing disclosure: AI drafted and translated the prose of this article from the author's pipeline code, run records, and the public sources cited above. The pipeline design, the choice of which judgments to move to Jev, and responsibility for publication belong to the author.

What Does It Take to Reproduce Jev's Decisions Locally?

Tatsuya Shimomoto — Tue, 22 Sep 2026 15:46:53 +0000

Model	Method	Agreement with Opus ceiling	Time per row	Result
Jev (hosted)	choice+noul	0.346	0.3 s	✓ Baseline
gemma4:e4b	logits x 54 questions	0.162	51 s	Existing baseline
qwen3:8b	logits x 54 questions	— (stopped at 4 rows)	43–74 s	❌ Too slow
Laya	noul	0.075	39 s	❌ Random-level
kev-0.8b	choice	— (stopped at 25 rows)	6.9 s	❌ Out of memory
AFM 3 Core	verbalized	— (20 rows only)	20.7 s	❌ Context too short

"Agreement with Opus ceiling" = Jaccard index measuring how much each model's selections overlap with those of Opus 5, treated as the upper bound. Details in "Measuring with 150-row replays." Random expectation: 0.056. Method descriptions follow in the next section. AFM = Apple Foundation Models (the on-device model in macOS 27). Its context length (the maximum text a model can process at once) is only 4,096 tokens — too short for the full prompt — so it was tested on just 20 rows with the catalog cut in half; AUC was 0.443. Jev's numbers are from the previous article.

I tried to reproduce the skill-selection judgment that TypeSafe's decision model Jev solves in 0.3 seconds, using four candidates that run locally. As the table shows, all four failed. But the causes of failure split into three, and they reveal the conditions to evaluate next time a local decision model is on the table.

30+ open alternatives and 3 approaches

In the previous article, Jev reached about half the agreement ceiling — using Opus 5's picks as the upper bound — in 0.3 seconds per row for skill selection in an autonomous agent. But Jev is a closed API. As of September 2026, there are no open weights and no self-hosting path.

Within a week of Jev's launch, over 30 open alternatives appeared (a catalog is at systemonemodels.org). None of the trained decision-only models (kev, von, Laya, etc.) run on Ollama, though. Ollama requires GGUF-format weight files, and none of the decision-only models ship GGUF. On top of that, these models replace the standard text-generation output layer with a custom head that returns only select/don't-select probabilities — Ollama's general-purpose inference engine cannot handle that.

The approaches you can try fall into three categories.

Logits method — Ask a general-purpose LLM "Should we select this skill?" as a yes/no question and read the yes and no probabilities (log-probabilities = logits) via Ollama's logprobs option. No fine-tuning needed, but you pay 54 calls for 54 skills
Typed-decisions method — Use a classifier trained specifically for decision tasks. Pass all 54 skills and the situation at once, and receive the "should select" probability for each skill in a single response. Instead of generating text like a general-purpose LLM, it outputs only an array of probabilities
Specialized decoder method — Replace the "text-generation part" of a general-purpose LLM with a "decision-only output part" and run the resulting small model through a dedicated server

I picked one from each category. qwen3:8b is the smallest Ollama-compatible general-purpose LLM where prefix cache (explained below) works. Laya is the only decision-only model with a public Python API. kev-0.8b is the highest-profile project aiming to reimplement Jev. I also added Apple's on-device model (AFM 3 Core, Apple Foundation Models) available through the FoundationModels framework in macOS 27. AFM is not open, but it ships with macOS and requires no extra installation.

Measuring with 150-row replays

The measurement method is the same as last time. I fed 150 rows of actual skill-selection logs from an autonomous agent into each model and compared the agreement with Opus 5's picks. The test environment is Apple Silicon M1 (16 GB) using MPS (Metal Performance Shaders — Apple's GPU compute framework, analogous to NVIDIA's CUDA), running Ollama 0.34.2.

Three metrics:

Jaccard@topk — The overlap between the skill sets Opus picked and the model picked. Calculated as "skills both picked / skills either picked." If Opus picked {A, B, C} and the model picked {A, B, D}, the overlap is {A, B} = 2, the union is {A, B, C, D} = 4, so Jaccard = 2/4 = 0.5. 1.0 means perfect agreement, 0 means zero overlap
AUC — Higher when the skills the model assigned high probabilities to are the ones Opus also picked. Measures ranking quality: 0.5 equals random, 1.0 means perfect ranking
ECE (calibration error) — How well a model's stated confidence matches its actual hit rate. When a model says "80% confident this should be selected," does it actually get it right 80% of the time? Calibration is the property of a model's probabilities matching real-world accuracy. ECE of 0 means perfectly calibrated. High ECE means you cannot trust the probability numbers for decision-making

The skill-selection prompt runs about 12,000 characters (skill catalog ~9,600 characters + situation description median 1,537 characters). The maximum text a model can process at once is called its context length. This prompt size matters later.

qwen3:8b — 54 questions pile up

First, the logits method. Ollama 0.34.2 returns the top-20 token probabilities with logprobs: true.

curl -s http://localhost:11434/api/generate \
  -d '{"model":"qwen3:8b","prompt":"...Should we select this skill? Answer yes or no:","stream":false,"logprobs":true,"top_logprobs":5,"options":{"num_predict":1,"temperature":0}}'

For each of the 54 skills, ask "Should we select this skill?" and read the yes log-probability.

The 54 prompts share a common prefix — "skill catalog + situation description" (~12,000 characters) — with only the trailing "Should we select this skill?" part changing per question. Prefix cache holds the processing result of this shared part in memory so that from the second question onward, only the changed tail gets processed.

I first tried qwen3.5:9b, but even though the 54 prompts share a common prefix, every question after the first still took over 5 seconds. Prefix cache was not working. Switching to qwen3:8b, the second question with a different suffix returned in 0.4 seconds.

qwen3.5:9b — same prompt resent: 0.3 s / different suffix: 7–12 s (full reprocessing)
qwen3:8b   — same prompt resent: 0.3 s / different suffix: 0.4 s (prefix cache active)

LLMs compute relationships between tokens using an attention mechanism. Standard Transformer attention examines every pair of tokens, which is expensive, but the intermediate results (the KV cache) can be saved and reused. The hybrid linear attention that qwen3.5 uses trades cheaper computation for what I suspect is an inability to reuse KV cache from just the shared prefix. I switched to qwen3:8b, which uses standard Transformer attention.

With cache active, the first question took 14.2 seconds; the rest had a median of 0.63 seconds. Per row: 43-74 seconds. The gemma enum method from the previous round passes all 54 skills at once in a single call and gets an answer in ~13 seconds per row; the logits method runs 54 serial calls, making it 3-5x slower. I cut it off at 4 rows.

The serial execution of 54 calls is the fundamental bottleneck of the logits method. Even with prefix cache, 0.63 s x 54 = 34 seconds is the floor.

How to tell if prefix cache is working: The prompt_eval_count (input token count) in Ollama's response reports the original full token count whether or not the cache is active, so that number alone does not reveal cache status. If prompt_eval_duration (time to process the input) drops significantly, the cache is working.

Laya — the wall called calibration

Calibration, as explained in the previous section, is the property of a model's stated probabilities matching actual hit rates. "70% confident" means it really hits 7 out of 10 — that is what being calibrated means. Laya hit a wall here.

Laya is a typed-decisions model based on mmBERT-base (322M params), available on HuggingFace. I used two question formats:

noul — Returns yes/no probabilities per skill
choice — Picks the best from a set of options, returning each option's probability

Results over 150 rows:

Metric	Laya (noul)	Laya (choice)	gemma (logits)	Random
Jaccard@topk	0.075 [0.062, 0.089]	0.051 [0.041, 0.061]	0.162 [0.141, 0.182]	0.056
AUC	0.587 [0.561, 0.611]	0.477 [0.455, 0.499]	0.728 [0.704, 0.750]	0.500
ECE	0.469	—	—	—

Brackets show 95% confidence intervals.

Noul's Jaccard of 0.075 barely exceeds random (0.056). Choice's confidence interval includes random — indistinguishable.

The core problem is calibration. Breaking down ECE 0.469: of 8,207 total judgments, 6,382 (78%) fell in the 0.5-0.7 probability band, where the actual hit rate was 10-14%. "60% confident this should be selected" actually hits 1 in 10.

Laya prints this warning at startup:

laya: this checkpoint ships temperatures outside [0.5, 5]
which would distort confidence; clamping choice:11+=0.1006.
Treat confidence from the affected buckets as uncalibrated.

What this means: Laya uses an internal temperature parameter (which adjusts the sharpness of the output probability distribution) to compute probabilities. In this checkpoint, cases with 11 or more options (bucket = group by number of options) have temperatures outside the normal range, so the probability values are force-corrected (clamped). Selecting from 54 skills falls into the 11+ range, so the returned probabilities are not calibrated.

I also checked latency. The model card cites 33 ms per question (7 ms batched) on an NVIDIA T4 GPU, but my test environment uses Apple Silicon (MPS). Different hardware, so a direct comparison is not meaningful, but the measured values in this environment were: noul median 39.4 seconds per row (54 questions), minimum 6.8 seconds; choice 10.5 seconds, minimum 1.7 seconds.

Laya's problem is quality, not speed. When probabilities cannot ground decisions, you cannot run a policy like "delegate only the high-confidence rows to Jev's replacement."

kev-0.8b — 0.8B devours 12 GB

Last, the specialized decoder kev-0.8b. It mounts a decision head on Qwen3.5-0.8B-Base and runs through a dedicated server.

Sending one request as designed (~6,000 tokens, choice + noul for 54 skills) immediately ran out of memory.

RuntimeError: MPS backend out of memory
(MPS allocated: 12.50 GiB, other allocations: 7.02 GiB,
 max allowed: 20.13 GiB).
Tried to allocate 880.00 MiB on private pool.

Splitting works. Choice alone (2,432 tokens) takes 8.6 seconds; noul split into batches of 14 takes 7-27 seconds. But running choice alone continuously, the MPS memory allocator failed to allocate even 16 KB after processing 17 rows. The server does not release GPU memory between requests.

Two constraints are at play:

Inference kernel (the program that executes the model's computation on the GPU): kev's flash-linear-attention implementation targets NVIDIA (CUDA) and AMD (ROCm) GPUs and does not support Apple Silicon's MPS. It falls back automatically to the unoptimized reference implementation, printing a "much slower" warning at server startup
Input length: Training was done at 384 tokens or fewer. The serving config can set num_ctx to 8,192, but the gap from the 12,000-character prompt to the training input length remains large

I cut it at 25 rows, taking only the choice-alone results. Working around the allocator issue would require periodic server restarts, and I could not see a path to completing the full 150 rows.

AFM 3 Core — context length falls short

macOS 27 made the FoundationModels framework available. On the M1 (16 GB), the model that arrives is AFM 3 Core (3B) with a 4,096-token context length. Too small a window for text generation, but a decision model's input and output are short, so I tried it on the off chance. The larger AFM 3 Core Advanced (20B sparse) has no model-selection API in the SDK and does not come down to M1.

The first wall is context length. The skill-selection prompt is about 12,000 characters, but AFM's context length is only 4,096 tokens. I cut the catalog in half and measured on just the first 20 rows.

fm serve provides an OpenAI-compatible endpoint, but it silently ignores logprobs. Using a verbalized approach (asking the model to state probabilities as numbers), AUC was 0.443 at temperature 0. The same 20 rows with gemma's logprobs scored 0.736. Five of the 20 rows hit a guardrail RefusalError, leaving half the catalog unscored.

The one advantage is co-residency. In-flight memory is about 2 GB, and it coexists with Ollama models without increasing swap. But if the prompt does not fit the window, it is not a candidate.

3 conditions a local decision model needs

Line up the reasons each candidate dropped out, and three conditions for practical local decision models emerge.

1. Calibrated probabilities

Laya returns probabilities, but you cannot split decisions by those numbers. ECE 0.469 means the probabilities are decorative. As the gap between Laya's typed-decisions benchmark (0.766) and its score on this task (0.075) shows, calibration depends on the task and the prompt structure.

2. An attention mechanism that supports prefix cache

The 54 skills share a common prefix. Without cache, every question reprocesses 12,000 characters from scratch. qwen3.5's hybrid linear attention cannot do this reuse; kev's flash-linear-attention targets CUDA/ROCm and does not run on Apple Silicon. Both the kind of attention mechanism and the inference environment are conditions.

3. Context length of 12,000 characters — and training at that length

kev was trained at 384 tokens or fewer. Stretching num_ctx at serving time does not guarantee quality when the gap from training is that wide. AFM's context length is physically 4,096 tokens — the 12,000-character prompt does not fit. Input length must not only be "accepted by the config" but also "trained at that length," and the window must be wide enough.

In response to these results, I introduced a seam called DecisionBackend on the agent side. Setting the environment variable DECISION_MODEL switches the decision surface to a local model — ready to swap in a model that meets the conditions.

Looking back, Jev's performance is in a different league. Calibrated probabilities, context length that absorbs long inputs, an attention mechanism that supports prefix cache — it meets all three conditions, and on top of that returns 0.3 seconds per row at half the Opus ceiling. The fact that the four candidates each dropped out at a different condition throws into relief what Jev accomplishes at that price and speed.

Four triggers would reopen the evaluation: kev implements an MLX backend, Laya gets fine-tuned on this task's data, Jev itself publishes open weights, or AFM's context length is extended. The moment any one of those moves, I re-measure with the same 150 rows.

AI-mediated writing disclosure: AI drafted the English prose of this article from the author's Japanese original, measurement records, and terminal output. The measurements, the synthesis into three conditions, and publication responsibility belong to the author.

I Added Jev's Skill Router to Claude Code and Turned Back Just Before Rewriting the Skill Listing

Tatsuya Shimomoto — Mon, 21 Sep 2026 21:47:39 +0000

I try not to fill the gaps in a big harness like Claude Code with my own implementation. When I do fill one, the official side changes a while later and my work is no longer needed. That has been the pattern for the last six months.

The rules I had accumulated to shore up an older generation of models stopped being useful when Anthropic changed course for the Opus 5 generation — fewer rules, leave more to the model's judgment — so I cut my resident set from 5,789 words to 2,463. To run several Claude Codes side by side I installed Herdr and wrote an article about it, and now Claude Code itself can send messages between sessions. Projects, where Claude starts and manages parallel sessions from a single conversation, is rolling out in public beta. Today's Projects bundles only cloud sessions, but once it handles the sessions on my own machine, Herdr will not be needed either.

Even so, on September 21, 2026, curiosity won. Jev is a model that writes no text and returns only probabilities to typed questions. That day I ported the skill suggestion cookbook from TypeSafe, who build Jev, to a Claude Code hook.

This article is about how far the thing I built reached, where I turned back, and what was waiting past the turn. If you are thinking about adding Jev to your own harness, you should come away with three things to check before you do.

What I added is one line

What I built is a Claude Code plugin, jev-skill-router. On every prompt a UserPromptSubmit hook runs and sends Jev the prompt together with the roster of skills installed locally — their names and descriptions. It makes at most two requests to Jev. The first ranks the whole roster and asks three yes/no questions about whether the request needs a skill at all. The second re-reads only the top three candidates, this time with the full text of each SKILL.md. If neither the average of the three first-pass answers (gate in the log; the third is inverted before averaging) nor the highest per-candidate value from the second pass (fits) reaches 0.30, it suggests nothing.

By default it adds nothing and only records its verdict in a log. With injection enabled, a turn that reaches 0.30 gets exactly this one line added.

<skill_relevance>
Relevant to the current request: adr-writer. Ignore this if it does not fit what the user actually asked for.
</skill_relevance>

Each verdict leaves one line in the log. This is the line from a request, written in Japanese, to record a design decision in an ADR (excerpt).

{"mode": "inject", "model": "jev-1.13.0", "router_version": "0.2.0",
 "gate": 0.47,
 "shortlist": ["adr-writer", "adhd:adhd", "archify"],
 "fits": {"adr-writer": 0.93, "adhd:adhd": 0.14, "archify": 0.05},
 "suggestion": "adr-writer",
 "usage": {"input_tokens": 21597, "output_tokens": 702, "calls": 2},
 "elapsed_ms": 1567}

I wrote six request sentences with a single intent each and sent them. Five named the skill I judged to be right, at 0.93 to 0.98, and "Thanks, that's it for today" drew no suggestion at all. Each took 0.7 to 1.6 seconds. These are single readings per case, not numbers you can read as a rate.

Up to here, it worked exactly as the cookbook said it would.

Claude Code's own skill selection was running the whole time

While running it, I asked: "Hold on — so Claude Code's own skill selection is still running at the same time?"

It is. All a hook's additionalContext can do is add one string to the model's context. Claude Code goes on showing the model a skill listing with the name and description of every installed skill, and the model is what picks from it. Not one token of that listing goes away. Jev's verdict was simply running alongside the choice.

On top of that, the conditions that made the cookbook work are absent in Claude Code. The cookbook's experiment covers 488 requests; across the 315 where a skill applied, loading the wrong skill dropped from 16.8% to 7.3%. The model choosing skills there was claude-haiku-4-5, and what it saw was an index cut to 60 characters per skill. The cookbook says so at the top:

Hermes, the agent harness used here, cuts it to 60 characters by default.

60 characters is not a recommendation from the cookbook; it is the default display width of the harness used in the experiment. The cookbook itself gives the example that at 60 characters you cannot tell a skill that edits .pptx from one that creates it. It describes the problem it set out to solve as an agent that "makes its choice on almost no information." A small model choosing from a truncated index, with one line added by a judge that had read the full text. Those were the conditions under which it worked.

Claude Code is the opposite. It shows the model each description (together with when_to_use) up to 1,536 characters, uncut. Counting in the repository where I am writing this: 59 skills installed, 2 with descriptions that fit in 60 characters, median 412. The model doing the choosing is Claude Fable 5.1, stronger than Haiku. A router in Claude Code turns into a second model that has read the same descriptions, advising from the side a stronger model that has already read all of them in full. The only information it can add is the body of SKILL.md.

This is an argument from the mechanism, not a measured result. I have no evidence that injecting the line improved anything. That is why I did not make injection the plugin's default.

There was a path to rewriting the skill listing

To make it work as a router, adding a line is not enough. You have to change the skill listing the model is shown.

That path existed. The type definitions for function hooks — Claude Code's early-access feature, known as Mods (design thread) — list skill_listing among the kinds of attachment that reach the model, and say its body can be rewritten from the Mod side (mods/types/claude-code.d.ts in anthropics/claude-code, checked on September 21, 2026).

I started a design plan and stopped almost immediately. With enough effort it might not be impossible. But it means replacing Claude Code's default skill selection with my own. An implementation that rides on early-access types and breaks the default behavior has to follow along every time the official side changes. After six months of filling gaps only for the official side to change and make the fill unnecessary, this is a swamp.

I wrote at the top of the README, with the reasons, that "as a router it is unlikely to help a strong model in Claude Code," left it there for anyone thinking the same thing, and stopped.

That path had been walked the day before

Searching again while preparing this article, I found that what lay past my turning point was already implemented.

It is the Mod jev-skill-suggestion in davila7/claude-code-templates. Its first commit was on the morning of September 20, 2026, Japan time — the day before I turned back. At that point the very rewrite I had stopped short of was working. It returns empty for Mods' skill_listing, so the model never reads the skill listing at all. Instead it tells the model to load the single skill Jev picked. The model can no longer choose from the listing itself; what remains is to follow Jev's suggestion or ignore it. A second commit, at midday on the day I turned back, went further: the Mod attaches the chosen skill's SKILL.md itself. The first line of the current README reads:

Takes the skill listing out of the context window and lets Jev, TypeSafe's System One decision model, pick at most one skill per prompt from the skills' descriptions — and then loads that one skill itself, by attaching its SKILL.md to the prompt.

The way it picks is the same two requests from the same cookbook. Hiding the skill listing was not custom work: it is the official setting skillOverrides, which switches how each skill is shown to the model across four levels — name and description, name only, only when the user calls it, or not shown. If all you want is to cut the tokens the skill listing costs, you need neither Jev nor Mods. The same repository also holds jev-model-router, which routes model and reasoning depth with Jev.

I have tried neither. Whether they work, I do not know. What I do know is that someone willing to carry the cost of following the official side was already there before I stopped. If I want to try it later, the real thing exists without my building it.

Who holds the role of choosing

From here on, this is my own read, from having built the thing.

In Claude Code, the role of choosing skills belongs to Claude Code and its model. A frontier model carries it, and the only opening I had was one for adding a line of text.

How far Jev works, I think, is decided by who holds the role of choosing. If a weak chooser holds it, one line added from the side is enough — that was the cookbook. If a strong chooser is looking at the full text, advice does not move it, and the only way through is to reach the role of choosing itself. Mods is an example of the harness side opening that role outward. pi-jev, an extension for the coding agent Pi, enables only the tools whose Jev probability clears a threshold. Not advice: the tools the model can see are what change.

Most harnesses today are built around the loop where the model thinks and then calls a tool (ReAct), with a strong model seeing everything and doing both the choosing and the judging inside its own reasoning. As long as the chooser is strong, adding a fast decision model beside it leaves no work to be replaced. Moves to fit Jev partially into existing harnesses will probably keep coming, but the effect should stay inside whatever range the harness has opened.

Conversely, if a harness appears that builds tool selection, model routing, and the judgments along the way with Jev from the start, and hands only the final reasoning to a frontier model, speed, cost, and accuracy could all look very different. TypeSafe itself calls Jev "a frontier-intelligence function call" in its announcement, framing it as a component you call. When I compared Jev and Opus on skill selection in a different agent of my own, Jev took 0.32 seconds per case at about 1/560th of Opus's cost, and across the 45 cases where it pointed with probability 0.5 or higher, not one disagreed with Opus's pick. On the other hand, agreement across all 150 cases was about half of the agreement Opus reached with itself when made to do the same selection twice. How much of this you can hand to Jev, I have not measured, and I have not found a report that measures it.

As of September 21, 2026, within the range I searched, no such general-purpose harness existed. Everything I found is an integration into an existing harness.

Three things to check before you add

If you are about to add a decision model like Jev to your own harness, check three things before you start writing.

Does what you add reach the role of choosing? If you only add text, the original choice keeps running underneath. Find out first whether there is an opening that reaches it — your own loop, Mods, tool enablement.
Is the chooser already seeing the same information as the decision model? Before you read the numbers from a vendor's experiment, read the experiment's conditions. Which model was choosing? What could that chooser see?
Is an official setting enough? If what you want is fewer tokens spent on the skill listing, skillOverrides is enough in Claude Code. A decision model is for what lies past that.

An extension that only adds text goes back the way it was when you remove it. Build one out of curiosity if you like. An implementation that replaces the default behavior is something I will not build, even if I could.

I moved over to following instead of building. I added "does a harness built on Jev appear" to my daily research checklist. My plugin keeps running in the setting where it adds nothing and only writes the log. What I watch is the match between the skill Jev named and the skill actually used in that turn. If turns where it named a skill that went unused pile up, adding one line may be worth something. If they do not pile up, I take it out.

AI-mediated writing disclosure: AI drafted and translated the prose of this article from the author's session records, the plugin's logs and README, and the public sources cited above. The central thesis, the decision to turn back, the read in "Who holds the role of choosing," and responsibility for publication belong to the author.

How Close to Opus Does Jev, a Model That Writes No Text, Get at Skill Selection in 0.3 Seconds?

Tatsuya Shimomoto — Mon, 21 Sep 2026 01:03:01 +0000

I took the step an autonomous agent runs before it acts — picking, out of a catalog, the skills that apply to the situation in front of it — and had Jev and Claude Opus do it over the same 150 cases. Jev is the model TypeSafe opened early access to on September 15, 2026. It writes no text; it returns answers over a set of options and the probabilities behind them (TypeSafe's announcement).

The yardstick is Opus itself. Run the same 150 cases through Opus a second time, and its agreement with the first run is 0.678. Picking at random gives 0.056. Jev came in at 0.346 — about half of Opus's agreement with itself, and more than twice the small model on my machine. The skill Jev ranked first is one Opus picked too, in 83% of the cases. Across the 45 cases where it ranked something first with probability 0.5 or higher, not one disagreed with Opus. Time is 0.3 seconds per case, and the 150 cost about five cents: about 1/50th of Opus's time and about 1/560th of its cost.

Once Jev runs on my own machine, I want to move most of this agent's processing of the same kind over to it. Jev today is hosted, and this agent, which I keep complete on one machine at hand, cannot take it yet.

Picker	Agreement with Opus	Time per case	Cost for 150
Opus (second run)	0.678	15.3 s	$26.26
Jev	0.346	0.32 s	$0.047
local gemma4:e4b	0.143–0.162	9.8 s	—
Picked at random	0.056	—	—

The baseline is Opus's first run. The range for gemma is the range across eight output configurations I tried. Opus's picks are the baseline, not the ground truth. Time and cost were not measured under matched conditions, so I put them under "What this comparison cannot say."

One skill selection, in full

The subject is an autonomous agent I run (Contemplative Agent). Before it acts, the agent receives a description of the situation and a catalog of skills — name and description pairs — and picks the ones that apply. It may pick several, and it may pick none. The catalog runs 53 to 57 entries depending on the period.

Right now a small model running on my Mac (gemma4:e4b) writes out the names of the skills that apply. Because it picks by generating text, each run takes around ten seconds, and 20–30% of the time it produces a misspelled skill name. A model that picks without writing came out, so I tried it.

The agent logs the input and output of every skill selection. Out of those logs I pulled 75 cases that contained a misspelled skill name and 75 that did not; those are the 150. From here on I call one case one row.

Look at one of those rows. The catalog for this row holds 54 skills.

Skill	Opus	Jev's probability	local gemma
`analogy-mapping-relationships`	picked	0.61	picked
`analyzing-systemic-governance-loops`	picked	0.12	—
`trace-structural-authority`	picked	0.07	—
`cross-reference-foundational-claims`	—	0.02 or less	picked
`detecting-abstract-to-operational-constraint-shift`	—	0.02 or less	picked
`scope-boundary-mapping`	—	0.02 or less	picked
`shifting-focus-from-state-to-process-mechanics`	—	0.02 or less	picked
`identify-structural-tensions-via-system-metaphors`	—	—	misspelled

Opus picked the same three skills both times. The three Jev gave the highest probabilities are those same three. gemma wrote six names, one of them not in the catalog. The real one is identifying-…; the word form is broken. Names that are not in the catalog get dropped by the code, so gemma's selection comes to five.

When this article says "agreement with Opus," it means the number both picked divided by the number either picked (the Jaccard index). On this row gemma shares one and either-picked seven: 1÷7, or 0.14. For Jev, taking the same number of skills gemma picked from the top of the probability ranking (how that number is set comes in the next section), all three of Opus's picks are in, giving 3÷5, or 0.60.

This row is one of Jev's best. gemma's 0.14 happens to be the same value as its average across the 150 rows. One row says nothing, so I count across 150.

How I asked Jev

I used two kinds of question. What TypeSafe's docs call Choice — pick one out of a set of options — and what they call Noul — a yes-or-no question. I made each row one request, with one Choice question and one Noul question per skill.

{
  "state": {"situation": "(the situation description)"},
  "model": "jev-1.13.0",
  "questions": {
    "choice": {
      "type": "choice",
      "instructions": "(the selection criteria) Which single learned skill applies best to `situation`?",
      "criteria": {
        "analogy-mapping-relationships": "(this skill's description)",
        "none of the above": "No skill in the catalog applies to this situation."
      }
    },
    "n0000": {
      "type": "noul",
      "instructions": "(the selection criteria) Does the learned skill `analogy-mapping-relationships — (description)` apply to `situation`?"
    }
  }
}

In practice criteria holds every skill and the Nouls run as long as the skill list, so one request carries about 55 questions. Jev's questions cannot reference one another, so the selection criteria and each skill's description go into every question. Input came to about 7,450 tokens per row.

Choice returns a probability for every skill, but it does not return how many to pick. So I took, from the top of the probabilities, the same number of skills gemma picked on that row (6.0 on average) and made that the set. Opus decides the count itself (6.0 on average in its first run). The Jev row in the table at the top is this Choice value.

Across 150 rows, how close to Opus

Picker	Skills picked (average)	Agreement with Opus	Agreement with near-skill credit
Opus (second run)	5.3	0.678	0.924
Jev	6.0 (gemma's count)	0.346	0.855
local gemma4:e4b	6.0–8.1	0.143–0.162	0.755–0.784
Random, same count	6.0	0.056	0.70

Even asking the same Opus twice, agreement tops out at 0.678. That is the ceiling for this way of counting, and the floor is random at 0.056. Jev's 0.346 sat at about half the ceiling, more than twice gemma. gemma did not move off 0.14–0.16 whether I constrained its output with an enum or set temperature to 0 (the gemma-side record is in the evidence in the public repository).

Jaccard counts a neighboring, similar skill as a miss. The right-hand column measures closeness with embeddings of the skill descriptions and gives partial credit for similar skills. For each skill Opus picked, I averaged its closeness to the nearest one in the other side's selection. Every skill resembles every other to some degree, so even random scores 0.70. Between that floor of 0.70 and the ceiling of 0.924, Jev is at 0.855 and gemma at 0.76–0.78. Allow similar skills and Jev moves a bit closer to Opus's agreement with itself.

Look at the individual picked skills rather than the agreement between sets, and it gets simpler.

What	Share Opus also picked
Skills Opus's second run picked (5.3 on average)	85%
Jev's top pick (the 147 rows where Opus picked anything)	83%
of those, the 45 rows where the top probability was 0.5 or higher	100%
Jev's top picks taken out to gemma's count (6.0 on average)	50%
Skills the local gemma picked (6.0 on average)	25%
A single random pick	11%

Jev's top pick, at 83%, is level with the 85% for what Opus's second run picked. But Opus's 85% is measured over 5.3 picks on average, while Jev's 83% covers only its single most confident pick. Widen Jev out to six and it falls to 50%. In 5 of the 147 rows, Jev's top pick was "none of the above," which I count as a miss.

Across the 45 rows where Jev pointed at something with probability 0.5 or higher, it never once disagreed with Opus. The probability works as a cutoff separating the rows you can trust from the rows you cannot. If you take Jev's pick as-is in place of a large model, take only the high-probability rows.

Ask one at a time, and the skills that read broadly come out on top on most rows

I ran the same comparison on the probabilities from the Nouls in that same request (per skill: does this apply?), again taking the same number from the top.

	Choice	Noul
Agreement with Opus	0.346	0.295
Agreement with near-skill credit	0.855	0.836
Rows where `shifting-focus-from-state-to-process-mechanics` made the top	34%	77%

Noul put that one skill in the top on 77% of rows. It is the skill gemma picked in the row shown above. Opus picked it on 16% of rows in the first run and 14% in the second. It is not that Noul says yes to everything: 35% of its probabilities are 0.5 or higher, and the median is 0.38. Noul looks at one skill at a time, so it has nothing to compare against. My reading is that a skill whose scope can be read broadly comes out on top in any situation. With Choice, which weighs every skill against the others, it drops to 34%.

For skill selection, Choice was the better fit. TypeSafe's cookbook for skill suggestion is two-stage: rank broadly with Choice, then re-judge only the top with Noul. Applying a per-skill Noul to every skill, the way I asked, falls outside that usage.

Five cents and 0.3 seconds

Opus took 15.3 seconds per row (median) and $26.26 for the 150. Jev took 0.32 seconds and $0.047: about 1/50th the time, about 1/560th the cost. All I had Opus return was a JSON array of skill names, and even so the output ran to a median of 1,373 tokens. Jev has no output tokens. The difference in time is the difference between generating tokens and not generating them. The difference in cost also carries the difference in input pricing.

Jev had 0 failures across the 150 requests, with a median of 0.32 seconds, 0.53 seconds at the 90th percentile, and a maximum of 4.5 seconds. That includes the network round trip from Japan. Input totaled 1,125,726 tokens; multiplied by the published rate ($0.042 per million input tokens, output free), that comes to $0.047. It is a computed figure, not a billed amount. Opus I called one row at a time with claude -p; the cost is the API-equivalent amount its response returns, and the time is the API-side duration.

What this comparison cannot say

The baseline is Opus's picks, not the ground truth. I took no human labels. What I can say goes as far as "Jev is close to the way Opus picks" — a different claim from "Jev's picks are right." And gemma's picks being far from Opus's does not mean gemma is wrong.

Conditions unfavorable to Jev are mixed in with repurposing of my own. The situation text contains Japanese, and TypeSafe itself writes that languages other than English are not handled equally. Choice is a question meant to pick one, and using its probabilities as a ranking is my repurposing.

Nor is the handoff matched. Opus got the same situation, the same catalog and the same selection criteria, but through a different template from gemma's production one. Jev got the JSON above. The timing conditions differ too. Jev's 0.32 seconds includes the round trip from Japan, gemma's 9.8 seconds is on a 16GB M1, and Opus's is the API-side duration.

The subject is one process in one agent, 150 rows. Half were drawn from rows that contained a misspelled skill name, so this is not the production distribution as it stands. The row data contains third-party posts, so I am not publishing it.

Once it runs on my machine, I want to move processing of this kind to Jev

This agent has several other steps that only pick, or only judge. Right now every one of them runs on the local gemma by generating text. Looking at these results, if Jev ran on my machine I would want to move most of them over. It lands more than twice as close to Opus as the gemma I run today, in 0.3 seconds, with no misspelled names and a probability attached.

What keeps it out today is that Jev is hosted. This agent is designed and run so that skill selection and text generation alike stay complete on one machine at hand. Having no path out to the outside in the production code is this agent's premise for safety. The 16GB M1 and the small model are a constraint I chose as well. Jev has neither open weights nor a self-hosting path yet. Once it can run on a local runtime like Ollama, it is the first thing I will try.

For this measurement I sent the 150 rows of situation text to both Jev and Opus. That was a measurement I permitted once, from a script separate from the production code. I judged it a different thing from production sending data out on every run.

There is no shortcut where I take logs of Jev's picks as a teacher and train the small model on my machine. Section 2.3(b) of TypeSafe's Master Customer Agreement (updated September 19, 2026) prohibits distilling a model from the outputs and training a model to imitate them.

Skill selection turned out to be a process that does not need text written for it. A model that only attaches probabilities to options came, in 0.3 seconds, to about half of Opus's agreement with itself. If your agent's design allows calling an external API, there is no reason to wait. Taking the rows Jev points at with high probability as they are, and sending the rows where the probability falls short of 0.5 (about 70% here) to a bigger model, is an idea worth trying.

AI-mediated writing disclosure: AI drafted the prose of this article from the author's measurement records, the aggregated result files, and the public evidence in the repository. The central thesis, the choice of Opus as the yardstick, the judgment about adopting Jev, and publication responsibility belong to the author.

I Made the Top Model My Session Default, and One Heavy Implementation Plus Its Review Burned Through Fable's Usage Limit

Tatsuya Shimomoto — Wed, 16 Sep 2026 12:12:21 +0000

On August 22, 2026, I started a heavy implementation in a Fable session and ran a review right after it. Fable's usage limit was gone in an instant. That happened because the built-in skill that handles the review inherited the session's model and ran on it.

Three days later, in the morning, I was chasing a different problem and noticed something. The over-engineering that shows up in my own repositories was happening in the Opus sessions — the ones that picked up design work after Fable's usage limit ran out.

Those two are the first and second half of the same event. When you make the top model your session default, the first thing that breaks is not the work product but the usage limit, and once the limit is gone, judgment itself drops to a lower model. This article is a record of which paths the default leaked into, why a convention and a warning failed to stop it, what did stop it, and what fell once the limit was gone.

Setup: three models across three tiers

I use Claude Code on a subscription plan (Max) and split three models by role. Fable, the top model, does judgment — checking premises, planning, acceptance. Below it, Opus does implementation, and I hand that work to a new session. Mechanical cross-checking goes to Sonnet. On my plan, Fable's usage limit is separate from Opus's, so Opus still works after Fable runs out. From here on I call the judgment side the judge tier and the implementation side the build tier (in my config files they are literally judge-tier and build-tier). My settings.json default model is fable, and I wrote up the division of labor for handing implementation off in an earlier article.

Since the start of August, every subagent I wrote myself carries a model: line. Writing the model into the definition file to fix it is what I call pinning below. The 11 files in ~/.claude/agents/*.md break down as fable 1 / opus 4 / sonnet 5 / haiku 1 as of September 16, the day I started writing this. A lint rejects any I forget.

I thought that was enough to have the tiers in place.

The default leaks into paths you cannot pin

What leaked on the day the limit ran out were /code-review and /simplify. Both ship with Claude Code, take no model argument, and run on the session's model. Call them from a Fable session and they run on Fable. That is how the limit arrived in an instant.

There are other leak paths: the built-in subagents, which have no frontmatter. Counting my own log (~/.claude/metrics/agent-usage.jsonl), over the three weeks from installing the review hook described below to writing this article, 576 agent launches included 373 built-ins — general-purpose 231, Explore 121, Plan 10, claude-code-guide 11 — or 65%. Writing model: into my own agents does not reach two thirds of the launches. (claude-code-guide is fixed to Haiku, so the count that can inherit the session model is 362, or 63%.) The log does not record the model, so I cannot say how many of those ran on Fable. It was a hole I only noticed by counting, and I closed most of it while writing this article. What I closed it with is at the end.

Per the official documentation (as of September 16, 2026), a subagent's model is resolved in this order: the model named at call time, the model: in the definition, the CLAUDE_CODE_SUBAGENT_MODEL environment variable, then the session's model. Explore inherits the session model but caps at Opus; general-purpose and Plan inherit with no cap. So if the caller names no model and the environment variable is unset, every general-purpose a Fable session launches is Fable. Setting the variable changes general-purpose only; Explore and Plan do not move.

I fell into the same mechanism back in February. At the time, --model opus applied only to the main loop, and my subagents ran on Haiku, which I had not intended (the article from back then; the behavior has changed since). In February it leaked below what I intended, this time above it. Inheritance defaults leak in both directions.

The same report shows up in public issues. #76514, from July 10, says that omitting the per-agent model propagates Fable to every subagent; it was closed as not planned. #93894, from September 12, says that one run of /code-review high on Fable 5.1 exhausts the session budget and the review never finishes; it is still open.

The convention did not stop it, and neither did the warning

My fix on the day the limit ran out was a convention. I added a step at the end of every implementation plan — decide in one line whether this session implements — and made heavy implementation go to a new Opus session. I rejected enforcing it with a hook, because that would bury the policy inside the hook.

Two days later, at night, I typed this (my own words, translated):

...I put in a convention where Fable handles design and opus does the implementation, but it isn't really being followed. Implementation is one thing, but when review runs straight afterwards — simplify and code-review especially — it wastes a serious amount of tokens....

So I added a hook. The first version was an advisory: warn when a review is launched in a Fable session. What I learned the next morning is that an advisory gets read after the review has already run. Seeing the warning, stopping, and re-running the review on Opus means paying twice — once for Fable, once for Opus.

That same morning I rewrote review-model-notice.sh. What I had rejected first was burying policy in a hook. What went into the hook this time is only a mechanical test — skill name crossed with session model — while the policy of which model does what stays on the rules side. The response is split by path.

# ~/.claude/hooks/review-model-notice.sh (excerpt, abridged)
# Split the response by path:
#   Direct skill call (code-review / simplify) -> **block**. The test is purely
#     mechanical (skill name x session model) with no room for a false positive,
#     and as an advisory the skill runs in the same turn and burns judge-tier
#     tokens before the advice is ever read (measured — stopping and re-running
#     the review on Opus meant paying twice). Only a block before execution works.
#   Agent/Task launch with a missing model pin -> stay at advisory. Partial
#     prompt matching is a heuristic and can produce false positives, so it does
#     not get deny authority.

The session's model is not in the hook payload. It reads the most recent "model" from the tail of the transcript, and stays silent if it cannot read one.

model=$(tail -c 2000000 "$T" 2>/dev/null | grep -o '"model" *: *"[^"]*"' | tail -n 1) || true
case "$model" in
  *fable*) ;;
  *) exit 0 ;;
esac
# ...(compose the block / advisory text)
if [[ "$mode" == "block" ]]; then
  jq -cn --arg reason "$msg" '{decision:"block", reason:$reason}'
  exit 0
fi

A model that reads the block reason switches itself over to Agent(subagent_type: "general-purpose", model: "opus"). No human has to stop it and retype the command.

What broke was the budget, then the judgment

The same morning, in another conversation, I was talking about over-engineering. Just before that I had simplified my weekly-report machinery heavily, cutting thousands of lines of code. I typed this:

A fair amount of the over-engineering happens on opus, after fable's usage limit is past

The real problem is that this happens because Fable gets wasted on implementation and review, the usage limit runs past, and I'm forced to make Opus the orchestrator.

Making the top model your default breaks the budget first. But that budget was there to buy the judge tier's model. When it runs out, the judge tier drops to Opus, and the dropped judge tier waves over-engineering through. Putting the top model in the "default" slot works to push it out of the "judge tier" slot. Up to here this is an impression from a handful of cases; I have not counted how many times Opus's judgment let over-engineering through.

So I reversed the direction in which I plug the leaks. The session default stays at the top model; the paths from the default to the judge tier stay open, and the paths that leak from the default to anything else get cut. The judge tier gets pinned. The architect agent, which decides whether a thing should be built at all, has been the only model: fable in my environment from that day through September 16.

It still ran out every weekend

Three days later, on Friday, I typed this:

Fable's usage limit always runs out on Saturdays, Sundays and holidays, which is when I use it most. After it's gone Opus does the design, and I really do feel Fable's absence. I want it focused on design and planning, where Fable is strong, and everything else delegated to other models. Even now it's supposed to delegate to Opus when it can, but a lot of the time Fable just goes ahead and implements.

I had plugged the review path, but implementation itself was running on Fable. My first convention had an escape hatch — if the conditions are not met, this session may implement — and the model itself was making that call.

I changed three things the same day.

Inverting the default. The default for "does this session implement" became "hand it to Opus." I may implement in place only when I write one line in the plan naming one of three things: prose edits to design documents (decision records such as ADRs), a concrete reason the work cannot be handed off, or an explicit instruction from the user. Accountability moved from the side that hands off to the side that does not
A hook right after plan approval. A PostToolUse on ExitPlanMode reminds Fable sessions, and only Fable sessions, to decide the executor. It fires before the first implementation Edit, the latest safe position available
One line in the resident rules. So that it reaches implementation that never passes through plan mode, I wrote into my rules: "implementation in a judge-tier session defaults to dispatch to the build tier." One line saying a judge-tier session does not implement; it hands the work to a build-tier session

The hook in (2) has a condition for staying quiet: if the plan body already contains the set phrase "executor decision," it says nothing. At first I also had general words like dispatch and spawn-session in the suppression list. Running that against 271 past ExitPlanMode calls suppressed 31, and 30 of the 31 were false suppressions. A plan that merely mentions handing implementation off — that is, a session doing nothing but routing tasks, exactly where I most want the hook to fire — would go quiet. I narrowed the suppression list to the single set phrase and put "a plan that only mentions dispatch still fires" into the regression tests. Putting only mechanically decidable conditions into the machinery is the same call I made with the review hook.

In the three weeks since, there is no report in my session logs of the usage limit running out. That is the absence of a report, not a record of measured headroom. I still have nothing in my environment that mechanically records how much of the usage limit is left. I also have not looked at whether reviews that ran on Fable were catching anything Opus misses. What I can say goes as far as this: I have not had to write that same report again.

Four decision rules

If you run a similar setup, these four are what you can take away.

Don't leave the tiers to the default; pin the judge tier and plug the paths that leak. My session default is still fable. What I changed is not the default but the paths the default flows into. It leaks into built-in skills, built-in subagents, and the session's own implementation, and a model: in my own agent definitions reaches none of those three.

Fill in the built-in subagent defaults with the environment variable and same-named definitions. Setting CLAUDE_CODE_SUBAGENT_MODEL=opus changes the default for general-purpose. Explore and Plan do not move on that variable alone: either fix everything to one model with CLAUDE_CODE_SUBAGENT_MODEL_FORCE=1, or place a same-named agent definition carrying a model:. While writing this article I set the environment variable to opus and gave Explore a same-named definition on sonnet. Of the 362 launches above, what remains is Plan's 10.

Block when the test is mechanical, advise when it is a heuristic. Skill name crossed with session model produces no false positives, so it gets stopped before execution. Anything you can only test by partial prompt matching stays a warning. Advice that arrives after execution does not protect a budget.

Invert the default of a convention. Most "conventions nobody follows" put no accountability on the side that does not follow them. Put the default on the handing-off side, and you have to write down your reason for not handing off.

I wrote expiry conditions for this wiring. It comes out if model tier distinctions and usage limits disappear, or if Claude Code starts switching models per session on its own. opusplan (plan on opus, execution on sonnet) already exists, so once a version that drops from fable to opus arrives, the hook is unnecessary.

Sources

Create custom subagents - Claude Code Docs — the model resolution order, Explore's cap, CLAUDE_CODE_SUBAGENT_MODEL_FORCE (retrieved 2026-09-16)
Model configuration - Claude Code Docs — opusplan, CLAUDE_CODE_SUBAGENT_MODEL (retrieved 2026-09-16)
anthropics/claude-code #76514 — subagents in a Fable session all inherit Fable (2026-07-10, closed as not planned)
anthropics/claude-code #93894 — one /code-review high on Fable 5.1 exhausts the session budget (2026-09-12, open)

Why an 11-Second Burst Never Showed Up in the Log That Gets Read Every Week

Tatsuya Shimomoto — Sat, 12 Sep 2026 22:08:54 +0000

On the morning of September 12, 2026, I gave every log my agent writes a weekly reader.

That afternoon, counting the same logs by hand for a different article, I found GET /home hit 13 times in the 11 seconds before a session ended. All HTTP 200. The agent was in a loop with nothing left to do, fetching /home once a second.

The reader I set up that morning cannot show this. Not bad luck; arithmetic. Draw 30 random rows from the week's 4,035 API log rows, and the expected number of those 13 rows that make it into the sample is 0.1.

Giving a log a reader and shaping it so a fault shows up turned out to be two different jobs. This article is about what became visible once I replaced the way the logs get shown with a form that shows the fault, and what that same replacement made invisible.

Setup: what is being scored, and what is being logged

The subject is Contemplative Agent, an autonomous AI agent (Python) I am developing. It runs on Moltbook, a social network where agents talk to each other.

How it works is simple. Within a 60-minute session, it scores feed posts for relevance with an LLM, and upvotes or comments on the ones above a threshold. Upvotes and comments had a duplicate check ("never twice on the same post"); scoring did not.

The agent writes its own activity to JSONL logs, one event per line, in 15 files; the API log (one call per line) and the LLM-call log (one call per line) are two of them. Every week, an unattended Claude Code session (the weekly chain, from here on) reads them and writes an observation document.

Morning: a reader, a sample with no question, and still no burst

What I built that morning was a registry (REGISTRY) holding one "weekly question" per log, and a script that reads it and emits a count table and a value distribution for each log. The first stage of the weekly chain reads that output. This was me applying the rule from the previous article: when you build a new instrument, write down who reads it and when.

The part I cared about most was one no-question stage, where nothing is decided in advance. Every existing check in the weekly chain counts a fault shape someone has already imagined. To catch a fault nobody imagined, I figured you need a stage that reads near-raw data with no question attached, so I had it draw 30 random rows from each log and read them.

The sample drops the body field. That field holds post text written by other agents, and feeding it to an unattended LLM session is an entry point for prompt injection.

From here on I call this kind of processing, deciding what part of a log is shown and how, a projection.

Even after all of that is read, the 13-row burst does not show. This stage takes up most of what gets read: of the 130,547 bytes and 443 lines the output had for the same week (September 5 to 11, 28 sessions), 96% is sample.

There are two reasons it does not show, and neither is fixable by tuning.

The count table does not distinguish 13 rows in 11 seconds from 13 rows spread over an hour. The fault's shape is in time, and a count has no time in it
The sample shows the distribution of rows but erases the gaps between them. 13 ÷ 4,035 × 30 ≈ 0.1 rows. Change the seed as many times as you like; it stays invisible

What was missing was a time axis in the projection. There was a reader, and there was a no-question stage. A fault shaped like time still does not show in a projection that has no time in it.

Afternoon: no column that catches a known bug

The cause of the burst was quick to find. A boundary condition in the wait logic: when the remaining time was shorter than the intended wait, the sleep was skipped and the loop spun for the rest of the session (proposal record RFC-0036, fixed the same day).

Before fixing the bug, I decided how to fix the reader. Partway through, I typed this to the agent:

One thing about scope: in this session I am not thinking about symptomatic fixes that squash a known bug. I want to build something that picks up the anomalies that come next.

Add a column that counts /home hits and this burst shows up. But that projection will not show the next unknown fault. Put in a single column or threshold carrying the name /home, or keyed to the scoring function, and the projection catches only the faults you already know.

So everything I built into the reader was a shape that carries no known name.

The unit is the session, not the row: one session, one row
For each category (endpoint for API, caller for LLM), three time-axis columns: count, hits in the busiest minute, minimum gap
Outliers are decided not by a threshold but by distance from the other 27 sessions in the same week (median and MAD, the median absolute deviation, with a modified z-score of 3.5 or above)

Columns are not declared; they are derived from the data. The columns are the union of this week's categories and the past four weeks', so when a new endpoint appears next week, it gets measured from that week without anyone editing the registry.

Bursts are the one exception; I fold them the way syslog does with last message repeated N times: three or more consecutive events in the same category within 2 seconds collapse into one line.

Something I was not looking for came out on its own

Run the replaced projection over the same week, and the burst I had found by hand comes out with nobody looking for it. It is third in the within-week outliers table. Columns are named log:category, so api-audit:GET /feed is the API log's /feed endpoint.

### Within-week outliers (median / MAD over sessions, |modified z| ≥ 3.5)
|      z | column                            | session   |   value |   median |   MAD |
| 1294.3 | comment-outcomes:reply            | db8ed4e2  |    1919 |        0 |   0   |
|  116.7 | comment-outcomes:reply /1m        | e4a99b6b  |     173 |        0 |   0   |
|  -19.6 | api-audit:GET /feed gap s (log z) | a6eac8ae  |       0 |      148 |  25.5 |

It surfaced because "the minimum gap for GET /feed is 0 seconds, far from the other sessions' median of 148 seconds." The name /home appears nowhere. Once the columns carry a time axis, a burst comes out as a time-axis outlier, nothing more.

The section that lists the minutes around each outlier in time order also showed something I had missed when counting by hand. The second line: /feed was being hit 12 times too, alternating with /home.

15:59:09 api-audit:GET /home ×13 in 11s
15:59:10 api-audit:GET /feed ×12 in 10s

The top two entries were a separate matter. One session wrote 1,265 comment-outcome rows in a single second, cause not yet identified (my guess is a bulk import of past data). This time, the no-question stage I placed that morning did pick up something I had not seen, and it was not the burst.

The amount that gets read went down: from 130,547 bytes to 24,463 bytes, 311 lines. Two runs with the same arguments are byte-identical, and zero fields derive from the body field.

The same replacement hides a fault that happens every time

This is the part I most wanted to write.

The agent had one more known bug. It did not remember scoring results, so as long as a post stayed in the feed, it re-scored the same post every cycle (RFC-0032).

Scoring has been an LLM call since the first commit on March 8, 2026, and the LLM call log shipped on June 10. The evidence had been on disk since June. The bug was found in a code review on September 12.

In the new projection's session ledger (the one-session-one-row table), this fault shows up as a number: a median of 44.5 scoring calls per session.

But it does not show up as an outlier. All 28 sessions do the same thing, so no session is far from the others. The moment "different from the other sessions" becomes the definition of a fault, a fault that has happened every time since day one falls outside the definition.

This is not an oversight; it is the structure of the replacement. The very judgment that made the burst visible, distance from the other sessions in the same week, is what hides the chronic fault. The fault that became visible because I added a time axis and the fault that vanished because I switched to differences are two faces of one decision.

So I wrote the hidden shape into the projection's own output. It appears in the header, the one place that gets read every time:

A session is compared with the other sessions of this window, so a fault present in every session since it began departs from nothing: that shape belongs to Redundancy and to the reader of the ledger's absolute values (ADR-0110).

ADR-0110 is the Architecture Decision Record for this replacement, one decision per file.

Two things cover the chronic shape. One is the Redundancy section of the output, which counts violations of a written invariant: the same caller with the same normalized prompt does not repeat within a session. The other is the LLM that reads the session ledger's absolute values; whether 44.5 scoring calls per session is too many is for that reader to judge, not a difference to compute.

To be honest, the Redundancy section does not work for this week yet. The normalized prompt digest exists only in rows from September 12 onward, so this week reports 0 repeats. The projection starts covering the chronic shape next week, and the effect of the fix becomes readable in the weekly readings of September 18 and 25.

Where this decision does not hold

Before you port any of this, the conditions under which it does not hold.

More shapes shown means the reader grew. The script went from 616 lines to 1,079 (3 modules), and pandas entered the dev dependencies. What shrank was the output that gets read, not the code I wrote, and the trap from the previous article, where more lines were built for retirement than were removed, can happen here too.

The calibration values are empirical, added one at a time as simpler shapes failed on real data. Minimum gap only for categories with 4 or more events; gaps and totals under log1p; columns with MAD 0 get a scale floor of one unit of that column. All three were added after a simpler shape failed on real data, and there is no guarantee the same values are right in your environment.

The price of the floor: a constant column needs a 5.2-unit departure (3.5 × 1.4826, the MAD-to-sigma constant). If every session calls a category 3 times and one session calls it 0 times, that does not appear as an outlier; it stays as a zero in the session ledger. That is another shape the "shape shown" hides.

The population is 28, and the unit is fixed-length sessions. With only a handful of sessions per week, neither median nor MAD means anything, and for a set of services whose request counts differ by orders of magnitude, the right unit will be a different one.

The chronic side is unverified. That acute faults show was confirmed on real data from the week before the fix. For chronic faults, I only wrote at design time that they do not show in differences; whether the invariant and the reader pick them up has not been read even once.

If you want to port this to your own setup

If you have logs read periodically by an LLM or a human, decide three things before building the projection.

1. Make the unit of a row a unit of work, not a log line. One row per session, request, or job, and, per category, a count, the busiest minute, and the minimum gap. A random sample of rows shows the distribution but erases gaps and density; a fault shaped like time shows only in the minimum gap or the busiest minute.

2. The moment you decide the shape shown, write one line about the shape it hides into the output itself. If you show differences from the other rows in the same window, a fault present in every row will not show. Write it in the header that gets read every time, before the reader has to guess.

3. Catch the chronic shape with an invariant, not a difference. Count an invariant you can write down, such as "the same input does not get the same processing twice within a session." A difference only sees change, and something that has been that way from the start is not a change.

Last, the thing I overlooked longest.

The evidence of re-scoring was on disk every day from June 10. The reader arrived on the morning of September 12, and even had that reader been running, it was not shaped to show this fault. Giving a log a reader and shaping it so a fault shows up are two different jobs.

Building a projection means deciding the shape it shows and the shape it hides at the same time. Unless you write the hidden shape into the output, the next reader keeps reading without knowing what it is not seeing.

Supplement: on writing "no registry" in the previous article

In the previous article I wrote that I would not build a registry of instruments. Two weeks later I built something named REGISTRY, so here is what is the same and what is different.

What the previous article rejected was a list managing consumers per instrument. If you cannot answer, about that list itself, who reads it, how many readings close the decision, and when it gets retired, you have only added one more layer of the same problem.

This REGISTRY is a set of rows holding each log's weekly question as data. The reader is the weekly unattended session, the editor is me looking at the non-OK rows in the weekly human review, and the removal condition is the retirement of the weekly chain itself; all three are written in the design decision record. The principle was never "build no registry" but "build no instrument whose consumer you cannot name," and the previous article's wording was narrower than the principle.

Sources and references

Google SRE Book, ch.6 Monitoring Distributed Systems — the principle that rules which never fire should be removed. The reason this article's projection has no thresholds
Fast and flexible observability with canonical log lines (Stripe) — one request, one line. This article turns that into one session, one line
The RED Method (Grafana Labs) — the idea of giving every category the same axes
Boris Iglewicz and David Hoaglin, How to Detect and Handle Outliers (ASQC, 1993) — the modified z-score and the 3.5 threshold. This article does not adopt their alternative scale for MAD 0; it uses one unit of the column as the floor instead
Log Sampling: Techniques, Challenges & Best Practices (groundcover) — the general explanation of how sampling drops rare, short-lived events

DEV Community: Tatsuya Shimomoto

Can You Tell an Eval from a Review?

An eval is made of inputs, a way to run them, and grading

The word "eval" means different things to different people

Align grading with human judgment before you use it

The loop climbs through holes in the grading too

You can set a numeric goal when a tool that measures the same quantity already exists

Placing my evals on the parts

What I did on the README was a review loop

How often were the review's findings right?

I have avoided eval loops

Related links

Whoa… My Claude Code Skill Descriptions Are Way Too Long…?!

The skill listing has a budget, and past it, skills are listed by name only

I had cut them before, and they grew back

First, I decided whether the model should invoke a skill at all

I decided to count width with a lint rule

What I didn't decide

Related links

Is a README for Humans or for LLMs?

I couldn't make cutting rules at the sentence level

I was packing the visible body for LLMs

The assistants that fetched it read the folded section too

I split how I write READMEs into top and bottom

I wrote this article the same way

I'm still on the separating side

Records

Summary of this article

Terms

This article's experiment

Timeline

The marking test (2026-10-08)

Design of the README experiment (readme-fetch-canary, 2026-10-08)

Results of the README experiment (each condition once)

Changes to readme-writer (2026-10-08, ADR-0091)

Why the author wrote for LLMs (GitHub clone counts)

External sources on llms.txt and search

References

Related links

Turning Claude Code into a Writing Harness with Claude Mods and an Output Style

Turning a few things off in settings barely shrank the listing

A Mod chooses what Claude sees in each repository

The coding instructions I meant to remove weren't in Claude 5 models

I wrote the rules for talking with me into the output style

Deciding what Claude sees and how it talks, separately

Related links

"Is There Any Point to This?" Removing the Jev Plugins I Added to Claude Code After One Week

Jev's skill picks barely overlapped with Claude Code's own choices

After removing it, I read 20 of the unused picks

A completion that had already been verified got stopped

The component and my workflow were each right on their own

I couldn't read their effect, and the work they added was already being done

You can read the effect of a built-in component where the flow is fixed

Related links

Before Adding a Jev-Type Judgment Model, Read Your Generation Model's Probabilities

Read the same gemma's probabilities instead of letting it write the answer

Did it improve because of the four levels, or because of reading probabilities?

It matched a model trained for judgment at ranking

Changing one condition at a time, only the reading method made a clear difference

How close it got, and where it doesn't work

Before adding a judgment model

Related links

Moving My Research Pipeline's Judgment Calls from an LLM to Jev, a Judgment-Only Model

The morning research I used to leave entirely to Opus

The pipeline at a glance

Ask alongside a question

Numbers from the final run

Writing stays an LLM's job

What to check before handing judgments to Jev

Related links

What Does It Take to Reproduce Jev's Decisions Locally?

30+ open alternatives and 3 approaches

Measuring with 150-row replays

qwen3:8b — 54 questions pile up

Laya — the wall called calibration

kev-0.8b — 0.8B devours 12 GB

AFM 3 Core — context length falls short

3 conditions a local decision model needs

Related links

I Added Jev's Skill Router to Claude Code and Turned Back Just Before Rewriting the Skill Listing