DEV Community: LYR

Are you still picking that setting by gut feel?

LYR — Mon, 03 Aug 2026 14:29:52 +0000

See the shape of the curve, not a single point — the habit called Sweep

Thresholds, image resolutions, batch sizes — most of them get picked by gut feel. Try one or two, take whichever looks better.

The moment verification got cheap, I stopped deciding that way. Instead of picking a single point, I sweep the range end to end and look at the shape of the curve. The sweet spot, the knee, the cliff — none of them show up until you measure the shape. A single point tells you nothing about the shape.

And whether you can sweep at all comes down to how cheap verification is. When one run is expensive, people fall back on gut feel. Make verification cheap first — that comes before anything else.

Engineering is full of magic numbers (concrete values sitting in the code with no clear justification behind them). Thresholds, image resolution, batch size, timeouts. How do you decide those values? To be honest — mostly, by gut. You try one or two at best and take whichever looks better.

When the harness made verification cheaper by an order of magnitude, I stopped deciding this way. Instead of picking one point, I started to sweep the range end to end and look at the shape of the curve. That is a Sweep.

A curve shows you what a single point can't

Take the image resolution used when OCR detects text. "Higher resolution means higher accuracy" — that's what intuition says. So if you're going by gut, you pick the higher one without hesitating.

But when I varied the resolution across a range and measured accuracy (CER, lower is better) and speed together, this is the curve that came out.

Resolution	Error rate (CER, lower is better)	Speed
960	9.4%	75 ms
736	10.7%	40 ms
480	35% (collapse)	18 ms

Figure 1: resolution sweep for OCR detection. Latency on the x-axis, error rate on the y-axis. 736 is the sweet spot for speed and accuracy: against 960 it trades +1.3pt of error rate for half the latency. Drop all the way to 480 and the error rate spikes into collapse.
Note: n=6, a small sample = a directional check.

The middle one, 736, was the obvious "sweet spot." Against 960, the error rate goes up by a mere 1.3 points (9.4% → 10.7%), and the speed halves. Drop to 480, on the other hand, and accuracy collapses (the error rate goes from 10.7% to 35% — more than tripling). The bend in the curve — the knee — sits exactly at 736.

Had I picked 960 by gut, I'd have gone on paying double the time for a sliver of accuracy. Had I picked 480, the text wouldn't be readable at all. This sweet spot only became visible once I measured both ends and the middle. Measure a single point and you will never find it.

Curves have a "knee"

There's one more thing only a curve will show you: the knee — the bend past which doing more barely improves anything.

When I swept how quality rises as you add more training data, reading-comprehension accuracy on the difficult cases went flat past a certain amount of data (a very small amount). However much data you add, the hard cases don't improve. Past the knee, the investment is wasted — a signal that a different lever (the model's capacity) was what it needed.

This too is invisible from one or two points. Only when you draw the curve do you see the boundary where "nothing beyond here works."

Why sweeping is possible now

Sweep is an old idea. But maybe not that many people do it thoroughly in practice. The reason is simple: it's expensive. Trying eleven thresholds by hand means eleven verification runs. That's a number of runs that makes you question your sanity.

What changed is that the harness took the human out of verification. Once you've baked the frame sequence to disk, you can run eleven configurations in parallel, with zero manual work. Where a person used to hold the device and record once per configuration, everything now runs in one batch. The moment verification got cheap, "sweep a surface and look at the shape" became more reliable and better value than "pick one point by gut." Sweep was the first use I found for the slack the harness created.

Lessons

Don't set magic numbers by gut. Thresholds, resolution, batch size — most of the time there's no reason why the value is that particular number.
Sweep the range and look at the curve, not one point. The sweet spot and the cliff only become visible once you measure the whole shape. The knee especially — the bend where "nothing beyond here works" — is the signal to move to a different lever.
Whether you can sweep comes down to how cheap verification is. When a single verification run is expensive, people fall back on gut feel. Make verification cheap first. Then sweeping becomes the obvious thing to do.

Next time: what you see when you read that cloud of swept points against several objectives at once (speed × accuracy × cost) — on to Pareto. Sweeping and reading the frontier are two sides of the same coin.

Appendix: raw data

Sweep	Result	Conditions & caveats
Resolution for OCR detection	960 = error rate 9.4% / 75ms, 736 = 10.7% / 40ms (sweet spot), 480 = 35% / 18ms (box detection collapses)	Portrait orientation, JA+EN, n=6 small sample = a directional check
Input dimensions of the text-recognition model	128×480 is Pareto-best across every dataset	3 datasets × 7 dimensions. The others either flicker or make timing worse
The knee in data volume	Comprehension on the hard cases goes flat past N≲25	Capacity, not more data, is the rate limiter. The knee was visible because I swept
Preprocessing width	The width that's optimal for short subtitles (Live) collapses when used on long lines (Page)	One fixed value breaks another mode = the optimum is mode-dependent

Originally published at LYR Performance Note #021 — part of the series “Measure & tune — measure it right, then fit the settings”. The full set is at lyr.jp/en/research.

A general-purpose model is too wasteful a vessel for subtitles

LYR — Mon, 03 Aug 2026 14:28:50 +0000

A floor compression couldn't fill — the next thing to cut isn't parameters, it's generality

My own model with no compression applied at all — raw, 8 billion parameters (8B) — still left 15 critical mistranslations out of 97. On the same hard cases, a general-purpose model more than three times bigger (27 billion = 27B) had 4. No matter how I varied the strength of the compression, those 15 didn't move.

Figure 1: taking compression all the way off didn't bring the floor back (97 Korean→Japanese hard cases, blind). The self-hosted 8B had 15 critical errors even raw, with no compression at all — statistically indistinguishable from medium compression (14), and from heavy compression too. The general-purpose 27B, meanwhile, had 4 — significantly fewer than the raw 8B (p=0.015).
Note: judged blind by a large model from a different family, in the same round. Effective n=97 after excluding items suspected of overlapping the training data. The counts are critical errors only (subject and meaning mix-ups and the like — translations that don't work as a subtitle), not average quality. The three compression levels are INT4 / INT8 / BF16 (= no compression), in that order. The difference between heavy compression and raw was also not significant (p=0.22).

So the thing setting this floor — where the worst case bottoms out — was the model's size (capacity), not the compression. Every time quality fell apart, I had suspected compression first, and acted on it — the clearest case being smoothing out compression error with a repair called rotation, which took catastrophic failures from 64% to 24%. That line of attack died here.

But before falling back on "make it bigger," there's one more suspect: the vessel (the architecture) itself. Subtitling is a job of short sentences, tight constraints and bounded context — and yet we're using, as-is, a general-purpose LLM designed for free generation of arbitrary length. The hypothesis: cut the waste that generality costs, and a smaller model than today's should produce better subtitles than today's.

And what I'm after isn't only speed. What LYR is aiming at is the experience of subtitles at the level a professional human translator would produce, running over the video you're watching right now. Professional quality itself is available — if you wait. But that wait stacks up: booking a translator, a rough pass, proofreading, timing, rights clearance — weeks after release, and for some titles months. And — by far the bigger share is work that never gets translated into your language at all. Professional subtitles only get made for the languages and titles where the economics work, so for a lot of people "wait and you'll get it" isn't even on the table. Which leaves the viewer choosing between not watching when they want to and settling for crude automatic subtitles. Real time, on the other hand, is available right now — if you give up on quality. It only becomes magic when both hold at once.

So what I'm watching are two things: the note a native speaker gave me — "the meaning comes through, but the emotion is stiff" — and the speaker's gender, obvious from one look at the screen and impossible to recover from the source text. My read is that these two only come within reach once you change the vessel.

Let me be honest about this. This piece isn't a record of measurements; it's a design hypothesis. I have a reasonable read on the architecture, but I have no data points on my own language pairs yet. So at the end I'll also write down what result would make me throw the hypothesis away.

Where things stand — out of borrowed AI, and small and fast as far as that goes

The setup, briefly. LYR is a product that translates text on your phone in real time. It gets used three ways — Live, translating the subtitles running across a video one line at a time; Page, translating the text on the screen in one batch; Manga, translating the dialogue in comic panels. One and the same translation model does double duty across all three.

With that, let me fold the road so far into one paragraph — with the caveat that this article is only about the model. In practice, most of the speed and cost came out of things outside the model — queues, distance, on-device processing, the number of calls themselves — and in plenty of cases those mattered more (I'll leave them to the infrastructure and OCR categories). Here I trace just one layer, the model, straight down.

Restricted to the model, the first step was lining up third-party translation APIs side by side and picking the fast one (some AI gets disqualified before you ever measure quality). Then I hit the wall that borrowed AI disappears through retirement and price increases, and moved to my own GPU. From there I confirmed that compressing to a quarter barely costs any smarts, and pinned down that what decides generation speed isn't the parameter count but the byte count of the weights. On the quality side, I took an un-fine-tuned small model from 42% to 84%, and got to where narrowing to the single point of Live subtitles lets an 8B draw level with a 27B.

So "you don't need a general-purpose giant; for one trick, a small model is enough" is already partly demonstrated. This article is about what comes after that. Can I push that narrowing to one trick past the training data, all the way into the vessel (the architecture)?

The hypothesis — foundation models are too multi-task for subtitles

Today's LLMs write poetry, write code, do math. To be that general-purpose, they're built to emit text of any length in any format.

The subtitle job is the exact opposite.

What a general-purpose LLM assumes	What the subtitle job actually is
Output length is free (a few words to dozens of pages)	1–2 lines. Bound to a length that fits the display time
Output format is free	The format is fixed. Return the translation, nothing else
Context can be made as long as you like	Only the few utterances either side
A response can take seconds	Real time, keeping up with the speech
Solve any task in the world	Produce one translation

I call this mismatch the generality tax. You pay it in two ways.

First, you pay it as wasted data reads. Every time the model emits one character, it re-reads its own insides (the weights = a vast pile of numbers) in full. That's why generation speed is decided not by how heavy the computation is but by how much data gets read (generation speed is decided by the byte count of the weights). The numbers that exist for poetry and code and math get read out too, on every single character of subtitle. You're paying freight, on every character, for abilities you never use.

Second, you pay it as a forced choice between speed and quality. As things stand, pick a fast model and the translation gets rough; pick an accurate one and it gets slow. That exchange rate looks immovable. But it isn't a law that comes with the subtitle job. It's just that the general-purpose vessel we happen to be using sits on that particular rate. Change the vessel and there's room for the rate itself to change (the Pareto view).

I don't think this is LYR's problem alone. Now that the "drop a foundation model into the product as-is" phase has gone around once, recutting the vessel per use case ought to be ground a lot of teams are stepping onto at the same time. I'm looking at it from the single vantage point of subtitles.

But "smaller at the same quality" isn't unconditional — two counterexamples

Let me put the measurements that are inconvenient for the hypothesis first.

Counterexample 1: specialization is a reallocation, not a free lunch. The same training that took Live (video subtitles) from a 90 to a 96 success rate knocked the other two — Page (whole-screen batch translation) and Manga (dialogue in comic panels) — down by 4 each (A small specialist drew level with the strongest model in service; the mechanism is mode mixing). As long as one vessel does double duty across several jobs, whatever you gain has been carved out of somewhere else.

Counterexample 2: there's a layer specialization can't fill. Handing the model perfect context didn't fill in deep comprehension ("add context and it gets smarter" was half a lie), and comprehending dropped subjects and idioms in Japanese demanded capacity itself (spotting the rate limiter in the language you're worst at). "Fixable by polishing" problems and "needs capacity" problems are different animals.

So the question becomes: of the gap that remains, how much is capacity, and where does the generality tax start? Saying "change the vessel and everything gets better" with the two still mixed together is giving up on diagnosis. So I separated them first.

The verdict — the floor was capacity, not quantization

The first move in separating them was ruling quantization out as a suspect.

First, quantization in one line. The insides of an AI model are a vast collection of numbers. Quantization holds those numbers on a coarser scale and repacks the model into something lighter (compressing to a quarter barely costs any smarts). It's like re-saving a photo as a JPEG — the size drops dramatically, and a little detail is lost. The production model runs in that compressed state, because it's faster and cheaper.

And every time quality fell apart, compression is what I suspected first. I've observed both cases for real: the compression conditions were mismatched between training and production, and compression was inflating catastrophic failures (the floor). In the latter, a repair called rotation took catastrophic failures from 64% to 24% — but it never reached the pre-compression level (12%). Was that remaining half leftover compression damage, or something else entirely? Which raises the question: if I take compression all the way off, does the floor come back? Put another way — with a raw 8B that hasn't lost a single bit of detail, do those critical errors go away?

I had it translate 100 Korean→Japanese hard cases (97 effective, after removing items that overlapped the training data) and judged them blind to which configuration produced which translation. The comparison was the same self-hosted 8B run at three levels — heavy compression / medium compression / no compression (raw) — against the general-purpose 27B. What I counted wasn't an average score but the number of critical errors — translations that mix up the subject or the meaning and don't work as a subtitle at all. Averages dilute exactly this kind of rare catastrophic failure until you can't see it (raise the floor, not the ceiling).

The result is Figure 1 at the top. Here are the numbers again, on their own.

Configuration	Critical errors (of 97)
Self-hosted 8B, heavy compression (the production setting)	no difference from raw
Self-hosted 8B, medium compression	14
Self-hosted 8B, no compression (raw)	15
General-purpose 27B	4

Here's how to read it. Taking compression off entirely didn't reduce the critical errors. So quantization is not what set this floor. Since the 27B keeps it to 4 on the same hard cases, what's left is the difference in model size. A spread of 15 against 4 is hard to explain as chance wobble.

That separated the levers available to me into three.

Lever	Where it acts	Status
Compression (quantization)	Generation speed = how much data gets read	Done as a quality move. It works on speed; it does nothing for the floor
Size (capacity)	The comprehension floor = critical errors	It works — but it's a move back toward the expensive "bigger is smarter" direction
The vessel (the architecture)	The generality tax = speed, and the quality ceiling (the cap on how good it gets at its best)	Untouched. This is where the next investment goes

The conclusion of the experiment is that I stopped betting on quantization. It's still live as a speed lever, but it wasn't a tool for raising the quality floor. The investment moves from compression to the vessel.

What I expect from specialization isn't only speed

Here's the point I least want misread. Talking about the value of a specialist model purely in terms of latency is too narrow. What I expect from recutting the vessel is three independent improvements.

① Emotional register. When I had a native speaker read translations off the actual device, the verdict that came back was: "the meaning comes through, but the emotional parts are stiff." This isn't a mistranslation. The meaning is right. And still, as a subtitle, it catches your attention — and in that instant the viewer steps outside the story. If the ideal translation is a subtitle you never notice is there, this isn't decoration; it's the main keep.

And register doesn't move however you tune the instructions (the prompt). I know by now that the only way is to retrain the model's insides (the register the 4B struggled with showed up at 8B is the same story). A general-purpose model's register is pulled toward the general-purpose average. Getting off that average is the first motive for changing the vessel.

② Accuracy from translating while looking at the video. How much better does the translation get if you hand it perfect information about the scene? I've measured that elsewhere. What worked condensed down to three things: the speaker's gender, how proper nouns are read, and dropped subjects (13 improvements against 0 regressions).

The biggest of the three was the speaker's gender. Japanese exposes who is speaking constantly, through pronoun choice (ore vs watashi) and sentence-final particles (~daze vs ~dawa), while the English source carries none of that information.

So when the model has to infer gender from the text alone, it drops to 34% right and 21% wrong. Wrong is worse than "don't know" — it invents a fact that doesn't exist and contaminates the translation.

But this isn't a hard problem in principle. Look at the screen and it's obvious who's talking. There's the saying that a picture is worth a thousand words. In translation it bites harder than that — some of those thousand words were never in the source text to begin with. A model that only follows text can re-read the missing information as many times as it likes and never recover it. No amount of stacked context reaches what one frame would settle. In fact, simply training gender information into the model raised the rate at which it correctly switches first-person pronouns and sentence endings to match a male or female context from 19% to 74% (a first demonstration on a separate evaluation set). As long as the vessel has no entrance except text, there's no way to put this information in. Building that entrance is the second motive for changing the vessel.

③ Latency. The third is the freight a general-purpose model pays on generation. That's known ground (covered elsewhere), so I won't repeat it.

What matters is that the three are independent of each other. Which means that if it works, you don't get an addition — the whole curve lifts.

The next vessel — a "top-heavy" subtitle-specialist model

The insides of a model split broadly in two: the Encoder (the reading side), which reads the input and grasps the meaning, and the Decoder (the writing side), which writes that out as a translation. Today's models give the two roughly the same thickness. And the subtitle job gets pushed through that same allocation.

What I'm considering is changing that allocation. A big Encoder, a thin Decoder. The subtitle job isn't "talk at length and freely," it's "render it short and correct" — so the natural allocation puts the weights thick on understanding and thin on writing out.

Figure 2: rebuilding the vessel — changing the allocation between Encoder and Decoder. On the left is today, with roughly the same thickness given to the Encoder (the reading side) and the Decoder (the writing side). Every character emitted re-reads all of it. On the right is the plan. The Encoder gets bigger to raise the comprehension floor (and gains a second entrance, video), and the Decoder gets thin. The model as a whole actually gets bigger. It still doesn't get slower, because the only thing that runs per character is the thin Decoder. What gets cut is generality, not capacity.
Note: the heights are schematic ratios and do not represent the actual parameter allocation. Both sides are at the same scale. This is a design at the plan stage; there is no implementation and no measurement yet.

The reason I expect this shape to work is that the number of times each side runs is asymmetric. The Encoder runs once per input; the Decoder runs on every character it emits. So thinning the Decoder drops latency alone, while the thickness of the understanding stays.

The three shifts each push a different axis. The aim isn't to pick a winner, it's to move the trade-off curve itself.

Figure 3: the aim isn't to "pick a winner," it's to move the curve. As long as you use a general-purpose LLM, speed and quality get traded along the same curve (solid line). Fit the vessel to subtitles and maybe that whole curve moves up — that's the hypothesis (dashed line).
Note: the dashed line is an unmeasured hypothesis, not measured points. The three points on the solid line are likewise a conceptual diagram showing relative position; the coordinates are not exact measured values.

Let me separate what's backed by evidence from what isn't.

What's backed by evidence. "Thick Encoder, thin Decoder" is an architecture already demonstrated in machine translation (there's prior work where even a single-layer Decoder beats strong methods in the same speed band). On the video side, I've confirmed for myself that feeding in gender information switches the register correctly (19% → 74%). None of the parts are new. The only new thing is binding them into one model around the single point of subtitles.

What isn't. That prior work was demonstrated mainly on English↔German, and Japanese↔English/Korean is the language combination this kind of speedup struggles with most (word order differs so much that the writing side carries more of the load). So there's no guarantee that "thinner without losing quality" carries over to my own languages as-is. My estimate is that the realistic answer isn't extreme thinning but something like keeping it at 1–2 layers + having it imitate a larger model's translations as material (distillation) + routing only the hard cases out to a larger model.

And the thing I should be most honest about: the upper bound on the lift is limited to what you recover by dropping generality. As Figure 1 showed, part of the Korean→Japanese floor is set by model size, and changing the vessel leaves it there. What the vessel buys back is speed and register; the comprehension floor can only be filled by raising size, or by a design that routes the hard cases away.

Open questions — I have a read on the architecture; I have no data points yet

Everything above is what I've thought through to some degree. What follows is what still needs thinking.

Move	What it would settle	Status
Sweep the thickness of the Decoder (the writing side)	Hold the Encoder fixed and vary the Decoder across 1, 2, 4, 6 and 12 layers, then draw the frontier of quality (register included) against speed. Measure how thin it can go in my own languages	Not started
Graft on a video entrance	Add a part that reads video to today's 8B, and see whether the speaker's register really is solved by video	Not started
Where the training data comes from	How to build material that pairs video with translations. The text-only recipe doesn't carry over as-is	Not designed

Let me put the falsification conditions up front too. If thinning the Decoder (the writing side) costs more quality than what dropping generality recovers — concretely, if the thinned vessel loses on quality to a general-purpose model at the same speed — I throw the hypothesis away. If adding video doesn't move the register metric, I cut the video shift loose. Because the three shifts are independent of each other, one of them failing leaves the other two standing. Rather than asking "is this whole plan right?" as a single question, I can verify them one at a time and drop only the ones that don't hold.

That's as far as I can go for now. The intuition that "foundation models are too wasteful for subtitles" has an explanation behind it — the generality tax — and a measurement that ruled quantization out as a suspect. But I don't yet hold a single piece of evidence that changing the vessel wins. Keeping that line sharp is, I think, the condition for measuring the next thing right.

Lessons

"Faster through compression" and "smarter through size" are different levers, and you must not talk about them as one. With compression off entirely, the critical errors stayed at 15; the model more than three times bigger had 4. The moment the quality floor turns out not to move, compression drops out of the quality toolbox — even though it stays alive as a speed move. Hold each lever together with where it acts.
Don't talk about a specialist model's value in terms of speed alone. A general-purpose model's register is pulled toward the general-purpose average, and a vessel that only accepts text has no entrance for video. Speed, register and video are three axes that act independently — and because they're independent, there's a chance of moving the whole trade-off curve rather than adding up gains.
Write "I don't have the data yet" plainly. Having a read on the design and having it work under your own conditions are different things. The prior work was demonstrated on a different language combination, and there's no guarantee it carries over as-is. Putting a hypothesis down as a hypothesis and fixing the falsification conditions first finishes faster than starting to build on momentum.

Appendix: raw data

A. Separating quantization from capacity (the data behind Figure 1)

Item	Detail
Task	Korean→Japanese, a 100-item hard-case set (items suspected of overlapping the training data excluded → effective n=97)
Comparison	Self-hosted 8B (INT4 / INT8 / uncompressed BF16), general-purpose 27B
Metric	Number of critical errors. Subject and meaning mix-ups and the like — translations that don't work as a subtitle. Not average quality
Judging	Blind, by a large model from a different family, same round

Configuration	critical	vs BF16
Self-hosted 8B, INT8	14 / 97	Statistically indistinguishable (almost no discordant pairs)
Self-hosted 8B, INT4	—	No significant difference (p=0.22)
Self-hosted 8B, BF16 (no compression)	15 / 97	—
General-purpose 27B	4 / 97	p=0.015 against 8B-BF16 (about a 3.8x difference)

How to read it: the floor doesn't move with or without compression, so compression isn't its cause. The gap to the 27B remains, so the cause is model size (capacity). → Quantization is done as a quality move; the investment moves to capacity / video / the shape of the vessel.

B. Why change the vessel (the three shifts, and the evidence so far)

Shift	Expected effect	Evidence so far	Not yet confirmed
Drop the generality tax	Accuracy, speed	A general-purpose LLM (decoder-only, reader and writer fused into one) is built for free-form generation of arbitrary length. Subtitles are short, fixed-format, bounded-context (how much data gets read decides speed)	Measurements in my own languages
Add video	Register, accuracy	The benefit of perfect context condenses into gender, proper nouns and dropped subjects (13 improvements, 0 regressions, context vs. capacity) / from the English source alone, gender is 34% right and 21% wrong / training on gender information takes the follow-through rate 19% → 74% (a first demonstration on a separate set)	The video-input graft itself
Thin the Decoder	Speed	"Thick Encoder + thin Decoder" is already demonstrated in machine translation (mainly EN↔DE)	JA↔EN/KO differ far more in word order and are the combination this speedup struggles with most. No guarantee of extrapolation

C. Caveats

Figure 2 is a schematic of the plan; there is no implementation and no measurement. The dashed line in Figure 3 is likewise an unmeasured hypothesis, and the points on the solid line are a conceptual diagram showing relative position — the coordinates are not measured values.
The n=97 in A is small. The p-values come from paired comparisons on the same set; read them as a hint about direction.
The upper bound on the lift is what you recover by dropping generality. Part of the Korean→Japanese floor is set by model size and remains after changing the vessel (→ the assumption is that a design routing hard cases to a larger model fills the rest).

Originally published at LYR Performance Note #032 — part of the series “Models — smaller, faster, sharper”. The full set is at lyr.jp/en/research.

Generation speed is decided by the byte count of the weights, not the parameter count

LYR — Mon, 03 Aug 2026 14:27:48 +0000

4B-INT8 and 8B-INT4 came out at the same speed — and get one thing wrong in how you measure, and 2.35x looks like "no difference"

Measured on the same GPU, 4B-INT8 (weights about 4.0GB) and 8B-INT4 (about 4.3GB) came out at nearly the same speed. 5.98ms against 6.50ms per token. Even though the parameter counts differ by 2x.

The reason is simple: every time it generates one token, it reads out the model's weights in full. What decides it isn't the amount of compute and isn't the parameter count — it's the number of bytes it has to read. Which is also why, when you pick a GPU, you look at memory bandwidth and not compute performance.

This law comes with premises, though. In fact, my first measurement got the method wrong and I misread a "should be 2x faster" as 1.3x — and under a different measurement method, a 2.35x gap disappeared, and then inverted.

Figure 1: the byte count of the weights against generation time per token. Color is the parameter count (green = 4B, red = 8B). The ordering swaps around — 4B, 4B, 8B, 4B — and yet generation time is cleanly monotonic in byte count. 4B-INT8 (4.0GB) and 8B-INT4 (4.3GB) in particular land at nearly the same speed, despite a 2x difference in parameters.
Note: a comparison held constant on the same GPU, batch=1, the same quantization kernel, and with LoRA merged.

"A 4B should be 2x faster" only came out at 1.3x

The first measurement went like this. Swap the 8B model for a 4B and you halve the weights you read, so it should be close to 2x faster. But what I measured stopped at 1.24–1.36x.

"Going to 4B doesn't buy you as much as you'd think" — if I had drawn my conclusion right there, I would have gotten the whole model-size decision wrong.

The cause wasn't the model. It was that the setups I was measuring weren't held constant. The 4B side alone was carrying a double handicap.

A different quantization kernel (one of them was a scheme that unpacks less efficiently on that GPU generation)
The adapter was still running bolted on the outside (every generation step pays an extra small matrix multiply)

Re-measure with the same quantization scheme as the 8B side, and with the adapter merged into the model itself — 2.3x. It had followed the law all along.

Hold the serving setup constant before you compare. A comparison that isn't held constant makes 2x look like 1.3x.

Every token reads all of the weights

Why does the byte count decide it? The mechanism is plain.

When a language model generates one token, it touches every weight in the model exactly once. So at batch=1 (generating for a single user only), the read out of memory dominates rather than the compute. The amount read is the byte count of the weights, so generation speed is inversely proportional to that byte count.

Figure 2: why the byte count decides it. Every token you make reads the weights sitting in VRAM from beginning to end, once. The compute units spend more time waiting for that read to finish than working, and what's rate-limiting is not compute but memory bandwidth. So 2.6GB of weights (INT4) is 4.04ms and 7.6GB (FP16) is 9.49ms — the time stretches in proportion to how much you read.
Note: this is the batch=1 case (generating for a single user). Process several users together and one read can be shared, which changes the story.

The same law carries straight over to quantization bit width. Bit width is byte count.

Precision	Weight size	Per token	vs FP16
FP16	7.6GB	9.49ms	1.00x
INT8	4.3GB	6.00ms	1.58x
INT4	2.6GB	4.04ms	2.35x

The size ratio is 2.92x but the measured value is 2.35x. The gap is what's left over — the parts that aren't quantized (reads and writes of the cache and the activations). Byte count decides everything, but there are reads and writes beyond the model's weights.

And as long as memory bandwidth is the rate limiter, the number to look at when choosing a GPU is bandwidth too. Not compute performance, and not VRAM capacity.

Get one thing wrong in how you measure and the 2.35x disappears

Now for the real point. Same model, same GPU, measured three times changing nothing but the measurement method. The results split three ways.

How it was measured	FP16	INT4	What it looks like
Stock inference path	40ms	39ms	Level (no difference)
Inference server (eager execution)	20.9ms	22.2ms	Inverted (quantizing makes it slower)
Inference server (graph execution)	9.49ms	4.04ms	2.35x (correct)

In the first, the fast quantization kernel never got used and compute was the rate limiter. In the second, the fixed per-step overhead dominated, and on top of that only the unpacking cost of quantization got added. Only in the third did the fixed cost come out and the bandwidth-bound shape appear.

The speed of quantization only shows up when "fast kernel × graph execution × batch=1 bandwidth-bound" all line up. Drop any one of the three and it doesn't just look like it "doesn't work" — it looks actively harmful. When you're reading benchmark numbers, check first that all three line up.

Open questions: a law has a domain where it holds

This law isn't unconditional. I've confirmed two conditions, by measurement, where it breaks.

The first is when you're handling several users at once. Process the generation in a batch and several requests can share a single read of the weights. Throughput scales with bandwidth, but the wait for any one user stays proportional to the byte count. Everything in this article is about the speed of generating for a single user.

The second is when the weights get too small. Quantize a small model hard and put it on a fast GPU, and the weight read finishes so quickly that fixed costs like kernel launch become the dominant term. In practice, 4B-INT4 came out at 168 against 177 tok/s across the top two generations of data-center GPU — barely any difference at all, even with 3.3x the bandwidth. Measure the same 8B-INT4 on a previous-generation consumer GPU and it ran at exactly half the speed of the 4B (= plainly bandwidth-bound).

So don't try to confirm this law with a comparison that spans GPUs. If one side is bandwidth-bound and the other is fixed-cost-bound, the same law will hold in one place and fail in the other.

Lessons

Generation speed is decided by the byte count of the weights, not the parameter count. 4B-INT8 and 8B-INT4 came out at nearly the same speed. So design around "light in bytes" rather than "a small model," and choose GPUs on memory bandwidth rather than compute performance.
Hold the serving setup constant before you compare. A difference in nothing but the quantization kernel and how the adapter was handled turned 2x into 1.3x. Before you attribute a gap to model size, hold the way it's run constant.
A law has a domain where it holds. Step outside the batch=1, bandwidth-bound premise and neither size nor bandwidth buys you anything. In comparisons that span GPUs, suspect that the premise itself has changed.

This is the manifesto's first principle — Measure First — in its inference-performance form. Same model, same GPU, and still, one choice in how you measure makes 2.35x look like "no difference," or like an inversion.

Appendix: raw data

All measured at batch=1 (generating for a single user). TPOT = generation time per token.

Item	Measured	Conditions & caveats
Monotonicity of bytes and speed	4B-INT4 2.6GB / 4.04ms < 4B-INT8 4.0GB / 5.98ms ≈ 8B-INT4 4.3GB / 6.50ms < 4B-FP16 7.6GB / 9.49ms	Same GPU, same quantization kernel, adapter merged. Monotonic in byte-count order, not parameter-count order
Effect of halving the parameters	8B-INT4 76 tok/s → 4B-INT4 172 tok/s = 2.3x	With the same quantization and merged adapters held constant. Consistent with 4B-INT4 at 166 tok/s on a different GPU
Mis-measurement (setup mismatch)	4B served with a bolted-on adapter + a different quantization scheme → 91 tok/s = 1.24–1.36x over the 8B	A double handicap: the efficiency gap between quantization kernels plus adapter overhead. The 8B side was partly under the same conditions, so it's asymmetric
Effect of bit width	FP16 105 / INT8 167 / INT4 247 tok/s	Against a size ratio of 2.92x, the measured value is 2.35x. The gap is the reads and writes that aren't quantized (cache, activations, embeddings) plus fixed kernel cost
The measurement trap	stock path 40 / 39ms (level) → eager execution 20.9 / 22.2ms (inverted) → graph execution 9.49 / 4.04ms (correct)	The 2.35x only comes out once fast kernel, graph execution and batch=1 bandwidth-bound all line up
Outside the domain (fixed-cost-bound)	4B-INT4 came out 168 against 177 tok/s across the top two generations of data-center GPU (no response to 3.3x the bandwidth)	The weights are too small, so fixed costs like kernel launch dominate. 8B-INT4 is plainly bandwidth-bound on a previous-generation consumer GPU
Side benefit	Once the quantization schemes were held constant, a bug where characters from another language crept in also went away	The quantization scheme can affect not just speed but the language consistency of the output

Originally published at LYR Performance Note #030 — part of the series “Models — smaller, faster, sharper”. The full set is at lyr.jp/en/research.

Let another AI write your teacher data

LYR — Mon, 03 Aug 2026 14:26:47 +0000

How I had a large AI, not people, write the "gold data" that trains a specialist

The "gold data" that trains a specialist AI is something humans write — I threw that assumption out. I had a large AI write it, and a small AI copy it out. Japanese→English, the direction the small model was worst at, jumped 42% → 84% on straight imitation alone. Zero human translators.

The ceiling above that sat not with the teacher but with the student's comprehension capacity (the teacher's own translations passed almost across the board). Lump every ceiling together as "the teacher is bad" and you pick the wrong move.

And one pitfall. Hand the teacher surrounding context with the best of intentions, and the student learns to invent subjects. Build the teacher data under the same input conditions as production — the way kindness backfires only becomes visible once you measure.

Training a small AI into "a specialist at one job" takes a large volume of gold data. A stack of model answers saying "for this input, this translation is the right answer." For translation, that's a pair — the source text and its model translation.

And nearly everyone assumes the same thing — that those right answers are something humans produce. You hire professional translators, have them translate sentence by sentence, and use that as the model answers. The quality is solid, but it's slow. It's expensive. And for low-resource languages you can't get hold of a professional in the first place. This "produce the gold data" step looks like the biggest bottleneck in training a specialist model.

I threw that assumption out. I had the gold data produced by another AI, not by people.

Have the large AI write the model answers

The method is simple. What I want to train is a small, lightweight model (4B = 4 billion parameters). What I have on hand is a far larger, far smarter general-purpose model (27B class).

So — have the large one write the right answers and the small one copy them out. Pour the source text into the large model, take the translations that come out as "model answers," and have the small model imitate them. In machine learning this is called distillation. The side producing the right answers is the teacher model; the side learning by imitating it is the student model. The title of this article — "Let another AI write your teacher data" — means exactly what it says: have the teacher model write the gold data (= the teacher data), and have the student copy it out.

The effect showed up plainly. The un-fine-tuned small model was hopeless at Japanese→English, with accuracy of about 42%. It got more than half of them wrong. So I had the large teacher write about 14,000 translations (clean, a single generation at temp0.1, and — as described below — with no context), rolled those into about 5,000 training examples, and had the small model imitate them —

Accuracy: 84%. +42 points.

Worth adding: this 84% came out with no elaborate selection in between, just imitating the teacher's plain translations as they came. Tricks like "generate N candidates and pick the good one" are the next lever, the one you pull after this. Straight imitation on its own gets you this far.

(The scoring is a reference-free binary judge. Rather than a human yardstick, an independent AI scores each translation "usable / not usable," and I compared before and after training under identical conditions. These are point estimates from a single run, so the absolute values wobble by a few points.)

One training run took a model that was basically an amateur up to a level that draws level with the large general-purpose model. Not one human translator was hired. An AI mass-produced the gold data.

The pitfall — imitation has a ceiling

It would be a nice story if it ended there, but reality kept going. This approach has a clear ceiling. The 84% on Japanese→English plateaued right there. Look into the remaining 16% and most of the failures wore the same face — hallucination. Inventing a subject that isn't in the source, supplying an object that was never there. As prose it's fluent and looks plausible enough. But it says things the source doesn't say.

This is where attribution goes wrong. "The teacher taught it lies, so the student lies too." But for Japanese→English, that wasn't it. The teacher (27B class) is strong at Japanese→English: score the teacher's translations with an independent binary judge and they pass almost across the board. The one getting it wrong wasn't the teacher — it was the student. Hand the 4B student a hard source sentence with a dropped subject and an idiom tangled into it, and it can't read it all the way through; trying to make the pieces add up, it slips in a fluent lie. So this ceiling didn't come from the teacher: it was hallucination coming out of the 4B student's "comprehension capacity." Which means no amount of cleaning up the model answers, and no amount of piling on data, moves the capacity wall itself (KTO and CPO below run into the same wall).

That said, "the teacher's lies are inherited by the student" isn't a thing that never happens. It happens clearly in two other situations. One is when the teacher itself is weak in the language — move into a language it's weak in and the bigger the general-purpose model, the more abruptly it falls apart. There the teacher's model answers become the rate limiter, and the more the student imitates, the more it gets contaminated. The other is "when you hand the teacher context," covered in the next section. Both are genuine teacher-side contamination, confirmed by measurement. What the ceiling actually is differs from situation to situation — lump them together as "the teacher is bad" and you pick the wrong move.

The teacher's first shot isn't the ceiling — best-of-N

Even if the Japanese→English ceiling is on the student's side, there's still headroom on the teacher's side. Because the teacher's "first shot" isn't necessarily the best the teacher can do.

Have the teacher translate the same source N different ways and take only the single best one. This is best-of-N. The yardstick for picking isn't agreement with a reference translation (a human's model answer) but QE (Quality Estimation — scoring the quality of a translation without looking at a right answer), which looks only at the quality of the translation itself. The payoff and the pitfalls of this selection mechanism get a separate article, so I'll keep it to the essentials here.

In my comparison of levers, best-of-N selection is where I've placed the highest-ROI "next move" (projected +10 to 15 points). But to be honest, that isn't a measured result from running the Japanese→English track end to end — it's my read from the design. Where I have actually made best-of-N work and measured it is a different mode, Live: there, retraining on re-picked model answers gave +2pt (and +6pt, 90→96, together with the preference training stacked on top) (details). The value of "picking from candidates" is real, but I should be clear that the double-digit pt on Japanese→English is still a projection.

Selection is all about the "yardstick"

best-of-N is not a cure-all. If the yardstick you select on is off, you end up worse than not selecting at all. On a different dataset I had it select on "closeness to the official subtitles," and the alignment-based selection (73–77%) lost to the plain single generation with no selection at all (82%). The official subtitles I used were a translation in a different direction, from a different source language, and every bit the candidates were optimized toward them was fidelity to the source given up. A QE judge, which never looks at the reference translation, does the opposite: it correctly fails those differently-directed official translations, and avoids the trap automatically.

The details are in the separate article, but the lesson is one line — it matters less how many candidates you roll than whether the yardstick you select on points at the kind of good you actually want.

KTO slipped; my bet is on CPO

After the student has copied out good model answers, there's one more stage of polish. Show it a "good translation" and a "bad translation" and train the judgment itself — "this one is better." That's preference learning.

I tried KTO (Kahneman-Tversky Optimization) first. You attach nothing but a binary good/bad label to each individual translation — a method that's light to implement. I drew up the design, ran it through the pipeline, and trained.

The result — almost nothing. 84% became 85%. That sits inside the judge's margin of error. The content of the translation changed in 14 of 100 cases, with zero regressions among them, but not one instance of the main cause of failure (hallucination from the student's capacity) went away. A signal as coarse as binary good/bad can't fill in the 4B model's comprehension ceiling — the controlled experiment put that in my face.

So I moved my bet. To CPO (Contrastive Preference Optimization). Instead of a bare binary, CPO ranks several candidates — {teacher translation, student translation, reference translation} — and trains on the difference itself, "A is better than B." It's a method with a track record on translation tasks (ALMA-R — the study that demonstrated CPO for translation), and it lets you etch in the "gap" between the good translation you picked with best-of-N and the ones you didn't. But let me be honest about this: I have not yet run CPO end to end on the Japanese→English track — I've decided to adopt it on the strength of ALMA-R's results, and that's as far as it goes. If I have to point at measured evidence that preference learning actually worked, it's in a different mode, Live, where Live-CPO contributed to the 90→96 push (details).

The lesson is that with preference learning, how you do it makes all the difference in the world. Same "teach it good from bad," and the coarse binary (KTO) slips, while the ranked version that shows the difference (CPO) is where I'm putting my money. But to press the point: CPO doesn't get over the comprehension ceiling itself either. It's a region where a difference only appears once you pile up an order of magnitude more (preference data in the tens of thousands), and really breaking that ceiling means going deeper into how the distillation is done. Honestly, I haven't climbed that one yet.

The backbone, in order: first build the foundation with straight imitation (42→84), then re-pick better model answers with best-of-N selection (the projected highest ROI), and on top of that etch in the differences with CPO.

An incidental finding — "give it context and it gets smarter" was backwards

Let me record one more moment where an assumption flipped.

If you want the teacher to write good right answers, surely handing it the surrounding context makes it translate more intelligently — that's the natural thought. Human translators translate more accurately with context too.

But when I trained the student on model answers produced with context injected into the teacher's prompt, the student learned to hallucinate subjects. Subjects the teacher had filled in from context, the student now filled in on its own — in production, where there is no context. At actual translation time the LYR app doesn't hand the model that kind of conversation history the way the teacher got it. Training conditions and production conditions were mismatched.

The fix was to deliberately withhold context from the teacher as well and have it write the model answers with "no context," exactly as in production. Kindness meant to make it smarter backfires in production. Build the teacher data under the same input conditions as production — an iron rule I only learned by measuring.

Thin and wide — 61 languages at once

One last property of this methodology, and the one that matters most in business terms.

Having an AI write the right answers makes adding a language essentially free. With human translators you'd re-hire a specialist per language; the teacher model spits out model answers for 61 languages at zero additional labor cost.

In practice, a single training run covered all 61 languages, thin and wide. The result was a clean gradient in resource volume.

Language	Approx. pass rate	Where it sits
Spanish	~91%	plenty of parallel text = easy
Arabic, Vietnamese, Portuguese	~82%	mid-resource
Korean, Thai	~77%	mid-resource
Hindi	~59%	somewhat low-resource
Swahili	~9%	lowest-resource = the data floor

(Same training run, held-out evaluation (scored on unseen data not used in training), binary judge. The point estimate for each language wobbles by a few points.)

The pass rate is a direct gradient of "how much parallel text exists for that language on the internet." The easy languages come out high whether you do anything or not; the weaker the language, the thinner the data both teacher and student have to work with, and the lower it lands. And while hitting 84% on Japanese→English, it also held 80% on English→Japanese (it doesn't get locked into one direction).

That gradient is also a map that shows at a glance where to invest next. The languages at the top are already fine. The bottleneck is always the weakest language. But — what the rate limiter is there can't be known without diagnosing the content of the failures language by language. Is it the student's capacity, the teacher's quality, or does the base simply not have the language at all? You can't lump it together as "the bottom of the list is always the teacher's fault" (Swahili is a case where the base doesn't have the language itself = the data floor). How to run that diagnosis, and the different moves each rate limiter calls for, are covered in a separate article.

Once you've laid down this "teacher-data production line," the rest is knocking out the weak languages one at a time. The fuel for training a specialist — gold data — keeps getting supplied thin, wide, and cheap by an AI instead of by people.

Lessons

Gold data can be produced by an AI instead of by people. Have a large teacher model write the model answers and a small student copy them out (distillation). On Japanese→English, the direction it was worst at, straight imitation alone took accuracy from 42% to 84% (+42pt). Zero human translators. The step usually taken to be the biggest cost of training a specialist — producing the right answers — can be automated. (The manifesto's second principle, Hunt the Constraint: get past the bottleneck by a different route.)
What the ceiling actually is differs from situation to situation. Stalling at 84% on Japanese→English was hallucination from the 4B student's comprehension capacity, not from the teacher (the teacher's own translations passed almost across the board). What does get inherited from the teacher's lies are "languages the teacher is weak in" and "when you hand the teacher context" — and those I confirmed by measurement. Lump every ceiling together as "the teacher is bad" and you pick the wrong move.
Build the teacher data under the same input conditions as production. Hand the teacher context with the best of intentions and the student learns to invent subjects. A mismatch between training conditions and production conditions turns straight into a defect. And the next move for pushing quality up is selection, not generation (best-of-N) — but select on quality alone, because selecting on closeness to a reference translation pulls you off in a different direction.

Appendix: raw data

Shared conditions: student = my own lightweight model (distilled 4B). Teacher = a general-purpose large model (27B class). Evaluation compares before and after training under identical conditions with a reference-free binary judge (PASS/FAIL). Point estimates come from a single run = absolute values wobble by a few points.

Item	Result	Conditions & caveats
Distillation SFT (imitation)	base 4B Japanese→English ~42% → 84% (+42pt)	Teacher generated 13,786 translations (temp0.1, no context = production parity, no selection) → 5,090 training examples. English→Japanese held at 80% too
Where the ceiling sits	The teacher's translations pass almost across the board = the one getting it wrong is the student	The hallucination is bound by the 4B's comprehension capacity
Multilingual held-out (61 languages)	Spanish 91 / Arabic, Vietnamese, Portuguese 82 / Korean, Thai 77 / Hindi 59 / Swahili 9	Pass rate = a gradient in resource volume. Swahili's 9 is the base barely having the language at all (diagnosis)
Selection (best-of-N QE)	Projected highest ROI in the lever evaluation (+10 to 15pt), not yet run on Japanese→English	Where it worked in measurement was Live's SFT-aug +2 (+6 including CPO)
gold-alignment trap	Anchoring on the reference translation pulls you off in a different direction and falls apart	reference-free QE correctly fails the non-aligned official translations and avoids the trap automatically
Preference learning (KTO)	null: 84→85% (within judge noise)	14/100 translations changed, 0 regressions, main cause of FAIL unchanged. A binary signal doesn't fill in the comprehension ceiling
Preference learning (CPO)	Adopted, but not yet trained on Japanese→English	Grounded in ALMA-R's results. The measured case is Live-CPO's 90→96. The real difference surfaces at the tens-of-thousands scale
Handling of context	Train on model answers made with context handed to the teacher, and the student learns subject hallucination	Production doesn't hand over conversation history under the same conditions = the teacher data has to be built with no context

Related: The details of quality selection are in Letting an AI pick a translation better than the "right answer". The basics of training a small model on this teacher data are in Training a small AI into a specialist at one job. Diagnosing the weakest language, where the teacher itself becomes the rate limiter, is in Spotting the rate limiter in your weakest language.

Originally published at LYR Performance Note #020 — part of the series “Models — smaller, faster, sharper”. The full set is at lyr.jp/en/research.

A small specialist drew level with the strongest model in service

LYR — Mon, 03 Aug 2026 14:25:45 +0000

You don't have to fight the giants head-on — narrow to one task and sharpen it, and a far smaller model can draw level. It just isn't free

A model about a third the size (8B) drew level with the strongest class currently in service (27B). But only after narrowing to a single task — Live subtitles. 90 → 96, tied with its own teacher.

That draw came with a price tag. The price of taking Live up +6 was Page and Manga each breaking by −4. Specialization isn't a free lunch; it's a choice about allocation.

Which means the question to settle first isn't "should I specialize" but "which mode is the main battleground, and what do I do with the others." The ways out — remixing the data, per-mode routing, scaling capacity — each cost something different in shape.

There's an unspoken pecking order in the world of machine translation. The more parameters, the smarter. So if you want the best quality, use the biggest model.

The teacher model LYR runs in production is a general-purpose model with 27 billion parameters (27B) — one of the strongest currently in service. The specialist model LYR raises in-house has about a third of that: 8 billion parameters (8B).

How far could I push that 8B on Live subtitles — the job of translating the one-line-at-a-time captions running along the bottom of a video, in real time? That's what this article is about. The short version: the 8B drew level with the 27B. But that draw came with a price tag.

The assumption — "a small model can't beat a big one"

Here's where it started. I measured the rate at which Live subtitle translations actually pass as usable (overall = the share where meaning and naturalness are good enough for real use), side by side across models.

Model	Live	Page	Manga
8B (sharpened for general use)	90	93	92
27B (in service, the teacher)	96	98	99
Scout (the predecessor)	97	86	90

On Live, the 8B was at 90. The 27B at 96. A 6-point gap.

Read plainly, that looks like a difference in capacity. The 8B's capacity is too small to handle Live's hard cases — idioms, picking the wrong subject. If that's true, there's only one move: make the model bigger. It's a tempting conclusion.

But is that 6 points really "the capacity wall"? Or is it a question of how it was sharpened? Running to a bigger model without checking is sprinting in the most expensive direction available. So first I looked at what the failures actually were.

Measured — the gap wasn't capacity

Opening up the Live items the 8B was failing on, one by one, the culprits came in three kinds. Literal translation of idioms (rendering "isn't like you" word for word), picking the wrong subject, and word hallucination (adding words that were never in the source).

Of those, idioms and subjects are not a question of capacity. They're the kind of mistake that goes away if you show the model enough correct model answers. So maybe what the 8B lacked wasn't capacity but high-quality model answers specialized for the Live job.

So I sharpened it in two stages.

Stage one: make the model answers better. Have the 27B teacher translate the same sentence N different ways, then pick the single best among them with quality estimation (QE = automatically scoring a translation by looking at the translation alone). The translations picked out by this "best-of-N QE" become the 8B's model answers (→ Getting the AI to pick a translation better than the gold reference). That's what makes the teacher data for idioms and subjects high quality.

Stage two: teach preference with pairs of good and bad translations. Build Live-specific pairs — "this translation is good, this one is bad" — and bake them in with CPO (training that corrects the model's preferences directly from contrastive good/bad example pairs). Live's pairs draw the sharpest line between good and bad, and the separation in training was where it worked hardest.

The result —

Model	Live	Page	Manga
8B (sharpened for general use)	90	93	92
8B (Live-specialized)	96	89	88
27B (in service, the teacher)	96	98	99

Live went 90 → 96. Tied with the 27B (96), and within a hair of Scout (97). The 6-point gap had gone to zero. And register held at 98 — still 27B-class (99). Split into its parts: the higher-quality model answers gave +2, the Live-specific CPO +4. What closed the gap wasn't capacity, it was specializing on the single point that is Live.

A model about a third the size had drawn level with the strongest model in service. If the story ended here, it'd be a satisfying win. But there are columns in that table I can't pretend not to see.

Specialization isn't free — it's a choice about allocation

Look at Page and Manga on the Live-specialized version. Page 93→89, Manga 92→88. Both broke by −4.

Figure 1: Before and after Live specialization (overall % across the three modes). Live rises 90→96, but the other side falls: Page 93→89, Manga 92→88. On a zero-based y axis all three modes cluster at a high level, so the changes are small — but they point in opposite directions.
Note: the y axis is overall %, from zero. Point estimates on the same evaluation set; the number of evaluation items (n) wasn't recorded.

The training that took Live up +6 dragged Page and Manga down −4. This isn't an accidental side effect. The 8B does double duty across three modes in a single model, and when you skew it toward one side (Live's single-line, one-at-a-time subtitles), the distribution for Page/Manga — which translate a block at a time — gets pushed out. I observed the same shape of breakage — a regime shift (the training data's distribution skews to one side and the mode on the other side falls apart) — over and over: with Manga-heavy data, and with training that included context. The mechanism of mode mixing itself, with all the numbers, is in Don't stop at measuring a single mode. The −4 from Live specialization here is one instance of that general rule.

So Live 90→96 is not, properly speaking, "6 points out of nowhere." It's carved out of Page and Manga and handed to Live — a reallocation. And not even a clean, even trade at that (Live +6, against −8 combined across Page and Manga). Specialization is not a free lunch. It's choosing where to put the weight inside a limited capacity. The right posture is to suspect that whenever something goes up, something you didn't measure has gone down.

So what do you do with that −4?

Once the trade-off is visible, the move stops being a binary of "specialize or don't." It becomes a problem of designing the allocation.

Remix the Live data back into the general data and retrain — instead of adding single lines as single lines, blend them in a form that doesn't wreck the block distribution, and dilute the interference.
Split the airframe per mode — Live gets the specialist airframe, Page/Manga the general-purpose airframe, switched by role. Don't make them fight over the same capacity.
Scale up capacity — the only unconditional way to eliminate the interference at the root, but it's also a return to the expensive direction of "bigger is smarter." Reaching straight for capacity on problems that sharpening fixes, like idioms and subjects, is giving up on diagnosis. (Some jobs do genuinely demand capacity, though — deep reading comprehension in Japanese, for one. Where specialization can't reach, I'll leave to Spotting the rate limiter in the language it's worst at.)

Whichever one you pick, the starting point was the same: don't say "I won" while looking only at the targeted mode. Had I stared only at Live's 96, I'd have gone around announcing "the 8B has caught up with the 27B" without ever noticing the −4 on Page and Manga. That's half the truth at best.

Lessons

You don't have to fight the giants head-on. Instead of challenging 27 billion parameters at general-purpose work, narrow 8 billion onto a single point — Live subtitles — and the success rate goes 90 → 96, drawing level with the strongest model in service. At about a third of the capacity, you can match it on the one trick you aimed at.
But specialization is a choice about allocation, not a free lunch. The same training that took Live up +6 broke Page/Manga by −4. As long as one model does double duty across several jobs, whatever you gained was carved out of somewhere else (and not necessarily as a clean, even trade).
So don't decide win or lose from the metric you targeted alone. Whether specialization worked can only be judged by laying the target's gain next to what the non-targets sacrificed. Measuring one side only fakes a success.

Appendix: raw data

Shared conditions: self-hosted 8B model (a distillation line taught by the 27-billion-parameter 27B general-purpose model). The three modes — Live / Page / Manga — measured side by side on the same evaluation set. Numbers are overall (= the share of translations good enough for real use, %); listed separately is register (how natural the voice sounds, %). Judged by Claude (Max plan) in a single judge round. The number of evaluation items (n) wasn't recorded, so read each % as a point estimate — a signal, not a settled result (nothing here is established at a significance level. The direction of the −4, however, has been observed repeatedly when specializing other modes → Don't stop at measuring a single mode).

Step	Detail	Quantity & conditions
L-1	Merge the 8B (30k distillation)	—
L-2	Additional Live SFT (further training on model answers selected by best-of-N QE)	Live gold 522 items, loss 0.34 / acc 0.95
L-3	Live-CPO (correcting preference directly from contrastive good/bad examples)	516 Live pairs, rewards/acc 1.0 = the sharpest good/bad separation
L-4/L-5	Evaluation → judge	Same evaluation set, same judge round

Main results (overall / register, %, same set):

Model	Live	Page	Manga
8B general CPO	90 / 100	93 / 98	92 / 98
8B LiveSFT	92 / 99	92 / 98	87 / 98
8B LiveSFT+CPO	96 / 98	89 / 100	88 / 100
Scout (the predecessor)	97 / 98	86 / 91	90 / 96
27B (new, the teacher)	96 / 99	98 / 100	99 / 100

Point	Number	Conditions & caveats
Live 90→96	Tied with the 27B (96), within a hair of Scout (97)	Breakdown: SFT-aug +2 (idioms / register improvement), Live-CPO +4 (reweight). register 98 holds 27B class (99)
Remaining defect	Word hallucination persists	Outside the coverage of a small 522-item aug. The gains on idioms / register outweighed the hallucinations left over, reaching overall 96
Trade-off	Page 93→89, Manga 92→88 (−4 each)	Live-specialized training drifted the training distribution toward the Live side

The general rule of mode mixing (the −4 regime shift, reproduced with three moves on 2026-07-14): beyond Live specialization, the same shape of degradation on the opposite side showed up with Manga-heavy data and with context-included training. All the numbers and the mechanism (drift in the learned distribution of entry counts / styles) are deferred to Don't stop at measuring a single mode. The −4 from Live specialization here is one of those three moves.

Ways out: ① retrain with the Live data mixed into the general set without wrecking the distribution, ② per-mode routing (Live = the specialist airframe / Page-Manga = the general-purpose airframe), ③ capacity scaling (the only move that unconditionally eliminates the interference, but expensive).

Originally published at LYR Performance Note #019 — part of the series “Models — smaller, faster, sharper”. The full set is at lyr.jp/en/research.

Let the AI pick a translation better than the "right answer"

LYR — Mon, 03 Aug 2026 14:24:43 +0000

Instead of imitating one sacred right answer, have it produce N candidates and pick the best — but it's no cure-all

There is one right answer, and it is sacred — that article of faith in AI training cracked. Have the teacher produce N candidates and let a quality estimator pick among them, and what you get is a set of model answers that beats the teacher's own first shot (+2 on Live; 90→96 together with the CPO that followed).

But the thing I'm sure of is the flip side. If the yardstick you select with is off target, you lose outright to a single clean answer. When I selected by closeness to a right answer that faced the wrong direction, 82% fell to 73–77%.

More volume doesn't change that. Data inflated to 8,553 examples by raising the temperature lost to a clean 3,301. The question to ask wasn't "what should N be" — it was "does this yardstick point at the kind of good I actually want?"

When you teach an AI something, the most straightforward approach is this: prepare one correct set of model answers and have it imitate them. For translation, you show it a great many pairs that say "the right answer for this sentence is this translation." Straight out of the machine learning textbook, and it's how I worked for a long time.

Behind it sits a quiet article of faith: the assumption that "there is one right answer (gold), and it is sacred." Raise the quality of the model answers and you raise the quality of the student. So you carefully prepare one good right answer —

Then one experiment cracked that faith. Having the model produce N candidates and picking the "best one" gave better translations than having it imitate a single right answer. And another experiment from the same period taught me, just as clearly, that this way of picking is no cure-all — get the yardstick wrong and you lose outright to one clean set of model answers. This article is about both sides.

Where it started — one weak mode I just couldn't close

LYR's small model has three places it gets used (modes): Live, which translates video subtitles in real time; Page, which translates the whole screen; and Manga, which translates comics.

Of the three, Live was stubbornly the weak one (what I'm training here is the mid-size one among the small models — the 8B specialist). Same small model, and yet Page and Manga were roughly level with the large model above them, while Live alone was left well behind. Looking at the causes one case at a time, the shape of the failures came into view: idioms translated literally (reading "isn't like you" as the opposite of 「あなたらしくない」, the idiom it should have produced), the wrong subject picked, words inserted that were never there.

The first move that comes to mind is "collect more gold data for Live and have it imitate more of it." But who writes that gold data? The larger model above it — the teacher — writes it (the distillation from last time). You have the teacher translate once and use that as the model answer. Except the teacher's own translations sometimes missed the idiom. If the model answers miss it, the student misses it too.

The realization — the teacher's "first shot" isn't necessarily its best

So I changed the framing. Stop having the teacher translate exactly once and then worshipping that as the one right answer. Instead —

Have the teacher translate the same sentence N different ways. Then mechanically pick the "best translation" out of that set.

That's the idea behind best-of-N (take the single best of N candidates). The crux is how you decide which translation is "best." What I used was a quality estimator (QE = quality estimation): a mechanism that scores "how good is this translation?" without ever looking at the right answer. Instead of matching a candidate against a right answer, it estimates how well the translation itself came out. Score the N candidates with QE and adopt the top scorer as the "new model answer."

I retrained Live on those model answers. It worked — modestly. Re-imitating the model answers rebuilt with best-of-N moved Live's pass rate +2. Stacking CPO on top of that (preference optimization that learns "the difference between a good and a bad translation" instead of imitating one right answer — a method with a track record in machine translation) added +4: +6 in total, 90→96 — level with the large model's 96. register (how natural the speech style sounds) held at 98 against the 100 it had before specialization, still at large-model level.

Let me draw a line here, to be safe. Putting those three models side by side (small general-purpose / Live-specialized / large) is the subject of the sister article A small specialist drew level with the strongest model in service. What matters here is one point: of that +6, the part best-of-N moved is the +2 — not a magic strike, but a solid step.

What matters is the mechanism. The teacher's "first shot" is not a ceiling on the teacher's ability. Roll it N times and somewhere in there is a translation better than the first one. Pick that one up with QE and you have model answers that beat the teacher itself. The same mechanism is also the heart of the process of having an AI build the teacher data. The right answer wasn't one thing — it was something you grow out of N candidates.

Figure 1: the gold faith, and best-of-N QE. Against imitating one right answer (left), rolling N candidates and picking the best with a quality estimator (right). The teacher's first shot is not a ceiling on the teacher's ability. But if the yardstick you select with is off target, you lose to one clean answer.

Another direction — and the "next move" I haven't stacked yet

Live shouldn't be the only place this selection idea works… or so I'd expect. But here I have to draw an honest line.

For Japanese → English — the direction LYR polishes so people can "read Japanese manga and anime in English" (this one is small too, but a lighter 4B airframe) — the un-fine-tuned accuracy was only around 42%. Run a chain of distillation training over it and it jumped to 84% — the same level as a large model, or as a powerful previous-generation model.

But that 84% is not a best-of-N result. What's inside it is plain distillation: straightforward imitation of the teacher's single most confident output (one low-temperature sample). best-of-N QE is the "next move" you stack on top of that — the lever my design notes rate as having the highest return on investment (a projected +10–15 points) — but I haven't run it yet. The full picture and the numbers for that direction I'll leave to the sister article Get a different AI to build your teacher AI.

So what I can say "from measurement" about best-of-N right here is only that +2 on Live. A jump like 42→84 is not yet best-of-N's doing — it's an expected value, not a track record.

One of my usual notes of caution here. These are LLM-as-judge verdicts, and the number of evaluated cases is limited (for some experiments I didn't even record the count). Read the point estimates as a strong signal that something worked, not as numbers you can grip down to the decimal place and declare settled.

The pitfall — if the yardstick is off target, you lose to one clean answer

Up to here I could end on "best-of-N is universal magic." But let me be honest about this: with this same method, I fell flat once.

On a different dataset, I used closeness to the official subtitles (gold) as best-of-N's "selection yardstick." "Pick the candidate closest to the real subtitle and it must be the best one" — intuitively that sounds right. The result: it lost to the straightforward single output with no selection at all.

How the model answers were made	Pass rate
The teacher's most confident single output (lower temp, no selection)	82%
best-of-N selection by "closeness to the official subtitles"	73–77%

The naturalness of the speech style (register) fell too, from 87% to 79%. I rolled candidates and selected among them, and ended up worse than not selecting at all.

Why? Because the "official subtitles" I used were a different thing entirely. Those subtitles were the Korean → Japanese translation of a Korean drama, and what I was translating was the English source text. The source language was different, and on top of that they carried the mismatch particular to subtitles (alignment noise between screen and dialogue). Select candidates by closeness to that off-target right answer, and the model abandons faithfulness to the English source and gets pulled toward content that came from the Korean. If the yardstick is skewed, the more candidates you add, the more carefully you optimize in the skewed direction.

I stepped in the same hole going for volume, too. When I padded the model answers (the SFT training data) out to 8,553 examples with many variants at a higher temperature, temperature noise came along with them and the result lost to a clean 3,301 examples built at a single temperature. This isn't about increasing the candidate count N; it's about how many examples I inflated the training data to — more was not better.

The lesson isn't that best-of-N is wrong. It's that the validity of the yardstick (the QE, the selection anchor) is everything. It worked on Live because QE is a yardstick pointed the right way — it estimates "how good this translation is" directly. It fell over on subtitles because I used a yardstick pointed the wrong way: closeness to a right answer facing a different direction. best-of-N is a device that amplifies the quality of your selection criterion as-is — a good criterion beats the teacher, a bad one leaves you worse off than not selecting.

This is also the same story as That "accuracy" is a definition you chose, which I wrote earlier. Picking "good" with a QE means picking one definition of "good." If that definition misses the mark, then no matter how carefully you select among the candidates, what gets selected is off the mark.

Putting this in business terms

Carefully building one set of gold data is an easy investment to understand. But however good that one set is, it's bound by the ceiling of the teacher's first shot.

best-of-N × QE is a mechanism for pushing that ceiling up with no extra model and no data purchase. Roll the same teacher N times and pick the good ones — that's all. On Live it did produce a real +2. And my design notes rank it as the highest-ROI lever I have (a projected +10–15 points) — though the version that stacks it up big is still a "next move," not something I've run.

But this device's performance is decided by how you measure "good." With an accurate yardstick it beats the teacher; with an off-target one it loses even to a single clean answer. So the question to ask before adopting best-of-N isn't "how many candidates should I generate?" It's "does the yardstick I select with really point at the kind of good I want?"

Breaking the faith that "there is one right answer and it is sacred" is a good thing. But you mustn't replace it with another faith: "just roll candidates and you win." What belongs on the ground where the old faith stood is the discipline of continually doubting your definition of "good."

Lessons

Don't worship the one right answer. The teacher's first shot is not a ceiling on the teacher's ability. Roll N of them and select with a quality estimator (QE), and you can build model answers that beat the teacher itself. On Live this selection did produce a real +2 (90→96 together with the CPO that followed), and as a lever to stack up big I rate it the most promising one I have (not yet run).
best-of-N is no cure-all — the yardstick you select with is everything. Select by closeness to an off-target right answer (wrong direction, alignment noise) and you lose to a single clean answer with no selection at all (82% → 73–77%). Before you increase the candidate count, verify that the selection criterion is valid. A bad yardstick is worse than not selecting.
Direction over volume — picking "good" means picking a definition of "good." Inflating to 8,553 examples with a higher temperature still lost to a clean 3,301. Piling on more, or rolling more, is not value in itself. The question before you adopt it isn't "what should N be" but "does this yardstick point at the kind of good I want?"

Appendix: raw data

All LLM-as-judge (with sample spot-checks by hand). The number of evaluated cases is limited; read the point estimates as a strong signal, not as settled.

Experiment	Result	Conditions & caveats
Live specialization (best-of-N SFT + Live-CPO)	Live 90 → 96 (SFT-aug +2, Live-CPO +4)	overall / register %, same evaluation set. The large model (the strongest currently in service) is Live 96
The trade-off it came with	The price: Page 93→89 / Manga 92→88 (−4)	Live-specialized training drifted away from the general batch. Expected to be resolved by retraining on a mix, or by per-mode routing
Japanese → English (different model, 4B)	un-fine-tuned ~42% → 84% with plain distillation SFT	best-of-N not used. The ceiling is hallucination (comprehension-capacity-bound)
Assessment of best-of-N SFT	Projected to have the highest ROI (+10–15pt)	Not yet run. The only measured result is the +2 on Live above
The gold-alignment trap	No selection 82% → selection by chrF closeness 73–77% (register 87→79%)	The yardstick's reference faced a different direction (KO→JA), plus alignment noise. Faithfulness to the English source was lost
Padding out the volume	8,553 examples at temp0.7 < 3,301 at temp0.1	This is the training-data example count, not the candidate count N. Temperature noise made it worse
A note on preference optimization	Binary preference (KTO) on its own was null (84→85%, within judge noise)	Preference at the string level doesn't close the comprehension-capacity wall. CPO adopted as the backbone stacked on top of best-of-N SFT

Related: the basics of distillation and SFT/CPO are in Raising a small AI into a specialist at one job. Using best-of-N to generate teacher data is covered in Get a different AI to build your teacher AI, and the full picture of Live specialization in A small specialist drew level with the strongest model in service.

Originally published at LYR Performance Note #018 — part of the series “Models — smaller, faster, sharper”. The full set is at lyr.jp/en/research.

Swahili from 0% to 30%. But the value wasn't the score — it was that the failures changed

LYR — Mon, 03 Aug 2026 14:23:41 +0000

Spotting the real bottleneck in the language AI is worst at — look at what the failures are made of, not the absolute score

Swahili is one of the languages AI is worst at. Translation accuracy went from 0% to 30%. But the value wasn't in the score — it was that the failures themselves changed.

From "strings that aren't even words" (= the model doesn't have the language) to "fluent, but the meaning slips" (= the teacher is bad). The score is just as low either way, but it is a strong clue that the rate limiter has moved off capacity and onto the teacher.

And in low-resource languages, a huge teacher doesn't necessarily help. Tripling the teacher's size produced no difference, and a smaller teacher designed for low-resource languages won instead (9 wins / 5 losses). About $15 per language jumps a model from "doesn't have it" to "usable but rough."

Figure 1: the distribution of Swahili failures. Between before and after the fundamentals were poured in, the pass count moves only from 0 to 3. But the substance of the failures was swapped out: from strings that aren't even words (= the model doesn't have the language) to fluent output whose meaning slips (= the teacher is bad). That's the signal that the rate limiter moved from capacity to the teacher side.
Note: n=10. Read the point estimate as a signal, not a settled result.

Translation AI has languages it is good at and languages it is hopeless at. English and Spanish are easy — there are mountains of parallel text online. And then there are languages with over 100 million speakers that AI simply cannot translate. Swahili is the poster child.

When I had LYR's own model translate into Swahili, accuracy was 0%. Not one of the 10 items passed. This is where most teams conclude: "too much to ask of a small model — either scale it up or give up."

But that isn't what this article is about. It's not about the score being 0% — it's about how that 0% was wrong. I looked inside the failures, and the rate-limiting step (the bottleneck) showed itself.

There are two kinds of "0%"

Start by looking at how the base model got Swahili wrong. What came out when it tried to translate was this: "Kwa kari", "H,h...".

That isn't even Swahili. It isn't even words — it's garbled characters. The model simply doesn't have the Swahili language. It isn't mistranslating; it has no vocabulary to speak with. It's like asking a baby to read a contract aloud. Seven of the ten items were this kind of "string that isn't even a word," and three barely took the shape of a language. Passes: 0.

What matters here is that the same 0% can be completely different in kind. Is it 0% because the meaning was misread, or 0% because the language isn't there at all? The first can be polished out. The second is a problem that comes before polishing. The number on the score tells you nothing about which one you have.

Pouring in the fundamentals

If the substance of the failure is "the language isn't there," there's only one move: pour the fundamentals of that language into the model.

The method: before teaching it to translate, have it read a huge amount of Swahili prose (continued pretraining = adding a language to the base after the fact). A hundred million words of clean Swahili collected from the web, read through and nothing else. And to keep it from outputting only Swahili (forgetting the languages it already had), I mixed in a quarter as much Japanese and English. Half a day of GPU time; this reading step (continued pretraining) alone cost about $13. Add requantization and the step that trains translation in (imitation and CPO), and it comes to about $15 per language in total — one taxi ride.

Then I had it translate again. Accuracy: 0% → 30%.

Let me be honest about this. 30% is still a long way from the 60% ship line. On the number alone, it's "still not good enough." Anyone who stops the story here would call this investment a failure.

But the 30% wasn't what I was actually looking at.

The "shape" of the failures changed

As Figure 1 at the top shows, the number (the pass count) moved only from 0 to 3. But the "shape" of the failures was replaced wholesale.

Before: the main culprit was "strings that aren't even words" — the language isn't there.
After: non-word output all but disappeared. Failures like handing back the Japanese as-is, or emitting some other language, were zero. What comes out is always fluent, correct Swahili. It's just that the meaning sometimes slips — flipping "worthless" into "valuable," or letting a few English words leak in.

That is the real return on this investment. The model acquired the Swahili language. From "broken" to "usable but rough." The failures moved out of the language layer and into the meaning layer.

And that move is itself the diagnosis. When the main culprit shifts from "strings that aren't even words" to "fluent but the meaning slips," it means the rate limiter has moved off the model's capacity and onto something else. Onto what?

The rate limiter was on the "teacher" side

"Fluent, but the meaning slips." I had a suspicion about that particular way of being wrong.

The small model is trained by imitating model answers — translations produced by a larger model above it (the teacher). That's the distillation from last time. If the student translates fluently but misses the meaning, then — maybe the teacher's model answers are missing the meaning too.

The way to check is a controlled experiment. Leave the student (base) completely untouched and change only the teacher side. If the scores go up, that's a signal that the rate limiter is on the teacher, not the student.

What I changed first wasn't the teacher's "model" but its "recipe." Translating Japanese→Swahili directly falls apart, so I routed it through English (Japanese→English→Swahili). Sending it down the "wide road" that is English restores reading comprehension.

The result — that recipe change alone gave 9 wins / 1 loss out of 13 (3 ties). Not a byte of the student changed. That said, only 10 of the 13 were decisive, so I can't call it settled. Even so, the signal that the rate limiter sits on the teacher's "recipe" and not on the student's capacity is quite strong.

Figure 2: a controlled experiment that changed only the teacher's "recipe." Not a byte of the student (base) changed; only the teacher's generation path was swapped, and the same 13 cases were run head-to-head. English pivot wins 9 / ties 3 / direct wins 1, and the pass rate rises from 15% to 23%. Because the student is held fixed, this difference points not at the student's capacity but at the rate limiter being on the teacher side.
Note: 10 of the 13 were decisive. Because n is small, a strong signal rather than a settled result.

If I had looked only at that first score of "30%" and concluded "the model is too small," I would have invested in the direction of adding capacity (= a bigger model, expensive). But at least at this entry point (Japanese→Swahili), what worked was changing the teacher's recipe (= cheap). Get the diagnosis wrong and I'd have sprinted full speed at the expensive option.

The counterintuitive part — the "teacher" is bad at this language too

There's one more pitfall here.

"If the teacher is the rate limiter, just bring in a bigger, smarter teacher" — that's the natural thought. What the model is imitating is a huge general-purpose model. Those can do anything. Surely Swahili is nothing.

But the huge general-purpose models are startlingly weak on low-resource languages. This isn't my discovery; it has been demonstrated over and over in the literature. There is a report that even the very best general-purpose model beat a dedicated translation model only 5.5% of the time on English→other-language translation. Another survey found that general-purpose models lost to older translation-only models in 84% of 204 languages.

The giant models that look omnipotent are omnipotent only on the "wide paved road" of English and the major languages. Turn onto an unpaved road like Swahili and a dedicated translation model passes them without effort. The reflex of "have it imitate a big model and you'll be fine" doesn't work here.

So I actually made the teacher bigger. I took the translation-only model from 3B to 10B — more than 3x. The result: no significant difference (3 wins / 3 losses). More parameters left both the English leakage and the meaning flips exactly where they were. What helped wasn't size but a different line of model designed for low-resource languages (English→Swahili passes went 0/7 → 5/7). Though that 5/7 rests on only 7 items too — "it helped" isn't a verdict; read on the same yardstick as the 3-3 split, it's a strong signal.

"Make the teacher bigger" isn't something you can buy in a low-resource language. What you can buy is "pick a teacher that fits the language."

The rate limiter differs by direction

The diagnosis taught me one more thing. The easy way to trip here is that "touching the teacher" actually covers two kinds of intervention. Changing the teacher's "recipe" (the English pivot) and making the teacher "model" itself bigger or stronger (strengthening the teacher) are completely different moves. Mix them together and the attribution of the rate limiter looks inverted. Here are the four things I tried around the teacher, re-sorted by kind of intervention (all small n = read "it helped / it didn't" as a signal, not a verdict).

#	Intervention	What changed	Direction	Result (n)	What it tells you
①	English pivot	the teacher's recipe (ja→en→sw)	ja→sw	9 wins / 1 loss, 3 ties (13 total, 10 decisive)	comprehension recovered = the teacher's output had plateaued
②	3x the teacher	the teacher model's size (3B→10B, same design)	Swahili output	3 wins / 3 losses, 14 ties (no significant difference)	more size makes no difference
③	switch to a different line	the teacher model's design (to NLLB, built for low-resource)	en→sw	pass 0/7→5/7 (n=7)	the right design does move it
④	switch to a different line	the teacher model's design (to NLLB)	ja→sw	tie (teacher-independent)	even a strong teacher doesn't move it

Read the table by kind of intervention and the identity of the rate limiter splits cleanly by direction.

English→Swahili: improve either the teacher's "recipe" or its "model" and it goes up (teacher-bound = ③ 0/7→5/7). This is a problem of spelling and phrasing on the output side.
Japanese→Swahili: the English pivot, which changes the "recipe," works (① 9 wins / 1 loss), but no amount of making the "model" bigger or stronger gets past the plateau (②③④). The deep understanding it takes to "read through" dropped subjects and idioms in Japanese demands the model's capacity itself (capacity-bound).

So the previous section's "the teacher's 'recipe' is the rate limiter" (① ja→sw) and "strengthening the teacher model doesn't help" (④ the same ja→sw) don't contradict each other. What works is the "recipe"; what plateaus is "strengthening the model" — same direction, different intervention, different rate limiter. Had I rolled these two into a single "swap the teacher," I'd have gotten the attribution wrong. Had I filed the whole thing under "low-resource languages are just hard," I would never have seen the split at all.

Low-resource isn't a tail you give up on

Let me put that in business terms.

For most teams, low-resource languages are the "long tail you cut off." Few users, AI is bad at them, the cost doesn't add up. The edge of the distribution.

For LYR it's different. "There's content you can't see because language is in the way" — breaking that is why LYR exists. The languages the world is worst at are exactly where LYR should be strongest.

And what this round showed is that this isn't willpower — it pencils out economically. About $15 per language jumps a model from "doesn't have this language" to "usable but rough." That stacks up straightforwardly as "$15 × number of languages" — line up 50 hard languages and it's $750; 100 and it's $1,500. Less than one engineer-day of salary. After that, you diagnose the rate limiter and invest on the correct side. As long as the diagnosis is right, low-resource languages can be promoted to first-class citizens cheaply, one at a time.

The starting point was always the same — look at what the failures are made of, not at the number on the score.

Lessons

Look at the "distribution" of failures, not the absolute score. The same "0%" is one thing when the model doesn't have the language and another when it misread the meaning. Same number, opposite moves. Only by looking inside the failures do you find out what the rate limiter really is. (The manifesto's first principle, "Measure First" — except that you measure how it got it wrong, not what score it got.)
The change in the "shape" of the failures is what gives you the diagnosis. After pouring in the fundamentals, the failures moved from "strings that aren't even words" to "fluent but the meaning slips." That's more important information than the score going up — because it's a strong clue that the rate limiter moved from capacity to the teacher side (recipe, model choice). Read the shift, not the size of the gain.
In low-resource languages a "big teacher" doesn't necessarily help — and they aren't a tail to give up on. Tripling the teacher produced no difference; the design built for low-resource won. What you should buy isn't "size" but "a design that fits the language." About $15 per language jumps it from "doesn't have it" to "usable but rough." After that, separate the rate limiter out direction by direction, and knock out the cheap side (teacher choice) first.

Appendix: raw data (per experiment)

Shared conditions: our own 8B model, Swahili (0% at INT4 quantization / 10% at BF16 = base-bound, already established). Evaluated at the deploy quantization as shipped, comparing before and after within the same judge round as a controlled comparison. Every judgment is small n = read the point estimates as a signal, not a settled result.

Experiment	Result	Conditions & caveats
① base injection (continued pretraining)	pass 0.00 → 0.30. Failures shift from "strings that aren't words 7/10" to "fluent but the meaning slips"	n=10. mono ~100M tok + 25% ja/en replay, ~10.8 hours. Japanese echo and wrong-lang both went to zero
① cost	CPT alone ~$13; with requantization + translation SFT + CPO included, ~$15/language in total	—
② the teacher's "recipe"	direct (pass 0.15, 1 win) → English pivot (pass 0.23, 9 wins)	base held fixed, only the generation path swapped. Non-tie n=10, so a strong signal rather than settled
③ teacher size	dedicated MT 3B→10B gives 3 wins / 3 losses, 14 ties = no significant difference	10B leaves the English leak and the meaning flips just as they were
③ teacher design	a different line designed for low-resource: 9 wins / 5 losses, en→sw pass 0/7→5/7	the advantage is design, not size
④ the rate limiter differs by direction	en→sw improves (teacher-bound) / ja→sw doesn't (capacity-bound)	reading Japanese (dropped subjects, idioms) demands capacity
literature (the teacher's own weakness)	even the best general-purpose model wins only 5.5% against dedicated MT on en→xx, and is behind on 84% of 204 languages	arxiv 2404.13813 / 2309.07423. In low-resource, the reflex of "distill from a giant model" doesn't work

Related: the basics of distillation and SFT/CPO are in Raising a small AI into a specialist at one job. (Note: the Swahili CPT base forgets other languages, so shipping keeps them separate by switching the base per language. That serving design is a separate article.)

Originally published at LYR Performance Note #017 — part of the series “Models — smaller, faster, sharper”. The full set is at lyr.jp/en/research.

I cut the bottom-rank rate from 64% to 24%, and held the top-rank loss to 12pt

LYR — Mon, 03 Aug 2026 14:22:40 +0000

Cutting the worst translations beat chasing the best ones — a product's value is in its floor, not its ceiling

I cut the bottom-rank rate hard, from 64% to 24% (−40pt), and held the drop in top-rank rate to a small 24% → 12% (−12pt). For the 3 firsts I gave up, I handed back 10 lasts (out of 25).

What breaks the user experience in a subtitle product isn't the average, and isn't the best — it's a single worst case. So I moved my target from the ceiling (peak quality) to the floor (the one worst case). I sold a little ceiling and bought floor.

The average agreement rate (89–96%) doesn't capture that improvement at all. If the problem is in the tail, the average dilutes it. The ceiling and the floor move independently — what I should have decided first was which of the two I wanted to move.

How it was measured: 25 hard cases translated by three configurations (compression only / +rotation / before compression), then ranked blind. Bottom-rank rate / top-rank rate = the share of cases where a configuration came last / first among the three (each sums to 100% across the three configurations). I use the bottom-rank rate as a proxy for catastrophic failure — I confirmed by eye that most of the translations that fell to last place were critical breakdowns.

Figure 1: top-rank rate vs. bottom-rank rate. The share of cases where each configuration ranked first / last among the three. Adding rotation cuts the bottom-rank rate from 64% to 24% (−40pt), while the price — the drop in top-rank rate — stays at 24% → 12% (−12pt). Before compression, shown for reference, is the mirror image (top-rank 64% / bottom-rank 12%).
Note: held-out drama, 25 hard cases, blind ranking. Ranks are relative, so the top-rank rates sum to 100%, as do the bottom-rank rates.

I compressed my own small model to a quarter of its size (quantization). It got faster and cheaper. Quality, on average, barely moved. On paper there was nothing to complain about.

But when I actually lined the translations up and read them, every so often something made me flinch. A Japanese line comes back still in Japanese. A character's gender swaps. A proper noun turns into something else entirely. The frequency is low. But in subtitles, one of these is enough to shatter immersion instantly.

Same average, completely different experience. Chasing that discrepancy ended up forcing me to rebuild how I use the word "quality" at all. What was mattering to the product was not the ceiling (peak quality) but the floor (the one worst case).

The average makes the difference disappear

First, what was actually happening.

The heavily compressed model (INT4 = rounding the weight values to 4 bits) was wildly hit or miss. I took 25 deliberately difficult lines (held-out hard cases), translated them with three configurations, and had a stronger model rank them blind. Every number below is a relative rank within those 25 cases — not an incidence rate across production as a whole. Here's what came out.

The compressed model's top-rank rate (the share where it gave the best translation): 6 of 25 = 24%
The compressed model's bottom-rank rate (the share where it gave the worst translation): 16 of 25 = 64%

Sometimes it produced the best output; often it produced the worst. The spread is wide — the technical term is bimodal (it splits into a good peak and a bad peak). Superb when it hits, an accident when it misses. As subtitles, that's unusable.

And yet none of this danger shows up in the average. Here is the token agreement rate before and after compression.

Metric	Before compression	After compression (INT4)
Agreement rate (general text)	89.5%	89.8%
Agreement rate (JA→EN)	95.6%	95.3%

The difference is essentially zero. Had I looked only at these numbers and concluded "no degradation," I would have missed the failures entirely.

Why is the average so blunt? The loss from compression concentrates in the just-over-10% of "borderline" calls (the story I wrote in Part 1). The remaining 90% return the same answer as before compression, so the average agreement rate gets diluted by that 90%. The average waters down the tail — the bad side. What mattered was the just-over-10% buried inside it.

Why a single echo is fatal

Why weight "the one worst case" that heavily?

The north star of subtitle translation is "subtitles with no presence." The translation dissolves naturally into the screen and the user forgets they're even reading a translation — that's the best state. But the moment a Japanese line comes out still in Japanese (echo = the failure where the model returns the source untranslated), you snap out of it: "ah, this is machine translation." Immersion isn't a continuous quantity. Once it breaks, it goes back to zero.

This has the same structure as the "critical breakdown" I wrote about in the article on the bottleneck. Even if 99 out of 100 are perfect, one echo decides the impression of the whole experience. Users don't experience the average. They experience the worst case.

So the metric to chase wasn't average quality. It was the floor — quality at its worst. A catastrophic-failure rate is nothing more than that floor measured upside down (the lower the rate, the higher the floor).

So how do you measure it? The headline number of this article (64% → 24%) is the share of those 25 hard cases where a model produced the worst translation of the three — the bottom-rank rate. Because these are relative ranks, coming last doesn't automatically mean the output was critical (even before compression, 12% end up last). But when I read the last-place translations one by one, most of them were critical breakdowns: gender flips, mangled proper nouns, subject and object swapped, echo. So the bottom-rank rate can be used as-is as a proxy for catastrophic failure — and that's what "catastrophic-failure rate" means from here on.

Don't raise the ceiling; raise only the floor

So how do you reduce catastrophic failures?

What worked here was a correction called rotation. If you mathematically "rotate" the model's weights before compressing, the rounding error doesn't pile up on a handful of values; it spreads thin across all of them. Borderline calls become less likely to be flipped by error — it's a technique that suppresses breakdowns in the tail on purpose.

Figure 2: what rotation does to the error. Without rotation, rounding error concentrates on a few weights and borderline calls flip. With rotation, the error spreads thin and evenly across the whole model, and the big collapses stop.

I added it and re-measured the same 25 cases.

	Compression only	+rotation	(ref.) Before compression
Residual echo	4	0	0
Bottom-rank rate (= proxy for catastrophic failure)	64% (16 cases)	24% (6 cases)	12% (3 cases)
Top-rank rate	24% (6 cases)	12% (3 cases)	64% (16 cases)

Read this carefully. The top-rank rate went down (24% → 12%). Rotation does not make the model "smarter at its best." But the bottom-rank rate fell from 64% to 24%, to roughly a third (mean rank moved 2.40 → 2.12 as well). −12pt given up against −40pt gained — what came back is far larger than what I handed over. Echo went from 4 to 0. Quality at its worst — the floor — moved much closer to before compression (12%).

Perplexity on general text (ppl = a measure of the model's "lack of confidence," lower is better) tells the same story: compression had degraded it to 10.65, and rotation brought it back down to 9.95, essentially level with the 9.94 from before compression. In exchange for a slight drop in the top-rank rate, the broken floor — quality on the worst side — recovered.

As a product, that was enough. For subtitles, removing ten worst translations does more than adding one best translation. I dropped the ceiling and chose the floor.

(The "ceiling or floor" choice itself came up in the Pareto article as "how to pick an operating point." There I laid them out as method A = compression only and method B = rotation. This article digs into the measurement behind that single point. Where you sit is decided not by which one is "better," but by which axis the product needs.)

The ceiling doesn't come back. And automatic metrics hide the failures

First, this isn't a clean win. What rotation restores is the floor; the ceiling doesn't come back — it still falls short of before compression (top-rank rate 64%, mean rank 1.48). The peak quality lost to compression cannot be recovered by this correction.

And one more thing. The scariest part of this episode wasn't the technical side. It was that the metric I trusted first had made the failures invisible.

An agreement rate of 89–96% — a respectable number. Had I used it as the acceptance criterion, the compressed model would have passed as "essentially no degradation" and gone to production spitting out echoes. It's not that the metric was wrong. And it's not that averages are bad either — ppl (perplexity) is an average too, and it captured the floor's recovery just fine. The question is what the metric dilutes. Token agreement rate waters down the small number of cases where the failures happen with the vast majority of "nothing wrong here," so the worst cases I most wanted to see became structurally invisible. When the problem lives in the tail, question what your average is diluting, and if you have to, count the worst cases directly.

In the end, what caught the failures was a plain manual job: reading the generated translations by hand and counting critical breakdowns one at a time. I still haven't automated it. If the automatic metric is blunt, the only option is to count the tail directly. Shift what you measure one notch, from the average to the worst case. That was all it took — and I very nearly failed to see it.

Lessons

A product's value is in its floor, not its ceiling. In a product where a single breakdown ruins the experience, the floor is the KPI. Write your acceptance criteria as "worst-case rate at or below X%," not "average score of X or higher" — users don't experience the average, they experience the worst case.
Ask what your average metric dilutes. Not every average is bad (ppl captured the floor's recovery). But a token agreement rate of 89–96% was watering down the just-over-10% where the failures happen with the vast majority of "nothing wrong here." If the problem is in the tail, switch to a metric that counts worst cases directly.
The ceiling and the floor move independently. I cut the bottom-rank rate hard, from 64% to 24% (−40pt), and held the drop in top-rank rate to 24% → 12% (−12pt). For the 3 firsts I gave up, I handed back 10 lasts. Decide "which part do I want to improve" first, rather than "make the whole thing better." Aim at the wrong place and even the right technique comes up empty.

The manifesto's third principle — Optimize the Right Proxy for UX, and within it "a good proxy looks at the one worst case (the tail), not the average" — was shoved in my face in its plainest form by this episode. In speed or in quality, what ends up mattering is always out at the edge of the distribution.

Appendix: raw data

Scope: our own 8B model; held-out drama, 25 hard cases; free generation identical to production, ranked blind by a stronger model.

Metric	Compression only → +rotation	Conditions & caveats
Bottom-rank rate (proxy for catastrophic failure)	64% → 24%	Approaching the 12% of BF16 before compression
Residual echo (source copied verbatim)	4 → 0	—
Mean rank / ppl	2.40 → 2.12 / 10.65 → 9.95	ppl is essentially level with the 9.94 of BF16 before compression
Shape of the distribution	6 firsts, 16 lasts (bimodal) → 3 firsts, 6 lasts	Still short of BF16 before compression (16 firsts, mean rank 1.48) = the ceiling doesn't come back, but the floor does
Bluntness of the average metric	Agreement rate, general 89.5→89.8%, JA→EN 95.6→95.3% (Δ≈0)	What's being fixed is the tail — proper nouns, pronouns — and the average agreement rate dilutes it = don't use it to decide
What rotation is	A correction that rotates the weights so quantization rounding error is spread out	A finishing step for recovering the floor, not the ceiling

General rule: always evaluate a compressed model at the same quantization and free generation as production, and make the worst-case rate, not the average, the primary KPI (same root as the evaluation principle in quantization eats into fine-tuning).

Originally published at LYR Performance Note #016 — part of the series “Models — smaller, faster, sharper”. The full set is at lyr.jp/en/research.

"Add context and the model gets smarter" was half a lie

LYR — Mon, 03 Aug 2026 14:21:38 +0000

What RAG and injecting context into the prompt fill in is "missing knowledge" — a shortfall in comprehension capacity stays open even with the answer right in front of the model

"Add context and the model gets smarter" was only half true.

The gaps it fills are the ones you could look up. Speaker, proper nouns, dropped subjects — context filled all of those (13-0). But deep reading comprehension and register don't close even with the answer sitting right in front of the model — that is a question of capacity, not of raw material.

In a production-equivalent full factorial, the loss rate against the 27B stayed at 76% / 75% / 76% no matter how I supplied the context. And the most expensive move (baking it into the weights) couldn't beat the cheapest one (pasting it into the prompt on the spot) — 94% identical outputs. Which half of the problem you have is something you can tell apart before you invest.

"If the model isn't smart enough, give it more context" — that's the most popular prescription in AI development right now. RAG (retrieving documents and handing them to the model along with the prompt) is an extension of the same idea. Give it the surrounding information and the model will "read the context" and answer correctly.

I assumed the same thing. In subtitle translation, my own small model (8B = 8 billion parameters) loses to a large model (27B = 27 billion) on reading comprehension. So — if I hand it the lines before and after — surely the small model catches up? Before investing, I measured.

The result split cleanly down the middle. Half of it was true, and half of it was a lie. And if I hadn't caught the lie, I was about to pour weeks into a training investment that would have bought nothing. And more than that — which half your problem falls on is something you can tell apart before you invest. If it's a gap you could look up, context fills it cheaply; if it's a shortfall in comprehension capacity, only size moves it. That distinction is what decides whether you bet on RAG or on the model.

First — context really did work

The first experiment was to measure the upper bound: if the context were perfect, how much better does the small model get? Speaker gender, the correct reading of proper nouns, the dropped subject — I prepared the state of the scene omnisciently, then pitted the version handed perfect context against the current version handed none, on the same 8B model.

The judging was pairwise (put two translations side by side and pick the better one, item by item; the sign wobbles less than it does with an overall score). Measured across four episodes of one drama, 86 items —

With perfect context: 40 wins / without: 22 wins / 24 ties. 65% on a decisive-only basis (ties excluded from the denominator), statistically significant (p≈0.03). Context really did work. So far, the prescription holds.

— But break down what worked

The trouble started here. I split out what had worked, dimension by dimension, and recounted.

Dimension that worked	Record
Speaker gender (ore/watashi — speech style)	8-0
Proper nouns (fixing the reading and the transliteration)	3-0
Restoring dropped subjects (who, to whom)	2-0
— Total (clean)	13-0 (p≈0.0001)
Register that fits the scene	4-4 (a wash)
Word choice	7-10 (if anything, worse)
Semantic correctness (negation, polarity)	0-1 (no movement)

(The dimensions aren't exhaustive, and one item can fall under several of them. So the dimensions don't add up to the overall 40 wins / 22 losses.)

As you can see, almost all of what worked was concentrated in the top three dimensions — speaker gender, proper nouns, dropped subjects. And those three have something in common: every one of them is something you could look up. Who is speaking, how the name is read, who the dropped subject is. It's all there in the script if you go and look — these are gaps in knowledge.

On the other side, register for the scene was a wash and semantic correctness didn't move at all. Put the raw material — the context — right in front of it, and the 8B still can't translate the difference. This isn't a gap in knowledge; it's a matter of capacity for comprehension and expression.

Incidentally, until not long ago I had concluded that "context doesn't help — it's a draw." That was because I was measuring impoverished context with a wobbling score. Mix a component that works (gender) and one that doesn't (register) into a single overall score and they cancel each other out, and it looks like a draw. Only by splitting it out by dimension did the 13-0 sign become visible (the manifesto's first principle, Measure First — how you measure included).

The main question — does context let the small model catch the large one?

The upper-bound experiment told me that with perfect context there are dimensions where it works. Now the practical question: if I hand it the surrounding lines, does the 8B catch the 27B? This is the fork in the investment decision.

I ran my own 8B against the large 27B on 434 manga items, Japanese→English. Exactly the same prompt form as production, generation randomness pinned to zero (temp0 = the same answer comes out every time; this keeps the experiment's variables clean). On top of that I compared three ways of supplying the context, with everything else held constant:

A: no context (current)
B: context injected into the prompt on the spot
C: context baked into the weights by training (context-aware CPO = a version trained on preferences with the context attached)

Then I ran each against the 27B and looked at the loss rate against the 27B (decisive-only basis, ties excluded).

Figure 1: the loss rate against the 27B doesn't move with how you supply the context. No context 76% / injected into the prompt 75% / baked in by training 76%. However I supplied it, the gap didn't budge.
Note: 434 manga items, temp0, decisive-only basis (ties excluded).

No context 76% → injected into the prompt 75% → baked into the weights 76% (this full factorial is all one and the same temp0 run). However I supplied the context, the gap to the 27B didn't budge. On this temp0, decisive-only basis, the number of items where the 8B "won strongly" against the 27B was, across all three conditions, zero.

Even with the answer right in front of it, it can't use it

Why doesn't it move? It clicked once I lined up the outputs from the version regenerated with the context actually handed over (B, injected into the prompt). Even with the correct context right in front of it, the 8B can't put it to use. All of the following are outputs produced with that context.

「太陽に溶かされて」 ("melted by the sun" — Japanese leaves the subject unsaid; the preceding line, a poem about wings, had been handed over): the 8B picked the wrong subject and gave "I'd be melted"; the 27B correctly gave "they melt".
「天文部を頼む」 ("take care of the astronomy club") (the context says the request is being made to you): the 8B translated the direction backwards; the 27B correctly gave "I want you to help".
The character name 「相川仁」 (the given-name kanji 仁 has several valid readings): the 8B missed the reading with "Masaru"; the 27B gave "Jin".

The raw material — the context — was handed over identically in both cases. The gap shows up anyway. Which means the ability to use context is itself rate-limited by the model's capacity. Put the knowledge right in front of it, and if the capacity to read it isn't there, it can't be used.

There was a clinching symptom, too. Given long context, the 8B broke the output format on 19 items (the 27B, on 4). The limit of capacity shows up not only in accuracy but as a drop in robustness like this.

"Baking it into the weights" didn't beat "handing it over on the spot" by an inch

Last, I tested the most expensive option — baking the context into the weights (C). Instead of putting it into the prompt on the spot (B), train preference on data that carries the context, and maybe the model learns to use context better. That was the hope.

The result was blunt. The trained version (C) and the hand-it-over-on-the-spot version (B) produced 94% identical outputs at temp0 (the 6% that differed was trivial). Going to the trouble of training on 256 pairs of material got me effectively the same thing as pasting the context into the prompt on the spot. → This training was not adopted.

Context engineering is necessary. But "hand it to the prompt on the spot" is enough — there was no value in baking it into the weights. And above all: however you engineer the context, an 8B on its own doesn't reach a 27B. Closing that gap was the job of capacity — a bigger base — not of context (the manifesto's second principle, it can still be cheaper and faster — here: don't optimize the context alone. The rate limiter was on the capacity side, not the context side).

Lessons

Context solves missing knowledge; the wall in front of understanding is capacity. Gaps you could look up — speaker, proper nouns, dropped subjects — get filled by context or RAG (13-0, significant). But deep reading comprehension and register don't get filled by handing over material — they need model capacity. RAG is a specific cure for the former and does nothing for the latter. Separate the two before you invest.
Don't measure what works and what's capacity-bound mixed together. Mix a dimension that works (gender 8-0) with one that doesn't (register 4-4) into a single overall score and they cancel out, it looks like a draw, and you read the sign wrong. Only by splitting it out by dimension can you see what works and what doesn't (the manifesto's first principle). I did in fact measure them mixed once, and wrongly concluded that context doesn't help.
Sometimes the most expensive option doesn't beat the cheapest one. Baking context into the weights couldn't beat pasting it into the prompt — 94% identical outputs. And on top of that, handing over the knowledge is useless if the model can't read it — with the correct context right in front of it, the 8B still missed subjects and proper nouns. Measure the cheap move and the expensive move on the same footing, before you invest.

Add context and the model gets smarter — that's only half true. What gets filled is the gaps you could look up. The wall in front of understanding is, in the end, model capacity, and just as chasing smarts alone leaves you with something unusable in real time, chasing context alone won't get you over the capacity wall.

Appendix: raw data

Only the measurement conditions, judging methods, and cross-checks against a separate run that aren't in the body.

Experiment	Result	Conditions & caveats
Upper-bound experiment (perfect context vs none)	40 wins / 22 losses / 24 ties, p≈0.030	four drama episodes, 8B-SFT, temp0, pairwise, n=86. All four episodes point the same way
Split by dimension (the side that worked)	gender 8-0 + name 3-0 + subject 2-0 = 13-0 (p≈0.0001)	spot-checked (ore→watashi, transliteration fixed, subject restored)
Split by dimension (the side that didn't)	register 4-4 wash / word-choice 7-10 (slightly worse) / polarity 0-1	one item can span several dimensions, so the totals don't match the upper bound's 40/22
The main full factorial (temp0)	loss rate against the 27B = no context 76% / injected into the prompt 75% / context trained in 76%	434 manga items, production template matched, pairwise. Strong 8B wins in the same run = 0
Cross-check against a separate run	raw wins and losses n=419 → 76% on a decisive-only basis	agrees with the temp0 run independently. The discrepancy in strong wins (0 vs 3) is a run-to-run difference
Training doesn't beat injection	trained vs injected on the spot: 94% identical outputs at temp0 → not adopted	context-aware CPO, 256 pairs. The raw effect of context is 72:57 (56%), but the 27B is helped just as much
Format robustness	the 8B produced broken JSON on 19 items with long context (27B: 4)	the cost of long context isn't quality alone

The output examples in the body (「太陽に溶かされて」, 「天文部を頼む」, 「相川仁」) are all outputs from the version regenerated with the context injected into the prompt.

Originally published at LYR Performance Note #015 — part of the series “Models — smaller, faster, sharper”. The full set is at lyr.jp/en/research.

The "97% improvement" I threw away

LYR — Mon, 03 Aug 2026 14:20:37 +0000

The flashier the number, the harder you look at the raw output before you kill it

The automated measurement came back with good news: junk English down 97%. I didn't take the number. I threw it away.

Going through the raw output one page at a time, the 97% turned out to be two measurement bugs on top of each other. (1) Randomness in the compression step occasionally produces a broken instance, and (2) the aggregator's parse-failure fallback was counting the entire raw text. Measured fairly, all three came out about the same (5 / 7 / 9 words).

But don't bury the technique itself along with the number. "The metric is rejected; whether the technique works is still open." Drawing the line between what you know and where your knowledge stops is what leaves you room to measure it properly next time.

Up to the previous parts, I had compressed (quantized) my own small model to make it fast and cheap. Quality barely drops — except that in the just-under-10% of "borderline" cases, it occasionally falls apart. One form that breakdown takes is raw English leaking into the translation (stray English words left sitting inside the Japanese output).

To kill this "leftover English," I tried a repair technique. The automated measurement came back with dramatic good news. Junk English down 97%.

I didn't take that number as it stood. This article is about why I threw away a spectacular 97%.

The good news: 97%

I applied the repair technique (a post-processing step that recovers the accuracy lost to compression by rearranging the model's internal weights; I'll call it rotation from here on) to a hard-compressed model and had it translate 30 pages of manga. Here's what the aggregate looked like.

Metric	Before repair	After repair	Change
Junk English (words)	968	33	−97%
Pages with junk English left	19/30	5/30	−74%
Agreement with the reference model¹	81.6%	85.8%	+4.2pt

And lined up next to the un-compressed full-precision version (FP16), the story looked perfect.

Full-precision version's junk: 3 words (essentially clean)
Hard-compressed version: 968 words (compression blew junk up by +965 words)
Repaired version: 33 words (935 words recovered, back to roughly the full-precision level)

"Compression catastrophically multiplies the junk, and repair brings it back to full-precision levels" — a storyline as clean as a textbook. Take it as it stands, and you can conclude that the repair technique is worth shipping.

The number was too good

What stopped me was that −97% was far too clean. Real model breakdowns don't line up that obediently in one direction.

There was something else: two separate measurements disagreed. Measure the same repair technique with a general-text metric (perplexity) and the recovery stops at 29%. A 97% recovery on the translation task only — when two metrics are orders of magnitude apart, one of the two measurements is usually broken.

So before trusting the aggregate, I decided to look at the raw output itself. I had all three models (full-precision, compressed, repaired) translate 30 pages of manga, lined the outputs up in a browser, and went through them page by page with my own eyes.

Re-measuring it fairly

Looking at them, it was obvious immediately. Two corrections were needed.

Correction 1: recount. The aggregator was counting even the model's "talking to itself" as English. Before returning a translation, the model writes a reasoning scratchpad internally (a working area it closes with </think>). Pull out just the translation — excluding the scratchpad — and recount, and the junk in the full-precision and repaired versions dropped to single digits.

Correction 2: re-roll the instance. But the compressed version's 968 didn't disappear from recounting alone. That instance was emitting raw English into the translation itself, not into the scratchpad (hole 1, below). Re-run the same compression step, draw a healthy instance, and the translation-only count fell to 7 words.

Here's the result after both corrections (the compressed column is the value from the re-rolled healthy instance).

	Full-precision	Compressed*	Repaired
Junk English (words)	5	7	9
Pages with junk left	4	5	8

* A re-rolled healthy instance. Not the unlucky instance that emitted 968.

All three, about the same. Compare healthy instances against each other and count only the translation, and neither compression nor repair moves the junk much at all. Both the 968 and the 33 were illusions that stood only until those two corrections. The −97% vanished without a trace.

Figure 1: one phenomenon, measured with two yardsticks. The automatic metric made it look like a huge gap — 968 words for the compressed version, 33 for the repaired one — but recount the raw translations alone and all three sit at 5 to 9 words, barely different. The −97% was a product of the yardstick.

Two holes

So where did that 968 come from? The two bugs compounded.

Hole 1: randomness in the compression. The process of compressing a model includes a "calibration" step that tunes the rounding using a small number of samples, and that's where randomness enters. Same procedure, and yet every run produces a slightly different model. One unlucky time, a "broken" compressed version was born, and it emitted raw English instead of a translation. Re-run the same process and all 30 pages were normal, with 7 words of junk. The 968 was nothing more than one unlucky draw.

Hole 2: the aggregator's fallback. The program that counts junk was designed to parse the model's structured output (JSON) and pull out just the translation. But when parsing fails, it falls back to counting the English in the entire raw text. The reasoning scratchpad, echoes of the prompt, everything gets counted as English. The chain goes like this.

broken instance → JSON parse fails → the fallback counts English across the whole raw text → explodes to 968

An unlucky instance (hole 1) times an over-counting fallback (hole 2) multiplied into an enormous fake difference. Either one on its own wouldn't have broken this spectacularly.

Ironically, the supposed "winner" — the repaired version — had a defect of its own. On 29 of 30 pages, the </think> tag from the reasoning scratchpad had leaked out ahead of the translation. The aggregate said not a word about that regression.

Keep only what's real

Once the noisy numbers were thrown out, this is all the "real" I had left.

Perplexity on general text: 10.65 → 10.42
Agreement with the reference model: 81.6% → 85.8%

Small, but solid improvements. Nothing spectacular.

And one more thing: I was careful about how I threw it away. I did not conclude that "the repair technique doesn't work." Both the </think> leakage and the failure to reduce junk could well come from a confound in my own measurement procedure (I measured through a makeshift evaluation path rather than the proper server). So I closed it out this way — "the −97% automatic metric is rejected; whether this repair works in production is still open, and it needs to be measured properly again."

Don't confuse "the measurement was wrong" with "the technique was wrong." Being fair means not killing the technique as collateral damage — and that matters as much as killing the flashy number.

Lessons

The flashier the number, the more you look at the raw output before killing it. A number as clean as −97%, or metrics disagreeing by orders of magnitude, isn't a victory — it's an alarm. Don't accept good news until you've looked at the raw output upstream of the aggregate, page by page, with your own eyes.
Pin the seed on non-deterministic steps, and suspect the aggregator's fallback. If randomness is in the mix, a single run tells you nothing but instance-to-instance variation (an improvement that doesn't reproduce isn't an improvement). And a fallback on parse failure will quietly manufacture an enormous fake signal — look at what it's actually counting once it has swallowed the error.
Close out measurement errors and technique errors separately. When you throw away a false victory, don't bury the technique with it. Drawing the line — "the metric is rejected, the technique is still open" — is what leaves you room to measure it properly next time.

And underneath all of this sits the manifesto's first principle, Measure First — but measure it right. A wrong measurement is worse than no measurement. Because it lets a false victory ship as a real one.

Appendix: raw data

Scope: a 4B model compressed hard (INT4) + learned rotation repair, translating 30 pages of manga, measured on an A100.

Item	Number	Conditions & caveats
Automatic metric (wrong)	junk English 968→33 words (−97%), pages with junk left 19/30→5/30	against the FP16 baseline = FP16: 3 / compressed: 968 / repaired: 33
Fair re-measurement (translation text only)	FP16=5 / compressed=7 / repaired=9 words (pages with junk left 4/5/8) = all three about the same	the compressed version's 7 is a re-rolled healthy instance, not the instance that emitted 968
Hole 1: non-determinism	the compressed version occasionally goes degenerate and emits raw English	regenerate and it's 30/30 clean. In production the instance-to-instance risk remains
Hole 2: the aggregator's fallback	on JSON parse failure it counts the English across the whole raw text	the fallback manufactured an enormous fake signal
Improvements that survive as real	wikitext ppl 10.65→10.42, teacher-forced agreement 81.6→85.8%	recovery on general text stops at about 29% of the gap (orders of magnitude away from the "97%" on the translation task = a sign one of the two is broken)
Side effect of the repaired version	the `</think>` tag showed up ahead of the JSON on 29 of 30 pages	never appears in the aggregate
Unresolved	possible confounding from the measurement path (a non-standard serve)	needs re-evaluation on the proper serve path + free generation + pairwise judge + a fixed calib seed. Not "rejected" but "requires a proper evaluation"

Originally published at LYR Performance Note #014 — part of the series “Models — smaller, faster, sharper”. The full set is at lyr.jp/en/research.

"Agreement with the reference model" = how well the compressed version reproduces the next word that the un-compressed full-precision version (FP16) produces (top-1 agreement rate under teacher forcing). The +4.2pt means "it moved a little closer to FP16's behavior," not that the translation itself got better. ↩

The AI that was smart on the bench suddenly started making mistakes in production

LYR — Mon, 03 Aug 2026 14:19:35 +0000

The one reason "the test score" and "the real-world skill" came apart

A fine-tuned 8B model was a star pupil on the benchmark, and then lost to the smaller 4B in a production-equivalent setup (21 to 11). It looked like the limit of the model's capacity.

The culprit was neither the model nor the training. It was that the precision I trained at and the precision production ran at were different. Putting the same adapter on both bases and comparing them, the verdict came back 23 to 2: it was the environment mismatch.

Once I re-trained with production's compression priced in, the 8B came out ahead of the 4B. Hold the conditions you train the skill under and the conditions you serve it under to the same thing. That alone was enough for the skill that had gone missing to show up intact.

I was holding a winning hand.

I wanted better translation quality. Which means the bigger model — an 8B (8 billion parameters) remembers more than a smaller 4B and can read subtler context. And in fact the 8B I'd trained with the fine-tuning from last time was posting unimpeachable numbers on my own benchmark. All that was left was to put it into production. That's what I thought.

But when I loaded it into a production-equivalent setup and re-measured quality, that 8B lost to the 4B that was supposed to be the smaller one. On one evaluation, 21 to 11. The order is backwards. The bigger model is supposed to be the smarter one — and yet in production the smaller one is winning.

For a moment I was about to conclude: put Live translation on the 4B, plain and simple. The bigger model just isn't suited to this use case.

That was the mistake. The 8B hadn't gotten weaker. The 8B I was measuring and the 8B I put into production were two different things.

The model's "precision" was different on the bench and in production

Before the reveal, one piece of vocabulary. Quantization is the compression technique that takes the model — that "huge mountain of numbers" — and re-stores it with fewer bits so it gets lighter. I covered it in detail in Part 1, but the gist is this: to run it fast and cheap on a production GPU, you compress the model before you load it.

That's where the pitfall was. Without ever deciding to, my workflow had ended up like this.

Fine-tuning (training) happens at full precision. In BF16, a format that holds the numbers finely, I train the translation skill into the model.
Production (deployment) runs compressed. I load it in INT4, a format shrunk to a quarter of the size, and take the win on speed and cost.

In other words, the form the skill was trained into and the form the customer got were different. The benchmark measures the full-precision form; production runs the compressed form. It's as if two different people had written the exam paper and the real one.

Same adapter, different base — and the result flipped

Confirming the suspicion was anticlimactically easy. Take the exact same fine-tuning (the adapter), put it on two bases, and have them fight over the same 40 items.

Base A = full precision (BF16). The same form it was trained in.
Base B = compressed (INT4). The form deployed to production.

(An adapter is a "skill-correction part" you bolt onto a base model after the fact. The base changed; the correction bolted on top stayed identical down to the last bit.)

The result was blunt. Judging I handed to opus (put the two translations side by side in front of a third-party high-capability AI and have it decide which one is better).

Same adapter, base only differs, out of 40 items, opus verdict	Wins	Ties	Losses	Serious defects
Full precision (BF16) = same as training	23	15	2	3
Compressed (INT4) = production deployment	2	15	23	22

The same skill was supposedly trained into both, and merely compressing the base dropped 23 wins to 2. And when I read through the defects one by one, subject mix-ups (getting whose line it is wrong) and polarity flips (turning "doesn't" into "does") — the kind of error that breaks the backbone of a sentence — came entirely from the compressed side. The 3 defects left on the full-precision side were all minor ones of another kind.

The good scores on the bench didn't reproduce in production. Not because of capability, and not because the training had failed. The form I measured and the form I ran were different. That was all.

Why compression makes the skill go dull

You might find that odd. In Part 1 I wrote that "compressing it barely costs you any smarts (−1 point of quality)." Surely that's a different story.

There's no contradiction. The two act in different places.

The skill you train in through fine-tuning is a correction that sits thin and wide across the whole model. "In this context the subject is this one," "this tone gets translated like that" — a delicate overlay whose job is to tip close calls onto the right side. And compression shaves off exactly that close-call margin first. Round the numbers coarsely and the majority doesn't move, but the borderline decisions flip easily. The "subject" and the "negation" that fine-tuning had carefully tipped the right way get tipped back the other way by compression's rounding error.

Here's the analogy. When I trained the skill in, I drew the lines with a high-precision ruler — one with fine gradations. In production I swapped it for a ruler with coarse gradations and tried to copy those lines across. The finer the line, the less of it a coarse ruler can capture.

The fix: train it in the production form

If that's the cause, the fix is obvious. Re-train the fine-tuning on the same compressed base you deploy to production. Draw the lines with the coarse ruler from the start. Build the skill with the rounding error already priced in, and that skill shows up in production intact. This way of "holding training and production to the same precision" is what's called quant-aware training (training with production precision priced in).

I re-measured on the same 40 items.

Live translation, 40 items, opus verdict	Wins	Ties	Losses
8B re-trained at production precision vs 4B in production form (INT8)	16	15	9
8B re-trained at production precision vs the old, mismatched 8B	20	12	8

The clearest effect was in the head-to-head against the old, mismatched 8B. It came out ahead 20 to 8, and counting only the items where the verdict was clear-cut, it was 10 to 0 — every subject and polarity error stayed on the old side, and the re-trained side was clean. Swap the opponent for "the 4B in its production form" and the 8B still showed no regression (16 wins / 15 ties / 9 losses, n=40. With 15 ties that isn't enough to declare a winning record, but it is at least not losing). The assumption that "it loses to the 4B" fell apart right here. On Manga translation, it also held echo (copying the source verbatim) at zero across 434 items.

I withdrew the "Live goes on the 4B" decision. The 8B had never been the weak one.

Before you suspect the model, suspect how you're measuring

What stung most is that I came within an inch of blaming the model's capacity. "The 8B isn't suited to this use case." It's a plausible hypothesis, and concluding it would have settled the matter quickly. But the real culprit wasn't inside the model. It was in the procedure I used to measure the model. The environment for training and the environment for production didn't match — not a model problem, a system problem.

"An AI that's supposed to be smart suddenly turns stupid in production." When you see that, the first thing to suspect isn't the model's ability. It's whether the bench and production are really running the same conditions.

Lessons

Suspect the measurement before you suspect the model. When an AI that's supposed to be smart starts making mistakes in production, what to suspect is not the model's capacity but the measurement procedure. Put the same adapter on both forms — the training one and the production one — and you can separate whether the culprit is the model or the environment (it was the latter, 23 to 2).
Hold the conditions you train the skill under and the conditions you serve it under to the same thing. Fine-tuning and compression are separate steps, but get the interaction between them wrong and the skill you trained in disappears in production. Only once you hold everything from training to serving constant, as one continuous system, does the quality actually arrive.
The bigger the design decision, the harder you should check that the playing field is level. A switch from 8B to 4B nearly got decided by a single measurement mismatch. Before you conclude "it isn't suited to this," make sure the conditions you're comparing are genuinely fair.

Appendix: raw data

Conditions: all Live, 40 items, opus pairwise verdict (win / tie / loss).

Experiment	Result	Conditions & caveats
Diagnosis (same adapter, different base)	BF16 base = 23 wins / 15 ties / 2 losses; INT4 base = 2 wins / 15 ties / 23 losses	Serious defects 3 vs 22. Subject errors and polarity flips all on the INT4 side
Fix confirmed (8B re-trained at production precision)	vs 4B-INT8 = 16/15/9 (no regression); vs old mismatched 8B = 20/12/8	10 to 0 on the items with a clear-cut verdict. quant-aware SFT-aug→CPO
Manga	echo (copying the source verbatim) held at 0/434	no side effects
The wrong conclusion I withdrew	"Live goes on the 4B" (the old 8B lost 21 to 11)	an apparent loss produced by the BF16-training → INT4-production mismatch

Originally published at LYR Performance Note #013 — part of the series “Models — smaller, faster, sharper”. The full set is at lyr.jp/en/research.

Taking a small AI's accuracy from 42% to 84% — and on some tasks, level with the giants

LYR — Mon, 03 Aug 2026 14:18:33 +0000

Drawing level with a general-purpose giant on a single task — SFT, CPO, and learning to doubt the "gold data"

Accuracy 42% → 84%. On some tasks, level with the giant model currently in service (96). And the model stays small.

What worked wasn't "making it smarter" — it was narrowing it to one job. Most of a giant model's smarts are things LYR's task never needs. Training comes in two stages: model answers for the form, preferences for the flavor.

But there's a line around what you can train in. The form of how things are said can be trained in cheaply, but deep comprehension and emotional register need capacity (size). The register the 4B struggled with showed up the moment I moved up to 8B.

Figure 1: Japanese→English accuracy. The base 4B sits at 42%. Have it imitate the 27B teacher's translations (distillation SFT) and it jumps to 84%, reaching the same level as the retired original model.
Note: reference-free binary judge, point estimate from a single run.

Last time I made my own small model fast and cheap. But that's only half the job. Left as it is, a small model loses on quality to the giant models.

By how much? Hand an un-fine-tuned small model (4B = 4 billion parameters; from here on, 8B = 8 billion and 27B = 27 billion for the large ones) a Japanese→English translation and its accuracy was only 42%. Unusable in practice.

So how do you close that gap? The answer is Figure 1 above — I pulled that 42% up to 84%, the same as the retired original model, and on some tasks all the way to level with the giant model currently in service. And the model stayed small.

This is about the recipe — and about the wall the recipe can't get over.

A giant AI is a jack-of-all-trades

First, why does a small model lose? And why is there room for it to win at all?

GPT, Claude, the large Qwens — today's giant models do everything: poetry, legal summaries, code, medical questions, and translation too. They're giant precisely because of that range.

But what LYR needs is one thing: subtitle translation. It never once uses the ability to write poetry, or to write code. Most of that general-purpose intelligence is dead weight on LYR's task (overspec).

Which means — if you narrow it to a single job, even a small model ought to reach the giants on that one point. Give up generality, get specialization back. That was the bet behind "specialization."

Training comes in two stages — model answers, then preferences

So how do you narrow it? You apply a treatment called fine-tuning, in two stages.

This is actually the same path ChatGPT took to get smart. A base language model → show it model answers so it follows instructions (SFT) → polish it with human preferences (RLHF). LYR followed the same order.

1. SFT: show it model answers, in bulk

Stage one is SFT (supervised fine-tuning). In short: show it a huge number of model answers and hammer the form of the task into it.

I had the larger giant model (27B) produce the model answers. Have the giant translate a few thousand items, then have the small model imitate those translations. This is called distillation. Picture the student copying out the teacher's answer sheet.

The effect was dramatic. Japanese→English accuracy went 42% → 84% (+42 points). That is the same level as the retired model I had been running in production until recently. Stage one filled in most of the gap.

That said, SFT has a ceiling. The student imitates the teacher too faithfully — it copies the teacher's phrasing tics wholesale (always rendering "look" as the same literal 「見て」, say). And above all, what the model fundamentally cannot understand doesn't get fixed by imitation.

2. CPO: put a good translation and a bad one side by side

Stage two is CPO (preference optimization — fine-tuning that teaches the model "preferences"). After model answers comes preference.

The method: show it a good translation and a bad one as a pair, and have it learn "this one is better." It's the stage that corresponds to human feedback (RLHF) in ChatGPT's case. The sense of "this is what natural sounds like" and "this isn't," which model answers alone can't convey, gets trained in by contrast.

When I applied this, specialized to Live subtitles, subtitle translation quality went from 90 → 96 (that 90–96 is a score for overall subtitle quality; the 42→84 earlier was accuracy on Japanese→English translation — note that these are different yardsticks). That's level with the giant model currently in service (96), and within touching distance of the retired original (97). Literal renderings of idioms ("it isn't like you" now coming out as the idiomatic 「あなたらしくない」) and picking the wrong subject got corrected, deliberately, one after another.

Doubting the "gold data"

This was the biggest discovery of the whole thing.

In stage-one SFT, you use the teacher's (27B's) answers as "model answers = the right answer." Naturally, I assumed that one answer was the gold data, and that training meant getting closer to it.

At some point I doubted that. What if I had the teacher produce several translations instead of one, and re-picked the best of them on quality? The result — the best candidate disagreed with the "gold data" 86% of the time.

In other words, the teacher's first-shot answer — the thing I had been calling "the right answer" — was not the best one in 86% of cases. It was verbose, or too literal, and a better candidate sat elsewhere in the set.

The lesson: stop preparing one right answer and training toward it. Generate a lot of candidates and select on quality itself (QE = quality estimation). There is no single "correct translation" for a subtitle. (I've stepped in this trap — where chasing the gold data actually makes things worse — in another setting too.)

And the method has a second payoff: it produces the SFT model answers and the CPO material (good/bad pairs) at the same time. That 90→96 above was in fact a combination: stage-two CPO applied with this "material re-picked from candidates." What was doing the work wasn't CPO as a technique so much as the quality of the material you feed it.

How this meshes with quantization

One last warning. The smarts you went to the trouble of training in can be shaved away by last time's quantization. Fine-tuning's effect sits "thin and wide" across the whole model, so aggressive compression tends to take it out first.

So hold fine-tuning and quantization to the same conditions. And always evaluate at the quantization you deploy in production. Celebrating "it got smarter" in the uncompressed state means nothing if the gain is gone after compression.

What you can train in, and what you can't

Fine-tuning is not a cure-all, though. As I stacked up rounds of training, the walls you can get over and the walls you can't separated out clearly.

Transferable by fine-tuning: the "form" of how things are said. Phrasing, idioms, brevity, formatting — the form of how to say it can be pulled up toward the larger models by fine-tuning, even on a small model.
Needs capacity (size): deep comprehension, and emotionally loaded register. Who the subject is, which gender, whether a negation applies, what the proper nouns are (= reading comprehension). And a register that matches the temperature of the scene. These two can't be filled in by teaching good/bad alone; they need the model's capacity. In fact, emotionally loaded register only reached giant-model level (98–100%) — a struggle at 4B — once I moved up to 8B.

To back that up, I ran a controlled experiment. Push the preference training hard on its own and Japanese→English accuracy (comprehension) goes 84 → 85% — inside the judge's noise. The form of how things are said moves; the ceiling on comprehension does not.

To sum up — the "form" of how things are said is cheap to reach for, even on a small model. But "deep comprehension" and "emotionally loaded register" are the model's capacity itself. The recipe and the capacity (the model's size) have to be designed as a set. Specialization isn't magic; it was the work of telling apart which parts you can reach by training and which parts need capacity.

Lessons

Most of a giant model's smarts are things your task doesn't need. Narrow to one job and even a small model gets there on that one point. Training comes in two stages — model answers (SFT = distillation) for the form, preferences (CPO) for the flavor.
Doubt the "gold data." Generating candidates and selecting on quality (best-of-N) gets you further than fitting to a single right answer. The best translation is usually not the "right answer" you prepared.
What you can train in is the form of how things are said; what needs capacity is deep comprehension and emotional register. The form is cheap to train in, but comprehension and feeling need capacity (size). Design the recipe and the size together.

So now my own small model is fast (last time), cheap, and — on one job — smart. It draws level with a general-purpose giant AI on LYR's task and nothing else. It's the last piece needed to let go of borrowed AI.

Appendix: raw data

Experiment	Result	Conditions & caveats
SFT (distillation)	base 4B Japanese→English ~42% → 84% (+42pt)	27B teacher, reference-free binary judge. Multilingual too, thin and wide from a single training run (ES 91 / PT 82 / KO, TH 77 = a gradient in resource volume). The ceiling is the 4B's comprehension capacity
best-of-N QE	700 source → 2,800 candidates (N=4, temp0.7) → selection. best≠gold 86% (451/522)	Selecting on quality is the right call. Selection that pulls toward gold with chrF and the like is a chasing trap
CPO (ALMA-R)	general Page 86.5→91.6 / Manga 83.7→86.9 / Live-specialized 90→96	+4 is the same order as the single-seed noise floor of ±3-5pt = directional. Live 96 is level with the 27B currently in service
register	98–100% = 27B class with 8B-CPO	The register that struggled at 4B shows up once you raise capacity = the register weakness is capacity-bound
Control for the limit	binary preference alone 84→85 (within judge noise)	Preference at the string level doesn't fill in the 4B's comprehension ceiling. It was only cleared by moving up to 8B

Originally published at LYR Performance Note #012 — part of the series “Models — smaller, faster, sharper”. The full set is at lyr.jp/en/research.