GeneLab_999

Posted on Sep 22

I built a Jev-style decision model for Japanese in two days. Claude Code A/B-tested my architecture and killed it.

#ai #machinelearning #nlp #opensource

TL;DR

TypeSafe's Jev introduced "System One" models: no text generation, you send a state and typed questions, you get typed answers with calibrated probabilities in one forward pass. Open reproductions followed within weeks. The most complete one is Laya (ModernBERT-large, 421M).
I benchmarked laya-multilingual on 300 Japanese business emails. choice works. score (ordinal) and bool are below the majority-class baseline.
The score failure is a positional bias: the first-listed option is chosen 0 to 1 times out of 300, under five different orderings and wordings. It reproduces in English (0/290). The English laya checkpoint does not have it. Filed as NandhaKishorM/laya#131.
I built a Japanese one, sokudan-ja-310m, on ModernBERT-Ja. On the same 300 items with unseen schemas it beats laya-multilingual on all three primitives, three seeds. Model, code, benchmark and training-data recipe are public.
Day 1 produced a model that ignored the input text. Day 2 fixed the data and it still failed. The actual cause was my architecture. Claude Code, which I had handed the spec to, ran the A/B that proved it.

Model: https://huggingface.co/GeneLab/sokudan-ja-310m
Code: https://github.com/hiroki-abe-58/sokudan

What a System One model is

If you have ever asked an LLM "which department should handle this ticket, answer with one word" and then written a regex to parse the reply, you know the problem. You are using a text generator as a classifier and paying for it in latency, cost and parse failures.

A System One model skips the generation. Three question types:

type	asks	returns
choice	pick one of N options	choice, per-option probabilities, confidence
score	rate on an ordinal rubric	score, distribution, confidence
bool (Jev calls it noul)	is this statement true	P(true)

Multiple questions go in one request and are evaluated against the same state independently. Laya implements this as an encoder with one [MASK] token per option; the hidden state at each mask is scored and softmaxed within the question. Because the answer space is defined at request time, new schemas need no retraining. On an RTX 5090 it runs at 22 ms per item.

The benchmark

I generated 300 Japanese business emails with a local LLM (Qwen3 30B-A3B), label-conditioned: the prompt says "write a slightly irritated billing inquiry that hints at cancellation", so the generation conditions are the gold labels. No annotation. Three questions per document: 4-way department routing (choice), 3-level urgency (score), "hints at churn" (bool).

Baselines, all on the same 300 items with the same scoring code: laya-multilingual, laya (English checkpoint fed Japanese), majority class, random, and the generating LLM itself as a classifier (an upper bound, not a fair comparison, since it wrote the data).

I did not benchmark Jev itself. TypeSafe's Master Customer Agreement (2.3(b)) prohibits using the service or its output to develop a similar product, and this is one.

Results for laya-multilingual

	choice acc	score RPS (lower is better)	score acc	bool acc	bool AUROC
laya-multilingual	0.747	0.232	0.443	0.543	0.523
majority class	0.380	0.197	0.460	0.703	—
random	0.253	0.201	0.403	0.513	—

choice is fine: twice the majority baseline, three times random. If you need multi-class routing in Japanese, this checkpoint works today.

score is worse than always predicting the majority level. The confusion matrix explains why:

gold \ predicted	soon	work is blocked
not urgent (77)	58	19
soon (138)	78	60
work is blocked (85)	30	55

The lowest level is never predicted. My first guesses were "weak on Japanese negation" or "ordinal training is broken". Five more conditions ruled both out:

condition	options (in listed order)	first option chosen	argmax counts
A original	not urgent / soon / blocked	0 / 300	[0, 167, 133]
B reversed	blocked / soon / not urgent	0 / 300	[0, 50, 250]
C relabeled	low / mid / high	1 / 300	[1, 291, 8]
D relabeled, reversed	high / mid / low	1 / 300	[1, 0, 299]
E four levels	not at all / not urgent / soon / blocked	0 / 300	[0, 78, 37, 185]

Compare B and D. The last-listed option means "not urgent" in one and "high" in the other. Both attract most of the mass (250 and 299). The model is answering by slot, not by label.

Then I ran the same five conditions on 290 English emails, because English is a training language for this checkpoint and "it can't read the input" should not apply:

model	Japanese (n=300)	English (n=290)
laya-multilingual	0, 0, 1, 1, 0	0, 0, 0, 0, 0
laya (English checkpoint)	13, 56, 8, 1, 110	65, 74, 0, 5, 4

Still zero. In English, laya-multilingual also gets score RPS 0.340 (random is 0.197) and bool AUROC 0.355, which is below 0.5, meaning the ranking is inverted. The English checkpoint picks the first option 22 to 26 percent of the time under the same conditions. This is not a language gap; it is something specific to the multilingual checkpoint.

Practical consequence: ordinal rubrics almost always list the lowest level first. On this checkpoint, score will almost never return the lowest level regardless of input.

I verified my harness against the official laya package with laya.load() / agent.predict() straight from the README; probabilities match to float precision. Repro: docs/baseline_en.md.

Temperature scaling, for the record, fixes ECE (0.148 to 0.087) and changes no argmax at all. As expected.

So I built one

sokudan-ja-310m: sbintuitions/modernbert-ja-310m backbone (MIT), full fine-tune, one <mask> per option, softmax within the question. Two design choices worth mentioning:

score is not a multi-class softmax. It uses a cumulative-link parameterization that keeps the CDF monotone for any number of levels K, decided at request time: each level's marker contributes a non-negative increment via softplus, and P(y > k) is a sigmoid over the running sum. K=2 and K=7 share the same head.
All training data is gold-labeled synthetic data from the same label-conditioned generation. No Laya or Jev outputs anywhere.

Results on the same 300 items, three seeds, mean ± SD. The benchmark's three schemas were never in training:

	choice acc	score RPS	score acc	bool acc	bool AUROC
sokudan-ja-310m	0.847 ± 0.009	0.090 ± 0.023	0.763 ± 0.088	0.788 ± 0.010	0.789 ± 0.043
laya-multilingual	0.747	0.232	0.443	0.543	0.523
majority class	0.380	0.197	0.460	0.703	—

Seed variance on score accuracy is large (0.663 / 0.800 / 0.827). The distributed weights are seed 0, which is the worst of the three on that metric and the best on bool AUROC. Seed 0 was designated before any of the three finished training. Seeds 1 and 2 are revisions in the same repo.

Under the five position-bias conditions, sokudan picks the first option 45 to 98 times out of 300 (i.e. normally), across all seeds.

The part I actually want to write about

Day 1 ended with a model whose bool AUROC on the benchmark was 0.513. Diagnostics showed something worse than "weak": replacing the input text with an empty string improved bool accuracy (0.623 to 0.703). The model was reading the question and emitting a prior. My training data's bool questions were all mechanical derivations ("does this document belong to category X", "is urgency at least level k") and none required reading for intent. "Hints at churn" is an intent question. You cannot transfer a skill you never trained.

Day 2 morning: Claude Code and I redesigned the data. Twenty-four attributes baked into the generation prompt as conditions, split into three tiers: surface (mentions a deadline), explicit attitude (states dissatisfaction), implicit intent (implies it is about to decline). Five attributes held out entirely so we had an unseen-schema validation set that was not the benchmark. Claude Code caught two things in the verification pass that I would not have: implicit-intent attributes imply each other, so independent 50/50 sampling creates false labels that are actually true; and discarding verifier disagreements removes hard cases asymmetrically, teaching "always yes". Both fixed.

Day 2 afternoon: trained on the new data. Held-out bool AUROC 0.509. Every held-out attribute at chance, including "ends with a question mark". And choice and score were worse than Day 1 despite 2.6 times the data.

Here is the diagnostic that mattered. On attributes the model had trained on, evaluated on unseen documents, only 4 of 14 were above 0.7, and the top two were "contains numbers" and "mentions a deadline". Lexical. Every attitude and intent attribute sat at 0.5 even though the model had seen thousands of examples of each. Before asking why it did not transfer, we had to ask why it had not learned.

My architecture encoded the state and the question separately and merged them in a two-layer cross-attention head. The selling point was that the state is encoded once per request and broadcast to all questions, so latency does not grow with question count. I had rejected Laya's joint layout (question and state in one sequence) on the grounds that ModernBERT has global attention only every third layer and a 128-token local window, so a question block placed far from the state could not see it.

Claude Code proposed the A/B and ran it: same data, same mix, same epochs, same learning rate, the only change being joint encoding with no cross-attention head.

	cross-attention (mine)	joint (Laya's layout)
held-out bool AUROC, unseen schema	0.506	0.872
implicit-intent tier	0.486	0.786
explicit-attitude tier	—	0.995
trained attributes on unseen docs	4 of 14 above 0.7	14 of 14 between 0.96 and 1.00
seconds per epoch	464	330

Ablating the state (empty or shuffled) drops joint to 0.485 / 0.504. It is reading the text.

My local-attention argument was correct and irrelevant. The mean sequence length in this data is 160 tokens. The "question block at position 3000" I was worried about never occurs. Meanwhile, the only held-out attribute joint failed was "ends with a question mark" (0.688), which is a positional question and exactly where a 128-token window hurts. The concern was real; I had aimed it at the wrong place.

I retracted the design, the latency claim, and the paragraph in the architecture doc that explained why joint would not work. v0.1 ships joint. Claude Code wrote the retraction section with the A/B numbers and a note on where the original argument's premise failed.

Two things I put in the spec that earned their place: never fabricate a number, write TBD (no estimate ever leaked into a doc), and stop at the gate (three sub-majority checkpoints were stopped before publication). One thing I did not put in the spec: permission to refute the spec's author. It did that on its own.

Limits

On the benchmark, bool under-predicts true (mean P(true) 0.125 vs a gold rate of 0.297). AUROC 0.789, so the ranking works; set your threshold to your prior. Temperature does not shift this.
Calibration helps choice and bool ECE and hurts score RPS (0.090 to 0.149). Default is uncalibrated; temperatures.json is included; refit on your own validation set.
Long inputs: held-out bool AUROC drops from 0.904 on short documents to 0.730 at 400 to 800 tokens. Local attention.
Latency scales with question count under joint encoding. usage.backbone_passes in the response tells you how many.
Single benchmark, 300 items, all synthetic from one LLM. Real-data validation is next.

DEV Community