I didn't fix the bug: contributing to a 20k-star ML repo by measuring it

Last week I opened an issue on laya, an open-source decision model with a little over 20k stars. The issue reported that one of its checkpoints almost never picks the first-listed option on ordinal questions. About 36 hours later, three things had happened. The maintainer had documented the limitation in a release. He had asked me to run the before/after on the retrained checkpoint. And he had merged a regression check I wrote. That was four days after my first measurement.

Nothing is fixed. The fix is a retrain, and the retrain hasn't happened yet. What I contributed is narrower. I think it is also the kind of thing an outsider is best placed to contribute: a measurement nobody could wave away, and a tool that says whether the fix worked.

The fastest way I've found to contribute to an ML repo you don't maintain: measure it, make the measurement impossible to explain away, and then hand the maintainer a tool.

The timeline, in UTC:

• Sep 20 — first Japanese baseline run
• Sep 22, 05:56 — issue #131 opened
• Sep 22, 06:40 — condition F posted (position vs. label, explained below)
• Sep 22, 07:15 — a third party, AlKor13, traces it to the weights
• Sep 22, 10:56 — my labelled A/B test of his hypothesis
• Sep 22, 20:40 — the maintainer's decision
• Sep 23, 07:25 — v0.3.7 ships with the limitation in the README
• Sep 23, 14:42 — PR #259 opened
• Sep 23, 17:48 — PR #259 merged

What laya is, and why I was measuring it

laya is a non-autoregressive "System 1" decision model. You give it some text and a set of typed questions: choice, score for ordinal levels, and noul for yes/no. It answers all of them in one forward pass, with probabilities and no text generation. It ships several checkpoints, including an English laya and a laya-multilingual.

I'm building a Japanese model in the same space (sokudan). Before writing any training code, I wanted a baseline: how well does the existing multilingual checkpoint handle Japanese?

I built 300 label-conditioned Japanese business emails. The labels were fixed first. Then a local LLM wrote an email to match them, and any email containing the label words themselves was thrown out and regenerated. Each email got three questions: department (4-way choice), urgency (3-level score) and whether the customer is hinting at cancelling (bool).

The results split by question type. choice worked: 0.747 accuracy against a 0.380 majority-class baseline. score lost to the majority baseline (RPS 0.232 vs 0.197, lower is better). bool also lost to the majority baseline (0.543 vs 0.703), with an AUROC of 0.523. That is barely better than ranking at random.

One comparison I did not run is TypeSafe's Jev, the hosted API laya benchmarks itself against. Its customer agreement prohibits using the service or its outputs to develop similar products, and my project is a similar product.

The number that didn't fit

"Worse than majority" is a finding, not an explanation. The confusion matrix for score was more interesting. The lowest level, "not urgent", was never predicted in all 300 emails, even though 77 emails had it as the gold label.

At least three explanations fit that. The ordinal head could be degenerate. The model could be reading position rather than content. Or this particular Japanese phrase could be the problem.

To separate them, I re-ran the same 300 emails under five option schemas: original order, reversed, reworded (low / medium / high), reworded and reversed, and four levels. In all five, the first-listed option was chosen 0 or 1 time out of 300.

The cleanest pair was the original order against the reversed one. "Not urgent" was chosen 0 times when listed first and 250 times when listed last. The emails were the same, the words were the same, and only the slot had changed.

Making it hard to explain away

Before reporting, I tried to list the replies a busy maintainer could reasonably give, and to answer each one with data before he had to ask.

"Your harness is wrong." I re-ran the benchmark using only the two lines from the README, laya.load() and agent.predict(), with no wrapper of mine in between. Then I compared the results to my saved probabilities. The maximum difference was 0.0 after the probability floor I apply, and 2.2e-16 on the raw values.

"It's a Japanese problem." I built 290 English emails with the same label weights, the same generator and the same rejection rules, and ran the same five conditions. The multilingual checkpoint went 0 for 290 in every one of them. English is one of its training languages, so "it can't read the input" doesn't explain this.

First-listed option chosen, per condition (original / reversed / reworded / reworded reversed / four levels):

checkpoint	Japanese (n=300)	English (n=290)
laya-multilingual	0, 0, 1, 1, 0	0, 0, 0, 0, 0
laya (English)	13, 56, 8, 1, 110	65, 74, 0, 5, 4

The English checkpoint picks the first slot in most conditions, on the same data through the same harness. So the task isn't the cause, and neither is the harness. It comes down to one checkpoint. I opened #131 with a title that said exactly that.

(A correction to myself: in the issue I summarised the English checkpoint as picking the first option "22–26% of the time". That holds for the two original orderings. It is 0 in one condition and under 2% in two others. The table above is the honest version.)

"Is it the position or the word?" Within any single fixed ordering, "the model rejects slot 1" and "the model rejects whichever word sits in slot 1" predict the same table. Someone on X asked whether the effect survives shuffling. Twenty-two minutes later I had condition F: the option order is shuffled per item, with a fixed seed. The results:

• The first slot was chosen 0 times out of 300.
• Choices by slot were [0, 149, 151].
• Choices by label were 75 / 93 / 132.
• Each label had sat in the first slot 90 / 109 / 101 times.

So every label lost exactly the picks it had while it was in slot 1. Slots 2 and 3 split evenly, so this isn't a preference for the last slot either. One run was enough to separate position from label.

Someone else went deeper

What happened next was the best part, and it wasn't my work. AlKor13, who is not a maintainer, read the raw marker logits straight off the forward pass, before temperature and softmax.

On the identical code path, swapping only the checkpoint made the hole appear or disappear. Then he ran the control I wish I'd thought of: three identical options, so the inputs differ only by position. The English checkpoint came out essentially flat. The multilingual one came out sharply position-dependent. That settled it: the cause is in the weights, not the code.

He went one step further. Score options are rendered with a level N: prefix, and dropping that prefix made the slot-0 suppression vanish from the raw logits. But he stated the limit of his own result. Without the prefix the input is off-distribution, so the recovery might just be the learned prior getting scrambled. Settling that would take labelled data.

Testing someone else's hypothesis, and my own

I had the labelled data, so I ran three renderings on both benchmarks: A is the shipped level N: rendering, C drops the prefix, and D uses word ordinals. My harness reproduced agent.predict() to within 4.6e-5 under condition A. I compared the renderings with paired McNemar tests on identical items:

bench	change	score accuracy	items that flipped correctness	p
ja	A → C	0.447 → 0.513	56.7%	0.145
ja	A → D	0.447 → 0.570	38.3%	0.0007
en	A → C	0.266 → 0.428	56.9%	0.0003
en	A → D	0.266 → 0.293	19.3%	0.350

Two of the four comparisons are significant, and they are different conditions in different languages. D wins in Japanese and C wins in English, and each is null in the other language. The claim this supports is "the shipped rendering isn't the best one for this checkpoint". It does not support "drop the prefix and it's fixed".

I'll admit the accuracy column alone had me halfway to writing the second claim. The paired test is what stopped me, and I said so in the thread.

The column that matters is the flip rate. Under C, 57% of items change correctness, while headline accuracy moves by 6.7 to 16.2 points. A cosmetic change to the option string nearly re-rolls the prediction. That is an instability, not a mitigation. As a control, removing the prefix makes the English checkpoint worse on the English set (0.583 → 0.500). So level N: isn't harmful in general. The multilingual checkpoint has learned something wrong about that specific pattern.

The maintainer's call

About fifteen hours after I opened the issue, the maintainer, NandhaKishorM, replied to both of us. He called the thread "a model of how to run a bug down" and made three decisions:

1. The bug is in the weights, not the code.
2. He would not ship a new default rendering. He cited the 57% flip rate: changing the rendering would swap one instability for another.
3. The real fix is in training. Ordinal options need the same position balancing that choice options already get.

The next morning, v0.3.7 shipped with a line in the README's limits section. It says laya-multilingual has a position bias on score questions, that English score questions should go to the English checkpoint, and that users should validate score outputs on their own data for other languages. The issue stays open until a retrained checkpoint lands.

I offered to run the same A–F and A/C/D before/after on that checkpoint. He said he'd ping me when it's ready.

Handing over a tool

That offer had a weak point: it depended on me. My benchmarks are CC BY 4.0, so anyone can rerun them. But the question the retrain has to answer doesn't need labels at all: did the slot prior go away? So I wrote PR #259:

python research/eval/presentation_checks.py --model <retrained checkpoint> [--subfolder multilingual]

It uses ten short English support messages that are fixed in the file, with no dataset and no labels. It runs two checks:

• score_slot0_identical is AlKor13's identical-option control turned into a gate. It measures slot 0's raw logit minus the slot mean, over K = 3, 4 and 5 identical levels, and requires the result to be at least −0.20.
• score_first_slot_permuted puts the three real levels in all six orders for each message, so every level sits in every slot exactly twice. A model whose answer doesn't depend on order picks the first slot in exactly 1/3 of decisions. The gate is at least 0.15.

Results on the checkpoints that ship today:

checkpoint	slot-0 logit (gate ≥ −0.20)	first-slot rate (gate ≥ 0.15)	verdict
laya (English)	+0.664	0.217	PASS
laya-multilingual	−0.492	0.017	FAIL

I fixed the threshold conditions before the real 10-message run, and I didn't move them afterwards. There were two conditions. First, in every leave-one-out subset, multilingual must fail and English must pass. Second, each checkpoint's worst leave-one-out value must clear its gate by at least 0.10 logit (slot 0) or 0.05 (first-slot rate). The tightest margin turned out to be the English first-slot rate, at 0.054.

The gate is one-sided on purpose. With identical options, the English checkpoint isn't flat either. It leans toward early slots, and the lean grows with K. A two-sided "no position effect" gate would fail the checkpoint everyone agrees is fine. So the gate asks only the narrower question #131 is about: is slot 0 suppressed?

The harness checks itself before it gives a verdict. It first compares its logit path against Agent.system_one on every message. On the shipped checkpoints, the maximum |Δp| was 4.98e-5, which is just the API's 4-decimal rounding. If the difference exceeds 1e-3, the script exits with code 2 and reports no verdict. Exit 0 means pass, 1 means a check failed, and 2 means the harness disagrees with the package. There are 69 offline tests using scripted logits. For example, a scripted slot-0 hole fails both checks, and an order-invariant model scores exactly 1/3.

The PR also lists its own limits. Passing is not accuracy. The check covers English only, score only and ten short messages. And the thresholds were set on CPU with fp32.

Before opening the PR

I spent more time reading the repo than writing the script. Tests there are plain scripts, not pytest. CI doesn't load model checkpoints. Research code lives under research/. A new test has to be registered in both ci.yml and release.yml, and an open PR, #184, hadn't yet settled how research tests get wired in. So I didn't wire mine in. I said so in the PR and offered to do it once #184 lands. Nothing under laya/ changed, and I added no new dependencies.

The same afternoon, another contributor, Rukafuu, had proposed a general metamorphic robustness framework in #244, and the maintainer had invited them to start with option permutation. That was their work to do. I kept my check specific to #131 and noted that it could move into their framework once it exists. Their PR, #269, has since been merged.

259 was merged about three hours after I opened it. The maintainer said a label-free check answering one question was "exactly what #131 needs". He also said that separating a failed check from a harness disagreement in the exit codes made it "easy to trust". That second point is the one I'm proudest of, because it's the part nobody asked for.

On using Claude Code

I used Claude Code throughout: for the harness, the offline tests and drafting the PR text. The commit carries a Co-Authored-By line. The decisions about what to measure, what to claim, what to take back and where to stop were mine, and so are any mistakes.

What I'd tell myself next time

• Measure before you report, and check your harness against the official package before you measure anything that matters. "Your harness is wrong" is the cheapest possible dismissal. Rule it out first.
• Answer the obvious objections before anyone raises them. Is it the language? Run it in English. The harness? Check parity. The word? Shuffle per item. Each of those came back as one line in the thread instead of a round-trip.
• Take back your own overclaims, out loud. McNemar turned my "the prefix is the problem" into "the rendering is unstable". The weaker claim is the one the maintainer could act on.
• Credit precisely. The most important control in this story came from AlKor13. My PR is his idea turned into a gate, and the PR says so.
• Don't take work someone else was just handed. #244 was Rukafuu's. Keeping my check narrow cost nothing and avoided stepping on them.
• Don't claim the fix. I found it, reported it, and merged the tool that will say whether the retrain worked. That's the whole claim.

Where it stands

131 is open, and the retrain is pending. When it lands, presentation_checks.py gives the first answer, and I'll run the labelled A–F and A/C/D before/after on top of it.

What I posted the day it merged:

• Repo: https://github.com/NandhaKishorM/laya
• Issue #131: https://github.com/NandhaKishorM/laya/issues/131
• PR #259: https://github.com/NandhaKishorM/laya/pull/259
• Related: #171, #244
• My data and repro scripts: baseline_ja.md, baseline_en.md