Laya's README reports a raw ECE of 0.466 on the typed-decisions benchmark before temperature scaling. The method it's named for - Reinforcement Learning for Calibrated Decisions - is supposed to be what makes those probabilities mean something. So I went looking for the RL term's contribution.
I found a suspect with a very solid alibi: it was doing nothing.
Atomic answer: Laya's RL term is an evolution-strategies estimate of the gradient of a noise-smoothed proper score. As the noise scale goes to zero it converges to the cross-entropy gradient the model already computes. At practical noise levels it makes the model over-sharp at inference, and across 30 training runs it never improved a single accuracy number over plain cross-entropy.
TL;DR
Laya's RL reward is a score-function / evolution-strategies estimator of
∇ E_ε[R(softmax(z+ε))]. For the log score, that gradient→ t − softmax(z)asσ → 0-which is exactly the CE gradient.The noise-smoothed objective's optimum satisfies
E_ε[softmax(z*+ε)] = t. Inference reports the noise-freesoftmax(z*), which is provably sharper than the target (I prove it for binary questions).On the real benchmark, fitted temperature rises monotonically with the training noise scale:
1.14 → 2.42asσgoes0.25 → 4. Accuracy does not move.Across CE-only, RL+CE, RL-only, a σ sweep, a reward-weight sweep, and a reward-composition sweep, CE-only is at least as good on every metric I track. The only thing that raised accuracy was doubling the input token budget.
This is the short version. The deep version - the estimator derivation, the proof, and the per-run tables - is in the companion post: Inside RLCD: The Estimator, the Proof, and the 30 Runs That Changed My Mind.
First, what Jev and Laya actually are
TypeSafe's Jev is a "System One" model: you hand it unstructured state, it returns a probability distribution over a closed, typed set of options - a choice among named options, an ordinal score, or a binary noul (yes/no) probability. No text generation. The headline claim is calibration: an answer given with 0.8 confidence should be right about 80% of the time.
The method credited for that, RLCD, has no published paper, no reward function, no dataset, and no calibration figure. That's the whole reason this project exists - I wrote about Jev from the outside here: Jev, Explained: The AI That Refuses to Write a Word.
Laya is the only open reproduction of the same interface. It publishes code, checkpoints, and an evaluation on LocalLLaMA/typed-decisions: 400 test cases, 2,000 decisions, four workflows. Its fine-tuned checkpoint reports 0.766 accuracy, Brier 0.062, ECE 0.213. Against Jev's published 0.727. That's the target I reproduced and then took apart.
The method, in one paragraph
Laya encodes each question as its own row - [CLS] <type> question: <instr> [SEP] [MASK] opt0 [MASK] opt1 ... [SEP] <state> [SEP] - runs a ModernBERT-large encoder, a small transformer head, and reads one logit out of each option's [MASK] position. Training adds noise to the logits, scores the noisy distributions with a proper scoring rule (log score + spherical + RPS), and applies a REINFORCE-style update next to a soft cross-entropy term on the teacher's probability distribution.
Here's the full training step:
z ← model(x) # one logit per option
for g in 1..G: # G = 4 noise samples
ε_g ~ N(0, σ²), zero-mean over options
q_g = softmax(z + ε_g)
r_g = R(q_g, t) # proper score vs the teacher target t
adv = (r − mean(r)) / std(r)
L_rl = −mean(adv · log N(z+ε | z, σ²)) # score-function estimator
L_ce = −Σ t · log softmax(z) # soft cross-entropy
loss = w_rl · L_rl + w_ce · L_ce
σ anneals from 0.4 to 0.1 over four epochs. After training, one temperature per question type is fitted on a held-out slice.
The finding: the RL term is a smoothed cross-entropy gradient
Write J_σ(z) = E_ε[R(softmax(z+ε), t)]. The score-function identity gives
∇_z J_σ(z) = (1/σ²) · E[ R(softmax(z+ε), t) · ε ]
which Laya estimates with four samples and a group-mean baseline. For the log score, ∇R = t − softmax(z). So as σ → 0, the RL estimator's expected value is t − softmax(z) — the exact negative CE gradient. Laya is paying for a Monte Carlo estimate of a gradient that the CE term next to it computes exactly.
That alone would be a curiosity. The problem is the noise: it doesn't vanish, it biases the optimum.
The over-confidence is real, and it grows with the noise
The smoothed objective's stationary point satisfies E_ε[softmax(z*+ε)] = t. Averaging a softmax over logit noise flattens it. So the model's noise-averaged prediction matches the target, while the noise-free prediction that inference actually reports is sharper than the target. Over-confident by construction.
I checked this two ways. The toy first: minimise the smoothed log score for t = [0.7, 0.2, 0.1] and look at both predictions.
| σ | noise-free softmax(z*) | noise-averaged E[softmax(z*+ε)] | target |
|---|---|---|---|
| 0.5 | [0.723, 0.187, 0.090] | [0.698, 0.202, 0.100] | [0.7, 0.2, 0.1] |
| 1.0 | [0.780, 0.154, 0.066] | [0.696, 0.205, 0.099] | [0.7, 0.2, 0.1] |
| 2.0 | [0.905, 0.071, 0.024] | [0.706, 0.196, 0.098] | [0.7, 0.2, 0.1] |
The averaged column tracks the target. The noise-free column is the one inference ships, and at σ=2 it puts 0.905 on a class the teacher gave 0.70.
Then the real thing. Train with fixed noise and fit the temperature:
| σ | fitted T (noul) | NLL vs targets | accuracy |
|---|---|---|---|
| 0.25 | 1.14 | 0.866 | 0.777 |
| 0.5 | 1.16 | 0.871 | 0.772 |
| 1 | 1.38 | 0.890 | 0.764 |
| 2 | 1.72 | 0.928 | 0.769 |
| 3 | 2.08 | 0.956 | 0.775 |
| 4 | 2.42 | 0.987 | 0.774 |
Fitted temperature and NLL climb monotonically with σ. Accuracy is flat. The temperature is the model's own tell that its raw outputs are too sharp - and it rises exactly as the theory predicts.
30 runs later: the loss doesn't matter, the input budget does
I ran the full ablation. Reproduced Laya: 0.773 accuracy, Brier 0.054. CE-only: 0.782, Brier 0.052. RL-only: 0.769. Then a reward-weight sweep and a reward-composition sweep. Nothing beat CE-only.
| configuration | accuracy | Brier | NLL |
|---|---|---|---|
| CE-only | 0.782 | 0.052 | 0.861 |
| RL+CE (Laya) | 0.773 | 0.054 | 0.866 |
| RL-only | 0.769 | 0.054 | 0.866 |
| CE-only, 1024/256 tokens | 0.789 | 0.0495 | 0.858 |
The only thing that moved accuracy was matching the token budget the checkpoint card documents (1024 context / 256 option tokens), which my port had under-provisioned at 512/192. That's the option-token starvation Laya warns about: many options share a fixed budget, so a large option set loses tokens per option.
What this means if you're building typed-decision models
If your targets come from a data engine you control, cross-entropy on those distributions is already a proper-score optimum. You don't need RL for it.
Logit-noise smoothing is not a free regulariser. It has a direction, and the direction is over-confidence. If you use it, temperature scaling is undoing your training, not just cleaning up after it.
Report the fitted temperature. It's a cheap, honest signal that an upstream stage distorted the distribution.
Check your option-token budget before blaming the loss. It was the only accuracy lever I found.
How to reproduce
Everything is public. The code, configs, and result files are in LakoreAI/sev. Each run is a single A100/A5000 fine-tune of Laya's own checkpoint.
If you build decision models for a living, I'd like to hear whether the fitted temperature signal matches what you see in production. It's the cheapest diagnostic in this whole post.
Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.
Top comments (0)