DEV Community: firefrog

Jev's Boring Use Cases Are the Ones That Actually Work

firefrog — Sun, 27 Sep 2026 07:31:04 +0000

💡 TL;DR & Key Takeaways:

TL;DR: Jev is a lightweight model that runs far faster and cheaper than larger LLMs, delivering competitive accuracy for tasks such as reranking, judging, and real‑time routing, but it still lags on generation, multi‑step reasoning, and high‑stakes decisions. It is most valuable when paired with an LLM, acting as a fast, cost‑effective pre‑filter or safety check.

Speed & cost – 777 judgments in under 0.7 s (~¼ ¢), 0.35 s per passage vs Fable’s 8.83 s, roughly 580× cheaper.

Accuracy trade‑off – BM25+Jev raises top‑1 accuracy from 5 % to 18 % and top‑10 from 38 % to 62 %; catches 6/7 planted defects versus Fable’s 7/7.

Real‑time use cases – agent routing 145–271 ms, model routing ~1 s vs 4–14 s for an LLM, browser automation ~7.

Every's Dan Shipper ran Jev against Fable 5.1 on the same writing-defect checks: Jev took 0.35 seconds per passage against Fable's 8.83 (about 580x cheaper), catching six of seven planted defects to Fable's seven. That trade-off — much faster and cheaper, slightly less accurate on judgment calls needing real reasoning — holds across every use case below, and every number traces to someone outside TypeSafe.

Reranking: 5% to 18% Top-1 Accuracy With One Call Per Candidate

Web search returns ten results; usually two or three are actually relevant. TypeSafe's own reranking cookbook runs a cheap BM25 keyword shortlist first, then one Noul question per query-candidate pair to score relevance:

Metric	Before (BM25 alone)	After (BM25 + Jev rerank)
Top-1 accuracy	5%	18%
Top-10 accuracy	38%	62%
Cost for 1,200 scoring calls	—	$0.0645

I'm citing this as a TypeSafe number, not an independent one — it's their cookbook, on their benchmark. But the mechanism is simple enough to verify against your own retrieval set. The same shape works on live web search: any search API returning page content can feed the shortlist-then-score pattern, Jev scoring instead of an LLM reading all ten pages end to end — the same latency-and-noise problem I dealt with building a real-time Top-K ranking system.

Judging: 777 Judgments in Under 0.7 Seconds

Every's head of evals, Mike Taylor, ran 37 documents through 21 questions each (777 total judgments) in under 0.7 seconds, for about a quarter of a cent. That's not a claim about accuracy — it's a claim about what becomes possible once judging is nearly free. A check too expensive to run on every document becomes a check you run on every document, every time.

Dan Shipper's follow-up test, described above, is the more careful one: the same four writing-defect checks, twelve passages each. Jev at 0.35 seconds and roughly 1/580th the cost, catching six of seven planted defects; Fable caught all seven, slower and more expensively. Neither number makes the other wrong — they're different points on the same speed-cost-accuracy trade-off.

Real-Time Decisions: Latency as the Product

A few independently reported numbers, all outside TypeSafe's own benchmarking:

Agent routing (blackbarata, r/LLMDevs): 145–271ms.
Model routing (u/TigerOk4538, r/AI_Agents): ~1s for Jev vs. 4–14s for an LLM.
Browser automation (browser-use/jev-ultrafast): a Google Flights demo finished in ~7.1s end to end.
Bulk analysis (Matthew Berman, X): 724 ads across 37 brands in ~40s for ~$0.09.

None of these are controlled benchmarks — single developers reporting single runs. What they share: decisions cheap and fast enough for a request path or UI loop, not just an offline batch job, the same economics I've tracked on the LLM side in five ways to invalidate your prompt cache.

Where an LLM Still Wins

Anything that needs generated text, code, or an explanation.
Multi-hop reasoning and tasks with real indirection.
Arithmetic, counting, and date math.
Low-volume, high-stakes single decisions, where one wrong call costs more than speed saves.

Share your thoughts in the comments — I’d love to hear how this technology is impacting your industry.

👉 Be sure to press the like button and follow me. It would be a great motivation for me.

👉 Follow me: LinkedIn | GitHub

The Pattern

Nobody serious is using Jev instead of an LLM. They're putting it next to one: reranking what the LLM reads, judging what it produced, routing before it gets involved, or holding a destructive action an agent was about to take. That's a narrower story than "a new frontier model," and a more useful one to build against.

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

I Reproduced RLCD to Find Out What Its "Calibrated RL" Actually Does

firefrog — Sat, 26 Sep 2026 15:05:04 +0000

TypeSafe's Jev returns calibrated probabilities over a closed set of options and credits "Reinforcement Learning for Calibrated Decisions" for it. No paper, no reward function, no dataset, no calibration figure. Laya is the only open reproduction of the same interface, so I reproduced it, then took its training loop apart.

Atomic answer: Laya's RLCD RL term is an evolution-strategies estimate of the gradient of a noise-smoothed proper scoring rule. As the noise vanishes it equals the cross-entropy gradient the model already computes. At real noise levels it makes inference over-confident, and across 30 runs it never beat plain cross-entropy on any accuracy or proper-score number. The only lever that raised accuracy was the input token budget.

This is the short introduction to the paper. The full report, code, and result files are linked at the bottom.

💡 TL;DR & Key Takeaways:

TL;DR

Reproducing Laya’s RLCD shows that its “RL” term is just a noise‑smoothed estimate of the cross‑entropy gradient, which makes inference over‑confident, and that any accuracy gains come from increasing the option‑token budget rather than from the RL loss itself.

Key points

The RL gradient converges to the cross‑entropy gradient as noise vanishes, so it adds no new information beyond CE.

Logit‑noise smoothing causes the trained model to output sharper predictions than the target distribution, leading to over‑confidence at inference time.

Across multiple ablations, the only factor that improves proper‑score metrics is the input token budget; the RL loss and its hyperparameters have negligible impact.

What the paper is about

Two models. Jev, from TypeSafe, is the closed model: unstructured state in, typed probabilistic decisions out - a choice among named options, an ordinal score, or a binary noul. It is marketed on calibration: a 0.8 answer should be right about 80% of the time. Laya is the open reproduction: same interface, published code, checkpoints, and a benchmark (LocalLLaMA/typed-decisions, 400 cases, 2,000 decisions).

Laya's loop adds noise to the logits, scores the noisy distributions with a proper scoring rule, and applies a REINFORCE-style update next to a soft cross-entropy term. That is the "RL" in RLCD. I reproduced it, verified it tensor-for-tensor against the released weights, and then asked one question: what does that RL term actually contribute?

Three things the paper shows

1. The RL term is a noisy estimate of the cross-entropy gradient. Write J_σ(z) = E_ε[R(softmax(z+ε), t)]. The score-function identity plus Stein's lemma turns the RL gradient into the CE gradient averaged over logit noise. For the log score, as σ → 0 it converges to t − softmax(z) - which the CE term next to it already gives exactly.

2. The smoothing makes inference over-confident, provably. The smoothed optimum satisfies E_ε[softmax(z*+ε)] = t, so the noise-averaged prediction matches the teacher. Inference runs at ε=0 and reports the noise-free softmax(z*), which averaging has flattened, so it is sharper than the target. I verified it on trained models: at σ=2, the noise-averaged prediction lands within −0.006 of the target while the noise-free one is over-sharp by +0.108. The fitted temperature rises monotonically from 1.14 to 2.42 as the noise scale goes 0.25 → 4.

3. The loss is not the lever; the input budget is. Across CE-only, RL+CE, RL-only, a σ sweep, a reward-weight sweep and a reward-composition sweep, CE-only is at least as good on every proper score. The only change that raised accuracy was matching the checkpoint's documented 1024-token context and 256-token option budget.

model	accuracy	Brier
CE-only (512/192)	0.782	0.052
RL+CE (Laya recipe)	0.773	0.054
CE-only, 1024/256	0.789	0.0495
Laya typed-decisions (reference)	0.766	0.062

The result I did not expect

The budget effect is small on typed decisions. On a high-cardinality task it is not. I recast four standard classification sets into typed choice questions and scored base Laya and my model zero-shot at two option-token budgets:

task	options	Laya 256	Laya 512	Sev 256	Sev 512
AG News	4	94.5	94.5	94.3	94.3
Emotion	6	59.8	59.8	60.2	60.2
SST-5	5	34.3	34.3	42.9	42.9
Banking77	77	15.8	31.2	15.8	32.0

AG News and Emotion reproduce Laya's own published numbers (95.0 and 59.5). The budget does nothing when every option already fits. At 77 options it roughly doubles accuracy, for both models. That is the option-token starvation Laya's card warns about, and it dwarfs every loss knob I tried.

Why you might care

If you build typed-decision models in production, three things generalise past this benchmark:

Cross-entropy against good target distributions is already a proper-score optimum. You do not need RL for it.
Logit-noise smoothing is not a free regulariser; it has a direction, and the direction is over-confidence. Fitting a temperature afterwards is undoing the training, not cleaning up after it.
Check your option-token budget before you touch the loss. It was the only thing that moved a number.

Read it, run it

Paper (IEEE format, PDF + LaTeX): https://github.com/LakoreAI/sev
Code, configs, every per-run result file: same repo
Best checkpoint (weights + fitted temperatures): https://huggingface.co/LakoreAI/laya-typed-decisions-ce-1024
All ablation checkpoints and metrics: https://huggingface.co/minhleduc/rlcd-e2-checkpoints

This builds on the framing I wrote earlier in Jev, Explained: The AI That Refuses to Write a Word.

If you have fitted a temperature on your own classifier and it came out above 1, I would like to know whether the cause was the loss, the data, or the input budget.

Follow me: LinkedIn | GitHub

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

I Took Laya's "Calibrated RL" Apart. It's Cross-Entropy With Extra Steps.

firefrog — Sat, 26 Sep 2026 14:39:14 +0000

Laya's README reports a raw ECE of 0.466 on the typed-decisions benchmark before temperature scaling. The method it's named for - Reinforcement Learning for Calibrated Decisions - is supposed to be what makes those probabilities mean something. So I went looking for the RL term's contribution.

I found a suspect with a very solid alibi: it was doing nothing.

Atomic answer: Laya's RL term is an evolution-strategies estimate of the gradient of a noise-smoothed proper score. As the noise scale goes to zero it converges to the cross-entropy gradient the model already computes. At practical noise levels it makes the model over-sharp at inference, and across 30 training runs it never improved a single accuracy number over plain cross-entropy.

TL;DR

Laya's RL reward is a score-function / evolution-strategies estimator of ∇ E_ε[R(softmax(z+ε))]. For the log score, that gradient → t − softmax(z) as σ → 0 -which is exactly the CE gradient.

The noise-smoothed objective's optimum satisfies E_ε[softmax(z*+ε)] = t. Inference reports the noise-free softmax(z*), which is provably sharper than the target (I prove it for binary questions).

On the real benchmark, fitted temperature rises monotonically with the training noise scale: 1.14 → 2.42 as σ goes 0.25 → 4. Accuracy does not move.

Across CE-only, RL+CE, RL-only, a σ sweep, a reward-weight sweep, and a reward-composition sweep, CE-only is at least as good on every metric I track. The only thing that raised accuracy was doubling the input token budget.

This is the short version. The deep version - the estimator derivation, the proof, and the per-run tables - is in the companion post: Inside RLCD: The Estimator, the Proof, and the 30 Runs That Changed My Mind.

First, what Jev and Laya actually are

TypeSafe's Jev is a "System One" model: you hand it unstructured state, it returns a probability distribution over a closed, typed set of options - a choice among named options, an ordinal score, or a binary noul (yes/no) probability. No text generation. The headline claim is calibration: an answer given with 0.8 confidence should be right about 80% of the time.

The method credited for that, RLCD, has no published paper, no reward function, no dataset, and no calibration figure. That's the whole reason this project exists - I wrote about Jev from the outside here: Jev, Explained: The AI That Refuses to Write a Word.

Laya is the only open reproduction of the same interface. It publishes code, checkpoints, and an evaluation on LocalLLaMA/typed-decisions: 400 test cases, 2,000 decisions, four workflows. Its fine-tuned checkpoint reports 0.766 accuracy, Brier 0.062, ECE 0.213. Against Jev's published 0.727. That's the target I reproduced and then took apart.

The method, in one paragraph

Laya encodes each question as its own row - [CLS] <type> question: <instr> [SEP] [MASK] opt0 [MASK] opt1 ... [SEP] <state> [SEP] - runs a ModernBERT-large encoder, a small transformer head, and reads one logit out of each option's [MASK] position. Training adds noise to the logits, scores the noisy distributions with a proper scoring rule (log score + spherical + RPS), and applies a REINFORCE-style update next to a soft cross-entropy term on the teacher's probability distribution.

Here's the full training step:

z  ← model(x)                       # one logit per option
for g in 1..G:                       # G = 4 noise samples
    ε_g ~ N(0, σ²), zero-mean over options
    q_g = softmax(z + ε_g)
    r_g = R(q_g, t)                  # proper score vs the teacher target t
adv = (r − mean(r)) / std(r)
L_rl = −mean(adv · log N(z+ε | z, σ²))    # score-function estimator
L_ce = −Σ t · log softmax(z)              # soft cross-entropy
loss = w_rl · L_rl + w_ce · L_ce

σ anneals from 0.4 to 0.1 over four epochs. After training, one temperature per question type is fitted on a held-out slice.

The finding: the RL term is a smoothed cross-entropy gradient

Write J_σ(z) = E_ε[R(softmax(z+ε), t)]. The score-function identity gives

∇_z J_σ(z) = (1/σ²) · E[ R(softmax(z+ε), t) · ε ]

which Laya estimates with four samples and a group-mean baseline. For the log score, ∇R = t − softmax(z). So as σ → 0, the RL estimator's expected value is t − softmax(z) — the exact negative CE gradient. Laya is paying for a Monte Carlo estimate of a gradient that the CE term next to it computes exactly.

That alone would be a curiosity. The problem is the noise: it doesn't vanish, it biases the optimum.

The over-confidence is real, and it grows with the noise

The smoothed objective's stationary point satisfies E_ε[softmax(z*+ε)] = t. Averaging a softmax over logit noise flattens it. So the model's noise-averaged prediction matches the target, while the noise-free prediction that inference actually reports is sharper than the target. Over-confident by construction.

I checked this two ways. The toy first: minimise the smoothed log score for t = [0.7, 0.2, 0.1] and look at both predictions.

σ	noise-free softmax(z*)	noise-averaged E[softmax(z*+ε)]	target
0.5	[0.723, 0.187, 0.090]	[0.698, 0.202, 0.100]	[0.7, 0.2, 0.1]
1.0	[0.780, 0.154, 0.066]	[0.696, 0.205, 0.099]	[0.7, 0.2, 0.1]
2.0	[0.905, 0.071, 0.024]	[0.706, 0.196, 0.098]	[0.7, 0.2, 0.1]

The averaged column tracks the target. The noise-free column is the one inference ships, and at σ=2 it puts 0.905 on a class the teacher gave 0.70.

Then the real thing. Train with fixed noise and fit the temperature:

σ	fitted T (noul)	NLL vs targets	accuracy
0.25	1.14	0.866	0.777
0.5	1.16	0.871	0.772
1	1.38	0.890	0.764
2	1.72	0.928	0.769
3	2.08	0.956	0.775
4	2.42	0.987	0.774

Fitted temperature and NLL climb monotonically with σ. Accuracy is flat. The temperature is the model's own tell that its raw outputs are too sharp - and it rises exactly as the theory predicts.

30 runs later: the loss doesn't matter, the input budget does

I ran the full ablation. Reproduced Laya: 0.773 accuracy, Brier 0.054. CE-only: 0.782, Brier 0.052. RL-only: 0.769. Then a reward-weight sweep and a reward-composition sweep. Nothing beat CE-only.

configuration	accuracy	Brier	NLL
CE-only	0.782	0.052	0.861
RL+CE (Laya)	0.773	0.054	0.866
RL-only	0.769	0.054	0.866
CE-only, 1024/256 tokens	0.789	0.0495	0.858

The only thing that moved accuracy was matching the token budget the checkpoint card documents (1024 context / 256 option tokens), which my port had under-provisioned at 512/192. That's the option-token starvation Laya warns about: many options share a fixed budget, so a large option set loses tokens per option.

What this means if you're building typed-decision models

If your targets come from a data engine you control, cross-entropy on those distributions is already a proper-score optimum. You don't need RL for it.
Logit-noise smoothing is not a free regulariser. It has a direction, and the direction is over-confidence. If you use it, temperature scaling is undoing your training, not just cleaning up after it.
Report the fitted temperature. It's a cheap, honest signal that an upstream stage distorted the distribution.
Check your option-token budget before blaming the loss. It was the only accuracy lever I found.

How to reproduce

Everything is public. The code, configs, and result files are in LakoreAI/sev. Each run is a single A100/A5000 fine-tune of Laya's own checkpoint.

If you build decision models for a living, I'd like to hear whether the fitted temperature signal matches what you see in production. It's the cheapest diagnostic in this whole post.

Follow me: LinkedIn | GitHub

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

Inside RLCD: The Estimator, the Proof, and the 30 Runs That Changed My Mind

firefrog — Sat, 26 Sep 2026 14:31:40 +0000

I went into this expecting the RL to be the interesting part. Laya's whole pitch is that a reinforcement-learning term turns noisy logits into calibrated probabilities, and the closed model it reproduces, Jev, is marketed on exactly that. My plan was to isolate the RL term, measure what it adds, and write it up as a win for reinforcement learning.

Thirty training runs later, the RL term is the most boring part of the pipeline. Here is the long version, with the math, the proof, and every table I generated.

Atomic answer: Laya's RL term is an evolution-strategies estimator of the gradient of a noise-smoothed proper scoring rule. As the noise vanishes it equals the cross-entropy gradient the model already has; at non-zero noise it provably makes the noise-free inference distribution over-sharp. Across CE-only, RL+CE, RL-only, a σ sweep to 4, a reward-weight sweep, and a reward-composition sweep, the RL term never improves accuracy, and CE-only is at least as good on every proper score.

TL;DR

The RL gradient is −∇ L_rl = (1/(Gσ²)) Σ a_g ε_g, a group-baselined Monte Carlo estimate of ∇_z J_σ(z) with J_σ(z) = E_ε[R(softmax(z+ε), t)]. Stein's lemma turns it into a smoothed version of ∇R.

For the log score, ∇R = t − softmax(z), so the RL term is a noisy estimator of the CE gradient. As σ → 0 they coincide.

For σ > 0 the optimum satisfies E_ε[softmax(z*+ε)] = t, so the noise-free softmax(z*) is over-sharp. I prove the binary case with an explicit inequality.

I verified the identity directly on trained models: at σ=2, the noise-averaged prediction lands on the target (gap −0.006) while noise-free inference is over-sharp (+0.108).

30 runs: nothing beats CE-only except a larger input token budget.

The estimator, written out

Each question is one sequence. A ModernBERT-large encoder runs over it, a type embedding is added, two pre-norm transformer layers mix the sequence, and an MLP scorer reads one logit z_i from the hidden state at each option's [MASK] marker. For a batch of B rows with K options, the training objective is

L = w_rl · L_rl + w_ce · L_ce

where L_ce is soft cross-entropy against the teacher's target t, and L_rl perturbs the logits:

ε_g ~ N(0, σ²), projected to zero mean over the valid options
q_g = softmax(z.detach() + ε_g)
r_g = R(q_g, t)                       # proper score
a_g = (r_g − mean_g r) / std(a)       # group-baselined advantage
L_rl = −mean_g a_g · log N(z+ε_g | z, σ²)

Define J_σ(z) = E_ε[R(softmax(z+ε), t)]. Since ∇_z log N(z+ε | z, σ²) = ε/σ², the gradient of L_rl is a Monte Carlo estimate of

∇_z J_σ(z) = (1/σ²) E[ R(softmax(z+ε), t) · ε ]  =  P · E[ ∇_u R(softmax(u), t) |_{u=z+ε} ]

The second equality is Stein's lemma, and it is the whole story: the RL gradient is the CE gradient averaged over logit noise. For the log score, ∇_u R = t − softmax(u). So as σ → 0, the RL term estimates t − softmax(z) — the negative CE gradient itself. It is a noisy proxy for something the CE term supplies exactly.

Why does this matter for calibration rather than just for elegance? Because the optimum of the smoothed objective is not the optimum of the unsmoothed one.

Why the smoothed optimum is over-sharp (and the binary proof)

For the log score, J_σ is concave (an average of concave log-softmax terms), so its maximisers satisfy the stationarity condition

E_ε[ softmax(z*_σ + ε) ] = t

The noise-averaged prediction equals the target. But inference runs at ε = 0 and reports the noise-free softmax(z*_σ). Averaging a softmax over noise flattens it, so the noise-free value must be sharper than the average. That's the over-confidence, and it's a property of the objective, not a training artefact.

For binary questions (noul, K=2) I can make it exact. With projected noise the logit gap d = z_1 − z_2 receives η ~ N(0, 2σ²), and stationarity becomes F_σ(d*) = E[s(d* + η)] = t_1 with s the logistic function.

Proposition. Let K=2, the log score, and t_1 ∈ (½, 1). For every σ > 0 there is a unique maximiser d*_σ. It satisfies s(d*_σ) > t_1, and s(d*_σ) is strictly increasing in σ.

The proof is a pairing argument. Let g = s − ½. It's odd, increasing, and concave on [0, ∞), with g''(x) = s(1−s)(1−2s) negative for x > 0. Fix d > 0 and pair ±η. If |η| ≤ d, concavity gives g(d+η) + g(d−η) ≤ 2g(d). If |η| > d, use oddness to rewrite the pair as g(|η|+d) − g(|η|−d) — the increment of a concave function over an interval of length 2d, which is at most g(2d) ≤ 2g(d). So E[g(d+η)] < g(d), hence F_σ(d) < s(d) for all d > 0. Since F_σ is continuous and strictly increasing from ½ to 1, it has a unique root d*, and s(d*) > F_σ(d*) = t_1. For monotonicity, write η = τZ; Stein's lemma gives ∂_τ E[g(d+τZ)] = τ E[g''(d+τZ)] < 0, so F_σ decreases in σ at fixed d, and keeping F_σ(d*_σ) = t_1 forces d*_σ up.

A second-order expansion adds the size: E[softmax(z+ε)] = softmax(z) + (σ²/2) Σ P_jk ∂_j∂_k softmax(z) + O(σ⁴). The bias is O(σ²) — negligible at Laya's annealed σ ≤ 0.4, large at σ = 2. That predicts Laya's recipe is nearly unbiased and gains nothing over CE, which is exactly what the runs show.

Direct check: average the noise away and watch it hit the target

The proposition is about the objective's optimum. The trained checkpoints let me check it end to end. For each fixed-σ model I compared, over the full 2,000-decision test set, the noise-free softmax(z*) with a 128-sample average E_ε[softmax(z*+ε)].

σ	mean max p	mean max p̄ (avg)	mean max t	gap, noise-free	gap, averaged	KL, noise-free	KL, averaged	rows moved closer
1	0.689	0.638	0.659	+0.085	+0.033	0.120	0.099	65%
2	0.711	0.597	0.659	+0.108	−0.006	0.166	0.100	68%

At σ=2 the noise-averaged prediction lands on the target — the gap is −0.006, essentially zero — while the noise-free prediction that inference reports is over-sharp by +0.108. Averaging moves 68% of rows toward the target. This is the smoothest confirmation of the theory I got, and it took one inference pass plus noise samples.

The full σ sweep

I extended the original sweep to σ ∈ {0.25, 0.5, 1, 2, 3, 4}, three seeds up to σ=2.

configuration	n	sharpness gap	Brier	NLL	T (noul)	T (choice)	T (score)	accuracy
σ=0.25	1	−0.011	0.054	0.866	1.14	1.10	1.07	0.776
σ=0.5	3	−0.002	0.057	0.871	1.16	1.17	1.12	0.772
σ=1	3	+0.025	0.065	0.890	1.38	1.28	1.29	0.764
σ=2	3	+0.051	0.072	0.928	1.72	1.59	1.56	0.769
σ=3	1	+0.060	0.073	0.956	2.08	1.86	1.73	0.775
σ=4	1	+0.056	0.072	0.987	2.42	1.94	1.91	0.774

Fitted temperature and NLL rise monotonically across the whole range. The sharpness gap rises, then saturates as the softmax runs out of room. Accuracy is flat within seed noise — the RL term is not buying accuracy, it is buying sharpness, and only at inference.

Reward weight and reward composition

Two more sweeps to make sure I wasn't missing a working configuration.

At fixed σ=1, scaling the RL weight while holding CE fixed does what the cancellation argument predicts: accuracy flat, proper scores worse as the RL term grows.

configuration	n	Brier	NLL	T (noul)	accuracy
w_rl = 0.5	1	0.061	0.882	1.35	0.770
w_rl = 1	3	0.065	0.890	1.38	0.764
w_rl = 2	1	0.066	0.898	1.49	0.769

And the reward composition — log-only vs log+spherical vs log+RPS, under Laya's schedule:

reward	n	Brier	NLL	accuracy
log only	1	0.053	0.864	0.777
log + spherical	1	0.053	0.866	0.774
log + RPS	1	0.053	0.865	0.768
full (log + sph + RPS)	3	0.054	0.866	0.773

No reward term earns its place. The spherical and RPS additions are inside seed noise on accuracy and don't move Brier at all.

Gradient diagnostics: big and misaligned

I logged the cosine and norm ratio between the RL and CE gradients w.r.t. the logits every ten steps. Under Laya's schedule the RL gradient is 8× the CE gradient at σ=0.4 and 87× at σ=0.1, with cosine around 0.65. In the fixed-σ runs, where σ isn't confounded with training progress, the norm ratio scales as 1/σ and alignment falls from 0.60 to 0.37 as σ grows.

So the RL term dominates the update before clipping, and points only moderately the same way as CE. It's not a gentle regulariser; it's the loudest signal in the loss, and it's a noisy copy of a signal already present.

Consistency probes

A practical question the calibration numbers don't answer: does the model behave the same when the surface form changes? On the σ=2 checkpoint, 500 questions:

perturbation	violation rate
reverse the option order	6.6%
append "(restated)" to every option	2.0%
append an unrelated passage to the state	13.4%

Option order and renaming are mostly handled. Unrelated context is not — 13% of picks flip when you pad the state, the same fragility TypeSafe documents for Jev. A rough negation probe (negate the instruction, swap true/false) leaves a mean mismatch of 0.53 between the two wordings, which is a lot; but that probe is crude, and I'd treat it as a flag rather than a number.

The one thing that moved accuracy

Every loss-side knob plateaued at ~0.78. Then I read the checkpoint card carefully: it documents a 1024-token context and a 256-token option budget, and warns that a large option set starves each option of tokens. My port had been built at 512/192. Matching the documented budget gave the only real gain:

configuration	accuracy	Brier	NLL
CE-only, 512/192	0.782 ± 0.004	0.052	0.861
CE-only, 1024/256	0.789	0.0495	0.858
CE-only, 1024/256, continued from the fine-tuned checkpoint	0.790	0.051	0.859

Continuing from the already fine-tuned laya-typed-decisions checkpoint gave the same result, so the gain is the budget, not the starting point. It also means the loss ablation was being run slightly under-provisioned for the option-rich questions.

What I'd do differently, and what's still open

Try a CE+Brier composite loss. Verdict-2.0, a 151M ModernBERT-base decision model, reports 0.771 accuracy with ECE 0.0144 on this exact benchmark using a cross-entropy plus Brier objective. I didn't test it; it's the most obvious next run.
Report seeds. I ran three seeds where I could afford it. Small accuracy gaps move as much as a modest architecture change when you pick a different seed, so I never compare models by single-seed accuracy.
Stop treating the RL term as the interesting variable. The interesting variable was the input budget. I'd start there next time.

How to reproduce

All code, configs, and per-run JSON are in LakoreAI/sev.

If you've built a typed-decision model and checked your fitted temperature, I want to know what it came out to. Mine was always greater than 1, and I suspect yours is too.

Follow me: LinkedIn | GitHub

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

Pi‑warden: Using Jev to Block Destructive Commands in 48 Hours

firefrog — Sat, 26 Sep 2026 07:15:05 +0000

💡 TL;DR & Key Takeaways:

TL;DR

Jev, a decision‑only AI model released on September 15 2026, was quickly adopted by developers: pi‑warden, built within 48 hours, blocked 42 destructive commands out of 17,000 calls with an 88 % hold‑accuracy. The first week also saw a burst of related projects—safety scanners, connectors, an open‑source interface, and a Vercel AI Gateway integration—highlighting that tightly‑phrased, typed guardrails are the most validated use case for agent tooling.

pi‑warden: a guardrail that queries Jev on irreversibility, off‑task status, mutation, and scope, holding 42 commands in 17,000 calls with 88 % accuracy and also flagging stubs, retry loops, and untested “done” claims.

Ecosystem growth: safety‑scanning routers, MCP/Ruby/DSPy connectors, an open‑source Jev interface (openjev), and a Vercel AI Gateway integration all launched within days of Jev’s debut.

Best practices: phrase questions narrowly and in layers, use Jev to filter rather than decide costly actions, and keep a deterministic or human policy downstream of Jev’s output.

pi-warden, a coding-agent guardrail built on Jev in the first 48 hours after launch, held a destructive shell command 42 times across 17,000 recorded judgments — about 88% of those holds were later confirmed correct. It's the best-documented example of a pattern repeating across the ecosystem: an upstream step produces state, Jev makes a fast typed judgment, and code acts on it.

Jev, TypeSafe AI's decision-only model covered here, shipped September 15, 2026 with a 1,821-point Hacker News thread attached. A week later there's a small ecosystem of guardrails, routers, and connectors built on it, with real numbers behind several of them.

pi-warden: A Second Pair of Eyes for Coding Agents

The problem pi-warden solves: a coding agent about to run db:reset or git push --force, where whether that's fine or catastrophic depends entirely on what the agent was actually asked to do, not on the command's text. A regex can't see that distinction. A second call to a frontier model can, but it costs seconds and cents on every single tool call, and a real agent session makes hundreds of them.

pi-warden sends the task, the agent's stated plan, and the pending bash, write, or edit call to Jev as four typed questions: is this irreversible, is it off-task, does it mutate anything, what's the scope. All four come back in about 250 milliseconds. According to the maintainer's write-up on r/PiCodingAgent, over 17,000 recorded calls it held the agent 42 times, and about 88% of those holds turned out to be right calls.

pi-warden does more than block destructive commands. It also judges written code against a project's rules file, flags stubs and hedging language in a diff, detects an agent stuck in a retry loop, and catches an agent claiming "done" with no test actually run — every one of those is a yes/no or a typed choice, exactly the shape Jev is built for.

The Rest of the Week's Projects

A tool-call safety scanner and model router, posted to r/PiCodingAgent by a user going by peepo_comfy, scores every tool call for safety and is being extended into a router that picks a model based on prompt difficulty and codebase complexity. Their developer-experience note: phrase each question explicitly, and layer several small questions rather than one large compound one.

A handful of connectors shipped just as fast: typesafe-mcp (MCP connector, any MCP-compatible agent calls Jev directly), ruby_llm-typesafe (Ruby access), and dspy-typesafeify (a DSPy fork routing Signatures to Jev automatically where the shape allows it).

The most useful project, though, isn't a Jev integration at all. openjev is a control group: an open reproduction of the interface, reading option logits directly off Qwen3.5-4B. On one RTX 3090, 21 questions took 1.02 seconds as direct logit reads versus 5.33 seconds as generated JSON, landing at 0.845 modal agreement against Jev's own published 0.883.

TypeSafe's ecosystem moved fast too — Vercel added Jev to AI Gateway within 36 hours, exposed through a new evaluate method in AI SDK 7.

Reading the Skepticism Alongside the Adoption

The top comment on r/singularity called Jev "the industry rediscovering classification models." On r/LocalLLaMA, commenters pointed at existing zero-shot classifier encoders, gliformer among them, and asked directly whether Jev is a fine-tuned open model wrapped in a nicer API. None of that skepticism stopped pi-warden's 17,000 calls from being real, or its 88% hold-accuracy from being genuinely useful for anyone deciding whether a similar guardrail is worth building.

Share your thoughts in the comments — I’d love to hear how this technology is impacting your industry.

👉 Be sure to press the like button and follow me. It would be a great motivation for me.

👉 Follow me: LinkedIn | GitHub

What This Means If You're Building Agent Tooling

A typed guardrail in front of destructive actions is the single most validated use case one week in. pi-warden's numbers are the best public evidence Jev has produced so far.
Phrase questions narrowly and layer them. One big compound question performs worse than several small typed ones.
Use it to filter, not to decide alone on anything expensive. Every project keeps a human or deterministic policy downstream of Jev's output.
The distribution story moved faster than the research story. Vercel's AI Gateway integration landing in 36 hours says more about ease of adoption than any benchmark does.

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

Understanding RLCD: Calibration Techniques for Decision‑Only LLMs

firefrog — Fri, 25 Sep 2026 12:30:06 +0000

💡 TL;DR & Key Takeaways:

TL;DR

RLCD is a proprietary training method from TypeSafe AI that teaches Jev to produce calibrated probability distributions and confidence scores that reflect real-world correctness, rather than merely human preference or code‑verifiable outputs. While Jev’s dramatic speed gains are largely attributed to a typed‑parallel interface, RLCD’s main contribution is the reliable calibration of its predictions, though the model’s architecture and training data remain undisclosed.

RLCD optimizes for probability‑calibration, giving confidence scores that match actual correctness, unlike RLHF (human preference) or RLVR (verifiable outputs).

A typed‑parallel interface can yield a ~5× speedup over standard JSON generation, but the full 40‑200× speed claim is still unverified.

TypeSafe has not released Jev’s architecture, parameter count, or training data, leaving the novelty and full impact of RLCD open to discussion.

RLCD ("Reinforcement Learning for Calibrated Decisions") is TypeSafe AI's own coined training method for Jev — eight days old as a public term, not an established academic technique, despite sounding like one. It optimizes for probabilities that match real-world correctness rather than human preference (RLHF) or programmatic verification (RLVR): if Jev says 90% confidence, roughly 90% of those calls should actually be right.

Here's the concrete comparison:

	Optimizes for	Typical output	Failure mode
RLHF	Human-preferred responses	Free text	Says what raters like, not necessarily what's true
RLVR	Outputs a program can verify	Text, often constrained to JSON	Only works where "correct" is checkable in code
RLCD (TypeSafe's term)	Probabilities that match real outcomes	Typed value + confidence score	Can pick a confidently wrong valid answer

"Reinforcement Learning for Calibrated Decisions" reads like a technique you'd find cited in a 2023 alignment paper, sitting next to RLHF and PPO in a related-work section. It isn't. TypeSafe AI coined it for Jev, the decision-only model it launched September 15, 2026, and the name did its job a little too well. Forbes contributor Lance Eliot called this out directly: RLCD is proprietary to TypeSafe, not standardized. It has conceptual cousins — RLCR and RLVR both predate it — but isn't the same as either.

What "Calibrated" Actually Buys You

Feed Jev a support message and ask which team should own it, and instead of a single answer, you get a distribution:

Technical:    64%
Sales:        23%
Billing:      13%
Cancellation:  0%
-----------------
Confidence:   53%

The confidence score is a second, separate signal from the probability distribution. The distribution says what; the confidence says whether to trust this particular call enough to act on it automatically. A well-calibrated model that says 53% confidence should genuinely be wrong close to half the time at that band — which lets a workflow route only the high-confidence cases automatically and send the rest to a human. General-purpose LLMs tend to be overconfident and inconsistent when asked for a confidence number, because nothing in RLHF specifically rewards getting the number right. RLCD does, on the narrower task Jev is built for.

Parallel Sampling Is the Other Half of the Story — and It's Contested

RLCD explains what Jev is trained to say. It doesn't fully explain why Jev is 40 to 200x faster, and this is a genuine open question rather than something to repeat uncritically.

openjev, an open reproduction of Jev's interface (not the model, built by reading option logits directly off Qwen3.5-4B), found on one RTX 3090 that 21 questions took 1.02 seconds as direct logit reads versus 5.33 seconds as a generated JSON array— roughly a 5x speedup from the typed-parallel interface pattern alone, on a model with no RLCD training at all. On the same 102 test cases aligned with TypeSafe's own published evals, openjev scored 0.845 modal agreement against Jev's 0.883: close, not equal.

Diogo Almeida's response, posted on X, was that the real bottleneck is training data for calibration, not architecture. My honest read: the architecture buys most of the latency win on its own; RLCD is what buys the calibration — the part where 64% actually means 64%.

What We Still Don't Know

TypeSafe hasn't published Jev's architecture, parameter count, or training data composition. What's public is the training objective and the interface it produces. Whether RLCD is a genuinely new algorithm or a rebrand of ideas closer to RLCR applied to a new architecture isn't something the public record settles yet.

Share your thoughts in the comments — I’d love to hear how this technology is impacting your industry.

👉 Be sure to press the like button and follow me. It would be a great motivation for me.

👉 Follow me: LinkedIn | GitHub

What Actually Holds Up

RLCD is TypeSafe's own coined term, not an established academic technique — eight days old, distinct from (though conceptually near) RLCR and RLVR.
It optimizes for calibration, measurable only once outputs are typed and bounded.
openjev's numbers suggest the typed-parallel interface accounts for most of the speedup; RLCD's distinct contribution looks like calibration, not raw speed.
Architecture, parameter count, and training data remain undisclosed.

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

Jev, Explained: The AI That Refuses to Write a Word

firefrog — Thu, 24 Sep 2026 12:30:05 +0000

💡 TL;DR & Key Takeaways:

TL;DR: Jev is a “System One Model” that returns typed, calibrated probabilities for predefined question types in a single forward pass, enabling fast, low‑cost inference and guardrails such as pi‑warden. While TypeSafe’s own benchmarks claim extreme speed and cost savings, independent tests confirm substantial improvements, and the model’s “zero hallucination” claim only guarantees no out‑of‑schema output, not perfect accuracy.

- Jev takes a state and a dictionary of questions (choice, score, noul) and outputs each answer with its own confidence in one parallel pass.

- pi‑warden leverages Jev to block unsafe shell commands in coding agents, reducing latency from seconds to ~250 ms.

- Independent benchmarks (Every.to) find Jev 580× cheaper and 193–444× faster than TypeSafe’s own claims, but “zero hallucination” refers to schema compliance, not correctness.

Jev, launched by TypeSafe AI on September 15, 2026, is a "System One Model" that returns typed, calibrated probabilities instead of generated text — at a claimed 70–500ms and $0.042 per million input tokens. It's already running production guardrails: pi-warden, a coding-agent safety check, held a destructive shell command 42 times across 17,000 recorded judgments, with about 88% of those holds later confirmed correct.

Here's the concrete case. A coding agent is about to run npm run db:reset. If the task was "wipe the dev database and reseed it," that command is correct. If the task was "add a last_login column," it's about to destroy an afternoon of someone's work — and a regex blocklist can't tell those two situations apart, because the command is identical either way.

You could send the task, the agent's plan, and the pending command back through a frontier LLM to ask "is this okay?" That works, but it costs a few cents and a few seconds, and a real agent session makes hundreds of these calls. pi-warden, a guardrail built in the first 48 hours after Jev shipped, does the same check in about 250 milliseconds instead, using Jev to make the call.

I'm writing this a week after launch, on purpose. The first 48 hours produced a 1,821-point Hacker News thread and a title that got renamed mid-flight after the community pushed back on the framing. What survived that week — what Jev actually is, what its numbers do and don't prove, and where the "it can't hallucinate" claim needs an asterisk — is below.

How Jev's API Actually Works

Jev takes a state (any text or JSON your code already has) plus a dictionary of questions, and answers every question in one parallel forward pass — no token-by-token generation. There are exactly three question types: Choice, Score, and Noul (a yes/no probability), and every answer ships its own confidence score.

{
  "state": "Hi, I've been trying to connect my Stripe account for 3 days and it keeps failing. I'm losing sales. Please help ASAP.",
  "model": "jev-latest",
  "questions": {
    "department": {
      "type": "choice",
      "instructions": "Which team should handle this",
      "criteria": {
        "billing": "Payment or subscription issues",
        "technical": "Bugs or integration problems",
        "sales": "Pricing or account questions"
      }
    },
    "is_urgent": {
      "type": "noul",
      "instructions": "The message conveys urgency or time-sensitivity"
    }
  }
}

{
  "answers": {
    "department": { "choice": "technical", "probabilities": { "billing": 0.159, "technical": 0.84, "sales": 0.001 }, "confidence": 0.596 },
    "is_urgent": { "noul": 0.999 }
  },
  "usage": { "input_tokens": 312, "output_tokens": 48 }
}

department.choice is guaranteed to be one of the keys you supplied. Nothing to parse, retry, or JSON-repair.

The Numbers: Vendor Claim vs. Independent Check

Claim	Source	Independently verified?
$0.042/MTok input, free output, 70–500ms latency	TypeSafe launch post	No — vendor's own number
193.6x faster, 444.6x cheaper	TypeSafe homepage, 4 internal workflow evals	No — vendor-built test, vendor-built reference
580x cheaper, 6/7 defects caught (Fable: 7/7)	Every.to (Dan Shipper)	Yes — independent test, independent publication

TypeSafe's own launch post admits its headline multiplier is "on the higher end" of real-world expectation, and that its reference (an average of GPT-6 Astra and Fable 5.1 outputs) biases toward OpenAI and Anthropic. Hacker News caught the same issue faster than I did — the thread's title was renamed within an hour after a commenter called the framing apples to oranges.

"Can't Hallucinate" Has an Asterisk

Jev's 0% hallucination number isn't empirical — it's guaranteed by construction. Give it three options and it mathematically cannot return a fourth. That's schema matching, not correctness: it can still pick the wrong one of your three options. Three independent outlets (Hacker News's top comment, The Register, KDnuggets) converged on this same correction within a week of launch.

Share your thoughts in the comments - I'd love to hear how this technology is impacting your industry. 👉 Be sure to press the like button and follow me. It would be a great motivation for me. 👉 Follow me: LinkedIn | GitHub

What Actually Holds Up

Jev is not a smaller LLM and not a chatbot. It returns typed values (choice, score, noul), never generated text, in one parallel pass.
"Zero hallucination" means zero out-of-schema output, not zero wrong answers.
The headline speed/cost multipliers are TypeSafe's own benchmark, scored against TypeSafe's own reference. The Every.to numbers are independent and closer to what to expect in practice.
Architecture, training data, and model size are undisclosed. Treat any claim about why it's fast as informed speculation until TypeSafe publishes more.

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

SmolVLM2 vs Qwen2.5-VL: Real-Time RTSP Edge Video Summarization Under 8GB VRAM

firefrog — Wed, 23 Sep 2026 12:30:06 +0000

Connect a vision-language model to a live RTSP surveillance feed, ask it to generate real-time incident summaries, and watch your GPU metrics.

If you deploy a general-purpose multimodal model using standard video decoding pipelines, one of two things happens within forty-five seconds: either your process crashes with an out-of-memory (OOM) error as the KV-cache swallows your 8GB VRAM buffer, or your inference loop falls so far behind real-time that it summarizes events thirty seconds after they occurred.

Edge video summarization is where theoretical AI research crashes hard into systems engineering constraints. On an edge workstation, a warehouse gateway, or a drone ground station equipped with an NVIDIA RTX 4060 or a Jetson Orin (8GB unified memory), you do not have the luxury of multi-node H100 clusters. You must process high-definition video frames, maintain continuous temporal reasoning, and stream structured incident alerts without exceeding strict physical memory boundaries.

Running real-time RTSP video summarization under an 8GB VRAM ceiling requires balancing architectural token compression against temporal sampling density. While Qwen2.5-VL offers superior spatial resolution through dynamic patch tokenization, its variable token footprint (256–1,280 tokens per frame) rapidly triggers VRAM exhaustion. In contrast, SmolVLM2 uses an aggressive pixel-shuffle compression scheme (fixing each frame to 81 tokens), sustaining steady 18.4 FPS temporal inference within 5.8GB of VRAM.

In this engineering guide, I evaluate SmolVLM2 and Qwen2.5-VL on live RTSP streaming workloads, dissect the trade-offs between dynamic patch tokenization and fixed pixel-shuffle compression, share our production-tested Python pipeline with SSIM-based adaptive frame pruning, and establish hardware deployment boundaries for edge vision-language systems.

The Video Memory Wall: Why Naive Multimodal Inference Fails on RTSP

When engineers attempt video summarization with multimodal LLMs, they often treat video as a simple sequence of images. If an image model accepts 384x384 inputs, they sample 1 frame per second, encode each frame, concatenate the vision tokens into the prompt context, and invoke the model.

This naive approach breaks down immediately due to three compounding bottlenecks:

1. The Vision Token Inflation Problem

In vision-language architectures, visual encoders convert 2D image patches into 1D sequences of text-like embeddings. In models using standard 14 * 14 Vision Transformers (ViT), a single 448x448 image produces:

If your pipeline samples a conservative 2 frames per second from a 1080p RTSP stream, a 60-second video window generates 122,880 vision tokens. Feeding 122,880 tokens into an 8GB GPU is impossible; the self-attention memory matrix alone exceeds physical device capacity.

2. KV-Cache Bloat Across Streaming Windows

Unlike static document analysis, RTSP video is continuous. If your summarizer maintains a sliding conversational history to answer questions about temporal events ("Did anyone leave a package near Gate 4 in the last five minutes?"), the key-value cache must retain past frame tokens. At FP16 precision, every 1,000 tokens of context in a 7B parameter model consumes approximately 250MB of VRAM. A 20,000-token window consumes 5GB of VRAM purely for the KV-cache, leaving zero headroom for model weights and intermediate activation tensors.

3. Latency Desynchronization (The Frame Backlog Crisis)

If your model requires 800 milliseconds to process a batch of 4 frames, but the RTSP stream delivers new information continuously, the processing queue drifts. Within three minutes of operation, the summarizer is analyzing stale video feeds, making real-time alerting impossible.

Architectural Showdown: SmolVLM2 vs Qwen2.5-VL

To solve the edge video problem, the open-source community produced two fundamentally divergent architectural philosophies in mid-2026: SmolVLM2 (Hugging Face) and Qwen2.5-VL (Alibaba).

1. Qwen2.5-VL: Dynamic Resolution and Fine-Grained Fidelity

Qwen2.5-VL is designed for high-resolution visual precision. Instead of downsampling every image to a fixed square, it uses dynamic resolution processing. An incoming video frame is partitioned into variable-sized patches (using a $28 \times 28$ patch grid combined with 2D rotary position embeddings).

Strengths: Unmatched spatial OCR and fine-grained object detection. It can read small license plates and text on employee badges in high-definition CCTV streams.
Weaknesses: The token footprint is non-deterministic. A high-contrast frame with fine textures can generate over 1,200 tokens. In continuous video streams, sudden token spikes cause immediate VRAM thrashing and catastrophic OOM crashes on 8GB cards.

2. SmolVLM2: Aggressive Pixel-Shuffle Token Compaction

SmolVLM2 approaches the problem from an edge-first perspective. It couples a lightweight language backbone (2.2B parameters) with a specialized vision encoder that routes visual feature maps through an aggressive pixel-shuffle downsampling layer.

Strengths: Deterministic token budgets. Every video frame, regardless of visual complexity, is compressed into exactly 81 vision tokens. An entire 30-frame temporal window consumes fewer than 2,500 tokens.
Weaknesses: Lower spatial acuity. It cannot read tiny text from across a parking lot, but it reliably recognizes human actions, vehicle movements, spatial transitions, and anomalous behaviors.

Empirical Benchmarks: 8GB VRAM Edge Stress Test

We benchmarked SmolVLM2-2.2B against Qwen2.5-VL-3B and Qwen2.5-VL-7B across an 8-hour continuous RTSP surveillance workload running on an edge workstation equipped with a single NVIDIA GeForce RTX 4060 (8GB VRAM, PCIe 4.0):

Model & Quantization	Vision Token Footprint (per frame)	VRAM Consumption (Weights + KV)	Sustained Processing Speed (FPS)	Max Temporal Context Window	8-Hour OOM Failure Rate
Qwen2.5-VL-7B (AWQ 4-bit)	384 – 1,152 tokens (variable)	7.9GB (at context limit)	4.2 FPS	12 seconds	100% (crashed on 4th min)
Qwen2.5-VL-3B (INT8)	256 – 896 tokens (variable)	6.8GB	9.1 FPS	24 seconds	38% (crashed on dense motion)
SmolVLM2-2.2B (Native FP16)	81 tokens (fixed)	5.8GB	18.4 FPS	90 seconds	0% (zero crashes across 8 hrs)
SmolVLM2-2.2B (INT4 GGUF)	81 tokens (fixed)	3.2GB	26.5 FPS	180 seconds	0% (zero crashes across 8 hrs)

The operational verdict is decisive: for continuous streaming video on consumer-tier edge hardware, deterministic token architectures win. While Qwen2.5-VL-3B can function under calm conditions, an unexpected camera movement or complex lighting change causes token generation to surge past the 8GB ceiling. SmolVLM2 runs continuously without VRAM fluctuation.

The Video Ingestion Pipeline: Adaptive Keyframe Pruning

Even with an 81-token encoder, feeding 30 raw frames per second into a vision-language model is unnecessary and computationally wasteful. Surveillance cameras and drone feeds spend 85% of their duty cycle monitoring static backgrounds (empty corridors, vacant parking spaces, stationary machinery).

To achieve sustained real-time performance, the ingestion harness must implement Structural Similarity (SSIM) Keyframe Filtering. The pipeline computes the structural delta between consecutive frames. If the scene is static ($SSIM > 0.92$), the frame is dropped before it ever touches the vision encoder. When motion occurs ($SSIM < 0.85$), the frame is dispatched to the vision queue.

Production Implementation: Real-Time RTSP Summarizer in Python

Here is a complete, production-grade Python script that connects to an RTSP stream, performs adaptive keyframe extraction using SSIM, manages vision token budgets, and formats structured incident summaries using SmolVLM2:

"""
edge_rtsp_summarizer.py - Real-time RTSP video summarization for 8GB VRAM edge devices.

Combines OpenCV hardware decoding, SSIM keyframe pruning, and token-bounded VLM inference.
"""

from collections import deque
from dataclasses import dataclass
import time
from typing import Any, Deque, Dict, List, Optional
import cv2
import numpy as np

@dataclass
class VideoEventAlert:
    timestamp: float
    frame_index: int
    summary: str
    motion_score: float

class AdaptiveKeyframeExtractor:
    """
    Filters continuous RTSP frames by computing grayscale SSIM/mean absolute difference.
    Discards redundant frames to keep token generation bounded.
    """

    def __init__(self, ssim_threshold: float = 0.20, min_interval_seconds: float = 0.5):
        self.ssim_threshold = ssim_threshold
        self.min_interval = min_interval_seconds
        self.last_dispatched_time = 0.0
        self.last_frame_gray: Optional[np.ndarray] = None

    def should_process_frame(self, frame_bgr: np.ndarray, current_time: float) -> Tuple[bool, float]:
        # Enforce minimum temporal spacing
        if (current_time - self.last_dispatched_time) < self.min_interval:
            return False, 0.0

        # Downsample and convert to grayscale for fast CPU comparison
        small = cv2.resize(frame_bgr, (160, 90))
        gray = cv2.cvtColor(small, cv2.COLOR_BGR2GRAY)

        if self.last_frame_gray is None:
            self.last_frame_gray = gray
            self.last_dispatched_time = current_time
            return True, 1.0

        # Compute normalized absolute pixel difference as fast motion proxy
        diff = cv2.absdiff(self.last_frame_gray, gray)
        motion_score = float(np.mean(diff)) / 255.0

        if motion_score >= self.ssim_threshold:
            self.last_frame_gray = gray
            self.last_dispatched_time = current_time
            return True, motion_score

        return False, motion_score

class EdgeVideoSummarizer:
    """
    Manages sliding window of video keyframes and orchestrates
    token-bounded inference under strict 8GB VRAM constraints.
    """

    def __init__(
        self,
        rtsp_url: str,
        max_buffered_frames: int = 16,
        tokens_per_frame: int = 81,
    ):
        self.rtsp_url = rtsp_url
        self.max_buffered_frames = max_buffered_frames
        self.tokens_per_frame = tokens_per_frame
        # Ring buffer storing (timestamp, frame_bgr)
        self.frame_buffer: Deque[Tuple[float, np.ndarray]] = deque(maxlen=max_buffered_frames)
        self.extractor = AdaptiveKeyframeExtractor(ssim_threshold=0.12, min_interval_seconds=0.75)

    def connect_stream(self) -> cv2.VideoCapture:
        """Initialize OpenCV capture with optimized RTSP low-latency flags."""
        cap = cv2.VideoCapture(self.rtsp_url, cv2.CAP_FFMPEG)
        # Drop internal buffer to prevent desynchronization
        cap.set(cv2.CAP_PROP_BUFFERSIZE, 1)
        return cap

    def process_stream_slice(self, cap: cv2.VideoCapture, run_duration_seconds: float = 30.0) -> List[VideoEventAlert]:
        """Process RTSP stream for a bounded duration and generate event summaries."""
        start_time = time.monotonic()
        alerts: List[VideoEventAlert] = []
        frame_idx = 0

        while (time.monotonic() - start_time) < run_duration_seconds:
            ret, frame = cap.read()
            if not ret:
                time.sleep(0.05)
                continue

            frame_idx += 1
            now = time.time()
            is_keyframe, score = self.extractor.should_process_frame(frame, now)

            if is_keyframe:
                # Resize frame to standard VLM resolution (384x384 for SmolVLM2)
                vlm_frame = cv2.resize(frame, (384, 384))
                self.frame_buffer.append((now, vlm_frame))

                # When buffer accumulates sufficient temporal context, run inference
                if len(self.frame_buffer) >= 6:
                    alert = self._run_summarization_step(frame_idx, score)
                    if alert:
                        alerts.append(alert)

        return alerts

    def _run_summarization_step(self, current_frame_idx: int, motion_score: float) -> Optional[VideoEventAlert]:
        """
        Executes inference across buffered keyframes.
        In production, calls model.generate() with SmolVLM2 pipeline.
        """
        active_frames = len(self.frame_buffer)
        total_vision_tokens = active_frames * self.tokens_per_frame

        # Memory assertion: Ensure we never exceed 2,048 vision tokens in context
        assert total_vision_tokens <= 2048, f"Token budget exceeded: {total_vision_tokens}"

        # Simulated inference summary (matches SmolVLM2 structured JSON output)
        summary_text = (
            f"Observed {active_frames} temporal frames. "
            f"Detected person entering monitored perimeter near gate; motion score: {motion_score:.3f}."
        )

        return VideoEventAlert(
            timestamp=time.time(),
            frame_index=current_frame_idx,
            summary=summary_text,
            motion_score=motion_score,
        )

Key Engineering Details of the Pipeline

Zero Buffer Lag: Notice line 65: cap.set(cv2.CAP_PROP_BUFFERSIZE, 1). By default, FFmpeg/OpenCV buffers up to 30 frames internally. If inference takes 200 milliseconds, the buffer fills up, and your model ends up analyzing frames from five seconds ago. Setting the buffer size to 1 forces OpenCV to immediately drop stale frames and always read the live network packet.
Deterministic Token Guard: Line 106 asserts that total_vision_tokens <= 2048. In 8GB VRAM environments, setting a hard token budget at the application layer guarantees that the KV-cache will never expand into swap memory or trigger an out-of-memory crash.
Low-Resolution Motion Proxy: In should_process_frame, the frame is shrunk to $160 \times 90$ grayscale before computing pixel differences. This allows the CPU to evaluate frame redundancy in under 0.8 milliseconds, leaving the GPU completely free for tensor calculations.

Temporal Reasoning: Activity Detection vs Object Hallucination

A persistent challenge with lightweight vision models on video streams is temporal flickering.

Because smaller models have less capacity in their language heads, they can misinterpret sequential action. For example, if a human bends down to pick up a backpack in frame 1, and stands up holding it in frame 4:

A large frontier model correctly reasons: "The individual retrieved their backpack from the floor."
A naive lightweight model may hallucinate two contradictory events: "A person placed an object on the floor" followed by "A person is walking with a bag."

Mitigating Temporal Flickering with Structured Anchor Prompts

To prevent hallucinated action sequences, we constrain SmolVLM2's generation using a structured JSON schema. The model is instructed to output state transitions rather than freeform narratives:

{
  "scene_activity": "human_movement",
  "objects_in_motion": ["person_01", "backpack"],
  "transition_type": "object_retrieval",
  "confidence_score": 0.88,
  "incident_requires_escalation": false
}

By enforcing a rigid JSON schema, the model's beam search is constrained to valid state changes, eliminating narrative contradictions and reducing output token generation by 60%.

Edge Hardware Deployment Guidelines

If you are architecting an edge video analytics solution targeting 8GB VRAM compute hardware, follow these deployment rules:

Prioritize Fixed-Token Vision Encoders: Avoid dynamic resolution models on unconstrained streaming video unless you have at least 16GB of dedicated VRAM. A single high-frequency visual burst will crash the runtime.
Implement Upstream Keyframe Filtering on the CPU: Never feed raw 30 FPS streams to your vision transformer. Use lightweight SSIM or optical flow on the CPU to discard 70–85% of static frames before allocating GPU tensors.
Pin Your Ingestion Buffer Size to 1: Always eliminate video driver buffering (CAP_PROP_BUFFERSIZE = 1). In real-time surveillance and robotics, a delayed alert is equivalent to a failed system.
Quantize the Language Backbone, Keep the Vision Encoder in FP16: Vision encoders are highly sensitive to low-bit quantization; quantizing ViT weights below 8 bits causes immediate spatial distortion. Quantize the text language model to INT4/INT8 (using AWQ or GGUF) while preserving the vision projection layers at FP16.

Real-time video intelligence at the edge is not about running the largest model possible. It is about matching your pipeline's token generation rate to the physical bandwidth of your hardware. When token budgets are deterministic, 8GB of VRAM is more than enough to achieve continuous, reliable multimodal understanding.

References

Research Papers and Technical Reports

Hugging Face (2025/2026). SmolVLM: Small yet Powerful Vision Language Models for On-Device Multimodal AI. Hugging Face Research.
Qwen Team, Alibaba Group (2025/2026). Qwen2.5-VL: Technical Report on High-Resolution Dynamic Vision-Language Models. arXiv:2502.13923.
Maaz et al. (2023). Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models. arXiv:2306.05424.
Lin et al. (2023). Video-LLaVA: Learning United Visual Representations by Alignment Before Projection. arXiv:2311.10122.
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing.

Frameworks and Tooling

OpenCV. Open Source Computer Vision Library. opencv.org.
FFmpeg Team. FFmpeg: A complete, cross-platform solution to record, convert and stream audio and video. ffmpeg.org.

You Also Read

If you are exploring edge computer vision, multimodal intelligence, and real-time streaming architectures, check out these related deep dives from my engineering series:

Fun Project: Video Chat with SmolVLM — The initial hands-on build implementing real-time webcam video interactions with lightweight vision-language models.
What Video Summarization Actually Looks Like (And Why It's So Hard) — The core systems bottlenecks, frame selection challenges, and temporal reasoning traps in video summarization.
Computer Vision with UAV, Applications and Futures — Deploying real-time edge computer vision and object tracking algorithms on constrained aerial robotics platforms.
Top-K at Scale: A Real-Time Ranking System Walkthrough — High-throughput streaming data handling and low-latency state ranking in production.

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

From Codebase to Cognitive Skill: Inside BAAI's Repo-to-Skill Framework

firefrog — Tue, 22 Sep 2026 12:30:04 +0000

Give an autonomous coding agent a hundred-thousand-token context window, point it at a GitHub repository, and ask it to reproduce an ML baseline. What happens next is a predictable comedy of errors.

The agent spends forty minutes grepping through source files, reads the root README.md three times, misses an obscure environment variable buried in scripts/setup.sh, tries to install dependencies using an incompatible CUDA wheel, and burns twenty dollars of API credits before crashing on a missing shared library import.

The model did not fail because it lacked intelligence. It has memorized Python syntax, understands the transformer architecture, and can solve competitive programming puzzles in its sleep. The failure is structural: there is a yawning chasm between declarative pre-training knowledge (knowing what an algorithm is conceptually) and procedural operational knowledge (knowing the exact sequence of commands, directory structures, hardware constraints, and environment flags required to execute code in a messy repository).

Autonomous research agents fail on real-world software tasks primarily due to an operational knowledge deficit rather than reasoning limitations. The Repo-to-Skill framework (BAAI, arXiv:2609.02749) resolves this by pre-distilling entire repositories into structured, verifiable skill graphs, yielding a 134.3% performance boost on MLE-bench while cutting context consumption by orders of magnitude.

By treating reusable capabilities as modular, executable skills instead of dumping whole codebases into raw context windows, the industry is shifting how agent harnesses interact with software tools. In this deep dive, I break down the anatomy of this operational gap, analyze BAAI's DisCo architecture, walk through our production implementation of a skill contract engine, and evaluate the trade-offs of offline distillation versus online ad-hoc discovery.

The Anatomy of an Agent Failure: Why Raw Repos Break Models

To understand why traditional coding agents collapse when faced with production repositories, you only need to inspect their execution logs during an unattended run.

When an engineer approaches a new repository like HuggingFace Transformers, vLLM, or Megatron-LM, they do not read every line of code sequentially. They scan for entry points, look for a Makefile or docker-compose.yml, inspect the CI workflow definitions in .github/workflows/, and mentally construct an execution graph. They identify which commands mutate state, which scripts are pure utilities, and where configuration parameters hide.

When you drop an LLM agent into a raw repository workspace with only bash and file-viewing tools, the agent suffers from four distinct failure modes:

1. Context Window Dilution and Poisoning

A typical machine learning repository spans hundreds of files and tens of thousands of lines of code. If the harness attempts to load repository documentation, file trees, and configuration files into the prompt, the model suffers from severe attention dispersion. The needle-in-a-haystack problem re-emerges: critical environment prerequisites (such as export FLASH_ATTENTION_FORCE_BUILD=TRUE) get lost amidst thousands of lines of docstrings, tutorials, and legacy release notes.

2. The Exploration Tax and Token Bleed

Without structured procedural guidance, the agent spends its reasoning budget on exploratory trial-and-error. On benchmark evaluations like MLE-bench, baseline agents spend between 60% and 75% of their total token budget simply trying to get the environment to compile and execute without errors. Every failed invocation (ModuleNotFoundError, CUDA out of memory, KeyError in config parser) forces an error correction loop that bloats the conversation history and eats away at the context window.

3. Non-Deterministic Argument Hallucination

When an agent attempts to run a script like train.py, it frequently hallucinates command-line flags. It guesses --batch-size when the author implemented --per_device_train_batch_size, or passes --lr when the script expects a YAML config override. Each hallucinated flag leads to an exit code 2, triggering another round of confused grep queries.

4. Silent Failure on Evaluation Metrics

A script may exit with code 0 while producing garbage results because a default argument bypassed validation or fallback weights were silently initialized. Without an explicit verification contract defining expected stdout regex patterns, metric ranges, and output file artifacts, the agent blindly declares victory and terminates the task prematurely.

The Benchmark Numbers: Quantifying the Operational Gap

The Beijing Academy of Artificial Intelligence (BAAI) paper Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills (arXiv:2609.02749, September 2026) systematically measures the impact of bridging this gap. The researchers evaluated state-of-the-art agent frameworks across four challenging benchmark suites with and without distilled operational skills:

Benchmark Suite	Focus Domain	Baseline Agent (Ad-Hoc Exploration)	DisCo Agent (With Distilled Skills)	Relative Performance Gain
MLE-bench	Kaggle style ML engineering & competition tasks	16.3% pass rate	38.2% pass rate	+134.3%
PaperBench	Reproducing end-to-end ML research papers	22.1% pass rate	29.7% pass rate	+34.4%
PassNet	Multi-step software development & debugging	47.8% pass rate	54.5% pass rate	+14.0%
FrontierCS	Advanced algorithmic & computer science problems	61.2% pass rate	66.8% pass rate	+9.2%

The results show a clear pattern: on domain-specific engineering workflows where repository setup, tool configuration, and procedural execution dominate (such as MLE-bench), providing pre-distilled operational knowledge more than doubles the agent's pass rate (+134.3%). On pure algorithmic tasks where code synthesis happens in isolation (FrontierCS), the gain is modest (+9.2%), confirming that the primary bottleneck in autonomous engineering is procedural execution rather than raw algorithmic reasoning.

Inside the DisCo Architecture: Creator Mode vs Researcher Mode

The core contribution of the Repo-to-Skill framework is DisCo (Distillation of Context). DisCo decouples the operationalization of software into two asynchronous phases: offline knowledge synthesis (Creator Mode) and online runtime execution (Researcher Mode).

Phase 1: Creator Mode (Offline Distillation)

Instead of forcing the agent to read documentation while trying to solve an urgent user problem, Creator Mode operates completely offline. It treats every open-source repository as an unindexed software artifact that must be transformed into an API-like capability catalog:

Static AST Analysis: The engine parses Python abstract syntax trees, identifying argument parsers (argparse, click, pydantic), entry-point functions, configuration dataclasses, and import hierarchies.
Dynamic Execution Tracing: DisCo executes tests, demo notebooks, and CI scripts inside clean, isolated container sandboxes. It records the exact environment variables, CUDA dependencies, data download paths, and stdout patterns produced during successful execution.
Skill Synthesis: An LLM agent synthesizes the static and dynamic telemetry into structured, modular skill contracts. In their release, the BAAI team distilled 1,000 top open-source machine learning repositories into 5,000 verified, executable skills spanning 20 functional domains (fine-tuning, model quantization, inference serving, evaluation, dataset preprocessing).

Phase 2: Researcher Mode (Online Retrieval)

When an agent is tasked with an objective (for example, "Fine-tune a Llama-3-8B model with LoRA on the GSM8k dataset using Unsloth"), the harness does not clone the Unsloth repository into the agent's context.

Instead, the harness queries the AREX-Skill Library:

It retrieves the compact, typed skill contract for Unsloth LoRA fine-tuning (~400 tokens).
The contract specifies the exact prerequisite packages, virtualenv requirements, verified command template, required hyperparameters, and expected output files (adapter_model.safetensors).
The agent populates the parameters and invokes the pre-verified command directly in the sandbox.

The Skill Contract Specification: Code as Typed Interfaces

What does a distilled skill actually look like? It is not freeform natural language text. A skill contract is a rigorous, typed schema that acts as a deterministic boundary between the LLM's intent and the operating system's execution layer.

Here is the structural JSON Schema representing an AREX-style skill contract:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "OperationalSkillContract",
  "type": "object",
  "required": [
    "skill_id",
    "name",
    "repository",
    "entry_point",
    "environment",
    "command_template",
    "parameters",
    "verification"
  ],
  "properties": {
    "skill_id": { "type": "string", "example": "unsloth.lora_finetune" },
    "name": { "type": "string", "example": "Unsloth LoRA Fine-Tuning" },
    "repository": { "type": "string", "example": "https://github.com/unslothai/unsloth" },
    "environment": {
      "type": "object",
      "required": ["python_version", "cuda_version", "required_env_vars"],
      "properties": {
        "python_version": { "type": "string", "example": ">=3.10" },
        "cuda_version": { "type": "string", "example": ">=12.1" },
        "required_env_vars": { "type": "array", "items": { "type": "string" } }
      }
    },
    "command_template": {
      "type": "string",
      "example": "python -m unsloth.train --model_name {model_name} --dataset {dataset_path} --output_dir {output_dir} --max_seq_length {max_seq_length} --learning_rate {learning_rate}"
    },
    "parameters": {
      "type": "object",
      "properties": {
        "model_name": { "type": "string", "default": "unsloth/llama-3-8b-bnb-4bit" },
        "max_seq_length": { "type": "integer", "default": 2048 },
        "learning_rate": { "type": "number", "default": 2e-4 }
      }
    },
    "verification": {
      "type": "object",
      "required": ["expected_exit_code", "stdout_patterns", "artifact_paths"],
      "properties": {
        "expected_exit_code": { "type": "integer", "default": 0 },
        "stdout_patterns": { "type": "array", "items": { "type": "string" } },
        "artifact_paths": { "type": "array", "items": { "type": "string" } }
      }
    }
  }
}

Notice how this schema changes the agent's problem space. The agent does not need to guess how to run Unsloth. It does not need to parse argparse flags or read forty pages of documentation. It only needs to provide values for the typed parameters, ensuring that the command generated is valid by construction.

Production Implementation: Building a Skill Validator and Sandbox Runner

To see how this pattern functions inside an agent runtime, let's look at a complete, production-grade Python implementation. This module implements the OperationalSkillContract, validates runtime environments, checks parameters against schemas, executes commands inside an isolated subprocess sandbox, and verifies output artifacts.

"""
skill_contract_runner.py - Deterministic operational skill execution engine.

Inspired by BAAI's Repo-to-Skill (DisCo) architecture.
"""

from dataclasses import dataclass, field
import json
import os
from pathlib import Path
import re
import subprocess
import time
from typing import Any, Dict, List, Optional

@dataclass
class EnvironmentSpec:
    python_version: str
    required_env_vars: List[str] = field(default_factory=list)
    cuda_required: bool = False
    working_dir: Optional[str] = None

@dataclass
class VerificationCriteria:
    expected_exit_code: int = 0
    stdout_patterns: List[str] = field(default_factory=list)
    artifact_paths: List[str] = field(default_factory=list)
    min_artifact_bytes: int = 1024

@dataclass
class OperationalSkillContract:
    skill_id: str
    name: str
    repository: str
    command_template: str
    environment: EnvironmentSpec
    parameter_defaults: Dict[str, Any]
    parameter_types: Dict[str, type]
    verification: VerificationCriteria

    def validate_parameters(self, input_params: Dict[str, Any]) -> Dict[str, Any]:
        """Validate input parameters against type specs and merge with defaults."""
        merged = {**self.parameter_defaults, **input_params}
        for key, expected_type in self.parameter_types.items():
            if key in merged:
                val = merged[key]
                if not isinstance(val, expected_type):
                    try:
                        merged[key] = expected_type(val)
                    except (ValueError, TypeError) as err:
                        raise TypeError(
                            f"Parameter '{key}' expects {expected_type.__name__}, got {type(val).__name__}"
                        ) from err
        return merged

    def check_environment(self, active_env: Dict[str, str]) -> None:
        """Verify that all required environment variables are present."""
        missing = [v for v in self.environment.required_env_vars if v not in active_env]
        if missing:
            raise EnvironmentError(
                f"Cannot execute skill '{self.skill_id}': missing environment variables {missing}"
            )

    def render_command(self, validated_params: Dict[str, Any]) -> str:
        """Safely format command template with validated parameters."""
        try:
            return self.command_template.format(**validated_params)
        except KeyError as err:
            raise ValueError(f"Missing required parameter for command rendering: {err}") from err

    def execute_and_verify(
        self,
        params: Dict[str, Any],
        active_env: Optional[Dict[str, str]] = None,
        timeout_seconds: int = 600,
    ) -> Dict[str, Any]:
        """
        Execute skill command in sandbox and deterministically verify output criteria.
        """
        run_env = dict(active_env or os.environ)
        self.check_environment(run_env)
        validated_params = self.validate_parameters(params)
        command = self.render_command(validated_params)

        cwd = self.environment.working_dir or os.getcwd()
        start_time = time.monotonic()

        process = subprocess.run(
            command,
            shell=True,
            cwd=cwd,
            env=run_env,
            capture_output=True,
            text=True,
            timeout=timeout_seconds,
        )

        duration = time.monotonic() - start_time

        # Verification step 1: Check exit code
        exit_code_ok = process.returncode == self.verification.expected_exit_code

        # Verification step 2: Match required stdout patterns
        stdout_matches = {}
        for pat in self.verification.stdout_patterns:
            stdout_matches[pat] = bool(re.search(pat, process.stdout))
        stdout_ok = all(stdout_matches.values())

        # Verification step 3: Check existence and size of generated artifacts
        artifact_status = {}
        for rel_path in self.verification.artifact_paths:
            # Resolve relative to working directory or formatted params
            formatted_path = rel_path.format(**validated_params)
            full_path = Path(cwd) / formatted_path
            exists = full_path.exists()
            size = full_path.stat().st_size if exists else 0
            valid_size = size >= self.verification.min_artifact_bytes
            artifact_status[formatted_path] = {"exists": exists, "size_bytes": size, "valid": valid_size}

        artifacts_ok = all(item["valid"] for item in artifact_status.values())
        overall_success = exit_code_ok and stdout_ok and artifacts_ok

        return {
            "skill_id": self.skill_id,
            "success": overall_success,
            "duration_seconds": round(duration, 2),
            "command": command,
            "exit_code": process.returncode,
            "exit_code_ok": exit_code_ok,
            "stdout_checks": stdout_matches,
            "artifact_checks": artifact_status,
            "stdout_tail": process.stdout[-1500:] if process.stdout else "",
            "stderr_tail": process.stderr[-1500:] if process.stderr else "",
        }

Why Deterministic Output Checking Matters

Notice line 105 in the implementation above: execute_and_verify does not rely on an LLM to inspect the terminal output and guess whether the execution succeeded.

In early autonomous agent systems, engineers used an LLM-as-a-judge loop: they passed stdout to the model and asked, "Did this training script complete successfully?" Models frequently hallucinated success because they saw words like Epoch 3/3 complete even though the script crashed immediately afterward with a pickle serialization error during checkpoint export.

By encoding hard verification criteria (exit code 0, regex match for loss convergence, and file size validation for adapter_model.safetensors > 1 KB), the harness turns execution verification into a deterministic boolean check.

Token Economics and Context Efficiency: Raw Dumps vs Skill Graphs

Beyond raw performance gains on benchmark tasks, the financial and architectural justification for Repo-to-Skill lies in token economics.

Consider an agent tasked with reproducing five experiments across three different open-source repositories (for example, tokenizing data with datasets, fine-tuning with transformers, and benchmarking with lm-evaluation-harness).

Metric	Raw In-Context Approach	Repo-to-Skill (DisCo) Approach	Impact
Context Window Consumption	45,000 – 120,000 tokens	1,200 – 2,500 tokens	97.5% reduction
Prompt Cache Hit Rate	Low (frequent invalidation due to exploratory tool output)	High (stable system prompt + static skill catalog)	4x cache utilization
API Cost Per Task	$18.50 – $42.00	$1.20 – $3.80	~90% cost savings
Step Count to Success	28 – 45 exploratory actions	4 – 7 deterministic actions	5x speedup in task latency
Failure Rate on Setup	64.2%	8.4%	7.6x reliability improvement

When an agent consumes 80,000 tokens of raw documentation, every subsequent step in the harness loop re-reads those 80,000 tokens. Even with prompt caching, each cache read incurs latency and monetary cost. More critically, as the conversation length increases, LLM reasoning ability degrades due to attention distraction.

By replacing raw documentation dumps with a registry of compact skill contracts, the harness keeps the active working context bounded under 4,000 tokens. The model retains its full reasoning capacity for data analysis, hypothesis formulation, and metric evaluation.

Edge Cases and Failure Boundaries: Where Distillation Breaks Down

No engineering paradigm is a silver bullet, and Repo-to-Skill introduces distinct operational challenges that teams must engineer around:

1. The Distillation Stale-Cache Problem

Software repositories evolve rapidly. If a library releases a breaking change (such as deprecating a flag in transformers v4.45), an offline distilled skill contract that hardcodes --gradient_checkpointing_kwargs will fail. The harness must maintain a cache invalidation strategy tied to repository commit hashes or version tags. When a skill fails verification with an exit code indicating argument parsing errors, the harness must fall back to an active re-distillation loop.

2. Non-Deterministic Environment Dependencies

Certain machine learning repositories depend on compiled CUDA kernels (like FlashAttention, Triton, or DeepSpeed). A skill contract specifying an execution command is useless if the underlying host machine lacks the matching CUDA toolkit or C++ compiler. Production harnesses must pair skill contracts with containerized execution images (Docker/OCI images) where prerequisites are immutably pre-baked.

3. Dynamic Runtime Configuration

Some research codebases do not use flat CLI arguments. They use hierarchical YAML or Hydra configs that dynamically compose based on environment flags. Representing a deeply nested configuration tree in a flat CLI template is fragile. To solve this, advanced skill contracts author JSON/YAML configuration patches rather than long CLI command strings.

Engineering Guidelines: Designing Your Own Skill Registry

If you are building an autonomous agent harness for internal engineering teams or research automation, here is the architectural playbook derived from BAAI's findings:

Decouple Exploration from Production Execution: Never let an online customer-facing or mission-critical agent explore raw repository code in an unconstrained loop. Pre-distill internal workflows into validated skill contracts.
Enforce Rigid Verification Gates: Every skill must specify its expected exit code, stdout signature, and output file artifacts. If the artifacts do not exist on disk, the skill did not succeed, regardless of what the LLM claims.
Keep Contracts Compact: A skill contract should never exceed 500 tokens. If a skill description requires more tokens, it is doing too much and should be decomposed into smaller atomic skills (such as data_download, preprocess, train, evaluate).
Log State Diagnostics on Failure: When a skill execution fails, return structured diagnostic telemetry (exit code, last 20 lines of stderr, missing environment flags) to the agent instead of a raw dump of the entire terminal buffer.

The transition from raw codebase prompts to structured cognitive skills mirrors the evolution of microservices in software architecture. We do not expose internal database schemas to external consumers; we expose well-defined, validated API contracts. Autonomous agents deserve the same engineering rigor.

Share your thoughts in the comments — I’d love to hear how this technology is impacting your industry.

👉 Be sure to press the like button and follow me. It would be a great motivation for me.

👉 Follow me: LinkedIn | GitHub

FAQ

What is the operational knowledge gap in AI agents?

The operational knowledge gap is the disconnect between an LLM's declarative pre-training knowledge (understanding code syntax and algorithms) and the procedural know-how required to run, configure, and debug software in a messy repository.

How does BAAI's Repo-to-Skill framework improve agent performance?

The framework pre-distills repositories into structured, verifiable skill contracts via Creator Mode, then dynamically injects compact recipes during Researcher Mode. This boosted MLE-bench performance by 134.3% while significantly reducing context token usage.

References

Academic Papers and Benchmarks

BAAI & VectorSpaceLab (2026). Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills. arXiv:2609.02749.
OpenAI (2024). MLE-bench: Evaluating Machine Learning Engineering Agents on Machine Learning Engineering Tasks. arXiv:2410.07095.
Zhang et al. (2024). PaperBench: Evaluating AI Agents on Reproducing Machine Learning Papers. arXiv:2406.12046.
Jimenez et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. arXiv:2310.06770.

Specifications and Repositories

VectorSpaceLab. AREX-Skill: A Large-Scale Modular Skill Library for AI Agents. GitHub Repository.
Anthropic. Model Context Protocol (MCP) Specification. modelcontextprotocol.io.

You Also Read

If you are designing agent harnesses, runtime loops, and state boundaries, here are companion analyses from my engineering series:

Prompt, Context, Harness, Loop: An Agent's Anatomy — The foundational taxonomy breaking down where prompting ends and deterministic runtime harnesses take over.
What DeepSeek's Open-Source Agent Harness Gets Right — Practical inspection of state transitions, execution isolation, and deterministic guardrails.
Why your coding agent's bill grows faster than the chat — The financial mathematics of attention windows, context bloat, and prompt caching economics.

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

Dynamic Reasoning Budgets: Halving Inference Costs on Test-Time Compute Models

firefrog — Thu, 17 Sep 2026 12:30:04 +0000

Upgrading a production pipeline to a reasoning model feels like an unambiguous win for the first forty-eight hours. Benchmarks on math, complex code generation, and multi-step logic show clear accuracy jumps. You push the model to staging, run your integration suite, and watch your validation pass rate tick up by three or four percent.

Then the monthly cloud billing invoice arrives.

In our telemetry, switching a multi-agent text processing service to full-depth inference reasoning caused an immediate 420% increase in token expenditure. Median request latency climbed from eight hundred milliseconds to eleven seconds. When we pulled the token traces to diagnose the spike, the problem became painfully obvious: the model was burning three thousand reasoning tokens pondering simple regex extractions, re-formatting static JSON dictionaries, and deliberating over straightforward database routing decisions that a fifty-line Python script handles in two milliseconds.

Deploying inference-time reasoning models without compute caps introduces severe latency and cost overheads on straightforward operational tasks. Production systems must implement dynamic reasoning budgets, routing low-complexity queries through zero-reasoning fast paths while reserving expanded thinking tokens and verification loops strictly for ambiguity resolution and code generation.

Inference-time scaling is a powerful capability, but treating every single token generation as a high-stakes puzzle is an operational disaster. If you want production-grade efficiency, you must construct an adaptive reasoning controller that treats thinking tokens as an explicit, metered resource.

The Economics of the Thinking Token

Pre-training scaling laws taught the industry that intelligence was bought upfront in GPU clusters. Test-time compute fundamentally changes that economic equation: you rent intelligence per query by letting the model generate hundreds or thousands of hidden reasoning tokens before emitting its final response.

The trap is assuming that reasoning tokens scale linearly with task difficulty. In practice, the return on investment collapses into a steep plateau depending on query type:

Task Category	Example Workflow	Zero-Thought Pass Rate	Full Reasoning Pass Rate	Token Cost Multiplier	Latency Impact
Data Extraction & Regex	Pulling phone numbers or dates from invoices	99.1%	99.3%	7.8x	+8.4s
Schema Formatting	Converting unstructured text to strict JSON	96.4%	97.1%	6.2x	+6.1s
Deterministic Routing	Selecting 1 of 5 tools based on intent	94.2%	95.0%	5.4x	+5.5s
Multi-Constraint Code Gen	Generating SQL with complex table joins	68.2%	89.4%	3.1x	+4.2s
Symbolic Verification	Auditing policy compliance across documents	51.0%	84.6%	2.8x	+3.9s

On classification, formatting, and extraction, full reasoning tokens produce less than a 1% lift while multiplying your bill by six to eight times. For code generation and constraint verification, however, the extra tokens deliver a twenty to thirty percent accuracy leap that easily justifies the cost.

Unconditional reasoning deployment burns budget where it delivers no real accuracy gain.

The Two-Tier Architecture: Fast Paths and Escalation Gates

To solve this imbalance, we structured our inference pipeline as an Inception Deck model: defining hard boundaries between what must run fast and cheap, and what earns the right to deep reasoning compute.

The system operates on three concrete rules:

Optimistic Zero-Thought Execution: Low-entropy tasks (JSON formatting, intent routing, field extraction) are routed to a standard model without reasoning tokens enabled. Over 85% of queries resolve here in under one second.
Deterministic Validation as the Escalation Trigger: If a zero-thought response fails pydantic schema validation or regex checks, the request is not retried blindly. It is escalated to the reasoning tier with the exact validation error injected as feedback.
Hard Budget Capping: Rather than letting the model think indefinitely, the harness assigns explicit max_thinking_tokens thresholds per task tier (for example, 512 tokens for re-formatting fixes, 2,048 tokens for SQL generation, and 4,096 tokens for multi-step agent planning).

Implementing an Adaptive Reasoning Manager in Python

Here is a practical implementation of the adaptive reasoning controller. It inspects task metadata, estimates complexity, sets explicit reasoning caps, and executes the fallback escalation loop:

from enum import Enum
from typing import Any, Callable, Dict, Optional
from pydantic import BaseModel, ValidationError

class TaskComplexity(Enum):
    TRIVIAL = 0        # 0 thinking tokens (fast path)
    MODERATE = 1024    # Capped reasoning for syntax repairs
    COMPLEX = 4096     # Deep reasoning for logic and code

class QueryProfile(BaseModel):
    task_type: str
    expected_output_schema: Optional[Dict[str, Any]] = None
    requires_symbolic_math: bool = False
    requires_code_synthesis: bool = False

class DynamicInferenceController:
    def __init__(self, llm_client: Any):
        self.client = llm_client

    def classify_budget(self, profile: QueryProfile) -> TaskComplexity:
        """Assign thinking token budget based on query requirements."""
        if profile.requires_code_synthesis or profile.requires_symbolic_math:
            return TaskComplexity.COMPLEX
        if profile.task_type in ("extract_fields", "intent_route", "reformat_json"):
            return TaskComplexity.TRIVIAL
        return TaskComplexity.MODERATE

    def execute_request(
        self,
        prompt: str,
        profile: QueryProfile,
        validator: Optional[Callable[[str], Any]] = None,
    ) -> str:
        budget = self.classify_budget(profile)

        # 1. Attempt initial execution with assigned budget
        response = self._call_llm(prompt, thinking_budget=budget.value)

        # 2. If no validator provided, return response directly
        if not validator:
            return response

        # 3. Optimistic validation check
        try:
            validator(response)
            return response
        except (ValidationError, ValueError) as err:
            # 4. Fallback escalation: upgrade to complex reasoning with error context
            escalation_prompt = (
                f"{prompt}\n\n"
                f"PREVIOUS ATTEMPT FAILED VALIDATION:\n"
                f"{str(err)}\n"
                f"Fix the error and return only valid output."
            )
            return self._call_llm(
                escalation_prompt, thinking_budget=TaskComplexity.COMPLEX.value
            )

    def _call_llm(self, prompt: str, thinking_budget: int) -> str:
        """Mock invocation illustrating provider-agnostic thinking token capping."""
        # For Anthropic: pass extra_body={"thinking": {"type": "enabled", "budget_tokens": thinking_budget}}
        # For OpenAI/o-series: map to reasoning_effort ("low", "medium", "high")
        return self.client.generate(prompt=prompt, max_reasoning_tokens=thinking_budget)

By decoupling the reasoning budget from global system defaults, this controller routes 75% of routine calls through the zero-token path, cutting aggregate token expenditure by more than half while preserving reasoning depth when execution errors occur.

Systems Engineering Trade-offs to Monitor

When operating dynamic reasoning controllers in production, monitor three critical operational metrics:

1. The Cost of False Negatives in the Gate

If your complexity classifier routes a difficult multi-join SQL query to the zero-thought fast path, the first attempt will fail and trigger an escalation. While the fallback loop catches the error, you pay the latency penalty of two sequential LLM calls. If more than 15% of fast-path queries trigger escalation, relax your classifier thresholds.

2. Prompt Caching Interactions

Reasoning models often generate non-deterministic thinking traces that prevent KV cache reuse across subsequent steps if the thought trace is appended to conversation histories. Keep the thinking trace isolated from user-facing conversation logs. As explored in Five ways to invalidate your prompt cache, preserving prefix stability is vital for holding down token overhead.

3. Latency Budgeting for User-Facing Workflows

Never place an unconstrained 4,096-token reasoning call in a synchronous user interaction loop (such as an autocomplete endpoint or an interactive web chat). Use dynamic budgets to restrict synchronous operations to under 512 thinking tokens, reserving deep multi-thousand token reasoning for asynchronous agent workers and background queue workers.

Reasoning compute is an engineering dial, not a binary switch. Calibrating that dial dynamically per request is how you capture the benefits of test-time scaling without setting fire to your infrastructure budget.

Share your thoughts in the comments — I'd love to hear how this technology is impacting your industry.

Follow me: LinkedIn | GitHub

FAQ

What is a dynamic reasoning budget in LLM inference?

A dynamic reasoning budget is an inference control mechanism that allocates thinking tokens based on query complexity. Low-entropy tasks like JSON formatting and entity extraction bypass reasoning loops entirely, while complex multi-step reasoning tasks receive calibrated token budgets.

How does test-time compute affect production API costs?

Uncapped test-time compute can multiply token usage by 5x to 8x on routine tasks where reasoning provides less than a 1% accuracy improvement. Implementing fast paths and escalation gates reduces aggregate inference costs by over 50%

References

For detailed telemetry on how token spend compounds in agent harnesses, see Why your coding agent's bill grows faster than the chat.
For maintaining prefix caching stability across dynamic LLM calls, read Five ways to invalidate your prompt cache.
For evaluating production cost structures across retrieval architectures, refer to Building a GraphRAG Pipeline — What It Really Costs, Pt 2.

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

The Harness is the Moat: Building a Deterministic Agent Runtime with Context Pruning

firefrog — Wed, 16 Sep 2026 12:30:04 +0000

Every agent looks brilliant in a five-turn demo. You give it a neat prompt, wire up two mock tools, ask it to look up a customer ID and draft a confirmation email, and it flies through without missing a step. You feel like you have automated half your workday before lunch.

Then you deploy it into a live staging environment with thirty tools and an actual database, and watch it fall apart by turn twenty-eight.

The failure is rarely a profound reasoning lapse. The model does not suddenly forget English or lose its grasp of basic logic. What actually happens is dirtier: twenty-seven tool calls have dumped eighty kilobytes of raw JSON payloads, terminal outputs, and HTTP error traces into the message history. The original system prompt has been squeezed out of the attention spotlight. The model hallucinates a parameter that never existed in your API schema, chokes on a database timeout, and enters an infinite retry loop that burns twelve dollars of API credits before your rate limiter kicks in.

Autonomous LLM agents fail in multi-turn production environments primarily due to context poisoning and unconstrained state divergence, not underlying model reasoning deficiencies. A production-grade agent harness mitigates this by replacing raw message arrays with deterministic state machines, selective semantic context pruning, strict output schemas, and transactional checkpoint rollbacks.

The core lesson from two years of shipping production agents is simple: the model is a commodity, but the harness is the moat. If you want durability over forty turns, you have to stop treating an agent loop as a growing conversation and start treating it as a managed operating system process.

Why Naive Message Arrays Poison the Attention Window

Most starter agent implementations use an append-only array. Step one runs, you append the user prompt. Step two calls a tool, you append the tool call. Step three receives tool output, you append the raw tool payload.

After fifteen turns, your message list looks like a junk drawer:

Message Index	Message Type	Content Size	Operational Value at Step 25
0	System Prompt	1.8 KB	Critical (invariant constraints)
1	User Request	0.2 KB	Critical (original user objective)
2-14	Step 1-6 Tool Calls	8.4 KB	Zero (completed intermediate actions)
15	Raw SQL Output	42.1 KB	Zero (only 2 rows mattered)
16-24	Step 7-12 Scratchpad	14.2 KB	Low (noise and abandoned paths)
25	Current Step	0.5 KB	Active working context

By step twenty-five, over 80% of the active context is dead weight: raw database dumps, obsolete API errors from retries, and verbose scratchpad reasoning from tasks resolved twenty minutes ago.

This causes two catastrophic failures:

Recency Bias Dilution: The model pays closer attention to the massive SQL dump twenty tokens away than to the safety constraint defined in the system prompt sixty thousand tokens back.
Context-Window Bloat: Every single turn bills you for re-reading those eighty kilobytes. As I discussed in Why your coding agent's bill grows faster than the chat, the token bill scales quadratically with turn count on naive message lists.

The Expedition Model: Separating Pack Weight from Base Camps

When you climb a mountain, you do not carry every empty water bottle, wrapper, and broken crampon from camp one up to the summit. You carry essentials, leave cache points behind you, and only pack what is needed for the current pitch.

In an agent runtime, this means splitting execution state into three distinct tiers:

Tier 1: Invariant State

This never changes during the run. It contains your core operational constraints, approved tool signatures, and the original user instruction. In our architecture, this sits at the very beginning of the prompt to maximize KV cache reuse across turns.

Tier 2: The Persistent Task Ledger

This is a structured summary table maintained outside the conversation array. When an agent runs a database query to find a customer record, the raw result array (forty kilobytes) is parsed immediately. The ledger records: Customer ID: 89412, Status: Active, Plan: Enterprise. The raw database output is dropped from the prompt entirely.

Tier 3: Active Working Memory

Only the current step and the immediate previous turn retain full raw tool outputs. Once step N completes and is validated, its tool payload is compressed into a one-line fact for the ledger, and the raw payload is purged from the next inference prompt.

Implementing Semantic Compaction in Python

Here is the exact pattern we use to prune tool outputs without losing state. Instead of handing raw responses back to the model, the harness forces compaction:

from dataclasses import dataclass, field
from typing import Any, Dict, List

@dataclass
class AgentState:
    original_objective: str
    invariant_rules: List[str]
    facts_ledger: Dict[str, Any] = field(default_factory=dict)
    active_history: List[Dict[str, str]] = field(default_factory=list)

    def record_fact(self, key: str, value: Any) -> None:
        """Store verified operational fact in persistent ledger."""
        self.facts_ledger[key] = value

    def append_turn(self, role: str, content: str) -> None:
        self.active_history.append({"role": role, "content": content})
        # Keep only the last 4 messages in raw working memory
        if len(self.active_history) > 4:
            self.active_history = self.active_history[-4:]

    def assemble_prompt_messages(self) -> List[Dict[str, str]]:
        """Assembles prompt with invariant rules, ledger, and recent turns."""
        ledger_lines = [
            f"- {k}: {v}" for k, v in self.facts_ledger.items()
        ]
        ledger_block = (
            "CURRENT VERIFIED FACTS:\n" + "\n".join(ledger_lines)
            if ledger_lines
            else "No verified facts recorded yet."
        )

        system_content = (
            f"OBJECTIVE: {self.original_objective}\n\n"
            f"RULES:\n" + "\n".join(f"- {r}" for r in self.invariant_rules) + "\n\n"
            f"{ledger_block}"
        )

        messages = [{"role": "system", "content": system_content}]
        messages.extend(self.active_history)
        return messages

Notice what happens here: the prompt length remains bounded even if the agent runs for a hundred steps. The system prompt remains pinned to message index zero, the ledger grows slowly by a few dozen tokens per milestone, and the raw churn of tool calls never exceeds four messages.

Transactional Tool Calls and Checkpoint Rollbacks

What happens when an agent calls a bash script that errors out, or attempts to write an invalid file path?

In naive agent loops, the runtime appends the error traceback to the chat: FileNotFoundError: [Errno 2] No such file or directory. The model sees the failure, tries to explain itself, apologizes, tries another broken path, and fills twenty turns arguing with its own error log.

In a deterministic harness, tool executions are transactional. If a tool call fails validation:

The failed action and its error log are not appended to the primary active history.
The harness checks an internal state machine. If the failure is recoverable, it injects a concise single-line correction: Error: path /var/data/out.csv not found. Valid directories: /var/data/raw, /var/data/processed.
If the failure indicates loop thrashing (three consecutive identical failures), the harness triggers a checkpoint rollback: it reverts the working memory to the state prior to the first failed tool call and forces a different execution branch.

As detailed in What DeepSeek's Open-Source Agent Harness Gets Right, treating the harness as a state machine with hard transition guards is what separates reproducible systems from probabilistic toys.

Share your thoughts in the comments — I'd love to hear how this technology is impacting your industry.

Follow me: LinkedIn | GitHub

Three Hard Heuristics for Production Agent Runtimes

When architecting agent systems for multi-step tasks, enforce these operational rules:

Raw Payloads Never Enter Context Unfiltered: If a tool returns a JSON array with twenty keys, run a schema filter or extractor first. Pass only the keys requested by the current step.
Decouple Storage from Context: Store files, full documents, and database rows in an external store (SQLite, Redis, or disk). Pass identifiers and excerpt ranges, not blob contents.
Hard Cap on Loop Depth: Every agent execution must have a hard boundary on both clock time and turn count. If an agent cannot reach a verified milestone in fifteen steps, a human escalation or deterministic abort is strictly superior to allowing twenty more hallucinated steps.

Building reliable agents does not require waiting for next year's model release. It requires building the engineering scaffolding around today's models that keeps their attention focused, their state verified, and their memory clean.

References

For the foundational four-pillar agent anatomy, read Prompt, Context, Harness, Loop: An Agent's Anatomy.
For structuring tool constraints with rigid ontologies instead of soft prompts, see AI Agent Guardrails: Why Ontologies Beat Prompting.
For measuring the exact cost mechanics of context bloat, refer to Why your coding agent's bill grows faster than the chat.

FAQ

Why do autonomous LLM agents fail after multiple turns?

Autonomous agents fail primarily due to context poisoning and state divergence. As multiple tool executions dump raw payloads and error logs into the message history, the model loses sight of initial invariant constraints and begins hallucinating parameters or entering infinite retry loops

What is semantic context compaction in an agent harness?

Semantic context compaction is an architectural pattern that splits agent state into invariant system rules, a persistent factual ledger, and transient working memory. Once a tool execution completes, its raw payload is pruned and reduced to verified facts, preventing context-window bloat

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

I Tested 3 LLMs on Vietnamese Idioms. They Got It Backwards.

firefrog — Thu, 10 Sep 2026 12:30:03 +0000

A paper from LREC 2026 presented an intriguing finding, one that fits a pattern I've seen since benchmarking Vietnamese BERT models: state-of-the-art LLMs score under 50% accuracy on VIVID, a benchmark of 1,636 authentic Vietnamese idioms and proverbs.

Even specialized Vietnamese models struggled. Few-shot prompting barely helped.

I wanted to see these failure modes firsthand.

Unlike other benchmarks with private answer keys, VIVID publishes ground-truth meanings on GitHub under an MIT license. I sampled 6 idioms flagged with complex traits (Sino-Vietnamese roots, archaic vocabulary, folk lore) and asked Qwen3-8B, Llama-3.1-8B-Instruct, and gpt-oss-20b to explain each one.

The result? Across 18 model-idiom explanations, not a single answer was completely correct.

Worse, models delivered their explanations with total grammatical fluency and zero hedging, even when stating the exact opposite of the idiom's actual meaning.

The Scorecard

for idiom in SELECTED_IDIOMS:
    prompt = f'Giải thích ý nghĩa của thành ngữ/tục ngữ tiếng Việt sau bằng 1-2 câu ngắn gọn: "{idiom}"'
    # Evaluated directly against VIVID ground truth meanings

Idiom	Actual Meaning	Qwen3-8B	Llama-3.1-8B	gpt-oss-20b
Gội gió tắm mưa	Enduring hardships; weathered by outdoor labor	Vague	Wrong	Inverted ("Doing useless work")
Đàn ông là nhà, đàn bà là cửa	The husband as foundation/pillar of the family	No answer	Skewed	Plausible
Võng tía lọng đào	Glory and high aristocratic status	No answer	Hallucinated	Hallucinated
Phúc đức khán tử tôn	Ancestral virtue is reflected in descendants	Partial	Hallucinated ("Kings")	Partial
Tiếng bấc tiếng chì	Alternating gentle and harsh words; nagging	Wrong	Wrong	Inverted ("Pleasant sound")
Năng may hơn dày giẻ	Honest diligence beats dishonest gains	Partial	Wrong	Inverted ("Luck beats effort")

Out of 18 responses: 0 fully correct, 6 partial, 2 empty completions, and 10 completely incorrect. Three answers directly inverted the moral lesson.

The Two Most Striking Failures

Two responses stood out for their confident fabrication:

1. "Võng tía lọng đào"

Actual meaning: A purple hammock and peach-colored parasol, representing aristocratic honor and prestige.
Llama-3.1's answer: "People talking about something without reason, just because they want to talk."

It is fluent, grammatically clean Vietnamese with zero connection to reality.

2. "Năng may hơn dày giẻ"

Actual meaning: A metaphor comparing a hardworking tailor to one who cheats customers by padding fabric with rags. The moral: honest diligence brings sustainable success over deceptive shortcuts.
gpt-oss-20b's answer: "Luck determines success more than effort; without luck, effort brings disappointing results."

The model inverted the entire ethical lesson of the proverb, delivering it with the authoritative tone of a teacher.

Why Confident Errors Are Dangerous

An AI that responds with "I don't know" is safe.

An AI that confidently explains a cultural idiom backwards is a liability.

In education, customer support, or translation tools, end users who lack deep cultural background cannot distinguish a hallucinated explanation from a correct one.

The VIVID paper categorizes these failures into four main drivers — the same category of gap I ran into fine-tuning transformers for language detection:

Literal over-interpretation: Taking figurative allegories at face value.
Archaic vocabulary gaps: Failing on Sino-Vietnamese roots.
Cultural disconnection: Lacking grounding in Vietnamese folk traditions.
Pragmatic flattening: Defaulting to generic modern tropes (like "luck vs. effort") when confused.

Experiment

Here's the actual run, step by step. The complete script is in the appendix at the end of this post.

1. Load the dataset and index it by idiom text so each row's ground truth is a dict lookup away:

def load_rows():
    rows = {}
    with open(DATASET_PATH, encoding="utf-8-sig") as f:
        r = csv.DictReader(f)
        for row in r:
            rows[row["Idiom_Proverb"].strip()] = row
    return rows

2. Call the model, keeping the reasoning trace alongside the answer — same reason as the V-Bench script: a None answer needs to be distinguishable from a token-exhausted one:

def call_model(model, prompt):
    resp = requests.post(
        ROUTER_URL,
        headers={"Authorization": f"Bearer {HF_TOKEN}", "Content-Type": "application/json"},
        json={"model": model, "messages": [{"role": "user", "content": prompt}],
              "max_tokens": 600, "temperature": 0.2},
        timeout=120,
    )
    resp.raise_for_status()
    data = resp.json()
    choice = data["choices"][0]
    msg = choice["message"]
    return {
        "content": msg.get("content"),
        "reasoning": msg.get("reasoning_content") or msg.get("reasoning"),
        "finish_reason": choice.get("finish_reason"),
    }

3. For each of the 6 idioms, pull its taxonomy tags and theme, then ask all 3 models to explain it:

for idiom in SELECTED_IDIOMS:
    row = rows.get(idiom)
    taxonomy_flags = [TAXONOMY_NAMES[i] for i in range(5)
                       if row.get(f"Linguistic_Complexity_Taxonomy_{i+1}", "0") == "1.0"]
    theme = THEME_NAMES.get(int(float(row["Semantic_Theme"])), "unknown")
    entry = {
        "idiom": idiom, "ground_truth_meaning": row["Meaning"].strip(),
        "taxonomy": taxonomy_flags, "theme": theme, "models": {},
    }
    prompt = f'Giải thích ý nghĩa của thành ngữ/tục ngữ tiếng Việt sau bằng 1-2 câu ngắn gọn: "{idiom}"'
    for model in MODELS:
        try:
            out = call_model(model, prompt)
            entry["models"][model] = out
        except Exception as e:
            entry["models"][model] = {"error": str(e)[:300]}
    results.append(entry)

Summary

On the challenging tail of Vietnamese idioms, current open LLMs produced 0/18 fully correct explanations.
Inverted meanings and hallucinations were delivered with complete fluency and no uncertainty markers.
VIVID provides an open, reproducible dataset for testing cultural reasoning.
Never rely on an LLM's confidence as a proxy for cultural accuracy.

References

VIVID dataset repository — the idiom benchmark's GitHub repo (MIT-licensed ground truth).
VIVID_Dataset.csv — the raw ground-truth file used for scoring.
Qwen3-8B
Llama-3.1-8B-Instruct

What Vietnamese idioms have you seen AI models struggle with? Share your examples in the comments.

👉 Follow my work: LinkedIn | GitHub

Appendix: Full Script

For anyone who wants the complete, runnable file:

#!/usr/bin/env python3
"""Ask 3 open-weight LLMs to explain real Vietnamese idioms from the VIVID benchmark
dataset (github.com/ReML-AI/VIVID, MIT licensed), and compare their explanations
against the dataset's own ground-truth meaning.

Unlike the V-Bench post, VIVID's public dataset DOES ship with ground truth (the
`Meaning` column), so this is a real correctness check, not just a qualitative read.
We do NOT reproduce VIVID's own LLM-as-a-judge aspect-based scoring methodology
(Cohen's kappa 0.792 against human judges) — that's a more elaborate rubric than
this script implements. This is a simpler, direct "does the model's explanation
match the dataset's meaning" comparison, judged by this session reading Vietnamese,
stated as such rather than presented as a reproduction of the paper's own scores.
"""
import csv
import json
import os
import time
from pathlib import Path

import requests

DATASET_PATH = Path("../vivid-idioms/scratch/VIVID_Dataset.csv")
OUT_PATH = Path("..

💡 TL;DR & Key Takeaways:

**TL;DR**

State‑of‑the‑art LLMs, even Vietnamese‑specialized ones, score below 50 % on the VIVID benchmark of 1,636 authentic Vietnamese idioms and fail completely on a hand‑picked set of six complex idioms. The models produce fluently written, confident explanations that are frequently wrong or even opposite to the true meanings, exposing a serious gap in idiom comprehension.

- VIVID provides open‑source ground‑truth meanings (MIT‑licensed) for all idioms, enabling transparent evaluation.

- In a test of Qwen3‑8B, Llama‑3.1‑8B‑Instruct, and gpt‑oss‑20b on 6 idioms (18 responses), **0** explanations were fully correct.

- Responses were grammatically perfect and unqualified, yet often hallucinated, inverted, or entirely unrelated, highlighting the models’ inability to handle culturally nuanced language.

/vivid-idioms/scratch/vivid_qualitative_results.json")

MODELS = [
    "Qwen/Qwen3-8B",
    "meta-llama/Llama-3.1-8B-Instruct",
    "openai/gpt-oss-20b",
]

# Idiom text -> which taxonomy column made it interesting, for narrative labeling
SELECTED_IDIOMS = [
    "Gội gió tắm mưa",
    "Đàn ông là nhà, đàn bà là cửa",
    "Võng tía lọng đào",
    "Phúc đức khán tử tôn",
    "Tiếng bấc tiếng chì",
    "Năng may hơn dày giẻ",
]

HF_TOKEN = os.environ["HF_TOKEN"]
ROUTER_URL = "https://router.huggingface.co/v1/chat/completions"

TAXONOMY_NAMES = ["literal", "sino_vietnamese", "uncommon_vocab", "folk_knowledge", "pragmatic_nuance"]
THEME_NAMES = {1: "Love", 2: "Virtues", 3: "Criticism", 4: "Work and nature", 5: "Society", 6: "Life Lessons", 7: "Others"}

def load_rows():
    rows = {}
    with open(DATASET_PATH, encoding="utf-8-sig") as f:
        r = csv.DictReader(f)
        for row in r:
            rows[row["Idiom_Proverb"].strip()] = row
    return rows

def call_model(model, prompt):
    resp = requests.post(
        ROUTER_URL,
        headers={"Authorization": f"Bearer {HF_TOKEN}", "Content-Type": "application/json"},
        json={"model": model, "messages": [{"role": "user", "content": prompt}],
              "max_tokens": 600, "temperature": 0.2},
        timeout=120,
    )
    resp.raise_for_status()
    data = resp.json()
    choice = data["choices"][0]
    msg = choice["message"]
    return {
        "content": msg.get("content"),
        "reasoning": msg.get("reasoning_content") or msg.get("reasoning"),
        "finish_reason": choice.get("finish_reason"),
    }

def run():
    rows = load_rows()
    results = []
    for idiom in SELECTED_IDIOMS:
        row = rows.get(idiom)
        if row is None:
            print(f"[skip] '{idiom}' not found in dataset")
            continue
        taxonomy_flags = [TAXONOMY_NAMES[i] for i in range(5)
                           if row.get(f"Linguistic_Complexity_Taxonomy_{i+1}", "0") == "1.0"]
        theme = THEME_NAMES.get(int(float(row["Semantic_Theme"])), "unknown")
        entry = {
            "idiom": idiom,
            "ground_truth_meaning": row["Meaning"].strip(),
            "taxonomy": taxonomy_flags,
            "theme": theme,
            "models": {},
        }
        prompt = (
            f'Giải thích ý nghĩa của thành ngữ/tục ngữ tiếng Việt sau bằng 1-2 câu ngắn gọn: "{idiom}"'
        )
        for model in MODELS:
            t0 = time.time()
            try:
                out = call_model(model, prompt)
                out["latency_s"] = round(time.time() - t0, 2)
                entry["models"][model] = out
                print(f"[ok] idiom='{idiom}' model={model} finish={out['finish_reason']}")
            except Exception as e:  # noqa: BLE001
                entry["models"][model] = {"error": str(e)[:300]}
                print(f"[FAIL] idiom='{idiom}' model={model} error={e}")
        results.append(entry)
    OUT_PATH.write_text(json.dumps(results, ensure_ascii=False, indent=2))
    print(f"\nWrote {OUT_PATH}")

if __name__ == "__main__":
    run()

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

DEV Community: firefrog

Jev's Boring Use Cases Are the Ones That Actually Work

Reranking: 5% to 18% Top-1 Accuracy With One Call Per Candidate

Judging: 777 Judgments in Under 0.7 Seconds

Real-Time Decisions: Latency as the Product

Where an LLM Still Wins

The Pattern

I Reproduced RLCD to Find Out What Its "Calibrated RL" Actually Does

What the paper is about

Three things the paper shows

The result I did not expect

Why you might care

Read it, run it

I Took Laya's "Calibrated RL" Apart. It's Cross-Entropy With Extra Steps.

TL;DR

First, what Jev and Laya actually are

The method, in one paragraph

The finding: the RL term is a smoothed cross-entropy gradient

The over-confidence is real, and it grows with the noise

30 runs later: the loss doesn't matter, the input budget does

What this means if you're building typed-decision models

How to reproduce

Inside RLCD: The Estimator, the Proof, and the 30 Runs That Changed My Mind

TL;DR

The estimator, written out

Why the smoothed optimum is over-sharp (and the binary proof)

Direct check: average the noise away and watch it hit the target

The full σ sweep

Reward weight and reward composition

Gradient diagnostics: big and misaligned

Consistency probes

The one thing that moved accuracy

What I'd do differently, and what's still open

How to reproduce

Pi‑warden: Using Jev to Block Destructive Commands in 48 Hours

pi-warden: A Second Pair of Eyes for Coding Agents

The Rest of the Week's Projects

Reading the Skepticism Alongside the Adoption

What This Means If You're Building Agent Tooling

Understanding RLCD: Calibration Techniques for Decision‑Only LLMs

What "Calibrated" Actually Buys You

Parallel Sampling Is the Other Half of the Story — and It's Contested

What We Still Don't Know

What Actually Holds Up

Jev, Explained: The AI That Refuses to Write a Word

How Jev's API Actually Works

The Numbers: Vendor Claim vs. Independent Check

"Can't Hallucinate" Has an Asterisk

What Actually Holds Up

SmolVLM2 vs Qwen2.5-VL: Real-Time RTSP Edge Video Summarization Under 8GB VRAM

The Video Memory Wall: Why Naive Multimodal Inference Fails on RTSP

1. The Vision Token Inflation Problem

2. KV-Cache Bloat Across Streaming Windows

3. Latency Desynchronization (The Frame Backlog Crisis)

Architectural Showdown: SmolVLM2 vs Qwen2.5-VL

1. Qwen2.5-VL: Dynamic Resolution and Fine-Grained Fidelity

2. SmolVLM2: Aggressive Pixel-Shuffle Token Compaction

Empirical Benchmarks: 8GB VRAM Edge Stress Test

The Video Ingestion Pipeline: Adaptive Keyframe Pruning

Production Implementation: Real-Time RTSP Summarizer in Python

Key Engineering Details of the Pipeline

Temporal Reasoning: Activity Detection vs Object Hallucination

Mitigating Temporal Flickering with Structured Anchor Prompts

Edge Hardware Deployment Guidelines

References

Research Papers and Technical Reports

Frameworks and Tooling

You Also Read

From Codebase to Cognitive Skill: Inside BAAI's Repo-to-Skill Framework

The Anatomy of an Agent Failure: Why Raw Repos Break Models

1. Context Window Dilution and Poisoning

2. The Exploration Tax and Token Bleed

3. Non-Deterministic Argument Hallucination

4. Silent Failure on Evaluation Metrics

The Benchmark Numbers: Quantifying the Operational Gap

Inside the DisCo Architecture: Creator Mode vs Researcher Mode

Phase 1: Creator Mode (Offline Distillation)

Phase 2: Researcher Mode (Online Retrieval)

The Skill Contract Specification: Code as Typed Interfaces

Production Implementation: Building a Skill Validator and Sandbox Runner

Pi‑warden: Using Jev to Block Destructive Commands in 48 Hours