DEV Community: Bhushan Kinge

We Tested a 35B LLM Against Typed-Decision Models on 12,000 Real RFQs—Confidence Changed the Winner

Bhushan Kinge — Thu, 24 Sep 2026 19:39:35 +0000

91.9%. 89.6%. 78.0%.

Those were the primary-class accuracies of a typed-decision API, a 35B mixture-of-experts LLM, and a 421M open-weight decision model on the same real classification job.

But accuracy was not the result that changed the deployment decision.

The decisive question was:

Can the model tell us when its answer is safe enough to automate—and when a human needs to look?

To find out, I compared Jev, Qwen3.5-35B-A3B, and Laya 421M on 12,000 U.S. federal IT solicitations. The models were graded against behavioral evidence: what a reseller's sales team actually quoted.

The complete pipeline, aggregate metrics, figures, experiment design, and runbook are open in jev-laya-classification-bench.

Privacy note: The public repository contains aggregate results only. Solicitation samples, quote identifiers, gold-label files, and per-row predictions remain private.

The real job behind the benchmark

A federal IT reseller receives solicitations from SEWP, GSA MAS, and GSA 2GIT. Every opportunity must be classified before it can move through the CRM:

What is being purchased: hardware, software, support, services, or something else?
Is it a renewal, upgrade, replacement, or new purchase?
What solution domain owns it?
Does hardware come from distributor catalog SKUs, or does it require an OEM-configured build?
Is the text insufficient, an RFI, or brand-name-only?

These decisions are operational. Hardware may trigger distributor price lookups. A configured build may need an engineer and an OEM portal. Software can require publisher authorization. Services go to a statement-of-work workflow.

The goal was not to win a generic benchmark. It was to determine how much of this queue could be automated safely.

Three very different model paths

All three paths produced one shared label schema:

Path	Model	Execution
Jev	TypeSafe System One, jev-1.13.0	Hosted API with calibrated probabilities
Qwen	Qwen3.5-35B-A3B-FP8	On-prem vLLM with structured JSON
Laya	Convai Laya, 421M parameters	Open weights on an RTX 2000 Ada laptop GPU

Jev and Laya received byte-identical typed-question bundles. Qwen received a strict JSON schema, and its answers were mapped onto the same taxonomy.

All three labeled the 12,000-opportunity sample. Qwen produced 69 permanently malformed responses, leaving 11,931 rows in the all-source paired set.

Ground truth came from behavior, not a labeling exercise

When a sales rep quoted an opportunity, the product types on the latest quote lines provided the composition label. That yielded 741 paired opportunities with a single unambiguous primary class.

For fulfillment mode, configurator fingerprints and distributor information on quote lines produced 634 paired labels.

This gold is useful because it reflects real downstream behavior, but it is not perfect truth. Quoted opportunities are a selected subset, Hardware represents 77% of the single-class rows, and the fulfillment labels include an unvalidated heuristic. Those limitations matter throughout the interpretation.

The headline results

Source	Primary accuracy	Calibration error	Auto-accepted at a bounded 95% precision target	Fulfillment accuracy
Jev	91.9%	0.049	86.5%	65.0%
Qwen3.5-35B-A3B	89.6%	n/a: three confidence levels	not reached	71.0%
Laya 421M	78.0%	0.322	not reached	45.7%

Jev beat the much larger LLM by 2.3 percentage points on the primary class. Laya's shipped defaults trailed by 13.9 points.

Yet the biggest difference was not those two points between Jev and Qwen. It was calibration.

Confidence changed the winner

Jev's expected calibration error was 0.049. At a confidence cutoff of 0.94, it accepted 641 of 741 gold rows—86.5% coverage with 96.7% observed precision. The Wilson 95% lower bound remained at or above the 95% target along the cutoff envelope.

Qwen reported three prompt-defined confidence levels. Its “high” bucket covered 97.8% of rows, but precision was 90.1%, so no bucket reached the 95% target.

Laya produced 705 distinct confidence values on 741 rows, but those scores were not ranked well enough to reach the target at any useful cutoff.

That distinction is operationally enormous. A classifier that is slightly more accurate but badly calibrated still leaves you without a defensible automation boundary.

Nine prompt variants moved only five answers

I tested nine Jev variants across:

Twelve independent presence questions
One 12-way subclass choice
Hierarchical subclass choices
Title and description only
Added structured line items
An added 1,500-character attachment excerpt

All nine landed between 91.2% and 91.9% on the same 741 rows—676 to 681 correct.

The winner, A-S2, was chosen using a rule written before the experiments: accuracy first, then coverage at the bounded precision cutoff, then the cheaper input.

The practical lesson was unexpected: elaborate question structure and a small attachment excerpt did not measurably improve this top-level task. The question bundle itself consumed most of the tokens, and the tested attachment excerpt existed on only 5.6% of rows.

That is not proof that attachments never help. It is evidence that this particular excerpt strategy did not.

One queueing rule changed automation from 26.6% to 91.9%

The original design sent every row with any raised flag to human review. One flag—brand-name-only—fired on more than half the dataset.

In the gate simulation:

Policy	Auto-classified	Primary precision on scored accepted rows
Any flag forces review	26.6%	93.5%
Record flags as attributes	91.9%	93.8%

The stricter rule removed almost two thirds of the volume from automation and bought no measurable precision. When flags became attributes instead of automatic blockers, the remaining review queue was dominated by genuine Jev–Qwen disagreements.

This was one of the most useful results in the study: workflow policy can matter more than another point of model accuracy.

Fulfillment mode remained hard for everyone

Qwen led fulfillment-mode accuracy at 71.0%, followed by Jev at 65.0% and Laya at 45.7%.

That still fell far short of the operational target. Configured-build precision ranged from 19% to 36%, and the mixed category was effectively unsolved.

The likely reason is structural: the distinction between distributor SKUs and an OEM-configured build often lives inside a bill of materials, configurator quote, or spreadsheet attachment—not in the notice text.

There is a second uncertainty: the fulfillment gold itself uses an unvalidated “eight or more lines from one OEM” heuristic. Human validation is the next required experiment.

What happened with open-weight Laya?

Laya's 78.0% result describes the shipped defaults on this task, not the model's ceiling.

It also did several things well:

Zero runtime errors across 12,927 rows
Deterministic output
The same request schema as Jev
Stable single-row latency around 300 ms
93% Hardware precision
Near-Jev accuracy on the short, hardware-heavy 2GIT slice

The failure analysis produced concrete hypotheses rather than a vague “small model loses” conclusion.

Option definitions were silently truncated

Laya's sequence packer reserves a 192-token question head. The primary-class question had seven options with definitions written to separate hardware, software, support, and services.

With seven options, each definition was cut to 25 tokens, and the instruction could shrink to as few as eight tokens. The text designed to prevent Hardware-versus-Software confusion was exactly what got truncated.

Laya called 105 of 572 Hardware rows “Software.” Jev made that error seven times; Qwen made it 20 times.

Accuracy dropped before the context window filled

Laya reached 81.5% accuracy on states below 600 characters but only 65.8% from 600 to 1,200 characters—roughly 150 to 300 tokens and still inside its 512-token window.

Jev and Qwen did not show the same decline. That means simple end-of-window truncation cannot explain the whole effect.

Presence questions over-fired

Laya emitted an average of 4.05 components per row, compared with 1.95 for Jev and 1.16 for Qwen. Its RFI and text-insufficient flags fired on 63% and 71% of rows, while the discovery analysis found true RFI rates around 2–3%.

The public repository lists six follow-up experiments: increase the head budget, warn on truncation, try the typed-decision and multilingual checkpoints, fit per-question thresholds, split the primary question from the rest, and record actual token counts.

These are testable engineering questions—and contributions are welcome.

Cost and operational behavior

For all 12,000 opportunities:

Source	Cost or compute	Observed operation
Jev	$0.78 at list input-token price	p50 185 ms, p95 273 ms, zero errors
Qwen3.5-35B	About 80 minutes of one shared GPU	2.7 rows/s, 69 malformed responses
Laya 421M	Local RTX 2000 Ada laptop	p50 299 ms, p95 576 ms, zero errors

These are different deployment products, so the table is not a simple price race. Jev includes hosted operations. Qwen used an existing on-prem cluster. Laya ran unbatched at one row at a time; its separate CUDA study shows what a batched server can do.

The right architecture depends on volume, infrastructure, latency, privacy, and whether calibrated probabilities are part of the product requirement.

Agreement tells us where humans add value

Jev and Qwen agreed on 91.0% of 11,931 paired rows.

The 1,068 disagreements clustered around Hardware versus Other, Hardware versus Software, Software versus Maintenance & Support, and Services versus Software or Other.

Instead of reviewing random rows, those disagreements form a much more valuable blind-adjudication pool. The next planned step is human review of a stratified sample and 300 blinded Jev–Qwen disagreements.

Limitations I would not hide

Quote gold measures agreement with what reps quoted, not objective truth.
Hardware dominates the evaluation set; Services has only six rows and Maintenance & Support has 32.
Fulfillment gold contains an unvalidated heuristic.
Subclass, lifecycle, and domain answers do not yet have human gold.
Laya ran single-row with default configuration, no threshold tuning, and one checkpoint.
Qwen's three-valued confidence was imposed by the prompt.
The 69 malformed Qwen rows are excluded from paired metrics and are not random.
The study covers one organization, one domain, one Jev version, and one measurement period.

The results are useful because the boundaries are explicit—not because the benchmark pretends to be universal.

Reproduce or challenge it

Everything publishable is open:

Repository: bhushankinge/jev-laya-classification-bench
Full report: classification experiment report
Aggregate metrics: results/e2-full
Experiment runbook: pipeline/RUN.md
Companion CUDA study: laya-cuda-bench

The repository includes 52 passing tests, deterministic figure generation, a citation file, and the pre-registered selection rule.

If you build production classifiers, I would be especially interested in your answer to this:

Would you optimize for the highest raw accuracy—or for the largest slice of traffic that can be automated at a defensible precision bound?

How Fast Can a 421M-Parameter Decision Model Run? I Benchmarked Laya Across NVIDIA GPUs

Bhushan Kinge — Thu, 24 Sep 2026 19:15:49 +0000

One H100 NVL. A 421M-parameter decision model. 15.1 million decisions per day while staying inside a p99 ≤ 130 ms latency budget.

That number sounds impressive—but raw throughput is the easy number to publish. The useful question is harder:

How many typed decisions can one GPU sustain when tail latency, correctness, and cost all matter?

I built an independent, fully reproducible benchmark to answer that question for Laya, Convai's open-weights “System 1” decision models.

Disclosure: This is an independent benchmark, not an official Laya or Convai result. The harness, workload, raw per-request data, environment captures, notebook, and figures are all public.

What I tested

Laya turns a state plus a typed question into a decision. Instead of generating prose, an application can ask which team should own an opportunity, how time-critical it is, or whether it needs human review.

The benchmark used a frozen sample of 1,000 public SAM.gov contract-opportunity notices. Every request asked the same three questions—choice, score, and yes/no—so throughput is reported as decisions per second rather than requests per second.

I tested:

RTX 2000 Ada laptop, 8 GB
RTX PRO 5000 Blackwell laptop, 24 GB
RTX PRO 6000 Blackwell workstation, 96 GB
H100 NVL, 94 GB—whole and split into seven 1g.12gb MIG instances
PyTorch eager FP32, FP16, and BF16
torch.compile with max-autotune
ONNX Runtime CUDA and TensorRT FP16
The hosted Jev API and two Qwen3.5 vLLM configurations as context baselines

The two service-level objectives were:

p99 ≤ 50 ms for an inline guardrail-style budget
p99 ≤ 130 ms for a less aggressive server budget

The load generator used open-loop Poisson arrivals against a small dynamic-batching HTTP server. A point counted only if it achieved at least 90% of the offered rate, stayed under the latency SLO, returned no errors, and did not show a growing queue.

The headline capacity results

GPU and serving backend	p99 ≤ 50 ms	p99 ≤ 130 ms
RTX PRO 5000, TensorRT FP16	15 decisions/s	42 decisions/s
RTX PRO 6000, TensorRT FP16	not measured	146 decisions/s
H100 NVL, TensorRT FP16	105 decisions/s	175 decisions/s
H100, 7 × MIG 1g.12gb, eager FP16	not met	not met reliably

At the 130 ms SLO, those sustained rates translate to:

3.6M decisions/day on the RTX PRO 5000
12.6M decisions/day on the RTX PRO 6000
15.1M decisions/day on the H100 NVL

The H100 result was 175 decisions/s with a measured p99 of about 91 ms at the selected operating point—not a theoretical offline ceiling.

Five findings that changed how I think about serving small models

1. The latency knee is sharp

A GPU can look healthy right until it does not. One load step beyond sustainable capacity caused the batcher to fill, long-document batches to become expensive, and p99 latency to jump from roughly 100 ms into seconds.

The operational lesson is simple: size below the knee, not at “100% GPU.” Average latency hides this behavior; p99 exposes it.

2. TensorRT was not the universal winner

For fixed-shape throughput, torch.compile(mode=max-autotune) FP16 was fastest on every card at larger batches—about 1.3–1.7× eager FP16.

Dynamic serving told a more nuanced story:

On Hopper, TensorRT raised H100 capacity at p99 ≤ 130 ms from 93 to 175 decisions/s.
On the RTX PRO 6000 Blackwell, eager FP16 and TensorRT both reached 146 decisions/s.
On the Blackwell laptop's multilingual checkpoint, eager FP16 beat TensorRT under the same SLO.

Static microbenchmarks are useful, but they do not automatically predict the best dynamic-serving backend.

3. Seven MIG slices were worse than one whole H100 for this workload

Seven concurrent H100 1g.12gb slices provided excellent isolation—their measurements agreed within roughly 1 ms—but they were the wrong shape for 400+-token documents.

At the lowest tested aggregate load, seven slices reached about 49 decisions/s at p99 127 ms, but they missed the benchmark's achieved-rate gate. Higher loads missed the 130 ms SLO. The whole H100 served 175 decisions/s inside that budget.

MIG 1g may still be an excellent fit for short-prompt decision traffic. It simply was not the right fit here.

4. Faster means nothing if the answers change

Before timing a backend, I required it to reproduce the upstream FP32 answer on a parity set covering 16 cases and 63 typed questions.

Across four GPUs, three checkpoints, multiple precisions, and five backends, 74 of 74 backend-and-device rows passed all 63 questions. The test also checked public JSON equality, finite outputs, and steady-state allocator stability.

This is fidelity to upstream—not a claim of real-world task accuracy—but it prevents an optimization from quietly changing behavior.

5. A 10-million-decision day fit on one workstation GPU

I replayed a compressed 24-hour diurnal curve at each hour's real arrival rate on one RTX PRO 6000. It completed 138,863 requests—10 million individual decisions—with zero errors and an overall p99 of 111 ms.

Two peak-hour segments reached 132 and 143 ms. A deployment that must remain below 130 ms continuously would therefore need roughly 25% headroom at that volume.

What did it cost?

Using three-year card amortization, 100% utilization, and electricity at $0.12/kWh, self-hosted Laya cost approximately:

$0.67 per million decisions on the RTX PRO 5000
$0.66 per million decisions on the RTX PRO 6000
$1.86 per million decisions on the H100 NVL

The hosted Jev API came to roughly $6.8–8.2 per million decisions at its list token price for this workload. That comparison needs context: the Jev measurement includes the public network path from Arizona, and a hosted API removes hardware and operational work. At lower volumes—roughly below one million decisions per day in this cost model—the API can still be the economically sensible option.

The Qwen3.5 LLM baselines reached about 190–200 decisions/s per card, but their p99 latency was around one second, so they did not meet either serving SLO. These were throughput and latency comparisons only; I make no accuracy claim for the LLM baselines.

The limitations matter

This study deliberately does not claim universal latency or model quality:

It uses one English federal-procurement workload.
Parity proves fidelity to upstream outputs, not correctness of those outputs.
Server sweeps and replays used one run per configuration.
The RTX PRO 6000 sweep did not test below 50 requests/s, so its 50 ms capacity is unknown.
torch.compile was measured for fixed shapes but was not tested as a serving backend.
The serving implementation is a compact asyncio dynamic batcher over loopback, not Triton.
Jev latency includes the public internet; self-hosted Laya latency does not.

Every caveat, including failed experiments and follow-up work, is documented in the full report.

Everything is open

You can inspect or reproduce the entire study:

Repository: bhushankinge/laya-cuda-bench
Full report: REPORT.md
Harness and raw results: immutable benchmark snapshot
Upstream documentation contribution: Laya PR #429
Upstream project: NandhaKishorM/laya

If you work on GPU inference, dynamic batching, TensorRT, MIG, or decision models, I would love to see you reproduce the benchmark on another card or challenge the methodology. The most valuable next data points are L4/A10G-class GPUs, torch.compile under dynamic serving, and a production Triton implementation.

What GPU or serving backend should I test next?