How Fast Can a 421M-Parameter Decision Model Run? I Benchmarked Laya Across NVIDIA GPUs

One H100 NVL. A 421M-parameter decision model. 15.1 million decisions per day while staying inside a p99 ≤ 130 ms latency budget.

That number sounds impressive—but raw throughput is the easy number to publish. The useful question is harder:

How many typed decisions can one GPU sustain when tail latency, correctness, and cost all matter?

I built an independent, fully reproducible benchmark to answer that question for Laya, Convai's open-weights “System 1” decision models.

Disclosure: This is an independent benchmark, not an official Laya or Convai result. The harness, workload, raw per-request data, environment captures, notebook, and figures are all public.

What I tested

Laya turns a state plus a typed question into a decision. Instead of generating prose, an application can ask which team should own an opportunity, how time-critical it is, or whether it needs human review.

The benchmark used a frozen sample of 1,000 public SAM.gov contract-opportunity notices. Every request asked the same three questions—choice, score, and yes/no—so throughput is reported as decisions per second rather than requests per second.

I tested:

• RTX 2000 Ada laptop, 8 GB
• RTX PRO 5000 Blackwell laptop, 24 GB
• RTX PRO 6000 Blackwell workstation, 96 GB
• H100 NVL, 94 GB—whole and split into seven 1g.12gb MIG instances
• PyTorch eager FP32, FP16, and BF16
• torch.compile with max-autotune
• ONNX Runtime CUDA and TensorRT FP16
• The hosted Jev API and two Qwen3.5 vLLM configurations as context baselines

The two service-level objectives were:

• p99 ≤ 50 ms for an inline guardrail-style budget
• p99 ≤ 130 ms for a less aggressive server budget

The load generator used open-loop Poisson arrivals against a small dynamic-batching HTTP server. A point counted only if it achieved at least 90% of the offered rate, stayed under the latency SLO, returned no errors, and did not show a growing queue.

The headline capacity results

GPU and serving backend	p99 ≤ 50 ms	p99 ≤ 130 ms
RTX PRO 5000, TensorRT FP16	15 decisions/s	42 decisions/s
RTX PRO 6000, TensorRT FP16	not measured	146 decisions/s
H100 NVL, TensorRT FP16	105 decisions/s	175 decisions/s
H100, 7 × MIG 1g.12gb, eager FP16	not met	not met reliably

At the 130 ms SLO, those sustained rates translate to:

• 3.6M decisions/day on the RTX PRO 5000
• 12.6M decisions/day on the RTX PRO 6000
• 15.1M decisions/day on the H100 NVL

The H100 result was 175 decisions/s with a measured p99 of about 91 ms at the selected operating point—not a theoretical offline ceiling.

Five findings that changed how I think about serving small models

1. The latency knee is sharp

A GPU can look healthy right until it does not. One load step beyond sustainable capacity caused the batcher to fill, long-document batches to become expensive, and p99 latency to jump from roughly 100 ms into seconds.

The operational lesson is simple: size below the knee, not at “100% GPU.” Average latency hides this behavior; p99 exposes it.

2. TensorRT was not the universal winner

For fixed-shape throughput, torch.compile(mode=max-autotune) FP16 was fastest on every card at larger batches—about 1.3–1.7× eager FP16.

Dynamic serving told a more nuanced story:

• On Hopper, TensorRT raised H100 capacity at p99 ≤ 130 ms from 93 to 175 decisions/s.
• On the RTX PRO 6000 Blackwell, eager FP16 and TensorRT both reached 146 decisions/s.
• On the Blackwell laptop's multilingual checkpoint, eager FP16 beat TensorRT under the same SLO.

Static microbenchmarks are useful, but they do not automatically predict the best dynamic-serving backend.

3. Seven MIG slices were worse than one whole H100 for this workload

Seven concurrent H100 1g.12gb slices provided excellent isolation—their measurements agreed within roughly 1 ms—but they were the wrong shape for 400+-token documents.

At the lowest tested aggregate load, seven slices reached about 49 decisions/s at p99 127 ms, but they missed the benchmark's achieved-rate gate. Higher loads missed the 130 ms SLO. The whole H100 served 175 decisions/s inside that budget.

MIG 1g may still be an excellent fit for short-prompt decision traffic. It simply was not the right fit here.

4. Faster means nothing if the answers change

Before timing a backend, I required it to reproduce the upstream FP32 answer on a parity set covering 16 cases and 63 typed questions.

Across four GPUs, three checkpoints, multiple precisions, and five backends, 74 of 74 backend-and-device rows passed all 63 questions. The test also checked public JSON equality, finite outputs, and steady-state allocator stability.

This is fidelity to upstream—not a claim of real-world task accuracy—but it prevents an optimization from quietly changing behavior.

5. A 10-million-decision day fit on one workstation GPU

I replayed a compressed 24-hour diurnal curve at each hour's real arrival rate on one RTX PRO 6000. It completed 138,863 requests—10 million individual decisions—with zero errors and an overall p99 of 111 ms.

Two peak-hour segments reached 132 and 143 ms. A deployment that must remain below 130 ms continuously would therefore need roughly 25% headroom at that volume.

What did it cost?

Using three-year card amortization, 100% utilization, and electricity at $0.12/kWh, self-hosted Laya cost approximately:

• $0.67 per million decisions on the RTX PRO 5000
• $0.66 per million decisions on the RTX PRO 6000
• $1.86 per million decisions on the H100 NVL

The hosted Jev API came to roughly $6.8–8.2 per million decisions at its list token price for this workload. That comparison needs context: the Jev measurement includes the public network path from Arizona, and a hosted API removes hardware and operational work. At lower volumes—roughly below one million decisions per day in this cost model—the API can still be the economically sensible option.

The Qwen3.5 LLM baselines reached about 190–200 decisions/s per card, but their p99 latency was around one second, so they did not meet either serving SLO. These were throughput and latency comparisons only; I make no accuracy claim for the LLM baselines.

The limitations matter

This study deliberately does not claim universal latency or model quality:

• It uses one English federal-procurement workload.
• Parity proves fidelity to upstream outputs, not correctness of those outputs.
• Server sweeps and replays used one run per configuration.
• The RTX PRO 6000 sweep did not test below 50 requests/s, so its 50 ms capacity is unknown.
• torch.compile was measured for fixed shapes but was not tested as a serving backend.
• The serving implementation is a compact asyncio dynamic batcher over loopback, not Triton.
• Jev latency includes the public internet; self-hosted Laya latency does not.

Every caveat, including failed experiments and follow-up work, is documented in the full report.

Everything is open

You can inspect or reproduce the entire study:

• Repository: bhushankinge/laya-cuda-bench
• Full report: REPORT.md
• Harness and raw results: immutable benchmark snapshot
• Upstream documentation contribution: Laya PR #429
• Upstream project: NandhaKishorM/laya

If you work on GPU inference, dynamic batching, TensorRT, MIG, or decision models, I would love to see you reproduce the benchmark on another card or challenge the methodology. The most valuable next data points are L4/A10G-class GPUs, torch.compile under dynamic serving, and a production Triton implementation.

What GPU or serving backend should I test next?