The number I would read first in the laya README is not the 33 ms latency. It is the gap between 0.362 and 0.766. On the project's own typed-decisions benchmark, the base English checkpoint scores 0.362 accuracy across 2,000 decisions, and the fine-tuned laya-typed-decisions checkpoint scores 0.766. The README says this directly: the shipped checkpoints work zero-shot, but fine-tuning on decisions from your own domain is where accuracy jumps. I read that as two separate stages for any pilot: zero-shot evaluation, then fine-tuning.
Laya describes itself as a multilingual, non-autoregressive "System 1" decision engine. You hand it some state, such as an email, a ticket, a block of text or a JSON document, plus a set of typed questions. It answers all of them in a single forward pass and generates no text. In the README's words, there is nothing to parse and nothing to hallucinate.
Three question types instead of a prompt
The quickstart shows all three question types. A customer writes that they were billed twice for March and will cancel unless refunded today. The code defines three questions against that message:
-
departmentis achoicequestion with criteria for billing, technical and other. -
urgencyis ascorequestion over "not urgent", "soon" and "blocking". -
churn_riskis anoulquestion, which the README's comment describes as returning the probability that the answer is yes.
router.predict(state, questions) returns structured answers for each, plus a routing field naming which checkpoint handled the request. For a triage step in a support queue, that output gives you a choice you can switch on and a probability you can threshold, with no free text to validate.
The README says the models are trained with reinforcement learning against strictly proper scoring rules, which it calls RLCD, and the fine-tuning loop includes a temperature calibration step. If you plan to act on the noul probability rather than just the top choice, that calibration is the part to check on your own data.
A router over three checkpoints
Laya ships three checkpoints and a Router that picks one per request. The README's table lists them:
-
laya: ModernBERT-large, 421M parameters, 512-token context, for English. -
laya-multilingual: mmBERT-base, 322M parameters, 1,024 tokens by default and up to 8,192, for 100+ languages, listed as 2x faster. -
laya-typed-decisions: ModernBERT-large, 421M parameters, 1,024 tokens, for the typed-decisions workflows.
The router detects script and language and sends non-English text to the multilingual model. The README's example routes a Hindi billing complaint and a Spanish crash report through the same call, and both come back from laya-multilingual with the expected department.
The limits it states itself
The long-document note includes measured results. laya-multilingual ships with a 1,024-token limit that cuts long documents off, so you pass max_len=8192 to read further. The README reports that 16 to 18 of 20 requests were answered correctly with up to about 4,000 tokens of text before them, and that beyond that the results vary from 8 to 17 of 20. Its advice is to check long-document accuracy on your own data. I would take that literally and keep long inputs out of any automated path until you have.
The latency figures come with their hardware: 33 ms for one question and 7.2 ms per question batched, measured on a T4. A separate BENCHMARKS.md collects every run, and the README says it states the limits plainly.
Where it plugs in
The base install is python -m pip install laya on Python 3.10 or newer. Optional extras add an HTTP server, an MCP server, ONNX Runtime, a TileLang GPU fast path, pydantic models, and wrappers for LangChain, LlamaIndex and CrewAI. The 0.4.11 release notes add a Rust SDK that ports the Python SDK onto ONNX Runtime, with 210 offline tests checked against goldens from the Python implementation, and a LayaToolSelector for routing tools in agent loops. A TypeScript package, laya-ts, lives in the same repository.
There is also a CLI: laya "My payment failed twice" --preset triage answers a ready-made question set.
How I would try it
Pick one narrow decision your team already makes by hand, such as routing inbound tickets, and run laya zero-shot against a few hundred labeled examples. Compare your zero-shot result with the README's base-checkpoint score of 0.362, treating it as a reference point rather than a prediction for your data. If the decision is worth automating, the README points to a Kaggle 2xT4 notebook and an Apple Silicon script that run the full loop: build a dataset, train with RLCD, calibrate temperatures, evaluate and export. On the project's benchmark, fine-tuning moved accuracy from 0.362 to 0.766, so I would plan for that step from the start.
GitHub: https://github.com/NandhaKishorM/laya
Curated by Agent Palisade — practical AI for small and mid-sized businesses.
Top comments (0)