Research scope: This study combines one benchmark we ran on November 2, 2026 (100 Banking77 test messages, 10 intents, six models, one request each) with official TypeSafe documentation read the same day and official list prices verified October 5, 2026. Costs are list prices multiplied by the tokens each API reported; they are not Tokely invoices.
Short answer: Jev 1.14.0 classified 97 of 100 Banking77 messages correctly, landing between Claude Haiku 5.5 at 98 and Gemini 2.5 Flash-Lite at 96. Its median response time was 0.62 s, compared to 1.19 s to 3.41 s across four small LLMs. At official list prices, 1M requests cost $19.78 on Jev, which sits below three of the four LLMs and runs 1.1x the cost of GPT-4.1 nano at $17.78.
What is Jev?
Jev is a System One decision model from TypeSafe AI, an AI lab based in San Francisco. Instead of generating freeform text, Jev answers typed questions about a piece of content designated as the state and returns probabilities. TypeSafe introduced System One models and Jev on September 15, 2026, in a blog post by founder Diogo Almeida. The name System One references Daniel Kahneman's description of fast, intuitive System 1 human cognition, while Jev takes its name from the economist William Stanley Jevons.
TypeSafe says Jev is trained using Reinforcement Learning for Calibrated Decisions, an approach it calls RLCD. This training target aims at output probabilities where higher confidence directly matches higher observed accuracy. Two versions of the model are currently active: jev-1.13.0 and jev-1.14.0, which became available on October 14, 2026. The alias jev-latest routes directly to jev-1.14.0.
Calls to Jev support four question types. A noul question produces the probability of a yes or true outcome, expressed on a continuous scale from 0 to 1. A choice question selects one option from up to 255 named candidates, accompanied by individual probabilities and an overall confidence score. A score question places the input onto an ordered scale of up to 10 distinct levels, while a box question places a bounding box over an image, a capability restricted to jev-1.14.0 and marked experimental by TypeSafe.
API consumers interact with Jev via POST /v1/systemone using a JSON payload containing model, state, and questions. The questions field accepts an object keyed by user-defined identifiers. Requests accept standard limits of 64k tokens per request and 32k tokens per question, expanding to 262k tokens when callers provide the TypeSafe-Danger-Zone-Long-Context header. TypeSafe prices Jev at $0.042 per 1M input tokens, which equals $42 per billion tokens, while billed output tokens remain free.
What did we measure?
Our benchmark evaluated 100 customer support messages drawn from the Banking77 test set, published by PolyAI under the CC BY 4.0 license. We sampled 10 specific intents at random using random seed 77, selecting 10 individual messages for each intent. The selected intents include card linking, card payment with a wrong exchange rate, contactless not working, country support, disposable card limits, exchange charge, fiat currency support, getting a virtual card, top-up limits, and Visa or Mastercard. Several pairs represent semantically adjacent issues, most visibly exchange charge alongside card payment with a wrong exchange rate.
Every test message was dispatched once across six distinct model configurations through the Tokely API on November 2, 2026. We evaluated Jev 1.14.0 and Jev 1.13.0 using a single choice question configured with the 10 intents as its evaluation criteria. Alongside Jev, we submitted the identical messages to four low-cost LLMs: Claude Haiku 5.5, GPT-6 Luna, Gemini 2.5 Flash-Lite, and GPT-4.1 nano. Each LLM received a system prompt enumerating the same 10 intents, set to return a structured JSON object at temperature 0 with a cap of 400 output tokens.
The complete run yielded 600 individual requests, every one of which resolved successfully with an HTTP 200 status code. Calls executed sequentially, originating from a Tokely server in Finland. Latencies are full round trips measured from that server. Cost calculations apply developer list prices against the exact token volumes recorded by each respective provider API.
Finding 1: Jev answered in 0.62 seconds, 1.9x faster than the fastest LLM
End-to-end response time per request, median and 90th percentile, 100 Banking77 messages per model, measured November 2, 2026 from a Tokely server in Finland.
Response latencies clustered into two distinct performance tiers across the six evaluated engines. Median response times registered at 0.60 s for Jev 1.13.0, 0.62 s for Jev 1.14.0, 1.19 s for GPT-4.1 nano, 1.58 s for Gemini 2.5 Flash-Lite, 1.90 s for GPT-6 Luna, and 3.41 s for Claude Haiku 5.5. Jev 1.14.0 cleared the benchmark 1.9x faster than GPT-4.1 nano and 5.5x faster than Claude Haiku 5.5 at the median.
Tail latencies separated the two groups further. At the 90th percentile, Jev 1.14.0 clocked 0.74 s and Jev 1.13.0 recorded 0.80 s, while GPT-4.1 nano reached 1.57 s, GPT-6 Luna hit 4.30 s, Claude Haiku 5.5 reached 6.79 s, and Gemini 2.5 Flash-Lite extended to 8.93 s. Nine in ten Jev requests finished within 0.80 s, and the slowest took 1.63 s, while the LLM tails stretched from 1.57 to 8.93 seconds.
TypeSafe quotes baseline end-to-end latencies between 70 ms and 500 ms for its own internal measurements. TypeSafe says its published evaluations are generally run from the US West Coast. Our recorded latencies include the network time from our server in Finland.
GPT-6 Luna is a reasoning model: on average, 27.6 of its 43.9 output tokens per request were reasoning tokens.
Finding 2: 97% accuracy, between Claude Haiku 5.5 and Gemini 2.5 Flash-Lite
Share of 100 Banking77 test messages (10 intents) where each model returned the labelled intent, November 2, 2026.
Accuracy remained high across five of the six evaluated endpoints. Claude Haiku 5.5 achieved top accuracy with 98 correct intents out of 100, followed immediately by Jev 1.14.0 and Jev 1.13.0 at 97 correct answers each. Gemini 2.5 Flash-Lite followed closely with 96 correct classifications, GPT-6 Luna reached 94, and GPT-4.1 nano fell behind with 86.
Jev 1.14.0 produced three misclassifications across the 100 benchmark items. The model assigned a fiat currency support question to top-up limits, routed a card linking message into getting a virtual card, and placed a contactless payment problem under card linking. GPT-4.1 nano struggled primarily on fine distinctions between foreign currency fees, sorting 5 of the 10 wrong-exchange-rate queries into the broader exchange charge bucket.
Model output formatting introduced edge cases on generative endpoints that did not appear with Jev. GPT-6 Luna failed an exact match check on one request by returning "getting virtual card" with spaces instead of the exact label from the prompt. By contrast, a Jev choice response maps strictly to one of the predefined criteria keys supplied in the payload, eliminating schema parsing drift.
These accuracy differences should be interpreted within standard sampling margins. Across a target run of 100 individual evaluation samples, deviations of one or two correct answers fall fully within expected statistical noise. Both Jev models demonstrated intent discrimination capabilities equivalent to current small LLMs.
Finding 3: The free-output trade
List price of 1M requests at the mean input and output tokens each API reported in our run. Jev output tokens are free.
We define "the free-output trade" as the economic dynamic observed in our benchmark where Jev billed 1.8x to 3.3x more input tokens than the LLMs for the same message, but returned its answer with zero billed output tokens. This trade-off dictates whether Jev or a minimal LLM yields lower costs for a given classification task.
Average token consumption varied sharply by architecture on the Banking77 workload. Jev logged an average of 471.1 input tokens and 118.8 output tokens per query, though none of its output tokens accrued billing charges under TypeSafe's pricing rules. Among the LLMs, Claude Haiku 5.5 averaged 232.5 input tokens and 45.6 output tokens, GPT-6 Luna consumed 143.8 input tokens and 43.9 output tokens, Gemini 2.5 Flash-Lite processed 268.4 input tokens and 15.0 output tokens, and GPT-4.1 nano required 144.8 input tokens alongside 8.2 output tokens. Jev and the LLMs received the same message and the same 10 intents; the difference is in how each API counts its tokens.
At official developer list prices, handling 1M such classification queries totals $19.78 on Jev. GPT-4.1 nano came in slightly lower at $17.78 per 1M requests, representing 0.90x the expense of Jev. The remaining three models cost substantially more: Gemini 2.5 Flash-Lite totaled $32.84 (1.66x Jev), GPT-6 Luna totaled $36.32 (1.84x Jev), and Claude Haiku 5.5 accumulated $46.03 (2.33x Jev).
Output token rates exert a major influence on LLM expense profiles. Billed output tokens represented 49.5% of total spend for Claude Haiku 5.5 and 60.4% of spend for GPT-6 Luna, while settling between 18% and 19% for Gemini 2.5 Flash-Lite and GPT-4.1 nano. Through Tokely, Jev input tokens list at $0.0294 per 1M tokens, a 30% reduction from TypeSafe's direct list price, bringing the total expense for 1M identical requests down to $13.85.
Finding 4: At TypeSafe's own example, the gap is wider
List price of 1M decisions at 428 input and 73 output tokens, the usage of TypeSafe's quickstart request. Official list prices; GPT-6 Astra and Claude Fable 5.1 ($7,930) are left out of the chart.
The pricing variance between Jev and general-purpose LLMs expands when evaluating TypeSafe's reference workload. In the quickstart example published in TypeSafe's developer documentation, a single text input evaluated against three questions consumes 428 input tokens and generates 73 output tokens. Calculating pricing under the assumption that LLMs process those exact token quantities highlights the extreme ends of provider pricing scales.
Under that matched-token assumption, executing 1M decisions costs $17.98 on Jev, compared to $25.18 on Qwen-Flash at 1.4x the cost. GPT-5 nano reaches $50.60 (2.8x), while Gemini 2.5 Flash-Lite and GPT-4.1 nano both reach $72.00 (4.0x). Claude Haiku 5.5 and GPT-6 Luna both land at $79.30 (4.4x), followed by GPT-4o mini and DeepSeek V4.1 Flash at $108.00 (6.0x), GPT-5.6 Luna at $173.20 (9.6x), Gemini 3.1 Flash-Lite at $216.50 (12.0x), and GPT-5 mini at $253.00 (14.1x).
At the top of the provider spectrum, GPT-6 Astra and Claude Fable 5.1 would bill $7,930 for that same 1M request volume, or 441x the cost of Jev. This is close to the "444.6x cheaper" figure that TypeSafe shows on its home page.
Our empirical benchmark explains why that theoretical 441x figure functions primarily as an upper bound. In our benchmark, the small LLMs used fewer input tokens than Jev for the same message, and the measured cost of the small LLMs ranged from 0.9x to 2.3x Jev.
Finding 5: Confidence tells you which answers to trust
Jev 1.14.0 on 100 Banking77 messages: share of messages at or above each confidence threshold and accuracy on those messages.
Jev returns an explicit confidence metric ranging from 0 to 1 with every choice response. The value reaches 1 when the model assigns all probability density to a single candidate, and drops toward 0 when probability distributes uniformly across the available options. This property allows developers to construct automated routing rules based on numerical certainty.
In our Banking77 evaluation, Jev 1.14.0 produced 91 classifications with a confidence score of 0.9 or greater. Out of those 91 high-confidence predictions, 90 were correct, establishing a 98.9% accuracy rate for that segment. The model issued 7 predictions with confidence values between 0.5 and 0.9, yielding 5 correct classifications and 2 errors.
Setting an automated execution threshold at 0.7 confidence would have approved 94% of incoming customer messages while maintaining a 98.9% accuracy mark on the automated volume. The remaining 6 ambiguous items would have been flagged for secondary evaluation. TypeSafe's documentation suggests initial baseline thresholds of 0.7 for yes outcomes and 0.3 for no outcomes on noul questions, and advises testing thresholds on your own data. Across all 100 test items in our benchmark, Jev 1.13.0 and Jev 1.14.0 selected the identical winning intent label every time.
How do you call Jev through Tokely?
Tokely provides access to jev-1.14.0 and jev-1.13.0 via the standard POST /v1/systemone schema at https://api.tokely.me/v1/systemone using a single Tokely API key. Tokely accepts text states with noul, choice and score questions, up to 64 per request. Developers can bundle multiple queries into a single payload, a technique that TypeSafe's developer cookbook reports was 12.2x cheaper and 10.0x faster than issuing separate API requests in its internal testing.
curl https://api.tokely.me/v1/systemone \
-H "Authorization: Bearer $TOKELY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "jev-1.14.0",
"state": "Do you guys accept mastercard or visa?",
"questions": {
"intent": {
"type": "choice",
"instructions": "Which intent does this bank customer message express?",
"criteria": {
"visa_or_mastercard": "visa or mastercard",
"card_linking": "card linking",
"top_up_limits": "top up limits"
}
},
"is_urgent": {"type": "noul", "instructions": "The message conveys urgency"}
}
}'
A common mistake in unofficial guides is placing fields named question, options, or scale inside a question. The official TypeSafe API schema requires instructions, alongside a criteria field for choice and score evaluations. For choice questions, criteria must be formatted as an object mapping string identifiers to detailed descriptions, while score questions require criteria formatted as an ordered list of level definitions.
What should teams do with this?
Engineering teams should deploy Jev when the task requires discrete categorization, binary filtering, or rating scales, and where request latency affects production workflows. Routing pipelines, user intent classification, feature gating, and content moderation fit this profile directly. In our benchmark, Jev matched the accuracy of small LLMs while operating at roughly half or less of their median response times, all while returning strictly conforming structured objects.
Development teams should use the returned confidence scores to build layered fallback pipelines. Requests exceeding an empirically validated confidence cutoff can process without human intervention, while ambiguous items route automatically to human review queues or larger generative models. Calibrating this boundary on task-specific test sets allows systems to maintain high classification standards on production traffic.
Architects should avoid assuming Jev is universally the cheapest alternative across every possible classification workload. Jev tallies billable input tokens for every option description contained in its criteria definitions, meaning tasks with dozens of lengthy definitions can cost as much as lightweight LLMs like GPT-4.1 nano. Engineering teams should run cost audits using their own production prompt distributions before committing to a single architecture. Tokely provides a single unified API key to access Jev, Claude, GPT, and Gemini, enabling direct head-to-head testing across models: Jev 1.14.0 on Tokely.
Frequently asked questions
Is Jev an LLM? No: TypeSafe says System One models do not write replies, produce code or explain their reasoning; they return structured answers.
Can I call Jev with the OpenAI SDK? No, /v1/systemone is a different format; use plain HTTP, TypeSafe's Python or TypeScript SDK, or LangChain4j.
How much does Jev cost? $0.042 per 1M input tokens at TypeSafe, $0.0294 on Tokely, output free.
Which version should I use? jev-1.14.0 is the newest and the target of jev-latest; TypeSafe says it gives 60 days' notice before jev-1.13.0 is deprecated; pin a version in production.
Does Jev accept images? TypeSafe offers images and the box type as experimental features behind the TypeSafe-Danger-Zone-Images header; through Tokely, Jev takes text with noul, choice and score questions.
Methodology and limitations
This benchmark evaluated 100 messages spanning 10 intents drawn from a single English-language public dataset. Adjusting the task context, target language, prompt phrasing, or option counts could alter comparative rankings, and performance gaps of one or two correct answers fall within statistical noise. All tests executed as sequential, single-request calls on November 2, 2026, originating from a Tokely test host in Finland routing through the Tokely gateway engine. Five LLM requests experienced retries due to host CPU saturation on our test server, with reported latencies reflecting the timing of the final successful transaction.
LLM prompts contained the identical list of 10 intent descriptions embedded in a system instruction, evaluated at temperature 0 with a 400 output token limit, without prompt engineering or fine-tuning for any specific endpoint. Cost numbers reflect official list prices multiplied by the token counts reported in each API response, excluding provider volume discounts, prompt caching tiers, batch discounts, and localized taxes unless specifically noted. The quickstart comparison assumes LLMs consume token counts equivalent to Jev's format, which our empirical data demonstrates is not the case in practice. TypeSafe's latency and cost claims come from its own evaluations.
Data and citation
Benchmark datasets and cost breakdown sheets are freely available for external verification: per-model benchmark results can be accessed at jev-benchmark-2026-11.csv, while the list-price comparison across quickstart workloads is published at decision-cost-2026-11.csv. Primary documentation sources include the TypeSafe documentation, the official launch article Introducing System One Models and Jev, and the Banking77 dataset.
Cite as: Tokely (2026). Jev by TypeSafe vs Small LLMs: Accuracy, Speed and Cost per Decision (2026 Benchmark). https://tokely.me/blog/jev-typesafe-benchmark-2026
Originally published at tokely.me.
Top comments (0)