67.8% directional accuracy. 339 decisions. Net result after fees: -62.69.
That's the outcome from Waxmell114514/jev-trade, an open-source project that wired Jev directly into NQ futures order book data and traded every decision. A second project, egrm07/jev_bitcoin_backtest, tested ten different data representations on BTC/USD 5-minute bars and ran a five-gate validation framework. Best holdout AUC after 65 days out-of-sample: 0.503 — statistically indistinguishable from random.
Both projects lost money. Neither result is a verdict on Jev's calibration quality. They're a diagnosis of where Jev is being placed in the system.
This post covers three things: what Jev actually does under the hood, why high hit rate doesn't prevent losses at the execution layer, and a specific data infrastructure problem that applies to any AI decision layer — not just Jev.
What Jev Is (and Is Not)
Jev, built by TypeSafe, skips something all standard LLMs do as a matter of course: it never generates text token by token to produce a structured answer.
Standard LLM classification workflow:
- Process prompt → build KV cache
- Generate
{, then", thens, thene... each token requires moving the cache through GPU memory - Parse the output string → extract
"positive"
A three-way classification decision means roughly 12 tokens of autoregressive generation. Each token is a full forward pass, and the bottleneck isn't raw compute — it's memory bandwidth. Moving the KV cache back and forth is the slow part.
Jev's approach: one forward pass, shared KV cache across all batched questions, probabilities read directly from a numeric output head. The answer options aren't text the model writes out — they're dimensions in the computation graph.
Three question types:
| Type | How it works | Returns |
|---|---|---|
| Multiple choice | Pick from predefined options | option + probability + confidence |
| Scoring | Evaluate against a rubric | score + probability + confidence |
| Judgment | Is this statement true or false | 0–1 probability |
No format validation. No JSON parsing retries. No structured output overhead.
One architectural note worth understanding: Jev's option probabilities are not independent. Add an irrelevant option and the others shift. This is the behavior of a classifier doing joint computation over the full option set — not a model scoring each option in isolation. It maps inputs to a predefined decision space in a single pass, more like a discriminative classifier than a generative model reasoning through an answer.
The Calibration Story: What RLCD Actually Claims
TypeSafe trained Jev with RLCD — Reinforcement Learning for Calibrated Decisions. The stated goal: when the model outputs 70% confidence, the event should actually occur roughly 70% of the time.
This matters more than it sounds, and the difference from standard RLHF is specific:
| Standard Training (RLHF) | Calibration Training (RLCD) | |
|---|---|---|
| Optimization target | Human preference score | Calibration error |
| Signal | "Is this answer good?" | "Does 70% confidence → 70% correct?" |
| What the model learns | Produce outputs humans rate highly | Make probability statements accurate |
A standard LLM can output confidence: 0.95 with no statistical basis — because high-confidence-sounding language correlates with human preference ratings. RLCD targets calibration error directly.
Independent test results:
| Source | Test | Calibration Error | Accuracy |
|---|---|---|---|
| webofmike | 60 tool-call risk cases | 0.0712 (latest) / 0.0505 (preview) | 91.7% |
| archerhume.com | MMLU 1,200 questions | 0.0313 ECE | 84.6% (MMLU-Pro) |
5–7 percentage points of calibration error is genuinely better than a vanilla LLM. The direction is right.
One caveat to state plainly: TypeSafe hasn't published the RLCD training methodology. The calibration results are independently measured, but the mechanism isn't verifiable. If you're using Jev's probabilities for position sizing or risk management, that uncertainty needs to be in your model.
The BTC/USD Backtest: Ten Representations, Zero Edge
egrm07/jev_bitcoin_backtest is methodologically careful. Five validation gates:
- Permutation testing
- Multiple comparison correction
- Holdout validation (out-of-sample)
- Sharpe ratio confidence intervals
- Buy-and-hold comparison
Dataset: Binance BTCUSDT 5-minute bars. Development window: March–July 2026. Out-of-sample holdout: July 15–September 19, 2026 (~65 days).
Ten data representations tested: raw OHLCV, percentage returns, technical indicators, text descriptions, ASCII charts, and combinations.
Results:
| Metric | Value |
|---|---|
| Holdout AUC (all 10 representations) | 0.471–0.503 |
| Best strategy return | -15.73% |
| BTC buy-and-hold (same period) | +25.55% |
| Total model API cost | $2.58 |
| Statistical significance | None found |
Random guessing has AUC 0.5. The range 0.471–0.503 straddles that floor.
The NQ Order Book: High Accuracy, Still Underwater
Waxmell114514/jev-trade used synthetic NQ order book data replayed against real Kraken data. 339 decisions, 67.8% directional accuracy.
| Metric | Value |
|---|---|
| Total decisions | 339 |
| Hit rate | 67.8% |
| Gross P&L | +28.90 |
| Fees | -91.60 |
| Net P&L | -62.69 |
| Break-even fee threshold | < 0.316 bps |
The break-even threshold — below 0.316 basis points — is lower than what any realistic trading venue offers at this decision frequency. High directional accuracy still loses because transaction costs are a fixed drag that doesn't care how right you are.
The structural problem: forcing a trade on every Jev decision turns a calibrated probability model into a randomized execution engine. Zerve.ai's 2026 quant research report states it directly:
LLMs don't generate alpha. They don't surface research directions that produce real signal.
A recent arXiv paper (2608.20304) goes further: after statistical calibration, LLM feature contributions collapse to zero. A near-zero-cost baseline (headline count) outperformed every LLM feature tested. The paper proposes a calibration viability checkpoint — verify that the LLM feature has real predictive power before building the inference pipeline.
Jev outputs a probability, not a strategy. The signal extraction and position sizing layer is still on you.
The Real Bottleneck: Market Data Infrastructure
Where does Jev belong in a quant system?
| Layer | Good fit | Bad fit |
|---|---|---|
| Information processing | News sentiment, earnings tone, regime classification, candidate screening | — |
| Signal execution | — | Per-tick direction calls → orders |
But there's a deeper problem the backtests don't surface.
Jev accepts text. Quant systems run on structured data: prices, volumes, session states, order book depth. Jev has no market data interface. It doesn't know whether the open price field is populated during pre-market hours. It doesn't know that US extended-hours data carries different field structures than regular session. It doesn't know that a data snapshot from 09:28 ET has different semantics than one from 09:35 ET.
If your pipeline feeds Jev a raw market snapshot without explicitly encoding session state, the probability you get back has no attributable data context.
The trade_session field: existence vs. value
Here's a concrete example. The US market trading sessions endpoint returns structurally different objects depending on which session is active:
import requests
resp = requests.get(
"https://api.tickdb.ai/v1/market/trading-sessions",
params={"market": "US"},
headers={"X-API-Key": "YOUR_KEY"}
)
# Response:
# {
# "market": "US",
# "trading_sessions": [
# {"begin_time": 400, "end_time": 930, "trade_session": 1}, # pre-market
# {"begin_time": 930, "end_time": 1600}, # regular — no trade_session field
# {"begin_time": 1600, "end_time": 2000, "trade_session": 2} # after-hours
# ]
# }
The regular trading session (09:30–16:00) has no trade_session field. Pre-market carries trade_session: 1. After-hours carries trade_session: 2.
| Session | trade_session field | Correct detection method |
|---|---|---|
| Pre-market 04:00–09:30 | = 1 | field exists AND value is 1 |
| Regular 09:30–16:00 | absent | field does NOT exist |
| After-hours 16:00–20:00 | = 2 | field exists AND value is 2 |
If you write if data.get("trade_session") == 0 to detect regular hours, your code raises a KeyError during regular session because the field isn't there. The correct check is field existence, not field value:
session_info = trading_sessions_response["trading_sessions"]
def get_current_session(current_time_hhmm: int, sessions: list) -> str:
for s in sessions:
if s["begin_time"] <= current_time_hhmm < s["end_time"]:
# Detect by field existence, not value
if "trade_session" not in s:
return "regular"
elif s["trade_session"] == 1:
return "pre_market"
elif s["trade_session"] == 2:
return "after_hours"
return "closed"
This isn't a quirk of one API. Every market has session-specific field structures. Hong Kong equities have a 60-minute lunch break with a data gap at noon. Futures roll dates change contract liquidity structure overnight. If your AI decision layer doesn't know which session the snapshot came from, it's reasoning from incomplete context.
Building the audit trail with timestamps
resp = requests.get(
"https://api.tickdb.ai/v1/market/kline",
params={"symbol": "AAPL", "interval": "1d", "limit": 3},
headers={"X-API-Key": "YOUR_KEY"}
)
# Each bar includes:
# {
# "time": 1789531200000, # Unix milliseconds — this is your audit trail
# "open": "332.53",
# "high": "335.48",
# "low": "330.70",
# "close": "332.41",
# "volume": "35981000"
# }
Each bar carries a Unix millisecond timestamp. When Jev returns a 72% probability on a direction call, there's a question you need to be able to answer: which bar, which session state, which fields were populated? If you can't trace the probability back to a specific, complete data state, the decision isn't auditable — and any retrospective backtest you run is disconnected from what actually happened.
This is not a Jev problem. It's a data infrastructure problem that lives upstream of any AI inference layer.
The Speed Numbers in Context
TypeSafe's headline: 193.6x faster, 444.6x cheaper than GPT-5.6 Terra.
| Source | Baseline | Speed | Cost |
|---|---|---|---|
| TypeSafe (self-reported) | GPT-5.6 Terra | 193.6x | 444.6x |
| PearPages (independent) | Equivalent intelligence baseline | ~25x | ~76x |
| Near Here (independent) | Mistral Small 4 | ~5x | ~8.6x |
| webofmike (measured p50) | — | 421.6ms | — |
5–25x faster depending on the comparison. TypeSafe's 193x is against the slowest, most expensive possible baseline. The measured p50 of 421ms is the operational number for planning purposes.
TypeSafe's stated end-to-end latency is 70–500ms; 421ms sits in the upper half of that range. For the information processing use case — batch news classification, earnings tone scoring, universe screening — this matters. For the execution layer, speed is irrelevant because the edge doesn't exist there.
If Jev Is in Your Stack: Three Things to Check
1. Where in the system? Information processing → reasonable. Per-tick execution with a forced trade on every decision → expect negative returns regardless of hit rate.
2. Is your data layer session-aware? Before feeding a market snapshot to Jev, encode session state explicitly in the text input: "09:42 ET, regular session, fields present: OHLCV, no pre/post-market quotes". Don't leave the model to infer what data was available.
3. Does your decision log have timestamps? Every Jev call on market data should log: which bar, which session, which fields were active. Without this, the probability output is not auditable, and any backtest is answering a different question than what happened in production.
The calibrated probability is real. The trading strategy still has to be built on top of it — with the full data infrastructure stack that entails.
Sources: TypeSafe Jev documentation; archerhume.com architecture reverse-engineering; APUS open-source replication report; webofmike 60-case tool-call risk benchmark; PearPages speed/cost analysis; Near Here 50-decision content moderation benchmark; GitHub egrm07/jev_bitcoin_backtest; GitHub Waxmell114514/jev-trade; Zerve.ai LLMs in Quant Research (2026); arXiv 2608.20304 LLM Calibration-Induced Degeneracy in Financial Forecasting; arXiv 2501.19047 Understanding Model Calibration; TickDB API (tested 2026-09-21). Free market data API access: tickdb.ai
Top comments (0)