JEV assisted LLM Trading

We gave our LLM trading bot an auditor that never sees the chart

Every hour, on the hour, a vision model looks at three forex charts and decides whether our bot should buy, sell, or sit out. It has been doing this with real money since May. As of last Saturday, something finally grades its reasoning.

This post is about the grader.

The bot, briefly

It's a Python script called gemini_snap_flash. Once an hour it sends a one-minute candlestick chart plus daily and weekly context images to Gemini, gets back UP or DOWN and a one-line thesis, and trades EURUSD, GBPUSD, or USDJPY through MT5 on funded prop accounts. One position per pair, fixed stop and take-profit, and hard rails around it: 2% daily loss cap, 0.6% daily profit cap, and an external watchdog that kills the process after four losses in a row.

The rails matter. The first $10k account blew through its max drawdown on September 9 and is gone. The fleet is two $100k accounts now, run as separate instances with separate state.

Here's the part most AI-trading posts skip. The bot already logs a lot. signal_log.csv has every decision, executed or skipped, with the model's reasoning attached. trade_analysis.csv has every close with MFE/MAE in pips and R-multiples. What nothing had was a check on the decision itself: does the thesis actually argue for the direction the model picked? When the account died, I could read every trade and every excuse, and none of it told me whether the reasoning was any good.

What JEV is

JEV is TypeSafe's System-One model (jev-latest), served at api.typesafe.ai/v1/systemone. It's text-only. No vision, no market opinions. You POST it a "state" string plus a set of questions, each with instructions and pass/fail criteria, and it returns structured verdicts:

{
  "state": "<the thing being judged>",
  "model": "jev-latest",
  "questions": {
    "shadow_direction_consistency": {
      "type": "noul",
      "instructions": "Based only on the provided state (a recorded hourly vision decision for a live FX scalper), is the recorded directional call a clear, self-consistent directional commitment coherent with its own thesis?",
      "criteria": {
        "true": "The recorded direction is explicit (UP or DOWN) and the recorded thesis is coherent with and supports it.",
        "false": "The direction is ambiguous, contradicts the thesis, or no clear directional commitment is recorded."
      }
    }
  }
}

The shape isn't an accident. A noul question is a pass/fail check the model answers one way against criteria you wrote — not an open-ended essay. That's what makes the verdict usable as a watchdog: it stays binary and countable, so you can line thousands of them up against the trades they judged and watch the incoherence rate move. An essay grader couldn't do that; it would drown you in prose and give you nothing to plot.

So it's a proofreader, not a fortune teller. It can't tell you whether UP is right about EURUSD. It can tell you when UP arrives with a thesis that quietly argues DOWN.

The shadow harness

The new file is shadow.py, about 430 lines, plus tests. The design has one hard rule: the trading path must not know it exists unless we say so.

Everything is gated behind SHADOW_MODE=1. When that's unset, the harness is a complete no-op; the only cost on the hot path is a single os.getenv() call, and the tests assert that zero network calls happen. When it's on, for every hourly vision decision the harness:

1. Dumps the exact chart images that went to the vision model into shadow_out/<instance>/<timestamp>/, and records a sha256 of the prompt text for later lookup.
2. Runs exactly one JEV call over a text rendering of those same inputs, hard-timeboxed at 20 seconds on a daemon thread. The state is capped at 30k characters to stay inside the model's context.
3. Appends one JSON line per decision to shadow_log.jsonl, including the JEV verdict and its latency.

A little later, an attach_outcome() call patches the most recent unfinalized line for that pair with what actually happened: EXECUTED, FAILED, or ORDER_FAILED, plus the ticket. It only touches lines written in the last 15 minutes, so a slow hourly cycle can't corrupt an older decision's record.

Each instance writes to its own log, so the fleet's accounts never mix.

The switch is currently off

Worth saying plainly: SHADOW_MODE is not set in any launch config yet, so zero shadow calls have happened in production. The dashboard probe ledger shows nine calls, all from us poking at the API while building the plugin, and the Shadow Harness tab reads NOT ENABLED until the first row lands. The harness is new and the tests are new, and we want a few market-open cycles of confidence before turning on a collector that sits in the decision path.

What we expect the data to answer

Three questions, in order of interest.

First, the big one: does coherence predict money? Every verdict lands in the same record as the real action and, downstream, the real P&L. If incoherent trades lose measurably more often than coherent ones, JEV graduates from observer to entry filter, and we skip or downsize flagged decisions. If it doesn't, we drop it. That's the point of measuring instead of believing.

How a verdict gets scored is fixed before the first data point. The pass/fail bar is pre-registered in code — minimum sample sizes, a minimum counterfactual edge, a ceiling on JEV errors — and the backend and the dashboard banner read the same constants, so the goalposts can't move after the data arrives. That same tab is where we watch it: it shows one of five states — NOT ENABLED, INSUFFICIENT, SHIP CANDIDATE, KILL, HOLD — next to the histogram of disagreement edges and a per-day series. The edge itself is computed counterfactually: on rows where JEV withholds endorsement, we flip the sign of what the vision-side trade actually realized. That's a mirror approximation, not a P&L replay, and it's labelled as one; orders that never resolved are counted separately, never scored as zero. And if JEV's disagreements turn out to correlate with winners, the shadow signal dies first.

Second, drift detection. The bot retries Gemini up to six times with exponential backoff, and we've swapped models before (3.5-flash to 3.6-flash in July). Degraded or fallback responses usually get sloppier before they get unprofitable. A rise in inconsistent verdicts is a cheap alarm — a shadow call is text-only, input-only pricing, a fraction of a cent per decision, and every record carries the JEV latency and error status, so degradation shows up in the record itself before the drawdown alert does.

Third, model comparison. Since the harness records each prompt's hash and the decision but stays agnostic about which model produced them, the same record can score an OpenRouter alternative against Gemini on identical charts. Evidence beats vibes when picking a model.

What it can't do

The limits are real and worth naming. JEV sees the text of the decision, never the chart pixels, so it audits self-consistency only. It will happily bless a confident, well-written, wrong call. It evaluates after the fact, so today it can't stop a trade, which is deliberate: any gating step is a later change with its own risk to review. And it's one question right now. Coverage is only as good as the question set, and one question is a start, not a panel. The API shape makes a panel cheap, though: several independent pass/fail checks can ride in a single request and get combined in code, and our dashboard probe already does that. Adding questions to the harness is a config change, not a rewrite.

The whole thing is built to answer one question, and the answer is allowed to be no. If coherent trades don't make more money than incoherent ones, JEV gets dropped and we've lost a few cents a decision and a couple of weeks. If they do, the auditor stops being a proofreader and becomes a filter. Either way the data decides, and the bar was written before the first verdict landed. That's the part I'm actually proud of — not the harness, the willingness to let it be wrong.

Comments

[mthburnsbarber-web]

The pre-registration discipline on the scoring bar before the first verdict lands is the thing most backtesting efforts skip entirely. Moving goalposts after you've seen the data is how a useless signal gets adopted because of survivorship in the evaluation window. Locking the minimum counterfactual edge and sample size in code, and having the dashboard read the same constants, is the right way to make it actually falsifiable.

The noul question shape is interesting — binary pass/fail criteria makes it composable and countable in a way prose evaluation never is. You said the API shape makes a panel cheap, and that part seems under-explored. Grading coherence is one thing, but the same binary structure lets you check other auditable properties independently — does the thesis cite the timeframe it's supposed to trade, does the pair specificity match the instrument — without entangling them in a single judgment.

The honest naming of limits helps too. An auditor that validates self-consistency isn't the same as one that predicts direction, and conflating them is exactly how these tools get overpromised. "Either way the data decides" is the right frame. What's the latency penalty looking like on the shadow thread when it actually runs?

[chowderhead]

testing will start this week, ill comment here and let you know- thanks for bring this up its a good metric to measure.

[Ono Saburo]

Hello Salika,
I hope you're doing well.
I have a good business idea that I'd love to discuss with you in more detail.
To give you some background, a friend of mine started this business with a U.S.-based partner three years ago. Since then, he's been paying his partner between $8,000 and $10,000 per month, and the business has been working well.
If you're interested in learning more, I'd be happy to share the details.
Whatsapp: [phone]
Telegram: @ono0319
Best regards,
Ono