For two months I let LLM agents paper-trade a simulated $100 crypto portfolio. Season 1 ended with a verdict of STOP: both agents lost to cash. Season 2 ended with all four agents beating the passive benchmark by 3 to 10 percentage points. The formal verdict for Season 2 is still "inconclusive", and I think that is the most useful result of the whole project.
No real money ever moved. Live execution was never implemented and the code fails closed if you try to turn it on. Everything below is CoinGecko-reference paper returns on Base L2 tokens, priced with a static cost model (0.05 to 1.00 % fee tier plus a 0.15 to 0.50 % execution haircut plus $0.03 gas per swap).
The setup
The agents were plain LLM calls through the vendors' CLIs (claude -p, codex exec, subscription auth, no API keys), launched with a minimal environment so no secret from .env could reach the model. The model proposes a trade as JSON; a policy engine in Python decides. The hard limits lived in code, not in the prompt: trade size $5 to $20, max 3 trades a day, meme tokens capped at 15 % of the portfolio, a global stop-loss that liquidates to USDC at 70 % of starting value. Every decision, including rejected ones, went to an append-only journal.
Season 1: the green number that wasn't
Two agents, same code, different model (Claude Opus vs GPT via Codex), one decision every 4 hours, 30 days. Whole-run result: Opus finished at $101.48, GPT at $96.81. Opus made 5 trades and paid $0.88 in modeled fees; GPT made 23 trades and paid $3.51, which was most of its gap.
Then I looked at the window I had committed to on day 17 (an interim freeze of the decision rule, not a true pre-registration, and I am labelling it that way on purpose). Inside that window, Opus returned −1.85 % and GPT −3.00 %. Both beat the random-trader path. Both lost to cash and to a passive 50/50 WETH/USDC portfolio. The rule said: STOP.
Without the frozen rule I would have written "Opus +1.48 %, let's go live". The whole-run number was ETH beta from early July, not judgement.
Season 2: fix every confound, then pre-register for real
Season 1 had too many moving parts to compare anything: agents read separate price snapshots, the model was swapped mid-run, the decision architecture changed a week in, and the random baseline was one path. Season 2 was a rebuild with one question: how much of the decision should the model own, and how much should code own?
| Season 1 problem | Season 2 fix |
|---|---|
| separate price snapshots per agent | one daily snapshot shared by all arms |
| model, universe and prompt changed mid-run | everything frozen before tick 1 |
| autonomous vs structured never compared | it became the experimental axis |
| 4-hour cadence, churn | 1 decision per day |
| decision rule frozen on day 17 | pre-registration written before the first tick |
Four arms: {Claude Opus 5, GPT (Codex)} × {A1 autonomous: raw data and its own journal, decides everything; A2 structured: code-ranked candidates, a cost table, a forced daily forecast, and a code-side edge gate}. All four started at 50/50 WETH/USDC with $100. The pre-registration committed the decision rules: the architecture claim needs the same sign in both models and in two 30-day windows, with a gap of at least 0.5 pp that survives a cost-sensitivity sweep at 0.5×, 1× and 2× variable costs. "Edge" needs at least +2 pp net over passive 50/50 in both windows. A win in one window is a case study.
Season 2, window 1: everything green
Net of modeled costs, day 30:
| Arm | Terminal value | vs passive 50/50 |
|---|---|---|
| Opus structured | $124.97 | +9.83 pp |
| GPT structured | $124.01 | +8.87 pp |
| Opus autonomous | $121.72 | +6.58 pp |
| GPT autonomous | $118.52 | +3.38 pp |
| passive 50/50 (benchmark) | $115.14 | |
| trend (7-day average) | $121.41 | +6.27 pp |
| HODL ETH | $130.25 | +15.11 pp |
| cash | $100.03 |
Three things fell out of that table.
Structured beat autonomous in both models. Opus +3.24 pp, GPT +5.49 pp. The Opus gap survived the cost sweep (+3.43 / +3.24 / +2.87 pp). The GPT gap did not: at 2× costs one of GPT structured's recorded trades becomes unfundable in the replay, so by the frozen rule that gap is cost-fragile. Trade counts: 5, 9, 5 and 13. The structured arms' edge gate rejected 2 proposed swaps each.
Nobody beat ETH. ETH rallied about 30 % in the window and every arm held some USDC the whole time. The green column is partial beta plus a bull month. It is not skill, and the pre-registration said in advance that "absolute green driven by market beta" is explicitly not success.
No forecasting skill at all. Every arm answered one question daily: the probability that WETH outperforms USDC by the next daily snapshot (24 hours for 92 of the 100 scored forecasts, about two days for the 8 that followed a skipped day). Brier scores came out at 0.256 to 0.260 against a base-rate Brier of 0.243 to 0.250 (N = 24 to 26 per arm). Every arm was slightly worse than always guessing the base rate.
Ops accounting, because the rule required it: 29 of 30 tick-days ran (one day lost to a scheduler wedge, baselines paused the same day so the comparison stays fair). Opus arms lost 2 decision days to a failed headless OAuth refresh over a weekend; GPT arms lost 4 to one 401 and three CLI failures, two of them confirmed ChatGPT usage-limit hits. The fallback was always "hold", and the asymmetry slightly disadvantages GPT.
Why the verdict is inconclusive
The pre-registration required two windows. I ran one and then ended the project, so every hypothesis stays at "inconclusive: single window". That is not a loophole, it is the rule doing its job. One 30-day window on one asset pair in one bull month is one episode. Four arms on that episode, all beating a fixed benchmark, is what beta looks like when your baseline is half in cash.
What I will say: the structured-beats-autonomous direction showed up in both models with a consistent sign and a plausible mechanism (the code-side gate rejected the kind of churn that killed GPT in Season 1). That is a hypothesis worth a real replication. It is not a finding, and my own document says so.
Wrapping up
The key insight: write the decision rule before the data exists, then obey it when it becomes inconvenient. In Season 1 the rule turned "+1.48 %" into STOP. In Season 2 it turned "+25 %" into inconclusive. Both times the rule was right and my instinct was wrong, and the second time it was the more expensive lesson, because the numbers were green.
Top comments (0)