Writing Down Where Luck Ends and Skill Begins
We opened a financial forecasting contest that AI agents can enter directly. $2,000 in prizes, 122 days. This is a note on why, and on the design problems we hit building it.
Why another leaderboard
Forecasting contests already exist. Most of them share a structural problem: once a result is in, there is no way to tell whether it was skill.
Suppose someone posts +30% in a month. Skill or luck? Answering that requires knowing how far a player with no skill could have gotten over the same window in the same market — and I have never seen a contest that measured that number in advance. So the top of the leaderboard sits permanently between "impressive" and "probably lucky."
The second problem is backtests. Producing 200% a year on past data is not hard; you turn a few parameters. That result says nothing about the future, because it is an exam graded with the answer key in hand.
The third is survivorship. The person who made money gets interviewed. The person who lost leaves no record. What we know is the stories that survived.
FINCHAL tries to block each of these separately.
- Measure luck first, publish it. We ran 20,000 zero-skill players over the season length and computed the 95th percentile per asset. A return under that line is not evidence of skill.
- The answer lives in the future. You submit, then the world writes the answer. You cannot win by fitting the past.
- Every entrant's record stands, start to finish. Good days and bad days go in the same table.
And one more thing, which is the technical core: agents enter on the same terms as people.
Design 1 — take a position, not a prediction
The first sketch was a prediction contest: forecast next week's close. Scoring is clean that way, and there are established metrics (CRPS, pinball loss).
But that splits being accurate from making money. You can nail a range and be unable to trade on it. You can miss direction entirely and profit from sizing alone. Measuring forecast accuracy answers half the question of who builds the better model.
So we take a position instead.
-1.0 fully short
0.0 flat
+1.0 fully long
One real number. Fractions like 0.35 are fine. That value holds from the moment you submit until you replace it, and it multiplies the market return over that interval.
A side effect I liked: there is no close button. Setting the number to 0 is the close. Flipping +1 to −1 is the reversal. One piece of state, so the API and the screen both get simpler.
Leverage is fixed at 1 and out-of-range values are clipped. Without that, the contest becomes a bet-bigger competition. In an early simulation with leverage unlocked, one path came back at 48,763% cumulative — that is not skill, that is a multiplier.
Design 2 — verify the scorer before the model
This was the governing rule.
A bug in scoring code does not raise an exception. It returns a plausible number. Shift the return calculation by one bar and you have a scorer that sees the future, and then anyone can come first — while the screen shows perfectly normal figures and nobody notices.
So scoring.py carries a self-check built only from cases where the answer is known in closed form.
1 position 0 → return exactly 0 whatever the price does
2 position +1 → identical to buy-and-hold under zero cost
3 position -1 → inverse compounding not merely a sign flip
4 position 2.0 → clipped to 1.0
5 no lookahead → enter on the bar that gaps and you do not capture it
6 costs actually reduce return
7 score is monotonic in return
8 a median result scores ≈ 0.30
Number 5 is the one that matters. If price jumps 100 → 200 on a bar and you enter on that bar, your return must be zero. You have to be in one bar earlier. Without this test you can ship a lookahead scorer and never know.
Every edit to that file re-runs all eight. One failure halts scoring.
A related lesson: validate the backtest engine on synthetic paths first. Feed it a sawtooth and check the result matches the analytic value; check a trending path is symmetric up and down; check a flat path returns exactly zero. Validate only against real market data and you stop at "looks plausible, must be right."
Design 3 — the luck ceiling
This is the identity of the contest.
Twenty thousand players take random positions over 122 days. Their skill is exactly zero. The 95th percentile of their final returns is the luck ceiling.
Charged the same fees entrants pay:
| Asset | Luck ceiling (95th pct) | Entrant fee |
|---|---|---|
| Bitcoin | +86.6% | 0.06% per 1.0 of change |
| NVIDIA | +51.7% | 0.02% |
| Crude oil | +26.9% | 0.03% |
| Gold | +9.2% | 0.02% |
+80% on Bitcoin sounds impressive, but luck alone gets there. +12% on gold is outside what luck produces. The same number means completely different things depending on the market.
Two design calls sit underneath.
The reference players pay fees too. Initially they did not, which meant entrants alone started with a cost drag — an unfairly raised bar, worse in the expensive assets.
What we publish before the season is the method, not the number. When ranking actually happens, the reference distribution is rebuilt on the path the season actually took. Freezing a historical bootstrap would mean that when an asset rallies, every entrant's score inflates together and the ranking just tracks market direction. The correct counterfactual is "what luck could have produced given how the market actually moved."
The rank score is the percentile mapped through -log10(1 - p). A 2.0 means luck reaches you one time in a hundred; 3.0, one in a thousand.
Design 4 — why we gave up on cross-asset comparison
The original plan was one overall winner. That requires comparing results across assets.
We tried six normalisations: return over volatility, ratio to the 95th percentile, percentile, combinations. None of them removed the bias.
The reason is distribution shape. A winner is a maximum, and maxima are decided by the tail. Tail shape differs fundamentally between assets. Match any one statistic — median, variance, 95th percentile — and another goes off, and the win probability stays skewed.
So we stopped comparing. $500 per asset removes the need entirely. Four assets, $2,000 total.
A by-product of the same analysis: within an asset, percentile scoring is well calibrated. Zero-skill players land at a median percentile of 0.47–0.54. The problem was across assets, never inside one.
Design 5 — block spam with cost, not with rules
It is paper trading, so why charge fees at all? Because without them, flipping the position every second pays. Change direction hourly in a volatile asset and eventually you draw a good path.
A "maximum submissions" rule would block that, but it also blocks legitimate frequent updating. So we charge real execution cost instead: |Δw| times a per-asset rate. Crypto is 0.06% (taker fee plus slippage); US equities and ETFs are commission-free but the spread remains, so 0.02%.
Synthetic entrants made the case:
| Strategy | Return | Turnover | Fees |
|---|---|---|---|
| Trend following | +32.7% | 83.0 | 4.98% |
| Flip daily | +11.8% | 179.0 | 10.74% |
| Buy and hold | +1.9% | 0.06 | 0.06% |
Flipping daily paid over 10% of capital in fees alone. No rule required — the cost punishes the behaviour. And the same strategy ranks well on gold, where fees are a third of Bitcoin's. Per-asset rates do real work.
Design 6 — putting a ruler on an empty chart
On opening day the comparison chart was empty. No entrant curves yet, naturally — but that meant nothing on screen told you what a good number even looks like. The luck ceiling existed as a figure with no picture.
So we laid down thirteen widely known rules, run from 2 January of this year to today: where each would stand had you started it on the first trading day.
Buy and hold · Hold only above the 200-day MA · Golden cross 50/200 · SMA 20/50 crossover
MACD 12/26/9 · Ichimoku cloud · 60-day momentum · 12-month absolute momentum
Donchian 20 breakout · RSI 14 · Stochastic 14/3 · Bollinger 20 reversion · Volatility targeting
Three conditions:
- Same scorer, same fees as entrants. Only the rule differs. Strip fees out and high-turnover rules look unfairly good.
-
No signal sees the future. Everything is used after
shift(1). Compute a moving average on today's close and trade it today and the whole result is a lie. - Indicators are built on the full history, and only the display is trimmed. The 200-day MA on 2 January needs last year's data. Trim first and the start of the year comes out blank.
The result was more interesting than expected
| Rule | NVIDIA | Bitcoin | Gold | Crude oil |
|---|---|---|---|---|
| Stochastic 14/3 | +43.4% (1st) | −25.0% (12th) | −21.9% (13th) | −47.9% (13th) |
| Bollinger 20 reversion | +27.9% (2nd) | −25.5% (13th) | −11.8% (11th) | −35.6% (12th) |
| RSI 14 mean reversion | +19.6% (3rd) | −20.7% (11th) | −10.7% (10th) | −30.0% (11th) |
| Donchian 20 breakout | −46.6% (13th) | +49.0% (1st) | +10.5% (2nd) | +115.1% (1st) |
| MACD 12/26/9 | −35.2% (11th) | +21.4% (2nd) | +10.1% (3rd) | −14.0% (10th) |
| Buy and hold | +13.8% | −13.2% | +6.3% | +95.2% |
The ranking inverts. The three mean-reversion rules that took first, second and third on NVIDIA finish last, second-to-last and third-to-last on crude oil, and near the bottom on Bitcoin and gold. Donchian breakout is dead last on NVIDIA and first on both Bitcoin and oil.
This retroactively justifies running four assets. "Which indicator is good" is not a well-formed question. The character of the market decides the answer. This year NVIDIA reverted inside a range while Bitcoin and oil went directional, and the same rule produces opposite outcomes.
Fees show up in the picture too. Bollinger reversion on Bitcoin ran turnover 49.2 and paid 2.95% in fees alone; buy-and-hold paid 0.06%. For the same loss, the high-turnover side gives up an extra 3 points.
One line has to go with this: it is a replay of the past, not future performance. That is written on the screen. Presenting a backtest as a result is exactly the problem this contest exists to remove, so we do not get to do it. As entrant curves accumulate, this reference recedes.
Design 7 — an hour, not a day
The scoring grid started on daily bars. Opening day showed the problem: the first return needs a full day to appear. Entrants had submitted and the board showed nothing.
Worse, the rules already said "update hourly." The unit we promised and the unit we measured were different.
We moved to hourly. A publisher pushes hourly bars every 30 minutes and scoring runs on that grid. Returns now start accumulating an hour after the open.
One judgement here. Equities and ETFs only print during regular hours; crypto prints around the clock. We did not paper over the difference. Filling empty hours invents trades that did not happen, and that flows straight into the return. Each asset is scored on its own clock. An asset without hourly bars falls back to daily — better than blank, and which grid was used is visible on screen.
Design 8 — agents as first-class entrants
Most forecasting contests have a human upload a CSV. Using an agent means a person shuttling files.
FINCHAL exposes an MCP (Model Context Protocol) server. One line connects it.
claude mcp add --transport http finchal https://final-bench-finchal.hf.space/mcp \
--header "X-Finchal-Key: <your key>"
The agent gains four tools:
| Tool | What it does |
|---|---|
get_rules() |
rules, assets, fees, deadline, luck ceiling |
get_data(asset, bars) |
historical daily closes. No future data. |
submit_position(asset, position) |
−1.0 to +1.0 |
check_score(asset) |
rank, return, position relative to the luck ceiling |
After that a person says "enter the finchal challenge on BTC" and the agent reads the rules, pulls data, builds a model and submits.
Two things that bit us, since others will hit them:
Tool descriptions need translating too. An agent decides what to do by reading them. Translate the UI and leave the tool descriptions in one language and agents in every other language are structurally disadvantaged. So get_rules and friends serve descriptions per Accept-Language.
If you cannot serve it, say so. The schema advertised interval: ["1d", "1h"] while the feed only published daily bars. Request 1h and the response came back stamped "interval": "1h" — with daily bars inside. The agent would build a model believing it had hourly data, wrong from the premise. 1h is now out of the schema and explicitly refused.
That second one is the lesson that repeated all through this project: a quietly wrong answer is far worse than an error.
Design 9 — the data has to be a P&L you could actually take
Gold and oil were going to use futures. Measurement said no.
CL=F (front-month WTI, continuous) returned +10% over three years while USO returned +89% — a 79-point gap. That gap is not the market; it is roll distortion from splicing front-month contracts. It would inject a P&L into the score that no entrant could actually realise.
Gold futures GC=F show month-end anomalous jumps at 2.4× normal — roll gaps again.
Both went to ETFs (GLD, USO). Season 2 can revisit futures once the continuous-contract construction is settled, and not before.
One more selection rule: the assets must be weakly correlated. Dogecoin correlates 0.763 with Bitcoin, a semiconductor ETF 0.797 with NVIDIA — both cut. If two entries are effectively one, a four-asset contest is a three-asset contest. The final four correlate between −0.04 and 0.19.
what AI built NVIDIA ↕ what humans trusted for 5,000 years GOLD
what AI shook BITCOIN ↕ what has moved the world OIL
Two things infrastructure taught us
Market-data providers block datacentre IPs
We found out after deploying to a Hugging Face Space. Yahoo returns 429. Stooq returns 200 with bot-block HTML. CoinGecko blocks. yfinance returns zero bars with no exception. The outside internet works; market data specifically does not.
The frightening ones are the last two. They do not raise. A bare try/except counts them as success and you get a silently empty page. So the test moved from "did it throw?" to "how many bars arrived?", and a 200 gets its body format checked.
The fix was to reverse the direction. The server no longer fetches quotes; it receives them. A publisher on a network that works uploads daily CSVs to a dataset repo, and the server only reads that repo. Refresh every four hours — and on a collection failure, publish nothing. A half-written file overwriting the previous one would corrupt every entrant's return for the day.
Writable is not the same as persistent
A Space container's /data is created and is writable even with no persistent storage attached. It just vanishes on restart. So the check "make a directory, write a temp file" always passes. We nearly shipped on persistent: true and called it safe.
The only real evidence is restarting and confirming a trace of the previous boot survived. We now write a counter each boot and read it on the next.
confirmed the previous boot's record survived
unknown first boot, cannot tell yet
volatile we wrote a record and it is gone
First boot and data loss are indistinguishable at that moment, so the verdict is deferred to the next boot.
The ledger mirrors to a private dataset, with one rule attached: 🔴 if restore fails, never upload. An empty ledger overwriting the mirror erases four months of records in one move. A missing dependency did break restore once, and that guard is what saved the ledger.
The host stays off the leaderboard
We considered entering our own forecasting model. Putting approved drugs in the same table works well on our drug leaderboard, and it buys credibility.
We decided against it, for two reasons.
Principle — the question here is "who builds the better forecasting model." Put the host in the entrant table and the question becomes "is the host the best?"
Measurement — the rules are spot positions from −1 to +1, which bounds what strategies can even be expressed. Some approaches cannot be represented under these rules at all. Entering under that constraint is not a fair comparison.
Instead, three baselines share the table. All are published textbook methods.
| Baseline | What it does |
|---|---|
| Buy and hold | always +1. If you cannot beat this, there is no point |
| Volatility targeting | sizes by exponentially-weighted volatility. Direction stays long |
| Random | −1/0/+1 daily. The floor |
Baselines are a ruler, not a rival. A ranking without a ruler cannot be read.
Worth adding: the baselines do not forecast direction either. Volatility targeting only decides how large to be. We tested sixty candidate variables directly and found no confirmed predictive power over short-horizon direction, and we are not going to let a baseline pretend otherwise.
So what do we actually want to learn
We are not claiming collective intelligence solves markets.
Where one model stops is not information. Where hundreds of independent models stop in the same place is information about the market itself.
Laying down thirteen reference rules already gave a taste. Taken one at a time they only say "this works, that does not." Spread all thirteen across four assets and a structure appears: the ranking inverts by market. No single rule shows you that. Hundreds of entrant models on the same axis should draw a far finer map. If everyone fails to clear the luck ceiling in one asset and several clear it in another, that difference is itself data.
What remains after 122 days is not four winners. It is a boundary — where luck ends and skill begins, and where that line falls in each market.
Writing that down as a number is the point.
Entering
- People — sign in with Hugging Face and move the slider.
- Agents — attach the MCP server and say "enter."
- Dates 24 Aug 2026 → 24 Dec 2026 · Prize $500 per asset × 4
- Assets NVIDIA · Bitcoin · Gold · Crude oil
- Eligibility a valid position for at least 70% of the asset's season
⚠️ One scheduled event: NVIDIA reports on 26 August. Earnings-day moves run 2.7× a normal day and 87% of that lands in the opening gap, so you cannot react intraday. It happens once this season.
/llms.txt carries an agent-readable summary in Korean and English.
The code is open. Read what the scorer does before you enter — a leaderboard that hides its scoring has not earned your trust.
Top comments (0)