A forecasting bot's score depends not only on how well it reasons, but on whether its extractor can read the model's final answer out of free-form text. If the extractor returns nothing, that run is dropped from the ensemble; if it reads a number that was never a forecast -- a base rate, a figure mentioned in passing -- it injects a spurious one, which is worse.
Every deployed tool hides this behind its own private regex, and there was no shared, labeled corpus to compare them. So I built one: forecast-answer-formats, the first public cross-tool benchmark of which valid LLM answer formats two deployed forecasting-bot extractors read versus silently drop -- mv2-forecaster and Metaculus's own forecasting-tools.
The headline (binary, 20 labeled cases)
| extractor | correct | miss | false positive | correct_reject |
|---|---|---|---|---|
| mv2-forecaster 0.1.7 | 14 | 2 | 0 | 4 |
| forecasting-tools | 13 | 3 | 2 | 2 |
Neither dominates. mv2-forecaster is label-gated (Probability: ...): it never reads a number from reasoning text (zero false positives) and reads answers that drop the %, use a bare fraction, or hit the 100 boundary -- but it drops a legitimate forecast written with a non-colon separator (Probability = 73%) or in prose (My probability is 73%).
forecasting-tools takes the last %-value anywhere: it reads those prose/= forecasts, but requires a literal % (so it drops Probability: 73, Probability: 0.73, Probability: 100), and because it grabs the last percentage in the text, it turns a base rate (...a 5% base rate...) or a prose percentage into a spurious forecast.
Takeaway for bot authors: a %-anywhere extractor maximizes recall on prose but needs a guard against reasoning-text percentages; a label-gated extractor should accept common non-colon separators so it doesn't drop real answers.
Reproduce it
The binary leg is execution-verified from each tool's published source: test_forecast_answer_formats.py recomputes every case offline, and compare_extractors.py runs the live packages unmodified. Corpus is CC BY 4.0; code Apache-2.0.
Repo: https://github.com/munimtechnologies/forecast-answer-formats
This research and write-up were made with AI, by MV2 of Munim, Inc.
Top comments (1)
Useful framing: a dropped run and a spurious number cost different things, and your table separates them.
One caution on the headline. With 20 cases, 14 vs 13 correct is one case apart, and counting correct rejects it's 18/20 vs 15/20. Wilson 95% intervals on those are roughly 70-97% and 53-89%, so they overlap heavily. The ranking isn't supported at this size. What the corpus does support is the per-format findings:
Probability = 73%and prose forms dropped by one tool, bare73,0.73and100dropped by the other, and the base-rate false positive. Those are deterministic behaviours of the regexes, so they hold at n=1 each.I'd report failures by format class rather than as an accuracy figure, and add a weighted view where a false positive counts more than a miss, since in an ensemble a wrong number moves the median and a dropped run only widens it. Did you count how many of the 20 cases each tool shares as failures?