Key Takeaways
- OpenAI o1 achieved an 83% success rate on the 2024 American Invitational Mathematics Examination (AIME), up from 13% recorded by GPT-4o, according to OpenAI’s technical report.
- DeepSeek-V3 scored 90.2% on the MATH benchmark using a Mixture-of-Experts architecture and reinforcement learning, compared to OpenAI o1’s 94.4%, at a reported training cost of $5.6 million.
- Epoch AI’s FrontierMath benchmark, built with over 60 expert mathematicians, shows leading models from OpenAI, Google and Anthropic solving fewer than 2% of novel PhD-level problems, exposing a hard ceiling that inference-time scaling alone may not break. The gap between 94% on high school math and under 2% on PhD-level problems is the most honest summary of where AI reasoning actually stands. OpenAI‘s o1 and DeepSeek-V3 have both made real progress on competition-level mathematics, but a benchmark released by Epoch AI in late 2024 makes clear that research-level mathematics is a different problem entirely.
The Transition to Inference-Time Reasoning
Both o1 and DeepSeek-V3 move beyond standard next-token prediction by allocating extra compute at inference time. The idea, sometimes called “System 2 thinking,” is that the model generates an internal chain of thought before committing to an answer. It can backtrack, check steps and revise before the user sees any output. According to OpenAI’s technical report, o1 solved roughly 83% of problems on the 2024 AIME, a competition-level test for high school students. GPT-4o, without that structured reasoning time, managed 13%.
DeepSeek-V3 takes a different architectural route. Its technical report, released in late 2024, describes a Mixture-of-Experts (MoE) framework that activates roughly 37 billion parameters per token from a total of 671 billion. Trained heavily on mathematical proofs and code, it scored 90.2% on the MATH benchmark against o1’s 94.4%. The closing gap between closed-source and open-weights models suggests the core recipe, reinforcement learning combined with chain-of-thought data, is now well understood outside of OpenAI’s walls.
Training cost is the other number worth noting. The DeepSeek-V3 technical report states the model was trained for approximately $5.6 million on H800 GPUs. That figure, if accurate, puts high-tier mathematical reasoning within reach of organisations that cannot afford nine-figure compute budgets. The trade-off is latency: inference-time reasoning burns tokens internally, so users get higher accuracy but slower responses. For R&D and engineering workloads, that is usually the right trade.
Saturating the GSM8K and MATH Benchmarks
The benchmarks that shaped AI research for the past several years are no longer useful for distinguishing between leading models. GSM8K, a set of roughly 8,000 grade-school word problems, has been effectively saturated, with multiple models now scoring above 95%. The MATH benchmark is approaching the same ceiling. At that point, a score difference of a few percentage points tells you almost nothing about whether a model genuinely reasons through a problem or has simply absorbed similar structures during training.
The 2024 AIME holds up better as a test. Solving around 12 of 15 problems places a model in the top tier of high school competitors in the US. What makes these problems hard is their multi-step structure: an error in step three can silently corrupt the answer in step ten. The self-correction loop built into o1’s reasoning chain is the main reason its AIME scores jumped. It identifies its own errors before the user sees the output, working the way a careful mathematician checks a proof.
Data contamination is a real concern with any public benchmark. GSM8K and MATH have been available for years, and variations of their problems have likely appeared in training sets. The 2024 AIME results are more credible on this front because the problems were published after the models’ training cutoffs, which suggests the reasoning capability is genuine rather than memorised. That distinction matters for engineering and physics applications where problems rarely resemble anything in a textbook.
FrontierMath and the PhD Ceiling
Less than 2% is the number that matters here. That is the combined solve rate of leading models from OpenAI, Google and Anthropic on FrontierMath, a benchmark Epoch AI released in late 2024 in collaboration with over 60 expert mathematicians, including several Fields Medalists. The problems are entirely original, not available on the public internet, and often require hours or days for a human PhD to work through. There is no contamination shortcut here.
The failure mode is specific. Current models can follow a chain of logic step by step but struggle with what might be called global strategy: seeing the shape of a proof before writing any of it down. Many FrontierMath problems require an unconventional leap or the combination of two unrelated areas of mathematics. Models tend to produce proofs that read correctly sentence by sentence but contain a fatal flaw at the core. Scaling inference-time compute pushes the token budget higher, but it does not solve this problem.
Epoch AI’s data shows that even with a very large number of attempts at a single problem, models rarely arrive at a correct answer. This points to a representational gap, not just a compute gap. One active research direction is pairing LLMs with formal verification systems like Lean or Isabelle, which act as a mathematical sandbox: the model proposes a proof step, the verifier rejects it instantly if it is logically invalid, and the model adjusts. Google DeepMind and OpenAI are both working in this space, though how far either has progressed is not yet clear from public material.
Economic and Operational Implications of Math Reasoning
Strong math performance matters to builders because it correlates with reliable multi-step reasoning across domains. A model that can work through a competition-level problem is more likely to handle a complex financial audit or a multi-file software refactoring task without dropping a step. The correlation between math benchmark scores and coding performance is frequently cited in technical evaluations, though a precise figure varies by evaluation method and no single authoritative source pins it down cleanly.
The cost structure is still a constraint. Running o1 is meaningfully more expensive than GPT-4o because the provider charges for the internal reasoning tokens, even though the user never sees them. A hard problem can generate thousands of internal tokens to produce a short final answer. That makes o1-class models poorly suited to high-volume, low-margin use cases like customer service chat, but well matched to R&D, structural engineering and legal analysis where accuracy justifies the cost. DeepSeek-V3 offers a lower-cost path into this tier, though many enterprise teams remain cautious about data handling and the provenance of its model weights.
The near-term direction for builders is adaptive routing. Rather than running a powerful reasoning model on every query, an orchestration layer, of the kind already appearing in tools like GitHub Copilot and Cursor, selects the model based on task complexity. Simple arithmetic goes to a small, fast model. A complex optimisation problem gets escalated to a reasoning-heavy one. This is a tractable engineering problem today, and it is where practical agentic workflow design is heading. Whether any current model can eventually contribute to genuinely new mathematics, rather than solving known problem types more reliably, remains an open question. The FrontierMath numbers suggest that milestone is further away than the AIME results might imply. For more on AI agents and automation tools, visit our AI Agents section.
Originally published at https://autonainews.com/openai-o1-scores-83-on-aime-math-while-frontiermath-stumps-all-models/
Top comments (0)