Ernesto Herrera Salinas

Posted on Jun 25

AGI, Are We There Yet? A Follow-Up

#ai #openai #writing

In my previous article, AGI, Are We There Yet?, I asked a simple question with a complicated answer:

Are we at AGI yet?

At the time, OpenAI’s o3 had just made headlines for its performance on ARC-AGI-1. Depending on the compute budget, o3 reached 75.7% on the semi-private evaluation set and 87.5% in a much higher-compute configuration, according to ARC Prize’s analysis of the o3 result.

That was a serious milestone. It was not just another benchmark bump. ARC was designed to test something closer to fluid intelligence: the ability to solve novel problems, not just repeat patterns from training data.

But my conclusion then was cautious:

This was a pit stop on the road to AGI, not the destination.

A year and a half later, that still looks like the right call.

The story since then has not been “AGI arrived.” The story is more interesting than that. Frontier models have become much better at reasoning, coding, tool use, multimodal understanding, long-context processing, and agent-like workflows. They are more useful, more capable, and more economically relevant than the systems we were discussing even a short time ago.

But they are still not robustly general in the human sense.

So where are we today?

We are not at AGI.

But we are no longer talking about simple chatbots either.

We are now in the era of frontier reasoning-and-agent systems.

The short version

Question	Current answer
Did o3’s ARC-AGI-1 result matter?	Yes. It was a real milestone.
Did it prove AGI had arrived?	No. ARC Prize itself warned against that interpretation.
Have frontier models improved since then?	Yes, significantly. Especially in reasoning, coding, multimodality, long context, and tool use.
Are old benchmarks still enough?	No. Many are saturated or noisy.
Are newer benchmarks still difficult?	Yes. ARC-AGI-2, ARC-AGI-3, HLE, and long-horizon task evaluations still expose major gaps.
Are we at AGI?	No. We are at increasingly general-purpose, increasingly agentic systems, but not robust human-like general intelligence.

What the previous article claimed and how it reads today

The previous article defined AGI as the broad adaptability of a human mind, contrasted it with narrow AI, and argued that o3’s ARC result was important but not conclusive.

That framing still holds.

Claim from the original article	Current evidence	Updated reading
AGI should imply broad adaptability, not just task-specific skill.	François Chollet’s On the Measure of Intelligence frames intelligence around skill-acquisition efficiency and fluid generalization, not static benchmark skill.	Still correct. Benchmarks are useful, but not equivalent to AGI.
o3’s ARC-AGI-1 result was a milestone.	ARC Prize reported o3 at 75.7% under the public compute budget and 87.5% in a high-compute configuration.	Still correct. It was a major jump.
Passing ARC-AGI-1 should not be treated as AGI.	ARC Prize explicitly said the result did not mean AGI had arrived and later introduced harder ARC-AGI benchmarks.	Still correct, and even more strongly supported now.
Chain-of-thought is not necessarily a faithful window into reasoning.	Research like LLMs Do Not Think Step-by-step In Implicit Reasoning and later chain-of-thought monitorability work suggests caution.	Still correct. Reasoning traces are useful, but not proof of internal cognition.
Scaling laws matter, but do not guarantee AGI.	OpenAI’s Learning to reason with LLMs and Google’s Gemini 2.5 report show gains from both training-time and inference-time compute.	Correct, with an update: the frontier now also scales through test-time reasoning and tool use.

The o3 ARC result aged well, but not as proof of AGI

The o3 result on ARC-AGI-1 was real and important. It showed that test-time compute, better reasoning methods, and stronger model scaffolding could dramatically improve performance on tasks that had resisted previous systems.

That matters.

But the caveats also mattered.

ARC Prize itself warned that passing ARC-AGI-1 should not be treated as “AGI achieved.” They noted that o3 still failed some tasks humans would find easy, that the result depended heavily on compute, and that the tested system had been trained using part of the public ARC training set.

That distinction is critical.

A model can perform extremely well on a benchmark without possessing the full flexible intelligence that the benchmark was trying to approximate.

And that is exactly what became clearer afterward.

ARC-AGI-1 started to saturate, so ARC-AGI-2 was introduced. This newer benchmark kept the original spirit of ARC but increased the difficulty around compositional reasoning, contextual rule application, and novelty.

The result?

As of the ARC-AGI-2 paper, frontier models were still below meaningful performance. OpenAI o3 scored around 3%, and other leading systems were similarly weak. The benchmark authors considered scores under 5% not meaningful.

Then ARC-AGI-3 went further. Instead of static puzzles, it introduced interactive environments where agents must infer goals, explore, learn rules, and adapt through interaction.

Again, frontier systems struggled. As of the ARC-AGI-3 paper, leading AI systems were below 1%, while humans could solve the environments reliably.

That is the cleanest update to my previous article:

o3 broke through ARC-AGI-1, but the broader ARC program did not confirm AGI. It confirmed that evaluation had to move.

That is progress, but it is not arrival.

Timeline: AGI-relevant milestones since the previous article

Date	Milestone	Why it matters
Dec 2024	ARC Prize reports o3 at 75.7% high-efficiency and 87.5% high-compute on ARC-AGI-1.	A major reasoning benchmark breakthrough, but not proof of AGI.
Jan 2025	Humanity’s Last Exam is introduced.	A harder, broad academic benchmark designed after older benchmarks became easier for frontier models.
Mar 2025	METR publishes work on measuring AI ability to complete long tasks.	Moves evaluation from static questions toward task duration and real-world work horizons.
Apr 2025	OpenAI releases o3 and o4-mini.	Reasoning models become more explicitly tool-using and agentic.
May 2025	Anthropic releases Claude 4.	Stronger agentic coding and long-running software workflows become a major frontier battleground.
May 2025	ARC-AGI-2 paper shows frontier systems still below meaningful performance.	Strong evidence that novelty and compositional generalization remain hard.
Jun 2025	Google publishes the Gemini 2.5 technical report.	Shows broad gains in coding, reasoning, multimodality, long context, and thinking-budget performance.
Mar 2026	ARC-AGI-3 is introduced for agentic intelligence.	Moves evaluation from static reasoning puzzles to interactive environments.
May 2026	METR publishes updated task-completion time horizon measurements.	Tracks how long frontier models can work reliably on realistic tasks.

The important shift is not just that model scores went up. The field also changed what it measures.

By early 2025, older tests were already too easy or too noisy. By mid-2025, the most capable models were being compared on coding agents, difficult science questions, long-context retrieval, multimodal reasoning, and expert-level exams. By 2026, benchmark designers were moving from static test items to interactive environments.

That evolution itself is strong evidence that “AGI” cannot be reduced to one benchmark win.

Benchmarks improved, but benchmarks also got weirder

Since the previous article, model scores have improved across a lot of hard benchmarks.

Frontier systems now perform strongly on:

GPQA
MMMU
AIME
SWE-bench
LiveCodeBench
Humanity’s Last Exam
long-context retrieval tasks
tool-use evaluations
multimodal reasoning evaluations

This is not trivial. These are not toy tasks. Some of these benchmarks test graduate-level science, difficult math, software engineering, visual reasoning, and multi-step problem solving.

But there is a problem.

The better models become, the less stable old benchmarks become as evidence.

MMLU used to be a big deal. Now it is mostly saturated. HumanEval used to be a strong coding signal. Now it is too easy for frontier models. GSM8K used to be a standard math benchmark. Now it does not tell us much about the frontier.

Even worse, some benchmarks contain errors. The paper Are We Done with MMLU? found that a non-trivial percentage of MMLU questions contain mistakes, with some subsets much worse than others.

Stanford HAI’s 2025 AI Index technical performance chapter also notes that traditional benchmarks like MMLU, GSM8K, and HumanEval have been saturating, while newer benchmarks like MMMU, GPQA, and SWE-bench became more important.

So the benchmark story today is mixed.

Yes, models are getting much better.

But also, the measurement problem is getting harder.

Mid-2024 generation snapshot

The following table shows the mid-2024 generation of strong models. These values are useful for historical context because they show how quickly the conversation moved from “chatbot” to “general-purpose assistant.”

Important caveat: these numbers are not perfect apples-to-apples comparisons. Different labs use different prompts, scaffolds, evaluation setups, and reporting conventions.

Model	MMLU	MMLU-Pro	GPQA	HumanEval	BFCL tool use	Long-context example
GPT-4	86.4	-	-	67.0	-	Multimodal text and image input supported
GPT-4o	89.1	74.0	53.6	90.2	80.5	NIAH 100.0
Claude 3.5 Sonnet	89.9	77.0	59.4	92.0	90.2	NIAH 90.8
Llama 3.1 405B	87.3	73.3	51.1	89.0	88.5	ZeroSCROLLS/QuALITY 95.2
GPT-3.5 Turbo	70.7	49.2	30.8	68.0	85.9	-

Sources: GPT-4 Technical Report, The Llama 3 Herd of Models

This table explains why the conversation changed so quickly in 2024. The strongest systems were no longer merely fluent chatbots. By mid-2024, leading models were posting near-saturated MMLU scores, strong code generation, usable tool calling, and credible long-context retrieval.

But even here, GPQA remained much lower than MMLU, which foreshadowed the field’s shift toward harder reasoning benchmarks.

Mid-2025 frontier reasoning snapshot

By mid-2025, the frontier had moved from “models that answer” toward “models that reason with tools.”

Model	LiveCodeBench	Aider Polyglot	SWE-bench Verified	GPQA	Humanity’s Last Exam	AIME 2025	LOFT <=128K	MMMU
Gemini 2.5 Pro	74.2	82.2	59.6 single / 67.2 multi	86.4	21.6	88.0	87.0	82.0
OpenAI o3 high	72.0	79.6	69.1 single	83.3	20.3	88.9	77.0	82.9
OpenAI o4-mini high	75.8	72.0	68.1 single	81.4	18.1	92.7	60.5	81.6
Claude 4 Sonnet	48.9	61.3	72.7 single / 80.2 high-compute multi	75.4 with extended thinking / 70.0 without	7.8	70.5	81.6	74.4
Claude 4 Opus	51.1	72.0	72.5 single / 79.4 high-compute multi	79.6 with extended thinking / 74.9 without	10.7	75.5	-	76.5
DeepSeek R1	70.5	71.6	57.6 multi	81.0	14.0	87.5	-	No native multimodal

Sources: Gemini 2.5 technical report, Claude 4 announcement and benchmark appendix

This table captures the current situation better than any single “AGI score.”

The strongest systems now span a much larger capability bundle than the previous article discussed:

frontier coding
difficult science QA
difficult math
long-context retrieval
multimodal understanding
tool use
agentic workflows

Gemini 2.5 Pro looks especially broad in public vendor reporting. OpenAI’s o3 and o4-mini look especially strong on reasoning-heavy tasks and multimodal perception. Anthropic’s Claude 4 family looks especially strong in coding and tool-using software workflows.

But the caution is important.

Benchmark wins are not stable enough to map directly to AGI claims. Different labs use different scaffolds and compute settings. Anthropic explicitly distinguishes no-extended-thinking benchmarks from benchmarks run with extended thinking and tool use. Google also notes that many non-Gemini results are provider self-reports.

These comparisons are useful, but only as capability indicators. They are not a definitive scoreboard for general intelligence.

The biggest shift: models now think with tools

In the previous article, I talked about chain-of-thought and the question of whether models are really reasoning step by step or just producing text that looks like reasoning.

That question is still open.

But the field has shifted.

The most important change is that frontier models are no longer only “answer generators.” They are increasingly systems that can use tools while reasoning.

Modern frontier models can call Python, browse the web, inspect files, work with images, write and execute code, use external tools, and continue across longer workflows. Some models are explicitly trained to spend more time reasoning before answering. Others allow configurable “thinking budgets,” where more inference-time computation can improve performance.

This is a major shift.

The frontier is no longer just about bigger pretraining runs.

It is now scaling along at least three axes:

Pretraining scale
Post-training and reinforcement learning
Inference-time reasoning and tool use

That third axis is especially important.

A model that can pause, inspect evidence, run code, test a hypothesis, revise its plan, and continue working is meaningfully different from a model that only predicts the next token in a single pass.

But this still does not mean AGI.

It means we are building more capable systems around language models. Some of the intelligence is in the model. Some of it is in the scaffold. Some of it is in the tools. Some of it is in the feedback loop.

That matters because when people say “the model solved it,” we now need to ask:

Did the model solve it from internal reasoning?
Did it solve it through tool use?
Did it solve it through repeated attempts?
Did it solve it because the benchmark was already familiar?
Did it solve it because the environment was carefully scaffolded?

Those are not nitpicks. They are the difference between measuring a model and measuring a whole engineered system.

Long-horizon work is improving, but still limited

One of the most useful ways to think about current AI progress comes from METR’s task-completion time horizon work.

Instead of asking, “What score did the model get on this benchmark?”, METR asks something more intuitive:

How long are the tasks that AI systems can complete reliably?

Their March 2025 paper, Measuring AI Ability to Complete Long Tasks, estimated that frontier models had reached a 50% task-completion horizon of around 50 minutes on their task suite. METR also reported that this horizon had been doubling roughly every seven months since 2019.

That is impressive.

It means the systems are not just answering trivia. They are becoming capable of completing longer, messier, more realistic tasks.

But it also shows the limitation.

A 50-minute task horizon is not the same as a full workday. It is not a week-long project. It is not open-ended research. It is not reliable autonomous operation in the real world.

METR’s later time-horizon measurements continue tracking this trend, but the core conclusion remains: the trend is important, and the current level is still not AGI.

This matches what many software engineers experience in practice. The models are extremely useful. They can write code, review code, explain systems, generate tests, help debug, and accelerate development. But they still need supervision. They still make confident mistakes. They still lose track of constraints. They still require someone with judgment to decide whether the answer is actually correct.

That is not a small limitation.

That is the boundary between assistance and autonomy.

Chain-of-thought is still not a solved window into reasoning

The previous article mentioned research suggesting that implicit chain-of-thought may not always reflect genuine step-by-step reasoning.

That concern has not gone away.

If anything, it has become more important.

As reasoning models become more capable, their intermediate reasoning becomes more valuable for debugging, safety, and evaluation. If a model explains its reasoning, we might be able to catch mistakes, deception, hidden assumptions, or unsafe intent.

But that only works if the reasoning we see is faithful.

The paper LLMs Do Not Think Step-by-step In Implicit Reasoning argues that when models are prompted to use implicit reasoning, they often do not appear to internally compute intermediate steps in a robust way.

Later safety work, such as A New and Fragile Opportunity for AI Safety, frames chain-of-thought as a useful monitoring surface, but also a fragile one.

That creates a strange situation.

The more powerful reasoning models become, the more we want to inspect their reasoning.

But the more strategically capable they become, the less we can naively assume their visible reasoning is complete, faithful, or safe.

This does not mean chain-of-thought is useless.

It means it should be treated as a tool, not as proof.

A model writing a beautiful explanation is not the same as a model having robust, grounded, human-like understanding.

So what is missing?

The missing piece is not raw capability.

Raw capability is clearly improving.

The missing piece is robust generality.

Humans are not impressive because we can memorize internet-scale text. Humans are impressive because we can enter new environments, infer goals, test ideas, learn from sparse examples, adapt under uncertainty, and transfer knowledge across domains.

That is the kind of ability AGI is supposed to capture.

Today’s frontier models are broad, but brittle.

They can be superhuman on some tasks and surprisingly weak on others. They can solve difficult coding problems but fail simple novel puzzles. They can reason through expert-level questions and still hallucinate. They can use tools but also misuse them. They can follow instructions but still lose track of the real goal.

This unevenness is important.

A system that is excellent across many benchmarks is not automatically generally intelligent. It may simply be very powerful across many benchmark-like distributions.

AGI requires more than breadth.

It requires:

transfer
reliability
adaptation
grounded planning
calibration
robustness under unfamiliar conditions

That is where the strongest evidence still says: not yet.

Why this still is not AGI

The cleanest reason is that novelty remains hard.

ARC-AGI-2 was built specifically to preserve human accessibility while increasing compositional difficulty, contextual rule application, and in-context symbol definition. The benchmark authors report that scores below 5% are generally not considered meaningful. Leading systems remained below that threshold.

ARC-AGI-3 raises the bar again in interactive environments, and its paper reports frontier systems below 1% as of March 2026.

If one’s AGI criterion includes fluid intelligence and open-ended adaptation, those numbers matter more than a high score on a saturated benchmark.

The second reason is that benchmark saturation and benchmark quality problems make many impressive scores less informative than they first appear. Stanford’s 2025 AI Index explicitly says traditional benchmarks are saturating. Separate work on MMLU found errors that can mislead evaluation and model comparison.

The third reason is that long-horizon reliability is still limited. METR’s time-horizon framing is helpful because it shows real progress, but also shows that current systems are not yet reliable for long, messy, open-ended work.

The fourth reason is that reasoning transparency remains unsettled. Reasoning models are impressive, but their internal cognition is not understood well enough to use as evidence of full AGI.

The best current description: general-purpose, not generally intelligent

Here is where I would place today’s frontier AI systems:

They are not narrow AI in the old sense.

They are also not AGI.

They sit in a middle category that is becoming more important:

general-purpose AI systems with emerging agentic capabilities.

They can operate across domains. They can use tools. They can reason more deeply than previous models. They can handle text, images, audio, code, long documents, and structured tasks. They can assist professionals in meaningful ways.

But they still do not have the kind of robust, open-ended, human-like adaptability that most serious definitions of AGI imply.

This is why saying “AGI is here” feels premature.

But saying “it is just autocomplete” is also outdated.

The honest position is uncomfortable because it does not fit the usual internet debate.

We are not at AGI.

But we are much closer to something economically disruptive than many people expected.

Bottom line

In the previous article, I treated o3’s ARC result as a milestone, not a destination.

That conclusion has aged well.

The milestone was real. The hype was excessive. The research since then has clarified the picture.

ARC-AGI-1 showed that frontier systems could make surprising jumps.
ARC-AGI-2 and ARC-AGI-3 showed that novelty, interaction, and compositional generalization are still hard.
METR’s work showed that long-horizon task completion is improving quickly, but also that current systems are still far from reliable long-term autonomy.
New reasoning models showed that inference-time computation and tool use matter a lot.
Safety research showed that more capable systems also require more serious evaluation.
Benchmark research showed that our measurement tools are struggling to keep up.

So, are we there yet?

No.

But the road changed.

The question is no longer whether AI can be useful across many domains. That has already been answered.

The question is whether these systems can become robustly adaptive, reliable, and autonomous outside the distributions where we train, scaffold, and evaluate them.

That is the real AGI question now.

And today, the factual answer is:

We are not at AGI yet. We are at increasingly general-purpose reasoning systems that can look shockingly intelligent in many contexts, while still failing in ways that reveal they are not generally intelligent in the human sense.

That may be less dramatic than “AGI is here.”

But it is much more useful.

And honestly, much more interesting.

Sources

Primary sources and technical reports used for this follow-up:

DEV Community