DEV Community

Chandan Kumar
Chandan Kumar

Posted on Originally published at technologyreview.com

METR’s Time Horizon Plot: AI Coding Soars, Judgment Lags

The most misunderstood graph in AI just got a major update—and it reveals a crucial gap: AI models are getting exponentially better at coding, but they still suck at tasks requiring human judgment.

METR, the AI evaluation nonprofit, published its latest time horizon plot, which measures how long an AI can autonomously complete tasks before failing. The plot shows that for software engineering benchmarks, the time horizon has been doubling roughly every seven months since 2021. That’s exponential growth. A model that could only handle tasks lasting a few seconds in 2021 can now handle tasks lasting over an hour.

Where the graph misleads

Most people look at that curve and think, “AI is taking over.” But the update includes a critical nuance: the plot only tracks tasks that have a clear right answer. Coding tasks fit that mold—compile or fail. Tasks requiring judgment, like deciding whether a news article is biased, are harder to automate because the answer is subjective.

METR specifically calls this out: models aren’t quite as good at tasks that require judgment as they are at coding. The time horizon for judgment-based tasks is shorter and improving more slowly. This isn’t a minor detail—it’s the difference between an AI that can write your unit tests and one that can review your PR for design flaws.

What this means for builders

  • If you’re building a coding agent, expect rapid capability gains. The trend suggests models will handle multi-hour coding tasks within a year or two.
  • If you’re building a decision-support tool, don’t assume the same trajectory. Judgment tasks require more data, better benchmarks, and likely new architectures.
  • When evaluating models, look at task-specific benchmarks, not aggregate scores. Aggregate plots hide the judgment gap.

The practical takeaway

The exponential curve is real, but it’s not uniform. Coding is the low-hanging fruit. Judgment is the hard wall. If you’re planning your AI stack, invest in coding automation now, but keep humans in the loop for any task that requires nuance. The graph update doesn’t change that reality—it just makes it measurable.

Source: MIT Technology Review

Source: https://www.technologyreview.com/2026/11/20/1149399/the-most-misunderstood-graph-in-ai-just-got-a-major-update/

Top comments (0)