Coding agents can write a pull request, run the tests and fix what breaks. Yet someone still watches them. The reason isn't that they write bad code. It's that they can't tell, before they act, what acting will do.
This post is about a kind of model we think is missing from almost every agent stack: a consequence model. It takes the state of the world and an action an agent is about to take, and returns one thing: the probability that the action works, and how it's likely to fail if it doesn't.
Every step is a bet
An agent works in a loop: read the context, decide, act, look at what happened, check, repeat. Each step can go wrong, and errors compound. If each step succeeds 95% of the time, a 20-step task succeeds about 36% of the time.
Now add a check that catches most bad steps before they land, so the agent can retry. Lift each step to 99% and the same 20-step task succeeds about 82% of the time. The agent didn't get smarter. It stopped walking off cliffs.
| Steps in the task | 95% per step | 99% per step |
|---|---|---|
| 10 | 60% | 90% |
| 20 | 36% | 82% |
| 40 | 13% | 67% |
Chance of finishing a task when each step succeeds 95% or 99% of the time. This is arithmetic (p to the power n), not a benchmark.
That's why long agent tasks are mostly a reliability problem. Catching errors before they happen moves the line further than producing slightly better answers.
Why a bigger language model doesn't fix it
A language model learns from text: what people wrote about the world. That's a record of what people say happens, not of what does happen when you act.
There's a precise version of this idea. Judea Pearl's ladder of causation separates seeing (what's observed) from doing (what happens if I intervene). The Causal Hierarchy Theorem shows that data about seeing almost always leaves questions about doing undetermined. Text gives models secondhand descriptions of interventions, so this is a structural limit rather than an absolute one. But it's why asking a language model "will this work?" mostly tells you how plausible the action sounds.
Confidence is the second problem. A model's stated confidence is hard to trust, and OpenAI's GPT-4 technical report showed the pre-trained model's probabilities were well calibrated and the post-trained model's much less so. An agent can't set a threshold on a number it can't trust.
What a consequence model is
Think of a friend who has watched thousands of people touch a hot stove, and tells you, before you reach out, how likely you are to get burned and how.
-
Goes in: what the agent is about to do, and where. For example:
git push --forceon this repository's main branch. - Comes out: how likely it is to work, and how it would fail: 7%, would overwrite teammates' commits.
Three properties make it useful:
- It reads and doesn't write. It returns a probability and a short label instead of generating an essay, so one call is cheap and fast enough to make on every step.
- It's calibrated. When it says 80%, the action works about 80% of the time, so a program can set thresholds: run above 0.9, ask a person between 0.5 and 0.9, block below.
- It's trained on outcomes, not descriptions. That's the part a general language model can't copy.
The shape isn't new: some decision models already return probabilities over a fixed set of options instead of text. A consequence model points the same idea at a different question: not "which of these is it?" but "what happens if I do this?"
The training data labels itself
New kinds of model tend to appear when a new kind of training signal becomes cheap. Image models followed millions of labelled photos. Language models followed the web. Chat models followed human preference ratings. Reasoning models followed answers that can be checked automatically, like math and code.
Agents now produce another one. Every test run, CI job, shell command, API call and reopened support ticket is an action with a recorded result. Nobody has to label it: the outcome is the label. Rerun a flaky test and you learn it was flaky. Merge a patch and you learn whether CI passed.
The loop a consequence model sits in (the thresholds are examples; each team sets its own):
- Propose: the agent picks its next action.
- Predict: the consequence model returns P(works) and the likely failure.
- Gate: run above 0.9, ask a person from 0.5 to 0.9, block below 0.5.
- Learn: what actually happened becomes the next training example.
What it makes possible
Agents you can leave alone
If an agent pauses only on the few steps it's unsure about, a person stops supervising one agent and starts clearing many agents' exceptions in the morning. Overnight runs become practical, because the risky steps wait for a human and the rest don't.
Selling outcomes instead of tokens
Intercom prices its Fin support agent at $0.99 per outcome (its pricing page, read October 11, 2026). A vendor can only take on that risk if it can predict which jobs it will complete. A consequence model makes that prediction possible for other kinds of work: quote a price per merged pull request, and accept the jobs you expect to land.
Search before acting
MuZero, a game-playing system, plans by searching many moves ahead inside a model of the game that it learned itself. With a model of consequences, an agent can do the same with deploys, migrations and refactors: weigh many plans, then run the one most likely to work.
Agents closer to production
Refunds, configuration changes, deploys and database changes are where agents would be most useful and are least trusted. A calibrated go or no-go on each action means people approve the risky tail instead of every step.
Where it pays first
Before any of that, there are narrow uses that are worth it on their own:
- "Is this CI failure mine?" A coding agent's push fails. Was it the change, or a flaky test? Today the agent either reruns a pipeline that takes many minutes or starts fixing a test it never touched. Every rerun labels the answer.
- Patch triage. An agent writes five candidate patches. Run the full test suite only on the one most likely to pass, and run its likely failures first.
- Auto-send for support replies. Predict whether a customer will reopen or escalate after an AI-drafted reply. Send the confident ones and hold the rest for a person.
- Routing by measured success. Pick the model with the lowest cost per expected success, instead of guessing how hard a request looks.
What exists today
Parts of this are in research and open source:
- Qwen-AgentWorld is a language world model that simulates tool calls, terminals, web pages and operating systems, with open weights for one of its sizes.
- WebDreamer has a model imagine the result of each web action before the agent takes it.
- R2E-Gym trains verifiers that judge coding-agent patches with and without running the tests, and finds the two work best together.
- SWE-RM studies verifiers that score patches without running them, and argues that calibration matters as much as accuracy.
What we haven't found is the piece in between: a small, calibrated model, trained on real outcomes, cheap enough to call on every step and offered as an API.
What could go wrong
- Outcomes can be specific to one system. A model trained on other people's repositories may not know yours. Conditioning on the environment and adapting per team is its own research problem.
- Calibration has to hold on real work, not only on benchmarks. A model that's confidently wrong is worse than no model, because people stop checking.
- Decision models can latch onto the wrong cue. Recent research found a constrained decision head following an option's name rather than the rubric attached to it. Consequence models will need the same scrutiny.
Why we're writing about it
ScaledThought runs inference for agents. Our view is that many of the next useful models won't be bigger language models, but smaller ones that each do one job well: decide, point to the source, estimate a number, predict a consequence. Consequence models are the one we think matters most, because they turn an agent from a tool someone operates into work you can hand off.
One note on data: we don't store prompts or completions, so nothing in this post describes training on customer traffic. A model like this would learn from outcomes that the people running agents choose to share.
If you're building in this direction, or you'd use a consequence model if one existed, we'd like to hear from you: hello@scaledthought.com.
Originally published at scaledthought.com/blog/consequence-models. I'm a founder of ScaledThought. Intercom and Fin are trademarks of Intercom; ScaledThought isn't affiliated with Intercom.
Top comments (0)