🪙 Anthropic study: a coin flip inside the agent shifted expert decisions
📅 29 October 2026 • The Decoder
Anthropic published a study of how an AI agent's stated position influences human expert judgment. 50 experts in causal inference worked through a realistic analytical task. At each of 5 decision points, a coin was flipped to assign which side the agent would argue, and the agent then argued that assigned side honestly, regardless of how the coin landed. The point of the design is that the agent's argument quality stays constant while only its assigned position changes, so any shift in human decisions can be attributed to the position itself rather than to the reasoning behind it.
The measured effect was large. Participants adopted the agent's position in 86 percent of the four analytical decisions, against 50 percent expected under pure chance. Net flip rates, meaning the gap in the share of participants taking the agent's side between groups shown opposite coin outcomes, ranged from 62 to 82 percentage points across tasks. The final recommendation differed by 17 percentage points between groups that saw opposite coin outcomes on the same problem. The paper notes this 17-point shift did not reach statistical significance under the pre-registered primary measure, while the 86 versus 50 percent effect on the four analytical decisions did. Because the side assignment was random, neither the difficulty of the task nor the strength of the arguments can explain the gap between the two groups.
Three plausible moderators showed no measurable difference: years of experience, self-assessed expertise, and stated trust in AI. Participants also completed the task faster and with fewer serious errors when working with the agent, so the influence operated alongside a genuine performance benefit rather than replacing it. The authors present both effects together, which matters because a speedup alone would have suggested shallower reviewing instead of a genuine benefit.
IMPACT: One model's randomly assigned position can shift a whole team's conclusion, which the authors describe as a false-consensus risk.
The practical implication for AI-assisted analysis workflows is concrete. When a team asks one model for a view, the model's first framing carries the weight of a coin flip with downstream consequences. Countermeasures include requiring an independent first-pass analysis before the model's view is shown, running an explicit devil's-advocate pass with a second model, and rotating models across decision rounds so no single framing anchors the group. The same logic applies to any repeated decision round in which one model speaks first.
Sources: the-decoder.com
Continue the conversation: https://t.me/AIAutomationArchitecture · Bluesky
Top comments (0)