DEV Community

News
News

Posted on

AllenAI releases XYBench: 8115 questions test whether AI assistants catch the wrong question

Infographic cover

๐Ÿ“… November 13, 2026 โ€ข AllenAI (Hugging Face blog)
AllenAI released XYBench, a benchmark of 8,115 questions testing how AI assistants respond to requests based on misconceptions (per AllenAI). The benchmark is built around the XY problem: a person wants to accomplish X, assumes approach Y will work, and asks for help with Y instead of asking about X directly. An example from the post: a user asks how to thicken frosting that keeps sliding off cupcakes, while the real problem is that the cupcakes are still too warm to frost (per AllenAI).
In such requests the stated question describes the assumed approach, and the actual goal sits behind it. The benchmark measures whether an assistant recognizes these cases and addresses the real goal rather than only the request as phrased (per AllenAI). The domains covered are programming and everyday activities such as cooking and home repair (per AllenAI).
XYBench won the Best Resource Paper Award at EMNLP 2026 (per AllenAI). It was developed with researchers at Carnegie Mellon, NVIDIA and Johns Hopkins (per AllenAI).
The composition is documented by source: 1,557 questions come from Stack Overflow, 184 from Stack Exchange, 102 examples were manually curated, and 6,272 questions were generated from WikiHow guides (per AllenAI). The WikiHow-generated part is the largest segment of the benchmark. The dataset card for the released benchmark lists 6,268 WikiHow questions, while the announcement and the EMNLP paper state 6,272. For public forum questions, the reference answers use the three highest-scored human answers plus the accepted answer.
Cases where answering the stated request was sufficient were filtered out, so the retained questions require going beyond the literal request (per AllenAI).
Five models were evaluated: Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro (per AllenAI). The stated finding: the models often suggested a solution but kept answers focused on the original mistaken approach. Unlike the human reference answers, all five were less likely to steer users away from the stated approach toward a solution suited to the actual goal (per AllenAI). The post presents this as a pattern observed across all five evaluated models (per AllenAI).
For reference, the public post does not include numeric per-model scores.
Sources: huggingface.co โ€” aclanthology.org

Key figures

IMPACT checklist


Continue the conversation: https://t.me/AIAutomationArchitecture ยท Bluesky

AI Automation

Top comments (0)