XYBench: Can AI Assistants Answer the Question Behind the Question?

type

Post

status

Published

date

Nov 14, 2026

slug

summary

XYBench: Can AI Assistants Answer the Question Behind the Question?

Good assistants are usually evaluated on how well they answer a clearly formed question. Real conversations are messier. People often arrive with a wrong assumption already baked into their request, and an assistant that answers the literal question may leave the underlying problem untouched. XYBench, a benchmark released by the AllenAI team together with researchers at Carnegie Mellon, NVIDIA, and Johns Hopkins, is designed to measure exactly this gap. It contains 8,115 real-world questions whose requests rest on misconceptions, and it tests whether models can spot those misconceptions and steer users toward what they actually need. The work received the Best Resource Paper Award at EMNLP 2026, and the [paper](https://arxiv.org/abs/2609.06842), [dataset](https://huggingface.co/datasets/allenai/XYBench-dataset), and [code](https://github.com/allenai/XYBench) are all publicly available.

The XY Problem

The benchmark takes its name from a familiar failure pattern in technical support and everyday advice-seeking. Someone wants to accomplish a goal, call it X, decides in advance that a particular method, Y, is the way to get there, and then asks for help with Y. If Y is a poor fit for X, an answer about Y can be perfectly detailed and still useless.

The benchmark's own example shows the trap. A baker asks how to thicken frosting that keeps sliding off cupcakes. The stated request is about frosting consistency, but the real issue is that the cupcakes are still too warm to frost at all, and no thickening trick will help until they cool. Because the baker is focused on making the frosting stick, the temperature problem never enters the question. A genuinely helpful assistant has to look past Y and address X.

Building the Benchmark

XYBench spans programming as well as everyday activities such as cooking and home repair. Its 8,115 questions come from four sources: 1,557 from Stack Overflow, 184 from Stack Exchange, 102 curated by hand, and 6,272 generated from WikiHow guides.

Every question is annotated with two pieces of information: the stated request (Y), meaning what the person explicitly asks for, and the intended request (X), the underlying goal that can be inferred from context.

Construction differed by source. For questions from public forums, the team relied on the three highest-scored human answers along with the accepted answer, where one existed, to establish the intended request and possible solutions. Cases where answering the stated request would have been sufficient were filtered out. For the WikiHow portion, Claude Sonnet 4.5 generated questions around plausible misconceptions embedded in procedural guides. These were then refined so that the surrounding context gives an expert enough information to infer the underlying goal without stating it outright, and examples where the literal request would largely resolve the problem were removed.

Measuring Whether an Answer Actually Helps

The authors illustrate the difference with a hypothetical exchange about a vacuum cleaner. A model might spend several paragraphs recommending a more powerful machine, then add a brief line suggesting a check for a clogged hose. If the clog is what is weakening the suction, clearing it could solve the problem outright. A more helpful reply would recognize that the user's real aim is getting the suction back and put its emphasis on diagnosing the cause.

Telling these responses apart requires a form of understanding studied in pragmatics, the area of linguistics that examines how context and communicative intent shape what an utterance means. XYBench turns this into three concrete measurements for every response:

Pragmatic solution: whether the response contains a solution to the stated request (Y) and a solution to the intended request (X), with each checked separately.

Pragmatic emphasis: whether the answer centers on the stated request, centers on the intended request, or balances the two.

Misconception identification: whether the response steers the user toward the underlying goal while discouraging, or at least avoiding, the mistaken approach.

Claude Sonnet 4.5 serves as the automated judge. It classifies the individual suggestions inside each response and assesses its pragmatic emphasis. To check the judge's reliability, two human reviewers independently assessed 500 model responses. Averaged across the reviewers, agreement with the judge reached 97.5–100% on whether a solution appeared, 84.6–93.3% on misconception identification, and 74.2–79.8% on pragmatic emphasis.

What the Experiments Show

Five models were evaluated on XYBench: Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro. The evaluation used a single-turn setup, with one response per question, and human reference answers were scored separately using the same metrics.

The pattern is consistent across all four data sources: every model was more likely to include a solution to the stated request. In practice, the models often did propose something useful for the user's problem, but kept their answers anchored to the original, mistaken approach. Compared with the human reference answers, all five models were less likely to pull users away from that approach and toward a solution matched to their actual goal.

In short, current models tend to answer the question as asked rather than the problem behind it.

Why It Matters

People bring very different levels of knowledge to AI systems, and they do not always know the best way to approach a problem, or how to phrase the question in the first place. An assistant that only rewards well-formed requests serves the users who need it least.

The team hopes XYBench gives researchers a way to measure whether model improvements translate into assistants that can recognize mistaken assumptions, explain why a proposed approach might not work, and suggest more effective alternatives. Those are the moments when an assistant is truly useful, even when the user does not know the right question to ask.

Original article: Hugging Face, https://huggingface.co/blog/allenai/xybench