I Tried to Measure That One Weird Failure Mode in [LLMs / Agents]

The failure mode I couldn't stop thinking about

Every time I use [a model / an agent] for [multi-step reasoning / code generation / tool use], I notice the same thing: [describe the failure in one or two sentences, e.g. "it confidently skips a step and the final answer looks right but isn't"].

It's easy to say "that happens sometimes." It's harder to say how often, under what conditions, and whether it's getting better. So I decided to build a benchmark for it.

What I built

• Task design: [what each test case asks the model to do]
• What counts as a failure: [your scoring rule, kept as objective as possible]
• Dataset size: [number of cases and how you created them]
• Tooling: [Kaggle Benchmarks / Python / etc.]

How I measured it

1. Wrote [N] test cases that isolate the failure mode
2. Ran them across [models you tested]
3. Scored each response using [exact match / rubric / programmatic check]
4. Repeated runs to check for variance

What I found

Model	Pass rate	Notes
[Model A]	[x%]	[observation]
[Model B]	[x%]	[observation]

The most surprising result: [your main finding].

What I learned

• Measuring a failure is very different from noticing one
• [A lesson about test design or scoring]
• [A lesson about the models themselves]

What I'd do next

• Expand the dataset to cover [edge cases]
• Test newer models as they're released
• Try [a mitigation, such as prompting or tool changes] and measure whether it helps

Try it yourself

[Link to your Kaggle notebook or repo]

If you've run into a failure mode of your own, I'd love to hear about it in the comments.