DEV Community

Cover image for Blaming AI? What If Your Product Is the Problem?
Wassim Chegham
Wassim Chegham

Posted on Originally published at linkedin.com

Blaming AI? What If Your Product Is the Problem?

When an AI coding agent struggles with a product, my first question is not which model we should switch to. It is: what did we make unnecessarily difficult? A missing setup step, an unclear command, or a tool response that says “failed” without explaining why can leave the agent guessing. Before reaching for another model, I want to understand whether the product gave it enough information to make the right decision.

Agents make mistakes, and their capabilities matter. But treating every failure as a model problem overlooks the other half of the interaction. Agent Experience (AX) is the experience an agent has when discovering a product's capabilities, understanding its interfaces, completing work, and recovering from errors. It extends Developer Experience rather than replacing it: humans still define goals, review results, and decide what is safe to ship.

TL;DR

  • AX is a product responsibility: making a tool available does not establish that agents can use it reliably.
  • Use Scope to test it: define a realistic task, inspect the evidence, and evaluate a targeted improvement.
  • Start with one workflow: test the experience behind a product promise, then use the findings to decide what to change.

Access is not the same as usability

Experienced developers quietly compensate for incomplete documentation. We recognize familiar errors, fill in missing steps, and spot outdated examples. An agent may expose those gaps differently. That does not make every failure evidence of bad design, but it makes the failure worth investigating.

If developers have to fill gaps in your documentation, agents may struggle with those same gaps.

A product can expose tools through the Model Context Protocol (MCP) and still leave an agent unsure which tool to choose, what an argument means, or whether an operation succeeded. Consider a hypothetical deployment tool that accepts a project name and returns “deployment failed.” Did authentication expire? Is the project missing? Did the build fail? A useful response distinguishes those cases, explains whether anything changed, and identifies the next step, including when a person must intervene.

I would apply the same thinking to a command-line interface (CLI), a software development kit (SDK), or an agent skill: reusable instructions for particular tasks. Required inputs, evidence of success, and permission boundaries should be explicit. More instructions are not automatically better; they need to resolve the uncertainty at the decision where it arises. Good AX helps agents act correctly, not simply do more.

Use Scope to investigate

To evaluate how coding agents use your product, use Scope. It is an open-source agentic experience evaluation platform that runs existing coding agents against tasks you define, evaluates their output against reusable criteria, and preserves evidence you can inspect. The question becomes: “Can agents use our product, where do they struggle, and did our change help?”

Saved, versioned profiles capture agent configurations such as models, skills, and MCP servers. Versioned codebases provide a known starting workspace. Logs, generated files, snapshots, and results for each criterion help explain the outcome. The Portal, CLI, and REST API support different workflows, but the underlying value is the same: a shared investigation rather than a conclusion based on whoever watched the last demo.

I would use the platform alongside existing tests, not instead of them. Tests can verify that an integration works; the execution evidence helps explain how an agent got there using your documentation and tools. Two passing outputs may conceal very different experiences: one straightforward, the other full of retries and wrong turns. That distinction can reveal friction a final score alone would miss.

Can an agent deliver on your marketing promise?

The connection to marketing is practical. A compelling product page can get an SDK considered, but incomplete instructions can derail adoption. Being discoverable is not the same as being usable. Engineering, developer relations, product marketing, and product owners therefore share an interest in the workflow behind the promise.

Imagine your product says, “Build your first integration in minutes.” I would turn that workflow into a representative task in Scope, supply the relevant documentation and tools, and define checks we can verify: the project builds, the integration returns the expected response, and a documented error case is handled correctly. The time claim needs separate timing evidence with clear start and stop boundaries; passing those checks would not prove it.

Marketing makes the promise. AX helps agents deliver on it. Scope tests whether they can.

Product marketers do not need to operate the platform themselves. Bring the promised workflow, the instructions customers receive, and a clear description of success. Engineers can configure the evaluation; together, you can investigate whether the interface, documentation, or message needs changing. This tests a product experience, not campaign conversions, sales uplift, brand preference, or AI-search visibility.

Turn evidence into a product decision

I would start with a question the team can act on: should we fix an onboarding step before promoting the workflow more widely? Choose an owner, a review point, and a provider-usage budget. A bounded investigation is easier to evaluate than an open-ended platform rollout.

For the comparison, hold the task, starting codebase, and agent configuration steady while changing one relevant surface. Keep criteria focused on outcomes, not your preferred implementation route, unless that route is what you are testing. A clearer example, better command help, or a more informative error may be the right fix.

For a product owner, the result should become an actionable issue: connect a failed criterion to execution evidence, identify the suspected friction, and write an acceptance criterion for the change. Prioritize it alongside customer feedback and product goals, not by score alone. Preserve useful tasks as regression evaluations so later changes can be checked against failures you already understand.

I would not claim improvement from one successful run. Agent behavior varies, criteria can miss important details, and automated judges can be wrong. Repeat evaluations and review the evidence. Results describe the tasks and configurations tested, not every agent or customer. Include error cases and permission boundaries, not just the happy path: can the agent finish correctly, and behave appropriately when it cannot?

Your first evaluation

If your team already has a Scope deployment, start in the browser, no local installation is needed. The first-run walkthrough explains how to select or create a project, define a task and criteria, choose an available agent configuration, and submit a request.

If you need to run the platform yourself, follow the local setup guide. The default Copilot setup requires Git, Node.js, pnpm, Docker Compose, mkcert, GitHub CLI, an active Copilot entitlement, and access to the evaluation models. Provider usage may incur charges or consume quotas; installing the CLI alone does not deploy the platform.

Agree on permitted data and credentials before running tasks. The local development stack is not a security sandbox: use a dedicated environment for untrusted work and do not expose it to the internet. Keep the first evaluation narrow enough that someone can inspect the result and decide what it means.

Try one workflow, then share what you learn

Try Scope on one workflow your product already promises to support. Use the access guide for an existing deployment or the local setup guide above. Inspect the evidence, make one targeted change, and evaluate again.

Contributing is optional. Engineers can bring reproducible failures, tests, or fixes; developer advocates, product marketers, and product owners can bring realistic workflows and clearer success criteria. The contributor guide explains how to share what you learn.

Conclusion: investigate before you blame

I want agent usability to become an ordinary part of product review, not something we investigate only after a failed demo. A better model may help, but it does not remove our responsibility to provide clear interfaces, useful errors, and a workable path from intent to outcome. The question is not whether the agent or the product deserves the blame. It is what the evidence tells us to improve.

Scope gives teams a practical way to investigate that question together. Pick one workflow, examine where it breaks down, and test whether a targeted change helps. Before switching models again, find out what your product could make clearer.

Top comments (0)