Teaching an AI agent to say no: the feasibility gate

Daniil Pokrovsky — Fri, 24 Jul 2026 04:06:03 +0000

The most expensive thing a coding agent can do is confidently implement the wrong thing.

I learned this the usual way, by pointing an agent at a ticket that said, in full: "Improve the onboarding flow." It ran for forty minutes, touched eleven files, and opened a PR that redesigned a form nobody had complained about. The diff was clean. The tests passed. It was also useless, because the ticket never said what "improve" meant, and the agent, being an agent, picked an interpretation and committed to it.

The PR got closed, the branch got deleted, and I got to explain to myself why I had just paid API prices for work a human would have refused to start.

A human engineer, handed that ticket, does something agents don't do by default: they push back. "Improve how? For whom? Is this the drop-off on step 2 or the confusing copy on step 4?" That pushback is real work. Half the value of a senior engineer is refusing to build until the question is answerable.

So the biggest single fix I've found for unreliable ticket automation is not a better model or a longer prompt. It's an explicit step where the agent is structurally required to say "no, not yet."

What the gate looks like

Before any code gets written, I run the agent once in read-only mode against the formatted ticket and the actual codebase. Its only job is to answer three questions in writing:

Is the requirement clear enough that two reasonable engineers would build roughly the same thing?
Does the change fit this codebase as it exists today, or does the ticket assume architecture that isn't there?
What decisions would I have to invent to complete this, and who should actually make them?

The output is a short markdown file written before any implementation. If the assessment comes back "clear," the pipeline proceeds to implementation and a PR. If it comes back with open questions, it stops and posts them to the ticket as a comment, addressed to whoever wrote it.

Two details matter more than they look.

The gate runs read-only. The assessment pass has no permission to edit files. The reason is incentives, not distrust of the model. An agent that can edit files while assessing will start "just fixing" things to make the ticket feasible, and now your feasibility check has side effects. Most coding agents have a native mode for this (Claude Code's plan mode can't write); use it instead of trusting a prompt that says "please don't change anything."

The questions go back to the ticket, not to a log. A feasibility report that lands in a file on the server is a feasibility report nobody reads. Posting "Before I can implement this, I need answers to: ..." as a tracker comment gets the author to answer, and over a few weeks it trains them. People start writing tickets that survive the gate on the first try, because getting bounced by a robot is mildly embarrassing in a way that getting bounced by a colleague isn't.

What a good rejection reads like

Here's a lightly edited assessment from my own backlog, for a ticket titled "Add rate limiting to the API":

This task is not ready to implement. Open questions:

Which API? The repo exposes a public REST API (src/api/) and an internal webhook receiver (src/webhook-server.ts). Rate limiting the webhook receiver would break the GitHub integration's retry behavior.

Per what? Per IP, per API key, and per account produce very different implementations. The codebase has no per-account request tracking today.

What should a limited request receive? 429 with Retry-After is standard, but the mobile client (per docs/clients.md) does not handle 429 and will surface a generic error.

Recommendation: split into "rate limit public REST per API key, 429 + Retry-After" and a separate client task.

Each question is grounded in something the agent found in the repo, not generic requirement-gathering theater. Question 3 is the kind of thing that becomes a production incident when an agent guesses. And the ticket author can answer all three in two minutes, so the ticket becomes implementable the same day.

The grounding is the whole trick, honestly. "Please clarify the requirements" is noise. "You said rate limit the API, there are two APIs, here are their paths" is a colleague.

Failure modes I hit building this

The gate that approves everything. My first prompt asked "is this task feasible?" and the model, eager to please, said yes to almost anything. Reframing helped: instead of asking for a verdict, ask for the list of decisions the implementer would have to invent. If the list is empty, proceed. If it has entries, those entries are the rejection. Models are much better at enumerating ambiguity than at judging it.

The gate that rejects everything. Overcorrect and the agent starts demanding a spec for one-line copy changes. The calibration that worked for me: a question only counts as blocking if guessing wrong would produce a PR the reviewer rejects. "Which shade of gray" is not blocking. "Which of the two APIs" is.

Trusting the agent to stop on its own. Sometimes the implementation run itself ends with the agent asking "Should I also migrate the old records? How would you like me to proceed?" and then the pipeline, seeing exit code 0, happily commits whatever half-state exists. I now scan the tail of the agent's output for decision-shaped questions ("which option," "should I," "your call") and treat a run that ends on them as a non-run: nothing gets committed, and the questions go to the ticket instead of into a PR. Committing past an unanswered question ships an answer nobody gave.

Cost worries. The gate is an extra model call, but a cheap one: read-only, no edit loop, a few minutes. Against one wasted forty-minute implementation run plus one wasted human review, it pays for itself the first time it fires. On my logs it stops roughly one ticket in five, exactly the tickets that used to produce the PRs I closed without merging.

The general lesson

If you're wiring up ticket-to-code automation, in-house or otherwise: budget as much design effort for the refusal path as for the happy path. The happy path is a demo. The refusal path is what makes it survivable on a real backlog, where a meaningful fraction of tickets are one-line wishes written in a hurry between meetings.

Agents can notice ambiguity fine. What they lack is a moment in the pipeline where noticing it is their entire job, and somewhere for the resulting questions to go. Give them both and everything downstream improves.

What I ended up building

This gate is built into DevIntern, my tool that takes tickets from Jira, Linear, Trello, and four other trackers all the way to reviewed pull requests. Every run starts with this feasibility check; unclear tickets get their questions posted back as tracker comments instead of becoming PRs, and the assessment is saved as feasibility-assessment.md alongside the run's other artifacts.

devintern PROJ-1842 --create-pr