Why Your AI-Generated Code Passes Tests but Fails Production

Ashwini Kumar — Mon, 18 May 2026 16:06:26 +0000

AI coding assistants are fast — but fast in the wrong direction is just a faster way to fail. Here's how specs and BDD scenarios turn AI into a reliable executor, not a fast guesser.

The views and opinions expressed in this article are solely our own and do not represent those of our employer, Amazon, or any of its affiliates. All content is based on publicly available information and our personal experience. Any references to products or tools are based solely on publicly available information. No confidential or proprietary information has been disclosed.

AI coding assistants have changed the economics of software development. A well-prompted AI can produce in an hour what used to take a day. Teams are shipping faster than ever. And a growing number of those teams are shipping the wrong thing faster than ever.

The problem isn't the AI. The problem is what you give it to work with.

AI Is Only as Good as Its Context

When you ask an AI to implement a feature with a vague prompt, it does exactly what you'd expect — it makes reasonable assumptions, fills in the gaps with general software patterns, and produces code that looks correct, compiles cleanly, and is wrong for your domain in ways that won't surface until production.

This isn't a flaw in the AI. It's a structural reality. AI generates code based on pattern recognition. It doesn't know your business rules. It doesn't know that a missing rent estimate should return a score with confidence: "low" instead of throwing an error. It doesn't know that negative cash flow shouldn't automatically fail a deal — it should reduce the score by a maximum of 30 points. It fills those gaps with the most statistically likely answer, which is plausible, passes a surface-level review, and violates your domain logic.

The solution isn't better prompting. It's giving the AI a contract to implement against.

That contract is a spec. The mechanism that enforces it is a BDD scenario suite. Together, they transform AI from a fast guesser into a fast executor — and that distinction is everything.

The Contract: Spec-Driven Development

A spec is not a PRD. It's not a Jira ticket. It's a precise, unambiguous description of what a system must do — inputs, outputs, rules, edge cases, error states — written before any code exists.

When an AI reads a spec, it stops guessing. Every rule is explicit. Every edge case is named. The AI has no gaps to fill because the spec filled them first.

Feature: Deal Score Calculation
  Input: property price, rent estimate, assumptions (rate, down payment, taxes)
  Output: score 0–100, grade A–F, confidence level

  Rules:
  - Grade maps: 80+ → A, 65–79 → B, 50–64 → C, 35–49 → D, <35 → F
  - Missing rent estimate → score still computed, confidence: "low"
  - Negative cash flow reduces score by max 30 points, not automatic failure
  - Price above $800k → flag "high price tier", no score penalty

Compare what an AI produces with and without this spec:

Prompt without spec: "implement deal score calculation"

AI output:
- Returns 0 for missing rent estimate          ← wrong
- Fails immediately on negative cash flow      ← wrong
- No price tier flagging                       ← missing
- Passes all generic unit tests                ← tests share the same blind spots

Prompt with spec: "implement deal score calculation per this spec: [spec above]"

AI output:
- Handles missing rent estimate, returns confidence: "low"   ✓
- Caps cash flow penalty at 30 points                        ✓
- Flags high price tier without penalizing score             ✓
- Grade mapping matches spec exactly                         ✓

Same AI. Same tool. The spec is the only variable. The spec didn't slow the AI down — it directed it. The AI's speed is now working for you instead of against you.

The Guardrail: BDD Scenarios

The spec tells the AI what to build. BDD scenarios tell the AI — and your CI pipeline — whether it built it correctly.

BDD scenarios are written in plain language (typically Gherkin) that describes system behavior from the outside. They are independent of implementation. The AI cannot game them by writing tests that match its own code, because they were written from the spec before the code existed.

Scenario: Missing rent estimate
  Given a property priced at $450,000
  And no rent estimate is available
  When the deal score is calculated
  Then a score is still returned
  And the confidence level is "low"

Scenario: Negative cash flow
  Given a property with negative monthly cash flow of -$400
  When the deal score is calculated
  Then the score is reduced by no more than 30 points
  And the deal is not automatically failed

Scenario: High price tier
  Given a property priced at $850,000
  And a monthly rent estimate of $4,500
  When the deal score is calculated
  Then the "high price tier" flag is present
  And the score is not penalized for price alone

When you give an AI these scenarios alongside the spec, it has a verifiable definition of done. It can run the scenarios against its own output, catch failures, and self-correct before a human ever sees the code. The scenarios are not just tests — they are the AI's feedback loop.

This is why BDD scenarios are more valuable in an AI-driven workflow than traditional unit tests. Unit tests written by AI share the same assumptions as the code AI writes. BDD scenarios written from the spec are assumption-free. They describe behavior, not internals.

AI as a Spec Stress-Tester

The relationship between AI and specs isn't one-directional. AI is also one of the most effective tools for finding gaps in a spec before implementation begins.

Give an AI your spec and ask it to break it:

Prompt: "Given this spec, what edge cases are missing?

Rules:
- Score is 0–100
- Missing rent estimate → confidence: 'low'
- Negative cash flow reduces score by max 30 points"

AI response:
- What if both rent estimate AND cash flow data are missing simultaneously?
- What if price is $0 or negative — validation error or score of 0?
- What if cash flow is exactly $0 — is that negative or neutral?
- What if compounding penalties push the score below 0?
- Is there a maximum rent estimate that should be flagged as suspicious input?

Every one of those questions is a potential production bug. Answered in the spec, they cost nothing. Discovered after deployment, they cost a sprint, a hotfix, and a post-mortem.> AI-assisted spec review is the highest-leverage use of AI in the entire development cycle — because it eliminates bugs before a single line of code is written.

The Full AI-Driven Workflow

Here's what a feature cycle looks like when AI, specs, and BDD scenarios work together:

Day 1 — Spec authored by humans, stress-tested by AI
  → Tech lead, PM, and senior engineer write the spec
  → AI is asked: "what edge cases are missing from this spec?"
  → Gaps are resolved before implementation begins
  → Spec committed to the repo alongside the code it governs

Day 2 — Scenarios written from the spec
  → Engineers and QA convert spec rules into BDD scenarios
  → AI generates adversarial scenarios: "write scenarios that would
     catch a naive implementation"
  → Every rule has at least one scenario; every edge case has one too

Day 3 — AI implements against spec + scenarios
  → AI is given spec and scenarios as context
  → AI implements to pass scenarios, not to satisfy a vague description
  → AI runs scenarios locally, self-corrects, opens PR with green scenarios

Day 4 — Human review, focused on architecture
  → Reviewer sees passing scenarios — behavioral correctness is already verified
  → Review focuses on approach, performance, and maintainability
  → Review is faster because the question "does this do the right thing?" 
     is already answered

Day 5 — CI enforces the contract forever
  → Scenarios run on every commit
  → Any future AI-generated change that breaks behavior fails the build
  → The spec is permanently enforced without anyone having to remember it

The human's role in this workflow shifts toward high-judgment work: writing specs, reviewing architecture, resolving edge cases. The high-volume work — implementation, test generation, gap analysis — goes to AI. That's the right division of labor.

Why AI-Generated Tests Are Not Enough

A common mistake teams make when adopting AI tools is asking the AI to write tests alongside the implementation. This feels productive. It produces coverage numbers. It doesn't protect you.

AI-generated tests share the same assumptions as AI-generated code. If the AI misunderstood the spec (or there was no spec), the tests will validate the wrong behavior confidently. You end up with a green test suite that proves the code does what the AI thought you wanted, not what you actually needed.

BDD scenarios written from the spec before implementation are the only tests that are truly independent of the AI's assumptions. They describe what the system must do from the outside. The AI cannot write them correctly without the spec — and if it tries, the gaps in the spec become immediately visible.

// AI-generated unit test (no spec):
it('returns 0 when rent estimate is missing')  // ← validates wrong behavior

# BDD scenario (from spec):
Given no rent estimate is available
When the deal score is calculated
Then a score is still returned        # ← catches the wrong behavior
And the confidence level is "low"

The BDD scenario catches exactly what the AI-generated test missed. That's not a coincidence — it's the structural difference between tests that describe implementation and tests that describe behavior.

The Compounding Advantage

The real payoff of this approach isn't in a single feature cycle. It's what happens to the codebase over time.

Every feature built with a spec and BDD scenarios adds a permanent behavioral contract to the codebase. Six months later, when an AI is asked to refactor the scoring engine, the scenarios tell it immediately whether the behavior survived. The AI doesn't need to read the original spec, find the original engineer, or guess at intent. It runs the scenarios. They pass or they don't.

This is the compounding advantage: every spec and scenario written today makes every future AI interaction with that code safer and faster. The codebase becomes a set of contracts that AI can execute against with confidence, rather than a codebase it has to interpret and guess at.

Teams that build this way are not just shipping faster today. They are making every future sprint faster too.

The Practical Setup

The tooling to run this workflow end to end exists today:

Spec files: Markdown committed to the repo, co-located with the code they govern. AI reads them as context on every implementation prompt.

Scenario files: Gherkin syntax, run by Cucumber (JS/Java/Ruby), pytest-bdd (Python), or SpecFlow (.NET). Committed to the repo, run in CI on every commit.

AI tools: Claude Code, GitHub Copilot, Cursor — any assistant that accepts file context. The spec and scenario files are the context. Quality of output scales directly with quality of spec.

CI enforcement: GitHub Actions, AWS CodePipeline, or any CI system. Failing scenario blocks the merge. No exceptions, no overrides.

The spec and scenarios are not documentation overhead. They are the interface between human intent and AI execution. Remove them and AI is fast and unpredictable. Keep them and AI is fast and contract-bound.

The Rule That Makes It Work

One rule, enforced without exception:

🚫 No scenario, no code.

If a behavior isn't described in a scenario, it doesn't get implemented — by a human or an AI. If the AI wants to handle a case the spec didn't cover, the scenario gets written first. That either updates the spec or surfaces a disagreement that gets resolved before the code exists.

This rule feels like friction. It is friction — once, upfront, where it costs almost nothing. It eliminates the friction that compounds: the rework, the regression, the hotfix, the post-mortem, the sprint lost to "that's not what I meant."

Closing

AI coding tools are not going to slow down. The teams that figure out how to direct them precisely will compound their advantage over the teams that are still prompting vaguely and reviewing hopefully.

The spec is the precision. The BDD scenarios are the enforcement. Together they give AI exactly what it needs to stop guessing and start executing — against a contract that humans defined, scenarios verify, and CI protects on every commit forever.

Fast AI with no contract is a liability. Fast AI with a contract is a force multiplier.

Write the spec first. Write the scenarios second. Let the AI do the rest.

About the Authors

**Written by Ashwini Kumar and Malcolm Jones — Software Engineers @ Amazon.

The views and opinions expressed in this article are solely our own and do not represent those of our employer, Amazon, or any of its affiliates.

DEV Community: Ashwini Kumar