Your AI Agent Passed the Tests. Did It Build the Product?

jlcases — Tue, 25 Aug 2026 15:59:17 +0000

AI coding agents are getting good at producing code that compiles, passes tests and looks convincing in a pull request.

That is useful. It is not enough.

A product team does not ship “a green test suite”. It ships a specific outcome for a customer who pays. That outcome started as a PRD, became user stories, acceptance criteria, security constraints and product decisions. The hard part is making sure those things still mean the same thing after an agent has touched the code fifty times.

The failure mode is specification drift

An agent starts with a clear request.

Then it encounters an awkward dependency, a failing test or a missing detail. To keep moving, it changes an implementation choice. Later it adapts a test. Later still it narrows the behaviour until the test passes.

Nothing necessarily looks alarming in a diff. Every local step can seem reasonable. But the product may no longer do what was requested.

That is specification drift.

The agent did not need to be malicious or “hallucinate” for this to happen. It only needed permission to optimise for the evidence nearest to it: the files, tests and commands currently in context.

For a throwaway prototype, that may be acceptable. For a customer-facing product, it is expensive.

Passing tests is execution evidence, not product evidence

Tests tell us something important: a defined check passed.

They do not automatically tell us that the original user story was delivered, that security constraints remain intact or that an agent did not quietly reinterpret the requirement to make the check easier.

I find it useful to separate two questions:

did the agent successfully execute the work
did the work still satisfy the product specification

The first question is about tool calls, diffs, builds and tests.

The second is about traceability. Which PRD requirement does this change serve? Which acceptance criteria were verified? What constraints must remain true? What changed when reality forced a trade-off?

Without that link, human reviewers spend their time reconstructing intent from code after the fact. That is where a large part of the cost goes.

The real bottleneck is validation

People often describe AI coding as a generation problem.

In practice, the slow part is validation.

An agent can produce a huge amount of code quickly. Someone still has to decide whether the code matches the product, whether a test is testing the right thing, whether an exception was justified and whether the implementation introduced a security or usability regression.

If those decisions only exist in a chat window or in somebody’s head, they disappear as soon as the next context window begins.

Then the team repeats the same work: reading, guessing, asking, checking and repairing.

Product work needs an evidence trail

A reliable AI software workflow should preserve more than code.

For every meaningful change, the team should be able to follow a short chain:

the requirement or user story
the implementation decision
the work the agent performed
the verification that proves the relevant outcome
any unresolved risk or explicit trade-off

That does not mean turning development into bureaucracy. It means keeping the product contract visible while agents operate.

The goal is not to stop agents from making decisions. It is to make those decisions inspectable, reversible and accountable to the original intent.

What I am building

This is the problem behind PAELLADOC.

The aim is to turn specifications, PRDs and user stories into a controlled product-building workflow where agents can work quickly without quietly rewriting the brief. Code, decisions, security constraints and verification should remain connected all the way to a deliverable a real client can use.

The useful standard is not “the agent finished”.

It is “the product we promised is demonstrably the product we built”.

Feedback is very welcome, especially from teams using coding agents on real client work.

The Evolution of Documentation in the AI Era: A CPTO's Perspective

jlcases — Sun, 13 Apr 2025 17:58:35 +0000

As a CPTO deeply immersed in the AI transformation of software development, I've observed a fascinating shift in how we approach technical documentation. The traditional documentation paradigm is breaking under the weight of modern development practices, and it's time we address this head-on.

The documentation crisis nobody talks about

Let's be honest: despite our best intentions, documentation has always been the unloved child of software development. We've all been there:

Outdated docs that no longer match the codebase
Endless meetings explaining what should have been documented
The classic "It works on my machine" because setup steps were missing
Documentation that explains "what" but never "why"

But here's the real crisis: As development velocity increases with AI assistance, our documentation practices aren't just staying static – they're falling further behind.

The AI acceleration paradox

The irony is palpable. AI tools are helping us write code faster than ever:

Copilot suggests entire functions in seconds
ChatGPT helps debug complex issues
AI-powered code review tools catch bugs early

Yet our documentation practices remain stuck in the pre-AI era. We're creating technical debt faster than ever, but our tools for managing and documenting this complexity haven't evolved.

The hidden cost of poor documentation in the AI era

The cost isn't just in lost time. In my experience leading technical teams, poor documentation manifests in:

Onboarding friction: New developers spend weeks understanding what could be learned in days
Knowledge silos: Critical context lives in Slack threads and people's heads
Technical debt accumulation: Without proper context, quick fixes become permanent solutions
Reduced AI tool effectiveness: AI assistants need context to provide relevant suggestions

The path forward: AI-first documentation

The solution isn't just throwing AI at the problem. We need a fundamental shift in how we think about documentation:

1. Context-aware documentation

Documentation that understands the codebase's evolution
Automatic updates based on code changes
Preservation of architectural decisions and their context

2. Just-in-time knowledge

Documentation that finds you, not vice versa
Contextual help based on what you're working on
Integration with development workflows

3. Living documentation

Documentation that evolves with your code
Automatic validation against the current codebase
Real-time updates based on team interactions

Practical steps for modern teams

As a CPTO, here's what I've found works:

Treat documentation as code
- Version control for docs
- Review processes for documentation changes
- Automated testing for documentation accuracy
Implement documentation-driven development
- Write the docs first
- Use docs to validate design decisions
- Make documentation part of the definition of done
Leverage AI responsibly
- Use AI to identify documentation gaps
- Automate routine documentation tasks
- Maintain human oversight for critical content

The future is already here

The tools and practices for better documentation exist today. What's missing is often the organizational will to prioritize it. As technical leaders, we need to champion this evolution.

Questions for the community

I'm curious about your experiences:

What's your biggest documentation pain point?
How are you using AI in your documentation workflow?
What would your ideal documentation system look like?

Let's discuss in the comments. Your insights will help shape the future of technical documentation.

DEV Community: jlcases