DEV Community: joyrswd

I Let AI Handle More of Its Own Review Feedback. Then I Had to Give the Work Boundaries.

joyrswd — Fri, 11 Sep 2026 09:35:29 +0000

In the previous article, I wrote about treating AI development rules as something that needed regression testing too.

AIDD Skeleton is my open-source repository template for governing AI-assisted development through repository-level rules.

That led to another question:

If an external reviewer finds a problem, how much of the response should the implementation agent handle on its own?

I did not want every review cycle to become:

review
↓
finding
↓
human: "fix this"
↓
review again
↓
human: "check this too"

The implementation agent already had the repository context. Ideally, one useful finding should let it investigate further on its own.

But more autonomy created another problem.

The better the agent became at finding related issues, the easier it became for the current task to expand indefinitely.

So the real problem became:

How do I let the AI reason broadly without letting it silently redefine the work?

I stopped treating review findings as isolated lines

When a reviewer reported a problem, I started asking the implementation agent to look beyond the exact reported location.

A finding might indicate a broader missed invariant or assumption.

Instead of:

finding
↓
fix the reported location

the process became closer to:

finding
↓
what assumption failed?
↓
where else does it matter?
↓
what belongs to the current work?

This became part of what I called Coherent Correction.

The important distinction was:

The investigation boundary can be wider than the modification boundary.

The agent may inspect sibling paths or higher-level design to understand a problem.

But discovering an improvement does not automatically authorize it to change that improvement now.

That boundary was introduced while restructuring the root governance.

docs: restructure root governance boundaries

A valid finding still might not belong to the current work

Once the agent started looking more broadly, it naturally found more things.

Some needed to be fixed now.

Some were good ideas, but unrelated to current acceptance.

Some might matter later.

So I separated two questions:

Is the finding valid?

and:

What should happen to it now?

At that stage, AIDD Skeleton used four dispositions:

Disposition	Meaning
`Accept now`	Handle it in the current work
`Reject`	Do not adopt it
`Defer`	Reconsider it later under relevant conditions
`Observe`	Keep watching because evidence is insufficient

The important part was not the names.

It was that discovery no longer implied authorization.

A reviewer could be correct without expanding the current task.

docs: govern feedback triage and review recall

Then I made the correction itself a review target

Looking at related surfaces helped, but another pattern remained.

The agent would correctly fix a finding, and the next review would find a problem introduced by that fix.

For example:

finding:
there is no fallback

correction:
add a fallback

Now there are new questions.

What if the fallback fails?

What if the primary path partially succeeds first?

Does the fallback still preserve the original invariant?

So I stopped treating a material correction as evidence that the review was finished.

The correction itself became something to challenge.

One of the rules was expressed as:

“Treat a material correction ... as a new adversarial surface.”

A finding was also treated as a clue about what the previous review had failed to consider.

For example:

"There is no real database evidence here."

could become:

"Are we treating simulated evidence as proof of real persistence elsewhere too?"

That missing review perspective could then be applied to related paths before another external reviewer had to find them one by one.

This became Adversarial Self-Review.

docs: shift review learning into adversarial self-review

When the same problem kept returning, I stopped fixing it locally

Even with broader inspection and self-review, similar findings sometimes kept appearing.

At some point, repeated findings become evidence themselves.

If several different surfaces expose the same or closely related cause, maybe the problem is not any single implementation.

Maybe:

an invariant is still implicit,
responsibilities are divided incorrectly,
the verification model is weak,
or the work itself was divided at the wrong boundary.

That became the trigger for Structural Reassessment.

The distinction matters.

With the first finding, the agent looks broadly:

Where else could this problem exist?

When the same cause keeps reappearing, it asks:

Why does this structure keep producing this problem?

The governance later expressed the trigger in terms of cases where:

“findings or corrections repeatedly expose the same or closely related cause”

I did not want every defect to trigger a repository-wide redesign.

Repeated related failures became the signal to move up a level.

Interestingly, this rule ended up applying to the governance itself.

While compressing the adversarial-review rules, external reviews repeatedly found different meanings that had been lost.

Instead of restoring missing sentences forever, the agent eventually reconsidered the structure of the review rules themselves.

docs: separate adversarial review responsibilities

"Not now" still needed somewhere to live

The feedback dispositions created one more problem.

What happens to Defer?

Keeping it only in a conversation means it may disappear with the session.

Putting it into formal project definition makes it look adopted.

Turning every review comment into an Issue felt excessive.

So I used workbench/.

A deferred item could be kept with:

why it was not being handled now,
the context in which it was found,
and what condition should cause it to be reconsidered.

But retention did not mean adoption.

retained ≠ adopted
retained ≠ promised
retained ≠ active work

This gave the agent an option between:

do it now

and:

forget it

It could preserve enough context for a later decision without silently turning that decision into a future commitment.

`workbench/` was starting to mean something different

Originally, workbench/ was mainly for temporary work:

investigations,
comparisons,
prototypes,
drafts,
validation output.

But deferred review findings introduced something different.

Now some retained material had:

a reason for not acting now,
a condition for reconsideration,
and continuity across conversations.

The repository was starting to preserve work state, not only temporary work material.

That also clarified what I actually wanted from AI autonomy.

Not an agent that simply did more.

An agent that could:

inspect broadly,
keep modification scope bounded,
challenge its own corrections,
recognize repeated structural problems,
and preserve work that should be reconsidered later.

At that point, workbench/ was starting to feel like the wrong name.

It was no longer just a workbench.

That led to the next structural change in AIDD Skeleton.

AI Development Rules Needed Regression Tests Too

joyrswd — Sat, 05 Sep 2026 02:44:05 +0000

I had already reached an uncomfortable conclusion:

Giving an AI coding agent better instructions in chat was not enough.

So I started moving those instructions into the repository itself.

In AIDDSkeleton, most of that governance lives in AGENTS.md files.

They define things like source-of-truth ownership, review behavior, evidence, scope, and lifecycle. Instead of reminding the AI about those rules in every prompt, I wanted an agent to read the repository, understand the rules, and make the right decisions on its own.

So the experiment was not:

Can I write a better prompt?

It was:

Can I change AGENTS.md and cause a fresh AI agent to behave differently without telling it what behavior I expect?

The repository became more consistent.

The agents seemed to behave better.

And then I ran into another problem.

How did I know the AGENTS.md changes were actually responsible for the improvement?

An AI agent can produce a better result for many reasons:

the task prompt was slightly different,
previous conversation context leaked the expected answer,
the agent saw later commits,
I accidentally hinted at the problem,
or the model simply took a different reasoning path that day.

At some point, “this seems better” stopped being good enough.

So I started regression-testing the rules themselves.

The repository was becoming part of the prompt

The project is AIDDSkeleton.

It is a repository skeleton for AI-driven development.

Most of its interesting behavior does not come from scripts or a framework.

It comes from repository-local governance.

AGENTS.md files tell an AI agent how to interpret the information around it: what counts as authority, which documents are sources of truth, what evidence is sufficient, how findings should affect current scope, and when work should or should not continue.

That distinction matters.

I was deliberately trying to reduce the amount of intelligence carried by the immediate task prompt.

A task could remain relatively ordinary:

Review this change.

or:

Fix this issue.

The more interesting behavior was supposed to come from the repository instructions the agent discovered and followed by itself.

That creates a strange testing problem.

If I change a TypeScript function, I can run a test.

If I change AGENTS.md and expect an AI agent to make a different autonomous decision, what exactly do I run?

Commit `08ded669` — feedback stopped meaning “do this now”

08ded669 — docs: govern feedback triage and review recall

One early change separated two things I had been mixing together:

How serious is this finding?

and

What should we do with it now?

A review comment could be valid without automatically becoming current work.

The governance started distinguishing dispositions such as:

Accept now
Reject
Defer
Observe

from classifications such as:

Blocker
In-scope deficiency
Follow-up

This made it harder for every interesting suggestion to silently expand the task.

More importantly, this was not supposed to require me to remind the agent:

Be careful not to expand scope.

The rule lived in the repository.

The agent was supposed to encounter a finding, interpret it under the current AGENTS.md, and decide for itself whether that finding belonged in the current work.

That exposed a deeper problem.

I could read the new rule and think:

Yes. This is clearer.

But that still did not tell me whether a fresh agent would actually make a different decision because of it.

So I started applying candidate governance to another repository that already used AIDDSkeleton-style rules and observing what the agent did.

That became the basic idea:

A governance change is not validated only by reading AGENTS.md.

It also needs behavioral evidence from an agent actually operating under that AGENTS.md.

My first regression tests had a bug

At first, the experiments looked promising.

I would give the agent a review finding and ask things such as:

Is there a common cause?
Are there sibling surfaces with the same problem?
Should this be reconsidered at a higher level?

The agent generalized.

It found related problems.

It produced broader corrections.

Great.

Except for one thing.

I had told it what kind of reasoning I wanted.

The candidate AGENTS.md might have been causing the behavior.

But the task prompt was carrying the same idea.

If I wanted to test whether repository governance caused an agent to look beyond the immediate symptom, I could not write:

Look beyond the immediate symptom.

I had effectively written the expected answer into the test input.

That meant the experiment could answer:

Can the model follow this hint?

But not:

Did the AGENTS.md change make the model do this on its own?

The regression test itself had a bug.

Commit `0611d61b` — a finding became more than a patch target

0611d61b — docs: shift review learning into adversarial self-review

The review model then changed more substantially.

A valid finding was no longer just something to patch.

It could also reveal something about the reasoning that had failed to detect the problem earlier.

The idea was roughly:

finding
   |
   v
what did we miss?
   |
   v
does that missed perspective matter elsewhere?

Again, the important part was that I did not want to put that sequence into every review prompt.

I wanted a fresh agent to receive an ordinary finding, read the repository governance, and independently decide whether the finding implied a wider review surface.

That sounded useful.

It also made the testing problem harder.

If I explicitly asked the agent to inspect sibling paths or look for a shared cause, I would no longer know whether AGENTS.md itself was responsible.

So the prompt had to become more neutral as the repository governance became more capable.

Freezing the past

The next version of the experiment started from a real historical commit.

Not the latest repository.

Not a manually reconstructed example.

An actual past state.

I treated that commit as a fixture.

historical consumer commit
          |
          +-- Control
          |     old AGENTS.md
          |
          +-- Treatment
                candidate AGENTS.md

Both worlds started from the same repository state.

The task prompt was kept materially the same.

The intended experimental variable was the governance.

The important part was also what the agent was not allowed to see.

Everything after the fixture was treated as nonexistent:

later commits,
later Pull Requests,
later review comments,
known fixes,
previous regression results.

In other words, I tried to create two artificial timelines from Git history.

The agent could see the past.

It could not see the future.

If a later review had already identified a bug, allowing the agent to discover that comment would destroy the experiment.

It would no longer be testing whether the candidate AGENTS.md helped the agent discover the issue.

It would be testing whether the agent could read the answer.

Control and Treatment did not behave the way I wanted

This was probably the point where I started trusting the method.

The candidate governance did not simply win every round.

In one MediaStorage scenario, the Control branch explored a collision problem more aggressively.

The original issue was effectively:

destination already exists

The Control agent spontaneously pushed further toward:

what if two writers race for the same destination?

The Treatment agent improved the requested surface, but in that run it did not expand as far into the race condition.

That was inconvenient.

It was also excellent evidence.

If every experiment had ended with:

new AGENTS.md good

I would have trusted the experiments less.

Instead, I had an actual behavioral concern.

The candidate governance seemed stronger in some kinds of generalization, but possibly weaker in immediate exploration breadth.

So I did not declare victory.

I changed the repository rules again.

This was regression evidence, not a benchmark

There is an important limitation here.

I was not running each fixture dozens or hundreds of times.

I was not estimating a statistically meaningful success rate.

And across different AI providers, I was not controlling sampling parameters such as temperature closely enough to claim that the runs formed a clean quantitative benchmark.

So results like:

Treatment expanded the review surface.

or:

Control found the race condition first.

should not be read as:

Treatment has a 73% probability of doing X.

I did not have that data.

The regressions were primarily adversarial behavioral checks.

I was asking questions such as:

Can the intended behavior appear without being prompted?
Does the candidate introduce a material behavioral regression?
Does it cause scope or authority violations?
Does a concern still reproduce after the governance is refined?
Does the final candidate retain the behavior seen in an earlier revision?

In practice, my judgments were closer to:

PASS
- intended behavior appeared without being hinted
- no material governance violation appeared

CONCERN
- useful behavior appeared
- but another material regression or scope issue appeared

FAIL
- expected behavior did not appear
- or the candidate caused a material governance violation

Those categories were not a formal scoring system from day one.

They emerged as I repeated the experiments.

That distinction matters.

I was building a regression method, not claiming to have built an LLM benchmark.

Commit `94d163b8` — a useful experiment that I deliberately removed

At one point I tried persistent review knowledge.

The idea was attractive.

If one task teaches the agent a useful lesson, why not save it in the repository and let later agents reuse it?

The experiment worked.

Almost too well.

The retained knowledge materially changed later reasoning.

A fresh agent could pick up that knowledge and go deeper without me prompting it to do so.

That was exactly the kind of repository-driven autonomy I was trying to create.

But it also exposed the danger.

The retained knowledge influenced decisions outside the original task strongly enough that the boundary between:

useful prior knowledge

and:

accidental design authority

became difficult to control.

So I removed it.

94d163b8 — docs: remove knowledge base governance

That commit changed how I thought about successful AI testing.

A test did not need to prove that a new form of autonomous behavior was good.

It could tell me:

This repository instruction clearly changes what later agents do, but the side effects are not bounded well enough. Do not ship it.

That is still a successful test.

The test became part of the development loop

By this point the workflow looked suspiciously familiar.

change AGENTS.md
      |
      v
start agent against consumer fixture
      |
      v
observe autonomous behavior
      |
      v
find unexpected behavior
      |
      v
change AGENTS.md again
      |
      v
rerun regression

That is just software development.

The implementation language happened to be English.

And the thing being implemented was partly agent behavior.

Commit `e5033e6d` — refactoring natural language broke behavior

e5033e6d — docs: compact adversarial self-review escalation

Next I tried to make the review rules in AGENTS.md shorter.

The existing responsibilities were compressed into a smaller structure.

The new version looked cleaner.

I believed the semantics were preserved.

Then review started finding things I had lost.

The compacted version had weakened or dropped details such as:

diagnosing why a problem escaped detection,
retaining the concrete context that exposed it,
reopening review when later evidence changed the basis,
keeping review proportional and bounded,
reopening broader review after a broad correction.

This was a useful reminder that natural-language refactoring can fail exactly like code refactoring.

A human reader can look at two versions of AGENTS.md and decide that they “basically say the same thing.”

An agent may not behave as though they say the same thing.

The behaviorally important conditions can disappear even when the prose looks cleaner.

Eventually I stopped treating “fewer bullets” as a goal.

Commit `1f5dafd6` — structure won over compression

1f5dafd6 — docs: separate adversarial review responsibilities

The rule was reorganized around five responsibilities:

1. Initial adversarial review
2. Signal assimilation
3. Correction challenge
4. Review continuation / re-entry
5. Structural reassessment

The AGENTS.md text became slightly larger again.

But the responsibilities became clearer.

At that point I had another problem.

Most of the behavioral evidence had been collected against earlier candidate commits.

The repository instructions had changed during review.

So the final merge candidate needed to go back through the consumer test again.

A previous agent behaving correctly under an earlier AGENTS.md did not prove that a fresh agent would behave correctly under the final one.

Evidence from an earlier revision was not enough.

The final `AGENTS.md` went back into the frozen past

This time I did not need another full Control/Treatment A/B.

The earlier comparisons had already given me a baseline.

What I wanted to know was simpler:

Does a fresh agent operating under the actual final candidate still behave well in the historical fixture?

So I ran it again.

The agent got the candidate repository governance.

It got the historical consumer state.

It did not get hints about the behaviors I hoped to see.

It did not get later review comments.

It did not get the historical answer key.

This time the earlier breadth concern did not reproduce.

The agent independently moved from an existing-key collision problem into concurrent same-key publication.

That was encouraging.

It meant the wider reasoning appeared without me putting that reasoning into the immediate prompt.

Then it did something even more interesting.

It explored media reads and introduced requirements around:

avoiding whole-artifact buffering,
backpressure,
client-disconnect cancellation,
handling storage read failures after response start.

Technically?

Reasonable.

Possibly even good engineering.

And that was exactly the problem.

The AI autonomously found a good idea it was not authorized to adopt

I went back to the historical requirements.

Those read-performance behaviors had not actually been required.

The approved design did not establish them as current acceptance criteria.

The agent had crossed a subtle boundary on its own.

It had moved from:

Find deficiencies in the adopted design.

to:

Improve the design because this would be better.

This is one of the failures I now find most interesting in AI-driven development.

The generated result can look better and still be wrong.

As repository instructions make an agent more autonomous, finding more things is not enough.

The agent also needs to distinguish:

this is a real deficiency in current work

from:

this is a good idea for some other scope

A benchmark based only on code quality could easily reward the wrong behavior here.

I sent one finding back into the same session

I did not tell the agent to revert the change.

I did not tell it which parts to keep.

I gave it one narrower challenge:

Are these new read-performance contracts actually required by the existing approved scope, or were useful improvements promoted into formal design without sufficient authority?

The agent re-evaluated its own previous work under the same repository governance.

It removed the new read-performance requirements.

But it did not revert everything from the previous correction.

The collision changes stayed.

Why?

Because those were supported by an already-adopted invariant:

do not silently overwrite an existing key

Concurrent publication could violate that invariant.

So collision safety remained an in-scope correction.

The new read-performance contract did not.

The resulting behavior was effectively:

finding
   |
   v
re-evaluate authority and scope
   |
   +-- read performance requirements
   |      no existing requirement
   |      -> remove
   |
   +-- collision safety
          existing invariant
          -> keep

That was a much stronger result than simply finding more bugs.

The agent had expanded its own review, overreached, received a finding, and then separated the justified part of its own work from the unjustified part.

That was much closer to the kind of autonomous development behavior I wanted AGENTS.md to produce.

The harness is still mostly manual

Another limitation is operational.

There is no fully automated CI harness behind these experiments yet.

Git gives me reproducible fixtures and branches, but the full loop still involves manual orchestration:

select historical fixture
      |
      v
prepare Control / Treatment state
      |
      v
apply candidate governance
      |
      v
start a fresh agent
      |
      v
retain commits, diffs, and reports
      |
      v
compare observed behavior

The mechanical parts are reproducible.

The behavioral assessment still involves human judgment, often assisted by another AI review.

That has a real cost.

A consumer regression is much heavier than running a normal unit test.

But at this stage, I think that is acceptable.

Automating the loop before the evaluation criteria stabilized would have meant automating my assumptions too.

The experiments were still teaching me what should count as a signal, what should count as overreach, and which differences actually mattered.

Once those criteria become stable enough, much more of the loop could be automated.

A future harness could:

restore a fixed fixture commit,
apply a selected AGENTS.md candidate,
launch a fresh agent with a fixed neutral task,
retain the transcript, diff, commits, and reports,
repeat the same fixture N times,
classify predefined behavioral outcomes,
compare Control and Treatment rates.

That would also make it possible to say things I cannot honestly say today, such as:

Control:
  intended behavior in 3 / 10 runs

Treatment:
  intended behavior in 8 / 10 runs

I am not there yet.

For now, the method is closer to an adversarial regression suite than an automated statistical benchmark.

What I learned from black-box testing `AGENTS.md`

The obvious lesson is that prompts can be tested.

But that is not really the interesting part.

The more important lesson is that repository instructions behave like executable policy once an AI agent is expected to act on them autonomously.

Changing a few lines in AGENTS.md can alter:

what an agent inspects,
which nearby problems it notices,
what it considers evidence,
what it decides belongs in current scope,
when it reopens review,
when it stops,
and what it believes it is allowed to change.

Those ar

The Repository Was Organized. The Documentation Still Kept Spreading.

joyrswd — Mon, 31 Aug 2026 10:55:13 +0000

By the beginning of August, AIDD Skeleton had its basic shape.

There were five places for project information:

plans/
etc/
workbench/
references/
products/

The first problem had been structural.

When development happens through conversations with AI, where should requirements go? Where should unfinished investigations go? Where should implementation live?

Giving each responsibility a place helped.

But some of the problems I mentioned in the first article had already hinted that structure alone would not be enough.

TRACEABILITY.md had started growing too easily.

The source of truth could disagree with the implementation — and sometimes the source of truth itself was the thing that needed correction.

At the time, these looked like separate early problems.

They weren't.

They were the first signs of a much deeper question:

It wasn't enough to decide where documentation belonged. I also had to decide what that documentation was allowed to mean.

The first warning was documentation that could grow forever

In the first article, I mentioned that the first pull request in AIDD Skeleton was about preventing TRACEABILITY.md from growing without bound.

Prevent TRACEABILITY.md from growing without bound — August 1, 2026

At the time, I treated it mostly as a local documentation problem.

Traceability was useful because it connected requirements, design, implementation, and verification.

But it was also an obvious place to keep adding more information.

More implementation units.

More tests.

More verification records.

More relationships.

More historical context.

Each addition could be justified.

Together, they could turn one useful document into an ever-growing index of the entire project.

What became clear later was that TRACEABILITY.md was not a special case.

The same pattern could happen everywhere.

A status document could start carrying implementation history.

A testing document could accumulate execution results.

A design document could absorb facts discovered from the current code.

An investigation could eventually look like a project decision simply because nobody had removed it.

The repository did not just need rules about where documents belonged.

It needed rules about what those documents were allowed to mean.

The source of truth could be wrong too

Another event from the first stage of the project became more important in hindsight.

If the implementation disagreed with the documentation, the obvious reaction was:

The code is wrong. Make it match the source of truth.

But that assumes the source of truth is actually correct.

So AIDD Skeleton introduced a different order of diagnosis:

Check whether the source of truth is complete, current, and internally consistent.
Then check whether the implementation conforms to it.
Then check whether verification actually demonstrates that conformity.

Review sources of truth before implementation — August 1, 2026

At first, this looked like a rule for resolving documentation drift.

Later, I realized it implied something much broader.

Documentation was not automatically trustworthy because it lived in the right directory.

That changed the questions I was asking.

Not:

Where should I put this file?

But:

Is this actually an adopted decision?

Is this describing what the system should be, or what it happens to be today?

Is this a requirement, a design decision, an implementation observation, or evidence from one execution?

Those distinctions turned out to matter much more than the directory names.

`plans/` needed a much stricter meaning

The largest early ambiguity was inside plans/.

At first, it was tempting to keep both the current state and a proposed target state there.

That sounds useful.

You can see where the project is now and where it is supposed to go.

But it creates an authority problem.

Suppose a document says:

Current:
Authentication uses approach A.

Target:
Authentication will use approach B.

Has B been approved?

Is it still being discussed?

Can the AI begin migrating toward it?

What happens if another document still describes A as the intended design?

The distinction between describing a possible future and adopting that future as the project definition was too important to leave implicit.

By August 13, AIDD Skeleton made the boundary much clearer:

workbench/  -> candidate state, investigation, proposed change
plans/      -> adopted project definition

Clarify source-of-truth change handling — August 13, 2026

This meant a proposed design did not become authoritative just because someone had documented it carefully.

It remained in the workbench until it was actually adopted.

That rule solved more than a file-placement problem.

It separated two fundamentally different states of knowledge:

We are thinking about this.

and

The project has decided this.

For AI-driven development, that difference is enormous.

A human developer can often recognize that a beautifully written target design is still only a proposal.

An AI agent should not have to infer that from tone.

The repository needed to represent the distinction explicitly.

Then implementation knowledge started leaking back into design

Another problem appeared from the opposite direction.

Suppose an AI investigates an existing application.

It reads the code.

It discovers file paths, framework conventions, database structures, library behavior, deployment details, and internal APIs.

That information can be extremely useful.

But useful does not automatically mean normative.

If the current implementation is copied into a design document, the design gradually stops describing what the system must mean.

It starts describing how this particular version happens to be implemented.

Eventually, future implementations become constrained by yesterday's implementation details.

So another boundary had to become explicit:

Can the project still be reconstructed semantically from its normative documentation without access to the current implementation?

That question led to a much stronger separation between normative and non-normative information.

On August 20:

Tighten normative documentation boundaries — August 20, 2026

Requirements, design, and testing were increasingly treated as documents that could constrain future implementation.

Current implementation observations could still be valuable.

But they should not silently inherit that authority.

This was one of the points where I realized that "documentation" was too broad a category.

Two Markdown files could look almost identical while having completely different responsibilities.

One might define what future code must preserve.

The other might simply record something useful I learned today.

Treating both as equally authoritative was dangerous.

Testing had the same problem

Testing introduced another version of the same confusion.

A testing specification might say:

The system must reject an expired token.

That is a normative statement.

It defines what must be verified.

Now suppose I run the test today and record:

PHPUnit 12.3.1 passed this test on commit abc123.

That is useful too.

But it is not the testing specification.

It is evidence from a particular execution.

The difference matters because one of them can remain valid while the other becomes stale.

The requirement to reject expired tokens may still be valid six months later.

The fact that one test passed six months ago does not prove that the current system still does.

Yet AI-generated project documentation can easily mix the two.

Testing intent, execution output, screenshots, logs, observations, and verification summaries can all look like "testing information."

So AIDD Skeleton began separating them explicitly.

On August 21:

Separate execution evidence from normative testing — August 21, 2026

Then the boundary was refined further:

Refine execution evidence boundaries — August 21, 2026

And the next day, active verification evidence was routed through the workbench instead of being allowed to quietly become permanent project truth:

Route verification evidence through workbench — August 22, 2026

Tighten verification evidence lifecycle — August 22, 2026

The underlying lesson was simple:

Evidence has a lifecycle.

A specification can remain valid.

An execution result can become stale.

A useful investigation can stop being useful.

A temporary artifact can deserve promotion into durable reference material.

Another can simply be deleted.

Keeping everything forever is not traceability.

Sometimes it is just accumulation.

I had been treating documentation as files

By this point, the original five-directory model still looked almost unchanged.

plans/
etc/
workbench/
references/
products/

But my understanding of what lived inside those directories had changed substantially.

At the beginning, I had mostly been asking:

Where should this go?

Now the questions looked more like:

Has this been adopted?

Does this constrain future implementation?

Is it an observation or a decision?

Is this testing intent or execution evidence?

How long is this evidence valid?

If this information is retained, why is it still useful?

If it is moved, what other documents still depend on it?

The repository structure had not become much more complicated.

The information model had.

And that was probably unavoidable.

The more development I delegated to AI, the less comfortable I became relying on distinctions that existed only in my head.

A human developer might look at a draft and intuitively understand:

"We're still discussing this."

An AI agent needs that state to be represented somewhere.

A human might see an old verification report and know:

"That was from before the refactor."

An AI agent needs enough provenance and lifecycle information not to mistake it for current proof.

A human might recognize that a design section merely describes the current framework implementation.

An AI agent may treat it as a requirement unless the repository tells it otherwise.

The goal was no longer simply to preserve information.

It was to preserve its meaning.

Less documentation became a feature

This also changed how I thought about documentation quality.

Previously, completeness often meant adding information.

Now I was increasingly interested in whether information deserved to exist in the durable project record at all.

Does this fact belong in the source of truth?

Does it only matter while we are investigating something?

Can it be derived from another authoritative source?

Has it become stale?

Does keeping it create another place that must be synchronized forever?

A documentation system can fail because information is missing.

It can also fail because information never leaves.

The goal was not maximum documentation.

The goal was minimum ambiguity.

That distinction became increasingly central to AIDD Skeleton.

And eventually it led to a much stranger problem.

I had spent weeks tightening the rules that told AI how to classify, retain, promote, and remove project information.

Those rules were becoming powerful enough to cause significant changes when applied to an existing repository.

Which raised an uncomfortable question:

How did I know the rules themselves were right?

It was no longer enough to inspect AIDD Skeleton and decide that the rules looked reasonable.

If the framework was supposed to govern real projects, eventually those rules would have to be tested against real projects too.

And if applying them produced a bad result, I would need to resist the easiest explanation:

"The project was wrong."

Sometimes the thing that needed debugging might be the framework itself.

That became the next stage of the project.

AIDD Skeleton is still evolving:

AIDD Skeleton on GitHub

The first article in this development history covered how conversations with AI led to the initial five-part repository structure and the creation of workbench/.

This article covers what happened next: realizing that organizing documents was not enough. Their authority, lifecycle, and evidentiary scope also had to be controlled.

The next stage began when I started applying those rules to other repositories and asking a different question:

Can the framework itself fail the test?

AI disclosure: This article was written with AI assistance. The narrative was reconstructed from the AIDD Skeleton Git history and my development conversations, and the resulting account was reviewed against the repository history before publication.

How Conversations With AI Slowly Became a Development Framework

joyrswd — Sat, 29 Aug 2026 13:46:03 +0000

I didn't start AIDD Skeleton because AI was bad at writing code.

Quite the opposite.

I was already using AI as a development partner.

We would discuss requirements, compare approaches, investigate problems, make design decisions, implement changes, and review the results.

The conversation itself worked surprisingly well.

The problem was everything around the conversation.

As a project continued, more and more things had to survive beyond the current chat:

decisions we had already made,
ideas we had considered but not adopted,
requirements,
design,
implementation,
experiments,
reference material,
environment configuration,
verification results,
and the current state of the project.

If all of that was handled casually, the repository could become just as chaotic as the conversation history I was trying not to depend on.

So the original idea was fairly modest:

I wanted a predictable format where a developer and an AI agent could keep building through conversation without slowly turning the project into a mess.

That was the beginning of AIDD Skeleton.

It started as four places

Before creating the public repository, my first rough structure was very simple:

docs/
etc/
prototypes/
src/

The idea was mostly about separating responsibilities.

docs/ held the formal project documentation.

etc/ held development and execution environment configuration.

prototypes/ held temporary experiments and proofs of concept.

src/ held the formal implementation.

At that point, I was still thinking in fairly familiar repository terms.

Documentation goes here.

Source code goes there.

Experiments should not be confused with production code.

Environment configuration should have its own place.

The important part was not the names themselves.

It was the idea that an AI agent should not create new top-level structures simply because they were convenient for the task immediately in front of it.

Before creating something, it should first ask:

What responsibility does this belong to?

That was the first boundary I wanted the repository to provide.

Reference material turned out to be a different kind of information

The four-way split quickly exposed another category.

Projects often contain material that matters, but is not itself a project decision:

documents supplied by a user,
sample data,
screenshots or designs,
external specifications,
PDFs,
information from another system.

Putting those materials beside formal project documentation made their status ambiguous.

A supplied specification can influence a requirement without automatically becoming the requirement.

A sample file can be evidence without becoming part of the implementation.

So I added a separate area:

references/

The structure became:

docs/
etc/
prototypes/
references/
src/

That distinction was small, but important.

I wanted the repository to distinguish between:

information the project received

and:

information the project had actually adopted.

`docs/` and `src/` described file types better than responsibilities

The next problem was naming.

docs/ sounds like a place for documents.

But what I wanted to manage there was not "documents" as a file type.

It was the project's adopted knowledge:

requirements, design, testing expectations, project state, and other information that the implementation was supposed to follow.

Likewise, src/ sounded narrower than what I meant.

The formal output of a project could include implementation, tests, generated artifacts, packages, applications, or other deliverables.

So I changed the names:

docs/ → plans/
src/  → products/

The structure became:

plans/
etc/
prototypes/
references/
products/

By the time I created the public AIDD Skeleton repository on July 29, 2026, this was already the model I was using.

Reference:

Initial English AIDD skeleton — July 29, 2026

This was the point where the repository started moving away from classification by file type and toward classification by responsibility.

A src/ directory could still exist.

A tests/ directory could still exist.

They just belonged inside the area responsible for formal project outputs.

The first public version was still mostly about places

At that point, the public repository had five top-level areas:

plans/
etc/
prototypes/
references/
products/

Each had a different responsibility.

plans/ was for adopted project documentation and sources of truth.

etc/ was for execution-environment configuration.

prototypes/ was for temporary experiments.

references/ was for externally supplied material.

products/ was for formal implementation and other project outputs.

This may look like a directory convention, but the motivation was more defensive than aesthetic.

AI agents are very good at solving the problem directly in front of them.

A framework expects src/?

Create src/.

A tool wants scripts/?

Create scripts/.

A new service looks independent?

Create another top-level directory.

Each individual decision can be perfectly reasonable.

The repository as a whole can still gradually lose any coherent ownership model.

So the rule was intentionally strict:

Before creating something, classify what responsibility it serves.

If something did not fit anywhere, the first response should not automatically be to invent a sixth category.

Maybe the classification itself needed to be reconsidered.

A few hours after the public repository was created, I also simplified the README around an intentionally minimal workflow:

Clone the repository.
Open it in an AI development environment.
Send the first message.

The conversation could begin normally.

The difference was that the project underneath it already had a shared structure for the developer and the agent.

At this stage, I thought the main problem was organization.

That assumption was challenged almost immediately.

Documentation could become part of the problem

One of the first things I added was traceability.

If requirements, design, implementation, and verification were related, it seemed useful to record those relationships.

But a single global traceability document had an obvious failure mode.

More requirements.

More implementation units.

More tests.

More identifiers.

More relationships.

More history.

The document intended to make the project easier to understand could eventually become another source of complexity.

So one of the first public changes was specifically about preventing TRACEABILITY.md from growing without bound.

Reference:

Prevent TRACEABILITY.md from growing without bound

This introduced an important constraint very early:

More documentation does not automatically create more order.

Documentation itself needs boundaries.

The repository also needed rules for distrusting its own documentation

Another early problem appeared when documentation and implementation disagreed.

Suppose the requirements say one thing, the design says another, and the implementation behaves differently again.

A tempting response is:

The code doesn't match the documentation. Fix the code.

But that assumes the documentation is correct.

I did not want an AI agent to silently repair implementation against an outdated, incomplete, or internally inconsistent document.

So I added an explicit ordering:

Check whether the relevant source of truth is complete, current, and internally consistent.
Check whether the implementation conforms to that validated source.
Check whether verification actually demonstrates that conformity.

Reference:

Review sources of truth before implementation

That was an important change in what I thought the repository needed to do.

Folders alone were not enough.

The agent also needed rules for deciding what information it was allowed to trust.

`prototypes/` turned out to be too small an idea

The next structural change came from the way conversations actually produced intermediate work.

Originally, prototypes/ was intended for things like:

proofs of concept,
technical spikes,
UI experiments,
alternative implementations used to answer a question.

That seemed reasonable.

But AI-assisted development produced many useful intermediate artifacts that were not prototypes.

An investigation.

A comparison.

A design draft.

An implementation plan.

A transformed copy of supplied material.

Temporary analysis.

Context needed to continue the work in another conversation.

These things were useful, but they were not formal project knowledge.

Putting them into plans/ created a much more dangerous ambiguity:

an unfinished thought could start looking like an adopted decision simply because it lived beside the real sources of truth.

So prototypes/ became workbench/.

prototypes/
    ↓
workbench/

Reference:

Replace prototypes with a project workbench

The distinction became roughly:

plans/       adopted project knowledge
workbench/   work that is still being investigated, prepared, or evaluated

A prototype could live in the workbench.

So could an investigation.

So could a candidate design.

So could temporary context for continuing work later.

The important rule was:

Something did not become authoritative merely because an AI had written it down.

The structure at the time of the first article

By the time I wrote about AIDD Skeleton in early August 2026, the top-level structure had become:

plans/
products/
references/
workbench/
etc/

The names had changed from the earliest sketch, but the underlying separation was becoming clearer.

formal project knowledge    → plans/
formal project outputs      → products/
supplied source material    → references/
intermediate project work   → workbench/
execution environment       → etc/

The repository was not trying to replace familiar structures such as src/, tests/, or application directories.

It was trying to put those structures inside a clearer ownership model.

The question was no longer simply:

What kind of file is this?

It was becoming:

What responsibility does this information have?

Verification needed its own boundary too

Another rule I wanted from the beginning was a strict separation between implementation and verification.

Creating a file does not prove that the program works.

Passing a syntax check does not prove that a feature works.

Passing a unit test does not prove behavior that the test never exercised.

So completion reports were expected to separate things such as:

Changes:
- Implemented the expense registration API
- Added input validation
- Added unit tests

Verification:
- Unit tests passed
- Static analysis passed

Not verified:
- Behavior against a production-like database
- Concurrent use by multiple users

The important rule was simple:

Do not claim a broader result than the available evidence demonstrates.

That rule was tightened during the same early development period.

Reference:

Tighten verification evidence scope

This mattered because AI-generated completion summaries can sound much more certain than the underlying verification actually justifies.

I wanted that boundary to exist in the repository rather than depending on the wording of a particular conversation.

Turning it into a repository template

The initial workflow was deliberately simple.

The goal was not to require a large setup prompt or a special orchestration system.

A developer could create a repository from the template, open it in an AI development environment, and begin the conversation normally.

The repository itself supplied the shared structure and working rules.

On August 7, 2026, I updated the README to make GitHub Repository Template usage the primary path.

Reference:

Make template usage the primary README path

The basic idea remained the same:

The conversation should not have to carry the entire project.

Requirements, design, implementation, experiments, supplied material, environment configuration, and verification all needed places and boundaries that could survive the current session.

What AIDD Skeleton was trying to solve at this stage

The early version of AIDD Skeleton was still fairly small in concept.

It was not an attempt to automate every part of software development.

It was an attempt to establish a few durable boundaries:

adopted project knowledge should be distinguishable from work in progress,
external material should be distinguishable from adopted requirements,
formal implementation should be distinguishable from experiments,
implementation should be distinguishable from verification,
AI working rules should live with the repository rather than only in conversation history.

The directory names changed several times while I was figuring out those boundaries.

The progression was roughly:

docs        → plans
src         → products
prototypes  → workbench
references  → added as a separate responsibility
etc         → retained

The important part was not the final names.

It was the shift from asking:

Where should I put this file?

to asking:

What responsibility does this information have?

That was the point where what began as a repository layout started to feel more like a development framework.

AIDD Skeleton:

https://github.com/joyrswd/AIDDSkeleton

DEV Community: joyrswd

I Let AI Handle More of Its Own Review Feedback. Then I Had to Give the Work Boundaries.

I stopped treating review findings as isolated lines

A valid finding still might not belong to the current work

Then I made the correction itself a review target

When the same problem kept returning, I stopped fixing it locally

"Not now" still needed somewhere to live

workbench/ was starting to mean something different

AI Development Rules Needed Regression Tests Too

The repository was becoming part of the prompt

Commit 08ded669 — feedback stopped meaning “do this now”

My first regression tests had a bug

Commit 0611d61b — a finding became more than a patch target

Freezing the past

Control and Treatment did not behave the way I wanted

This was regression evidence, not a benchmark

Commit 94d163b8 — a useful experiment that I deliberately removed

The test became part of the development loop

Commit e5033e6d — refactoring natural language broke behavior

Commit 1f5dafd6 — structure won over compression

The final AGENTS.md went back into the frozen past

The AI autonomously found a good idea it was not authorized to adopt

I sent one finding back into the same session

The harness is still mostly manual

What I learned from black-box testing AGENTS.md

The Repository Was Organized. The Documentation Still Kept Spreading.

The first warning was documentation that could grow forever

The source of truth could be wrong too

plans/ needed a much stricter meaning

Then implementation knowledge started leaking back into design

Testing had the same problem

I had been treating documentation as files

Less documentation became a feature

How Conversations With AI Slowly Became a Development Framework

It started as four places

Reference material turned out to be a different kind of information

docs/ and src/ described file types better than responsibilities

The first public version was still mostly about places

Documentation could become part of the problem

The repository also needed rules for distrusting its own documentation

prototypes/ turned out to be too small an idea

The structure at the time of the first article

Verification needed its own boundary too

Turning it into a repository template

What AIDD Skeleton was trying to solve at this stage

`workbench/` was starting to mean something different

Commit `08ded669` — feedback stopped meaning “do this now”

Commit `0611d61b` — a finding became more than a patch target

Commit `94d163b8` — a useful experiment that I deliberately removed

Commit `e5033e6d` — refactoring natural language broke behavior

Commit `1f5dafd6` — structure won over compression

The final `AGENTS.md` went back into the frozen past

What I learned from black-box testing `AGENTS.md`

`plans/` needed a much stricter meaning

`docs/` and `src/` described file types better than responsibilities

`prototypes/` turned out to be too small an idea