DEV Community: drmikecrowe

First Drafts All the Way Down

drmikecrowe — Mon, 19 Oct 2026 11:46:06 +0000

This post was written with AI assistance (Claude) for structure, formatting, and pulling the numbers out of my logs. The process, the opinions, and the voice are my own.

If your AI workflow is plan, build, tests pass, ship, you are shipping first drafts.

I don't mean that as a dig at the models. I mean it literally. The model hands you a plan with total confidence, then hands you code with a green test run and the same total confidence, and both of them are first drafts. Nobody would ship a junior engineer's first draft without a review. We've somehow decided the frontier model gets a pass.

I didn't want to argue that from vibes, so I measured it. Over the last month I ran 83 stories through my AI SDLC process across three repositories. Opus 5.5 wrote almost every plan and implementation, which is the best model I can reasonably afford to run all day. Separate review agents, mostly on Sonnet, checked the work at every step. Here's what they found:

Four out of five plans. Half of the green builds. One plan in four heading in the wrong direction. That's the latest model I can afford, working inside a process built to make it succeed.

Where this came from

I read Itamar Gilad's Evidence-Guided, and it changed how I think about delivering software. The book is about product management, but the core move applies everywhere: stop deciding on opinion and confidence, and combine human judgement with evidence. Your confidence in an idea should match the evidence you have for it, not how good it sounds in the meeting.

Here's the thing: a frontier model is the most confident colleague you will ever work with. It never says "I think." It says "Done." Gilad's book taught me to be suspicious of confidence without evidence, and it turns out that's exactly the right instinct for AI-written software.

We can't read our way out of this

This week Anthropic published how they redesigned source control. They went from about 2,500 merged changes a day in May 2026 to over 30,000 a day in September, and Git couldn't keep up, so they built their own system (Scion). Their line that stuck with me: "the latent demand for merging code is practically infinite, and every bottleneck we remove immediately leads to a further increase in the amount of code we produce."

Think about what that means for code review. If the company that makes the model can't keep its version control on top of the volume, nobody is reading all those diffs. I'm certainly not reading all of mine. For most of my career, reading the PR was the quality process. That doesn't scale to this.

So the standard has to change. The new standard is evidence that we met the plan and the story, assembled by the process and checked by something other than the thing that wrote the code. I read the evidence. I don't read every line.

Step 0: you need real tests first

I want to be blunt about this one: you won't get acceleration from AI without thorough testing. Tests are the only thing that steers the model. Without them, every one of the checks below has nothing to stand on, and "it compiles" becomes your definition of done.

And "we have tests" isn't the bar either. I wrote about this in The Tests That Would Not Notice. In this data set, 46 implementations ran mutation testing (1,055 mutants), and 8 of them had passing tests that wouldn't notice a broken line until new tests were written. Testing is step 0. Good testing is step 0 done right. Nothing in the AI SDLC works until it's there.

The process, briefly

Every story goes through the same steps. The model writes, and a separate agent checks:

Story. The author answers questions until the acceptance criteria are testable. A story check grades whether it's ready to plan.
Plan. The model writes the plan. An adversarial plan review hunts for places a builder would build something the story didn't mean. A plan check grades whether an engineer who never saw the conversation could build it. A person rules on the Decide calls, the choices that belong to a person and not the model.
Build. Tests first, then code, until the gauntlet (tests, lint, types, and whatever else the repo declares) is green. Then an adversarial code review tries to prove the diff wrong.
Evidence. The model writes the evidence. An evidence-vs-plan check holds it against the plan's scenarios, and an evidence-vs-story check holds it against the story's acceptance criteria.
PR. Only then does it go up for review.

The checks are cheap and fast. The median judge run takes under a minute.

Using /plan without a review is a first draft

The plan is where I expected the model to shine. It reads the story, reads the code, and writes a clean, numbered plan. It looks done. It almost never is.

The adversarial plan review raised gaps on nearly every plan it read. In the 31 plans that logged what happened to them, 124 gaps were fixed in the plan and 66 were declined with a written reason. That second number matters to me. The review is advice, not law, and the model pushes back when the review is wrong. But 124 real fixes across 31 plans is about four holes per plan that would otherwise have become code.

Planning roughly doubled in agent time. Nine extra minutes per plan is the cheapest fix in this whole process, because every hole caught in the plan is one that never gets built, tested, reviewed, and thrown away.

So when someone tells me they run /plan and go straight to build, my answer is that they're building from a first draft.

The plan pointed the wrong way

If I could put one number from this whole exercise on a slide, it would come from this section.

The review agents check whether a plan is complete and buildable. They can't check whether it's heading in the right direction, because the direction lives in someone's head: the customer, the business, how the product is actually used. So the process makes the model stop and ask. Every choice that belongs to a person becomes a Decide call: the question, the options, and the model's recommended default. A person rules on each one before the plan is approved.

Read that last tile again. One plan in four was confidently pointed somewhere it shouldn't go. The problem wasn't missing detail. The plan was wrong about what we needed. The review agents grade whether a plan is complete and buildable, and a plan can be perfectly buildable in the wrong direction.

The model's default was right 90% of the time, which is good. But the 10% it got wrong are the expensive ones. A missed edge case costs you a fix. A plan pointed the wrong way costs you the whole build, built correctly, tested, reviewed, and thrown away. If your process lets the model make those calls silently, you won't find out until the demo.

Testing doesn't mean it's ready

Now the build. By the time the adversarial code review runs, the gauntlet is green. Every test the model wrote passes. By the usual definition, the work is done.

Here's what kind of defects they were:

A few of them, in plain words:

Security (GH-120): extra request fields could overwrite checked values like parent, project or summary, because the guards ran before the fields were merged in. Tests passed.
Error handling (GH-566): if launching a child process failed in an unexpected way, the worker died silently and the child sat there waiting forever. Tests passed.
Test gap (GH-159): a new test file was never wired into any test layer, so it never ran and proved nothing. Every test that did run passed, which is exactly the problem.

That last one is my favorite, in the way a scary movie is your favorite. Green was a lie, and nothing in the test run could have told me.

Matching the plan isn't matching the story

This is the control I'm proudest of, and the one I'd push hardest on any team doing AI-driven delivery.

At sign-off the model writes the evidence: here's each requirement, here's the test or run that proves it. The evidence-vs-plan check holds that evidence against the plan. The evidence-vs-story check holds the same evidence against the story's acceptance criteria. Most AI workflows stop at the first one, if they have it at all. The plan is the model's own document, so of course the code matches it.

That second tile is the scary one. The plan was built. The build succeeded. Every test passed. The evidence-vs-plan check said ready. And the evidence-vs-story check said no, because the plan never covered everything the story asked for.

My clearest example is GH-566. The story asked for something to work while a pod was running. The tests proved it worked while an agent was running in the same process. Plan matched, tests green, evidence complete, and we'd built the wrong thing. Only the gate that went back to the original story caught it.

(16 is a small base. I added the evidence-vs-plan check to the process later, so only 16 implementations went through both.)

What it costs

None of this is free, so here's the bill:

The judges run on Sonnet, which is about half the per-token price of the Opus model doing the building. Review took about 10% of the agent's working time and 6% of the cost. Forty cents to catch a defect before a PR is the best deal in my whole workflow.

What still got through

After all of that, I went back through every issue and pull request filed after these stories merged, looking for bugs that traced back to one of them.

One was a merge that duplicated an import another story had already added. The other (GH-120) had two bugs in a Jira adapter that a later issue reported without naming the story, so I had to trace them back through git. Not zero. But I'll take 2 out of 83.

People still matter

The machines found most of the volume. They checked every plan, every diff, and every piece of evidence, and they're cheap and fast enough to do it on every story. But the 17 Decide calls where a person overruled the model are the ones no agent could have caught, because the right answer wasn't in the code or the story. It was in someone's head. Those are the calls about what we're building and why, and I don't want a model making them. People approve, agents check.

What this doesn't prove

It's still a small (but significant) sample. 83 stories is enough to see a pattern, not enough to call it a law.
Escapes only count what someone reported. A bug nobody filed an issue for isn't in my 2%.
Some reviews never block. The adversarial plan review is advice, and 66 of its gaps were declined. I counted only the ones that changed the plan.
Timing is the agent's working time, with time spent waiting on me left out. It's an estimate built from the logs, not a stopwatch.

The new definition of done

Here's where I've landed. The model isn't the problem. It's very good, and it gets better every few months. The problem is treating its first output as its final output because it arrived confident and green.

Done used to mean "the tests pass and someone read the diff." With this much code coming at us, done now means the evidence shows we built what the plan said, and the plan covers what the story asked, checked by something other than the thing that wrote it. And none of it works unless the tests underneath are real.

If you want help building this kind of process for your team, that's what we do at Pinnacle Solutions Group.

Have your own numbers on how often AI output is right the first time? I'd love to compare notes. Hit me up on GitHub or wherever you found this post.

The Tests That Would Not Notice

drmikecrowe — Sat, 10 Oct 2026 09:00:00 +0000

This post was written with AI assistance (Claude) for structure and formatting. The data, the bugs, and the opinions are entirely my own.

Until I started building out an AI-driven SDLC this year, I had never run a mutation test. I'm not sure I could have told you what one was. I've been writing software for a long time, I've shipped plenty of code with a coverage badge on it, and I treated that badge as the answer to "is this tested?" It isn't, and it took an agent pipeline to make me find out.

Here's the thing: both projects in this post already had significant unit tests and good coverage. Neither one was a neglected codebase. Mutation testing and adversarial review still found a pile of problems in both, and that's the point I care most about. When an AI writes most of your code, your tests are the only thing that proves it's moving in the direction you intended. A suite that runs every line but wouldn't notice a wrong one can't do that job, so improving test quality has gone from a nice-to-have to the core of how I keep AI-written code honest.

I've written about where AI belongs in software delivery and about the PR that fooled me into adding adversarial review. Mutation testing came out of the same work. Once agents were writing most of the code, I needed checks that didn't depend on me reading every line, and "the tests pass" turned out to be a much weaker statement than I'd assumed.

Mutation testing in two lines

A mutation tool makes a small change to your code (flip < to <=, delete an argument, swap a string), reruns your tests, and checks that some test fails. If no test fails, the mutant "survived", which means your tests would not notice that change if a human or an agent made it for real.

That's the whole idea. Coverage tells you a line ran. Mutation testing tells you whether anything would notice if that line were wrong.

The scenario

Here's the scenario: a client app with a Python FastAPI backend and a TypeScript/SvelteKit frontend. It had a healthy test suite by the usual measures (about 80% statement coverage on the frontend) and nobody had any reason to think it was undertested. I added mutmut on the backend and StrykerJS on the frontend, then used the survivor lists to decide which tests to write.

The scope:

Backend (mutmut 3.8): 32 Python files, about 2,200 non-blank lines of production code, 1,766 mutants.
Frontend (StrykerJS 10): 16 TypeScript files, about 1,180 non-blank lines, about 1,070 mutants.
Total: about 3,400 lines in 48 files, roughly 2,800 mutants.

Stryker only mutated .ts files. The .svelte components were out, for reasons I'll get to in the gotchas.

The numbers

Measure	Before	After
Backend mutants detected (killed or timed out)	65% (1,139 of 1,757)	82% (1,447 of 1,766)
Backend survivors	613	319
Frontend mutants killed	257	723
Frontend survivors	464	141
Frontend mutants with no covering test	345	202
Backend tests	294	399
Frontend unit tests	501	695

Some per-file moves on the frontend: a schema validator went from 10% to 77%, an auth module from 34% to 89%, a types module from 15% to 97%. On the backend, the LLM client dropped from 131 survivors to 43 and the index loader from 101 to 42.

Both "before" columns were taken after I fixed two tool problems, so they aren't a raw first run. A raw first run would have looked much worse, and most of that would have been environment noise.

What it actually found

None of this was known before I started. Where a survivor led straight to the bug I'll say so, and where it only pointed me at the right module I'll say that instead.

Three real application bugs

The streaming client missed the standard end-of-stream signal. It read finish_reason only from the delta object, but OpenAI-compatible streams put it on the choice. A quick probe showed tokens after the stop signal still came through:

# before the fix
['Hello', 'AFTER-STOP']
# after the fix
['Hello']

The impact is bounded because a [DONE] marker normally follows, but a proxy that omits it would leak trailing content or never stop. The existing test passed because it used a non-standard stream shape. A surviving mutant in the stream parser led directly to this one.

Two of the three chunking strategies silently dropped any text before the first heading. The baseline strategy kept it; heading-aware and hybrid didn't. So switching strategies would quietly lose content. It's latent (no current page has text above its first heading, and production uses the baseline), but it was sitting there waiting. A surviving mutant in the section splitter led to it, and the new tests fail on the old code and pass on the fix.

A greedy frontmatter regex made a full index rebuild impossible for a third of the content. Any page body with a markdown horizontal rule (---) got swallowed into the YAML block and crashed the parser. That hit 123 of 354 source pages. Production kept working only because the committed index predated those pages; a rebuild would have failed. The fix was one character, * to *?. The survivor list sent me into that module, but I found the bug by probing real data while sizing the chunking problem. The tool narrowed where to look and I made the find.

The test suite had bugs too

A fixture removed a shared auth override, so component auth tests returned 401 depending on test order. The default order hid it, and nine tests failed under a different order. mutmut runs the suite in its own order as a "clean run" check, which is how this surfaced.
A depth-filter test could not fail. Its fixture page was small enough to be filtered out as a stub anyway, so the test passed with the depth filter deleted entirely.

The surprise: a slow tool forced me to look at a slow suite

Mutation testing reruns your tests for every mutant. A suite that takes 48 seconds is a minor annoyance when you run it once, and a real problem when the tool runs it against hundreds of mutants. So before I could get useful sweeps, I had to find out why the tests were slow.

The first answer was embarrassing. Two retry tests slept for real, 7 seconds each, because nobody had mocked sleep() in the exponential backoff. The tests were sitting through the actual retry delays. Patching the sleep took those seconds out, and it also let me assert the backoff schedule, which nothing had checked before.

The second answer was worse. Every test-client startup posted spans and events to a real collector and an Application Insights endpoint, about a second of network time per test. Our unit tests were sending telemetry to production monitoring and nobody knew. A single environment variable (dummy mode in the monitoring library) stopped it.

Between the two fixes, the backend suite dropped from 48.6 seconds to about 27. Neither came from a surviving mutant. They came from the cost of running mutation testing at all, which made a slow suite too expensive to ignore. Some of the value came from the tool, and some came from the work the tool forced me to do around it.

Gaps with no bug, but nothing guarding them

The retrieval core (cosine-similarity search) had no tests at all, zero-vector and ordering behavior included.
The LLM client's request URL, headers, payload keys, and defaults were untested.
The embeddings client's batching and order preservation were untested.
JWT validation had holes around required claims, exact issuer matching, identity-claim fallbacks, and key-rotation refresh.
The chat markdown renderer, which sanitizes untrusted model output, had no tests.

That last one should make anyone nervous. It's the code standing between an LLM's output and your users' browsers, and the coverage report gave no hint it was bare.

How I added it

Install with the project's existing package managers: mutmut as a Python dev dependency, Stryker and its Vitest runner as npm dev dependencies. Commit the lockfiles.
Scope it to production code only. Exclude tests, generated output, and vendored code.
Smoke-test one backend module and one frontend file first.
Fix environment problems until a clean run works.
Run the full sweep and read the results as a ranked list of "tests that would not notice."
Triage. Separate log-message and string-literal survivors from logic survivors, because their value is wildly different.
Write tests for the logic survivors and run each against the unmodified code. A new test that fails there has found a bug.
Re-run the affected modules and compare.

Here are some of the things that drove me crazy

The tools copy your tree. mutmut runs tests in a copy of the repo, so every file the app reads at import time or in tests (config, content, data) has to be listed to get copied.
A dangling symlink crashed the copy. Listing source directories one at a time worked around it.
A macOS-only crash looked like a real result. Proxy auto-detection crashed forked workers that built an HTTP client, and mutmut reported 383 mutants as "segfault." Setting the no-proxy variable fixed it. Without that, the LLM and embeddings modules would have looked untestable.
Timeouts count as detected. A mutant that breaks a loop makes a test hang until the time limit, and the tool scores that as a kill. About 170 mutants ended that way, and they dominate the runtime.
Stryker can't score Vitest Browser Mode tests. All the component tests here use it, so five files sit at 0% even though they have tests. That's also why I didn't mutate the Svelte components: every mutant would have shown "no coverage."
import.meta.glob breaks instrumentation. Stryker rewrites its string argument, so four files that use it are excluded.
No type-checker plugin, so some frontend survivors are mutants that wouldn't even compile.
Run time isn't constant. The backend sweep took about 6 minutes alone and about 35 minutes once when other jobs overlapped. I never pinned down why.

What it cost

Raw wall clock was about 19 hours, but most of that was an overnight gap and a few stretches where I walked away. Capping idle gaps at 20 minutes puts active time at about 4 hours. I sent 12 prompts total. The machine time is small (about 6 minutes for the backend sweep, 2 for the frontend), so nearly all the elapsed time went into writing the tests the survivors pointed at.

The output was about 300 new tests, roughly 2,300 lines of test code, 230 lines of tooling and CI config, and 5 lines of application fixes. Five lines. That ratio tells you where the work is.

On tokens: about 190,000 output tokens and 57 million cache-read tokens across 286 turns, and that's a lower bound because the transcripts miss part of the session. The cache-read number looks scary but it's the conversation being re-read each turn, billed far below fresh input. What kept it from ballooning was redirecting every command's output to a file and reading only summaries. Unfiltered mutation output will eat a context window alive.

Why this changed how I think about review

This is where the client work connects to my own projects. In harnessed, I went back through 42 PRs over six weeks and tallied every finding by whatever layer caught it, counting only what the PR body names explicitly:

Look at the third bar. "Author re-reading" is me reading the PR, which is the traditional human code review I've done my whole career. It caught 18 findings. Adversarial review caught 114 and mutation testing caught 54. Together those two layers found more than nine times what my own careful reading did.

I'm not throwing out human review. I still read every PR, and I still approve every merge, because people approve and agents check. But for most of my career, reading the diff was the quality process, with tests, lint, and types as a safety net underneath. This chart says the safety net and my reading together account for a sliver. Tests, lint, and types combined found 12.

The cost side matters too. The whole mechanical gauntlet on one PR ran in 81 seconds. An adversarial review round is about 5 minutes. A mutation run is the expensive layer at about 25 minutes. A quarter of my findings for 25 minutes of machine time is a trade I'll take every time.

What this doesn't prove

I'm not claiming a code review could never find the client-app bugs. But my team cross-reviewed that code, and we missed every one of them. As we accelerate with more AI-generated code, that kind of miss is only going to get more common.
About half the 319 remaining backend survivors look like log-message or string changes, but that's an unverified heuristic.
I counted the ~170 timeouts (141 in the chat router alone) as detected and didn't investigate them.
I wrote the CI workflow and helper commands but haven't run them end to end yet. I ran the tools directly.
I deliberately set no score threshold. There's no baseline to justify one, and the timeouts and browser-only code need a decision first.
The harnessed numbers are floors. A finding counts only if the PR body names what caught it, so anything the suite caught earlier never shows up.

Run it as an audit, not a gate

If you take one thing from this: point a mutation tool at a codebase you think is well tested and read the survivor list. Don't make it a CI gate on day one, because the first runs mostly measure your environment problems. Treat the output as a ranked list of places your tests wouldn't notice a change, and go write the tests it asks for.

I spent years trusting a green coverage report. Four hours with mutmut and Stryker found three bugs, a test that couldn't fail, and my unit tests chatting with production telemetry. That was a well-tested codebase. If the AI is going to write the code, the tests are how I steer it, and I want tests that would actually notice when it drifts.

Have your own mutation testing stories, or a survivor that turned into a real bug? Hit me up on GitHub or wherever you found this post.

The PR That Fooled Me

drmikecrowe — Wed, 26 Aug 2026 09:00:00 +0000

This post was written with AI assistance (Claude) for structure and formatting. The PRs, the bug, and the opinions are entirely my own.

The best-looking PR in my repo had a data-loss race in it, and the model that wrote it reviewed its own work and found nothing.

I've written before about where AI belongs in software delivery, and the short version of that post is: people approve, agents check. This post is the origin story. It's three weeks of commit history in one public repo, and I'm going to walk you through it in order, because the order is the argument.

Act one: the PR that looked perfect

On July 28 I merged PR #169 in harnessed. Opus wrote most of it. Go look at the PR body, because the body is the point.

(A word about the repo, since I'm about to link into it four times and I haven't formally announced it. harnessed is a tool for people who want different configurations of their agentic harness in container isolation: you compose recipes of skills, MCP servers, and services into a named stack, and it launches in a pod with nothing leaking in from your host config. Toolchains come from mise, every dependency is pinned, updates respect a seven-day minimum release age so a poisoned package gets caught before it reaches a build, and images get Snyk and Socket scans at build time. It's my daily driver and I'm dogfooding it hard, but it's alpha: today it supports Claude Code and omp, with more harnesses planned. If that's a tool you've been wanting and you'd like to help shape it, come on in. Otherwise, the announcement post is coming when it's ready for you.)

It has a measured-cost table justifying the design (a one-line edit to an install script cost 307 seconds through the container build and 4.3 seconds natively, so the installs moved to runtime volumes). It lists two premises that were tested and disproved along the way, with links to where each was recorded instead of quietly dropped. It documents four bugs found during the work, each fixed with a test that fails without the fix.

That is a better PR writeup than most humans produce. I know because I'm one of the humans. When your AI hands you work at that level of polish, with benchmarks and disproved hypotheses and regression tests, skepticism starts to feel rude. I got comfortable. The polish was doing exactly what polish does.

Act two: a second opinion I didn't ask nicely for

Four days later I put up PR #185, a rework of the CLI verbs. Same workflow, same model, same confident writeup: 1,766 tests passing, every removed test accounted for in a table, docs updated.

This was also the first PR I ran through CodeRabbit, mostly out of curiosity. I had already asked Opus to review its own work. It found nothing worth stopping for.

CodeRabbit found this:

Two concurrent --recipe invocations with the same recipe set derive the same name. Both can observe preexisting=False before either mints, so both return a non-None minted_dir. [...] If one of the two concurrent launches fails _build_stack while the other is still using or building the same stack, the failing process deletes a manifest the other process depends on.

A classic check-then-act race, in the one code path where concurrent same-name launches aren't exotic (sharing one stack across repos is the design goal, so two repos starting together is the expected case). A failed build could delete a manifest another live launch was using. Data loss, sitting inside a PR whose test table I had just admired.

Here's the part that actually rattled me. The race wasn't even introduced by #185. It was pre-existing code that the refactor made visible. The model that wrote it, the model that refactored it, and the model that reviewed both had walked past it every time. A differently-trained model looked at it once and stopped.

That's when the lesson landed. The author always finds a reason the work is right. That's true of me, and it turns out to be true of a frontier model reviewing its own output. Self-review isn't review.

Act three: moving the fight upstream

I didn't fix the race in #185. I scoped it out deliberately (that PR was about CLI grammar, and mixing a concurrency fix into it is how you get neither reviewed well) and recorded it as issue #287.

By the time that issue got picked up in mid-August, my process had changed shape. Read the issue comments in order and you can see the new machinery:

A DECISION comment recording the fix approach, with current line numbers, so the implementer doesn't re-derive the research.
A full SPEC posted as a comment, before any code existed, with a line I now consider load-bearing: "Approval of this spec is the gate that authorizes implementation."
The spec itself went through adversarial review before I approved it, and the review changed it. Poking holes in a document costs minutes. Poking the same holes in an implementation costs a rewrite.
A human (me) applying the spec-approved label, timestamped.

That third point deserves a beat. When I started running adversarial reviews against code, the payoff was catching bugs. When I pointed the same treatment at specs, the payoff was bigger: every hole caught at spec time is a hole that never becomes code. The mint-race spec that got implemented was not the spec I first wrote.

Act four: what "done" looks like now

The fix landed as PR #414, and its body is an EVIDENCE document. Not a description of the change. Proof, organized so a reviewer can attack it:

A spec-to-test mapping: eight scenarios from the approved spec, each mapped to a named test.
A concurrency test that deterministically fails against the old code (two threads claim ownership) and passes with the fix (one does). Not "we added tests." A reproduction first.
Five hand-planted mutants, all killed, because if your test suite can't catch a deliberate mistake it wasn't going to catch an accidental one.
Two rounds of adversarial review, findings fixed and re-checked.
And a section titled "Not proven": the container build path the suite never executes, the NFS caveat on advisory locks, the mutation tool that wouldn't run in a worktree, the fact that the final SHA's adversarial pass was substituted rather than re-run.

That last section is the one I'd defend hardest. Evidence that can't tell you what it doesn't cover isn't evidence, it's marketing. The "Not proven" list is what makes the rest of the document worth believing.

What I actually learned

The confidence is a constant. The quality is not. Every PR in this story arrived with the same certainty. One of them carried a data-loss race. Nothing in the presentation distinguishes them, which means the presentation carries no information. You have to attack the work.

Self-review doesn't work, for silicon or for me. The reviewer needs to carry none of the author's assumptions. A fresh session helps. A different model with different training helps more (with one caveat: frontier models share training data and failure modes, so even two-model review has correlated blind spots; it's a design goal, not a guarantee).

Upstream beats downstream. Adversarial review of code catches mistakes. Adversarial review of the spec prevents them, at a fraction of the cost.

This costs tokens, and it's the cheapest spend in the budget. Every adversarial pass is a second model re-reading finished work. I meant it when I called them the cheapest tokens I spent all year: the alternative was shipping the race and paying for it during someone's demo.

If you want the team-scale version of this, with gates and human sign-off and the process we're building around it at Pinnacle Solutions Group, that's the People Approve, Agents Check post. This one is just the receipt trail: one repo, three weeks, and the PR that taught me to stop trusting a clean explanation.

Have your own "the AI sounded so sure" stories? Hit me up on GitHub or wherever you found this post.

People Approve, Agents Check

drmikecrowe — Mon, 24 Aug 2026 09:00:00 +0000

This post was written with AI assistance (Claude) for structure and formatting. The design, opinions, and specifics are entirely my own.

I think the industry is measuring the wrong thing, and I think the leaderboard is making software worse.

Every vendor pitch I've sat through in the last year answers the same question: how much of the work can the AI do? Percentage of code generated. Tickets closed without a human. Autonomy tiers, like a self-driving car. That's a fine question if you're selling a model, but it's the wrong question if you're responsible for an engineering organization, because it smuggles in two assumptions. The first is that the bottleneck is typing. The second, which nobody says out loud in the meeting but everybody hears, is that a successful AI rollout ends with fewer engineers.

Both are wrong, and the second one is expensively wrong.

Where this came from

This started with adversarial code review. I began having an independent reviewer, one carrying none of my assumptions, try to break my finished work instead of blessing it. Its entire brief was to find the input that makes the code fail. That alone was worth the tokens.

I'll admit why I needed it. I had drifted into trusting the high-end models. When Opus hands you code, it hands it to you with total confidence: no hedging, no "you may want to check this," just a clean explanation of why it did what it did. I got comfortable, and you can probably see where this is going. When I started pointing a second model at that output with instructions to break it, I was genuinely shocked at the gap between how certain the first model sounded and how well the code actually held up. The confidence is a constant. The quality is not. Nothing has made me more careful than watching a model I trust get taken apart by a second model with no stake in being right.

Then I moved the whole thing upstream. I was already writing specs first, so before any code existed I pointed the same adversarial treatment at the spec itself: find the requirement that can't be tested, find the two bullets that contradict each other, find the thing I assumed and never wrote down.

That turned out to be even more valuable. My specs were full of holes I could not see, because I was the one who put them there, and every hole caught at spec time was a hole that never became code. Reviewing the output catches mistakes; reviewing the intent prevents them. The tokens I spent arguing with a machine about what I actually meant were the cheapest tokens I spent all year.

That's the part I couldn't stop thinking about. If an adversarial loop does that for one engineer working alone, what does it do for a team? So we started building it into a real process at Pinnacle Solutions Group, the consulting firm I'm part of, and we're genuinely excited about where it's going. This is us thinking out loud about where AI belongs in software delivery, and I'd rather share it while it's still being shaped than wait until it's a polished case study.

Two disclaimers. Helping engineering orgs put this in place is work Pinnacle Solutions Group sells, so read the whole thing with that in mind. And it's running on a live client engagement right now, being developed with them and fitted to how they already work. I'm describing a process I believe in and am actively proving, not results I can hand you. When I get to what it costs, I'll be specific about what I don't yet know.

The personal arc, though, is public if you want to check my work. It played out in the open on a side project of mine: the review that converted me, where a differently-trained reviewer found a concurrency bug in code that self-review had blessed, and a complete evidence run, spec through adversarial review through proof, on the fix for that same finding.

Delivery breaks at the edges, not in the middle

Think back through the projects that went badly for stupid reasons, not the ones with genuinely hard technical problems. Anecdotally, across the ones I've watched up close, nearly all of them broke at one of two boundaries.

The first is the handoff from product to engineering. Somebody writes down what they want, and what they write down is incomplete: acceptance criteria you can't actually test, an assumption nobody stated, two requirements that quietly contradict each other. Nobody notices at write time. Somebody notices on day four of the sprint, in a meeting, after the money is already spent.

The second is the handoff from "the work is done" to "the work is proven." A change gets reviewed by looking at code. Looking at code answers a question (is this reasonable code?) that nobody actually needed answered. The question is whether this change does what product asked for, and how we know.

The middle, where the typing happens, is mostly fine. That's where every vendor is aiming.

So we aimed somewhere else, and not out of contrarianism. The speedup in the middle is real, but it's a commodity: every tool on the market delivers it, and your team collects it no matter whose logo is on the invoice. The failures at the edges cost sprints, not minutes, and nobody was selling anything there.

We put an AI agent at each of those two boundaries, and we gave it a job description with a hard limit on it: people approve, agents check. An agent in this design never approves anything, never merges anything, and never decides what gets built. It reads, it tests, and it reports, and the verdict belongs to a person every time.

Underneath, there's nothing exotic here. It's spec-driven development feeding test-driven development, wrapped in a workflow that makes both of them non-optional.

None of the individual pieces is new. Planting a deliberate fault to prove the tests would catch it is mutation testing, an idea from the 1970s. Coverage on changed lines is standard CI configuration. Requirements reviews, design reviews, spec-first, test-first: all decades old. What I'd defend as new is where they sit. When AI writes most of the code, plausible work arrives faster than anyone can judge it, and these old disciplines are how judgment keeps up.

Gate 1: AI that protects human judgment

The first gate sits between product and engineering. It runs on a story that's headed into refinement, and it answers one question: is this specified well enough to be refined?

An agent reads the story and puts it to this test: could an engineer prove, from what is written here, that the intent was met? That question does a lot of work. It forces every acceptance criterion to be verifiable rather than aspirational. It surfaces the assumption sitting under the third bullet that nobody wrote down. It catches the place where criterion two and criterion five can't both be true.

This is aimed squarely at helping the people who write those stories. Product and marketing are often not technical, and they are being asked to express intent precisely enough for engineers to manifest it. That's a hard thing to do with no feedback loop. Today the feedback arrives four days into the sprint, in a meeting, in front of people. This gate gives them the same feedback in writing, in minutes, from a machine that doesn't sigh.

A story that passes moves on to refinement. That's not an agent approving the work; it's an agent confirming that product has finished product's job, so engineering can start engineering's job. Nobody's clear, well-specified ticket sits in a queue waiting for a human to rubber-stamp what an agent already confirmed. What gets built is still decided by people in refinement, downstream of this gate, exactly as it was before.

A story that fails bounces back to whoever wrote it, with written findings. And not vague ones like "this needs work," but specific ones: criterion three has no observable outcome, here is what's missing; criteria one and four conflict under this condition; this assumes the user is already authenticated and the story never says so. Bad tickets bounce before they cost a sprint, and they bounce with a document instead of a meeting.

The part that matters most: the gate does not fill in the gaps. It finds them and hands them back.

The industry instinct is the opposite. Vague requirement? Let the model guess. It'll produce something plausible, and plausible is very close to correct, and the demo will go great. But deciding what the business actually wants is not a gap-filling exercise. It's the highest-judgment work in the building, done by the people with the context and the scars. Automating a plausible guess over the top of that doesn't remove the work, it just hides it until it's expensive.

So we deliberately preserve the human conversation at exactly the point where the human is the value. What we change is the price and the timing: earlier, cheaper, in writing, before anyone has spent a sprint on the wrong thing.

Between the gates: building a specification worth implementing

This is where spec-driven development actually lives, and it's the part I had wrong in my own head until recently. I'd lumped it all into "refinement," which is one word doing two jobs.

Refinement answers is this item ready: is it clear, is it testable, roughly how big, in what order. That's product's item, and Gate 1 is what protects it. Building the Implementation Plan answers a different question, how will we build this, and it belongs to the team. Those are two artifacts with two owners, and collapsing them is how you end up with a product owner refereeing a design argument.

Pre-assembling the context

Once a story clears Gate 1, an agent reads the codebase against it and assembles a context pack: which subsystems this touches, the patterns already in use there, the tests that cover that ground today, the prior art from the last time somebody did something similar.

This is small and it's cheap and I think it's one of the highest-leverage pieces in the whole process, because it gets produced while nobody is waiting on it. If your team still holds a refinement meeting, everyone walks in already knowing what the story touches instead of speculating out loud, which is most of what sizing actually needs, without paying for a full design on every item in the backlog. Later, it's the input the planning agent needs, already gathered.

When it runs is a knob, not a rule. A team that plans ahead can build the pack as soon as the story clears the gate. A team running lean can build it just in time, when an engineer picks up the story, so it reflects the codebase as it is that morning instead of three weeks ago. The orchestration bends to the team; the artifact is what matters.

Then the plan, when someone picks up the work

An engineer takes the story. An agent drafts the Implementation Plan from the approved Requirements and the context pack: approach, the tests that will prove each acceptance criterion, risks, and how we back it out. A second agent, with none of the first one's context, attacks that draft. The engineer resolves the findings, owns the result, and signs it. Then code gets written.

That agent draft is not a shortcut around the engineer. It's a better starting point than the engineer would produce alone under sprint pressure, because it actually read forty files instead of pattern-matching off four. What it cannot do is decide whether this approach is right for where the system is going, whether the rollback is real, or whether the risk list is honest. That's the signature.

And when the plan reveals the estimate was wrong, that's the process working. You find out on day one, in writing, with a reason. A plan can bounce a story back the same way Gate 1 does.

Why the agent may fill this gap and not the other one

Gate 1 refuses to fill gaps in the Requirements. Here an agent drafts a whole specification. That looks inconsistent. The difference is the cost of a wrong guess.

A wrong guess about product intent is unrecoverable. Nothing downstream can detect it. Every test passes, the evidence is clean, and you shipped the wrong thing correctly. A wrong guess about technical approach is checkable. The Requirements check it, an adversarial reviewer checks it, and so does what we call the gauntlet: an automated review that holds the implementation to a standard while it's built, with every changed line tested, the tests proven able to catch a deliberately planted fault, and every check the merge gate will run passing before the work is called done. So does the engineer who has to put their name on it. The blast radius is different, so the rule is different.

Gate 2: AI that forces human understanding

The gauntlet ends with output: check results, coverage of the changed lines, planted faults caught. That output says the code was built to standard. It does not say the right thing was built, and treating those as the same claim is exactly the conflation this gate exists to prevent.

That's what the Evidence document is for. Its shape is simple: each acceptance criterion from the Requirements, how the change satisfies it, and how that was verified. Tests mapped to specific criteria. The gauntlet's results. Screenshots where the change touches something a person looks at. And one section every Evidence document must carry: what is not proven. Proof that can't state its own limits isn't proof, it's marketing. The document isn't written at the end as paperwork; it accumulates as the work is built. Evidence is the proof that the acceptance criteria were met, assembled from what the gauntlet produced.

Gate 2 is the formal check on that proof. An agent validates the Evidence against the Requirements before the change is ever put up for review: does what the gauntlet proved actually prove what product asked for? Then the engineer must read the Evidence against the Requirements and personally decide whether the work meets the intent. The question isn't whether the tests are green; it's whether the thing does what was wanted. The AI assembles the proof, and the engineer renders the verdict and signs their name to it.

So a piece of work carries three named documents by the time it ships. Requirements says what and how we'll know. Implementation Plan says how, before the build. Evidence proves it. Each one carries a human approval, and each one was checked by an agent first.

But a signature is cheap. Nothing about signing a document stops an engineer from skimming it and signing anyway. If all this gate did was swap "skim a green build and merge" for "skim an Evidence doc and sign," I'd have added ceremony and called it rigor. A process that only asks for a signature gets signatures.

That's why the signature isn't the mechanism. The next section is.

The forcing function: every change becomes knowledge transfer

The engineer who signed the verdict then uses that same Evidence to walk reviewers and QA through the change, out loud, in front of colleagues. It's not a diff review, it's a narrative: here's what product asked for, here's how this satisfies it, here's how we know.

That's the part you cannot fake. You can sign a document you didn't read, but you cannot stand in front of two colleagues and explain a change you don't understand, and everyone in the room knows it within about ninety seconds. The Evidence document isn't the enforcement. It's the script for a conversation that is the enforcement.

"Out loud" flexes with the team. For a distributed group that means a live call or a recorded walkthrough rather than a conference room, and the rule holds either way: the engineer explains the change to people who can ask questions.

This is the thing every engineering leader is quietly afraid of: an engineer generating a change they don't understand, skimming a green build, and shipping it. I can't make that structurally impossible; anyone who tells you they can is selling. What I can do is put the moment of "explain this to your peers" between the AI's output and production, and make it the normal cost of shipping. Not understanding your own change stops being invisible.

And then think about what that does over a quarter. Every single change becomes a teaching moment, delivered by someone who is required to understand it well enough to defend it. The team learns how the system is changing, update by update, from a person rather than from a diff nobody reads. Institutional knowledge stops being a thing that lives in three people's heads and starts being a thing the process produces as a byproduct.

This is the same set of models everyone else is using, aimed at a different problem. The difference isn't the tooling; it's where the human sits, and what the process makes them do when they get there.

Your engineers matter more now, not less

Nothing in this design is aimed at needing fewer engineers. It's aimed at raising the quality of what your engineers ship when they work with AI. Those are different goals, and they lead to opposite decisions at nearly every fork.

A model will produce something plausible for almost any request. Plausible is the dangerous part: it compiles, reads well in review, passes the tests it was written alongside, and quietly does the wrong thing. The only reliable defense is a person who knows the system, knows what was actually asked for, and is equipped to tell the difference. No model removes that need, because judging whether the AI did what was intended is itself a judgment about intent.

So the volume of code goes up and the value of engineering judgment goes up with it. Every gate exists to put a competent engineer in front of the right question at the right moment, with the evidence in hand to answer it. Gate 1 hands them a request worth building. Gate 2 hands them the proof and makes them rule on it. The walkthrough keeps the whole team sharp enough to keep doing that as the system grows.

An org that cuts engineers to pay for AI ends up with more code and less capacity to judge it. That is the worst version of this. If your team gets faster, the win is that they take on work they couldn't before, not that there are fewer of them.

What this costs

This is the part vendors skip, so here it is up front: this process is slower than what the market currently expects from AI. Gate 1 adds a loop before refinement. The Implementation Plan adds a draft and an adversarial pass before code. Tests written from acceptance criteria are slower to write than tests written from finished code. Evidence takes real effort to produce, and the walkthrough takes real time from more than one person.

The walkthrough is the line I expect the most pushback on, and the pushback is fair: multiple colleagues per change, out loud, does not scale like a CI job. I'm not going to argue it away, because it's the deliberate purchase. Quality of what ships and a team that understands what shipped are the two things this whole design exists to buy, and the walkthrough is where both get paid for. How deep it goes, and for which changes, is one of the things the current engagement is calibrating.

It costs more tokens, too. Every adversarial pass is a second model reading work the first model already did. The context pack is a full codebase read before anyone has picked up the story. Attacking a spec, then attacking an implementation plan, then validating Evidence against the Requirements: none of that is free, and it is spend you would not have if you simply asked a model for the code and merged what came back.

I think it's the best money in the budget. I called those the cheapest tokens I spent all year and I meant it literally: you are buying rework you never have to do, at a price you can see up front. But it's a real line item, it's larger than a single-agent workflow, and a client should agree to it going in rather than discover it on the first invoice. Spend ceilings and usage alerts go in on day one for exactly that reason. Adoption shouldn't come with an open tab.

The middle is genuinely faster. An engineer with AI leverage and a clear spec outruns the same engineer without it, and it isn't close. But that gain is smaller than the number on the slide, because we are deliberately spending part of it on tests and proof.

That said, the paragraphs above make this sound slower than it is. Most of what this process standardizes should already be standard operating procedure in most organizations: tests tied to the requirements, review that checks intent, proof that the work does what was asked. I'm not piling new work on top of delivery. What I'm mostly doing is refusing to let AI's speed become the excuse to skip work that was always owed. The velocity this trims is the false kind: the perceived gains of merging plausible code fast and paying it back later as rework.

So what I'm claiming is confidence. Fewer sprints spent building the wrong thing, fewer changes that pass review and fail intent, and a team that understands what it shipped. If the number you're optimizing is lines per day, I'm the wrong call. If it's the ratio of shipped work that actually did what was asked, this is the trade I'd make.

What holds it up

The gates only work if what they're checking is real. Two supporting pieces do that work.

The gauntlet. Tests come from the acceptance criteria, not from the finished code, so they verify what was asked for instead of confirming whatever got built. The gauntlet's job is to prove that testing is real: every changed line covered, and planted faults confirming the suite would catch a genuine mistake. A suite that can't catch a deliberate mistake wasn't going to catch an accidental one.
Adversarial review. An independent AI reviewer, carrying none of the author's assumptions, is briefed to break the work: find the input that makes it fail. Confirmed findings get fixed and re-checked. This is where the whole thing started for me, and it's still the piece I trust most. Self-review doesn't work. The author always finds a reason the work is right. One caveat: two frontier models share training data and failure modes, so this independence is a design goal with limits. The reviewer has none of the author's context, and that catches a lot, but it can still share the author's blind spots.

Where this doesn't fit

This process assumes you know what you're building. Exploratory work, spikes, R&D where the spec is the thing you're trying to discover: gating that behind Requirements is backwards, and we don't try. The honest artifact there is a timebox and a writeup, and the process starts after.

It also assumes changes worth the ceremony. A dependency bump or a typo fix doesn't need an Implementation Plan and a walkthrough, and a process that pretends it does will get routed around within a month. Part of what we're calibrating with the current client is exactly that tiering: which changes get the full treatment, which get the gauntlet alone, and who decides.

And a three-person team probably shouldn't run this as written. The walkthrough is priced for teams big enough that knowledge silos are a real risk. Below that size everyone already knows every change, and the gates worth keeping are the two adversarial reviews.

How it lands in your org

We customize the gates to fit, and the fit runs in two directions. The gates can hook onto the states your team already has: no new methodology, nothing replaced, no migration. Or they can anchor a leaner workflow, because once agents build the context, draft the plan, and prove the work, some of your ceremonies stop earning their place on the calendar, and a team that notices is free to act on it. Which direction to take is your call. The gates hold either way.

And it's built to end. We lead the first run, coach the second, and step back to review only for the third. If your team still needs us on run four, we designed it wrong.

What to ask the next vendor, including us

If you're evaluating AI in your delivery process, the autonomy percentage on the slide tells you almost nothing about whether your delivery will improve. Three better questions, and I'd expect to answer all of them myself:

Where does the human judgment live, and what protects it? If the answer is that the AI fills in whatever product left vague, you're buying a machine that converts unclear thinking into shipped code at high speed.

How do you prove the work meets the intent? If the answer is "code review," ask what question code review actually answers. Then ask who on the team will be able to explain the change six months from now.

What does this slow down? Any honest process has a cost. A vendor who reports only acceleration is either not measuring or not telling you.

And ask what the vendor thinks happens to your engineers. If the pitch is a headcount line, walk. We didn't get here by asking how much of the job the AI could take. We asked where delivery breaks, and the answer was the same two places it's always broken. The machines are genuinely useful there. They just aren't in charge there, and they need better engineers around them than they did before, not fewer.

This is what I think about most days right now. We're building it with a client as I write this, shaping it to their process rather than handing them ours, and the parts that survive contact with a real team are the parts worth keeping. If you're wrestling with the same question of where AI belongs in your delivery process, and how you'd prove it worked, I'd genuinely like to hear how you're approaching it, whether or not you ever talk to us. Find me on GitHub or wherever you found this post.

Context Binoculars: Understanding LLM Context Windows

drmikecrowe — Sat, 13 Sep 2025 04:00:00 +0000

Remember as a kid (or in my case as an adult) when you pulled out binoculars and gazed around the room? It’s a bit disorienting for sure, but also fun. And I guess if you’ve ever wondered about what I do for fun, now you know.

It struck me the other day that this was the ideal representation for how LLM context works. While my demonstration you’re going to see below is extreme, this is the reality of what we’re facing when we’re trying to get LLMs to do big projects. The context window is everything, and you’re hearing more and more about context engineering in online articles. Ensuring that the LLM has exactly the right context greatly improves the accuracy of the results that you get out the other end.

There's not much more to this article. All I'm really trying to do is viscerally explain what the context window is and give you a feeling of why managing your context is so important.

Check out the visual demonstration here: context-binoculars

The Ultimate Design Review: Orchestrating AI with Task-Based Workflows - Part 6 of 6

drmikecrowe — Sun, 07 Sep 2025 00:00:00 +0000

Part 6 of 6: The AI-Assisted Development Workflow Series

This is the final installment in our six-part series on building an AI-assisted development workflow. We’ve set up our infrastructure, taught the AI our coding standards, and established a robust “memory bank” of project rules. Now, it’s time to put it all to the test with the ultimate challenge: a comprehensive, end-to-end design review of the entire codebase.

TL;DR

Want to see what we're building? I've created two example posts that show the actual output from this automated design review system:

Design Review Feedback Example - See the comprehensive analysis that the AI generates, including architectural strengths, critical issues, and detailed findings across all layers of the codebase.
Design Review Action Items Example - See how the AI transforms the analysis into a structured, actionable backlog with 18 major tasks and 72 subtasks, each with clear implementation steps and acceptance criteria.

These examples show the real output from running this system on an actual codebase. The AI identified critical architectural violations, generated specific improvement tasks, and created a comprehensive project plan - all automatically.

Series Overview:

Part 1: MCP Servers, Ports, and Sharing - Setting up the foundation
Part 2: ESLint Configuration Refactoring - Cleaning up tooling with AI
Part 3: Custom Architectural Rules - Teaching AI to enforce design patterns
Part 4: Task Orchestration - Managing complex refactoring workflows
Part 5: Project Rules for AI - Creating effective memory banks and guidelines
Part 6: The Ultimate Design Review - Putting it all together

The Challenge: From “Codebase” to “Action Plan”

How do you review an entire src/ directory? You can’t just feed it to an AI and say, “find problems.” The context is too large, the potential feedback is too broad, and the output would be a wall of text, not an actionable plan. To understand the issue, grab a pair of binoculars and use them in your office: Have a coworker toss an index card somewhere in your office and then search for that with the binoculars. If you have a really large code base, have your coworker put it in the office somewhere. That’s what your fighting:

The binoculars symbolize the AI context. It’s a fixed width window of information the AI can maintain
The index card is what you are asking the AI to find

My goal was to create a system that could:

Systematically analyze the entire codebase in manageable chunks.
Evaluate each chunk against multiple, specific architectural rules.
Produce a structured, prioritized, and actionable list of tasks.
Be entirely automated and repeatable.

To achieve this, I created a multi-layered prompting strategy that uses the TaskFlow MCP server to orchestrate the entire workflow. The process is driven by two key prompt files.

The Master Plan: `new-design-review.md`

The first file, @memory-bank/prompts/design-review/new-design-review.md, is the high-level orchestrator. It doesn’t perform the analysis itself; it instructs the AI on how to set up the entire project plan within TaskFlow tasks.

Here’s the strategy it lays out:

Create Two Buckets: The first thing it does is create two separate requests in the task server: one for the analysis process and one for the action items. This is a critical separation of concerns. The analysis tasks will be marked as “done” once the review is complete, but the action items will form a living backlog of work to be done.
Divide and Conquer: It then breaks the entire src/ directory into logical chunks: Core App Files, Components, Services, Machines, Hooks, etc. For each chunk, it creates a parent task in the “analysis” request.
The Mini-Review Subtasks: This is where the granularity comes in. For each parent task (e.g., DR Part 3 - Components (Dashboard)), the prompt instructs the AI to create four specific subtasks:

The Instruction Manual: `request-2-instructions.md`

If the first prompt is the “what,” then @memory-bank/prompts/design-review/request-2-instructions.md is the “how.” This file is a set of instructions for the AI on how to behave while it is executing the analysis subtasks.

Its most critical directive is this: You MUST create new tasks in the “Action Items” request for every issue you find.

It strictly forbids the AI from simply summarizing its findings. Instead, it must translate every identified improvement into a well-formed task, complete with a title, description, and severity, and add it to the other request bucket. This is what turns a qualitative review into a quantitative project plan.

The Results: An Actionable Backlog

After running the AI through this workflow, the taskflow-mcp server populates a YAML file with the results. The file from my design review, @DESIGN-REVIEWS/2025-08-19/tasks.yaml, is over 800 lines long, but here are a few snippets that show the power of this system.

First, you can see the completed analysis task for the “Core App Files,” with its four subtasks all marked as done:

- id: task-62
  title: DR Part 1 - Core App Files
  description: |-
    Analyze the following files for design patterns, code quality, and architectural consistency:nn- src/App.tsxn- src/appMachineContext.tsn- src/config.tsn- src/context/DashboardDataContext.tsx
    ...
  done: true
  subtasks:
    - id: subtask-71
      title: Overall Design Analysis
      done: true
    - id: subtask-72
      title: Effect Pattern Analysis
      done: true
    - id: subtask-73
      title: Component Styling Analysis
      done: true
    - id: subtask-74
      title: State Machine Analysis
      done: true

More importantly, the real output is the list of actionable tasks generated in the other request. The AI identified dozens of specific, granular improvements, each perfectly formatted according to the instruction prompt.

Here are a few examples of the generated tasks:

A high-priority refactoring task based on our Effect pattern rules:

- id: task-98
  title: Add Effect-based validation to ExcelService
  description: ExcelService.ts contains complex validation logic in generateExpenseReport function. Add Effect-based validation using E.succeed().pipe(E.filterOrFail()) pattern for report and expense data validation.

A task to fix a clear violation of our Tailwind CSS architecture:

- id: task-139
  title: Fix EmptyState.tsx UI component margin violations
  description: Remove margin classes from EmptyState.tsx component. The component contains margin classes mb-4, mb-2, mb-6 that should be handled by parent layout components. Read memory-bank/AGENTS.md and review memory-bank/project-rules/* before starting.

A task for a significant architectural improvement to break up a monolithic file:

- id: task-99
  title: Extract ExcelService helper functions to separate modules
  description: ExcelService.ts is a 485-line monolith with many helper functions. Extract helper functions into separate modules (excel-formatting.ts, excel-data.ts, excel-styles.ts) to improve maintainability and testability.

Conclusion: The Sum of the Parts

This automated design review is the culmination of our entire AI-assisted workflow. It seamlessly integrates:

The Agent Protocol (AGENTS.md): Provides the core principles for the AI’s analysis.
The Project Rules (project-rules/): Act as the specific, enforceable standards for the review.
The Prompting Strategy (prompts/): Orchestrates the complex workflow, breaking it down into manageable steps.
The Task Server (taskflow-mcp): Captures the output, transforming a review into an actionable project backlog.

By investing the time to teach the AI how we build software, we’ve unlocked the ability to automate one of the most time-consuming and critical parts of the development lifecycle. We’ve moved beyond simple code generation to a world where our AI partner can actively help us manage quality, reduce technical debt, and enforce architectural consistency at scale. It’s a powerful glimpse into the future of software development.

Project Rules for AI - Part 5 of 6

drmikecrowe — Sun, 07 Sep 2025 00:00:00 +0000

Part 5 of 6: The AI-Assisted Development Workflow Series

This is the fifth installment in a six-part series exploring how AI is transforming modern development workflows. In this series, I’ll walk through my journey of building an AI-assisted development environment, from basic infrastructure setup to advanced architectural enforcement and task orchestration.

In the previous posts, we’ve taught our AI assistant how to understand and enforce specific architectural rules. We’ve built a robust system for linting, testing, and managing complex tasks. But how do we ensure the AI behaves consistently and predictably over the long term? How do we give it a “personality” and a “memory” that aligns with our project’s philosophy?

The answer lies in creating a comprehensive set of guidelines that the AI can consult before every action. In my setup, this is split into two key parts: the Agent Protocol and the Project Rules. Together, they form the “brain” of my AI development partner.

The Agent Protocol: Defining the Personality

The first piece of the puzzle is the memory-bank/AGENTS.md file. I think of this as the AI’s constitution or its core programming. It doesn’t contain project-specific code rules; instead, it defines the AI’s high-level behavior, its interaction style, and its core principles.

You can see some of the key sections from my AGENTS.md file:

Core Philosophy & Attitude : This sets the tone. I want an “Expert Peer,” not a subservient assistant. It should be direct, proactive, and value clean code principles above all else.
Interaction Protocol : This defines how we communicate. No lectures, no fluff. It must clarify ambiguity and warn me if I’m asking it to do something that violates a convention.
Code Generation & Modification : This governs how it writes code. It should edit files directly, respect formatting, and, most importantly, follow a specific code review process.
Context & Memory Bank : This establishes the hierarchy of knowledge. The memory-bank/ is the single source of truth, more important than the codebase, conversation history, or its own external knowledge.

This file establishes a baseline of behavior. It ensures that no matter what the specific task is, the AI approaches it with the same professional, efficient, and principle-driven mindset.

The Project Rules: The Letter of the Law

If the AGENTS.md is the constitution, the files in memory-bank/project-rules/ are the specific, legally-binding statutes of the project. These are granular, often file-specific rules that the AI must follow.

These rules cover everything from technology usage to coding patterns:

00-pnpm.md: “Always use pnpm for package management. Period.”
02-effect-pattern.md: A detailed, multi-point guide on how to use the Effect library, including import styles, composition patterns, and where E.runPromise is allowed to be called (hint: only in XState machines).
08-tailwind.md: A treatise on our Tailwind CSS architecture, enforcing the strict separation of layout and style components.
09-state-machine-patterns.md: Defines the actor-based hierarchy for our XState implementation, ensuring actions are routed correctly.

These rules are not suggestions; they are directives. The AGENTS.md file instructs the AI to treat the memory-bank/ as its authoritative source of truth, and these files are the content of that memory. When I ask the AI to write a new component, it knows it must use CVA, it must not include margins, and it must get its available actions from a state machine actor—not because I told it to in the prompt, but because the rules demand it.

Putting the Rules into Practice: Prompting with Precision

Having this well-documented set of rules is powerful, but the real leverage comes from how I use them to guide the AI’s work. The rules aren’t just a passive library; they are an active part of my prompting strategy.

A perfect example is my process for a comprehensive design review, which will be the topic of the next blog post. I don’t just ask the AI to “review the code.” I give it a highly structured plan that explicitly references the project rules at each step.

Using a task planning tool, I create a master “Design Review” request, which is then broken down into smaller, logical chunks (e.g., “DR Part 1 - Core App Files”, “DR Part 2 - Components”). For each of these chunks, I create a series of subtasks, and this is where the magic happens. The description for each subtask points the AI to the exact rule it needs to enforce.

For example, the subtask for analyzing Effect patterns looks like this:

Title : “Effect Pattern Analysis”

Description : “Analyze Effect usage against memory-bank/project-rules/02-effect-pattern.md. Identify pattern violations, missing Effect implementations, and improvement opportunities. Generate specific action items.”

I do the same for other concerns:

Component Styling Analysis : “Review component styling against memory-bank/project-rules/08-tailwind.md…”
State Machine Analysis : “Assess state machine implementations against memory-bank/project-rules/09-state-machine-patterns.md…”

This approach transforms the memory bank from a simple knowledge base into a set of precise instruments. I can direct the AI’s focus with surgical accuracy, ensuring that each part of the codebase is evaluated against the correct, pre-defined standards. It’s a clear, explicit contract: “For this task, these are the rules that matter.”

The Killer Feature: The AI Code Review

This brings me to the single most valuable instruction in the entire system. Buried in the AGENTS.md file is a simple-sounding, yet profound, directive:

Code Review Process : After making ANY significant code changes, conduct an internal round table discussion with David Thomas (The Pragmatic Programmer), Andrew Hunt (The Pragmatic Programmer), and Uncle Bob (Clean Code). If they would approve, proceed; if not, iterate until they would approve.

This is where the magic happens.

This instruction forces the AI to pause and reflect. It takes its generated code, which already conforms to the strict Project Rules I’ve pointed it to, and then evaluates it against the higher-level principles of clean, pragmatic, and maintainable code embodied by these three legends of software engineering.

It’s a two-layer filter:

Does the code follow the rules? (e.g., Is the Effect pipe() structured correctly? Are the Tailwind classes semantic?)
Is the code good? (e.g., Is it simple? Is it maintainable? Is there a better way to do this?)

This internal dialogue has been a game-changer. The AI doesn’t just produce code that works; it produces code that is thought-out, clean, and aligned with the project’s deepest architectural and philosophical goals. It catches things I would miss in a manual code review, and it does it instantly. It’s like having a senior architect constantly looking over your shoulder, providing expert feedback in real-time.

By combining a high-level “personality” with a set of explicit, enforceable project rules, and then actively using those rules in our daily prompts, we create an AI assistant that is not just a tool, but a true partner in the development process.

In the final post of this series, we’ll put all these pieces together and see how this system performs in a full-scale, complex design review.

How I Taught My AI Pair Programmer to Be Our Team's Tailwind CSS Cop - Part 3 of 6

drmikecrowe — Sun, 07 Sep 2025 00:00:00 +0000

Part 3 of 6: The AI-Assisted Development Workflow Series

This is the third installment in a six-part series exploring how AI is transforming modern development workflows. In this series, I’ll walk through my journey of building an AI-assisted development environment, from basic infrastructure setup to advanced architectural enforcement and task orchestration.

We’ve all been there. You start a new project with Tailwind CSS, and everything is beautiful. The utility-first approach is fast, flexible, and keeps you right in your HTML. But as the project grows and the team expands, the CSS landscape can start to feel like the Wild West. Utility classes get sprinkled everywhere, components start to blur the lines between structure and style, and soon you’re overriding margins and fighting for specificity.

I love Tailwind, but I knew we needed to introduce some architectural discipline before things got out of hand. The problem is, architectural rules are only as good as their enforcement. Documentation gets stale, and nagging in code reviews doesn’t scale.

So, I had a thought. What if I could teach our team’s AI pair programmer to be our automated style cop? What if it could not only understand our rules but actively enforce them?

The Convention: A Three-Tier System for Sanity

First, we needed a clear, simple convention. We decided on a three-tier component architecture designed to enforce a strict separation of concerns:

Layout Components (src/components/layout/)
UI Components (src/components/ui/)
Feature & Page Components (src/pages/, etc.)

This system ensures that our UI components are truly reusable and that our layout logic is centralized and predictable.

The Solution: Hiring an AI for Architectural Enforcement

With the convention defined, I turned to my AI assistant. I described the three-tier system and asked, “Can you build a custom ESLint rule to enforce this?”

Working with the AI was like having a hyper-competent junior developer who knew the ESLint AST (Abstract Syntax Tree) better than I ever will. I provided the high-level architectural goals, and it handled the implementation details.

How the Custom Plugin Works

The resulting ESLint rule is a simple but powerful detective. For every component file, it performs three steps:

Location Check : First, it looks at the component’s file path. Does it live in src/components/layout/, src/components/ui/, or somewhere else? This determines which rulebook to apply.
Class Inspection : Next, it parses the JSX and extracts all the Tailwind classes from the className prop. It’s smart enough to handle string literals and basic template literals.
Rule Application : Finally, it applies the logic:
- If it’s a UI Component , it scans the class list for any margin patterns (m-*, mx-*, mt-*, etc.) and flags them as errors.
- If it’s a Layout Component , it checks for any non-layout classes (like bg-red-500, font-bold, etc.) and flags those.
- If it’s a Feature/Page Component , it does nothing. They are free to compose as they see fit.

The AI generated the rule file, a utility file to hold the class definitions, and even modified our main eslint.config.js to wire it all up.

The Results: The Moment of Truth

I ran the linter across our codebase, and it worked perfectly. It immediately flagged several components that were violating our new convention. Here’s a summary of what it caught:

AppLayout.tsx & Sidebar.tsx : These core layout components were caught using styling classes like bg-gray-100, text-purple-700, and font-bold. The linter correctly identified these as style concerns that should not be in a layout component.
EmptyState.tsx & ErrorState.tsx : These UI components had mb-4 and mt-2 margins, violating our “marginless UI” rule. The parent component should be responsible for this spacing.
FileDropzone.tsx & MainContent.tsx : These UI components were using mx-auto, another margin-related property that is the responsibility of a layout container.
StatCard.tsx : This UI component was using mt-1 and ml-1, which are small but important violations of the architectural pattern.

Each of these warnings was a clear, actionable task for us to clean up the codebase and align it with our new, more maintainable architecture.

Conclusion: More Than Just Code Generation

This experience was a powerful demonstration of how to use LLMs for more than just generating boilerplate or fixing simple bugs. By teaching the AI our team’s specific architectural patterns, we were able to turn it into a partner for enforcing code quality and consistency at scale. It automated the tedious work of code validation, freeing us up to focus on building features.

tures.

From Linter Chaos to Orchestrated Tasks - Part 4 of 6

drmikecrowe — Sun, 07 Sep 2025 00:00:00 +0000

Part 4 of 6: The AI-Assisted Development Workflow Series

This is the fourth installment in a six-part series exploring how AI is transforming modern development workflows. In this series, I’ll walk through my journey of building an AI-assisted development environment, from basic infrastructure setup to advanced architectural enforcement and task orchestration.

In my last post, I talked about how my AI pair programmer and I created a custom ESLint rule to enforce our new frontend architecture. It was a huge success. The linter, now armed with our specific rules, scanned the codebase and… gave us a giant list of things to fix.

This is a classic “good news, bad news” scenario. The good news? We had a precise, automated way to detect architectural drift. The bad news? We had a long, intimidating wall of terminal output listing every single violation.

/home/mcrowe/.../AlertDialog.tsx
  19:8 warning Unregistered class detected: data-[state=open]:animate-in
  19:37 warning Unregistered class detected: data-[state=closed]:animate-out
...
/home/mcrowe/.../EmptyState.tsx
  25:21 error UI Component contains a forbidden margin class: 'mb-4'
...
✖ 60 problems (2 errors, 58 warnings)

A raw list of errors is a starting point, but it’s not a plan. How do you manage this work? How do you distribute it? How do you ensure each fix is made with the right context? Just handing a developer (or an AI) a raw log is inefficient.

The Tool: TaskFlow MCP for Orchestration

This is where another tool in our AI-assisted workflow comes in: TaskFlow MCP. It’s an open-source task orchestration server designed specifically for managing development work between humans and AI agents.

Instead of just working from a simple prompt, TaskFlow allows us to break down a high-level request (like “Fix all these linter warnings”) into a structured plan of discrete, trackable tasks.

Crucially, as the user, I can insist that each task be configured with specific context and instructions. This is the killer feature. It turns a simple “fix this” command into a rich, context-aware work order.

The Process: Turning Chaos into a Plan

I fed the messy ESLint output to my AI assistant and gave it a new prompt:

"For each of the UI warnings below, use plan_task to create a ‘Fix UI layout vs. styling overlap’ request. For each violation, create a sub-task. Ensure that each task specifically includes these instructions:

Read memory-bank/AGENTS.md

Review memory-bank/project-rules/*"

The AI, using the TaskFlow MCP tool, did exactly that. It parsed the linter output, grouped the violations by file, and created a formal plan. Each task was not just a file path and a line number; it was a complete work package containing:

A Clear Title : e.g., “Fix AppLayout.tsx layout component styling violations”
A Detailed Description : Explaining exactly which classes were violating our architectural rules.
Embedded Context : The prerequisite instructions to review our project’s agent protocols and architectural rules before starting work.

The Result: An Actionable, Context-Aware Plan

Tasks are maintained locally

The output wasn’t a log file; it was a professional task report, ready for any developer—human or AI—to pick up and execute. It transformed a chaotic list of problems into a manageable, parallelizable set of tasks.

Here is the exact markdown file that the tool generated. It’s clear, organized, and ensures that whoever does the work has the full context required to do it right.

Task Status Report: Fix UI layout vs. styling overlap issues identified in ESLint warnings

Generated on: 2025-08-17

Overall Progress: 0%

Total Tasks: 9
Completed Tasks: 0
Approved Tasks: 0
Remaining Tasks: 9

Task Status

1. Fix AppLayout.tsx layout component styling violations (🔄 In Progress)

Description: Remove non-layout classes from AppLayout.tsx component. The component contains styling classes like bg-gray-100, rounded-md, hover:bg-gray-100, focus:ring-2, text-purple-700, ml-2, text-lg, font-bold, text-gray-900, bg-opacity-30, bg-black that should be moved to separate styling components or variants. Read memory-bank/AGENTS.md and review memory-bank/project-rules/* before starting.