DEV Community: drmikecrowe

The PR That Fooled Me

drmikecrowe — Wed, 26 Aug 2026 09:00:00 +0000

This post was written with AI assistance (Claude) for structure and formatting. The PRs, the bug, and the opinions are entirely my own.

The best-looking PR in my repo had a data-loss race in it, and the model that wrote it reviewed its own work and found nothing.

I've written before about where AI belongs in software delivery, and the short version of that post is: people approve, agents check. This post is the origin story. It's three weeks of commit history in one public repo, and I'm going to walk you through it in order, because the order is the argument.

Act one: the PR that looked perfect

On July 28 I merged PR #169 in harnessed. Opus wrote most of it. Go look at the PR body, because the body is the point.

(A word about the repo, since I'm about to link into it four times and I haven't formally announced it. harnessed is a tool for people who want different configurations of their agentic harness in container isolation: you compose recipes of skills, MCP servers, and services into a named stack, and it launches in a pod with nothing leaking in from your host config. Toolchains come from mise, every dependency is pinned, updates respect a seven-day minimum release age so a poisoned package gets caught before it reaches a build, and images get Snyk and Socket scans at build time. It's my daily driver and I'm dogfooding it hard, but it's alpha: today it supports Claude Code and omp, with more harnesses planned. If that's a tool you've been wanting and you'd like to help shape it, come on in. Otherwise, the announcement post is coming when it's ready for you.)

It has a measured-cost table justifying the design (a one-line edit to an install script cost 307 seconds through the container build and 4.3 seconds natively, so the installs moved to runtime volumes). It lists two premises that were tested and disproved along the way, with links to where each was recorded instead of quietly dropped. It documents four bugs found during the work, each fixed with a test that fails without the fix.

That is a better PR writeup than most humans produce. I know because I'm one of the humans. When your AI hands you work at that level of polish, with benchmarks and disproved hypotheses and regression tests, skepticism starts to feel rude. I got comfortable. The polish was doing exactly what polish does.

Act two: a second opinion I didn't ask nicely for

Four days later I put up PR #185, a rework of the CLI verbs. Same workflow, same model, same confident writeup: 1,766 tests passing, every removed test accounted for in a table, docs updated.

This was also the first PR I ran through CodeRabbit, mostly out of curiosity. I had already asked Opus to review its own work. It found nothing worth stopping for.

CodeRabbit found this:

Two concurrent --recipe invocations with the same recipe set derive the same name. Both can observe preexisting=False before either mints, so both return a non-None minted_dir. [...] If one of the two concurrent launches fails _build_stack while the other is still using or building the same stack, the failing process deletes a manifest the other process depends on.

A classic check-then-act race, in the one code path where concurrent same-name launches aren't exotic (sharing one stack across repos is the design goal, so two repos starting together is the expected case). A failed build could delete a manifest another live launch was using. Data loss, sitting inside a PR whose test table I had just admired.

Here's the part that actually rattled me. The race wasn't even introduced by #185. It was pre-existing code that the refactor made visible. The model that wrote it, the model that refactored it, and the model that reviewed both had walked past it every time. A differently-trained model looked at it once and stopped.

That's when the lesson landed. The author always finds a reason the work is right. That's true of me, and it turns out to be true of a frontier model reviewing its own output. Self-review isn't review.

Act three: moving the fight upstream

I didn't fix the race in #185. I scoped it out deliberately (that PR was about CLI grammar, and mixing a concurrency fix into it is how you get neither reviewed well) and recorded it as issue #287.

By the time that issue got picked up in mid-August, my process had changed shape. Read the issue comments in order and you can see the new machinery:

A DECISION comment recording the fix approach, with current line numbers, so the implementer doesn't re-derive the research.
A full SPEC posted as a comment, before any code existed, with a line I now consider load-bearing: "Approval of this spec is the gate that authorizes implementation."
The spec itself went through adversarial review before I approved it, and the review changed it. Poking holes in a document costs minutes. Poking the same holes in an implementation costs a rewrite.
A human (me) applying the spec-approved label, timestamped.

That third point deserves a beat. When I started running adversarial reviews against code, the payoff was catching bugs. When I pointed the same treatment at specs, the payoff was bigger: every hole caught at spec time is a hole that never becomes code. The mint-race spec that got implemented was not the spec I first wrote.

Act four: what "done" looks like now

The fix landed as PR #414, and its body is an EVIDENCE document. Not a description of the change. Proof, organized so a reviewer can attack it:

A spec-to-test mapping: eight scenarios from the approved spec, each mapped to a named test.
A concurrency test that deterministically fails against the old code (two threads claim ownership) and passes with the fix (one does). Not "we added tests." A reproduction first.
Five hand-planted mutants, all killed, because if your test suite can't catch a deliberate mistake it wasn't going to catch an accidental one.
Two rounds of adversarial review, findings fixed and re-checked.
And a section titled "Not proven": the container build path the suite never executes, the NFS caveat on advisory locks, the mutation tool that wouldn't run in a worktree, the fact that the final SHA's adversarial pass was substituted rather than re-run.

That last section is the one I'd defend hardest. Evidence that can't tell you what it doesn't cover isn't evidence, it's marketing. The "Not proven" list is what makes the rest of the document worth believing.

What I actually learned

The confidence is a constant. The quality is not. Every PR in this story arrived with the same certainty. One of them carried a data-loss race. Nothing in the presentation distinguishes them, which means the presentation carries no information. You have to attack the work.

Self-review doesn't work, for silicon or for me. The reviewer needs to carry none of the author's assumptions. A fresh session helps. A different model with different training helps more (with one caveat: frontier models share training data and failure modes, so even two-model review has correlated blind spots; it's a design goal, not a guarantee).

Upstream beats downstream. Adversarial review of code catches mistakes. Adversarial review of the spec prevents them, at a fraction of the cost.

This costs tokens, and it's the cheapest spend in the budget. Every adversarial pass is a second model re-reading finished work. I meant it when I called them the cheapest tokens I spent all year: the alternative was shipping the race and paying for it during someone's demo.

If you want the team-scale version of this, with gates and human sign-off and the process we're building around it at Pinnacle Solutions Group, that's the People Approve, Agents Check post. This one is just the receipt trail: one repo, three weeks, and the PR that taught me to stop trusting a clean explanation.

Have your own "the AI sounded so sure" stories? Hit me up on GitHub or wherever you found this post.

People Approve, Agents Check

drmikecrowe — Mon, 24 Aug 2026 09:00:00 +0000

This post was written with AI assistance (Claude) for structure and formatting. The design, opinions, and specifics are entirely my own.

I think the industry is measuring the wrong thing, and I think the leaderboard is making software worse.

Every vendor pitch I've sat through in the last year answers the same question: how much of the work can the AI do? Percentage of code generated. Tickets closed without a human. Autonomy tiers, like a self-driving car. That's a fine question if you're selling a model, but it's the wrong question if you're responsible for an engineering organization, because it smuggles in two assumptions. The first is that the bottleneck is typing. The second, which nobody says out loud in the meeting but everybody hears, is that a successful AI rollout ends with fewer engineers.

Both are wrong, and the second one is expensively wrong.

Where this came from

This started with adversarial code review. I began having an independent reviewer, one carrying none of my assumptions, try to break my finished work instead of blessing it. Its entire brief was to find the input that makes the code fail. That alone was worth the tokens.

I'll admit why I needed it. I had drifted into trusting the high-end models. When Opus hands you code, it hands it to you with total confidence: no hedging, no "you may want to check this," just a clean explanation of why it did what it did. I got comfortable, and you can probably see where this is going. When I started pointing a second model at that output with instructions to break it, I was genuinely shocked at the gap between how certain the first model sounded and how well the code actually held up. The confidence is a constant. The quality is not. Nothing has made me more careful than watching a model I trust get taken apart by a second model with no stake in being right.

Then I moved the whole thing upstream. I was already writing specs first, so before any code existed I pointed the same adversarial treatment at the spec itself: find the requirement that can't be tested, find the two bullets that contradict each other, find the thing I assumed and never wrote down.

That turned out to be even more valuable. My specs were full of holes I could not see, because I was the one who put them there, and every hole caught at spec time was a hole that never became code. Reviewing the output catches mistakes; reviewing the intent prevents them. The tokens I spent arguing with a machine about what I actually meant were the cheapest tokens I spent all year.

That's the part I couldn't stop thinking about. If an adversarial loop does that for one engineer working alone, what does it do for a team? So we started building it into a real process at Pinnacle Solutions Group, the consulting firm I'm part of, and we're genuinely excited about where it's going. This is us thinking out loud about where AI belongs in software delivery, and I'd rather share it while it's still being shaped than wait until it's a polished case study.

Two disclaimers. Helping engineering orgs put this in place is work Pinnacle Solutions Group sells, so read the whole thing with that in mind. And it's running on a live client engagement right now, being developed with them and fitted to how they already work. I'm describing a process I believe in and am actively proving, not results I can hand you. When I get to what it costs, I'll be specific about what I don't yet know.

The personal arc, though, is public if you want to check my work. It played out in the open on a side project of mine: the review that converted me, where a differently-trained reviewer found a concurrency bug in code that self-review had blessed, and a complete evidence run, spec through adversarial review through proof, on the fix for that same finding.

Delivery breaks at the edges, not in the middle

Think back through the projects that went badly for stupid reasons, not the ones with genuinely hard technical problems. Anecdotally, across the ones I've watched up close, nearly all of them broke at one of two boundaries.

The first is the handoff from product to engineering. Somebody writes down what they want, and what they write down is incomplete: acceptance criteria you can't actually test, an assumption nobody stated, two requirements that quietly contradict each other. Nobody notices at write time. Somebody notices on day four of the sprint, in a meeting, after the money is already spent.

The second is the handoff from "the work is done" to "the work is proven." A change gets reviewed by looking at code. Looking at code answers a question (is this reasonable code?) that nobody actually needed answered. The question is whether this change does what product asked for, and how we know.

The middle, where the typing happens, is mostly fine. That's where every vendor is aiming.

So we aimed somewhere else, and not out of contrarianism. The speedup in the middle is real, but it's a commodity: every tool on the market delivers it, and your team collects it no matter whose logo is on the invoice. The failures at the edges cost sprints, not minutes, and nobody was selling anything there.

We put an AI agent at each of those two boundaries, and we gave it a job description with a hard limit on it: people approve, agents check. An agent in this design never approves anything, never merges anything, and never decides what gets built. It reads, it tests, and it reports, and the verdict belongs to a person every time.

Underneath, there's nothing exotic here. It's spec-driven development feeding test-driven development, wrapped in a workflow that makes both of them non-optional.

None of the individual pieces is new. Planting a deliberate fault to prove the tests would catch it is mutation testing, an idea from the 1970s. Coverage on changed lines is standard CI configuration. Requirements reviews, design reviews, spec-first, test-first: all decades old. What I'd defend as new is where they sit. When AI writes most of the code, plausible work arrives faster than anyone can judge it, and these old disciplines are how judgment keeps up.

Gate 1: AI that protects human judgment

The first gate sits between product and engineering. It runs on a story that's headed into refinement, and it answers one question: is this specified well enough to be refined?

An agent reads the story and puts it to this test: could an engineer prove, from what is written here, that the intent was met? That question does a lot of work. It forces every acceptance criterion to be verifiable rather than aspirational. It surfaces the assumption sitting under the third bullet that nobody wrote down. It catches the place where criterion two and criterion five can't both be true.

This is aimed squarely at helping the people who write those stories. Product and marketing are often not technical, and they are being asked to express intent precisely enough for engineers to manifest it. That's a hard thing to do with no feedback loop. Today the feedback arrives four days into the sprint, in a meeting, in front of people. This gate gives them the same feedback in writing, in minutes, from a machine that doesn't sigh.

A story that passes moves on to refinement. That's not an agent approving the work; it's an agent confirming that product has finished product's job, so engineering can start engineering's job. Nobody's clear, well-specified ticket sits in a queue waiting for a human to rubber-stamp what an agent already confirmed. What gets built is still decided by people in refinement, downstream of this gate, exactly as it was before.

A story that fails bounces back to whoever wrote it, with written findings. And not vague ones like "this needs work," but specific ones: criterion three has no observable outcome, here is what's missing; criteria one and four conflict under this condition; this assumes the user is already authenticated and the story never says so. Bad tickets bounce before they cost a sprint, and they bounce with a document instead of a meeting.

The part that matters most: the gate does not fill in the gaps. It finds them and hands them back.

The industry instinct is the opposite. Vague requirement? Let the model guess. It'll produce something plausible, and plausible is very close to correct, and the demo will go great. But deciding what the business actually wants is not a gap-filling exercise. It's the highest-judgment work in the building, done by the people with the context and the scars. Automating a plausible guess over the top of that doesn't remove the work, it just hides it until it's expensive.

So we deliberately preserve the human conversation at exactly the point where the human is the value. What we change is the price and the timing: earlier, cheaper, in writing, before anyone has spent a sprint on the wrong thing.

Between the gates: building a specification worth implementing

This is where spec-driven development actually lives, and it's the part I had wrong in my own head until recently. I'd lumped it all into "refinement," which is one word doing two jobs.

Refinement answers is this item ready: is it clear, is it testable, roughly how big, in what order. That's product's item, and Gate 1 is what protects it. Building the Implementation Plan answers a different question, how will we build this, and it belongs to the team. Those are two artifacts with two owners, and collapsing them is how you end up with a product owner refereeing a design argument.

Pre-assembling the context

Once a story clears Gate 1, an agent reads the codebase against it and assembles a context pack: which subsystems this touches, the patterns already in use there, the tests that cover that ground today, the prior art from the last time somebody did something similar.

This is small and it's cheap and I think it's one of the highest-leverage pieces in the whole process, because it gets produced while nobody is waiting on it. If your team still holds a refinement meeting, everyone walks in already knowing what the story touches instead of speculating out loud, which is most of what sizing actually needs, without paying for a full design on every item in the backlog. Later, it's the input the planning agent needs, already gathered.

When it runs is a knob, not a rule. A team that plans ahead can build the pack as soon as the story clears the gate. A team running lean can build it just in time, when an engineer picks up the story, so it reflects the codebase as it is that morning instead of three weeks ago. The orchestration bends to the team; the artifact is what matters.

Then the plan, when someone picks up the work

An engineer takes the story. An agent drafts the Implementation Plan from the approved Requirements and the context pack: approach, the tests that will prove each acceptance criterion, risks, and how we back it out. A second agent, with none of the first one's context, attacks that draft. The engineer resolves the findings, owns the result, and signs it. Then code gets written.

That agent draft is not a shortcut around the engineer. It's a better starting point than the engineer would produce alone under sprint pressure, because it actually read forty files instead of pattern-matching off four. What it cannot do is decide whether this approach is right for where the system is going, whether the rollback is real, or whether the risk list is honest. That's the signature.

And when the plan reveals the estimate was wrong, that's the process working. You find out on day one, in writing, with a reason. A plan can bounce a story back the same way Gate 1 does.

Why the agent may fill this gap and not the other one

Gate 1 refuses to fill gaps in the Requirements. Here an agent drafts a whole specification. That looks inconsistent. The difference is the cost of a wrong guess.

A wrong guess about product intent is unrecoverable. Nothing downstream can detect it. Every test passes, the evidence is clean, and you shipped the wrong thing correctly. A wrong guess about technical approach is checkable. The Requirements check it, an adversarial reviewer checks it, and so does what we call the gauntlet: an automated review that holds the implementation to a standard while it's built, with every changed line tested, the tests proven able to catch a deliberately planted fault, and every check the merge gate will run passing before the work is called done. So does the engineer who has to put their name on it. The blast radius is different, so the rule is different.

Gate 2: AI that forces human understanding

The gauntlet ends with output: check results, coverage of the changed lines, planted faults caught. That output says the code was built to standard. It does not say the right thing was built, and treating those as the same claim is exactly the conflation this gate exists to prevent.

That's what the Evidence document is for. Its shape is simple: each acceptance criterion from the Requirements, how the change satisfies it, and how that was verified. Tests mapped to specific criteria. The gauntlet's results. Screenshots where the change touches something a person looks at. And one section every Evidence document must carry: what is not proven. Proof that can't state its own limits isn't proof, it's marketing. The document isn't written at the end as paperwork; it accumulates as the work is built. Evidence is the proof that the acceptance criteria were met, assembled from what the gauntlet produced.

Gate 2 is the formal check on that proof. An agent validates the Evidence against the Requirements before the change is ever put up for review: does what the gauntlet proved actually prove what product asked for? Then the engineer must read the Evidence against the Requirements and personally decide whether the work meets the intent. The question isn't whether the tests are green; it's whether the thing does what was wanted. The AI assembles the proof, and the engineer renders the verdict and signs their name to it.

So a piece of work carries three named documents by the time it ships. Requirements says what and how we'll know. Implementation Plan says how, before the build. Evidence proves it. Each one carries a human approval, and each one was checked by an agent first.

But a signature is cheap. Nothing about signing a document stops an engineer from skimming it and signing anyway. If all this gate did was swap "skim a green build and merge" for "skim an Evidence doc and sign," I'd have added ceremony and called it rigor. A process that only asks for a signature gets signatures.

That's why the signature isn't the mechanism. The next section is.

The forcing function: every change becomes knowledge transfer

The engineer who signed the verdict then uses that same Evidence to walk reviewers and QA through the change, out loud, in front of colleagues. It's not a diff review, it's a narrative: here's what product asked for, here's how this satisfies it, here's how we know.

That's the part you cannot fake. You can sign a document you didn't read, but you cannot stand in front of two colleagues and explain a change you don't understand, and everyone in the room knows it within about ninety seconds. The Evidence document isn't the enforcement. It's the script for a conversation that is the enforcement.

"Out loud" flexes with the team. For a distributed group that means a live call or a recorded walkthrough rather than a conference room, and the rule holds either way: the engineer explains the change to people who can ask questions.

This is the thing every engineering leader is quietly afraid of: an engineer generating a change they don't understand, skimming a green build, and shipping it. I can't make that structurally impossible; anyone who tells you they can is selling. What I can do is put the moment of "explain this to your peers" between the AI's output and production, and make it the normal cost of shipping. Not understanding your own change stops being invisible.

And then think about what that does over a quarter. Every single change becomes a teaching moment, delivered by someone who is required to understand it well enough to defend it. The team learns how the system is changing, update by update, from a person rather than from a diff nobody reads. Institutional knowledge stops being a thing that lives in three people's heads and starts being a thing the process produces as a byproduct.

This is the same set of models everyone else is using, aimed at a different problem. The difference isn't the tooling; it's where the human sits, and what the process makes them do when they get there.

Your engineers matter more now, not less

Nothing in this design is aimed at needing fewer engineers. It's aimed at raising the quality of what your engineers ship when they work with AI. Those are different goals, and they lead to opposite decisions at nearly every fork.

A model will produce something plausible for almost any request. Plausible is the dangerous part: it compiles, reads well in review, passes the tests it was written alongside, and quietly does the wrong thing. The only reliable defense is a person who knows the system, knows what was actually asked for, and is equipped to tell the difference. No model removes that need, because judging whether the AI did what was intended is itself a judgment about intent.

So the volume of code goes up and the value of engineering judgment goes up with it. Every gate exists to put a competent engineer in front of the right question at the right moment, with the evidence in hand to answer it. Gate 1 hands them a request worth building. Gate 2 hands them the proof and makes them rule on it. The walkthrough keeps the whole team sharp enough to keep doing that as the system grows.

An org that cuts engineers to pay for AI ends up with more code and less capacity to judge it. That is the worst version of this. If your team gets faster, the win is that they take on work they couldn't before, not that there are fewer of them.

What this costs

This is the part vendors skip, so here it is up front: this process is slower than what the market currently expects from AI. Gate 1 adds a loop before refinement. The Implementation Plan adds a draft and an adversarial pass before code. Tests written from acceptance criteria are slower to write than tests written from finished code. Evidence takes real effort to produce, and the walkthrough takes real time from more than one person.

The walkthrough is the line I expect the most pushback on, and the pushback is fair: multiple colleagues per change, out loud, does not scale like a CI job. I'm not going to argue it away, because it's the deliberate purchase. Quality of what ships and a team that understands what shipped are the two things this whole design exists to buy, and the walkthrough is where both get paid for. How deep it goes, and for which changes, is one of the things the current engagement is calibrating.

It costs more tokens, too. Every adversarial pass is a second model reading work the first model already did. The context pack is a full codebase read before anyone has picked up the story. Attacking a spec, then attacking an implementation plan, then validating Evidence against the Requirements: none of that is free, and it is spend you would not have if you simply asked a model for the code and merged what came back.

I think it's the best money in the budget. I called those the cheapest tokens I spent all year and I meant it literally: you are buying rework you never have to do, at a price you can see up front. But it's a real line item, it's larger than a single-agent workflow, and a client should agree to it going in rather than discover it on the first invoice. Spend ceilings and usage alerts go in on day one for exactly that reason. Adoption shouldn't come with an open tab.

The middle is genuinely faster. An engineer with AI leverage and a clear spec outruns the same engineer without it, and it isn't close. But that gain is smaller than the number on the slide, because we are deliberately spending part of it on tests and proof.

That said, the paragraphs above make this sound slower than it is. Most of what this process standardizes should already be standard operating procedure in most organizations: tests tied to the requirements, review that checks intent, proof that the work does what was asked. I'm not piling new work on top of delivery. What I'm mostly doing is refusing to let AI's speed become the excuse to skip work that was always owed. The velocity this trims is the false kind: the perceived gains of merging plausible code fast and paying it back later as rework.

So what I'm claiming is confidence. Fewer sprints spent building the wrong thing, fewer changes that pass review and fail intent, and a team that understands what it shipped. If the number you're optimizing is lines per day, I'm the wrong call. If it's the ratio of shipped work that actually did what was asked, this is the trade I'd make.

What holds it up

The gates only work if what they're checking is real. Two supporting pieces do that work.

The gauntlet. Tests come from the acceptance criteria, not from the finished code, so they verify what was asked for instead of confirming whatever got built. The gauntlet's job is to prove that testing is real: every changed line covered, and planted faults confirming the suite would catch a genuine mistake. A suite that can't catch a deliberate mistake wasn't going to catch an accidental one.
Adversarial review. An independent AI reviewer, carrying none of the author's assumptions, is briefed to break the work: find the input that makes it fail. Confirmed findings get fixed and re-checked. This is where the whole thing started for me, and it's still the piece I trust most. Self-review doesn't work. The author always finds a reason the work is right. One caveat: two frontier models share training data and failure modes, so this independence is a design goal with limits. The reviewer has none of the author's context, and that catches a lot, but it can still share the author's blind spots.

Where this doesn't fit

This process assumes you know what you're building. Exploratory work, spikes, R&D where the spec is the thing you're trying to discover: gating that behind Requirements is backwards, and we don't try. The honest artifact there is a timebox and a writeup, and the process starts after.

It also assumes changes worth the ceremony. A dependency bump or a typo fix doesn't need an Implementation Plan and a walkthrough, and a process that pretends it does will get routed around within a month. Part of what we're calibrating with the current client is exactly that tiering: which changes get the full treatment, which get the gauntlet alone, and who decides.

And a three-person team probably shouldn't run this as written. The walkthrough is priced for teams big enough that knowledge silos are a real risk. Below that size everyone already knows every change, and the gates worth keeping are the two adversarial reviews.

How it lands in your org

We customize the gates to fit, and the fit runs in two directions. The gates can hook onto the states your team already has: no new methodology, nothing replaced, no migration. Or they can anchor a leaner workflow, because once agents build the context, draft the plan, and prove the work, some of your ceremonies stop earning their place on the calendar, and a team that notices is free to act on it. Which direction to take is your call. The gates hold either way.

And it's built to end. We lead the first run, coach the second, and step back to review only for the third. If your team still needs us on run four, we designed it wrong.

What to ask the next vendor, including us

If you're evaluating AI in your delivery process, the autonomy percentage on the slide tells you almost nothing about whether your delivery will improve. Three better questions, and I'd expect to answer all of them myself:

Where does the human judgment live, and what protects it? If the answer is that the AI fills in whatever product left vague, you're buying a machine that converts unclear thinking into shipped code at high speed.

How do you prove the work meets the intent? If the answer is "code review," ask what question code review actually answers. Then ask who on the team will be able to explain the change six months from now.

What does this slow down? Any honest process has a cost. A vendor who reports only acceleration is either not measuring or not telling you.

And ask what the vendor thinks happens to your engineers. If the pitch is a headcount line, walk. We didn't get here by asking how much of the job the AI could take. We asked where delivery breaks, and the answer was the same two places it's always broken. The machines are genuinely useful there. They just aren't in charge there, and they need better engineers around them than they did before, not fewer.

This is what I think about most days right now. We're building it with a client as I write this, shaping it to their process rather than handing them ours, and the parts that survive contact with a real team are the parts worth keeping. If you're wrestling with the same question of where AI belongs in your delivery process, and how you'd prove it worked, I'd genuinely like to hear how you're approaching it, whether or not you ever talk to us. Find me on GitHub or wherever you found this post.

Context Binoculars: Understanding LLM Context Windows

drmikecrowe — Sat, 13 Sep 2025 04:00:00 +0000

Remember as a kid (or in my case as an adult) when you pulled out binoculars and gazed around the room? It’s a bit disorienting for sure, but also fun. And I guess if you’ve ever wondered about what I do for fun, now you know.

It struck me the other day that this was the ideal representation for how LLM context works. While my demonstration you’re going to see below is extreme, this is the reality of what we’re facing when we’re trying to get LLMs to do big projects. The context window is everything, and you’re hearing more and more about context engineering in online articles. Ensuring that the LLM has exactly the right context greatly improves the accuracy of the results that you get out the other end.

There's not much more to this article. All I'm really trying to do is viscerally explain what the context window is and give you a feeling of why managing your context is so important.

Check out the visual demonstration here: context-binoculars

How I Taught My AI Pair Programmer to Be Our Team's Tailwind CSS Cop - Part 3 of 6

drmikecrowe — Sun, 07 Sep 2025 00:00:00 +0000

Part 3 of 6: The AI-Assisted Development Workflow Series

This is the third installment in a six-part series exploring how AI is transforming modern development workflows. In this series, I’ll walk through my journey of building an AI-assisted development environment, from basic infrastructure setup to advanced architectural enforcement and task orchestration.

We’ve all been there. You start a new project with Tailwind CSS, and everything is beautiful. The utility-first approach is fast, flexible, and keeps you right in your HTML. But as the project grows and the team expands, the CSS landscape can start to feel like the Wild West. Utility classes get sprinkled everywhere, components start to blur the lines between structure and style, and soon you’re overriding margins and fighting for specificity.

I love Tailwind, but I knew we needed to introduce some architectural discipline before things got out of hand. The problem is, architectural rules are only as good as their enforcement. Documentation gets stale, and nagging in code reviews doesn’t scale.

So, I had a thought. What if I could teach our team’s AI pair programmer to be our automated style cop? What if it could not only understand our rules but actively enforce them?

The Convention: A Three-Tier System for Sanity

First, we needed a clear, simple convention. We decided on a three-tier component architecture designed to enforce a strict separation of concerns:

Layout Components (src/components/layout/)
UI Components (src/components/ui/)
Feature & Page Components (src/pages/, etc.)

This system ensures that our UI components are truly reusable and that our layout logic is centralized and predictable.

The Solution: Hiring an AI for Architectural Enforcement

With the convention defined, I turned to my AI assistant. I described the three-tier system and asked, “Can you build a custom ESLint rule to enforce this?”

Working with the AI was like having a hyper-competent junior developer who knew the ESLint AST (Abstract Syntax Tree) better than I ever will. I provided the high-level architectural goals, and it handled the implementation details.

How the Custom Plugin Works

The resulting ESLint rule is a simple but powerful detective. For every component file, it performs three steps:

Location Check : First, it looks at the component’s file path. Does it live in src/components/layout/, src/components/ui/, or somewhere else? This determines which rulebook to apply.
Class Inspection : Next, it parses the JSX and extracts all the Tailwind classes from the className prop. It’s smart enough to handle string literals and basic template literals.
Rule Application : Finally, it applies the logic:
- If it’s a UI Component , it scans the class list for any margin patterns (m-*, mx-*, mt-*, etc.) and flags them as errors.
- If it’s a Layout Component , it checks for any non-layout classes (like bg-red-500, font-bold, etc.) and flags those.
- If it’s a Feature/Page Component , it does nothing. They are free to compose as they see fit.

The AI generated the rule file, a utility file to hold the class definitions, and even modified our main eslint.config.js to wire it all up.

The Results: The Moment of Truth

I ran the linter across our codebase, and it worked perfectly. It immediately flagged several components that were violating our new convention. Here’s a summary of what it caught:

AppLayout.tsx & Sidebar.tsx : These core layout components were caught using styling classes like bg-gray-100, text-purple-700, and font-bold. The linter correctly identified these as style concerns that should not be in a layout component.
EmptyState.tsx & ErrorState.tsx : These UI components had mb-4 and mt-2 margins, violating our “marginless UI” rule. The parent component should be responsible for this spacing.
FileDropzone.tsx & MainContent.tsx : These UI components were using mx-auto, another margin-related property that is the responsibility of a layout container.
StatCard.tsx : This UI component was using mt-1 and ml-1, which are small but important violations of the architectural pattern.

Each of these warnings was a clear, actionable task for us to clean up the codebase and align it with our new, more maintainable architecture.

Conclusion: More Than Just Code Generation

This experience was a powerful demonstration of how to use LLMs for more than just generating boilerplate or fixing simple bugs. By teaching the AI our team’s specific architectural patterns, we were able to turn it into a partner for enforcing code quality and consistency at scale. It automated the tedious work of code validation, freeing us up to focus on building features.

tures.

The Ultimate Design Review: Orchestrating AI with Task-Based Workflows - Part 6 of 6

drmikecrowe — Sun, 07 Sep 2025 00:00:00 +0000

Part 6 of 6: The AI-Assisted Development Workflow Series

This is the final installment in our six-part series on building an AI-assisted development workflow. We’ve set up our infrastructure, taught the AI our coding standards, and established a robust “memory bank” of project rules. Now, it’s time to put it all to the test with the ultimate challenge: a comprehensive, end-to-end design review of the entire codebase.

TL;DR

Want to see what we're building? I've created two example posts that show the actual output from this automated design review system:

Design Review Feedback Example - See the comprehensive analysis that the AI generates, including architectural strengths, critical issues, and detailed findings across all layers of the codebase.
Design Review Action Items Example - See how the AI transforms the analysis into a structured, actionable backlog with 18 major tasks and 72 subtasks, each with clear implementation steps and acceptance criteria.

These examples show the real output from running this system on an actual codebase. The AI identified critical architectural violations, generated specific improvement tasks, and created a comprehensive project plan - all automatically.

Series Overview:

Part 1: MCP Servers, Ports, and Sharing - Setting up the foundation
Part 2: ESLint Configuration Refactoring - Cleaning up tooling with AI
Part 3: Custom Architectural Rules - Teaching AI to enforce design patterns
Part 4: Task Orchestration - Managing complex refactoring workflows
Part 5: Project Rules for AI - Creating effective memory banks and guidelines
Part 6: The Ultimate Design Review - Putting it all together

The Challenge: From “Codebase” to “Action Plan”

How do you review an entire src/ directory? You can’t just feed it to an AI and say, “find problems.” The context is too large, the potential feedback is too broad, and the output would be a wall of text, not an actionable plan. To understand the issue, grab a pair of binoculars and use them in your office: Have a coworker toss an index card somewhere in your office and then search for that with the binoculars. If you have a really large code base, have your coworker put it in the office somewhere. That’s what your fighting:

The binoculars symbolize the AI context. It’s a fixed width window of information the AI can maintain
The index card is what you are asking the AI to find

My goal was to create a system that could:

Systematically analyze the entire codebase in manageable chunks.
Evaluate each chunk against multiple, specific architectural rules.
Produce a structured, prioritized, and actionable list of tasks.
Be entirely automated and repeatable.

To achieve this, I created a multi-layered prompting strategy that uses the TaskFlow MCP server to orchestrate the entire workflow. The process is driven by two key prompt files.

The Master Plan: `new-design-review.md`

The first file, @memory-bank/prompts/design-review/new-design-review.md, is the high-level orchestrator. It doesn’t perform the analysis itself; it instructs the AI on how to set up the entire project plan within TaskFlow tasks.

Here’s the strategy it lays out:

Create Two Buckets: The first thing it does is create two separate requests in the task server: one for the analysis process and one for the action items. This is a critical separation of concerns. The analysis tasks will be marked as “done” once the review is complete, but the action items will form a living backlog of work to be done.
Divide and Conquer: It then breaks the entire src/ directory into logical chunks: Core App Files, Components, Services, Machines, Hooks, etc. For each chunk, it creates a parent task in the “analysis” request.
The Mini-Review Subtasks: This is where the granularity comes in. For each parent task (e.g., DR Part 3 - Components (Dashboard)), the prompt instructs the AI to create four specific subtasks:

The Instruction Manual: `request-2-instructions.md`

If the first prompt is the “what,” then @memory-bank/prompts/design-review/request-2-instructions.md is the “how.” This file is a set of instructions for the AI on how to behave while it is executing the analysis subtasks.

Its most critical directive is this: You MUST create new tasks in the “Action Items” request for every issue you find.

It strictly forbids the AI from simply summarizing its findings. Instead, it must translate every identified improvement into a well-formed task, complete with a title, description, and severity, and add it to the other request bucket. This is what turns a qualitative review into a quantitative project plan.

The Results: An Actionable Backlog

After running the AI through this workflow, the taskflow-mcp server populates a YAML file with the results. The file from my design review, @DESIGN-REVIEWS/2025-08-19/tasks.yaml, is over 800 lines long, but here are a few snippets that show the power of this system.

First, you can see the completed analysis task for the “Core App Files,” with its four subtasks all marked as done:

- id: task-62
  title: DR Part 1 - Core App Files
  description: |-
    Analyze the following files for design patterns, code quality, and architectural consistency:nn- src/App.tsxn- src/appMachineContext.tsn- src/config.tsn- src/context/DashboardDataContext.tsx
    ...
  done: true
  subtasks:
    - id: subtask-71
      title: Overall Design Analysis
      done: true
    - id: subtask-72
      title: Effect Pattern Analysis
      done: true
    - id: subtask-73
      title: Component Styling Analysis
      done: true
    - id: subtask-74
      title: State Machine Analysis
      done: true

More importantly, the real output is the list of actionable tasks generated in the other request. The AI identified dozens of specific, granular improvements, each perfectly formatted according to the instruction prompt.

Here are a few examples of the generated tasks:

A high-priority refactoring task based on our Effect pattern rules:

- id: task-98
  title: Add Effect-based validation to ExcelService
  description: ExcelService.ts contains complex validation logic in generateExpenseReport function. Add Effect-based validation using E.succeed().pipe(E.filterOrFail()) pattern for report and expense data validation.

A task to fix a clear violation of our Tailwind CSS architecture:

- id: task-139
  title: Fix EmptyState.tsx UI component margin violations
  description: Remove margin classes from EmptyState.tsx component. The component contains margin classes mb-4, mb-2, mb-6 that should be handled by parent layout components. Read memory-bank/AGENTS.md and review memory-bank/project-rules/* before starting.

A task for a significant architectural improvement to break up a monolithic file:

- id: task-99
  title: Extract ExcelService helper functions to separate modules
  description: ExcelService.ts is a 485-line monolith with many helper functions. Extract helper functions into separate modules (excel-formatting.ts, excel-data.ts, excel-styles.ts) to improve maintainability and testability.

Conclusion: The Sum of the Parts

This automated design review is the culmination of our entire AI-assisted workflow. It seamlessly integrates:

The Agent Protocol (AGENTS.md): Provides the core principles for the AI’s analysis.
The Project Rules (project-rules/): Act as the specific, enforceable standards for the review.
The Prompting Strategy (prompts/): Orchestrates the complex workflow, breaking it down into manageable steps.
The Task Server (taskflow-mcp): Captures the output, transforming a review into an actionable project backlog.

By investing the time to teach the AI how we build software, we’ve unlocked the ability to automate one of the most time-consuming and critical parts of the development lifecycle. We’ve moved beyond simple code generation to a world where our AI partner can actively help us manage quality, reduce technical debt, and enforce architectural consistency at scale. It’s a powerful glimpse into the future of software development.

From Linter Chaos to Orchestrated Tasks - Part 4 of 6

drmikecrowe — Sun, 07 Sep 2025 00:00:00 +0000

Part 4 of 6: The AI-Assisted Development Workflow Series

This is the fourth installment in a six-part series exploring how AI is transforming modern development workflows. In this series, I’ll walk through my journey of building an AI-assisted development environment, from basic infrastructure setup to advanced architectural enforcement and task orchestration.

In my last post, I talked about how my AI pair programmer and I created a custom ESLint rule to enforce our new frontend architecture. It was a huge success. The linter, now armed with our specific rules, scanned the codebase and… gave us a giant list of things to fix.

This is a classic “good news, bad news” scenario. The good news? We had a precise, automated way to detect architectural drift. The bad news? We had a long, intimidating wall of terminal output listing every single violation.

/home/mcrowe/.../AlertDialog.tsx
  19:8 warning Unregistered class detected: data-[state=open]:animate-in
  19:37 warning Unregistered class detected: data-[state=closed]:animate-out
...
/home/mcrowe/.../EmptyState.tsx
  25:21 error UI Component contains a forbidden margin class: 'mb-4'
...
✖ 60 problems (2 errors, 58 warnings)

A raw list of errors is a starting point, but it’s not a plan. How do you manage this work? How do you distribute it? How do you ensure each fix is made with the right context? Just handing a developer (or an AI) a raw log is inefficient.

The Tool: TaskFlow MCP for Orchestration

This is where another tool in our AI-assisted workflow comes in: TaskFlow MCP. It’s an open-source task orchestration server designed specifically for managing development work between humans and AI agents.

Instead of just working from a simple prompt, TaskFlow allows us to break down a high-level request (like “Fix all these linter warnings”) into a structured plan of discrete, trackable tasks.

Crucially, as the user, I can insist that each task be configured with specific context and instructions. This is the killer feature. It turns a simple “fix this” command into a rich, context-aware work order.

The Process: Turning Chaos into a Plan

I fed the messy ESLint output to my AI assistant and gave it a new prompt:

"For each of the UI warnings below, use plan_task to create a ‘Fix UI layout vs. styling overlap’ request. For each violation, create a sub-task. Ensure that each task specifically includes these instructions:

Read memory-bank/AGENTS.md

Review memory-bank/project-rules/*"

The AI, using the TaskFlow MCP tool, did exactly that. It parsed the linter output, grouped the violations by file, and created a formal plan. Each task was not just a file path and a line number; it was a complete work package containing:

A Clear Title : e.g., “Fix AppLayout.tsx layout component styling violations”
A Detailed Description : Explaining exactly which classes were violating our architectural rules.
Embedded Context : The prerequisite instructions to review our project’s agent protocols and architectural rules before starting work.

The Result: An Actionable, Context-Aware Plan

Tasks are maintained locally

The output wasn’t a log file; it was a professional task report, ready for any developer—human or AI—to pick up and execute. It transformed a chaotic list of problems into a manageable, parallelizable set of tasks.

Here is the exact markdown file that the tool generated. It’s clear, organized, and ensures that whoever does the work has the full context required to do it right.

Task Status Report: Fix UI layout vs. styling overlap issues identified in ESLint warnings

Generated on: 2025-08-17

Overall Progress: 0%

Total Tasks: 9
Completed Tasks: 0
Approved Tasks: 0
Remaining Tasks: 9

Task Status

1. Fix AppLayout.tsx layout component styling violations (🔄 In Progress)

Description: Remove non-layout classes from AppLayout.tsx component. The component contains styling classes like bg-gray-100, rounded-md, hover:bg-gray-100, focus:ring-2, text-purple-700, ml-2, text-lg, font-bold, text-gray-900, bg-opacity-30, bg-black that should be moved to separate styling components or variants. Read memory-bank/AGENTS.md and review memory-bank/project-rules/* before starting.