DEV Community: Michael Truong

The pipeline was green. The product was underspecified

Michael Truong — Mon, 17 Aug 2026 08:23:00 +0000

The checks all passed. I still would not have sent the resume.

I had been using a Cursor agent to implement a private facts → prose resume generator. Structured career claims in, recruiter-facing PDFs out. Generation, rendering, and ATS checks all stayed green. The PDFs looked plausible.

I treated that as enough.

Several product requirements were still wrong.

What implementation QA was checking

The useful split is structured facts on one side and disposable rendered artifacts on the other:

Layer	Holds	Does not hold
Structured facts	Stable claims (actions, outcomes, metrics, scope)	Resume bullet phrasing
Application config	Which facts to include, tone, theme, page length	New career claims
Generated output	Markdown and PDF resumes	Source of truth

Implementation QA in this workflow means the pipeline runs end to end and the automated checks pass. Generation succeeds. PDFs render. ATS scripts assert page counts, required sections, and a few structural rules about separators and headings.

Those checks are real. They caught broken builds and regressions I did not want to ship.

They did not answer a different question: did the specification describe the resume I actually wanted?

That gap showed up in three places. None of them failed the scripts at first.

When page count is not product fit

The workflow produced one-page and two-page variants for the same application. Both PDFs passed check:ats. Both stayed inside their page limits.

The two-page resume still read like a stretched one-pager.

Experience on the shorter version used concise evidence entries selected for a tight one-page fit. The longer version reused that same condensed slice, then filled the remaining space with additional facts. Page count was correct, but the shape was wrong: the original concise bullets never expanded into fuller evidence. A two-page resume should deepen the roles that already earned a place, not keep the one-page wording and pad with more items.

Mismatched typography could fake the same green result by enlarging text on the longer PDF. Unifying the shared typographic scale and margins removed that shortcut: a two-page count had to come from the evidence itself.

Implementation QA had no opinion about which facts belonged on which page length, or whether two pages meant more evidence or just larger fonts. It only knew the PDF had two pages.

When valid data reads wrong to a human

Not every miss was about page length.

One Program Lead role was technically valid in the data: correct dates, correct employer, correct title. In the Experience section it rendered with a Full-time employment label. On paper that is accurate enough for a schema. On a resume it reads like a sequential primary job when the role was actually concurrent with other work.

The requirement was not "store valid employment metadata." It was "make concurrent work legible to a recruiter scanning the ladder." Renaming the label to Concurrent program was a product fix, not a pipeline fix. No ATS script flagged the old wording.

When structural correctness stood in for finish

The last category looked optical. It was still underspecification.

I only noticed after opening the PDF: the contact block and Skills sidebar shared a column edge on paper, but the Skills heading sat a few points lower than Experience, so the two-column header row looked crooked even though every section and separator rule still passed. ATS-safe contact separators are a constraint, not a design.

The workflow had encoded structural correctness: page counts, required sections, separator rules, heading shape. It had never named a human visual-acceptance criterion. The scripts were never asked to stand in for a finish requirement the spec had never named: would I send this?

What manual implementation used to smuggle in

When I wrote this kind of tooling by hand, implementation and requirements review were harder to separate. Every intermediate decision was visible. Choosing a font size, picking which bullet to cut, or rewriting a concurrent-role label forced a product judgment in the same session as the code change.

Manual implementation was accidentally doing requirements QA. Every ambiguous decision eventually became my problem because I had to turn it into code myself. An agent can absorb that ambiguity instead.

The two-page PDF was that pattern in miniature. It looked finished. The missing product decision (whether two pages meant deeper evidence) never came back through me.

That speed is the danger now. Incomplete specification can arrive dressed as a finished product.

An earlier piece was about why building this system suddenly made economic sense. This is the other side of that shift: once implementation got cheaper, I needed to make requirements review more explicit.

What a requirements QA stage could look like

The portable version is a few jobs, not a particular toolchain:

Force the vague request into concrete behavior before implementation: what changes, what stays invariant, and what counts as done
Review against that written intent, not only the local diff
Accept the finished artifact as a product ("would I send this?"), not only as a green pipeline

The first two are where requirements QA is cheapest. If those questions stay implicit, an agent can execute the wrong thing extremely efficiently.

In my workflow that looks like:

plan                        (overall intent)
→ slice                     (concrete behavior)
→ review slices             (gaps and boundaries before any code)
→ for each slice:
    → implementation
    → review after code     (still matches plan?)
→ artifact acceptance       (ready to send?)

I commit the plan alongside the work so the intent lives in the repo, not only in the chat. A reviewer is not limited to the current diff. It can see where this slice is supposed to lead, and flag something that is locally correct but inconsistent with a later slice. Reviewing the slices can still challenge assumptions, missing requirements, and slice boundaries before any code exists.

The ladder above is how I recreate the interrogation that manual implementation used to provide implicitly. Planning and review still miss gaps that never made it into the written requirements. Artifact acceptance is the last gate for those: it catches product misses the earlier rungs never named.

Takeaway: Agents are good at satisfying the requirements they are given, quickly enough that a miss looks finished. Implementation QA proves the system did what you asked. Somebody still has to QA whether those requirements describe the product you actually wanted.

If you'd like to see the same gap between green checks and product acceptance on a product I keep revising in public, try Codenames AI.

AI changed the build-vs-buy threshold

Michael Truong — Thu, 13 Aug 2026 06:39:39 +0000

Building custom software to solve a two-afternoon problem sounded absurd.

A Riot Games recruiter reached out while I was still preparing to return to the job market. Suddenly I needed a current resume to send back, and I had roughly two afternoons to produce one.

Normally that is an obvious buy decision. Under a short deadline you are not optimizing for reuse. You are optimizing for a PDF in someone's inbox. A resume builder gives you templates, export, and enough polish to look professional without inventing infrastructure.

Historically, I built when the reuse justified the setup cost. I bought or assembled manually when I only needed the artifact.

The same rule still applied. What had changed was the cost.

AI had lowered not only how much it cost to build the first version, but how much it cost to keep revising the architecture underneath it.

What I built instead

I built a private facts → prose resume repository with Cursor.

The idea is to separate career evidence from application wording:

Layer	Holds	Does not hold
Structured facts	Stable claims (actions, outcomes, metrics, scope)	Resume bullet phrasing
Application config	Which facts to include, tone, theme	New career claims
Generated output	Markdown and PDF resumes	Source of truth

Career claims live once in structured YAML. Each application selects, reorders, and rephrases them. npm run generate renders recruiter-facing prose. npm run pdf prints it. npm run check:ats runs structural ATS checks on the output.

You do not need my private repo to apply the pattern. The useful split is structured facts on one side and disposable rendered artifacts on the other.

Before generating a resume, the workflow researched the company and role, then used that context to decide which evidence from my career inventory belonged in the application. The system knew about far more career evidence than any one resume should contain. The inventory stayed put. What changed was which slice mattered.

For a gaming company like Riot Games, the research made a university game-design award relevant enough to surface on a software engineering resume where it normally would not belong. The same run also produced a one-page and two-page resume plus a recruiter reply.

A week later, I used the same career inventory for an AI product engineering company. The underlying evidence had not changed, but what mattered had. The workflow selected a different slice of it. The reuse I had been optimizing for was already real.

Later it expanded past resumes into interview prep materials, a use I had not planned when the recruiter first wrote. Different company, different evidence selection. Different stage, different artifact altogether. The system that looked overbuilt for one reply kept finding uses.

The system did not start here

The repository did not begin with that model.

It evolved through increasingly useful abstractions across the same short build window:

A single resume document
A reusable resume template
Career facts separated from prose
Capabilities grouping related achievements
Evidence entries inside each capability
Application-specific selection over that inventory

The first workable version was not the last one. Because implementation and revision were cheap, I could keep moving instead of freezing at "good enough for tonight."

Without that cost shift, the rational stopping point would probably have been a reusable template or a lightly parameterized document. Fine for one application. Weak as career infrastructure.

That was the advantage of the build path here: room to discover the right abstraction after the first one works.

Two costs moved, not one

AI did not magically make build correct for every problem.

It lowered two costs at once:

Implementation cost: scaffolding the generator, themes, and checks stopped being a multi-week side project.
Architectural iteration cost: revising the underlying model (facts vs capabilities vs application selection) stayed cheap enough to do in the same session.

Buying still wins when the problem is narrow, the tool fits, and you will not reuse the result. Building still loses when maintenance will crush you.

What changed is the boundary. Problems that used to land firmly on the buy side can cross over when reuse matters and you can afford to iterate past the first design.

Buy was still faster for one send

This is not an argument that custom software always beats SaaS. I chose to build because the deadline still left room for something reusable. If the goal had been a single resume, buying an off-the-shelf builder would have been the faster path.

The build path made sense here because:

The career inventory is the stable core.
Research and role context drive application-specific selection.
The same system produced multiple resume variants and a recruiter reply.
It later expanded into interview prep materials.

Cheap iteration on both implementation and revision is what made exploring these abstractions affordable.

The same economics show up outside resumes: internal tooling, dashboards, documentation pipelines, code generators, personal workflow infrastructure. Anywhere the old math was "custom software is too expensive to build and revise", the revise term got smaller.

I suspect that generalizes beyond software engineering into knowledge work where bespoke systems used to lose to packaged tools on turnaround alone.

Takeaway: AI moves the build-vs-buy threshold by cutting both implementation cost and the cost of changing your mind about architecture. When reuse matters and you can iterate in the same sprint, building a small internal system can become rational where buying would previously have won on cost and turnaround.

If you'd like to see the same economics on a product I keep revising in public, try Codenames AI.

One skill per action looked like the safe boundary

Michael Truong — Fri, 07 Aug 2026 05:24:54 +0000

I started with a rule that felt like good engineering: one skill per action. Create a card here. Enrich it there. Reclassify it somewhere else. Each prompt got a clean boundary. Each file stayed small.

Then that decomposition started to fight the domain.

I ran into this while building an AI-assisted editorial workflow in Cursor, but the problem was not really about Cursor or publishing. It was about where an agent capability should begin and end.

In that setup, a skill is a markdown file the agent loads for a workflow. My editorial workflow uses those skills to create and manage Notion cards before drafting posts.

When every action gets its own skill

The first version of my inbox skill only created Inbox cards. That matched early usage: capture an observation, normalize it into a canonical shape, attach a small set of grounded references, and stop. The skill was create-only and treated each new card as immutable once it left Inbox.

As capture matured, the same card kept needing more work while it was still Inbox:

Create when a new observation arrived
Enrich when new evidence or framing changed the normalized shape
Reclassify when routing rules decided the card should be sparse vs rich, or a quick note vs a planned blog post

Those felt like three different jobs. They had different verbs. They had different retrieval triggers. Splitting them into separate skills seemed obvious.

The split failed in ordinary corrections. I would enrich an Inbox card with new evidence, then realize it should change from a quick note into a planned blog post. That meant a second skill invocation for reclassify, with a second copy of the same lifecycle rules. For a moment it was unclear which skill was still responsible for keeping the page as one current write-up instead of an accumulating edit history. Get the order wrong and you did the work twice: an enrich that left the card type stale, or a reclassify that ignored the evidence rewrite you still needed.

The deeper problem was ownership. All three operations touched the same owned object: a single Inbox card in a Notion database. They shared the same lifecycle gate (the card must stay in Inbox), the same mutation boundaries (never change lifecycle status, source path, or published URLs on existing pages), and the same routing rules for how sparse or rich the card should be and whether it was a quick note or a planned post. They also shared the same rule: after every update, the page must hold exactly one current normalized write-up and exactly one captured observation. Notion history is the revision log; the operational card is not an append-only audit log.

Treating Create, Enrich, and Reclassify as three skills meant three prompts trying to enforce one coherent capability. The boundaries were at the wrong layer.

Consolidation made the skill clearer

The fix was to stop pretending those were separate capabilities. One inbox skill now owns the full Inbox lifecycle: Create, Enrich, and Reclassify while the card remains in Inbox.

The operator still invokes one skill. The skill routes internally:

User-facing command	What it does
create inbox card	Creates an Inbox card with a canonical write-up and grounded references
enrich inbox card	Folds new evidence into the existing Inbox page as one current write-up
Same-thread continuation	Treats further capture in the same chat as enrich on the page just created

Reclassify is not a separate user command. It is detected inside enrich when the routing rules decide the card should be richer or thinner than before, or should shift from a quick note to a planned post. Sparse captures can become richer field reports. The skill rebuilds the current Inbox representation when those derived choices change; it does not append enrichment history sections.

That consolidation expanded the inbox skill from create-only immutability to Inbox-lifecycle ownership. The skill file grew, but the system got simpler: one place owns Inbox normalization, one shared rule set, one place with authority over the rules.

That was the opposite of what I expected. I thought a bigger skill file would feel heavier. Instead routing got easier. I stopped wondering which inbox skill to invoke for a correction vs a note-to-post change. I invoked the inbox skill, and the routing rules decided whether enrich included reclassify.

One capability, many internal operations

The incident suggests a short consolidate-vs-split test:

Same owned object under the same lifecycle gate. If lifecycle stage or artifact type diverges, stop consolidating.
Compatible mutation rules and safety boundaries. If the operations need conflicting write permissions or rejection rules that cannot share one authority, keep them separate.
Same type-and-shape rules across operations. Given the same input, if the operations would disagree about what kind of thing it should become, keep them separate.

Those are internal operations, not separate capabilities.

A parallel already existed elsewhere in the pipeline. A triage skill scores Inbox cards, recommends promotions out of Inbox, and archives the weakest rows. Those are different mutations, but they live inside one triage skill because they share the same queue-review ownership. I did not split score, promote, and archive into three skills. The operations differ; the owned workflow does not.

That comparison has limits. Triage promotion requires an explicit apply command after a dry-run report. Inbox enrich rejects cards that have already left Inbox. The internal gates differ without splitting ownership. The pattern is still recognizable: one skill per coherent capability, with internal routing between operations.

That does not mean every related action belongs in one skill. Scheduling, drafting, critique, and publishing own different artifacts and stop lines, so they remain separate.

The resource mattered more than the verb

You do not need Cursor to recognize the shape. A REST API does not usually become a separate service for every operation on the same resource.

POST creates. PATCH updates. DELETE removes. Different operations, same resource contract. Reclassification in my system is just another mutation of that same Inbox card resource.

Splitting those operations into separate agent skills was like building one service for create, another for update, and a third for delete. The endpoints looked clean in isolation. Ownership of the resource was fragmented.

Agentic workflows drift toward that decomposition because actions are easier to name than ownership. "Create card" and "enrich card" are vivid verbs. "Own Inbox normalization throughout the Inbox lifecycle" is accurate but abstract. The verbs made the skills easy to name. The resource revealed where the boundary actually belonged.

Your domains may decompose differently. The useful question is not "how many skills do I have?" but "what object does this capability own, and are these verbs operations on that object or different capabilities entirely?"

Takeaway: Start with one skill per action if that helps you ship. When multiple operations share an owned object, lifecycle, and compatible safety boundaries, consolidate them into one capability module and route internally. The risk was no longer a skill becoming too broad. It was one capability having multiple competing owners.

If you'd like to see the project behind these workflow experiments, try Codenames AI.

I expected pair programming with a Cloud Agent. I got a new hire.

Michael Truong — Fri, 31 Jul 2026 05:53:25 +0000

I thought a cloud coding agent was still pair programming: the same local conversation, just running somewhere else.

The first thing that surprised me was not the code. It was how little the run needed me.

My evidence is a first Cursor Cloud Agent run against a real monorepo. What initially looked like a Cursor feature turned out to be a different execution model. The run started over in a fresh environment, did the onboarding work, and left proof. Useful. Autonomous. And missing almost everything I already knew in the local session.

The wrong picture

In local Cursor I already had a planning thread: investigation, tradeoffs, and tools I had already authenticated in that session (including Notion MCP). When I pointed a Cloud Agent at Codenames AI, an npm-workspaces monorepo with an Express API, a Vite frontend, Playwright E2E, and CI, I expected continuity. Same decisions. Same auth. Same half-finished reasoning, just remote.

That assumption failed in the first hour.

What the first run actually did

The task itself was small: make the repository ready for future Cloud Agent runs and document the non-obvious setup in AGENTS.md. Completing it required establishing and proving the whole execution environment.

After that prompt, the agent worked without me sitting in the loop. It cloned the repo, installed dependencies, ran lint, typecheck, build, and tests, installed Playwright browsers, ran the E2E suite, played a Solo turn in the UI (a single-player practice game), captured screenshots and a walkthrough video, and opened a pull request.

That pull request added the AGENTS.md notes and recorded the verification trail from the clean VM, including a hello-world Solo turn in the UI. The change set was small. The behavior was large: autonomous environment setup plus artifacts local agent workflows rarely leave behind.

Two failures I did not expect

Local reasoning stayed on the laptop. Choices I had already made in Cursor (what mattered, what to skip, how I was framing the job) were not present in that new cloud task. Anything that depended on that judgment had to be re-established. The agent could reach the repo and CI. It could not inherit the argument I had already had with myself. Other Cursor flows can move a conversation into the cloud. This fresh task did not arrive with the local framing behind it.

Tool auth did not ride along. Notion MCP worked only after separate authentication for the cloud run. Local Cursor access was not session continuity. "The cloud can use MCP" and "the cloud already has my MCP sessions" turned out to be different claims.

Why it felt like hiring

Async handoff fit. Open-ended design debate in the cloud did not. The run wanted a bounded job and a definition of done, not a remote pair for figuring out the product.

You do not onboard someone by forwarding a half-finished Slack thread. You give them a machine, a checklist, and a brief. Missing context hurts more as autonomy increases, because a new clean-environment job starts without your local reasoning.

AGENTS.md mattered once the agent was in an unfamiliar checkout: non-obvious Node version floors, optional .env for local runs, E2E setup, font-sensitive visual snapshots. You do not need the file itself. The point is that this is briefing material for an execution worker, not a substitute for the planning conversation that happened on my laptop.

Why not another local worktree?

A second local git worktree can also provide parallel execution, and it has one obvious advantage: it can reuse the tools, credentials, caches, and machine state I already have. For a single developer, that can make the handoff cheaper.

But it leaves me managing another local workspace, and it keeps the result coupled to my machine: extra checkouts, local processes, port conflicts, and the risk that the result only works in my environment. A teammate cannot reproduce that run exactly without inheriting my laptop state.

A cloud agent starts with less inheritance, so the handoff matters more. In return, I get an isolated task that is easy to parallelize, with less "works on my machine" risk, and I can start or monitor work remotely without babysitting another local workspace.

Cursor was where I encountered the boundary clearly. Clean environments, onboarding, and missing laptop state are not new lessons if you have been shipping to remote servers for years. What was new was seeing that old systems idea reappear inside an AI coding workflow. The useful response is an explicit handoff, not a remote continuation of your session. Wherever coding agents become independently executable, I expect the same trade-off to appear. The boundary is architectural.

Where each side fits

Local workspace
(planning, judgment, brief)
        ↓
Cloud agent
(bounded execution in a clean environment)

That first run left open questions: whether later runs would reuse the onboarding work, and how the pattern would hold beyond one carefully bounded task.

Looking back, I kept choosing Cloud Agents for bounded execution: maintenance loops (including dependency upgrades), targeted bug fixes, and focused investigations such as tracing a review finding or validating a specific question. Not because the first-run costs disappeared, but because isolation and the shared task model reduced coordination overhead. The run still leaves a reviewable trail (screenshots, video, a PR), the same kind of demo evidence you would expect from another engineer handing work back.

The lasting surprise was that the clean boundary is not only a limitation. It is also the feature that makes the model scale.

Takeaway: A cloud agent gives up some inherited local context in exchange for isolation and scalable delegation. Brief it like a new hire, not like a remote continuation of your existing session. The context it needs must cross that boundary deliberately.

If you'd like to see the project behind these workflow experiments, try Codenames AI.

Active players looked real until we asked which sessions counted

Michael Truong — Thu, 23 Jul 2026 04:28:12 +0000

I've been building Codenames AI, a small web game where an LLM plays Codenames with you. Like most solo products, I glance at a Product Health dashboard when I want a quick read on whether anyone is actually playing.

One morning in June, three weeks after launching the site, the Active players tile said 64. Next to it sat 122 starts and restores. The number looked like traction. My first instinct was to treat it as confirmation and keep shipping.

That instinct did not survive the next question: which sessions were actually in that count?

The dashboard answered a wider question than I asked

I was reading Product Health as if every event in the project came from real players on the production site. The tile did not lie about its math. It counted distinct people who started or restored a game. What it could not tell me, from the chart alone, was which runtime those people were in.

I had reasons to trust the number:

PostHog init only ran when VITE_POSTHOG_KEY was set. Local Vite and Playwright runs did not ship that key, so I treated laptop and E2E traffic as silent by configuration. There was no analytics_environment property yet, and no environment-conditional init path. "Do not put the key in this build" was one guardrail.
Returning users looked safe too. On production, game state restores from origin-scoped localStorage, and PostHog keeps an anonymous ID on that same origin. Come back later and you still count as one Active player via game_restored. We do not call identify; continuity is browser storage on that host. I assumed testing on review URLs worked the same way: me again, already counted.

Outside PostHog, the acquisition picture did not match. Real arrivals were mostly organic Google Search. In Search Console, we had not yet hit the first “30 clicks from Google Search in the past 28 days” milestone. We had only just started posting on dev.to, so that channel was not a material source either.

Sixty-four unique players on a site that young, against a search funnel that had not cleared thirty clicks in a month, and early publishing that barely existed, was already a little suspicious. The starts/restores volume next to it made it worse. My working note was blunt: investigate further; something was minting unique players that real arrivals could not explain.

Review deploys were the hole

Review deploys (for us, Vercel preview URLs) look like the real app, often share the same analytics project key, and show up whenever you click a pull-request review link. They were not "local without a key," and they were not the same origin as codenames-ai.com.

A review hostname gets its own empty save store and its own anonymous PostHog identity, so a click-through during review can land as a new unique player (game_started) instead of folding into the production self I already knew. Without a way to separate those runtimes, that 64 was still a hypothesis about whether review-deploy traffic, and new identities on those hosts, were in the count.

That investigation became a concrete plan: stop treating every capture in the project as if it were production traffic.

An early cut disabled PostHog for E2E. Silencing one runtime would still leave review deploys sharing the key; we needed an explicit boundary instead of relying on some environments staying silent.

What should count as production?

Two fixes landed together.

Client tagging. On PostHog init, the frontend resolves an analytics_environment of production, preview, local, or e2e, then attaches it to every event and to the user profile. Hostname and the host’s build-time environment distinguish the runtimes.

Non-production traffic is excluded by dashboard filters, not by skipping PostHog init. Tagging every runtime, including ones we used to silence by omitting the key, is what makes the filter meaningful.

Dashboard filters. Product Health keeps events where analytics_environment = production OR not set, so older production events from before tagging remain visible. Newer views can use an exact production filter once tagging coverage is trusted.

The missing dimension wasn't another metric. It was the production boundary. Once that existed, Product Health could filter on it.

Healthy numbers resist questions

The harder lesson wasn't that the dashboard was wrong. It was that healthy-looking numbers are the least likely ones to get questioned.

While working on model experiments, failure exposed hidden assumptions. Here nothing looked broken, so curiosity had to do the same job: notice that the system was faithfully answering a different question than the one I thought I was asking.

How we ask the dashboard questions is a separate story. This post stays on the quieter failure mode: one project key, a review runtime that looked like production, and a number that looked clean until we asked.

What I'd check on the next dashboard

Start by asking what question the metric actually answers, not the one you hope it answers.
Compare it against an independent signal. If the numbers do not fit together, investigate before celebrating.
Look for missing dimensions that collapse different kinds of traffic into one KPI: environment, internal users, bots, staging, or another hidden segment.
Only then decide whether the fix is better tagging, better filtering, or a different metric altogether.

You do not need our dashboards or our app code to apply the pattern. Review deploys were the incident that exposed the gap here.

I cannot put a clean contamination percentage, from today’s data alone, on the period before we added tagging; the point is the missing question, not a guessed share of noise.

Takeaway: Production engineering isn't just responding to broken signals. It's occasionally distrusting reassuring ones. Metrics answer exactly the question you instrumented, not necessarily the one you think you asked.

If you'd like to see the project that inspired these lessons, you can try Codenames AI.

Model experiments became an architectural stress test

Michael Truong — Fri, 17 Jul 2026 15:41:21 +0000

I've been tuning Codenames AI, a small web game where an LLM plays Codenames with you. Clue generation is tightly constrained: one word, a count, optional intended targets, JSON on the wire, then deterministic validation before anything reaches the board.

As the project started attracting regular players, I wanted to improve the gameplay experience without blowing out costs. Moving one model generation from gpt-4o-mini to gpt-5-mini was my first instinct.

The default reasoning setting made responses an order of magnitude slower for this workload. Minimal reasoning looked like the obvious compromise: newer model, responsive gameplay.

I expected to compare clue quality, latency, and cost while the surrounding prompt, validator, and consumer contracts stayed put.

That last part was wrong.

The experiment stopped behaving like an A/B test

What showed up was structural, and it showed up in places that had been stable for months.

Validation failures started rising. Retries started rising. Entire candidate batches started failing before the game ever saw a clue. The sharpest signal came from a clue-selection path that had run untouched for months, and it hard-failed for the first time. They weren't latency regressions so much as architectural ones.

It is easy to read that as "minimal reasoning made the model worse." More often, the failures were exposing gaps in contracts that had looked fine under the previous model.

What each failure actually invalidated

Eventually every failure traced back to one of three layers:

Prompt contracts ask for exactly count targets and, in batch mode, several distinct candidates.
Deterministic validators reject target/count mismatches and filter invalid candidates before anything downstream runs.
Downstream consumers only see survivors. Empty batches retry with rejection feedback, then fall back if needed.

Those layers share one job: enforce the same invariants. The failures below cut across all three rather than mapping one to one.

Side commentary could kill an otherwise usable turn.

To pick a clue, one strategy (Strange mode) simulates how the AI guesser would respond to each candidate clue, then scores those simulated turns and keeps the best one. I thought those simulations would fail only when the guesses themselves were bad. After the swap, they could also fail because the model attached commentary about other words it had considered, including words that were not even on the board. Because the payload schema included that commentary, the validator had to treat it as part of the same all-or-nothing contract. A payload with usable guesses still got rejected, and when every candidate died that way, the turn came back as a controlled API failure instead of a clue.

Target cardinality had to match the clue count.

I thought my validator was protecting the game. Instead I discovered the previous model had been consistently producing outputs that satisfied those contracts.

Say the prompt asks for count: 2 and a targets array with exactly two unrevealed friendly codenames. Under the old model, a clue like {"word": "BUILDING", "count": 2, "targets": ["TOWER", "CASTLE"]} usually meant two real board words. After the swap, I started seeing the same shape with one valid target and one word that is not on the grid at all, or only a single target when count was 2. Valid JSON. Perfect keys. Intent status: invalid.

The validator rejects clues whose validated targets don't match count. Valid JSON wasn't enough.

Retries assumed the contracts were already specific enough.

I thought retries were simply robustness. Instead they became diagnostic tooling because they finally told me which invariant had actually failed. When a batch fails validation, the retry path can attach rejection feedback (failed clue words plus reason strings) so the next attempt is not a blind redo. That only helps if the contracts are specific enough to name the failure. Vague "try again" prompts hide whether you have a model problem or an underspecified invariant.

The failures showed up in the product, not just the logs.

Every rejected clue meant another retry before the player saw a move. On an AI spymaster turn, the game shows a clue, a count, and highlights the board words that clue is meant to cover. When the validated targets came back shorter than count, the UI looked broken: count: 2 with only one word highlighted. The AI guesser still trusted the clue count and started reasoning from a board state that never actually existed.

None of this required a different product thesis from schema-first validation. Valid JSON was never enough. The migration stress-tested whether prompt text, deterministic checks, and consumer assumptions still agreed after the model changed.

The uncomfortable part

On paper, the clue path already looked responsible. Prompt, validator, consumer. Clean separation.

The migration revealed a hidden layer:

Prompt
  ↓
Model capability
  (compensating for weak contracts)
  ↓
Validator
  ↓
Consumer

I expected to compare models. Instead I ended up comparing how much of my architecture each model had been compensating for.

While a more capable model kept quietly covering those weak contracts, the dashboards looked fine. Drop reasoning effort, and the same prompts start producing outputs that are honest about what you actually specified. Once that stopped happening, I was no longer measuring model quality. I was measuring how much of the gameplay experience had been resting on those hidden assumptions.

That is uncomfortable and useful. Apparent regressions (count mismatches, partial batches, more retries, collapsed guess simulations) are a signal to ask which layer was doing the work: the model, or the application.

Subjective "does this clue feel clever?" still matters for gameplay. It should not be the only scoreboard when the pipeline can reject an entire batch before the server ever picks a clue.

Treat migrations as compatibility tests

What I want out of a model swap now:

Align invariants across prompt, validator, and consumer. If the prompt says "exactly count targets," the validator must reject mismatches, and the API response shape must not pretend invalid intent is OK.
Keep structural correctness in deterministic code. Use the model for association quality. Use pure functions for board membership, cardinality, illegal clue shapes, and survivor lists.
Instrument validation failures by category. First-pass success rate, retry rate, and failure reasons tell you whether you tightened a contract or uncovered a real model gap.
Evaluate end-to-end workflow metrics, not only single-call latency or token price. Retries and fallbacks change the bill and the player experience; measuring only the happy path lies.

Takeaway: A model migration tests the model and the architecture around it. If prompt, validator, and consumer contracts do not enforce the same invariants, stronger models can mask weaknesses in those contracts until a cheaper or more literal model exposes them. The lesson is not really about which LLM you pick. It is about architectural coupling: the model itself had become part of the contract without me noticing.

If you'd like to see the project that inspired these lessons, you can try Codenames AI.

I fixed my AI reviewer. Then I kept solving the wrong problem

Michael Truong — Wed, 08 Jul 2026 05:09:51 +0000

I've been building an AI-assisted editorial pipeline for technical writing. Notion cards become markdown drafts in the repo, pass through review, then sync to dev.to.

Last month I shipped a post about the first big fix to my editor-critique reviewer skill: The AI reviewer scored 23/25 and missed the point. The problem was sequence. A score-first pass treated a polished rubric as the first lens and produced QA feedback when I needed editorial feedback. Reordering the skill so analysis precedes scoring fixed that.

I assumed the next improvements would come from rubric tuning. Longer prompts. Another scoring dimension. Sharper checklists.

That assumption was half right. The rubric still matters. But every useful fix after the baseline shared a different shape.

The pattern I kept missing

After I reordered analysis before scoring, reviewer failures kept arriving from different incidents. A critique that agreed with itself too easily. Drafts that grew every revision without getting shorter. A middle section that felt like a second article.

Each time I reached for the same lever: expand the rubric, add a rule, lengthen the prompt.

Incident 1: When the reviewer needs to argue with itself

editor-critique produced decisive scorecards and prioritized feedback, but the report rarely challenged its own conclusions. A draft could earn Ready to sync with medium items left unexamined.

Score-first review had failed because it judged too early. This failure was different: the primary critique could be thorough and still under-falsified.

The fix was another staged pass. After the primary critique drafts, freeze it. Run adversarial review that assumes the primary assessment is wrong until draft-supported counter-evidence proves otherwise. Then synthesize: change the publication recommendation only when falsification is material.

I added adversarial review, synthesis, and canonical report assembly as new skill steps. A follow-up pass tightened adversarial review with an anchor requirement: every counter-evidence bullet must name the frozen primary claim it challenges. No orphan hypotheticals like "title spoils thesis?" when the primary critique already praised title strategy.

Before:

Editorial read-through
→ Score
→ Critique
→ Post report

After:

Editorial read-through
→ Score
→ Primary critique
→ Adversarial review (frozen inputs)
→ Synthesis
→ Post report

That was the first time staging a different kind of reasoning into its own pass beat rubric expansion. Two more failures would repeat the same shape before I stopped treating it as coincidence.

Incident 2: When critique only adds

Self-falsification helped, but drafts were still growing. Investigation while critiquing Upgrades don't have to be a blind trust exercise showed feedback was consistently additive, but not subtractive. editor-critique found missing framing and evidence boundaries reliably. It did not ask what should be removed when new material arrived.

The result was layered drafts: an opening stacked on another opening, the same four-step investigation loop restated in three sections, a mental-model diagram that walked through event flow the prose had already established in the previous section.

The fix was not "be shorter" in the rubric. It was naming another cognitive job in the read-through: subtractive editing. Every paragraph should continue earning its place. Flag existing redundancy and addition-induced redundancy. Pair expansion recommendations with material that would become redundant if adopted.

A companion technique, single-owner ideas, lists 2–4 core ideas and flags when the same idea appears in multiple sections without new evidence. I codified subtractive editing in the skill file along with a test case that catches additive-only critique regressions and a lightweight subtractive pass in the human revision step.

The primary critique still owns expansion. Subtractive editing is a separate observational pass, not a rewrite engine.

Incident 3: When a section becomes its own article

The last failure pushed past critique mechanics into reader cognition. While critiquing draft variants in my editorial workflow, several middle-body sections were technically correct but felt wrong in context. In one draft, an implementation walkthrough interrupted the investigation arc. In another, a full section on validation tooling read like its own mini-article.

The failure mode was narrow: a section stopped advancing the reader's current question and temporarily made another explanatory thread the center of gravity.

Adding a rubric dimension for "section focus" would have been vague. What worked was an observational lens in the editorial read-through step: name the primary thread, name the secondary thread, decide whether to compress, delay, embed later, or leave as-is.

I codified this as a Secondary explanatory thread lens in the skill file. The rubric stayed the same. It simply added a named cognitive job: track whether prose is serving the reader's current question or drifting into a side article.

What stayed constant

Three incidents, three skill changes, one pattern. Across all three, a few constraints held:

The five-dimension rubric stayed mostly intact.
Read-only governance did not change: critique still does not write repo files or gate publish.
Each pass added another observational lens, not another scoring dimension.
The expensive part was naming the cognitive job precisely enough to operationalize in a skill file.

The recurring mistake was treating undifferentiated reasoning as one pass. Each fix changed the sequence, not the rubric weight. A capable reviewer can read before it scores and still under-read if falsifying primary judgment, displacing redundant prose, and tracking reader focus all compete in the same step.

Before you expand the rubric

List the failure modes that survived your last sequence fix.
For each one, name the cognitive job that failed (self-falsification, subtractive editing, reader-focus tracking).
Stage that job as its own observational pass with a frozen handoff to the next step.
Expand the rubric only if that observational pass still misses failures in production.

Once editor-critique understood before judging, the remaining improvements came from separating kinds of reasoning into distinct stages, not from a bigger rubric or a longer single pass. I suspect the pattern may generalize beyond editorial critique.

Takeaway: When a reviewer skill plateaus after a sequence fix, ask which cognitive jobs are still sharing one undifferentiated pass. Stage them before you expand the rubric.

If you'd like to see the project behind these workflow experiments, try Codenames AI.

Upgrades don't have to be a blind trust exercise

Michael Truong — Fri, 03 Jul 2026 09:02:30 +0000

I've been building Codenames AI as a solo project I want to keep alive. Renovate helps dependency upgrades move without maintenance eventually crushing momentum.

You don't need my exact setup to follow along.

Every project accumulates maintenance work. Framework upgrades are one place that work stalls, not because engineers do not know how to migrate, but because proving what actually needs to change takes time. That tradeoff shows up on a hobby repo as "I'll look at this when I have an uninterrupted evening." It shows up in production as major versions piling up while investigation competes with feature work.

That's the problem I was trying to solve.

Before AI, my realistic choices were narrow: trust the automation and hope, spend hours mapping release notes to my codebase, or leave the upgrade sitting.

I was not trying to invent a better review process.

I was trying to keep maintenance cost below available time.

AI changed the cost of investigation enough that I stopped treating it as something to postpone.

When Renovate opened a pull request for one of those framework majors, upgrading Vite from ^6.0.11 to ^8.0.0 and @vitejs/plugin-react from ^4.3.4 to ^6.0.0, my first instinct was still to treat it like migration work. Read the guides. Find the breaking changes. Plan the code changes. Validate the app. Then merge.

It was also the wrong starting assumption.

The useful work was not implementing the migration. The useful work was proving whether a migration existed for this repo at all.

The review gate was right

Renovate grouped the update as a frontend React/Vite major. My review policy sorts packages into low-risk and high-touch buckets: a patch-level bump to a type definition or a lint plugin can auto-merge, but anything that builds or serves the app (the bundler, the React plugin) is high-touch.

Both Vite and @vitejs/plugin-react sit in that high-touch bucket, so any version change routes to a human instead of auto-merge.

The pull request looked serious on paper:

Vite 8 release notes included explicit breaking changes.
The lockfile churn was large because Vite moved from the Rollup-centered dependency shape toward Rolldown packages.

The old workflow would have started with a migration plan. The evidence-first workflow started with a different goal:

Which documented breaking changes actually apply to this repository?

CI showed the branch built; it did not show the breaking changes were irrelevant to this repository. I ran the PR through an AI-assisted review and used the four-step checklist below to audit the result. My job was not to re-derive every fact by hand; it was to decide whether the evidence was enough to merge.

Investigation before implementation

The checklist covers four increasingly specific questions. For the Vite bump, the resulting evidence packet looked like this:

Inspect the upstream change.

Both packages shipped documented breaking changes. The review used those notes to name possible failure modes, not to assume which ones touched this repo. Vite 8's release notes called out, for example, SSR pipeline shifts and stricter import.meta.hot handling.
Map those changes to actual usage.

The packet reframed the question from "does Vite 8 have breaking changes?" to "does this app use the surfaces those changes break?" This repo has an ordinary Vite React setup with no custom SSR.
Identify custom risk.

The packet flagged one meaningful project-specific area: a small custom Vite plugin, cssBeforeModuleScript, that hooks transformIndexHtml to reorder the stylesheet and module-script tags. A bundler swap from Rollup to Rolldown could plausibly change that behavior, so this remained unresolved until the app was exercised.
Validate the app.

CI's test job had already been green; the packet still called for proof the custom-risk path and production build held. A ready Vercel preview closed that gap: it ran a production build through the new Vite with cssBeforeModuleScript included, and the rendered page was where broken stylesheet or module-script ordering would have shown up.

At that point, the recommendation changed.

The packet had not found migration work. It had found enough evidence that no migration was required.

Evidence over migration plans

The merged PR changed two files: frontend/package.json and package-lock.json.

No source files. No Vite config rewrite. No component changes. No test rewrites. No custom shim.

That is easy to misread as "the upgrade was trivial." It was not trivial. The pull request carried real risk signals. The absence of source changes only became meaningful after the investigation proved they did not require code changes.

Implementation only belongs once the evidence asks for it. The human job shifts from "please migrate this dependency" to auditing whether the packet answers the four steps above.

What this changed about manual review

Before this, "review manually" sounded like a parking lot. A major upgrade arrived, the automation refused to merge it, and the human picked it up later. Assembling the evidence packet by hand was often what stalled the review.

Now I treat "review manually" as an evidence-gathering lane. AI makes that lane practical: much of the packet assembly no longer has to happen in one sitting at your keyboard. That's what changed the economics. Investigation stopped being the expensive part that made maintenance easy to postpone. On the Vite bump, AI helped assemble the evidence; I audited whether it was enough to merge without migration work.

For low-risk patches the question stays simple: did CI pass and did the diff stay inside package files? For high-touch framework upgrades it gets richer: can the packet cover upstream changes, repo usage, custom risk, and real app validation well enough for a human to decide without first doing speculative migration work?

That does not remove judgment or the merge decision. It moves both earlier: gather the packet first, then audit whether implementation is actually required.

What I'd do on the next major upgrade

Classify the package honestly. Runtime and framework packages deserve more evidence than a patch-level dev tool bump.
Run the four-step loop above against release notes, repo usage, custom risk, and real app validation.
Treat zero source changes as a conclusion, not an assumption. If no implementation is required, say what evidence proved that.

Takeaway: When investigation is cheap enough to run, deferral stops being the default. Do the audit before you write the migration plan.

If you'd like to see the project behind these workflow experiments, try Codenames AI.

The AI reviewer scored 23/25 and missed the point

Michael Truong — Fri, 26 Jun 2026 15:14:45 +0000

I've been building an AI-assisted editorial pipeline for my technical writing. Notion cards become markdown drafts in the repo, pass through review, then sync to dev.to.

The motivation was simple: I already had a review loop I trusted for code. Open a PR, run Cursor's Bugbot against a review guide, fix what mattered, merge. I wanted the same rhythm for writing: draft, critique, revise, publish. So I built my own AI review skill called editor-critique.

I had also started adding HTML comments inside drafts, much like code comments. They captured the editorial intent behind a section, including why it opened where it did and why evidence sat where it did, without becoming part of the published post.

That made the review step look straightforward. Give the AI a rubric, score the draft, return prioritized feedback.

If the rubric was good, I assumed the critique would be good.

That assumption failed in a very specific way.

The first version of editor-critique did what I asked. It read a draft, applied five scoring dimensions, and produced a polished report. While reviewing my article, "The agent plan had every step except where to stop", it scored the piece 23/25 and mostly suggested polish.

It also missed the feedback I actually needed.

Valid rubric, shallow read

The draft did not need another pass on commas and section labels. It needed a colder editorial read.

A useful reviewer should have asked:

Does the title reveal the lesson before the incident earns it?
Does the article assume private repo context a dev.to reader will not have?
Are links to PRs, plans, and standards supporting evidence, or required reading?
Is governance framing outrunning what the incident actually proved?

Those are reader-journey questions, not formatting checks.

The score-first reviewer treated the rubric as the first lens. If the thesis was present, evidence was named, and the arc looked complete, the draft read as ready. The rubric turned critique into publication preflight: complete sections, reasonable voice, no obvious holes.

Useful, but not enough.

What changed in the sequence

I revised the reviewer skill so analysis precedes scoring.

Before:

Load draft
→ Score rubric dimensions
→ Generate critique

After:

Load draft
→ Editorial read-through
→ Score rubric dimensions
→ Generate critique

The rubric stayed. It stopped being the opening move.

Before scoring, the reviewer now reads visible prose like a cold dev.to audience member. It mentally strips author notes and asks whether the lesson still works if repo links and hidden rationale disappeared. Then it checks thesis timing, audience assumptions, reference framing, and speculation drift.

The annotation loop mattered here. Because the comments sat beside the sections they explained, critique could compare intent against effect: the note described what the section was trying to do, while the reader-facing paragraph showed whether it actually did it. Sometimes the article needed the edit. Sometimes the annotation exposed that editor-critique itself was reading the section too mechanically. Either way, the disagreement became useful training material for the reviewer skill.

Only after that read does it assign scores.

The output became more editorial. Instead of asking only "does this draft satisfy the rubric?", it started asking "what will break for the reader?"

On the same article, the revised reviewer surfaced title spoiling the lesson, private PR assumptions, weak framing for repo artifacts, and governance language potentially ahead of the evidence. The 23/25 pass had treated those as minor or invisible.

Why order beat rubric tuning

A rubric compresses judgment into categories: thesis, structure, evidence, voice, readiness. That compression helps consistency.

Compression too early can hide the problem.

Once the reviewer committed to a numerical assessment, the rest of the report tended to justify that assessment. A 23/25 draft needed 23/25 feedback, so the model organized its reasoning around why the piece was mostly ready instead of independently discovering what a reader would struggle with.

It is a little like running a linter before reading a design doc. The linter can confirm imports and formatting are clean. It cannot tell you whether the design makes sense. Start with the linter and the document can feel more complete than it is.

That is what happened here. The rubric was not bad. It was premature.

Once analysis came first, the same categories became more honest. "Evidence and specificity" could include link-only dependence. "Thesis and opening" could include title spoiling the lesson. "Publish readiness" could include whether prose survives without private repo access.

The score became a summary of the read-through, not a substitute for it.

QA review vs editorial review

The revision made me distinguish two kinds of AI review.

QA review asks: Did the artifact satisfy the stated criteria?

Editorial review asks: What will the reader misunderstand, miss, or not believe?

This was not completely new to me. In code review, I already used different Bugbot guides depending on what I wanted it to optimize for: security, game-state changes, UX regressions, or plan intent. The same diff could be reviewed through different lenses.

Writing turned out to have the same property as code review. A QA reviewer checks completeness and publishing criteria. An editorial reviewer reads for audience confusion and belief. The artifact stayed the same. The review lens changed.

Both matter. Broken frontmatter, missing sections, or absent takeaways still need QA. But if the reviewer starts and ends there, it can produce a confident report that never engages the reader's path through the article.

The first reviewer was not useless. It was doing QA under the name of critique.

The revised reviewer still scores, but it has to earn the score by reading first.

That sequencing shift moved output from "this article is mostly ready" toward "this article assumes too much context, reveals its lesson too early, and needs stronger in-narrative evidence before the governance argument about where an agent should stop lands."

That is the feedback I needed.

What I'd do on the next reviewer

For the next AI reviewer I build, I would design sequence before I tune rubric dimensions.

Start with an ungated read. Inspect audience, intent, risk, and evidence before scoring thresholds appear.
Make the rubric summarize the analysis. Scores should cite read-through observations, not invent them after the fact.
Separate checklist pass from judgment pass. "Is it complete?" and "is it good?" are different questions.
Force reader-impact language. Critique items should say what breaks for the reader, not only which rule was violated.
Let scores come last. Once a number appears, everything organizes around it.

This is not only about writing. I suspect the same pattern may apply to PR review, architecture review, incident analysis, and evaluation reports: if a reviewer scores before it understands, it overfits to the rubric and under-reads the situation.

The shape feels portable. Evaluation criteria are not enough. The order in which a reviewer thinks changes what it notices.

Takeaway: If your AI reviewer keeps producing technically correct but shallow feedback, do not only rewrite the rubric. Move analysis before scoring.

Editor's note (July 2026)

This article documents the first major architectural change to editor-critique: separating analysis from scoring. That sequence change held up, but it also exposed a new class of reviewer failures that couldn't be solved through rubric expansion alone. The follow-up, I fixed my AI reviewer. Then I kept solving the wrong problem, explores that next stage.

If you'd like to see the project behind these workflow experiments, try Codenames AI.

The agent plan had every step except where to stop

Michael Truong — Fri, 19 Jun 2026 06:29:47 +0000

I've been running multi-slice agent plans in the Codenames AI repo — Renovate migrations, content-pipeline skills, dependency upgrades. I split multi-PR work into slices (usually one pull request each), each backed by a markdown file with file paths, verification commands, and merge-safe acceptance criteria.

You do not need Cursor to recognize the shape: any agent workflow that can open branches, push commits, or merge PRs from a written plan has the same gap. In my setup I paste each slice into a fresh agent chat as a delegation prompt — not a ticket summary, but executable instructions — and start a new chat when that PR is ready.

I assumed the checklist was enough. The plan described what to build. I treated how far the agent could go as implicit.

Then an agent merged a pull request I expected to review first.

The merge that reframed planning

The trigger was mundane. During the first slice of a Renovate migration, an agent regrouped dependency buckets in renovate.json — config-only, no version bumps, no runtime behavior. It ran lint and typecheck, opened the pull request, and merged it.

The change itself was reasonable. Config-only renovate.json regrouping is exactly the kind of slice you'd want off your plate.

What surprised me was the absence of a documented stop line. The migration plan described the edit, the verification commands, and the acceptance criteria. It did not say whether the executing agent should stop at "open PR" or continue to "merge after green checks." The plan was an implementation spec. The agent treated it as permission to finish the job.

Implementation specs vs authority handoffs

Traditional engineering plans answer: what work should happen, in what order, with what verification?

Agent plans increasingly need a second answer: how much autonomy does the next actor get?

Those questions diverge the moment an agent can take repository actions — create branches, push commits, open pull requests, merge — instead of only recommending diffs in chat.

Question	Implementation plan	Authority handoff
What to change	File paths, diffs, acceptance	Same
How to verify	Commands, CI checks	Same
Where to stop	Often implicit ("human reviews")	Must be explicit
Who enforces limits	Code review habit	Plan recommendation + branch protection

A human teammate might read "prepare this for review" and stop. An agent reads a completed checklist and reasonably asks: "Verification passed — what's left?"

The first response wasn't the plan

My first reaction was not to rewrite the migration plan. It was to tighten the repository boundary.

Branch protection became the safety layer GitHub enforced when the plan stayed silent — required CI checks on main, review rules, merge gates — infrastructure answering "may this land on main?" regardless of what the agent thought the plan implied.

That helped. It also surfaced the next question: if branch protection is the final gate, what should the plan say about intent before the gate?

Repository guardrails and plan language solve different problems. Branch protection is authoritative — if merge is blocked, the agent stops. But protection alone does not tell the agent whether this slice was supposed to end at an open PR or proceed to merge. You still need the handoff to be legible before someone reviews the diff.

Making execution authority explicit

The follow-up was documentation, not a ban on agent merges.

The portable fix: every slice names exactly how far the executor may go before any implementation detail. We use two levels:

Level	Label	Agent instruction
Default	Open PR only	Do not merge. Stop after opening the PR.
Elevated	Merge granted	You may merge after documented verification passes.

Default is Open PR only. Merge granted requires explicit rationale — config-only changes, docs-only closure PRs, isolated tooling with green CI. Branch protection remains the final gate even when merge is recommended.

Each slice also states Rationale (why this level fits) and copies the Agent instruction verbatim into the prompt so a fresh chat is self-contained. A plan-level summary table at the top lets you scan a multi-PR plan and see where merge is elevated before you read file paths.

Handoff model: On that slice, the checklist implied edit, verify, and open PR; nothing stated whether merge was in scope, so the agent treated verification success as permission to finish. The chain we wanted spelled out: plan recommends authority → human accepts by executing the plan → agent follows the recommendation → branch protection enforces the final boundary.

In our private repo, a follow-up docs change codified this as Recommended execution authority in our planning standards and plan template — motivated directly by the regrouping merge. You do not need those files to apply the pattern; you need the label on every slice before the agent reads the checklist.

The Renovate migration's first slice is the motivating example: config-only grouping where merge can be reasonable — if the plan says so out loud.

What changed on the next slice

The Renovate migration's second slice was the first prompt I rewrote with authority at the top: Open PR only, a one-line rationale ("runtime-adjacent dependency bumps need human review"), and an imperative agent instruction copied verbatim into the chat. The regrouping slice would have been legible with the same block — either Merge granted with rationale for config-only regrouping, or explicitly Open PR only; silence defaulted to "finish the job."

I am not arguing for autonomous merge bots on every repo. The lesson is narrower: once agents act, plans delegate autonomy whether you write that down or not.

Human delegation has always been fuzzy — "take a pass at this" means different things to different people. Agent delegation punishes ambiguity faster because the agent will complete every step it can justify from the text in front of it.

The plan becomes the contract between author and executor. Implementation steps say what to build. Authority steps say how far to carry it.

Why not just forbid agent merges?

Fair pushback. If unexpected merges are the risk, disable merge capability and be done.

That misses what actually happened on the regrouping merge. The merge was not reckless — it was a config-only change with local verification and CI checks. Forbidding all agent merges would have blocked a useful outcome and pushed the work back to manual toil.

The interesting conclusion is not "agents should never merge." It is "agents need explicit authority boundaries."

Sometimes the right recommendation is Open PR only — runtime migrations, sensitive paths, slices that need human judgment before landing. Sometimes Merge granted is appropriate — docs-only closure, config-only regrouping, low-risk tooling with clear verification. The plan author chooses per slice. The agent follows the label. Branch protection catches mistakes either way.

Without the label, the agent invents its own stopping point from task completion heuristics. That is how you get surprised by a merge that was, by some readings, the correct next step.

What I'd do on the next agent plan

Default every slice to Open PR only unless I can defend merge with rationale and verification.
Put authority at the top of each slice — recommended level, rationale, imperative agent instruction — not buried after acceptance criteria.
Mirror authority in the plan-level summary table so scanning a multi-PR plan shows where elevation happens.
Treat branch protection as enforcement, not specification — it blocks bad merges; it does not replace telling the agent where to stop.
Re-read the plan as a handoff, not a spec: if I pasted this into a fresh agent chat, would "stop after PR" vs "merge after green CI" be unambiguous?

Prompt engineering still matters for implementation quality. It does not substitute for stating how much autonomy you are delegating when the executor can act on the repo.

Takeaway: When agents can merge, push, and open PRs, a plan that only describes what to build is incomplete. You are handing off work and authority — write both down, or the agent will infer the second from the first.

If you'd like to see the project that inspired these lessons, you can try Codenames AI.

One good example beat every AI writing rule I wrote

Michael Truong — Fri, 12 Jun 2026 07:38:06 +0000

I've been building an AI-assisted content pipeline around Codenames AI — field reports from the repo, drafted in markdown, synced to dev.to. The part I assumed would be hard was publish automation. The part that actually burned time was teaching the model how to sound like me.

The experiment

I started where most people start: the prompt. I wrote a Cursor rule with tone guidance, pacing notes, section shapes, and a list of things to avoid. If the draft felt flat, add another paragraph to the rule. If it over-corrected, tighten the rule. Iterate until the voice stabilizes.

That felt like the correct lever.

I assumed a longer, more detailed AI writing rule would produce better drafts. Voice felt like something you could specify in prose: a style encyclopedia with tone, pacing, and guardrails.

The failure loop

Each revision made the output worse in a different way.

The cycle was predictable: generate a draft, dislike the tone, add rules, get over-correction, revert partway, add different rules, hit a new failure mode. Some passes sounded like generic engineering docs: correct, but missing the observations that made the article worth reading. Others had no concrete details. Others followed every instruction and lost personality entirely.

The rule file kept growing. The drafts kept rotating through new ways to miss the mark.

The accidental discovery

The useful move, in hindsight, was deleting most of the rules.

I replaced the checklist with one shipped article: Schema first, prompt second: valid JSON wasn't enough. That post already had the shape I wanted — field report, wrong assumption up front, specific failures, tradeoffs, a single takeaway.

The Cursor rule shrank to a pointer: read the example, match the example.

"Write more like this article" beat "be direct, avoid metaphors, use short paragraphs, include a takeaway."

Drafts stopped sounding like engineering documentation. They started carrying the observations and pacing of a field report instead of a rule checklist.

Why the example transferred better

Rules describe voice from the outside. An example demonstrates it.

For long-form writing, an exemplar turned out to be closer to a spec than a style encyclopedia. Rule text and example text fail differently — a checklist compresses badly; an example carries decisions that are hard to encode as rules: pacing, level of detail, how much context to provide, and when to introduce examples.

When I asked for "direct engineer-to-engineer tone," the model complied literally and stripped the texture that makes a post readable. When I pointed at a finished article, it copied structural choices I hadn't thought to name: opening with context and a wrong assumption, using bold labels for contrast, ending sections with a concrete mistake instead of a principle.

The interesting part wasn't that the example contained better instructions. It contained decisions I didn't know how to describe.

I could recognize those choices when I saw them. I just wasn't very good at encoding them as rules.

What changed

Git history tells the story cleanly: the checklist-era rule peaked at 69 lines; the example-pointer rule landed at 23 lines.

After the switch, I spent less time fighting over-compliance and stripping generic phrasing. Voice became more consistent across drafts because the target was an article, not a growing instruction list.

Maintenance lesson: At 69 lines, the rule had enough instructions to contradict itself. A single canonical example stays honest. If the next post should sound different, update the example or add a second one for a new format. The rule stays an import statement.

Tradeoff: One example encodes one format. Field reports work; a tutorial or release note might need a second exemplar later.

Tradeoff: Examples can go stale. If the canonical post ages badly, future drafts inherit the wrong target. Treat the example like code you refactor, not like documentation you forget.

What I'd do differently next time:

Ship one article I'm proud of before investing in voice rules.
Point agents at that article.
Keep the Cursor rule as workflow plus a link, not a paraphrase of the example.
Add rules only for things examples can't carry: where files live, what not to paste into Notion, publish steps.

Prompt engineering still matters for facts, structure, and evidence gathering. For tone on long-form posts, though, one good example beat every style guide I wrote.

Takeaway

If your AI writing rules keep growing and the drafts keep getting worse, stop adding rules. Find an article that already sounds right and make that the spec.

Editor's note (August 2026)

The core lesson still holds: a growing style encyclopedia made drafts worse, and pointing agents at shipped writing worked better than encoding voice as rules.

The implementation has evolved. The editor-draft and editor-critique skills now calibrate against a short, human-maintained set of published field reports rather than one canonical structural example. The original schema-first article is still useful for tone and annotation style, but no longer defines the default structure.

The tradeoff remains the same: exemplars can go stale, so the set has to evolve with the writing.

If you'd like to see the project behind these posts, try Codenames AI.

Schema first, prompt second: valid JSON wasn't enough

Michael Truong — Thu, 04 Jun 2026 05:27:30 +0000

Over the last month I've been building Codenames AI, a small web game where an LLM plays Codenames with you. The guesser never sees unrevealed card identities. The server sends the board state and a clue; the model returns structured guesses with confidence scores and short explanations.

When I started, I assumed the hard part was prompting. I was half right. Getting something reasonable out of the model was fast. Making the system safe to expose to players was not.

My first milestone felt responsible: response_format: { type: "json_object" } on the chat completion, plus Zod schemas for the response body. If the JSON didn't parse or failed Zod, retry. Ship it.

Then I watched the model comply perfectly with the schema and still propose moves that would ruin a game.

Valid JSON, invalid game

Here's the distinction that mattered.

JSON schema (via Zod) answers: Did the model return the keys and types I asked for?

Domain validation answers: Is this output allowed on this board, for this clue, under these rules?

Those are not the same questions.

Three examples I hit while testing and running the game:

1. The model echoed the clue as a guess.

Codenames forbids guessing the clue word. The model would sometimes put it in guesses[] anyway—confidently, with a tidy explanation object. Zod was thrilled. The game was not.

2. The model hallucinated words that weren't on the board.

Perfect JSON. A guess list full of words that don't exist on the 25-card grid, or that were already revealed. Again, schema-valid.

3. The spymaster returned illegal clues.

Single-word clues can't match a codename, can't be a substring of one (or vice versa), and can't be near-miss spellings. The model regularly suggested clues that a human referee would reject. Valid JSON every time.

I spent too long fixing these by adding sentences to the system prompt. That helped a little. It did not help enough.

What actually moved reliability

The bigger wins came from code paths I treated as boring infrastructure.

Sanitization before trust. After Zod parses the guess payload, we strip clue echoes, off-board words, revealed cards, and duplicates, then realign the explanation array with whatever survived. The model can return whatever explanation it wants; the server decides which guesses survive validation.

Deterministic validators with explicit error strings. Clue validation returns things like "Clue cannot be a substring of a board word"—not "invalid." Those strings go back into the next attempt as rejectionFeedback, alongside an exclude list of clue words that already failed, so the next attempt could avoid repeating the same violations.

Post-processing for uncertainty. Even valid guesses get filtered by a confidence threshold before the client plays them. If nothing clears the bar, the API returns an empty guess list—the AI Guesser passes the turn rather than firing a weak pick. That's a product decision, but it only works because the earlier layers stopped nonsense from masquerading as success.

None of this required readers to know Codenames. It's the same shape as any LLM feature with invariants: inventory counts that can't go negative, user IDs that must exist, action enums that must match state machines.

Mistakes, surprises and tradeoffs

Mistake: Treating structured output as the guardrail. It only enforced shape.

Surprise: Sanitization outperformed prompt engineering for the dumbest failures (echoed clue, off-board tokens). Cheap deterministic filters beat another paragraph of "IMPORTANT RULES."

Surprise: Retry feedback with the reason a clue failed worked better than "try again." The model stopped repeating substring violations faster when the server named the violation.

Tradeoff: Retries burn tokens. Logging validation errors per attempt was essential to know whether we had a prompt problem or a missing rule.

Tradeoff: Sanitization can mask drift. If you silently drop bad guesses, monitor what you're dropping or you'll quietly turn the validator into the thing making all the decisions.

What I'd do on the next project

Define the wire shape (JSON + schema).
List domain invariants as pure functions with test cases
Add sanitization for the failure modes observed in the first 50 live calls.
Only then invest in prompt nuance—and feed validator messages into retries.

Prompt engineering still matters for quality. It is not a substitute for enforcement when the user can lose a game—or money, or data—because the model followed the JSON spec and ignored reality.

Takeaway: If your LLM integration stops at "parse JSON, call it a day," you haven't finished the feature. You've finished the demo.

If you'd like to see the project that inspired these lessons, you can try Codenames AI.