DEV Community: Michael Truong

Throwaway experiments are easy to start. Retiring one safely is not

Michael Truong — Thu, 10 Sep 2026 06:04:45 +0000

I was closing out a throwaway repo from an agent-workflow experiment. I had treated experiment repos as cheap to delete once the hypothesis felt answered. The prototype had to go because leaving both checkouts live gave later agents two competing sources of precedent. Deleting it meant deciding what had been validated, writing it down somewhere durable, and removing experiment surfaces only after that record was complete.

Standing up the narrow prototype had been genuinely fast. Portable agent policy, skills, and repo bootstrap had already made that part easy. Safe retirement was not.

What was validated before deletion

The experiment had tested one narrow claim: a working agent could turn captured learnings into retrospective summary cards in a single pass, without a dedicated second inference service. The prototype wired learnings into Notion, ran that synthesis step, and left card formats, retrieval defaults, and cap rules as unproven demo choices. Most of the wiring was unhardened prototype.

By retirement time, that conclusion was already in architecture notes. Everything else in the prototype wiring was discardable.

The audit question was not "did I copy every file?" It was "did I record every experimentally supported conclusion before I delete the experiment?"

The answer was yes. That cleared the way for the part where things actually got scary.

Three ways experiment retirement goes wrong

The first failure mode is preserving provisional choices because they feel unique.

When I asked an agent to plan the retirement, its first draft tried to absorb all the demo logic into architecture notes before deletion: card formats, retrieval defaults, cap rules, idempotency guesses. That treats retirement like archival absorption. The audit question above already ruled that out.

Agents completing a "retire safely" goal default toward documenting every visible artifact, because nothing marks validated conclusions versus provisional hacks unless you write an explicit discard list. An earlier architecture pass had already elevated some of that demo protocol as if it were product design; retirement needed de-specification, not more absorption.

Two different leftovers would have read as precedent. One run had produced five retrospective cards from eight captured learnings; that still looked like evidence even though the pipeline was unproven. A forty-card demo cap would have landed the same way: the next agent writes "last time we capped retros at forty cards" into architecture even when the experiment never validated that limit.

My retirement plan named what not to absorb so the real product could be designed from validated direction plus product constraints, not inherited defaults.

The second failure mode is deleting before you unlink.

Live GitHub and Notion links in architecture notes look harmless until the repo is gone and the URLs 404. Agents (and future-you) follow those links, infer missing context, or treat dead references as signals that something was lost mid-migration.

A docs-only absorption step had to land first: reframe the experiment in past tense, strip live experiment URLs, drop protocol details the experiment never validated, and point readers at the written record.

Only after architecture notes were self-contained did I delete the GitHub repo, the Notion databases, and the local checkout.

The third failure mode is treating search results as deletion targets.

When you have three related Notion databases (raw learnings, retrospectives, improvement ideas) linked by relations, you need a verified deletion set before you touch anything. My retirement plan listed exact database IDs and data source IDs, verified independently against the live workspace, and deleted them in reverse dependency order: improvement ideas, then retrospectives, then raw learnings.

Stray-page search was allowed only as inspection: find standalone pages that might belong to the experiment, report them, and do not delete anything that search returns unless you can prove it is experiment-only.

That rule exists because search is retrieval, not enumeration. A query that mentions "learning" or the experiment project name will surface unrelated pages. An agent under pressure to "clean up" can easily delete the wrong surface if search hits become the allowlist.

Why retirement needs semantics

Agentic workflows lower the cost of spinning up experiment repos, skills, Notion schemas, and multi-root checkouts to test a hypothesis in isolation. Nothing in the default toolchain lowers the cost of retiring them safely, so teams accumulate ambiguous surfaces that agents treat as canonical.

Those three failure modes are what retirement looks like without explicit decisions. The sequence that held up looked like this:

Step	What it did
Plan	Write down what will be deleted, in what order, and what must be captured first
Doc absorption	Make architecture notes self-contained: past tense, no live experiment links
Delete external	Delete GitHub repo, Notion DBs, and local checkout with exact ID allowlists, under human verification

Whether that retirement contract should live in a central registry, per-repo retirement plans, or something else is still an open design question.

Takeaway: Answering the hypothesis is not retirement. Capture validated conclusions in durable docs, make those docs self-contained, then delete external experiment surfaces from a verified allowlist under human verification.

The board came back. The highlights lied.

Michael Truong — Fri, 04 Sep 2026 02:26:58 +0000

I ship Codenames AI, a web game. Each game mode keeps its own save in localStorage. Reload the tab, switch to a different mode, come back later: the board, turn, clue history, and in-progress results all come back. That felt like a win until I added spymaster clue targeting.

While drafting a clue, I can click cards on my team. Those clicks sync the clue count and show which words I had in mind. They are visual intent only. They are not part of the clue submission payload.

I assumed that if I reloaded the same game, the UI could restore those highlights too. Same session, same cards, same mental model. On a single-mode refresh that was mostly harmless.

Switching game modes was not. Each mode loads its own 25-card board. Leftover targeting clicks stayed in memory. If a word on the new board matched a card I had highlighted in the previous mode, that new card lit up. I had never clicked it. The board was truthful. The UI was lying about what I was drafting.

Clearing those highlights when the board changed would have stopped the lie. That cheaper fix was not enough. I had already assumed a same-game reload should bring the clicks back. Keeping that assumption and stopping the leak meant persisting the clicks per mode, the same way we persist the board. That would have treated a thinking aid like a move. The collision forced the real question: should coming back restore those clicks at all?

What coming back restores

People now expect drafts to survive a reload. Google Docs made that the default: leave, come back, the paragraph is still there. Autosave is already on in this game. A lot of players never press Save. They just come back.

A saved snapshot carries the board, the turn, typed clue fields, history, and any in-flight result. It omits targeting clicks on purpose. The snapshot type documents that omission in gamePersistence.ts.

A clue you typed already works the Docs way. An unsubmitted AI-generated clue does not. Refresh asks the model again. The clue word and the highlighted targets can change (INSECT and one card can come back as BODYPART and two different cards). That generation has not crossed into accepted game state. A useful consequence is that a model experiment, upgrade, swap, or config change can take effect on the next reload instead of replaying the last output.

Human targeting clicks are the same shape: a thinking aid around a later submit, not a move on the board.

Restored from snapshot	Cleared or regenerated on restore
Board, revealed cards, team, outcome, and pending guess result	Human-intended target highlights
Human-typed clue word and count	Unsubmitted AI clue and targets
Submitted clue history	Manual clue-count override flag

That is why "persist domain state, discard UI state" is the wrong summary. Some UI state should persist (the typed clue). Some AI-generated state should not, even though it occupies the same fields. React state versus a domain object does not tell you enough. The snapshot has to encode what the product has accepted as true, not whatever happened to exist in the UI, and not because anyone pressed Save.

What restore actually does

Switching modes and reloading the tab both restore the saved game, then clear the drafting pose. Targets, the manual count override, and any cached AI overlay go with it. The same clear runs on new game and after a successful submit, so a leftover thinking aid cannot leak into the next turn.

So restore is not "rehydrate everything the component used to know." It is "restore the game, then clear the drafting pose."

The restored count is durable user work: the number you typed comes back. What does not come back is the live session flag that blocked auto-sync from targets. After reload, the first target selection re-derives count from the grid. That is the same interaction as changing targets in the same session without reloading (type 5 → change targets → derived count matches type 5 → reload → select a target → derived count). The override flag is transient drafting pose, not durable provenance like human-typed clue word vs unsubmitted AI clue.

The regression tests I care about are behavioral:

Grid reference: highlight on board A, switch to board B that shares the word, assert nothing is lit.
Provenance: after reload, a human-typed clue survives while an unsubmitted model-generated clue is regenerated. The test controls the model's second answer so replaying the saved output fails deterministically.
Count after reload: manually typed count survives reload; the first target selection afterward re-derives count from the grid (same as in-session target toggles).

Highlights only reflect choices made in the current drafting session on the current grid.

Same problem, different surfaces

The leftover highlight was not a styling bug. Dropping the clicks from the snapshot fixed the category: a thinking aid is not current truth.

If you are building agent UIs with drafts, wizards, or "thinking aloud" interactions, coming back poses the same question. Autosave does not settle what on the screen is current truth. Keep proposals and thinking-aloud hints ephemeral, even when they sit in the same inputs as accepted work. Document that acceptance boundary in the snapshot type so the next contributor does not "helpfully" persist whatever the component last held.

Takeaway: Coming back should restore what the product treats as current truth, not whatever happened to be on screen. Accepted work stays. Proposals and thinking-aloud hints do not, even when they share the same fields. The persistence boundary is semantic, not architectural. Test what coming back looks like, not whether the save still loads.

If you'd like to see the project that inspired these lessons, you can try Codenames AI.

I was solving agent portability at the wrong boundary

Michael Truong — Fri, 28 Aug 2026 01:12:13 +0000

Copying the last repo's agent setup into a new one worked at first. It also copied product-specific assumptions. Once I had several active projects, there was no longer a single canonical repo I could copy from. Every improvement now had several places it could drift.

I was doing that across Codenames AI, a portfolio site, and a resume generator. Shared workflows for planning, editorial work, dependency upgrades, review, and repo bootstrap had accumulated around them. Some still lived inside product repos simply because that was where they had evolved.

I wanted the next repo to start with the methodology already available, without cloning the implementation details of the last product.

My first instinct was to solve that with an MCP-shaped architecture. Extract the shared behavior into a separate repository. Expose it through a remote tool boundary. Every repo could call the same capability when it needed merge-safe planning rules or editorial workflow guidance.

That felt rigorous. One service. One contract. One place to version policy.

Why the MCP-shaped instinct looked right

The idea lived in notes and conversation: treat portable agent policy the way you would treat a remote tool.

Building it would have bought a clean contract. It would also have attached the baggage that belongs to real tools:

a runtime or service boundary
an MCP contract and deployment story
versioning and invocation decisions
"when do we call this?" routing inside every agent session

That overhead makes sense when the capability is genuinely external: query Notion, pull PostHog metrics, deploy through Vercel. It does not make sense when the capability is mostly operating methodology: how to slice plans, when to stop after opening a PR, how to keep merge-safe invariants explicit.

Planning standards and merge-safe workflows are agent policy and procedure. They are not remote resources waiting behind a tool boundary.

I never built that service. I did not need to.

What shipped instead: four durable scopes

The decomposition was the work: what should follow me into every repo, what should stay behind a tool boundary, what should stay with me as procedure, and what must live in the repo itself. I happened to implement that split in Cursor (user-level rules, MCP config, installable skills, repo files). The architecture is the scopes:

Scope	Role	What belongs here
Always-on policy	Invariants in every repo	Short rules: execution authority, repository topology, stop-after-open, tool preferences, pointer to planning methodology.
Shared tools	External tool boundaries	Notion, PostHog, Vercel after you authenticate the service.
Reusable procedures	Workflow installation	Full portable skills such as staged planning and new-repo bootstrap.
Repo files	What must live in the repo	Stable mechanics (local hooks, remote environment lifecycle, CI) and product knowledge (`AGENTS.md`, domain rules, product skills, review guides)

Always-on policy kept a pointer to the planning methodology instead of a second copy of it. Copying one product repo's harness into another just to match would have recreated the drift.

Shared tools stay behind that authenticated service boundary.

Reusable procedures stay at user level. The new repo does not store those skills. I run bootstrap once to write stable mechanics into repo files. That step does not copy the skill into the repo, and it does not write product knowledge because it varies by product.

The scopes also have different update semantics: policy changes flow across existing repos, while bootstrap changes become the baseline for new ones unless I explicitly migrate older repos.

The interview showed what still had to be reconstructed

The first serious cold-start test was a timed AI-native product-build interview. I used the same scopes in a genuinely new repo under time pressure.

The interview proved the methodology did not depend on my existing repos. It also showed that too much generic setup still had to be reconstructed in an empty one. The agent put instructions in the README instead of AGENTS.md. Hooks that should have wired the remote agent environment were not reliably set up. Setup that was obvious in my established repos was not obvious when an agent had to invent it under time pressure.

Policy, procedures, and shared tools were already available. What failed was leaving stable repo mechanics to be rediscovered. An empty repo still has to run those hooks and that remote environment. The following week I converted more of that baseline into deterministic bootstrap: known-good scripts and templates for local hooks, remote environment lifecycle, CI, and other baseline infrastructure, not another round of agent redesign.

The goal is not zero bootstrap. It is to stop spending agent reasoning on decisions I have already made.

Takeaway: Splitting the problem by ownership and lifecycle showed I did not need an MCP-shaped architecture. Policy could stay always-on, procedures could stay at user level, stable mechanics could be materialized by running bootstrap, and the agent could spend its reasoning on the product.

Editor's note (September 2026)

Since publishing this, I extracted the implementation behind this approach into an open-source workflow and shipped it as a Cursor plugin. It includes the portable agent policy, reusable skills, and deterministic repo bootstrap described above.

The implementation is available in the multipliers-dev/cursor-team-marketplace repository on GitHub.

The pipeline was green. The product was underspecified

Michael Truong — Mon, 17 Aug 2026 08:23:00 +0000

The checks all passed. I still would not have sent the resume.

I had been using a Cursor agent to implement a private facts → prose resume generator. Structured career claims in, recruiter-facing PDFs out. Generation, rendering, and ATS checks all stayed green. The PDFs looked plausible.

I treated that as enough.

Several product requirements were still wrong.

What implementation QA was checking

The useful split is structured facts on one side and disposable rendered artifacts on the other:

Layer	Holds	Does not hold
Structured facts	Stable claims (actions, outcomes, metrics, scope)	Resume bullet phrasing
Application config	Which facts to include, tone, theme, page length	New career claims
Generated output	Markdown and PDF resumes	Source of truth

Implementation QA in this workflow means the pipeline runs end to end and the automated checks pass. Generation succeeds. PDFs render. ATS scripts assert page counts, required sections, and a few structural rules about separators and headings.

Those checks are real. They caught broken builds and regressions I did not want to ship.

They did not answer a different question: did the specification describe the resume I actually wanted?

That gap showed up in three places. None of them failed the scripts at first.

When page count is not product fit

The workflow produced one-page and two-page variants for the same application. Both PDFs passed check:ats. Both stayed inside their page limits.

The two-page resume still read like a stretched one-pager.

Experience on the shorter version used concise evidence entries selected for a tight one-page fit. The longer version reused that same condensed slice, then filled the remaining space with additional facts. Page count was correct, but the shape was wrong: the original concise bullets never expanded into fuller evidence. A two-page resume should deepen the roles that already earned a place, not keep the one-page wording and pad with more items.

Mismatched typography could fake the same green result by enlarging text on the longer PDF. Unifying the shared typographic scale and margins removed that shortcut: a two-page count had to come from the evidence itself.

Implementation QA had no opinion about which facts belonged on which page length, or whether two pages meant more evidence or just larger fonts. It only knew the PDF had two pages.

When valid data reads wrong to a human

Not every miss was about page length.

One Program Lead role was technically valid in the data: correct dates, correct employer, correct title. In the Experience section it rendered with a Full-time employment label. On paper that is accurate enough for a schema. On a resume it reads like a sequential primary job when the role was actually concurrent with other work.

The requirement was not "store valid employment metadata." It was "make concurrent work legible to a recruiter scanning the ladder." Renaming the label to Concurrent program was a product fix, not a pipeline fix. No ATS script flagged the old wording.

When structural correctness stood in for finish

The last category looked optical. It was still underspecification.

I only noticed after opening the PDF: the contact block and Skills sidebar shared a column edge on paper, but the Skills heading sat a few points lower than Experience, so the two-column header row looked crooked even though every section and separator rule still passed. ATS-safe contact separators are a constraint, not a design.

The workflow had encoded structural correctness: page counts, required sections, separator rules, heading shape. It had never named a human visual-acceptance criterion. The scripts were never asked to stand in for a finish requirement the spec had never named: would I send this?

What manual implementation used to smuggle in

When I wrote this kind of tooling by hand, implementation and requirements review were harder to separate. Every intermediate decision was visible. Choosing a font size, picking which bullet to cut, or rewriting a concurrent-role label forced a product judgment in the same session as the code change.

Manual implementation was accidentally doing requirements QA. Every ambiguous decision eventually became my problem because I had to turn it into code myself. An agent can absorb that ambiguity instead.

The two-page PDF was that pattern in miniature. It looked finished. The missing product decision (whether two pages meant deeper evidence) never came back through me.

That speed is the danger now. Incomplete specification can arrive dressed as a finished product.

An earlier piece was about why building this system suddenly made economic sense. This is the other side of that shift: once implementation got cheaper, I needed to make requirements review more explicit.

What a requirements QA stage could look like

The portable version is a few jobs, not a particular toolchain:

Force the vague request into concrete behavior before implementation: what changes, what stays invariant, and what counts as done
Review against that written intent, not only the local diff
Accept the finished artifact as a product ("would I send this?"), not only as a green pipeline

The first two are where requirements QA is cheapest. If those questions stay implicit, an agent can execute the wrong thing extremely efficiently.

In my workflow that looks like:

plan                        (overall intent)
→ slice                     (concrete behavior)
→ review slices             (gaps and boundaries before any code)
→ for each slice:
    → implementation
    → review after code     (still matches plan?)
→ artifact acceptance       (ready to send?)

I commit the plan alongside the work so the intent lives in the repo, not only in the chat. A reviewer is not limited to the current diff. It can see where this slice is supposed to lead, and flag something that is locally correct but inconsistent with a later slice. Reviewing the slices can still challenge assumptions, missing requirements, and slice boundaries before any code exists.

The ladder above is how I recreate the interrogation that manual implementation used to provide implicitly. Planning and review still miss gaps that never made it into the written requirements. Artifact acceptance is the last gate for those: it catches product misses the earlier rungs never named.

Takeaway: Agents are good at satisfying the requirements they are given, quickly enough that a miss looks finished. Implementation QA proves the system did what you asked. Somebody still has to QA whether those requirements describe the product you actually wanted.

If you'd like to see the same gap between green checks and product acceptance on a product I keep revising in public, try Codenames AI.

AI changed the build-vs-buy threshold

Michael Truong — Thu, 13 Aug 2026 06:39:39 +0000

Building custom software to solve a two-afternoon problem sounded absurd.

A Riot Games recruiter reached out while I was still preparing to return to the job market. Suddenly I needed a current resume to send back, and I had roughly two afternoons to produce one.

Normally that is an obvious buy decision. Under a short deadline you are not optimizing for reuse. You are optimizing for a PDF in someone's inbox. A resume builder gives you templates, export, and enough polish to look professional without inventing infrastructure.

Historically, I built when the reuse justified the setup cost. I bought or assembled manually when I only needed the artifact.

The same rule still applied. What had changed was the cost.

AI had lowered not only how much it cost to build the first version, but how much it cost to keep revising the architecture underneath it.

What I built instead

I built a private facts → prose resume repository with Cursor.

The idea is to separate career evidence from application wording:

Layer	Holds	Does not hold
Structured facts	Stable claims (actions, outcomes, metrics, scope)	Resume bullet phrasing
Application config	Which facts to include, tone, theme	New career claims
Generated output	Markdown and PDF resumes	Source of truth

Career claims live once in structured YAML. Each application selects, reorders, and rephrases them. npm run generate renders recruiter-facing prose. npm run pdf prints it. npm run check:ats runs structural ATS checks on the output.

You do not need my private repo to apply the pattern. The useful split is structured facts on one side and disposable rendered artifacts on the other.

Before generating a resume, the workflow researched the company and role, then used that context to decide which evidence from my career inventory belonged in the application. The system knew about far more career evidence than any one resume should contain. The inventory stayed put. What changed was which slice mattered.

For a gaming company like Riot Games, the research made a university game-design award relevant enough to surface on a software engineering resume where it normally would not belong. The same run also produced a one-page and two-page resume plus a recruiter reply.

A week later, I used the same career inventory for an AI product engineering company. The underlying evidence had not changed, but what mattered had. The workflow selected a different slice of it. The reuse I had been optimizing for was already real.

Later it expanded past resumes into interview prep materials, a use I had not planned when the recruiter first wrote. Different company, different evidence selection. Different stage, different artifact altogether. The system that looked overbuilt for one reply kept finding uses.

The system did not start here

The repository did not begin with that model.

It evolved through increasingly useful abstractions across the same short build window:

A single resume document
A reusable resume template
Career facts separated from prose
Capabilities grouping related achievements
Evidence entries inside each capability
Application-specific selection over that inventory

The first workable version was not the last one. Because implementation and revision were cheap, I could keep moving instead of freezing at "good enough for tonight."

Without that cost shift, the rational stopping point would probably have been a reusable template or a lightly parameterized document. Fine for one application. Weak as career infrastructure.

That was the advantage of the build path here: room to discover the right abstraction after the first one works.

Two costs moved, not one

AI did not magically make build correct for every problem.

It lowered two costs at once:

Implementation cost: scaffolding the generator, themes, and checks stopped being a multi-week side project.
Architectural iteration cost: revising the underlying model (facts vs capabilities vs application selection) stayed cheap enough to do in the same session.

Buying still wins when the problem is narrow, the tool fits, and you will not reuse the result. Building still loses when maintenance will crush you.

What changed is the boundary. Problems that used to land firmly on the buy side can cross over when reuse matters and you can afford to iterate past the first design.

Buy was still faster for one send

This is not an argument that custom software always beats SaaS. I chose to build because the deadline still left room for something reusable. If the goal had been a single resume, buying an off-the-shelf builder would have been the faster path.

The build path made sense here because:

The career inventory is the stable core.
Research and role context drive application-specific selection.
The same system produced multiple resume variants and a recruiter reply.
It later expanded into interview prep materials.

Cheap iteration on both implementation and revision is what made exploring these abstractions affordable.

The same economics show up outside resumes: internal tooling, dashboards, documentation pipelines, code generators, personal workflow infrastructure. Anywhere the old math was "custom software is too expensive to build and revise", the revise term got smaller.

I suspect that generalizes beyond software engineering into knowledge work where bespoke systems used to lose to packaged tools on turnaround alone.

Takeaway: AI moves the build-vs-buy threshold by cutting both implementation cost and the cost of changing your mind about architecture. When reuse matters and you can iterate in the same sprint, building a small internal system can become rational where buying would previously have won on cost and turnaround.

If you'd like to see the same economics on a product I keep revising in public, try Codenames AI.

One skill per action looked like the safe boundary

Michael Truong — Fri, 07 Aug 2026 05:24:54 +0000

I started with a rule that felt like good engineering: one skill per action. Create a card here. Enrich it there. Reclassify it somewhere else. Each prompt got a clean boundary. Each file stayed small.

Then that decomposition started to fight the domain.

I ran into this while building an AI-assisted editorial workflow in Cursor, but the problem was not really about Cursor or publishing. It was about where an agent capability should begin and end.

In that setup, a skill is a markdown file the agent loads for a workflow. My editorial workflow uses those skills to create and manage Notion cards before drafting posts.

When every action gets its own skill

The first version of my inbox skill only created Inbox cards. That matched early usage: capture an observation, normalize it into a canonical shape, attach a small set of grounded references, and stop. The skill was create-only and treated each new card as immutable once it left Inbox.

As capture matured, the same card kept needing more work while it was still Inbox:

Create when a new observation arrived
Enrich when new evidence or framing changed the normalized shape
Reclassify when routing rules decided the card should be sparse vs rich, or a quick note vs a planned blog post

Those felt like three different jobs. They had different verbs. They had different retrieval triggers. Splitting them into separate skills seemed obvious.

The split failed in ordinary corrections. I would enrich an Inbox card with new evidence, then realize it should change from a quick note into a planned blog post. That meant a second skill invocation for reclassify, with a second copy of the same lifecycle rules. For a moment it was unclear which skill was still responsible for keeping the page as one current write-up instead of an accumulating edit history. Get the order wrong and you did the work twice: an enrich that left the card type stale, or a reclassify that ignored the evidence rewrite you still needed.

The deeper problem was ownership. All three operations touched the same owned object: a single Inbox card in a Notion database. They shared the same lifecycle gate (the card must stay in Inbox), the same mutation boundaries (never change lifecycle status, source path, or published URLs on existing pages), and the same routing rules for how sparse or rich the card should be and whether it was a quick note or a planned post. They also shared the same rule: after every update, the page must hold exactly one current normalized write-up and exactly one captured observation. Notion history is the revision log; the operational card is not an append-only audit log.

Treating Create, Enrich, and Reclassify as three skills meant three prompts trying to enforce one coherent capability. The boundaries were at the wrong layer.

Consolidation made the skill clearer

The fix was to stop pretending those were separate capabilities. One inbox skill now owns the full Inbox lifecycle: Create, Enrich, and Reclassify while the card remains in Inbox.

The operator still invokes one skill. The skill routes internally:

User-facing command	What it does
create inbox card	Creates an Inbox card with a canonical write-up and grounded references
enrich inbox card	Folds new evidence into the existing Inbox page as one current write-up
Same-thread continuation	Treats further capture in the same chat as enrich on the page just created

Reclassify is not a separate user command. It is detected inside enrich when the routing rules decide the card should be richer or thinner than before, or should shift from a quick note to a planned post. Sparse captures can become richer field reports. The skill rebuilds the current Inbox representation when those derived choices change; it does not append enrichment history sections.

That consolidation expanded the inbox skill from create-only immutability to Inbox-lifecycle ownership. The skill file grew, but the system got simpler: one place owns Inbox normalization, one shared rule set, one place with authority over the rules.

That was the opposite of what I expected. I thought a bigger skill file would feel heavier. Instead routing got easier. I stopped wondering which inbox skill to invoke for a correction vs a note-to-post change. I invoked the inbox skill, and the routing rules decided whether enrich included reclassify.

One capability, many internal operations

The incident suggests a short consolidate-vs-split test:

Same owned object under the same lifecycle gate. If lifecycle stage or artifact type diverges, stop consolidating.
Compatible mutation rules and safety boundaries. If the operations need conflicting write permissions or rejection rules that cannot share one authority, keep them separate.
Same type-and-shape rules across operations. Given the same input, if the operations would disagree about what kind of thing it should become, keep them separate.

Those are internal operations, not separate capabilities.

A parallel already existed elsewhere in the pipeline. A triage skill scores Inbox cards, recommends promotions out of Inbox, and archives the weakest rows. Those are different mutations, but they live inside one triage skill because they share the same queue-review ownership. I did not split score, promote, and archive into three skills. The operations differ; the owned workflow does not.

That comparison has limits. Triage promotion requires an explicit apply command after a dry-run report. Inbox enrich rejects cards that have already left Inbox. The internal gates differ without splitting ownership. The pattern is still recognizable: one skill per coherent capability, with internal routing between operations.

That does not mean every related action belongs in one skill. Scheduling, drafting, critique, and publishing own different artifacts and stop lines, so they remain separate.

The resource mattered more than the verb

You do not need Cursor to recognize the shape. A REST API does not usually become a separate service for every operation on the same resource.

POST creates. PATCH updates. DELETE removes. Different operations, same resource contract. Reclassification in my system is just another mutation of that same Inbox card resource.

Splitting those operations into separate agent skills was like building one service for create, another for update, and a third for delete. The endpoints looked clean in isolation. Ownership of the resource was fragmented.

Agentic workflows drift toward that decomposition because actions are easier to name than ownership. "Create card" and "enrich card" are vivid verbs. "Own Inbox normalization throughout the Inbox lifecycle" is accurate but abstract. The verbs made the skills easy to name. The resource revealed where the boundary actually belonged.

Your domains may decompose differently. The useful question is not "how many skills do I have?" but "what object does this capability own, and are these verbs operations on that object or different capabilities entirely?"

Takeaway: Start with one skill per action if that helps you ship. When multiple operations share an owned object, lifecycle, and compatible safety boundaries, consolidate them into one capability module and route internally. The risk was no longer a skill becoming too broad. It was one capability having multiple competing owners.

If you'd like to see the project behind these workflow experiments, try Codenames AI.

I expected pair programming with a Cloud Agent. I got a new hire.

Michael Truong — Fri, 31 Jul 2026 05:53:25 +0000

I thought a cloud coding agent was still pair programming: the same local conversation, just running somewhere else.

The first thing that surprised me was not the code. It was how little the run needed me.

My evidence is a first Cursor Cloud Agent run against a real monorepo. What initially looked like a Cursor feature turned out to be a different execution model. The run started over in a fresh environment, did the onboarding work, and left proof. Useful. Autonomous. And missing almost everything I already knew in the local session.

The wrong picture

In local Cursor I already had a planning thread: investigation, tradeoffs, and tools I had already authenticated in that session (including Notion MCP). When I pointed a Cloud Agent at Codenames AI, an npm-workspaces monorepo with an Express API, a Vite frontend, Playwright E2E, and CI, I expected continuity. Same decisions. Same auth. Same half-finished reasoning, just remote.

That assumption failed in the first hour.

What the first run actually did

The task itself was small: make the repository ready for future Cloud Agent runs and document the non-obvious setup in AGENTS.md. Completing it required establishing and proving the whole execution environment.

After that prompt, the agent worked without me sitting in the loop. It cloned the repo, installed dependencies, ran lint, typecheck, build, and tests, installed Playwright browsers, ran the E2E suite, played a Solo turn in the UI (a single-player practice game), captured screenshots and a walkthrough video, and opened a pull request.

That pull request added the AGENTS.md notes and recorded the verification trail from the clean VM, including a hello-world Solo turn in the UI. The change set was small. The behavior was large: autonomous environment setup plus artifacts local agent workflows rarely leave behind.

Two failures I did not expect

Local reasoning stayed on the laptop. Choices I had already made in Cursor (what mattered, what to skip, how I was framing the job) were not present in that new cloud task. Anything that depended on that judgment had to be re-established. The agent could reach the repo and CI. It could not inherit the argument I had already had with myself. Other Cursor flows can move a conversation into the cloud. This fresh task did not arrive with the local framing behind it.

Tool auth did not ride along. Notion MCP worked only after separate authentication for the cloud run. Local Cursor access was not session continuity. "The cloud can use MCP" and "the cloud already has my MCP sessions" turned out to be different claims.

Why it felt like hiring

Async handoff fit. Open-ended design debate in the cloud did not. The run wanted a bounded job and a definition of done, not a remote pair for figuring out the product.

You do not onboard someone by forwarding a half-finished Slack thread. You give them a machine, a checklist, and a brief. Missing context hurts more as autonomy increases, because a new clean-environment job starts without your local reasoning.

AGENTS.md mattered once the agent was in an unfamiliar checkout: non-obvious Node version floors, optional .env for local runs, E2E setup, font-sensitive visual snapshots. You do not need the file itself. The point is that this is briefing material for an execution worker, not a substitute for the planning conversation that happened on my laptop.

Why not another local worktree?

A second local git worktree can also provide parallel execution, and it has one obvious advantage: it can reuse the tools, credentials, caches, and machine state I already have. For a single developer, that can make the handoff cheaper.

But it leaves me managing another local workspace, and it keeps the result coupled to my machine: extra checkouts, local processes, port conflicts, and the risk that the result only works in my environment. A teammate cannot reproduce that run exactly without inheriting my laptop state.

A cloud agent starts with less inheritance, so the handoff matters more. In return, I get an isolated task that is easy to parallelize, with less "works on my machine" risk, and I can start or monitor work remotely without babysitting another local workspace.

Cursor was where I encountered the boundary clearly. Clean environments, onboarding, and missing laptop state are not new lessons if you have been shipping to remote servers for years. What was new was seeing that old systems idea reappear inside an AI coding workflow. The useful response is an explicit handoff, not a remote continuation of your session. Wherever coding agents become independently executable, I expect the same trade-off to appear. The boundary is architectural.

Where each side fits

Local workspace
(planning, judgment, brief)
        ↓
Cloud agent
(bounded execution in a clean environment)

That first run left open questions: whether later runs would reuse the onboarding work, and how the pattern would hold beyond one carefully bounded task.

Looking back, I kept choosing Cloud Agents for bounded execution: maintenance loops (including dependency upgrades), targeted bug fixes, and focused investigations such as tracing a review finding or validating a specific question. Not because the first-run costs disappeared, but because isolation and the shared task model reduced coordination overhead. The run still leaves a reviewable trail (screenshots, video, a PR), the same kind of demo evidence you would expect from another engineer handing work back.

The lasting surprise was that the clean boundary is not only a limitation. It is also the feature that makes the model scale.

Takeaway: A cloud agent gives up some inherited local context in exchange for isolation and scalable delegation. Brief it like a new hire, not like a remote continuation of your existing session. The context it needs must cross that boundary deliberately.

If you'd like to see the project behind these workflow experiments, try Codenames AI.

Active players looked real until we asked which sessions counted

Michael Truong — Thu, 23 Jul 2026 04:28:12 +0000

I've been building Codenames AI, a small web game where an LLM plays Codenames with you. Like most solo products, I glance at a Product Health dashboard when I want a quick read on whether anyone is actually playing.

One morning in June, three weeks after launching the site, the Active players tile said 64. Next to it sat 122 starts and restores. The number looked like traction. My first instinct was to treat it as confirmation and keep shipping.

That instinct did not survive the next question: which sessions were actually in that count?

The dashboard answered a wider question than I asked

I was reading Product Health as if every event in the project came from real players on the production site. The tile did not lie about its math. It counted distinct people who started or restored a game. What it could not tell me, from the chart alone, was which runtime those people were in.

I had reasons to trust the number:

PostHog init only ran when VITE_POSTHOG_KEY was set. Local Vite and Playwright runs did not ship that key, so I treated laptop and E2E traffic as silent by configuration. There was no analytics_environment property yet, and no environment-conditional init path. "Do not put the key in this build" was one guardrail.
Returning users looked safe too. On production, game state restores from origin-scoped localStorage, and PostHog keeps an anonymous ID on that same origin. Come back later and you still count as one Active player via game_restored. We do not call identify; continuity is browser storage on that host. I assumed testing on review URLs worked the same way: me again, already counted.

Outside PostHog, the acquisition picture did not match. Real arrivals were mostly organic Google Search. In Search Console, we had not yet hit the first “30 clicks from Google Search in the past 28 days” milestone. We had only just started posting on dev.to, so that channel was not a material source either.

Sixty-four unique players on a site that young, against a search funnel that had not cleared thirty clicks in a month, and early publishing that barely existed, was already a little suspicious. The starts/restores volume next to it made it worse. My working note was blunt: investigate further; something was minting unique players that real arrivals could not explain.

Review deploys were the hole

Review deploys (for us, Vercel preview URLs) look like the real app, often share the same analytics project key, and show up whenever you click a pull-request review link. They were not "local without a key," and they were not the same origin as codenames-ai.com.

A review hostname gets its own empty save store and its own anonymous PostHog identity, so a click-through during review can land as a new unique player (game_started) instead of folding into the production self I already knew. Without a way to separate those runtimes, that 64 was still a hypothesis about whether review-deploy traffic, and new identities on those hosts, were in the count.

That investigation became a concrete plan: stop treating every capture in the project as if it were production traffic.

An early cut disabled PostHog for E2E. Silencing one runtime would still leave review deploys sharing the key; we needed an explicit boundary instead of relying on some environments staying silent.

What should count as production?

Two fixes landed together.

Client tagging. On PostHog init, the frontend resolves an analytics_environment of production, preview, local, or e2e, then attaches it to every event and to the user profile. Hostname and the host’s build-time environment distinguish the runtimes.

Non-production traffic is excluded by dashboard filters, not by skipping PostHog init. Tagging every runtime, including ones we used to silence by omitting the key, is what makes the filter meaningful.

Dashboard filters. Product Health keeps events where analytics_environment = production OR not set, so older production events from before tagging remain visible. Newer views can use an exact production filter once tagging coverage is trusted.

The missing dimension wasn't another metric. It was the production boundary. Once that existed, Product Health could filter on it.

Healthy numbers resist questions

The harder lesson wasn't that the dashboard was wrong. It was that healthy-looking numbers are the least likely ones to get questioned.

While working on model experiments, failure exposed hidden assumptions. Here nothing looked broken, so curiosity had to do the same job: notice that the system was faithfully answering a different question than the one I thought I was asking.

How we ask the dashboard questions is a separate story. This post stays on the quieter failure mode: one project key, a review runtime that looked like production, and a number that looked clean until we asked.

What I'd check on the next dashboard

Start by asking what question the metric actually answers, not the one you hope it answers.
Compare it against an independent signal. If the numbers do not fit together, investigate before celebrating.
Look for missing dimensions that collapse different kinds of traffic into one KPI: environment, internal users, bots, staging, or another hidden segment.
Only then decide whether the fix is better tagging, better filtering, or a different metric altogether.

You do not need our dashboards or our app code to apply the pattern. Review deploys were the incident that exposed the gap here.

I cannot put a clean contamination percentage, from today’s data alone, on the period before we added tagging; the point is the missing question, not a guessed share of noise.

Takeaway: Production engineering isn't just responding to broken signals. It's occasionally distrusting reassuring ones. Metrics answer exactly the question you instrumented, not necessarily the one you think you asked.

If you'd like to see the project that inspired these lessons, you can try Codenames AI.

Model experiments became an architectural stress test

Michael Truong — Fri, 17 Jul 2026 15:41:21 +0000

I've been tuning Codenames AI, a small web game where an LLM plays Codenames with you. Clue generation is tightly constrained: one word, a count, optional intended targets, JSON on the wire, then deterministic validation before anything reaches the board.

As the project started attracting regular players, I wanted to improve the gameplay experience without blowing out costs. Moving one model generation from gpt-4o-mini to gpt-5-mini was my first instinct.

The default reasoning setting made responses an order of magnitude slower for this workload. Minimal reasoning looked like the obvious compromise: newer model, responsive gameplay.

I expected to compare clue quality, latency, and cost while the surrounding prompt, validator, and consumer contracts stayed put.

That last part was wrong.

The experiment stopped behaving like an A/B test

What showed up was structural, and it showed up in places that had been stable for months.

Validation failures started rising. Retries started rising. Entire candidate batches started failing before the game ever saw a clue. The sharpest signal came from a clue-selection path that had run untouched for months, and it hard-failed for the first time. They weren't latency regressions so much as architectural ones.

It is easy to read that as "minimal reasoning made the model worse." More often, the failures were exposing gaps in contracts that had looked fine under the previous model.

What each failure actually invalidated

Eventually every failure traced back to one of three layers:

Prompt contracts ask for exactly count targets and, in batch mode, several distinct candidates.
Deterministic validators reject target/count mismatches and filter invalid candidates before anything downstream runs.
Downstream consumers only see survivors. Empty batches retry with rejection feedback, then fall back if needed.

Those layers share one job: enforce the same invariants. The failures below cut across all three rather than mapping one to one.

Side commentary could kill an otherwise usable turn.

To pick a clue, one strategy (Strange mode) simulates how the AI guesser would respond to each candidate clue, then scores those simulated turns and keeps the best one. I thought those simulations would fail only when the guesses themselves were bad. After the swap, they could also fail because the model attached commentary about other words it had considered, including words that were not even on the board. Because the payload schema included that commentary, the validator had to treat it as part of the same all-or-nothing contract. A payload with usable guesses still got rejected, and when every candidate died that way, the turn came back as a controlled API failure instead of a clue.

Target cardinality had to match the clue count.

I thought my validator was protecting the game. Instead I discovered the previous model had been consistently producing outputs that satisfied those contracts.

Say the prompt asks for count: 2 and a targets array with exactly two unrevealed friendly codenames. Under the old model, a clue like {"word": "BUILDING", "count": 2, "targets": ["TOWER", "CASTLE"]} usually meant two real board words. After the swap, I started seeing the same shape with one valid target and one word that is not on the grid at all, or only a single target when count was 2. Valid JSON. Perfect keys. Intent status: invalid.

The validator rejects clues whose validated targets don't match count. Valid JSON wasn't enough.

Retries assumed the contracts were already specific enough.

I thought retries were simply robustness. Instead they became diagnostic tooling because they finally told me which invariant had actually failed. When a batch fails validation, the retry path can attach rejection feedback (failed clue words plus reason strings) so the next attempt is not a blind redo. That only helps if the contracts are specific enough to name the failure. Vague "try again" prompts hide whether you have a model problem or an underspecified invariant.

The failures showed up in the product, not just the logs.

Every rejected clue meant another retry before the player saw a move. On an AI spymaster turn, the game shows a clue, a count, and highlights the board words that clue is meant to cover. When the validated targets came back shorter than count, the UI looked broken: count: 2 with only one word highlighted. The AI guesser still trusted the clue count and started reasoning from a board state that never actually existed.

None of this required a different product thesis from schema-first validation. Valid JSON was never enough. The migration stress-tested whether prompt text, deterministic checks, and consumer assumptions still agreed after the model changed.

The uncomfortable part

On paper, the clue path already looked responsible. Prompt, validator, consumer. Clean separation.

The migration revealed a hidden layer:

Prompt
  ↓
Model capability
  (compensating for weak contracts)
  ↓
Validator
  ↓
Consumer

I expected to compare models. Instead I ended up comparing how much of my architecture each model had been compensating for.

While a more capable model kept quietly covering those weak contracts, the dashboards looked fine. Drop reasoning effort, and the same prompts start producing outputs that are honest about what you actually specified. Once that stopped happening, I was no longer measuring model quality. I was measuring how much of the gameplay experience had been resting on those hidden assumptions.

That is uncomfortable and useful. Apparent regressions (count mismatches, partial batches, more retries, collapsed guess simulations) are a signal to ask which layer was doing the work: the model, or the application.

Subjective "does this clue feel clever?" still matters for gameplay. It should not be the only scoreboard when the pipeline can reject an entire batch before the server ever picks a clue.

Treat migrations as compatibility tests

What I want out of a model swap now:

Align invariants across prompt, validator, and consumer. If the prompt says "exactly count targets," the validator must reject mismatches, and the API response shape must not pretend invalid intent is OK.
Keep structural correctness in deterministic code. Use the model for association quality. Use pure functions for board membership, cardinality, illegal clue shapes, and survivor lists.
Instrument validation failures by category. First-pass success rate, retry rate, and failure reasons tell you whether you tightened a contract or uncovered a real model gap.
Evaluate end-to-end workflow metrics, not only single-call latency or token price. Retries and fallbacks change the bill and the player experience; measuring only the happy path lies.

Takeaway: A model migration tests the model and the architecture around it. If prompt, validator, and consumer contracts do not enforce the same invariants, stronger models can mask weaknesses in those contracts until a cheaper or more literal model exposes them. The lesson is not really about which LLM you pick. It is about architectural coupling: the model itself had become part of the contract without me noticing.

If you'd like to see the project that inspired these lessons, you can try Codenames AI.

I fixed my AI reviewer. Then I kept solving the wrong problem

Michael Truong — Wed, 08 Jul 2026 05:09:51 +0000

I've been building an AI-assisted editorial pipeline for technical writing. Notion cards become markdown drafts in the repo, pass through review, then sync to dev.to.

Last month I shipped a post about the first big fix to my editor-critique reviewer skill: The AI reviewer scored 23/25 and missed the point. The problem was sequence. A score-first pass treated a polished rubric as the first lens and produced QA feedback when I needed editorial feedback. Reordering the skill so analysis precedes scoring fixed that.

I assumed the next improvements would come from rubric tuning. Longer prompts. Another scoring dimension. Sharper checklists.

That assumption was half right. The rubric still matters. But every useful fix after the baseline shared a different shape.

The pattern I kept missing

After I reordered analysis before scoring, reviewer failures kept arriving from different incidents. A critique that agreed with itself too easily. Drafts that grew every revision without getting shorter. A middle section that felt like a second article.

Each time I reached for the same lever: expand the rubric, add a rule, lengthen the prompt.

Incident 1: When the reviewer needs to argue with itself

editor-critique produced decisive scorecards and prioritized feedback, but the report rarely challenged its own conclusions. A draft could earn Ready to sync with medium items left unexamined.

Score-first review had failed because it judged too early. This failure was different: the primary critique could be thorough and still under-falsified.

The fix was another staged pass. After the primary critique drafts, freeze it. Run adversarial review that assumes the primary assessment is wrong until draft-supported counter-evidence proves otherwise. Then synthesize: change the publication recommendation only when falsification is material.

I added adversarial review, synthesis, and canonical report assembly as new skill steps. A follow-up pass tightened adversarial review with an anchor requirement: every counter-evidence bullet must name the frozen primary claim it challenges. No orphan hypotheticals like "title spoils thesis?" when the primary critique already praised title strategy.

Before:

Editorial read-through
→ Score
→ Critique
→ Post report

After:

Editorial read-through
→ Score
→ Primary critique
→ Adversarial review (frozen inputs)
→ Synthesis
→ Post report

That was the first time staging a different kind of reasoning into its own pass beat rubric expansion. Two more failures would repeat the same shape before I stopped treating it as coincidence.

Incident 2: When critique only adds

Self-falsification helped, but drafts were still growing. Investigation while critiquing Upgrades don't have to be a blind trust exercise showed feedback was consistently additive, but not subtractive. editor-critique found missing framing and evidence boundaries reliably. It did not ask what should be removed when new material arrived.

The result was layered drafts: an opening stacked on another opening, the same four-step investigation loop restated in three sections, a mental-model diagram that walked through event flow the prose had already established in the previous section.

The fix was not "be shorter" in the rubric. It was naming another cognitive job in the read-through: subtractive editing. Every paragraph should continue earning its place. Flag existing redundancy and addition-induced redundancy. Pair expansion recommendations with material that would become redundant if adopted.

A companion technique, single-owner ideas, lists 2–4 core ideas and flags when the same idea appears in multiple sections without new evidence. I codified subtractive editing in the skill file along with a test case that catches additive-only critique regressions and a lightweight subtractive pass in the human revision step.

The primary critique still owns expansion. Subtractive editing is a separate observational pass, not a rewrite engine.

Incident 3: When a section becomes its own article

The last failure pushed past critique mechanics into reader cognition. While critiquing draft variants in my editorial workflow, several middle-body sections were technically correct but felt wrong in context. In one draft, an implementation walkthrough interrupted the investigation arc. In another, a full section on validation tooling read like its own mini-article.

The failure mode was narrow: a section stopped advancing the reader's current question and temporarily made another explanatory thread the center of gravity.

Adding a rubric dimension for "section focus" would have been vague. What worked was an observational lens in the editorial read-through step: name the primary thread, name the secondary thread, decide whether to compress, delay, embed later, or leave as-is.

I codified this as a Secondary explanatory thread lens in the skill file. The rubric stayed the same. It simply added a named cognitive job: track whether prose is serving the reader's current question or drifting into a side article.

What stayed constant

Three incidents, three skill changes, one pattern. Across all three, a few constraints held:

The five-dimension rubric stayed mostly intact.
Read-only governance did not change: critique still does not write repo files or gate publish.
Each pass added another observational lens, not another scoring dimension.
The expensive part was naming the cognitive job precisely enough to operationalize in a skill file.

The recurring mistake was treating undifferentiated reasoning as one pass. Each fix changed the sequence, not the rubric weight. A capable reviewer can read before it scores and still under-read if falsifying primary judgment, displacing redundant prose, and tracking reader focus all compete in the same step.

Before you expand the rubric

List the failure modes that survived your last sequence fix.
For each one, name the cognitive job that failed (self-falsification, subtractive editing, reader-focus tracking).
Stage that job as its own observational pass with a frozen handoff to the next step.
Expand the rubric only if that observational pass still misses failures in production.

Once editor-critique understood before judging, the remaining improvements came from separating kinds of reasoning into distinct stages, not from a bigger rubric or a longer single pass. I suspect the pattern may generalize beyond editorial critique.

Takeaway: When a reviewer skill plateaus after a sequence fix, ask which cognitive jobs are still sharing one undifferentiated pass. Stage them before you expand the rubric.

If you'd like to see the project behind these workflow experiments, try Codenames AI.

Upgrades don't have to be a blind trust exercise

Michael Truong — Fri, 03 Jul 2026 09:02:30 +0000

I've been building Codenames AI as a solo project I want to keep alive. Renovate helps dependency upgrades move without maintenance eventually crushing momentum.

You don't need my exact setup to follow along.

Every project accumulates maintenance work. Framework upgrades are one place that work stalls, not because engineers do not know how to migrate, but because proving what actually needs to change takes time. That tradeoff shows up on a hobby repo as "I'll look at this when I have an uninterrupted evening." It shows up in production as major versions piling up while investigation competes with feature work.

That's the problem I was trying to solve.

Before AI, my realistic choices were narrow: trust the automation and hope, spend hours mapping release notes to my codebase, or leave the upgrade sitting.

I was not trying to invent a better review process.

I was trying to keep maintenance cost below available time.

AI changed the cost of investigation enough that I stopped treating it as something to postpone.

When Renovate opened a pull request for one of those framework majors, upgrading Vite from ^6.0.11 to ^8.0.0 and @vitejs/plugin-react from ^4.3.4 to ^6.0.0, my first instinct was still to treat it like migration work. Read the guides. Find the breaking changes. Plan the code changes. Validate the app. Then merge.

It was also the wrong starting assumption.

The useful work was not implementing the migration. The useful work was proving whether a migration existed for this repo at all.

The review gate was right

Renovate grouped the update as a frontend React/Vite major. My review policy sorts packages into low-risk and high-touch buckets: a patch-level bump to a type definition or a lint plugin can auto-merge, but anything that builds or serves the app (the bundler, the React plugin) is high-touch.

Both Vite and @vitejs/plugin-react sit in that high-touch bucket, so any version change routes to a human instead of auto-merge.

The pull request looked serious on paper:

Vite 8 release notes included explicit breaking changes.
The lockfile churn was large because Vite moved from the Rollup-centered dependency shape toward Rolldown packages.

The old workflow would have started with a migration plan. The evidence-first workflow started with a different goal:

Which documented breaking changes actually apply to this repository?

CI showed the branch built; it did not show the breaking changes were irrelevant to this repository. I ran the PR through an AI-assisted review and used the four-step checklist below to audit the result. My job was not to re-derive every fact by hand; it was to decide whether the evidence was enough to merge.

Investigation before implementation

The checklist covers four increasingly specific questions. For the Vite bump, the resulting evidence packet looked like this:

Inspect the upstream change.

Both packages shipped documented breaking changes. The review used those notes to name possible failure modes, not to assume which ones touched this repo. Vite 8's release notes called out, for example, SSR pipeline shifts and stricter import.meta.hot handling.
Map those changes to actual usage.

The packet reframed the question from "does Vite 8 have breaking changes?" to "does this app use the surfaces those changes break?" This repo has an ordinary Vite React setup with no custom SSR.
Identify custom risk.

The packet flagged one meaningful project-specific area: a small custom Vite plugin, cssBeforeModuleScript, that hooks transformIndexHtml to reorder the stylesheet and module-script tags. A bundler swap from Rollup to Rolldown could plausibly change that behavior, so this remained unresolved until the app was exercised.
Validate the app.

CI's test job had already been green; the packet still called for proof the custom-risk path and production build held. A ready Vercel preview closed that gap: it ran a production build through the new Vite with cssBeforeModuleScript included, and the rendered page was where broken stylesheet or module-script ordering would have shown up.

At that point, the recommendation changed.

The packet had not found migration work. It had found enough evidence that no migration was required.

Evidence over migration plans

The merged PR changed two files: frontend/package.json and package-lock.json.

No source files. No Vite config rewrite. No component changes. No test rewrites. No custom shim.

That is easy to misread as "the upgrade was trivial." It was not trivial. The pull request carried real risk signals. The absence of source changes only became meaningful after the investigation proved they did not require code changes.

Implementation only belongs once the evidence asks for it. The human job shifts from "please migrate this dependency" to auditing whether the packet answers the four steps above.

What this changed about manual review

Before this, "review manually" sounded like a parking lot. A major upgrade arrived, the automation refused to merge it, and the human picked it up later. Assembling the evidence packet by hand was often what stalled the review.

Now I treat "review manually" as an evidence-gathering lane. AI makes that lane practical: much of the packet assembly no longer has to happen in one sitting at your keyboard. That's what changed the economics. Investigation stopped being the expensive part that made maintenance easy to postpone. On the Vite bump, AI helped assemble the evidence; I audited whether it was enough to merge without migration work.

For low-risk patches the question stays simple: did CI pass and did the diff stay inside package files? For high-touch framework upgrades it gets richer: can the packet cover upstream changes, repo usage, custom risk, and real app validation well enough for a human to decide without first doing speculative migration work?

That does not remove judgment or the merge decision. It moves both earlier: gather the packet first, then audit whether implementation is actually required.

What I'd do on the next major upgrade

Classify the package honestly. Runtime and framework packages deserve more evidence than a patch-level dev tool bump.
Run the four-step loop above against release notes, repo usage, custom risk, and real app validation.
Treat zero source changes as a conclusion, not an assumption. If no implementation is required, say what evidence proved that.

Takeaway: When investigation is cheap enough to run, deferral stops being the default. Do the audit before you write the migration plan.

Editor's note (September 2026)

Since publishing this, I extracted the implementation behind this approach into an open-source workflow and shipped it as a Cursor plugin. It includes the governed Renovate merge ladder, portable classifier and maintainer skills, and the evidence-first review path described above.

The implementation is available in the multipliers-dev/renovate-workflow repository on GitHub.

The AI reviewer scored 23/25 and missed the point

Michael Truong — Fri, 26 Jun 2026 15:14:45 +0000

I've been building an AI-assisted editorial pipeline for my technical writing. Notion cards become markdown drafts in the repo, pass through review, then sync to dev.to.

The motivation was simple: I already had a review loop I trusted for code. Open a PR, run Cursor's Bugbot against a review guide, fix what mattered, merge. I wanted the same rhythm for writing: draft, critique, revise, publish. So I built my own AI review skill called editor-critique.

I had also started adding HTML comments inside drafts, much like code comments. They captured the editorial intent behind a section, including why it opened where it did and why evidence sat where it did, without becoming part of the published post.

That made the review step look straightforward. Give the AI a rubric, score the draft, return prioritized feedback.

If the rubric was good, I assumed the critique would be good.

That assumption failed in a very specific way.

The first version of editor-critique did what I asked. It read a draft, applied five scoring dimensions, and produced a polished report. While reviewing my article, "The agent plan had every step except where to stop", it scored the piece 23/25 and mostly suggested polish.

It also missed the feedback I actually needed.

Valid rubric, shallow read

The draft did not need another pass on commas and section labels. It needed a colder editorial read.

A useful reviewer should have asked:

Does the title reveal the lesson before the incident earns it?
Does the article assume private repo context a dev.to reader will not have?
Are links to PRs, plans, and standards supporting evidence, or required reading?
Is governance framing outrunning what the incident actually proved?

Those are reader-journey questions, not formatting checks.

The score-first reviewer treated the rubric as the first lens. If the thesis was present, evidence was named, and the arc looked complete, the draft read as ready. The rubric turned critique into publication preflight: complete sections, reasonable voice, no obvious holes.

Useful, but not enough.

What changed in the sequence

I revised the reviewer skill so analysis precedes scoring.

Before:

Load draft
→ Score rubric dimensions
→ Generate critique

After:

Load draft
→ Editorial read-through
→ Score rubric dimensions
→ Generate critique

The rubric stayed. It stopped being the opening move.

Before scoring, the reviewer now reads visible prose like a cold dev.to audience member. It mentally strips author notes and asks whether the lesson still works if repo links and hidden rationale disappeared. Then it checks thesis timing, audience assumptions, reference framing, and speculation drift.

The annotation loop mattered here. Because the comments sat beside the sections they explained, critique could compare intent against effect: the note described what the section was trying to do, while the reader-facing paragraph showed whether it actually did it. Sometimes the article needed the edit. Sometimes the annotation exposed that editor-critique itself was reading the section too mechanically. Either way, the disagreement became useful training material for the reviewer skill.

Only after that read does it assign scores.

The output became more editorial. Instead of asking only "does this draft satisfy the rubric?", it started asking "what will break for the reader?"

On the same article, the revised reviewer surfaced title spoiling the lesson, private PR assumptions, weak framing for repo artifacts, and governance language potentially ahead of the evidence. The 23/25 pass had treated those as minor or invisible.

Why order beat rubric tuning

A rubric compresses judgment into categories: thesis, structure, evidence, voice, readiness. That compression helps consistency.

Compression too early can hide the problem.

Once the reviewer committed to a numerical assessment, the rest of the report tended to justify that assessment. A 23/25 draft needed 23/25 feedback, so the model organized its reasoning around why the piece was mostly ready instead of independently discovering what a reader would struggle with.

It is a little like running a linter before reading a design doc. The linter can confirm imports and formatting are clean. It cannot tell you whether the design makes sense. Start with the linter and the document can feel more complete than it is.

That is what happened here. The rubric was not bad. It was premature.

Once analysis came first, the same categories became more honest. "Evidence and specificity" could include link-only dependence. "Thesis and opening" could include title spoiling the lesson. "Publish readiness" could include whether prose survives without private repo access.

The score became a summary of the read-through, not a substitute for it.

QA review vs editorial review

The revision made me distinguish two kinds of AI review.

QA review asks: Did the artifact satisfy the stated criteria?

Editorial review asks: What will the reader misunderstand, miss, or not believe?

This was not completely new to me. In code review, I already used different Bugbot guides depending on what I wanted it to optimize for: security, game-state changes, UX regressions, or plan intent. The same diff could be reviewed through different lenses.

Writing turned out to have the same property as code review. A QA reviewer checks completeness and publishing criteria. An editorial reviewer reads for audience confusion and belief. The artifact stayed the same. The review lens changed.

Both matter. Broken frontmatter, missing sections, or absent takeaways still need QA. But if the reviewer starts and ends there, it can produce a confident report that never engages the reader's path through the article.

The first reviewer was not useless. It was doing QA under the name of critique.

The revised reviewer still scores, but it has to earn the score by reading first.

That sequencing shift moved output from "this article is mostly ready" toward "this article assumes too much context, reveals its lesson too early, and needs stronger in-narrative evidence before the governance argument about where an agent should stop lands."

That is the feedback I needed.

What I'd do on the next reviewer

For the next AI reviewer I build, I would design sequence before I tune rubric dimensions.

Start with an ungated read. Inspect audience, intent, risk, and evidence before scoring thresholds appear.
Make the rubric summarize the analysis. Scores should cite read-through observations, not invent them after the fact.
Separate checklist pass from judgment pass. "Is it complete?" and "is it good?" are different questions.
Force reader-impact language. Critique items should say what breaks for the reader, not only which rule was violated.
Let scores come last. Once a number appears, everything organizes around it.

This is not only about writing. I suspect the same pattern may apply to PR review, architecture review, incident analysis, and evaluation reports: if a reviewer scores before it understands, it overfits to the rubric and under-reads the situation.

The shape feels portable. Evaluation criteria are not enough. The order in which a reviewer thinks changes what it notices.

Takeaway: If your AI reviewer keeps producing technically correct but shallow feedback, do not only rewrite the rubric. Move analysis before scoring.

Editor's note (July 2026)

This article documents the first major architectural change to editor-critique: separating analysis from scoring. That sequence change held up, but it also exposed a new class of reviewer failures that couldn't be solved through rubric expansion alone. The follow-up, I fixed my AI reviewer. Then I kept solving the wrong problem, explores that next stage.

If you'd like to see the project behind these workflow experiments, try Codenames AI.