I've been building an AI-assisted editorial pipeline for technical writing. Notion cards become markdown drafts in the repo, pass through review, then sync to dev.to.
Last month I shipped a post about the first big fix to my editor-critique reviewer skill: The AI reviewer scored 23/25 and missed the point. The problem was sequence. A score-first pass treated a polished rubric as the first lens and produced QA feedback when I needed editorial feedback. Reordering the skill so analysis precedes scoring fixed that.
I assumed the next improvements would come from rubric tuning. Longer prompts. Another scoring dimension. Sharper checklists.
That assumption was half right. The rubric still matters. But every useful fix after the baseline shared a different shape.
The pattern I kept missing
After I reordered analysis before scoring, reviewer failures kept arriving from different incidents. A critique that agreed with itself too easily. Drafts that grew every revision without getting shorter. A middle section that felt like a second article.
Each time I reached for the same lever: expand the rubric, add a rule, lengthen the prompt.
Incident 1: When the reviewer needs to argue with itself
editor-critique produced decisive scorecards and prioritized feedback, but the report rarely challenged its own conclusions. A draft could earn Ready to sync with medium items left unexamined.
Score-first review had failed because it judged too early. This failure was different: the primary critique could be thorough and still under-falsified.
The fix was another staged pass. After the primary critique drafts, freeze it. Run adversarial review that assumes the primary assessment is wrong until draft-supported counter-evidence proves otherwise. Then synthesize: change the publication recommendation only when falsification is material.
I added adversarial review, synthesis, and canonical report assembly as new skill steps. A follow-up pass tightened adversarial review with an anchor requirement: every counter-evidence bullet must name the frozen primary claim it challenges. No orphan hypotheticals like "title spoils thesis?" when the primary critique already praised title strategy.
Before:
Editorial read-through
→ Score
→ Critique
→ Post report
After:
Editorial read-through
→ Score
→ Primary critique
→ Adversarial review (frozen inputs)
→ Synthesis
→ Post report
That was the first time staging a different kind of reasoning into its own pass beat rubric expansion. Two more failures would repeat the same shape before I stopped treating it as coincidence.
Incident 2: When critique only adds
Self-falsification helped, but drafts were still growing. Investigation while critiquing Upgrades don't have to be a blind trust exercise showed feedback was consistently additive, but not subtractive. editor-critique found missing framing and evidence boundaries reliably. It did not ask what should be removed when new material arrived.
The result was layered drafts: an opening stacked on another opening, the same four-step investigation loop restated in three sections, a mental-model diagram that walked through event flow the prose had already established in the previous section.
The fix was not "be shorter" in the rubric. It was naming another cognitive job in the read-through: subtractive editing. Every paragraph should continue earning its place. Flag existing redundancy and addition-induced redundancy. Pair expansion recommendations with material that would become redundant if adopted.
A companion technique, single-owner ideas, lists 2–4 core ideas and flags when the same idea appears in multiple sections without new evidence. I codified subtractive editing in the skill file along with a test case that catches additive-only critique regressions and a lightweight subtractive pass in the human revision step.
The primary critique still owns expansion. Subtractive editing is a separate observational pass, not a rewrite engine.
Incident 3: When a section becomes its own article
The last failure pushed past critique mechanics into reader cognition. While critiquing draft variants in my editorial workflow, several middle-body sections were technically correct but felt wrong in context. In one draft, an implementation walkthrough interrupted the investigation arc. In another, a full section on validation tooling read like its own mini-article.
The failure mode was narrow: a section stopped advancing the reader's current question and temporarily made another explanatory thread the center of gravity.
Adding a rubric dimension for "section focus" would have been vague. What worked was an observational lens in the editorial read-through step: name the primary thread, name the secondary thread, decide whether to compress, delay, embed later, or leave as-is.
I codified this as a Secondary explanatory thread lens in the skill file. The rubric stayed the same. It simply added a named cognitive job: track whether prose is serving the reader's current question or drifting into a side article.
What stayed constant
Three incidents, three skill changes, one pattern. Across all three, a few constraints held:
- The five-dimension rubric stayed mostly intact.
- Read-only governance did not change: critique still does not write repo files or gate publish.
- Each pass added another observational lens, not another scoring dimension.
- The expensive part was naming the cognitive job precisely enough to operationalize in a skill file.
The recurring mistake was treating undifferentiated reasoning as one pass. Each fix changed the sequence, not the rubric weight. A capable reviewer can read before it scores and still under-read if falsifying primary judgment, displacing redundant prose, and tracking reader focus all compete in the same step.
Before you expand the rubric
- List the failure modes that survived your last sequence fix.
- For each one, name the cognitive job that failed (self-falsification, subtractive editing, reader-focus tracking).
- Stage that job as its own observational pass with a frozen handoff to the next step.
- Expand the rubric only if that observational pass still misses failures in production.
Once editor-critique understood before judging, the remaining improvements came from separating kinds of reasoning into distinct stages, not from a bigger rubric or a longer single pass. I suspect the pattern may generalize beyond editorial critique.
Takeaway: When a reviewer skill plateaus after a sequence fix, ask which cognitive jobs are still sharing one undifferentiated pass. Stage them before you expand the rubric.
If you'd like to see the project behind these workflow experiments, try Codenames AI.
Top comments (30)
This hits close to home. I went through something very similar with my series — kept reaching for the "sharper rubric" lever until I realized the problem wasn't the rubric. The "naming the cognitive job precisely enough" part is exactly where the leverage is. Appreciate you writing this up — validating. 👀
That's exactly the shift I was hoping to capture. I kept treating each failure as a missing rubric rule until I realized they were different kinds of reasoning competing in the same pass. Once the cognitive job had a clear owner, the fixes became much more obvious.
Different kinds of reasoning competing in the same pass" — that's a really clean way to put it. I found that once I gave each "kind of reasoning" its own seat at the table (and told them they're allowed to disagree), the quality bar moved way more than any single pass ever did. Have you experimented with having them disagree on purpose, or do you keep them in sequence?
I like the "each kind of reasoning gets its own seat" analogy.
Short answer: sequential, with one deliberate disagreement step.
Inside a single run, the passes don't argue live. They hand off frozen artifacts. The adversarial pass treats the primary critique like another editor wrote it and tries to falsify it with draft-supported counter-evidence. The burden of proof stays on the challenger, and an empty adversarial output is a perfectly valid outcome. Synthesis only changes the recommendation if the falsification is material.
There is still a feedback loop, but it's outside the skill. I review the output, revise the draft, then run the skill again. While the workflow is still evolving, I'd rather inspect each iteration before kicking off the next one than let reviewers debate each other without oversight.
A parallel-reviewer architecture with a synthesis step would be another interesting direction if there are ever too many seats at the table, but sequencing has been enough for what I've observed so far.
"Frozen artifacts, burden of proof on the challenger, empty adversarial output is valid" — that's a really clean discipline. I've been experimenting with something similar: the adversarial pass in my pipeline also works on static artifacts (the draft + the primary review notes), and it has two gears — "attack what's written" and "find what's missing." The missing-finding pass consistently produces more useful output than the attack pass, probably because the attack pass gets circular on well-defended positions. Have you observed the same?
Yes, I've seen something similar. I think it's mostly about where each cognitive job lives.
In my pipeline, the gap-finding work consistently produces more useful output too, but it's decomposed across multiple stages. An early refresh step catches freshness gaps. Context retrieval catches evidence gaps. Drafting shapes that evidence into a first narrative. The editorial read-through catches cold-reader gaps. The primary critique catches editorial judgment gaps before the adversarial pass ever runs.
That leaves the adversarial pass much narrower. It attacks frozen primary claims with draft-supported counter-evidence rather than looking for new gaps. If it's consistently finding lots of new gaps, I'd probably treat that as a sign that an earlier stage needs another cognitive job rather than making the adversarial pass broader.
Same shape, different domain — I've been writing something similar for the series. Two independent audit paths, one catches selection bias (cherry-picked data), the other catches silent drift (changed underfoot). Neither alone is enough. The cross-check is where the signal lives.
What you said about narrowing the adversarial pass by owning gap classes earlier — that's the same instinct. By the time it runs, it should almost be redundant. Which is the point.
Question: have you tried making it only verify, never discover? Give it the claims + evidence, tell it "pass or fail" — no hunting. Curious what happens to the output.
That's a good question. Looking at it again, I think we're actually a bit closer than I first thought.
The adversarial pass is already constrained to verifying the frozen primary critique. It can only challenge claims the primary reviewer actually made, and every challenge has to be backed by draft evidence.
The binary framing just lives one level later. Synthesis asks, "Did any of those challenges materially change the recommendation?" (
Recommendation revised: yes/no). So it's effectively deciding whether the adversarial findings actually matter, rather than passing or failing each individual claim.Your version pushes that binary decision down a level and verifies each claim independently. Mine still lets the adversarial pass identify which frozen claims can be falsified before synthesis decides whether any of those falsifications materially change the recommendation.
Good point — we're closer than it looked at first glance.
The synthesis-level binary wasn't a deliberate design choice actually, it came from a pattern I kept hitting: sometimes a single challenged claim looks survivable, but three related ones together change the recommendation. Per-claim pass/fail would miss that aggregate signal.
Still figuring out whether hiding the overruled challenges from the reader is a feature or a bug. Cleaner output, but the reader never sees the tension.
Anyway — thanks for this😄. Talking through this while I've been going through audits of the next series posts has been genuinely helpful. The review process already feels sharper. You've definitely had a hand in the next installments coming out faster. Great discussion.
Really like the anchor requirement here — that's the right instinct for traceability. One thing I keep noticing across posts like this: score, primary critique, adversarial review, synthesis — every pass is still the same model checking itself, just staged differently. When synthesis finally says "Ready to sync," does anything outside that pipeline verify the claim holds (a human spot-check, a different tool, a deterministic test)? Or does "Ready to sync" stay a self-report the whole way through?
Good question. The adversarial review and synthesis act as the verifier. The primary critique isn't trusted as final; it's treated as a hypothesis that has to survive a structured falsification attempt before synthesis can recommend "Ready to sync."
That's still the same model in one staged run, so I wouldn't call it independent verification. But I also don't think independent verification is the goal here, otherwise you immediately end up asking who verifies the verifier.
What the workflow is trying to do is separate the cognitive jobs: make a claim, try to falsify it, then reconcile the original critique with the counter-evidence before producing a recommendation.
Outside the pipeline, that recommendation is still advisory rather than an automatic publish gate. I review the markdown and preview before deciding whether to publish, but the structured critique makes that recommendation more trustworthy than a single-pass review.
Agreed on the cognitive-job separation — that part earns its keep. A structured falsification pass catches a different class of error than single-pass review, and it makes sloppy claims more expensive to produce. No argument there.
Where I'd push back gently is the regress framing. "Who verifies the verifier" only regresses if every verifier is another judgment. The regress ends at the first link that is a measurement rather than an opinion: a deterministic test, an execution readback, an artifact the claiming process can't rewrite. You don't need an infinite tower of verifiers — you need one anchor of a different kind than the thing being verified.
Concrete version from our own operations: one of our agents reported a native Windows launch as working. Every staged self-review passed, because every review stage consumed the same claim. One live run on the actual OS returned spawn ENOENT. The staged critique had raised the cost of a sloppy claim; the single cheap probe is what actually bounded it. Our rule since then: any claim that touches an external actuator (OS, network, subprocess) gets one real execution before it counts, no matter how many review passes it survived.
Which is why I think your preview step is doing more load-bearing work than "advisory" suggests — it's your measurement link. The failure mode worth watching: as the structured critique gets more trustworthy, the preview quietly becomes a skim. One thing that's worked for us is having the pipeline emit one falsifiable check a human can do in ten seconds (an exact diff stat, one anchor to click) instead of asking the human to re-review the whole artifact. It keeps the outside link real without redoing the pipeline's work.
That's a really useful distinction, and I think it clarifies why my workflow is split where it is.
In my earlier reply I was talking specifically about
editor-critique, which focuses on editorial judgments like thesis, structure, clarity, and evidence quality. Those claims are subjective, so the verifier there is another structured judgment, not a deterministic measurement.My next stage,
editor-publish, is deliberately different. It syncs the markdown, validates the publication payload, and produces a dev.to preview. That's where publication-related claims get tested: does it render correctly, do links resolve, and does the published result match what I wrote? I'm still reviewing the preview myself, but I'm observing an external artifact rather than re-arguing the draft.I hadn't articulated it this way before, but the boundary seems to be: judgment should challenge judgment. Measurement should verify claims about reality. I originally split the stages for practical reasons. Your comment helped me see the architectural principle behind it. Thanks for pushing on it.
"Judgment should challenge judgment. Measurement should verify claims about reality" — that's a cleaner formulation than the one we'd been using internally. I'll probably steal it.
One addition from operating experience: the boundary between the two is dynamic, not static. Claims that look like pure judgment often contain measurable sub-claims you can extract and push across the line. "Evidence quality" reads as editorial judgment, but it partially decomposes: the citation exists, the link resolves, the code sample runs, the quoted number matches the source. What's left after extraction is the genuinely subjective residue — and that residue is what judgment-vs-judgment is actually good at. Treating the extraction as an explicit step (settle the measurable parts first, argue only about the remainder) made our review arguments shorter, because the falsifiable parts were already closed by cheaper means.
There's external evidence for the narrower version of that split: a June paper on false success in LLM agents (arxiv.org/abs/2606.09863) found that a lightweight TF-IDF text detector recovered 4-8x more false-success cases than the best LLM judge at the same flag rate. That study tested a domain-calibrated text detector—not file, HTTP, or publish-state checks—so it doesn't validate your publish gate directly. But it does support keeping a generative judge from being the only monitor for completion claims. Your publish check is the operational extension: measure the state change where a direct state check is available.
Good exchange — your two stages made the principle concrete in a way abstract verification arguments usually don't.
That's a useful refinement. I hadn't thought about decomposing judgment-shaped claims quite that way.
I technically do a softer version of this already, but it sits upstream of drafting, not inside
editor-critique.editor-contextis a step that gathers the source material the draft is written from: card text, related cards, prior posts, and, when the card references shipped work, merged PRs and commits. It also classifies what's missing as blocking or non-blocking. Drafting won't proceed on blocking gaps.So it's not quite "ask the reviewer to reconstruct evidence afterwards." More like: establish the evidence first, then write from it.
The gap, as I see it, is that I don't yet treat decomposition as an explicit step.
editor-contextretrieves and curates, but it doesn't systematically settle measurable sub-claims inside a label like "evidence quality": does the citation exist, does the link resolve, does the quoted number match the source, before editorial judgment starts. Right now that's implicit in retrieval quality rather than a first-class output.Your framing suggests a cleaner evolution path: make that decomposition explicit in
editor-context, settle the falsifiable parts with cheaper checks, and only pass the subjective residue to drafting andeditor-critique. I already do a version of this at capture time for repo-anchored cards, andeditor-publishdoes it at the other end for publication state. The middle is where that decomposition could become explicit.I'm not sure I'd tighten it into a hard gate today because some cards are legitimately idea-first. But I like the direction. It gives me a way to evolve
editor-contextwithout changing what the later stages are responsible for."Some cards are legitimately idea-first" is, I think, the reason the explicit step shouldn't be a gate at all — it can be a label. When we made extraction explicit, the useful output wasn't pass/fail; it was that each claim leaves the step marked one of three ways: settled by a cheap check, measurable but unsettled, or genuinely subjective. Nothing blocks. An idea-first card just arrives at critique with "measurable-but-unsettled: 4" showing, and the critique stage knows which four arguments it should refuse to have — they're checkable, not arguable. The hard-gate question dissolves: you keep gating only where you already gate (publish state), and the middle stages inherit visibility instead of enforcement.
Two second-order effects, one observed and one still under test. Observed: the "measurable but unsettled" bucket is the cheapest audit trail we have. When something later turns out wrong, the first question is whether it sat in that bucket at draft time — "we knew it was checkable and didn't check" is a different failure from "we couldn't have known," and the label is what makes those two distinguishable after the fact. Under test: the ratio between the three buckets as a health signal. Our suspicion is that when "genuinely subjective" stays large for a claim family, it means extraction is failing rather than the claims being special — but we don't have enough history yet to say that firmly.
The cost caveat, since you mentioned not tightening it today: we don't decompose everything. Settling sub-claims has a latency price, so extraction only runs on claims the draft actually leans on — the ones that would be cited, quoted, or load-bearing in the argument. Peripheral texture stays unexamined, deliberately. That's the same blocking vs non-blocking call your editor-context already makes, applied one level down.
Staging each new kind of reasoning into its own pass instead of growing the rubric is the right instinct, since a longer checklist just gives the same biased reviewer more to agree with itself about. The adversarial pass that freezes the primary critique and forces every counter-point to name the claim it attacks is a clean way to kill orphan hypotheticals. I would be curious whether you score the two passes separately, since a judge that flips under adversarial framing is telling you something about the draft and about the judge.
Great question. The different passes aren't scored separately. Instead, the thing I'd measure independently is whether adversarial review causes synthesis to overturn the original publication recommendation. Giving the adversarial pass its own scorecard would turn it into a second editorial review, not a falsification pass.
I do like your point though. Tracking those recommendation overturns feels like a good feedback loop for evolving both the reviewer and the workflow over time.
The anchor requirement is the detail I'd steal from this. Without it, an adversarial pass drifts into vague skepticism that sounds rigorous but never lands on anything. One thing I'm curious about though. Once the pass is required to challenge frozen claims, does it ever manufacture weak objections just to have output? A reviewer told to argue can always find something to argue with, so I wonder how the synthesis step tells a material challenge from a dutiful one. Did "material" end up as concrete criteria in the skill file, or is it still a judgment call?
This is exactly the failure mode I was worried about when I added the pass.
The adversarial step isn't told to "find something to argue with." It's allowed to conclude that no convincing contradictory evidence exists. Every objection also has to anchor to a frozen claim from the primary review, otherwise it's dropped. Weak or speculative contradictions are supposed to be omitted, not listed for completeness.
"Material" itself is still a judgment call. The skill doesn't try to classify objections as material or not. Instead, synthesis asks a narrower question: did the adversarial review demonstrate a material misread or overlooked evidence that would reasonably change the publication recommendation? If not, the recommendation stays the same. The burden of proof is on the adversarial pass, even if it raises smaller points.
So dutiful arguing can still show up in the adversarial section; it just shouldn't flip the recommendation unless it changes that synthesis decision.
There's still judgment at the margin because "would reasonably change the publication recommendation" isn't really automatable. But it's anchored to a single decision artifact rather than open-ended contrarianism. That was the design bet.
That is a very familiar trap. It is easy to make the AI reviewer sound smarter while the actual bottleneck is still workflow design: what needs review, what needs deterministic checks, what needs human judgment, and what should never enter the queue.
I think that's where I ended up too. I started out trying to make the reviewer smarter, but most of the improvements ended up being about deciding where each cognitive job belonged instead.
Once those boundaries were clearer, the reviewer had a much narrower job to do. Looking back, that's probably why the story kept drifting from "smarter reviewer" toward "better workflow around it." The reviewer fixes were real; they just weren't the whole bottleneck.
That drift is familiar and usually healthy. Once the reviewer has a narrower job, the rest of the system can carry the deterministic proof, routing, and escalation. The reviewer then becomes one signal in the workflow instead of pretending to be the whole workflow.
This perfectly captures a trap I see even outside code reviews — in financial modeling and data tools, we do the same thing. We optimize the metric we can measure (output scores, pass rates, calculation accuracy) instead of stepping back to ask whether we're solving the right problem for the user. The insight about naming the cognitive job precisely is gold. Once you articulate what kind of thinking a step requires, the right tool or person for that step becomes obvious. Great writeup Michael!
I think that's exactly the broader lesson I stumbled into. Once I started naming the cognitive job each step was responsible for, the choice of tool became much more obvious.
Looking back, I was trying to optimize one oversized reviewer instead of asking whether it was doing several different kinds of thinking that should have been separated in the first place.
It's been interesting seeing people recognize the same pattern outside AI workflows. It's really a workflow decomposition problem that happens to show up in AI systems.
This is a common trap with AI workflows: the reviewer improves, but the workflow still optimizes the wrong bottleneck. Sometimes the fix is not a smarter judge, but a clearer source of truth, smaller work units, or a better definition of what should never reach review.
This is a useful lesson for any AI reviewer. Improving the bot is not always the same as improving the workflow. Sometimes the problem is upstream: unclear standards, weak source material, no decision owner, or too many review moments. A smarter reviewer can still amplify a broken process.
The anchor requirement for adversarial review is stronger than it may look at first glance. Requiring each counter-evidence bullet effectively creates traceability. That gives you something close to a lightweight argument graph: claim, challenge, synthesis decision. If you ever want to evaluate this systematically, that traceability could support metrics like challenge coverage, unsupported challenge rate, and recommendation-change rate.
Also, I'd advise to add different roles for the "reviewer", and even creating a "reviewer panel" - one agent for critique, one for defense, optionally adding moderator. As a bonus, this way you can solve much wider problems range than factually and stylistically correct article.
Yurii, sharp read, especially on traceability.
The anchor requirement was added for exactly that reason. Before it, adversarial bullets could drift into orphan hypotheticals even when the primary critique had already praised the same point. Now every counter-evidence bullet has to name the frozen primary claim it challenges. That creates something close to the lightweight argument graph you described: claim → challenge → synthesis.
The synthesis step makes that third node explicit. The publication recommendation only changes when adversarial review demonstrates a material misread, with the burden of proof on the falsification pass. If nothing convincing emerges, it simply says so.
On the reviewer panel: today it's intentionally one reviewer run with explicit role-switching rather than separate agents. The adversarial pass treats the primary critique as if another editor wrote it, but it's still the same session after the primary output is frozen. Those passes are tightly coupled. The output of one becomes the input to the next, so splitting them into separate agents today would mostly add coordination overhead.
A panel could broaden the problem space, but it also introduces coordination cost and handoff drift. So far I've found bigger gains from improving the evidence flow between stages than from adding more reviewers. I'd expect decomposition to make more sense once there are genuinely independent reviewers producing parallel critiques that need to be synthesized, rather than sequential passes over the same review.
On metrics: the structure is already there for challenge coverage and recommendation-change rate, but I haven't instrumented them yet. Verification today is still manual checklists and regression runs on real drafts. It's something I'd like to revisit once I have a larger corpus of production critiques to measure against.