The buried "Minor Adjustments" clause is the right diagnosis: a 600-word sign-off is not evidence that the ticket is done.
The article already flags the hole I would treat as the main one: told to write Skipped: None, the agent sometimes will. A labeled block is still a claim the model authored. I wouldn't let that claim be the record the reviewer trusts.
I'd have the host emit the Completed / Skipped table from the acceptance criteria and from whatever the run actually recorded — the diff, the named commands, the test results — so "Skipped: None" is not a sentence the agent is allowed to type into the gate. The agent's prose can annotate the table. It shouldn't be allowed to certify it.
A second read-only pass is useful, but only if that pass also isn't grading a writeup it wrote itself. The end state is whatever the recorded checks show, not the cheerful sign-off.
I'd question using summary length to decide that a task was too broad. A one-line configuration change can alter tenant isolation, while a long mechanical rename may be easy to inspect. For deciding review effort, would it help to ask which behaviors can change and how hard they are to reverse? That would keep the short reporting format useful without encouraging agents to make risky work look small by writing fewer words.
Yes to question 1, in its purest form: a Codex run reported a manifest file as written, and the file was never created. No buried clause, just a confident line in the summary.
The other version I keep hitting is green everything with the actual feature wrong. On one of my small data sites, tests passed and pages returned 200, but the site was still reading an old cache, the new SEO pages had no internal links pointing at them, and the home cards sent visitors to another site. None of that shows up in a summary or in CI.
So my rule isn't about the report format. A report is a claim, and nothing is done until I've walked the product as a visitor, page by page, in production. I like your Skipped/Deferred section, but as you say, an agent that didn't notice the gap will happily write "Skipped: None".
For people doing this on a team, who does the walk-through: the author of the PR or the reviewer?
The What Was Not Done section is the guardrail I would adopt first, because it targets the exact failure in your opening example. A dropped idempotency check should never sit four paragraphs down under Minor Adjustments. I also liked the point about keeping prompts small. When a change needs 600 words to explain, the review is already too wide to catch a buried caveat, and a short diff plus a clear skipped list feels much easier to check properly.
The Skipped/Deferred section is the right fix. One gap: "Skipped: None" is still a claim the agent writes about itself. What I'd add is a mapping against the ticket. List every acceptance criterion by ID, and next to each one either the test that proves it or the Skipped entry that explains it. A criterion with neither is your silent omission, and a reviewer can spot it in seconds without reading the 600 words. Your idempotency example would show up as an empty row before merge, not as an alert three days later.
Your coding agents are the engineers. Halfcycle is the rest of the team: it scopes, designs, plans and reviews production software, and runs the build inside your own Claude Code or OpenAI Codex.
The idempotency example is the right one, because it is a decision, not a bug. Nobody asked the agent whether that check could go, so it decided, and the summary is where the decision got filed.
What has worked for us: write down the constraints that must survive (idempotency, tenant isolation, audit logging) before the work starts, then have a separate reviewer check the diff against that list instead of reading the agent's own account. A summary from the agent that did the work is grading its own homework. That is a big part of why we built Halfcycle: it asks for those decisions up front and reviews against them, inside Claude Code or OpenAI Codex.
Have you tried asking the agent to list what it did not do? Curious whether that list comes out any more honest than the summary.
A page one of my agents built had to match a design master pixel for pixel, and its parity check passed every state while masking about 19% of the default screen. The masks covered the saved-search tabs and the save button, which had never been built. With the masks off and the sidebar back in, every state failed, 1.1 to 1.9% of pixels off against a 1% limit. A table mapping criteria to passing tests would've shown that row green. The fix records an unmasked score next to every masked one and ties each mask to the later step that removes it.
The required “Skipped / Deferred” section is valuable because it converts silence into a falsifiable claim. I’d make it acceptance-criterion keyed rather than only file keyed: every criterion should end as implemented, deferred with a blocker, or explicitly not applicable. That makes a reviewer compare the intended behavior to the result before reading the agent’s prose.
AI-generated summaries can save time, but important details often get overlooked when the original material is not reviewed carefully. Small technical points, exceptions, and supporting information can make a significant difference in the final result. The same applies to checking south-dakota-license-plate.org where accurate details matter when reviewing vehicle plate information. AI can be a useful starting point, but verifying the underlying information remains important. A careful review helps prevent misunderstandings and incomplete conclusions.
Sharing tutorials and API guides on omnichannel automation. We help devs build seamless end-user workflows across WhatsApp, Telegram, Meta (FB/IG), SMS, and webchat.
This is such an important perspective! We often focus on what AI coding agents successfully deliver, but rarely pay enough attention to what they silently leave behind. A polished summary and passing tests can create a false sense of confidence, while the most critical details get buried in the fine print. I especially like the idea of explicitly documenting what was skipped or deferred. At the end of the day, AI should make developers more productive, not make us less careful. Human judgment, accountability, and proper code reviews still matter more than ever. Great insights and a very practical reminder for anyone working with AI agents!
Building persistent memory for AI agents at Mnemoverse.
I write about the unglamorous half: forgetting, contradiction, and the moment an agent confidently repeats a fact you retracted.
Your opening example was a refactor ticket, so the dropped idempotency check was not invisible in the diff. It was a removed line. Most of the guardrails in this thread work from the requirement side: list the acceptance criteria, map each one to a test or a skipped entry. That catches what the ticket asked for. It does not catch a guard nobody thought to put in the ticket, because the code already had it.
For refactors and migrations there is a cheaper check on the other side of the diff. Take every removed condition, early return, retry, lock or key check, and require each one to appear again somewhere in the added lines, or to be listed as a decision with a reason. A guard that moved is fine. A guard that vanished is exactly the "Minor Adjustments" clause, except now it is a row a reviewer can scan in a minute, and it does not depend on anything the agent chose to write about itself.
The pairing does not need to be clever. Even a plain list of removed lines that contain if, raise, retry or lock, placed above the summary, puts the dropped check on the first screen instead of the fourth paragraph.
The "What Wasn't Done" section is the part I wish I'd added months ago. I started requiring a Skipped/Deferred block after an agent quietly dropped a retry timeout during a "simple" refactor. CI stayed green, and we only noticed when a downstream timeout started cascading. I also like the second read-only pass idea — I now run a separate review prompt that only gets the acceptance criteria and the diff, never the original agent's writeup.
We're a place where coders share, stay up-to-date and grow their careers.
The buried "Minor Adjustments" clause is the right diagnosis: a 600-word sign-off is not evidence that the ticket is done.
The article already flags the hole I would treat as the main one: told to write
Skipped: None, the agent sometimes will. A labeled block is still a claim the model authored. I wouldn't let that claim be the record the reviewer trusts.I'd have the host emit the Completed / Skipped table from the acceptance criteria and from whatever the run actually recorded — the diff, the named commands, the test results — so "Skipped: None" is not a sentence the agent is allowed to type into the gate. The agent's prose can annotate the table. It shouldn't be allowed to certify it.
A second read-only pass is useful, but only if that pass also isn't grading a writeup it wrote itself. The end state is whatever the recorded checks show, not the cheerful sign-off.
I'd question using summary length to decide that a task was too broad. A one-line configuration change can alter tenant isolation, while a long mechanical rename may be easy to inspect. For deciding review effort, would it help to ask which behaviors can change and how hard they are to reverse? That would keep the short reporting format useful without encouraging agents to make risky work look small by writing fewer words.
Yes to question 1, in its purest form: a Codex run reported a manifest file as written, and the file was never created. No buried clause, just a confident line in the summary.
The other version I keep hitting is green everything with the actual feature wrong. On one of my small data sites, tests passed and pages returned 200, but the site was still reading an old cache, the new SEO pages had no internal links pointing at them, and the home cards sent visitors to another site. None of that shows up in a summary or in CI.
So my rule isn't about the report format. A report is a claim, and nothing is done until I've walked the product as a visitor, page by page, in production. I like your Skipped/Deferred section, but as you say, an agent that didn't notice the gap will happily write "Skipped: None".
For people doing this on a team, who does the walk-through: the author of the PR or the reviewer?
The What Was Not Done section is the guardrail I would adopt first, because it targets the exact failure in your opening example. A dropped idempotency check should never sit four paragraphs down under Minor Adjustments. I also liked the point about keeping prompts small. When a change needs 600 words to explain, the review is already too wide to catch a buried caveat, and a short diff plus a clear skipped list feels much easier to check properly.
The Skipped/Deferred section is the right fix. One gap: "Skipped: None" is still a claim the agent writes about itself. What I'd add is a mapping against the ticket. List every acceptance criterion by ID, and next to each one either the test that proves it or the Skipped entry that explains it. A criterion with neither is your silent omission, and a reviewer can spot it in seconds without reading the 600 words. Your idempotency example would show up as an empty row before merge, not as an alert three days later.
The idempotency example is the right one, because it is a decision, not a bug. Nobody asked the agent whether that check could go, so it decided, and the summary is where the decision got filed.
What has worked for us: write down the constraints that must survive (idempotency, tenant isolation, audit logging) before the work starts, then have a separate reviewer check the diff against that list instead of reading the agent's own account. A summary from the agent that did the work is grading its own homework. That is a big part of why we built Halfcycle: it asks for those decisions up front and reviews against them, inside Claude Code or OpenAI Codex.
Have you tried asking the agent to list what it did not do? Curious whether that list comes out any more honest than the summary.
A page one of my agents built had to match a design master pixel for pixel, and its parity check passed every state while masking about 19% of the default screen. The masks covered the saved-search tabs and the save button, which had never been built. With the masks off and the sidebar back in, every state failed, 1.1 to 1.9% of pixels off against a 1% limit. A table mapping criteria to passing tests would've shown that row green. The fix records an unmasked score next to every masked one and ties each mask to the later step that removes it.
The required “Skipped / Deferred” section is valuable because it converts silence into a falsifiable claim. I’d make it acceptance-criterion keyed rather than only file keyed: every criterion should end as implemented, deferred with a blocker, or explicitly not applicable. That makes a reviewer compare the intended behavior to the result before reading the agent’s prose.
AI-generated summaries can save time, but important details often get overlooked when the original material is not reviewed carefully. Small technical points, exceptions, and supporting information can make a significant difference in the final result. The same applies to checking south-dakota-license-plate.org where accurate details matter when reviewing vehicle plate information. AI can be a useful starting point, but verifying the underlying information remains important. A careful review helps prevent misunderstandings and incomplete conclusions.
This is such an important perspective! We often focus on what AI coding agents successfully deliver, but rarely pay enough attention to what they silently leave behind. A polished summary and passing tests can create a false sense of confidence, while the most critical details get buried in the fine print. I especially like the idea of explicitly documenting what was skipped or deferred. At the end of the day, AI should make developers more productive, not make us less careful. Human judgment, accountability, and proper code reviews still matter more than ever. Great insights and a very practical reminder for anyone working with AI agents!
Your opening example was a refactor ticket, so the dropped idempotency check was not invisible in the diff. It was a removed line. Most of the guardrails in this thread work from the requirement side: list the acceptance criteria, map each one to a test or a skipped entry. That catches what the ticket asked for. It does not catch a guard nobody thought to put in the ticket, because the code already had it.
For refactors and migrations there is a cheaper check on the other side of the diff. Take every removed condition, early return, retry, lock or key check, and require each one to appear again somewhere in the added lines, or to be listed as a decision with a reason. A guard that moved is fine. A guard that vanished is exactly the "Minor Adjustments" clause, except now it is a row a reviewer can scan in a minute, and it does not depend on anything the agent chose to write about itself.
The pairing does not need to be clever. Even a plain list of removed lines that contain if, raise, retry or lock, placed above the summary, puts the dropped check on the first screen instead of the fourth paragraph.
The "What Wasn't Done" section is the part I wish I'd added months ago. I started requiring a Skipped/Deferred block after an agent quietly dropped a retry timeout during a "simple" refactor. CI stayed green, and we only noticed when a downstream timeout started cascading. I also like the second read-only pass idea — I now run a separate review prompt that only gets the acceptance criteria and the diff, never the original agent's writeup.