This is the part I stopped trusting at face value. I now require the agent to paste the exact commands it executed together with the pass/fail output for each step, and I do a quick manual spot-check of the resulting diff before marking the task done. A simple Stop hook that only runs the test suite is not enough when the suite itself misses edge cases; the printed command log lets me see whether lint, type-check, and build actually ran and succeeded. If any check is missing or shows a warning, I send the agent back with a concrete fix request. This extra step catches silent failures that a green test run would hide.
Printing the commands is a good check, because a green test run says nothing about what never ran. The spot-check on the diff is the part I wouldn't drop either.
Filling that section in changes how you use the agent, because it makes you name the edges of what it verified. Asking for it explicitly takes effort. An empty section on a change that touched the editor tells you more than a confident paragraph of prose. What made you put it last in the format but read it first?
It goes last because the agent can only say what it skipped once the rest of the report is filled in. I read it first because it tells me where to look before I trust anything above it.
A Stop hook is where done should be decided: the agent that wrote the code should not be the one declaring it finished. Put typecheck and build in the hook, so a run can't end on a failing check or on a claim alone.
Ran into this exact problem building visual review into an agent loop. The gotcha that got me: resizing a full page screenshot down to fit a token budget can make a layout bug invisible in the resized image even though it is obvious at full res, so the agent happily approves broken layout. I build a screenshot tool for this, the resolution math is on my blog at slimsnap.ai/blog/screenshot-downscaling, if it's useful.
That's a nasty one, because the check passes and the screenshot looks fine. Keeping a full-res crop of the area under review would avoid losing the bug in the resize.
@ahmad_dogar_7d098f5e48f72 Agreed, and keeping it fast is the point. A hook that takes a minute gets skipped on the first bad day, and then it checks nothing.
@ahmad_dogar_7d098f5e48f72 Yes. The agent can claim done, the hook decides. Typecheck and build are a cheap start, and tests that finish in seconds can go in the same hook.
This is the part I stopped trusting at face value. I now require the agent to paste the exact commands it executed together with the pass/fail output for each step, and I do a quick manual spot-check of the resulting diff before marking the task done. A simple Stop hook that only runs the test suite is not enough when the suite itself misses edge cases; the printed command log lets me see whether lint, type-check, and build actually ran and succeeded. If any check is missing or shows a warning, I send the agent back with a concrete fix request. This extra step catches silent failures that a green test run would hide.
Printing the commands is a good check, because a green test run says nothing about what never ran. The spot-check on the diff is the part I wouldn't drop either.
Filling that section in changes how you use the agent, because it makes you name the edges of what it verified. Asking for it explicitly takes effort. An empty section on a change that touched the editor tells you more than a confident paragraph of prose. What made you put it last in the format but read it first?
It goes last because the agent can only say what it skipped once the rest of the report is filled in. I read it first because it tells me where to look before I trust anything above it.
A Stop hook is where done should be decided: the agent that wrote the code should not be the one declaring it finished. Put typecheck and build in the hook, so a run can't end on a failing check or on a claim alone.
Yes, that's the split I want: the agent can claim done, but the hook decides. Typecheck and build are a cheap place to start.
Ran into this exact problem building visual review into an agent loop. The gotcha that got me: resizing a full page screenshot down to fit a token budget can make a layout bug invisible in the resized image even though it is obvious at full res, so the agent happily approves broken layout. I build a screenshot tool for this, the resolution math is on my blog at slimsnap.ai/blog/screenshot-downscaling, if it's useful.
That's a nasty one, because the check passes and the screenshot looks fine. Keeping a full-res crop of the area under review would avoid losing the bug in the resize.
@ahmad_dogar_7d098f5e48f72 Agreed, and keeping it fast is the point. A hook that takes a minute gets skipped on the first bad day, and then it checks nothing.
Exactly, and a skipped hook still looks like a safety net, which is worse than not having one.
@ahmad_dogar_7d098f5e48f72 Yes. The agent can claim done, the hook decides. Typecheck and build are a cheap start, and tests that finish in seconds can go in the same hook.
Yes, tests that finish in seconds fit there too. I'd keep anything slow out, so the hook stays fast enough that nobody wants to bypass it.