I watch a lot of agent demos, and the ones that convince me are rarely the impressive ones. Here is the checklist I use, in the order that filters fastest.
1. Does the video show the failure, or only the success?
A demo that only shows the run that worked tells me the author ran it once. A demo that shows a failure and then the fix tells me the author has run it thirty times. The second one is worth my attention and the first one is marketing.
2. Is there a diff at the end?
Show me the actual output: the file, the query result, the sent message. Not the agent's description of the output. Any claim that cannot be checked against an artefact is a claim about the model's tone, not its work.
3. What did the human do during the run?
Editing the prompt live, retyping the password, clicking the confirm button, "just moving this here" - each of those is work the demo silently attributes to the agent. The honest version puts the human's actions on the timeline too.
4. What happens on the second run?
Novel tasks are easy to make look good. Run the same task again tomorrow, on a machine in a different state, and see if it still holds. Most of my own automation failed on run 3, not run 1 - the environment had drifted and nothing had noticed.
5. How is it verified?
"Is this right?" answered by the agent is not verification. Verification is a command, a test, a hash, a row count - something that fails loudly. If the demo has no verification step, the reviewer is the verification step, and the reviewer is watching a video.
What makes me trust a demo instead
A short postmortem beats a slick video every time: what was the task, what broke, what the log said, what changed. Bonus if the author tells me what they gave up on, because that is the only part nobody fabricates.
I try to hold my own writing to this, and I fail at item 3 constantly - my logs make my own interventions visible, which is uncomfortable and useful in equal measure.
What would you add? The item I am least sure about is 4, because a task can hold on run 2 and still be a liability at run 20.
Top comments (1)
두 번째 실행에서 환경이 달라지는지 확인한다는 항목에 공감합니다. 실행 횟수뿐 아니라 로그인 만료·이미 처리된 항목·권한 변경 같은 시작 조건을 나눠 재실행하면, 우연히 성공한 데모와 반복 가능한 작업을 구분하기 더 쉬울 것 같아요.