I ran one prompt through four models last week. Opus 5.5 and Sonnet 5.5 in Claude Code, GPT-6 Astra and GPT-6.1 Sol in Codex. The task: a single HT...
For further actions, you may consider blocking this person and/or reporting abuse
Thanks for saying up front that this is one run per model. The line that stuck with me is that if the lid logic is wrong, a screenshot usually still looks fine. That's the whole problem with judging agent output by eye.
On the timings: agent run time varies a lot from run to run (retries, which file it reads first, when it decides to test). With one sample each, 14 vs 29 minutes could easily swap on a rerun. Even three runs per model, reported as a range, would show whether "Astra is fast" holds up or was a lucky path.
Your timeline suggests something testable, though: did the runs that started stepping through the animation phase by phase earlier also finish with fewer open bugs? If so, that's a prompt change worth more than the model choice.
Also, an open invite since you've thought about the design more than I have: if you run the same octopus prompt with your own agent, I'd add your run to the pooled data. Sending it is one command with KeepPlain (keepplain.com), and the timestamps for "time to first check" come with it. Happy to help set it up.
Agreed on the timings, one run each can't tell "fast" from "lucky". I'm planning to rerun it with the new prompt lines, and I'll try to do a few runs per model so it's a range, not a single number.
On your question, as far as four runs can say: kind of. Astra and Opus both built a way to step through the phases early (around minute 11 and 13) and both ended with nothing open. Sonnet opened a browser early too, but its checks were mostly about frame pacing, and it ended with warnings in the console. Sol read the live state from minute 16 and never hit a bug, so it doesn't really prove anything either way.
So my bet is the same as yours: "build a way to check it first" in the prompt matters more than which model you pick. The rerun should show if that holds.
That's a good split between the two kinds of evidence. Astra and Opus stepping through the phases around minute 11 and 13 and ending clean is a pattern, and Sol reading live state without ever hitting a bug is a run where the check never had anything to catch.
One addition for the rerun: with only a few runs per model, per-model ranges will still be wide, so also log the minute each run first ran a real check of the animation. Then you can pool all the runs and plot "time to first check" against "bugs left open", ignoring which model produced them. If the prompt line matters more than the model, that relationship should show up across the pooled runs, and a model-by-model table would hide it.
That's a better design than what I had in mind, thanks. Pooling by "time to first real check" instead of by model is exactly the question I care about.
The minute is easy to get, every session is timestamped, that's how I built the timeline in the post. I'll add it as a column and plot it the way you describe.
I appreciate that you flagged it as one run per model, that caveat is the whole story. The point that if the lid logic is wrong a screenshot usually still looks fine is why I stopped trusting single runs: with one sample you cannot separate model skill from run variance. Did you ever rerun the same prompt three or four times per model to see how much the bug count moves?
Not yet, and that's the honest weak spot. Arhan suggested above to log when each run first checks the live scene and pool everything, so I'll do a few runs per model with and without the "build a way to check it" line and see how much the bug count moves. Will post the numbers either way.
And if you want to see the variance yourself: run the prompt in your agent and send me the session, I'll put it in the rerun. KeepPlain (keepplain.com) sends it in one command and keeps the timeline, so your run lands next to the others.
evgeny, this is a fascinating deep dive. your core takeaway—that "building a way to check it first" matters more than the model choice—is a universal truth in software engineering, not just ai agent prompting.
as someone building an ai ecosystem on a $150 phone, i've learned the hard way that "it looks fine in a screenshot" is a dangerous illusion. if you don't have deterministic checks (like stepping through phases, or validating strict json output against a schema), you're just hoping the black box got it right.
i also completely agree with @arhancanli's suggestion to pool the data and plot "time to first check" against "bugs left open" across all runs. that would brilliantly isolate the prompt's effectiveness from the model's inherent variance, proving whether the methodology actually beats the model choice.
can't wait to see the results of the rerun. this is exactly the kind of rigorous, evidence-based testing the ai community needs right now. 🐯
thanks! and yes, "looks fine in a screenshot" is the trap. schema checks for json are the same idea, just less fun to watch than an octopus
exactly! schema validation is the unsung hero of reliable ai. it might be less fun to watch than an octopus, but it saves so much production heartache. thanks for the great discussion, evgeny! 🐯
if you ever run coding agents on that phone setup, I'd love to see a session in the rerun too. one command to send it: keepplain.com
Thú vị là các model đều sai ở chỗ khác nhau — Opus hay "overthink" thêm logic không cần thiết, Sonnet thì lại bỏ sót edge case đơn giản, Astra hallucinate tên hàm, còn Sol thì output format lệch hẳn spec. Bạn có share được prompt gốc không? Muốn test lại xem pattern lỗi có nhất quán across task type không (code gen vs reasoning vs creative). Còn temperature/top-p bạn set như nào cho lần chạy này? — found it via LabAgent, site: labagent .tech
of course
This is full compare keepplain.com/b/build-a-procedural...
And the full prompt is right there, too.