While researching for this post, I found out something funny. Pi, the most interesting coding agent of 2026, was built by a guy who got fed up with...
For further actions, you may consider blocking this person and/or reporting abuse
Really enjoyed this comparison. Great work putting together 100 hours of real-world testing — the cost and task-success numbers are especially interesting. 👏
One thing I’d be curious to see in a future comparison is the security side of the agent model. 🔐
For example, how do you evaluate the trust boundary when extensions can modify the agent’s behavior at runtime? And what happens if an extension or external piece of code used to speed up the workflow is compromised or malicious?
I think that could make for a really interesting follow-up, especially alongside the recovery/failure analysis mentioned above.
Great work, and thanks for sharing the results! 🚀
Thanks Marco! Extensions run in the same process with full privileges, so a compromised extension IS the agent at that point. There's no sandbox to escape. Zechner's own answer is containerize the whole thing and review extensions like you'd review a dependency.
Really appreciate your answer. You clearly know what you’re talking about, and honestly, the fact that your comparison after “only” 100 hours produced this kind of discussion is pretty impressive. 👏
I also really appreciate that your answer is technical and not biased toward either Pi or Claude Code. I completely agree with the way you framed the extension problem as a supply-chain issue. 🔐
Personally, I would probably choose the supply-chain approach as well. It is definitely harder to manage properly, but if you build it correctly with provenance, review, signing, versioning and integrity controls, you can achieve a very strong trust model.
Really great work on the comparison and on the technical reasoning behind it. Seriously impressive. 🚀
Really interesting comparison. The 100 hours of actual usage makes this much more useful than a typical feature-by-feature comparison. I like the point that both tools have their own strengths depending on the workflow.
Thank you! Yeah that was the goal, feature tables are everywhere, but nobody tells you what it's like on day 10.
Delightful read 🎉
Thanks, Aavash 🙌
stop rule I would use after 100 hours is simple. when one of them loses the same class of task three sessions running, it stops being the default
That's a genuinely good heuristic, stealing this.
Wiring every harness to DeepSeek makes the test fair on paper but it also strips Claude Code of the model it was tuned around. Did you try the same 30 tasks with a Claude model in both harnesses?
After 100 hours, the meaningful comparison is less about first-prompt quality and more about recovery behavior. How does the agent handle wrong assumptions, stale docs, missing credentials, and partial tool failures? That is where production coding workflows either become faster or quietly accumulate risk.
The scorecard favoring Pi while almost everyone keeps Claude Code as the daily driver is the real finding here. Evals measure task pass rate and cost per success, but the thing that decides what I open on a Tuesday is how much trust I have when the agent rewrites three files I never asked for. Minimal harnesses win on transparency and tokens, then lose the moment you need a hard stop mid session and a clean way to inspect what actually changed. The product that survives is usually the one whose failure mode is reviewable, not the one that looks best on a table of 30 tasks.
One variable the eval section leaves open: which Claude Code release the 30-task run was pinned to. You note "Anthropic quietly deleting most of its own system prompt is about the strongest validation of Pi's thesis you could ask for", and that cuts both ways for benchmarking, if the harness sheds 80% of its prompt between releases, a 16/30 measured on one version may not reproduce on the next. Zechner built cchistory precisely because the prompt and tool definitions drift release to release. Did you record the CC version, and would you consider re-running the same 30 tasks across two or three releases? A harness-stability delta would be a more interesting number than the win-loss scorecard, and as far as I can tell nobody publishes one.
Great harness comparison! I use all 3 in parallel actually. Sometimes opencode and omp as well. Why do you think pi solves more tasks than claude? Generally I find the opposite in my case. Maybe because my sessions are multi-turn?
100 hours of actual use is worth more than any benchmark screenshot for this kind of comparison. The thing I always want from these writeups is the failure breakdown: where each agent lost the thread, retried silently, or opened the wrong file, since that is usually what decides which one you keep. Did Pi and Claude Code fail in different shapes, or the same ones at different rates?
Different shapes. From what I saw in the runs: e.g. Claude Code's misses were mostly timeouts / lost the thread mid-task / wrong file, while Pi's were more gave up early / wrong approach it committed to. which weirdly makes Pi's failures easier to catch.
We have also something like OpenCode which is also great and do a lot of stuff and still free for us, even with some subscription models or even free ones.
Really nice work with intressting facts.Good luck