Full-stack dev sharing practical guides on WordPress, n8n automation, AI tools, Docker & self-hosting. Always experimenting with new tech to make life easier.
About the plan mode, that's actually one of the things I wanted to compare. I didn't explicitly enable a "plan mode" feature. The first step of the benchmark was simply to give the models the same functional brief and ask them to propose the architecture, project structure, scripts and API routes.
And interestingly, I was expecting them to ask questions before committing to an architecture. They didn't. They all produced their plan first and raised questions afterwards. That was quite revealing, because some of those questions could actually have affected the architecture.
About testing, though, there's an important limitation to my experiment: this was a PoC, not a production project. I never installed it on a server, so I didn't actually test the resulting application in real-world conditions.
The review was therefore a code review, not a substitute for functional testing. I had another model review the result for things like security, correctness, idempotency and completeness. That's useful, but it's obviously not the same thing as deploying the application and seeing what happens.
And I agree with you about debugging. That's probably one of the areas where I've seen AI coding become particularly frustrating. After several attempts, the model can start fixing its own previous fixes rather than the original problem. 😄
Starting over with a clean context can sometimes be the better option. And using a different model helps too — especially because you get a completely different reasoning path instead of asking the same model to keep defending its previous approach.
And yes, the panel is supposed to handle those four types of websites. 😄 But it's still part of the benchmark/PoC, so I haven't had the opportunity to actually test it in real-world conditions.
There is certainly still some work to do on it, but honestly, it's not a priority for me at the moment. Other things are more urgent!
Sorry, I misunderstood. You didn't mean plan mode, but rather wanted to check whether the models ask questions before planning. It sounds like you're describing the "Deep Research" mode in AI coding: when given a task, the AI first asks clarifying questions, then plans. Haha.
Some developers have shared that they usually work with a separate model to plan the architecture in detail, then hand the resulting spec to another model to write the code.
I've thought a lot about what programmers should do once AI coding becomes really capable. Then Paul Graham's words in Hackers and Painters inspired me: "If a hacker were a mere implementor, turning a spec into code, then he could just work his way through it from one end to the other like someone digging a ditch. But if the hacker is a creator, we have to take inspiration into account." This got me thinking: perhaps we can let AI handle the implementation, while we humans focus on the ideas(and the architecture). What are your thoughts? 😄
Speaking of turning ideas into reality —so you have the scripts but no front-end panel? That's pretty common. Now that AI coding makes turning an idea into reality much easier, the verification and real-world usage have become the new bottleneck.
Full-stack dev sharing practical guides on WordPress, n8n automation, AI tools, Docker & self-hosting. Always experimenting with new tech to make life easier.
Yes, I think you're very much onto something here — and actually, what you describe is pretty much how I work already. 😄
I don't see AI coding as replacing the creative part of programming. For me, it's almost the opposite: I come up with the idea, define the problem and the broad architecture, and specify what I want the system to do. Then I let the AI handle most of the implementation, and also do a first round of verification.
But I still review the result myself and test it myself. I think that distinction is important: AI can implement and verify its own work, but I don't think that should make us stop questioning the result.
In fact, I suspect that as AI gets better at implementation, the interesting part of programming will increasingly move towards the things that are harder to specify: having the idea in the first place, understanding the problem, making architectural choices, and knowing what "good" actually means.
And just to clarify one thing about the panel: there is a CLI interface for it. I simply haven't had the opportunity to try the whole thing on a completely clean VPS yet. So, like you said, the real-world verification is still missing.
And I really like your point about verification becoming a bottleneck. That's probably one of the interesting consequences of AI coding: once producing code becomes cheap and fast, the scarce resource isn't necessarily implementation anymore — it's knowing whether what was produced is actually the right thing.
Some comments have been hidden by the post's author - find out more
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Thank you, Julie! 😊
About the plan mode, that's actually one of the things I wanted to compare. I didn't explicitly enable a "plan mode" feature. The first step of the benchmark was simply to give the models the same functional brief and ask them to propose the architecture, project structure, scripts and API routes.
And interestingly, I was expecting them to ask questions before committing to an architecture. They didn't. They all produced their plan first and raised questions afterwards. That was quite revealing, because some of those questions could actually have affected the architecture.
About testing, though, there's an important limitation to my experiment: this was a PoC, not a production project. I never installed it on a server, so I didn't actually test the resulting application in real-world conditions.
The review was therefore a code review, not a substitute for functional testing. I had another model review the result for things like security, correctness, idempotency and completeness. That's useful, but it's obviously not the same thing as deploying the application and seeing what happens.
And I agree with you about debugging. That's probably one of the areas where I've seen AI coding become particularly frustrating. After several attempts, the model can start fixing its own previous fixes rather than the original problem. 😄
Starting over with a clean context can sometimes be the better option. And using a different model helps too — especially because you get a completely different reasoning path instead of asking the same model to keep defending its previous approach.
And yes, the panel is supposed to handle those four types of websites. 😄 But it's still part of the benchmark/PoC, so I haven't had the opportunity to actually test it in real-world conditions.
There is certainly still some work to do on it, but honestly, it's not a priority for me at the moment. Other things are more urgent!
Sorry, I misunderstood. You didn't mean plan mode, but rather wanted to check whether the models ask questions before planning. It sounds like you're describing the "Deep Research" mode in AI coding: when given a task, the AI first asks clarifying questions, then plans. Haha.
Some developers have shared that they usually work with a separate model to plan the architecture in detail, then hand the resulting spec to another model to write the code.
I've thought a lot about what programmers should do once AI coding becomes really capable. Then Paul Graham's words in Hackers and Painters inspired me: "If a hacker were a mere implementor, turning a spec into code, then he could just work his way through it from one end to the other like someone digging a ditch. But if the hacker is a creator, we have to take inspiration into account." This got me thinking: perhaps we can let AI handle the implementation, while we humans focus on the ideas(and the architecture). What are your thoughts? 😄
Speaking of turning ideas into reality —so you have the scripts but no front-end panel? That's pretty common. Now that AI coding makes turning an idea into reality much easier, the verification and real-world usage have become the new bottleneck.
Yes, I think you're very much onto something here — and actually, what you describe is pretty much how I work already. 😄
I don't see AI coding as replacing the creative part of programming. For me, it's almost the opposite: I come up with the idea, define the problem and the broad architecture, and specify what I want the system to do. Then I let the AI handle most of the implementation, and also do a first round of verification.
But I still review the result myself and test it myself. I think that distinction is important: AI can implement and verify its own work, but I don't think that should make us stop questioning the result.
In fact, I suspect that as AI gets better at implementation, the interesting part of programming will increasingly move towards the things that are harder to specify: having the idea in the first place, understanding the problem, making architectural choices, and knowing what "good" actually means.
And just to clarify one thing about the panel: there is a CLI interface for it. I simply haven't had the opportunity to try the whole thing on a completely clean VPS yet. So, like you said, the real-world verification is still missing.
And I really like your point about verification becoming a bottleneck. That's probably one of the interesting consequences of AI coding: once producing code becomes cheap and fast, the scarce resource isn't necessarily implementation anymore — it's knowing whether what was produced is actually the right thing.