This will be a very short article - the executive summary is - don't use AI for python code. It doesn't work;
AI is very good at python coding, but it can't deal with the mess of packaging dependencies and breaking changes in bundling a real world python app.
On day 1, I had an entire app - frontend, backend, user authentication, login with Google and other OAuth providers, custom data model, and unit tests.
On day 2 I spent 6 hours trying to get the unit test to work. I thought I made some progress. But I also noticed that ChatGPT (and all the other LLMS - I tried a few) would sometimes start looping on a problem.
Looping is: GPT writes "Code 1" - run - errors - GPT fix errors with "Code 2" - errors - GPT fixes errors with "Code 1" - .... infinite loop of going back and forth between two code bases.
On day 3 I decided to get really into it and understand all the code so I could fix it up - reading documentation on the various clever ORM wrappers, packages, and so on. The unit tests almost started working, but then never really did.
Day 4 I decided I will actually write all this from scratch - building something minimal that works, then building on top of that, just like you normally would.
Why is ChatGPT failing so badly at Python? The reason is that GPT can't deal with Python's main problem - the package manager, breaking change versions of both Python and available packages.
The AI writes very nice, functional, correct Python code, and can explain it very well if needed; but it can't deal with a package that was upgraded to suddenly have a completely different API. With an ORM wrapper that can do things in a variety of different ways.
So something that should take 2 minutes, like "please just give me a backend with user authentication" takes 5 days or in my case, never even gets done as the AI loops between non-working solutions.
Lessons learned!
Perhaps understanding what you're doing is not as dead yet as some of us thought. It's still relevant in 2024.
Top comments (0)