I’ve been doing a lot of code reviews recently, and I’ve noticed something about my experience during the reviews. And what it means for code in general, but specifically for AI generated code.
But before I do that, I want to take you back to the ’00s. A simpler time. Back then I started doing something I only read about in books – pair programming. Ok, that’s a lie. Everyone who grabbed someone and brought them to look at your code because it’s doing something weird – you’ve done pair programming.
But this was pair programming for like 80% of my work. And of course, not just me. This links directly to my topic – our understanding of the code. And in the age before AI, where teams really knew their code, that would be the biggest grade I’d give for code understanding.
Meaning, if you and I work both at the same time on the same task, and finish it. I’d say we’re both at the highest level of understanding of that feature – what it needs to do, how it works, what it depends on, what depends on it, and how it’s written and tested.
From here that level of understanding is going to drop. If you’re part of a team, but didn’t work on that feature, you probably know it exists. If we have dependencies, you probably know about them. But the rest is either non-existent or close to it.
If you’re on another team, that level of understanding drops, and if you’re on another project – you may not even be aware.
That makes sense, but we’re not talking about that today, I want to focus on my capability to do a proper review. And I go back to understanding for that. In order for me to give you proper feedback, catch your mistakes and offer alternatives, I need proper understanding. The more understanding I have, I can review better and make better changes.
So far, so good. It makes sense.
Two more things make sense. First the use of tools for review. Every tool, from syntax analyzer to the smartest security checker, work according to patterns, not understanding. All the tools check for generic stuff. Newer tools that use LLMs, can find more elusive patterns. But in the end, it’s like finding a missing semi-colon.
The second thing is async code review. Or PR review, as people call it. This is done not with the person who wrote the code, but at my spare(?) time. I pull the PR, look at the code and based on my understanding, I need to approve it. Note that at this point, I can’t ask questions, so guess what I do? Look for patterns. If I’m the team leader and I know the feature, I can review better. If I’m on another team – I will not. Not take risks. Not initiate another coding and code review. Who has time for that?
And that brings me to reviewing code my genie wrote.
Because if you think about it, I’m the team leader, and the code agent is on my team. I’m supposed to know more about the feature, but I’m reviewing code I didn’t write.
How well can I do it? There’s another part that contributes to this, which is the size of the diff. Humans, before AI know their limitations (most of them), and so they don’t write a lot of code, because they know what it’s like to review a lot of code. But even if they do, the reviewer, unless she gives up, can still ask questions.
Well, my genie does a lot of work and creates a lot for me to review. Now, I know I can’t review a lot of code in one go, so I make my genie work in chunks. Chunks I can read and understand. For example I break features into workflows, and ask the genie to do a workflow or two, including all tests and whatever’s needed.
Which is still a lot. And I tried to explore what happens that makes me give up. Yes I do give up sometimes.
It goes back to understanding. I’m reviewing pieces of code – tests, bits of FE, bits of the API code, bits of logic and bits of database code. I can review these at the lowest level – does it do what I need? Can I read it?
But when it comes to bigger things – like dependencies and side effects and performance – I give up. First there’s a lot of code to absorb, and then I need to piece everything that I know (or think I know) of how the rest of code works and designed. I’m building a code model in my mind. This helps me identify design issues, find the side effect traps. Maybe the performance issue hiding in there.
But, if it’s hard to build that puzzle, I’m not going beyond “does it look ok” and “do the tests run”.
So, I’m taking risks. Risking that there’s code in there that I don’t know what it does, how it works and what effect it might have on the system today. And tomorrow, I won’t even consider it for any co-dependence with my new feature, because I don’t understand how it works.
This accumulates over time. Within a few days or weeks – I won’t know how most of my app works.
So what do I actually do about it?
Two things, and neither of them is a fix.
The first one you’ve already seen. I keep the genie working in workflows, not in features. It makes the diffs smaller, and keeps them inside a context I can understand. The second one is a decision. Before I review a chunk, I decide what is important for me to go deep into, and what to go over lightly. Or not at all.
It doesn’t come close to the old level of understanding. But my overall feeling of understanding has improved.
Now, take that, and think about teams and groups. How much code and tests that nobody knows and understands can be accumulated in one year. Let alone three.
Do you understand this risk? And how are you handling it?
Originally published at testingil.com.
I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.
Top comments (0)