My Impression of AI in Programming

Introduction

In a previous post, I wrote the following about using AI in developing my new tool include-tidy (Tidy) that uses Libclang, a library I’d never used before:

What helped a lot was using AI (strictly speaking, an LLM), specifically Google’s Gemini (because I’m too cheap to pay for Claude, especially for a personal project that I have no intention of making any money from). While I may write a follow-up blog post describing my experience, I’ll state briefly that AI saved me from having to read a lot of the documentation, read the tutorials, post questions to a mailing list or Stack Overflow, and wait for answers (if any).

This is that aforementioned follow-up blog post. If you’ve been using AI for a while, none of what follows will likely be a surprise to you. Prior to using AI for Tidy, I’d only used it for small functions or to ask corner-case questions. Using it for Tidy was my first “big” use of AI.

If this is your first encounter with my blog or me, you might be wondering “Who is this guy, and why should I care what he thinks?” I’ve been tinkering with computers for over four decades, both as a hobby and professionally. Some programmers, especially those from older generations, are pretty skeptical about AI — often referring to it as “autocorrect on steroids.” While there’s a grain of technical truth to that, it’s still pretty impressive.

Prompting

While I think the term “prompt engineering” and its related “Prompt Engineer” title are vastly over-inflated (with the latter being along the lines of “Sanitation Engineer”), you do have to be detailed and specific — just like you would be if you were submitting a question on Stack Overflow.

Unlike Stack Overflow, you won’t be chastised for asking a question that’s been asked before nor for not reading the FAQ; you won’t be questioned as to why on Earth you would want to do that; you won’t get a condescending answer or comment; you won’t not get an answer; and you’ll get an answer in several seconds.

While all that sound great, the answer you get might be completely wrong. A couple of times, Gemini said I should use an API function that doesn’t exist. After me pointing that out, it did correct itself, however.

Also, the answer you get is often only based on your exact question. It’s probably best if you think of using AI like using a monkey’s paw lest you get responses that technically satisfy your request, but with unintended consequences.

For an example, here’s my very first prompt to Gemini to create Tidy:

Write a program using clang's API that, given either a C or C++ source file, prints every symbol (macro, constant, function, type, variable, etc.) and the file and line number where that symbol was declared, or "unknown" if it can not be determined.

And it printed out a few screens’ worth of C code showing what file from Libclang to #include, what functions to call to initialize and clean-up Libclang, how to parse a source file and iterate over its abstract syntax tree (AST). Sure, I could have ready Libclang’s tutorial, but Gemini’s response was tailored to what I wanted to do.

Of course the program it generated was nowhere near complete. It didn’t handle command-line options, configuration files, errors, colored output, or all the corner cases when analyzing C or C++ source files. These things would be fleshed out in the coming weeks. But I was off to a good start.

Mundane Tasks

One other nice thing about using AI is that you can have it do mundane code that you could do, but don’t want to be bothered because the code is boring. For example, Tidy needs to pre-scan command-line arguments looking for those preceded by -Xtidy to split them into two arrays: those that are Tidy-specific and those that should be passed along as-is to Libclang. Here was my prompt:

Given argc and argv for main in a C program, there are two types of options: (1) An -Xtidy option, that is an option with -Xtidy preceding it; and (2) Just an option. "Option" is either a short option or a long option.

I want to manipulate argc and argv such that all options that are preceded by -Xtidy are split off into a separate tidy_argv array along with tidy_argc leaving argv stripped of all -Xtidy and the immediately following option.

And it just did it correctly in several seconds, all without me having to debug the inevitable off-by-one errors had I written the code.

Experience

If you’re an experienced programmer, you’re able to look at the generated code and quickly tell whether it looks right. So for an experienced programmer, AI is quite the accelerant; if you’re an inexperienced programmer (or not even a programmer), then I suppose you’re at the mercy of the monkey’s paw.

Finding Bugs

Another thing you can do, when you’re finished with a particular .c or .cpp file, is upload it and ask, “Can you find any bugs in this code?” And it does. Of course you can’t tell whether it found all bugs, so you should still write unit and other tests.

It can of course report false positives as it did with my c_chan project. It thought my use of a particular mutex was wrong. Granted, it was a atypical way to use a mutex that perhaps would have also confused human programmers. But it also did find a few other real bugs including uninitialized structure members and noticing that the tv_nsec member of a timespec structure needs to be in the range 0–999999999, so I should add code to check for overflow.

What the AI Did Not Do

Like many codebases, the bulk of the code isn’t directly involved in doing whatever the main point of the codebase is. In Tidy’s case:

• Only 18% has to do with scanning for #include directives and mapping symbols to include files.
• 26% is just “utility” code (small functions that make writing the rest of the code easier).
• 25% is just for parsing configuration files (syntactically parsing TOML, semantically interpreting it, checking for errors, printing error messages with exact line and column).
• 15% for generic data structures (dynamic arrays, red-black trees).
• 9% for command-line parsing.
• 8% for test.

I used AI to assist with (not write) code using Libclang, not for utility code nor configuration files, only a tiny bit with the command-line (to split arguments), and not for test.

Implications

While my intent for this blog post is only to describe my impressions of AI for software development for me, I’m probably going to get asked in the comments what my think the implications are for software development as a whole, so I might as well just address that now.

While there certainly has been and will continue to be a lot of hype about AI, as I’ve said, it’s helped me a lot with Tidy. It’s a game-changer for software development. In the hands of an experienced programmer, it’s an accelerant. Since I’m no longer an inexperienced programmer, I can’t directly comment on what AI use is like for inexperienced programmers. It’s very likely also an accelerant, but inexperienced (or non-) programmers can’t spot wrong or security-omitting code when they see it. There will be more data breaches and lawsuits for sure.

You’ve undoubtedly heard (and perhaps unfortunately directly experienced) the rampant layoffs at tech. companies. As I commented, yes, there have been lots of layoffs, but I think AI is being used as a scapegoat to make CEOs not sound incompetent and companies not sound in trouble. The reasons for layoffs are the same as ever:

• The company stupidly over-hired in general.
• A new product or service didn’t pan out.
• The company is in financial trouble.

All of which make the CEO look incompetent. Instead by blaming AI, it makes the CEO look savvy by simultaneously adopting cutting edge technology while also cutting expenses — both of which are rewarded by Wall Street.

To that, I’d also add the following reason for layoffs. Incompetent CEOs:

• Might actually believe all the hype and think they can replace developers by AI.

Consider the following scenario. You’re the CEO of a tech. company that sells a flagship software product with a large customer base. You have 100 developers on staff. On average, you add one new feature and fix 10 bugs per month. You’d like to add more features and fix more bugs per month, but can’t afford to hire more developers. Now along comes AI and you have two options:

1. Continue to add one new feature and fix 10 bugs per month, but with fewer developers and expense.
2. Add two or even three new features and fix 30 bugs per month with the same number of developers using AI as an accelerant.

What sane CEO chooses option 1 thinking, “We’re happy with our current pace of development as are our customers”? Option 2 makes both existing customers happier and likely will be better at attracting new customers. Indeed, there has been some research that supports this.

So if you’ve been affected by a layoff, perhaps you can take some comfort (schadenfreude) in the likelihood that the CEO is incompetent.

Conclusion

AI is very likely here to stay. It’s clearly a developer accelerant. Sure, there will be shake-ups, buy-outs, and a few bankruptcies, because, at least currently, running AI hemorrhages money. Making AI profitable is another matter. Big players like Google that have an alternate cash cow (advertising) can fund AI indefinitely; Anthropic is only just now turning a profit for the first time ever. It’s unclear whether they’ll be able to continue to turn a profit. The next few years will tell.