Does GitHub Copilot actually raise bugs in code by 41%?

"Many developers say AI coding assistants make them more productive, but a recent study set forth to measure their output and found no significant gains. Use of GitHub Copilot also introduced 41% more bugs, according to the study from Uplevel, a company providing insights from coding and collaboration data."

I've seen this CIO article making the rounds on social, so I sacrificed my email address to get the Uplevel report this article is sourcing. It's two pages, mostly filler, with almost no data in it. Uplevel looked at about 800 devs total, only 351 of them in the TEST group with Copilot. I did some sample size calculators and if we assume 25 million devs around the world this is probably a decent enough sample size. They measured a 3 month span at the beginning of 2023 as a baseline and then again in 2024.

During this time PR volume in each period was measured. The report was not based on actual usage of Copilot, though, only on whether the devs had ACCESS to Copilot. In the end, the determination is that the devs with access did about the same as devs without access, but with more bugs.

I'm not saying the data is wrong, but looking at the depth of the study, amount of information in the report, and it looks to me like it might be a bogus attempt to build some buzz to sell Upsell product vs their competition.

If you hate what AI is doing to the developer experience and overall delivery, this type of article is going to support your already-held beliefs. That is what they are counting on.

So I have a few comments on this:

• Question it. When you see somebody write an article like this, assume there is an angle and go and read the actual source. Find out what it actually is about and not just the spin. This counts whether it is telling you AI is great or AI is bad!
• Feeling supported is good. If AI coding tools are working for you and are helping you feel better and you FEEL more productive, that might be just as important as actually being more productive?
• Learn to critically analyze. Don't blindly accept what AI tells you. Learn the skills to understand your code and debug. Test (and I mean learn to test your own stuff before you PR, not leave it all to others). Writing a line of code is one of the LEAST important things you do as a dev.