Using GitHub Copilot in a whole bunch of teams and projects, this is what we discovered

At BJSS we are super excited about new AI tooling coming out. We have been trying out various tools, however, the GitHub Copilot is a tool that we felt could give engineers instant benefits, we were also excited to see if the improved velocity improvements that GitHub has been advertising were true.

At the time of writing GitHub claims Copilot can help you achieve 55% faster coding. We of course know that being a software engineer is not just about coding, so we were not expecting a 55% efficiency, but we were expecting some exciting pace.

We gave Copilot to approximately 40-50 developers over 4 different clients. Before each team got going we gave them a feature overview and demo so that they could get up and running as fast as possible.

We then had biweekly check-ins and retrospectives to find out how developers felt that their work had been impacted by using AI-based tooling.

So what did we learn?

Here is some of the feedback the engineers gave us. Here is a picture of what the engineers told us by category.

Test Code

1. Creating new unit tests quite quickly
2. The few times I asked it to write some test code it gave me something that did cover the test but was very poor and difficult to understand
3. Was good at ‘generating’ unit tests that were similar to other tests in the file (eg. an undefined case test)

Productivity

1. It was Useful for quickly understanding blocks of code
2. Useful in suggesting code snippets that speed up productivity
3. It did not help much whilst working on bug tickets
4. Extra time was needed to determine if the suggestion is helpful
5. Useful for quickly changing simple patterns
6. Suggestions were often poor and stopped me from using IDE suggestions
7. Seemed to be a handy paste without having to find the thing to copy.
8. Existing IDE tools provide alternative useful functionality (not always compatible
9. Would occasionally be able to produce a function after writing a description for it, but would then have to analyse the function in detail to ensure it was what I expected
10. I cannot go back to life without Copilot, the accuracy when it comes to refactoring code has helped tremendously
11. I find it really useful for repetitive tasks which allows more time on the complex stuff
12. The main benefit I’ve had is when it autocompletes simple code that takes little brain power but sinks time to write. 13. Sometimes it’s quicker to write what I want code to do in plain English than write the code itself, and then with the text prompt Copilot takes care of the rest.
13. It’s a huge relief on cognitive load
14. It's good for repetitive tasks when it works. I've found it's sometimes quite slow for me and sometimes doesn't give suggestions when you expect it to.
15. I have found it super useful for updating pipeline files as well - if I add a bunch of vars, when i go to call those vars in the pipeline it remembers them and suggests them nicely

Quality

1. No effect on quality because the only suggestions I accepted were patterns I was already using
2. Code generated was of similar quality to existing code…
3. Handy to test on different approaches/patterns

Features and Integration

1. Chat is more useful than code complete
2. Chat was unable to help me with a specific issue
3. The chat not being available for JetBrains meant I couldn’t use it to all of its abilities
4. Useful feature: explain this
5. I disabled it in WebStorm but try it occasionally in VSCode.

Summary

From one snapshot we took, there was:

• 14 Positive sentiments
• 8 Negative sentiments
• 6 Neutral sentiments

Overall the sentiment was positive....

Users felt that chat features, explain, etc were of high benefit.
However, users of JetBrains/IntelliJ IDEs had a worse experience due to lack of chat feature available at the time and native auto-complete being missed.

GenAI tools were particularly useful for writing unit tests, however, were not as useful for bugs, migration of code, newer coding standards (eg. Vue 3), or emulating current repo coding styles. Code quality was unaffected but required verification and sometimes fixing to ensure it aligns with current standards and practices.

Closing thoughts

I must admit I was a little disappointed that this was not quite the HUGE success that I was anticipating, however, engineers did feedback that they missed the tool when they no longer had access to it.

The biggest gaps seemed to be a lack of understanding of the business context, project coding styles, and also an understanding of particular versions of language and frameworks.

Saying this, GenAI dev tools are constantly improving. Today I learned Amazon Q developer is now available as a general release. I am also keen to use GitHub Copilot Enterprise which also promises to address some of these concerns.

We also realised that it is not merely having a tool that speeds up AI-driven delivery, but changing and improving workflows whilst using GenAI tooling is also imperative.

We are working on this right now which we will share with you more soon!

Going forward

At BJSS we are confident that these tools are here to stay and are likely to radically change how we create software. We will continue to make assessments of new tools as they come out and will publish some of our findings. We are really excited about the productivity gains our teams will see in future!