DEV Community

loading...
Cover image for Some Experiments with GitHub Copilot

Some Experiments with GitHub Copilot

Rahul Banerjee
Comp Eng Student @uoft | My opinions are my own Add me on LinkedIn: https://www.linkedin.com/in/rahulbanerjee2699/
・4 min read

I recently got my hands the GitHub Copilot extension for VS Code and it's amazing (borderline scary)

The examples below are in Python

Function to get Pokemon Data

pokemonGithub

  • Was able to auto-complete my comment as well. Initially I never intended to save it to a json file.
  • Added some in-line code comments
  • Used an external library (requests) to make a request. Used Json to save the data.
  • Chose a decent file name on it's own
  • Was able to find a data source, surprisingly someone's github repo

Function to zip and unzip file

zipUnzip

  • Was able to import all the necessary libraries. Although it did import shutil and never used.
  • Used zipfile library to unzip/zip
  • For the second function, it didn't import the zipfile liibrary.
  • Was able to use the right parameters

Building a tictactoe game

tictactoe

  • Generated 64 lines of code
  • Was able to write functions for various purposes
  • Knew the winning combinations of a tictactoe board
  • Added Error Handling
  • Added print statements and ability to take input from user
  • Logic to check the result of the game

Although it wrote all the sub-functions, it never invoked them to actually build a playable game

Crypto Price

crypto

  • Added a parameter
  • Used a crypto api to get the data
  • Was able to return the correct column for the price

Build a streamlit app to display Github Repos

githubStreamlit

For this, I had to write multiple comments and it actually felt like I was pair-programming with the Copilot. However, most of the code was generated by Copilot.

  • Was able to get data from github api
  • Since I mentioned popular, it sorted the repos based on 'stars'. It is mind-boggling how it was able to relate 'stars' to popularity.
  • It was able to use an external library streamlit (streamlit is used to build web apps)
  • It also added a title and text to be displayed in the web app
  • It re-used the previously created function
  • For most parts, it was auto-completing my comments as well

General Observations

  • The variable and function names are pretty explanatory
  • Added relevant in-line code comments
  • Was able to use external libraries
  • Was able to get data from various data sources
  • The format of the code was clean with proper indentation and line breaks
  • It took me quite a few tries (trying out different comments) for it to actually use streamlit and build a simple app. In the end, I ended up importing the library and it started generating code using the library. However, sometimes it was able to generate code on its own as well
  • When trying to get the pokemon/crypto data, it often made suggestions that used beautiful soup to scrape the data. Web Scraping is not always the best option and in some cases, you might even end up breaking some laws.

Some weirdness

  • Sometimes it acted weird. For instance, at times the generated code contained local files paths for some other users,eg "Users/Projects/......."

  • I tried getting suggestions for a variable named api_key and it actually suggested a string with random keys. Ofc, it might actually be random but yeah that was weird.

  • At times it generated receptive code. Eg: When I was trying to generate code for streamlit, it generated the same two lines over and over again

  • It suggests a bunch of unnecessary imports at times

  • For some reason, it kept on generating code that used Dash although I specifically mentioned streamlit


My Views and a few questions

My Views are my own

It is sure going to improve a software developer's productivity. However, I don't see it replacing a software developer. Copilot often generated non-sensical and repetitive code. It didn't import the necessary libraries at times as well. It is basically like Kite or TabNine on steroids. 

A good analogy I can think of is Google Translate. It's been there for years but it has not replaced the need for an actual translator. You could translate an article from English to Japanese in a few seconds. However, you would still need somebody fluent in both languages to ensure that the translation is grammatically correct and it delivers the same message as the original article.

Another issue I can think of - Who would be liable for the code? If I used GitHub to generate some code and later get sued by someone for some reason. Can I put the blame on GitHub? GitHub will most likely make users agree to some terms and conditions which prevent them from being sued. So we would actually need somebody experienced in software to ensure that the generated code is safe to use.

Although GitHub Copilot is good for new projects, I am not sure if it would be as useful when working with an existing codebase. In an existing codebase, it would have to follow the existing coding style and be able to re-use already written code. I have not tried working with Copilot in an already existing project so I can't comment much

Assuming it is constantly learning as more users use it, how would it distinguish 'good' code from 'bad' or 'spaghetti code.

Conclusion

One thing GitHub copilot can guarantee is that it is going to make coding and software, in general, more accessible 💯

What are your thoughts on the Copilot? Did you try out anything cool with it? How do you think it is going to affect the Software/Data Science Industry? Let me know in the commentsPublishSave draftRevert new changes


Discussion (14)

Collapse
jonrandy profile image
Jon Randy • Edited

Looks very similar to TabNine which I've been using for ages - although Copilot seems to be totally OTT in comparison. TabNine's suggestions are fairly conservative, but incredibly helpful - it really does feel like it's reading your mind sometimes.

This level of completion though - especially the very large ones, is an absolute disaster for the levels of competency in the industry, and very bad for people who are learning. It's almost an automated StackOverflow copy-pasta machine.

Possibly useful in the right hands, but - as they say - with great power, comes great responsibility

Collapse
rahulbanerjee99 profile image
Rahul Banerjee Author

Agreed, I wouldn't recommend this tool to people who are learning. The generated code is just going to seem like a foreign language.

And as you aptly put it "with great power, comes great responsibilities".

Collapse
seyal84 profile image
AmjD(AJ)

Not really a bad tool for learners as well.

Thread Thread
jonrandy profile image
Jon Randy

An awful tool for learners

Thread Thread
seyal84 profile image
AmjD(AJ)

Lol not really. Depends what kind of a learner you are and how good are at learning new technologies.

So you support people using stackoverflow but not such tools lol. Double standards

Thread Thread
rahulbanerjee99 profile image
Rahul Banerjee Author

If you are learning a language like French, it’s okay to use a French to English Dictionary for translations and grammar rules and stuff. But if you just use google translate and claim you are learning French, lol you are in for a surprise.

And no, it doesn’t depend on what kind of learner you are. It’s a good short term solution to get your code working and stuff but in the long run you won’t be learning much.

Thread Thread
rahulbanerjee99 profile image
Rahul Banerjee Author

To add to my points above, when you use something form stack overflow, more often than not there is an explanation for the code and why it works. Copilot does no such thing.

Thread Thread
seyal84 profile image
AmjD(AJ)

True but not true.

I think being an experienced programmer you are just being over defensive.

If you think that in today's IT such tools aren't good for new learner then probably you aren't even aware of how quickly IT is changing.

New learners can use such tools to improve their codes and learn from it but these aren't just to fully rely on it.

On stackoverflow people post full codes or atleast solutions so it isn't as diff from copilot if you talk about general context

Collapse
andreidascalu profile image
Andrei Dascalu

@seyal84 no double standards at all. On SO you get discussions and a lot of the offered code comes with explanation (often with a degree of depth outright or through the discussions there).

Collapse
justjordant profile image
Jordan Taylor

So true, I do feel like this is just a more advance autocomplete like Kite, Tabnine. But as you said this can lead to possible some cases where people would be more relient on this, then actually know line by line that is going on, especially for those who are learning.

Collapse
shnydercom profile image
Jonathan Schneider (shnyder) • Edited

Have you tried googling some of the generated code? The tictactoe seems like an example where the AI ran on the natural language part (so the code comments) and the associated code is copied. I would actually prefer an AI to recommend me the full link to SO or GH so I can read the rest of the comments, or even contribute to OSS if the suggestion is faulty. All the copilot examples I saw don't modify the syntax tree much

Collapse
rahulbanerjee99 profile image
Rahul Banerjee Author

I haven’t searched it up but I will. I was under the impression that the tool was actually generating code but I think instead of generating code, the tools just suggests code snippets from public repos. So it’s just an over-hyped search engine Lol.

Collapse
shnydercom profile image
Jonathan Schneider (shnyder)

yeah I got the same feeling. Most devs think naming variables is hard, and this tool just has too many "good" names. Now "good" names mostly come from review processes, where the functionality was unclear before. Let us know what you find out :)

Collapse
cybeartron profile image
Cybear Tron

Salute!!!! Nice one here!! Thanks I wanted to know about the copilot myself this helped !!! :D