DEV Community: Erika Dietrick

Answering The Top 10 Questions I'm Asked About GitHub Copilot

Erika Dietrick — Mon, 09 Sep 2024 12:06:00 +0000

The advent of AI coding assistants has been somewhat surreal. Suddenly, after years of raiding Stack Overflow, dog-earring textbooks, and dumping all of my deepest darkest secrets to a rubber ducky, we have tools now that are smart enough to generate code for you.

This sounds great in theory, but in practice, plenty of questions remain to be answered.

(...And even if you're eager to jump in, how does one even know which coding assistant to choose?)

Well, we all have to start somewhere! I initially settled on learning GitHub Copilot - a leading AI coding assistant on the market (at least, as of February 2024, when my journey all started).

Earlier this year, I went on a winding journey of trial and error so that I could ultimately, and definitively, answer one question: "How useful or reliable is GitHub Copilot really?"

I gave a Cisco Live talk on this for your enjoyment, but in this article, I'll be answering the top 10 questions I get asked most frequently about adopting GitHub Copilot.

Let's dive in!

1. Are they training on your data?

If you tend to mistrust corporations and their handling of our data, I won't try to convince you otherwise. However, I can point you to GitHub Copilot's Responsible AI policy, which at this moment states that, "The model that powers Copilot is trained on a broad collection of publicly accessible code, which may include copyrighted code, and Copilot’s suggestions (in rare instances) may resemble the code its model was trained on." In settings, you can also disable prompt and suggestion collection.

2. Why Copilot over another coding assistant?

LLMs are constantly improving and competing with one another to be "the best" or gold standard. In most cases, I do not specifically advocate for using Copilot, but for using a coding assistant that was trained for this specific purpose so as to improve the accuracy of suggestions. Beyond this, whether or not you should use Copilot is likely "it depends." Are you working in GitHub? You may appreciate the GitHub-specific features. Do the alternatives you're considering work within your IDE? If not, Copilot will provide more contextually relevant responses.

3. Should new developers be using Copilot?

Yes, but after learning foundational concepts and writing their first functional app/script. While many may shudder that I would suggest a novice use an AI coding assistant, what I actually preach is that beginners use Copilot to learn and ask questions via Copilot Chat. Students already heavily rely on Google, and in my experience, Copilot Chat far exceeds Google in terms of quality of result and speed to finding a good result.

4. Do you have to pay for a subscription?

Yes. The subscription for individuals is currently $10 USD/month, but anyone can do a free trial, and it appears to be free for certain populations (i.e. students). There are also packages for Business ($19) and Enterprise ($39).

5. Is it just intelligent autocomplete?

The autocomplete is great, but it really just scratches the surface of why you would benefit from using Copilot in your workflow. Copilot can assist with document and test generation, with debugging, and with refactoring (i.e. improved variable names). The chat functionality is phenomenal for learning and is contextually relevant to your codebase. I also go into what I call "Comment-Driven Development" in my Cisco Live talk.

6. Can I actually trust the answers it provides?

It depends on what you're trying to do as well as what you mean by "trust." If by trust you mean blindly generate and accept code suggestions, the answer, at this time, will always be no. To boil down a long talk - the smaller the task that you're asking Copilot to complete, the more likely it is to generate something helpful. But at the end of the day, LLMs are using advanced machine learning algorithms to make predictions; no matter how effective those algorithms are, there is always a chance that the predicted - and provided - response is wrong.

7. Will it create entire applications from scratch?

No (at least, not at the time of writing). I've meticulously tested various prompts and - except in very basic use cases - was unable to scaffold entire scripts or apps without the need for significant review and modification. In my opinion, this makes code scaffolding not worth trying at this point in time.

8. Do you have to use it with GitHub, or can you use another repo?

Copilot works within your IDE; so as long as you're using a compatible IDE, you're good to go! (Keep in mind, however, that depending on the package you're subscribed to, you may be missing out on some GitHub-specific features, like pull request description and summarization.)

9. Is using Copilot just asking to introduce security vulnerabilities?

Copilot is trained on public data, meaning you do run the risk of adopting insecure code or code practices. That being said, this is really no different from when we were copy and pasting code from the internet. As you will hear me say often - what's most important here is that you are not blindly generating code without evaluating it. (And in fact, in my Cisco Live talk, I mention how Copilot can be used to _improve _code security.)

10. Do we still need to learn to code?

Absolutely. In order for code generation and assistance to be effective, we have to keep in mind that "assistance" is the key word here. You are still the brain. Your knowledge, skill, and experience are required, both for effectively leveraging Copilot and for evaluating the output it generates. (And as far as job security and all that... well... we're nowhere near being replaced quite yet.)

BONUS - 11. Does GitHub own my code if I use Copilot?

Not according to GitHub Copilot's Responsible AI policy. "If a suggestion is capable of being owned, our terms are clear: GitHub does not claim ownership."

Automate identification of uncommon DNS requests with Cisco Umbrella API

Erika Dietrick — Fri, 22 Mar 2024 18:25:37 +0000

Many corporate networks are processing massive amounts of internet traffic, which poses a monitoring and security challenge:

With so much activity, how do we know what should be investigated?
Better yet... how can we proactively identify internet traffic that is worth investigation before there's a security incident?
And most importantly... can we automate this?

We answer all of these questions for Cisco Umbrella users in this article.

Not an Umbrella customer? Check out our always-on sandbox.

What is Cisco Umbrella?
Generating an Umbrella Admin API Key
- Securely using the API Key and Key Secret
Planning our script
- Authenticating with Umbrella API
- Umbrella Reports API
- Leveraging Umbrella's global usage data
Filtering and formatting files for comparison
- Cleaning up our top_destinations.csv file
- Cleaning up the Top 1-Million CSV file
Finding uncommon domains
Removing old files
Running the script
Cisco DevNet sample code

What is Cisco Umbrella?

If you're unfamiliar with Umbrella, we colloquially refer to it as "internet security" or DNS security. While all Umbrella packages allow you to create DNS policies and access a variety of reports on your network's internet activity, other packages have additional features ranging from web policy to data loss prevention (DLP) policy to the Investigate API (proactive threat research).

The image above shows a small snippet of the Umbrella interface, in which I navigated to the Activity Search. (This is because I had just configured Umbrella, so my dashboard of the past 24 hours was empty.)

Generating an Umbrella Admin API Key

In the leftside menu, navigate to Admin > API Keys.

In the top right corner of the screen, you should then see a circular Add button. Click that, then fill out the information for creating a new Admin API Key.

The photo below provides an example of how to fill this out. What's important is that you can choose the correct scope (Reports > Aggregations: Read-Only) and that you choose an expiration date that isn't today.

When you've filled out the form above, click the Create Key button.

You'll now see an API Key and Key Secret, as shown above. Copy both of these -- they will only be displayed once.

Securely using the API Key and Key Secret

We'll need the API Key and Key Secret we just generated in order to communicate with Umbrella; but if we're using a version-controlled repository, hardcoding those credentials into the script and pushing them to the repository will expose our sensitive credentials to others.

While you can secure credentials in multiple ways, we'll create a .env file to store them in.

API_KEY=apikeygoeshere
KEY_SECRET=keysecretgoeshere

Next, to ensure this .env file is not pushed to our repository, we'll create a .gitignore file using this command:

touch .gitignore

Once the .gitignore file is created, we'll add the name of any files we want to be ignored inside the file -- in this case, only .env.

Finally, our script will need to access these credentials despite the fact that they aren't hardcoded into the script.

In Python, we'll include the following import statements to accomplish this:

import os
from dotenv import load_dotenv

Then, we'll load the credentials into the script:

load_dotenv()

# Environmental variables should contain your org's values in .env file.
client_key = os.environ['API_KEY']
client_secret = os.environ['KEY_SECRET']

Planning our script

Okay, so now we have an API Key and Key Secret from which we can retrieve DNS traffic from Umbrella. Now, we have 3 questions to answer:

What's required to authenticate with the Umbrella API?
Which API call should we be making to Umbrella?
How can we sift through the DNS traffic to determine what is "uncommon"?

Authenticating with Umbrella API

Authentication with any Umbrella API requires not just an API Key and Key Secret (which we generated earlier), but an access token, which expires after 1 hour.

In Python, we'll first need to import the requests library to make an API call:

import requests

Then, we write a function that generates an access token using the correct API endpoint. Because we'll run this script on a weekly basis, but the access token only lasts an hour, we'll make sure we call this function first each time the script runs.

# Relevant v2 Umbrella API endpoints
base_url = "https://api.umbrella.com"
access_token_endpoint = f"{base_url}/auth/v2/token"

# Generate new access token as these expire after 1 hour. Requires a valid and unexpired Umbrella API Key and Key Secret.
def generate_access_token():

    response = requests.post(url=access_token_endpoint,auth=(client_key,client_secret))
    access_token = response.json()['access_token']

    return access_token

Umbrella Reports API

By reviewing the Umbrella API documentation, we see that using the Reports API will retrieve information about the traffic coming through the Umbrella network; specifically, the getTopDestinations endpoint.

We'll first create a variable for the Top Destinations endpoint:

# Relevant v2 Umbrella API endpoints
base_url = "https://api.umbrella.com"
access_token_endpoint = f"{base_url}/auth/v2/token"
top_destinations_endpoint = f"{base_url}/reports/v2/top-destinations"

Next, we define the headers and parameters (as defined in the API documentation) before making the Top Destinations API call:

# Get the Top Destinations visited from 7 days ago until now. Top 1000 domains are returned.
def get_top_destinations(access_token):  

    headers = {
    "Authorization": "Bearer " + access_token,
    "Content-Type": "application/json",
    "Accept": "application/json"
    } 

    params = {
        "from": "-7days",
        "to": "now",
        "offset": "0",
        "limit": 1000
    }

    top_destinations_request = requests.get(top_destinations_endpoint, headers=headers,params=params)
    top_destinations = top_destinations_request.json()

    return top_destinations

You'll notice that we've set parameters to pull Top Destinations from 7 days ago until now. (Specifying now is a supported option based on documentation.)

Leveraging Umbrella's global usage data

The best way to determine what is abnormal or uncommon? Find a way to establish a baseline or "normal."

Fortunately, Umbrella posts a Popularity List daily. According to Cisco Umbrella, this list "contains our most queried domains based on passive DNS usage across our Umbrella global network of more than 100 Billion requests per day with 65 million unique active users, in more than 165 countries."

1-million of the most commonly queried domains should be a sufficient baseline of "normal."

We'll make a GET API call to retrieve the Umbrella Top 1-Million. (Yes, I'm realizing now I should have stored the URL in a variable to be consistent.)

# Download the Umbrella top 1 million destinations, unzip file, format file. 
def get_top_million():

    # API call to get Umbrella Top 1 Million as a zip file
    get_top_1million_zip = requests.get("http://s3-us-west-1.amazonaws.com/umbrella-static/top-1m.csv.zip")

This function isn't yet complete. Let's discuss what else needs to be considered before we finish this function.

Filtering and formatting files for comparison

We've now made two API calls: one to retrieve the top 1000 destinations seen by our Umbrella network over the past week, and one to retrieve the Top 1-Million domains seen by the Umbrella network globally.

We'll want to clean up these files so that they're easier to compare and the resulting file is meaningful.

Cleaning up our top_destinations.csv file

Our CSV file successfully returns our network's top destinations, but not all of those destinations are domains -- we'll also see IP addresses.

While those IP addresses may be worth investigating, they cannot be compared to Umbrella's Top 1-Million, which is a list of domains only. For this reason, we'll want to filter out IP addresses.

First we'll import Python libraries that will help us check for IP addresses and handle CSV files:

from IPy import IP
import csv

Next, we'll add logic that checks if something is an IP address.

def isIP(str):
    try:
        IP(str)
    except ValueError:
        return False
    return True

We'll then incorporate our logic in a new function that takes our csv file as a parameter. For each line of the csv, if the content isn't an IP address, we'll add it to a list called destinations_list.

After that, we'll write that "domains only" list to a new csv.

# If destination in Top Destinations is a domain, write it as a new line in a CSV called top_destinations.csv.
def top_destinations_to_csv(top_destinations):

    destinations_list = []

    for destination in top_destinations['data']:
        if not isIP(destination['domain']):
            destinations_list.append(destination['domain'])

    top_destinations_csvfile = open('top_destinations.csv', 'w')

    with open('top_destinations.csv', 'w', newline='') as top_destinations_csvfile: 
        filewriter = csv.writer(top_destinations_csvfile, delimiter=',',
            quotechar='|', quoting=csv.QUOTE_MINIMAL)
        for destination in destinations_list: 
            filewriter.writerow([destination])
        top_destinations_csvfile.close()

    return top_destinations_csvfile

Cleaning up the Top 1-Million CSV file

Our top destinations file is now just rows upon rows of domains, but how about our Top 1-Million file?

When we made our GET API call, we received a zipfile in return. We'll first import a library to work with that zipfile:

import zipfile

We then need to write that zip file to disk, create a fresh csv file to save the cleaned up version to, and unzip the file.

# Download the Umbrella top 1 million destinations, unzip file, format file. 
def get_top_million():

    # API call to get Umbrella Top 1 Million as a zip file
    get_top_1million_zip = requests.get("http://s3-us-west-1.amazonaws.com/umbrella-static/top-1m.csv.zip")

    # Write the zip file to disk
    open('top-1m.csv.zip', 'wb').write(get_top_1million_zip.content)

    # Create a new CSV file to write the cleaned up Top 1 Million to
    top_1million_csv = 'top-1m.csv'

    # Unzip the file
    with zipfile.ZipFile('top-1m.csv.zip', 'r') as zip_ref: 
        zip_ref.extractall('.')

The unzipped Top 1-Million file looks something like this:

We need to remove that rank order so that we can just compare domains. To do this, we import a Python library to help us format the csv:

import pandas

Finally, we drop that rank order column:

    # Removing rank order in first column so that we can compare domains to Top Destinations. 
    top_1million_csv = pandas.read_csv('top-1m.csv')
    first_column = top_1million_csv.columns[0]
    top_1million_csv = top_1million_csv.drop([first_column], axis=1)
    top_1million_csv.to_csv('top_1million_csv', index=False)

    return top_1million_csv

Finding uncommon domains

We're finally ready to find those uncommon domains! To do this, we compare both CSVs by opening and reading each line of the files. For each domain in our network's top destinations that does not appear in the Umbrella Top-1 Million, we write that domain as a line in our final CSV named uncommon_domains.csv.

# Compares each domain in top_destinations.csv to top-1m.csv (Umbrella's Top 1 Million) and returns any domains that are not in the Top 1 Million.
def find_uncommon_domains():

    top_destinations_file_path = "./top_destinations.csv"
    top_1million_file_path = "./top_1million_csv"

    uncommon_domains_file_path = "./uncommon_domains.csv"

    with open(uncommon_domains_file_path, 'w') as uncommon_domains_csv:

        top_destinations = open(top_destinations_file_path).readlines()
        top_1million = open(top_1million_file_path).readlines()

        for domain in top_destinations:
            if domain not in top_1million: 
                uncommon_domains_csv.write(domain)

    print(f"Uncommon domains have been written to uncommon_domains.csv in your current directory.")

    return uncommon_domains_csv

Removing old files

This part is a nicety, but let's remove all of the CSV files besides the resulting uncommon_domains.py to avoid confusion:

# Clean up files used to determine uncommon domains.
def clean_up_files():
    os.remove('top-1m.csv.zip')
    os.remove('top-1m.csv')
    os.remove('top_destinations.csv')
    os.remove('top_1million_csv')

Running the script

We'll write a main function that runs when the script runs, calling the relevant functions in order:

# Main function
def main():

    access_token = generate_access_token()
    top_destinations = get_top_destinations(access_token)
    top_destinations_csvfile = top_destinations_to_csv(top_destinations)
    cleaned_top_1million_csv = get_top_million()
    uncommon_domains_csv = find_uncommon_domains()
    clean_up_files()

    return uncommon_domains_csv

if __name__ == "__main__":
    main()

You can find an example of the output provided in the resulting uncommon_domains.csv below.

Cisco DevNet sample code

This is an official submission in Cisco Code Exchange, including a suggested use case, available here. You can also access the sample code directly on GitHub.