DEV Community: Emily Chen

Using GitHub Actions to Test Student Code Submissions

Emily Chen — Mon, 11 Jul 2022 02:37:06 +0000

Background

This blog is based on Bit Project's CounselorBot, which you can test out yourself!

We currently have two available courses:

Get started with this video tutorial or this walkthrough page.

Overview

GitHub Actions are often used for automated code operations, usually automatic deployments. However, they also have another handy-dandy use case: checking students' code submissions so you (being lazy) don't have to run them yourself!

In a future tutorial, we'll also talk about how output from GitHub Actions can be automatically commented on students' pull requests to provide feedback on what went wrong.

Outlining Our Plan

All of the workflow files used as examples in this tutorial can be found here.

Each "step" in our curriculum requires a different test. When a student (the user) commits to the repository, we need to...

Figure out which workflow file to run
Figure out which step the student is on

...in order to run the correct test.

Alongside actually running the tests, we will also be using GitHub Actions to determine the answers to these two questions.

Writing the YAML File

name: Getting Started
on:
  push:
    branches:
      - hello

The first section of the YAML file gives the GitHub Action a display name and configures when the action should run. In this case, it'll run when a commit is pushed to the hello branch.

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout Code
        uses: actions/checkout@v2

      - name: Setup Node Environment
        uses: actions/setup-node@v2
        with:
          node-version: '14'

      - name: Install Dependencies
        run: |
          npm install minimist
          npm install node-fetch

The next section provides other steps to "set up" our environment. We will mainly be using node to test code; npm dependencies can also be installed if they will be used in the tests.

      - name: Get Count
        id: vars
        run: echo ::set-output name=count::$(node ./.bit/scripts/getProgress.js --repo=${{github.event.repository.name}} --owner=${{github.repository_owner}} )

Our first "unique" step in this workflow involves using this script to retrieve the step that the student is currently on. This "count" value is an integer that correlates with steps in our curriculum.

Note: the echo ::set-output name=count:: part sets the output of the file that runs to a local variable named count, which will be accessed later with steps.vars.outputs.count.

Testing Different Types Of Submissions

Now that the environment for our workflow is set up, we can run different test files, which provide feedback on student code. With GitHub Actions, you are essentially just running commands on a terminal. This opens up options from running JavaScript files to testing webpages to calling Azure Functions.

All test examples used in this tutorial can be found in this directory and this directory.

Testing Program Files (JS, Python)

      - name: Step 1
        if: ${{steps.vars.outputs.count == 1 && github.event.head_commit.message != 'Update progress'}}
        run: |
          node .bit/tests/test.1.2.js --repo=${{github.event.repository.name}} --user=${{github.repository_owner}}

This step can be added onto the steps section of the YAML file for workflow configuration. A conditional statement is utilized to only run this test if the "count" value is correct. The command node .bit/tests/test.1.2.js would then execute this test.

try { hello = require('./../../week1/helloworld.js') }

The test first attempts to import the student's code in the repository.

let helloworld = hello()
let test_output = "Hello World"

If successful, it will attempt to execute the imported functions.

await functions.throwError(`Got: '${helloworld}', was expecting: '${test_output}'.`, user, repo)
console.log(`Got: "${helloworld}", was expecting: "${test_output}".`)
process.exit(1)

Note: Source code for the functions.throwError() method can be found here. It includes functions used to provide feedback to students and reduce repetition in tests.

Depending on its success, the test will provide feedback by throwing errors in the workflow.

Testing Webpages with Cypress

Cypress is a powerful testing tool that allows a simulation of a user's actions on a website.

describe('Testing Bunnimage', () => {
    it('Testing Week 4 Step 1', () => {
        cy.visit('bunnimage/index.html')
        cy.get('input[type="text"]').type('console.log("hi yall")')
        cy.get('input[type="button"]').click()
        cy.get('#output').contains('console.log("hi yall")❤️')
    })
})

A simple example of a test as shown here simulates typing console.log("hi yall"), clicking the specified button on the page, and checking to see if the output equals console.log("hi yall")❤️. In the workflow's output, feedback from Cypress is provided.

Testing APIs

Student coded endpoints can also be tested with HTTP requests run in the workflow. The key to this method is asking students to add "repository secrets" that can then be accessed during the workflow as environment variables using the below syntax.

      - name: Step 12
        if: ${{steps.vars.outputs.count == 12 && github.event.head_commit.message != 'Update progress'}}
        env:
          MORSE_ENDPOINT: ${{ secrets.MORSE_ENDPOINT }}
        run: |
          npm install node-fetch
          node .bit/tests/test.1.8.js --repo=${{github.event.repository.name}} --user=${{github.repository_owner}}

This example accesses the student's MORSE_ENDPOINT secret (this can be later accessed in the code with uri = process.env.MORSE_ENDPOINT).

try {
    const resp = await fetch(uri + "&plaintext=ilovebitproject", {
        method: 'GET'
    });
    var data = await resp.text()
    let test = JSON.stringify(data)
    functions.validateResponseStatus(resp, uri)
} catch (e) {
    console.error("We're having trouble making a request to your endpoint. Try again?")
    await functions.throwError("We're having trouble making a request to your endpoint. Try again?", user, repo)
    process.exit(1)
}

First, utilizing the student's endpoint, an HTTP GET request is made to see if the endpoint is alive.

if (data.length < 3) {
    console.error("No response... Try again!")
    await functions.throwError("No response... Try again!", user, repo)
    process.exit(1)
} else if (data == answer) {
    console.info("Yay!🎉 Success - thanks for helping us on this top secret mission. Welcome to the team.")
    console.info(`We got "${answer}" with the input of ilovebitproject`)
} else {
    console.error(`YIKES! We got "${data}" instead of "${answer}". Try again!`)
    await functions.throwError(`YIKES! We got '${data}' instead of '${answer}'. Try again!`, user, repo)
    process.exit(1)
}

If a response is received, the test attempts to determine if the output of the endpoint matches the correct "answer."

As demonstrated in this test shown above, multiple test cases can be at once with feedback returned to the student as output of the workflow.

Conclusion

GitHub Actions are extremely versatile as they allow developers to create their own environment, automating tasks and running scripts. Running on repositories, GitHub Actions can easily be incorporated with other services on GitHub: from bots, to branches, and to pull requests.

This use case demonstrates the power of GitHub Actions beyond automating deployment; extending into the area of learning and education.

Creating a File Sharing and Conversion Web App with Azure Functions

Emily Chen — Sun, 20 Dec 2020 01:44:38 +0000

If you or someone you know participated in this year's AP Collegeboard Exams, you probably recognize the stress of submitting handwritten work within a small time constraint.

Bunnimage aims to help alleviate that stress for students and others working at home. It takes an image as an input on an upload page and converts it into a PDF that is available at a download page.

Overview

In this tutorial, we'll be walking through:

Creating the "Upload" page and an HTTP Trigger Function that will upload the user's image to a storage container.
Setting up an Event Grid Subscription and a Function that converts the image into a PDF and stores it again.
- This is where the API will live!
Creating the "Download" page and an HTTP Trigger Function that retrieves the correct PDF.
Optional For those who are interested, we can add another Function to delete the files and keep our containers squeaky clean.
- Note: The diagram above excludes the optional deletion feature.

You can find a sample of the final product at my Github repository.

Before we start:

Make sure you have an Azure Subscription so we can utilize the amazing features of Microsoft Azure Functions (It's free!) 🤩
Register for an account on Online Convert (with the free version), as we will be using this API convert our images
If you want to host your website somewhere, check out Repl.it, or you can just have your project run locally

Step 1: Upload the image ⬆️

Creating a Function App

We're going to have a lot of triggers in this project, so let's get started by creating a Function App! Follow those steps to create the Function App, and then create the first HTTP trigger (this will upload our image).

Note: it will be helpful if you keep track of these strings for reference later in the project: Storage account name (found in "Hosting"), Function app name, Resource Group

Before we start coding the trigger, though, we need to install some npm packages/libraries.

Click on the "Console" tab in the left panel under "Development Tools".

Inside the console (shown on the right panel), type in the following commands:

npm init -y

npm install parse-multipart

npm install node-fetch

npm install @azure/storage-blob

Tip: The Azure Storage Blob client library is going to be a key piece of the project. After all, it's about blobs!

Setting up your storage account

This is the storage account you created when creating the Function App. If you don't know what it is, search "Storage Containers" in the query box in Azure portal.

We're going to need to create 2 containers: "images" and "pdfs." Think of these as folders in the account.

You will need to upgrade your storage account because Event Grid Subscriptions will only work with a v2 version. Follow this tutorial to upgrade it.

**Writing our First Azure Function to Upload an Image**

⬇ Some housekeeping...

For the function to work, we have to initialize the packages/libraries we installed in the beginning of part 1.
Take note of the process.env value being assigned to connectionstring in the code below (Line 3). Use this tutorial to add in your own secret strings from your storage container.
- The storage container is the one you created when you started your Function App. Navigate to it and find your secret strings here:
- Keep these safe, and use the connection string in the corresponding variable in the code.
- Note: You'll need to store other strings in environment variables later on in the tutorial

Let's start just by initializing a few variables we'll need.

⬇ The main block of code

Notice that we are able to name the file with the user's username in line 10 by receiving it from the header.
- Later on in the JS, we will send the username in the header of the request.
The parse-multipart library is being used in lines 4-11 to parse the image from the POST request we will later make with the frontend; refer to the documentation linked above.
Some if-else logic is used from lines 13-22 to determine the file extension.
We then call the uploadBlob() function in line 24.

⬇ Uploading the image blob to the "images" container

Notice the uploadBlob() function! This is what uploads the parsed image to the specified "images" blob container.
- Here's a YouTube Video to help explain the handy dandy library

Frontend: The "upload" webpage

Next, I created a static HTML page that will accept the image from the user and send to the Azure Function we just coded using Javascript.

Note: I removed unnecessary sections of my code because I wanted to make the webpage ✨fancy✨, but you can see the whole thing here.

Above we have:

Input box for the username (simple but insecure auth system)
Button to submit

However, a static HTML webpage can't make a request to the Azure Function itself, which is where we're going to cook up some JS. 😯

Frontend: Javascript for interacting with the Azure Function

This block of Javascript updates the preview thumbnail while getting the picture, gets the username, and sends them both over to the function we just coded.

First, loadFile() is called when the file input changes to display the thumbnail.

async function loadFile(event){
    console.log("Got picture!");
    var image = document.getElementById("output");
    // Get image from output 
    image.src = URL.createObjectURL(event.target.files[0])
    // load inputted image into the image src and display
}

Then, handle() is called when the file is submitted to POST the image and username. The image is sent in the body, and username is sent as a header. Lines 15-30

Be sure to change the function url on Line 19 to the one of your upload image Function!

Deploy your code

Try doing it locally with the live server extension for VS Code
Try Azure Web Apps
I personally used repl.it

Update CORS Settings

This is a crucial step!! 😱 If you don't change your CORS (Cross-origin resource sharing) settings, the POST request won't work. This tells the Function App what domains can access our Azure Function.

Options:

Recommended: Change it to a wildcard operator (*), which allows all origin domains to make requests
- Be sure to remove any other existing inputs before attempting to save with wildcard
Change it to the domain you are using to host your code

Home stretch! 🏃🏻‍♀️

It's finally time to test our first step that our app will make!

Navigate to your HTML page and submit an image

Go to the "images" storage container and check to see if your image is there!
Error? Check the log in your Function

Step 2: Convert The Image 🔄

Create another Azure Function

Yep... We need yet another Azure Function. (What can I say? They're pretty helpful.) This one will trigger when the image blob is stored, then convert it into a PDF, and store it in the "pdfs" container.

However, this time, it will be an Event Grid Trigger, so make sure you select the right one!

What's the difference?

Event Grid Triggers trigger based on an Event Grid Subscription, which we will create later in this step. Our trigger will fire when a blob is stored in the "images" container

HTTP Triggers fire when a GET or POST request is made to the endpoint (function URL)

Commercial Break 📺

Let's recap:

Step 1 ✅: We created the "Upload" page and an HTTP Trigger Function that uploaded the user's image to a storage container.
Step 2: We will create an Event Grid function that converts the image into a PDF by calling the Online Convert API and will upload the PDF to blob storage.

⚠😵WARNING😵⚠ Lots of code ahead, but it's all good! I split it into sections.

First off, the Online-Convert API!

We're going to need to get another secret key, except this time from the API. Here's how to get that.
Once again, save it in your environment variables so it's accessible.
Note: This API does have restrictions on the amount of conversions during 24 hours, so just be aware that you may get an error after reaching the limit.

⬇ This convertImage() function does exactly what it's called: convert the image by calling the Online-Convert API. Here's some documentation around how to use the API with Azure Blob Storage.

async function convertImage(blobName){
    const api_key = process.env['convertAPI_KEY'];
    const accountKey = process.env['accountKey'];
    const uriBase = "<https://api2.online-convert.com/jobs>";
    // env variables (similar to .gitignore/.env file) to not expose personal info
    // check out documentation 
    img = {
    "conversion": [{
        "target": "pdf"
    }],
    "input": [{
        "type": "cloud",
        "source": "azure",
        "parameters": {
            "container": "images",
            "file": blobName
        },
        "credentials": {
            "accountname": "bunnimagestorage",
            "accountkey": accountKey
        }
    }]
    }

    payload = JSON.stringify(img);

    // making the post request
    let resp = await fetch(uriBase, {
        method: 'POST',
        body: payload,
        // we want to send the image
        headers: {
            'x-oc-api-key' : api_key,
            'Content-type' : 'application/json',
            'Cache-Control' : 'no-cache'
        }
    })

    // receive the response
    let data = await resp.json();

    return data;
}

⬇To check the status of the conversion and determine whether we can store the PDF to blob storage yet, let's use this checkStatus() function that makes a request to the same https://api2.online-convert.com/jobs endpoint, except with a GET request instead of POST.

async function checkStatus(jobId){
    const api_key = process.env['convertAPI_KEY'];
    const uriBase = "<https://api2.online-convert.com/jobs>";
    // env variables to keep your info private!

    // making the post request
    let resp = await fetch(uriBase + "/" + jobId, {
        /*The await expression causes async function execution to pause until a Promise is settled 
        (that is, fulfilled or rejected), and to resume execution of the async function after fulfillment. 
        When resumed, the value of the await expression is that of the fulfilled Promise*/
        method: 'GET',
        headers: {
            'x-oc-api-key' : api_key,
        }
    })

    // receive the response
    let data = await resp.json();

    return data;
}

Then we can use the same uploadBlob() function from before to upload our object!

After this we get to the main section of our code.

⬇It gets the blobName, calls the functions, and downloads the PDF to be stored.

The blobName is retrieved from the EventGrid subscription subject* in lines 10-11
Because the API does not convert the image immediately, we need a while loop to repeatedly check for the status of the conversion in lines 21-36
The last portion is used to download the converted PDF by sending a GET request to the URI from the completed file conversion response. Lines 43-47

What's a subject? This part of the Event response contains information about what specific file caused the Azure Function to fire. For example: /blobServices/default/containers/test-container/blobs/new-file.txt where the file is new-file.txt

Now that the long block of code is done with, let's take a look at some responses you should expect from the API.

This is what you would get if the file is still converting 🤔
Here's what you would get when the conversion is complete! (yay) 🥳

In particular, there are 3 important pieces of the output we should examine:

update.status.code: This tells us whether its done processing or not
update.output[0].uri: This gives us the URL where we can download the PDF (used in the last GET request)
result.id: Gives the ID of the file conversion "job" so we can continually check for its status

Before we can test our code, we need one last step: the trigger!

Creating an Event Subscription

When the image blob is stored in the "images" container, we want the conversion from jpg/jpeg/png to pdf to begin immediately!

Just like how "subscribing" to a YouTube channel gives you notifications, we're going to subscribe to our own Blob Storage and trigger the Azure Function.

Tip: You'll want to keep the names for your storage account and resource group handy.

Search "Event Grid Subscriptions" in the search bar
Click "+ Event Subscription" in the top left
Fill in the form to create the Event Subscription:

If it asks you for a name, feel free to put anything you want - I named it "fileUploaded"
Under Topic Types, select "Storage Accounts"
The "Resource Group" is the Resource Group that holds your storage account
The "Resource" is your storage account name

Note: If your storage account doesn't appear, you forgot to follow the "upgrade to v2 storage" step

Under Event Types: filter to Blob Created

The "Endpoint Type" is "Azure Function"

The "Function" is the function we want triggered when an image is uploaded, so the convertImage function
Tweaking some settings...
Navigate to the "Filters" tab and "Enable Subject Filtering"
Change the "Subject Begins With" to /blobServices/default/containers/images/blobs/
- This way, the subscription will not trigger when a PDF is stored in the "pdfs" container. It will only trigger when something is stored in "images."

Congratulations! You have now subscribed to the "blob created" event in your "images" container that triggers the convert image function!

Upload a converted PDF to the "pdfs" container!

Now that we've connected our Functions and frontend together with an Event Grid Subscription, try submitting another image to check if it successfully uploads as a PDF into the "pdfs" container.

If you used my code and have the same context.log()s, you should get something like this when the PDF uploads:

Step 3: Downloading the PDF on the HTML page ⬇

Now that we have a PDF stored in the "pdfs" container, how will we get the PDF back to the user? You got it right, yet another Azure Function!

Create another HTTP Trigger - this one will return the PDF download URL to the frontend when triggered.

Commercial Break 📺

Let's recap:

Step 1 ✅: We created the "Upload" page and an HTTP Trigger Function that uploaded the user's image to a storage container.
Step 2 ✅: We will create an Event Grid function that converts the image into a PDF by calling the Online Convert API and will upload the PDF to blob storage.
Step 3: We will create a HTTP Trigger function that returns the PDF to the user when triggered by the "Download" page.
Step 4: Optional If you choose, create another HTTP Trigger function and modify other code to delete the image and PDF blobs from storage containers once they are unneeded.

Azure Functions: Check if the PDF is ready to be served 🍝

⬇First, it receives the username to get the correct PDF from the header of the request, which is made by the webpage. You will see this request later on in the JS of this step.

var fetch = require("node-fetch");
module.exports = async function (context, req, inputBlob) {
    context.log('JavaScript HTTP trigger function processed a request.');

    var username = req.headers['username'];
    var download = "<https://bunnimagestorage.blob.core.windows.net/pdfs/>" + username + ".pdf";

⬇Then, using the personalized URL, it performs a GET request to check if the PDF has been stored in the "pdfs" container.

    let resp = await fetch(download, {
        method: 'GET',
    })
    let data = await resp;
    if (data.statusText == "The specified blob does not exist.") {
        success = false;
        context.log("Does not exist: " + data)
    } else {
        success = true;
        context.log("Does exist: " + data)
    }

⬇The function then returns the URL for downloading the PDF and whether or not the PDF is ready for download to the webpage.

    context.res = {
            body: {
                    "downloadUri" : download,
                    "success": success,
            }
    };
    // receive the response

    context.log(download);
    context.log(data)
    context.done();
}

Frontend: Creating the Download HTML page

Once again, the "fancy" stuff is omitted.

⬆Like we created the "upload" page in Step 1, we now need a "download" page for users to receive the PDF.

This piece of code creates:

An input for the username Line 6
One button for refreshing to check if the PDF is ready Line 8
One button for downloading the file Line 9

Frontend: Downloading the PDF on the Webpage

Time to get bombarded with some lovely JS!

Part 1 ⬇:

Change the HTML on lines 2-4 to display the current status (whether it's looking for the PDF, whether it's ready for download, etc.)
Make a request on lines 9-16 to the HTTP Trigger Function we just coded, sending the username inputted on the HTML page along with it

Part 2 ⬇:

First we're going to find the link to download the PDF with data.downloadUri on line 1
Change buttons from "Refresh" to "Download" when PDF is ready for download
- How to do this? Remove the "Refresh" button Lines 10-11 and make "Download" visible Line 9
Set the onclick attribute of the "Download" button to call the getPdf() function with the unique username + link for download. Line 8
- The getPdf() function allows for immediate download with window.open(link) Lines 16-19

Amazing! You're done!

Here's the finished product in which I download the cute bunny shopping picture I uploaded earlier.

Congratulations! I hope this knowledge of Azure Functions helps you create even more fun apps!

If you're interested in augmenting this app, try using your new knowledge of Blob Storage, HTTP Triggers, the Node SDK (@azure/storage-blob), and some Stack Overflow to assist you to add a feature to delete the image and PDF blobs.