DEV Community: Ryan Mercadante

Monitoring Github events with Webhooks and Google Cloud Functions

Ryan Mercadante — Tue, 31 Mar 2020 15:32:50 +0000

In this post we are going create a Google Cloud Function and a Github Webhook to automate the deletion of inappropriate comments on Github issues. We'll create a webhook that will call an HTTP function every time an issue comment is created, edited, or deleted. In order to follow along with this tutorial you will need a Google Cloud Platform account and a Github account.

What is a Google Cloud Function?
Developing our Function in the Console
Developing our Function Locally
Creating our Webhook
Generating a Personal Access Token
Testing our Function
Deploying our Function
Wrapping up

What is a Google Cloud Function?

Google Cloud functions are single-purpose, serverless functions that can run on demand in your cloud environment in response to events. Events include HTTP events, Cloud Pub/Sub events, and Cloud Storage events. At the time of this writing there are also a few more events currently in beta. You then create a trigger in response to the event that is emitted, and bind that trigger to a function.

Cloud Functions are useful for situations where you don't want to spin up a full server to execute some logic. All of the infrastructure and software are managed by Google so that all you have to do is write the code. This is often referred to as Functions as a Service, or FaaS, and is not unique to Google. AWS Lambda and Azure Functions are just two of the many competitors in this space.

Developing our Function in the Console

There are two ways we could go about developing our function, in the console or locally in our development environment. First, I'll demonstrate how you would go about setting it up in the console, and afterward we will actually develop in our local environment.

Open up your cloud console and select a project or create a new one. Next, select Cloud Functions in the compute section of the navigation menu. Enable the API if it is not already enabled. You will also need to make sure you setup a billing account for the project to use Cloud Functions. Once you click on create function you'll see the following.

I gave this function the name test, left the memory allocation on the default of 256 MiB, and we are using the HTTP trigger type. We are also allowing unauthenticated invocations.

Next we have the advanced options. Choose the region closest to you to reduce latency. You get charged only while your function is running to the nearest 100 milliseconds. You can also set the timeout and the maximum function instances that you want running. This is useful because you can put a limit on how much your function can scale out, otherwise your function can scale out to as many instances as necessary. You will also need to select a service account that the function will assume as its identity.

It's nice to have all the options right in front of you, but writing code in the browser like that is definitely not ideal.

Developing our Function Locally

To develop locally we are going to use the Functions Framework. This will allow us to spin up a server for our function and invoke that function in response to a request. To get started, create a new folder that will have your function and run npm init. Next, run npm install @google-cloud/functions-framework node-fetch and add the following script to your package.json:

  "scripts": {
    "start": "functions-framework --target=deleteGithubComment"
  }

We'll need node-fetch to make a DELETE request to our github comment. Create an index.js file and add the following contents to it:

const fetch = require('node-fetch')

const inappropriateWords = ['enter','words','to','check','for']

exports.deleteGithubComment = async (req, res) => {
  const { repository_url, title } = req.body.issue
  const {
    id,
    body,
    user: { login },
  } = req.body.comment

  const bodyArray = body.split(' ')

  const url = `${repository_url}/issues/comments/${id}`
  const headers = {
    Authorization: `Token ${process.env.TOKEN}`,
  }

  let removeComment = false
  bodyArray.forEach((word) => {
    if (inappropriateWords.includes(word)) {
      removeComment = true
    }
  })

  if (removeComment) {
    try {
      await fetch(url, {
        method: 'DELETE',
        headers,
      })
      return res.status(200).json({
        user: login,
        message: `Removed inappropriate comment on issue "${title}."`,
        comment: body,
        deleted_message: body,
      })
    } catch (err) {
      return res.status(400).json({
        user: null,
        message: 'Error removing inappropriate comment.',
        comment: body,
        deleted_message: null,
      })
    }
  }

  return res.status(200).json({
    user: login,
    message: `No need to remove comment. Maybe you can log this information.`,
    comment: body,
    deleted_message: null,
  })
}

Our function is written in Node.js. Node cloud functions use express under the hood, so you will have the familiar request and response arguments. When this function is called, it checks the contents of the comment, and checks each word against an array of inappropriate words. To keep the tutorial family friendly, I removed the words I used and added a placeholder. If you wanted to get more advanced, you could use Google's AI services to better understand the meaning of the comment and maybe catch things that a simple word check would miss. Since this is just meant to get you started I will not be doing that.

If it finds a word in your array of inappropriate words, it will send a DELETE request to github to remove the comment using node-fetch. This requires getting a token from Github which we will cover in the next section. Before moving onto the next section, run npm start so the functions framework can start up a server for our function. This should start up on http://localhost:8080.

Creating our Webhook

Creating our webhook requires a publicly accessible URL, but because we haven't deployed our function yet, we don't have one. In order to get around this, we are going to install an npm package called ngrok which will create a tunnel to expose our localhost to the internet. Run npm install -g ngrok and once that is done, run ngrok http 8080.

Login to your Github account and select a repository that you want to use this function. Go to settings and select Webhooks.

Click on create webhook and fill out the form like I have done. Notice how I am using the the URL provided by ngrok which will tunnel to our localhost.

Click on Add Webhook and you're all set.

Generating a Personal Access Token

Go to your user settings and then click on Developer Settings at the bottom.

Select Personal access tokens and click on generate token. Enter some kind of descriptive note and select the repo checkbox.

Click on generate token and you'll be given your token. Make sure to copy it because you'll never be able to see it again after you leave the page.

Testing our Function

Head back to your code and create a .env.yaml file and add the token like so:

TOKEN: add-your-token-here

In order to get around installing the dotenv package for testing and uninstalling it for deployment, just replace

const headers = {
  Authorization: `Token ${process.env.TOKEN}`,
}

with the following:

const headers = {
  Authorization: `Token your-token`,
}

We will change this back before we deploy our function.

Now that you have everything set up you can test your function. Create a new issue in your Github repo and add a comment that shouldn't be removed by your function. Afterward, add a comment that is included in your inappropriate words array. You should see it get added, and once you refresh the page it should be deleted. You can use this setup to test other webhooks or functions that you have created.

Deploying our Function

Now that we have tested our function locally, we are going to deploy our function to Google Cloud. First, DO NOT forget to remove your token from your code and replace it with process.env.TOKEN. Next, from the root of your function directory run the following command:

gcloud functions deploy deleteGithubComment \
  --region us-east1 \
  --runtime nodejs10 \
  --trigger-http \
  --entry-point=deleteGithubComment \
  --env-vars-file .env.yaml \
  --allow-unauthenticated

This will deploy your function to the region us-east1 (you should change the region to the one closest to you) with the name deleteGithubComment, that uses the nodejs 10 runtime, and declares it as an HTTP function. The entry point is the function in your index.js file that you want to deploy and we are letting Google Cloud know that we have environment variables that are scoped to this function. We are also allowing unauthenticated function invocations but because this function requires information from Github, nothing will happen if you just go to the URL provided by the function.

Deploying will take up to two minutes but afterwards in the output you should see a URL for your function. The only thing left to do is to go back to Github and replace the ngrok URL in your webhook with the URL for your function. Now that you're deployed, test everything out once more to make sure it's still working. The first time you hit the endpoint, you'll experience what is called a cold start. If your function hasn't been run in awhile or is running for the first time, it will take a second or two for the server to spin up. After your function is called the server should stay active for some time before it's spun back down, meaning much faster response times.

Wrapping up

Cloud functions and Github webhooks can both be really powerful and you are only limited by your imagination on what you can do with them. I'd encourage you to take what you learned from this article and apply it to something else. Maybe you already have an application that could use some one-off logic wrapped up in a cloud function. Or maybe you want to automate some other aspect of Github, like emailing any user who creates a pull request with information about your project.

If anything was unclear or something isn't working right, leave me a comment down below or message me and I'll do my best to help you out. If you would like to get in touch for any reason, feel free to connect with me on LinkedIn, follow me on Twitter, or send me an email. Thanks for reading!

CI/CD with Google Cloud Build

Ryan Mercadante — Sat, 14 Mar 2020 20:46:02 +0000

In this post we are going to cover setting up CI/CD pipeline using google cloud build. If you need to set up a GCP account or need to deploy your application, I already wrote something that should get you up and running https://dev.to/ryanmercadante/deploy-your-mern-stack-application-to-google-app-engine-2g2c.

What is CI/CD?
Setup Google Cloud Build
Creating Build Triggers
Setting up Cloud KMS
Encrypting our Environment Variables

What is CI/CD?

CI/CD stands for continuous integration and continuous deployment/delivery depending on who you ask. It is the process of creating pipelines to automate certain tasks such as building, testing, merging, and deploying your code. Red Hat has a great article going in depth on the nuances between the integration, delivery, and deployment phases https://www.redhat.com/en/topics/devops/what-is-ci-cd, and I would highly recommend reading it if you're interested in understanding it better.

Setup Google Cloud Build

First head over to the GCP console and go to APIs and Services. You'll want to search the API library for Cloud Build API and enable it.

Next, go to Cloud Build in the tools section and click on settings. My application is deployed to app engine, so I am going to enable app engine. You should enable Cloud KMS as well, but we'll cover that more later in the tutorial.

If your application is deployed to app engine as well, you will also need to enable the App Engine Admin API back in the API library.

With that set up, let's head back to the Cloud Build page and go to Triggers. The first thing we need to do is to connect our repository.

Select where you keep your source code (I use GitHub).

Next it will ask you to authenticate with whichever provider you chose and then to select repositories to connect to cloud build. You should see a link to select repositories on GitHub (or whichever you use). You can select specific repos or all of them. I chose to only select the two projects that will be using cloud build at this moment. One is a react application and the other is an express API.

Once you click save you'll be redirected back to the page you were on. Select the repos you just gave permission to, and the checkbox saying you understand the warning they provide.

The last step to create a push trigger is optional and we are going to handle that in the next section, so skip that for now.

Creating Build Triggers

The next step in the process is to create our build triggers. Cloud Build offers 120 minutes of free build time per day, but after that it is $0.003 per minute. This is a very generous free tier, and for personal learning projects, even if you did go over the free tier limit it is very unlikely you would rack up any excessive cost. With this in mind though, we are only going to create one trigger for each repo I added. The trigger will install dependencies, build our app, and deploy it. You can also include other build steps like linting, testing, etc.

With that out of the way let's build our first trigger. Go to the cloud build dashboard and click the button to set up build triggers.

We are going to name the trigger push-to-master, and this needs to be unique within the project. There are three trigger types; branch, tag, and pull request. In this tutorial we are going to use the branch trigger type. In the branch section just type in master so it knows to only trigger when you push to the master branch. We are going to ignore a few optional sections which allow you to trigger based on changes to specific files. For our build configuration we are going to use a cloud build configuration file; cloudbuild.yaml. It should default the file location to the root of your project. This is what our file looks like.

steps:
- name: "gcr.io/cloud-builders/npm"
  args: ["install"]
- name: "gcr.io/cloud-builders/npm"
  args: ["run", "build"]
- name: "gcr.io/cloud-builders/gcloud"
  args: ["app", "deploy"]
timeout: "1600s"

This build uses two different cloud builders; npm and gcloud. Cloud builders are just container images that allow you to run commands within the context of the builder. The args field takes a list of arguments that the builder will run. So using the npm builder with the arguments run and build is the same as running npm run build in your terminal. If you click on the history tab, you should see a list of your previous builds.

You can further inspect the build process by clicking on the build Id in the build column.

Clicking into these tabs will give you more insight into what was going on behind the scenes. This is especially useful if your build failed.

That was the build trigger for my react app. The cloudbuild.yaml is going to look a little bit different for my express API because I am using environment variables that need to be encrypted. In order to do this properly, we are going to need to enable the Cloud Key Management Service (KMS) API.

Setting up Cloud KMS

Cloud KMS will allow us to create encryption keys to protect secrets and other sensitive data that we need to store in the Google Cloud Platform. More specifically, we are going to use it to encrypt our .env file because that has our database URI.

Search for Cloud Key Management Service (KMS) API in the API library and enable it. Before we can create a cryptographic key, we need to create a keyring. We are going to be using a gcloud command for this, otherwise you can create a keyring using the console.

Run gcloud kms keyrings create cloudbuild-keys --location global in your terminal to create your keyring. I named mine cloudbuild-keys but you can name it whatever you'd like. You should see this in the console.

Next we are going to create our key. Run the command gcloud kms keys create api-key --location global --keyring cloudbuild-keys --purpose encryption. You should see the key in the keyring you created.

If you want to use the console to create your keyring and keys, it's pretty straightforward and I have provided pictures to show you the console equivalent of our gcloud commands.

Encrypting our Environment Variables

Next, we are going to encrypt our .env file so that we can use it in our build. Just to reiterate, I have to do this for my deployed API because my environment variables include my database URI, which should be kept private. I didn't have any sensitive information for the client which is why none of these steps were necessary.

Run the following command in the root folder of your project but make sure to change out my project specific information with your project specific information.

gcloud kms encrypt \ --plaintext-file=config/config.env \ --ciphertext-file=config/config.env.enc \ --location=global \ --keyring=cloudbuild-keys \ --key=api-key

It is safe to commit your encrypted file to source code, but make sure you add your unencrypted file to your .gitignore.

Now that we have successfully encrypted our sensitive information, we can use it in our CI/CD pipeline. My cloudbuild.yaml file for my API looks like this. Once again you'll want to remove my project specific information and add yours instead. This will decrypt our config.env.enc at build time so that we can use the decrypted file in our workspace.

steps:
  # Decrypt our env file
- name: gcr.io/cloud-builders/gcloud
  args:
  - kms
  - decrypt
  - --ciphertext-file=config/config.env.enc
  - --plaintext-file=config/config.env
  - --location=global
  - --keyring=cloudbuild-keys
  - --key=api-key

  # Deploy our app
- name: "gcr.io/cloud-builders/gcloud"
  args: ["app", "deploy"]

timeout: "1600s"

Hopefully all of these steps worked for you and you have a fully functioning CI/CD pipeline for your application! If anything was unclear or something isn't working right, leave me a comment down below or message me and I'll do my best to help you out. If you would like to get in touch for any reason, feel free to connect with me on LinkedIn, follow me on Twitter, or send me an email. Thanks for reading!

Deploy your MERN Stack Application to Google App Engine

Ryan Mercadante — Sun, 01 Mar 2020 20:56:44 +0000

In this post we are going to walk through the steps of deploying a MERN stack application on the Google Cloud Platform. We will be deploying our React frontend and Node backend as separate services using Google App Engine, their Platform as a Service offering. In order to do this we are going to need to sign up for a GCP account. If you already have an account and have configured gcloud, you can skip the first couple of sections. Note: this tutorial assumes you have a demo application already built. If you do not, feel free to grab my code from https://github.com/ryanmercadante/Users-app-gcp

Signing up for GCP
Creating our Project
Setup the Cloud SDK
Deploying our React Application
Setup MongoDB using Atlas
Deploying our Node.js API
Recap

Signing up for GCP

Head over to https://cloud.google.com/ and you should be presented with a page that looks something similar to this:

Click on the ‘Get started for free’ button and follow the steps. You’ll need to create a gmail account if you don’t have one already. Afterward you should be presented with a screen to try Google Cloud Platform for free. They have a very generous free plan, which gives you $300 dollars worth of credits for free and no auto charge if you run out. You have to manually upgrade to a paid account.

The second page of signing up is fairly straight forward, just make sure under ‘Account type’ you sign up for an Individual account and not a Business account.

Upon completion of the form, you should be redirected to the google console homepage.

Creating our Project

Now that we have an account setup, let’s create a new project. In the top navigation bar you should see a dropdown to select a project. Click on that and select New Project in the top right corner. You can name the project whatever you’d like, and google will automatically create a globally unique project id for you. You can edit this now but once you create the project you won’t be able to change it. I usually like to edit my project ids so they are a little bit easier to remember, such as the project name with my initials at the end of it. We are not going to select an organization at this time. Click on create when you are done with this.

Head over to the navigation menu and select App Engine inside of the Compute section. First select download the SDK, we’ll be using this later to deploy our app to App Engine. When you are done install the SDK, come back to this page and select Create Application.

Select a region close to you (I will be using us-east1). On the next page we are going to select Node.js as our language, and Standard as our environment instead of Flexible. We will be using the standard environment so that our app can scale down to 0 instances when there is no traffic. This will allow us to run the app for free or for very little cost which is ideal in my case. You will want to use the flexible environment if you are using Docker containers and your app receives regular traffic, or if you want to use a programming language that the standard environment doesn’t support.

If you’d like to see a full list of differences between the standard and flexible environments head over to https://cloud.google.com/appengine/docs/the-appengine-environments.

If you haven’t already done so go ahead and download the SDK, otherwise select I’ll do this later.

Setup the Cloud SDK

Open up your terminal of choice and run ‘gcloud init.’ This will run you through the steps of configuring gcloud, their CLI tool.

Deploying our React Application

Congrats on making it this far, now we get to the fun part. Head into the root directory of your react application and create an app.yaml file. This is where we configure our application settings for App Engine. It contains information about our code, runtime, URL paths and more. This is what our app.yaml file is going to look like.

At the very least, our app.yaml needs to include a runtime, of which we are using nodejs10. The handlers will handle our static files that are in our build folder. If you haven’t created a build of your application yet you will want to do so now.
Because this is our first service that we are deploying, it will be deployed as the default service. Let’s go back to our terminal and run the command ‘gcloud app deploy’ from the root of our project. You will be asked if you want to continue, if you do enter Y for yes.

I am not deploying anything new so I have 0 files uploaded, but yours should be more than 0 if its your first time deploying your application or if you added something new since your last deploy.

Now that your frontend is done deploying, run ‘gcloud app browse’ and checkout your deployed app! If you head back over to your GCP console and to the App Engine dashboard, you’ll see some statistics about your app.

If/when you deploy your app again, app engine won’t automatically delete your previous version that you deployed. You can see the different versions of your app under the versions tab. This is useful if you want to split traffic between different versions of your application for things like A/B testing. I won't cover that in this tutorial, but you can read more about that here https://cloud.google.com/appengine/docs/standard/nodejs/splitting-traffic. App Engine is smart enough to move traffic to the most recently deployed version of your application. If you select the older versions you can delete them if you choose to do so.

Setup MongoDB using Atlas

Before we deploy our node API, we should setup a database using MongoDB Atlas. Go to https://www.mongodb.com/cloud/atlas and either sign in to your account, or create one. After signing up you’ll be presented with a page to create an organization.

Since this is just an account for small personal projects, I am going to name my organization test. Click next, and then add a new user and give them the role of Organization Project Creator, otherwise you won’t be able to create any projects.

Afterward you’ll be redirected to your dashboard. Select create a project, and give your project a name. On the next page, when you click to add a new member, the user you created earlier should be presented as an option. Select that user and give them the Project Data Access Admin role. Because this is just a demo app, I am not going to worry about giving this user admin privileges, but for a real application with potentially sensitive information, you should be diligent about how you assign database roles to users.

Now it’s time to build a cluster. Select build a cluster in the center of the screen and then select the free shared clusters. Make sure to select Google Cloud Platform and the region closest to you that clearly states ‘Free Tier Available.’

The M0 sandbox cluster tier is free forever so that is the one we’ll be selecting. You can rename the cluster to whatever you would like.

It should take a few minutes for your cluster to get setup, but once it does, click connect and whitelist 0.0.0.0. This will allow traffic from anywhere. This is fine for our demo app but for a production app you would probably never do this. You will also need to create a MongoDB user.

Next we need to choose a connection method. Select connect your application, and copy the connection string they provide for you to use in your application where needed.

Congrats! You now have your MongoDB cluster up and running. It has one primary node and two secondary nodes for data replication.

Deploying our Node.js API

Deploying our node api will follow similar steps we took to deploy our React frontend. We will create another app.yaml and place it in the root directory of our node application.

In the app.yaml for our react frontend, we didn’t name our service because it was our default service. Since this is not our default service, let’s just name it api. Once again we are use the nodejs10 runtime. Dealing with environment variables that are meant to be secret, such as our MongoDB URI, is a little bit tricky. The most straight forward solution I found was to create a separate yaml file for your environment variables, add that to your .gitignore file so it isn’t tracked, and use the includes element. Run the ‘gcloud app deploy’ command and your api will be deployed to App Engine. If you check the services tab, you should see both the default service and the api service, and how many version of each are deployed.

Both your frontend and backend are deployed now and you can go to the url that app engine created for you!

Recap

In this tutorial you learned how to deploy a React application, express API, and MongoDB database to Google Cloud Platform using Google App Engine and MongoDB Atlas. The steps you followed here are a great starting point for personal projects and I encourage you to build upon it. This application is most definitely not secure, and if you wanted to start building applications for clients utilizing GCP, https://cloud.google.com/appengine/docs/standard/nodejs/application-security is the first place I would start. In a future tutorial I plan to write about things like adding a custom domain, utilizing CI/CD, and more. If you have any questions, comments, or general feedback let me know in the comments below. If you would like to get in touch for any reason, feel free to connect with me on LinkedIn, follow me on Twitter, or send me an email. Thanks for reading!

DEV Community: Ryan Mercadante

Monitoring Github events with Webhooks and Google Cloud Functions

Table of Contents

What is a Google Cloud Function?

Developing our Function in the Console

Developing our Function Locally

Creating our Webhook

Generating a Personal Access Token

Testing our Function

Deploying our Function

Wrapping up

CI/CD with Google Cloud Build

Table of Contents

What is CI/CD?

Setup Google Cloud Build

Creating Build Triggers

Setting up Cloud KMS

Encrypting our Environment Variables

Deploy your MERN Stack Application to Google App Engine

Table of Contents

Signing up for GCP

Creating our Project

Setup the Cloud SDK

Deploying our React Application

Setup MongoDB using Atlas

Deploying our Node.js API

Recap