DEV Community: Brian Shaw

My submission to the GitHub Actions x DEV Hackathon | Automatic upload to Google Cloud Storage with GitHub Actions

Brian Shaw — Thu, 20 Aug 2020 14:29:55 +0000

My Workflow

This workflow uploads files to Google Cloud Storage. I use it for uploading my static personal web site, which is hosted from Cloud Storage, but it could be adapted for any use case. Simple modifications could make it work for other providers, too...

Submission Category:

DIY Deployments

Yaml File or Link to Code

bshaw / sha.ws

sha.ws

A personal website built with Hugo and the Congo theme.

Content License

Site content licensed under a Creative Commons Attribution 4.0 International License, except where indicated otherwise.

View on GitHub

Additional Resources / Info

I posted about this recently, so why not submit it!?

Automatic upload to Google Cloud Storage with GitHub Actions

Brian Shaw ・ Jun 24 '20

#cloud #devops #googlecloud #githubactions

Run the .NET Core SDK and Runtime from a Docker container

Brian Shaw — Wed, 08 Jul 2020 21:04:23 +0000

Overview

I run a lot of my tooling straight from Docker containers instead of installing them on my system.
Doing so helps me ensure I'm using the desired version(s) straight from the official maintainers, helps me keep my system tidy, and guarantees a consistent environment across machines.

I've started getting my feet wet with .NET Core in preparation for an upcoming project and noticed that neither Microsoft's documentation nor their Docker Hub page for .NET Core mention running the SDK or runtimes straight from the published images, so I've put together a simple wrapper to run the .NET Core SDK and Runtime from a Docker container:

https://github.com/bshaw/dotnet-docker/.

The script uses the official Microsoft .NET Core SDK image using the 3.1 tag (Debian 10).
It is based on (effectively forked from) the Docker Compose Run docker-compose in a container script, which properly handles volume mappings and various options based on the environment.

Installation

Place the wrapper script in your path and made it executable:

sudo curl -L --fail https://raw.githubusercontent.com/bshaw/dotnet-docker/master/run.sh -o /usr/local/bin/dotnet
sudo chmod +x /usr/local/bin/dotnet

Usage

Use the dotnet command as you would if it were installed locally:

❯ dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   3.1.301
 Commit:    7feb845744

Runtime Environment:
 OS Name:     debian
 OS Version:  10
 OS Platform: Linux
 RID:         debian.10-x64
 Base Path:   /usr/share/dotnet/sdk/3.1.301/

Host (useful for support):
  Version: 3.1.5
  Commit:  65cd789777

.NET Core SDKs installed:
  3.1.301 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.App 3.1.5 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.1.5 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

TODO: Multi version support

The dotnet version is specified as a variable in the script - you can easily update the file on your system depending on your requirements.
I may also update the script to support multiple versions.
Feel free to send a pull request if you get to it before I do.

Automatic upload to Google Cloud Storage with GitHub Actions

Brian Shaw — Wed, 24 Jun 2020 11:47:30 +0000

Overview

GitHub Actions enable simple workflow automation and CI/CD right from GitHub repos.

Uploading your content, whether a static website or any other artifact can be easily managed with an Actions workflow.

This article walks through everything required to successfully set up your workflow.

The examples are based on the this static website (sha.ws), which is uploaded to Google Cloud Storage for hosting.
The workflow is available in the git repo for the site: https://github.com/bshaw/sha.ws/blob/master/.github/workflows/google-cloud-storage.yml

Note: Most of these instructions use the GCP web console - you can just as easily use gcloud cli or your favourite language.

Role

We will be giving the GitHub Actions workflow runner access to our GCP project, so we must ensure to use a role which has only the absolutely necessary privileges.

Create a new role

Navigate to: IAM & Admin > Roles > +Create Role
Enter a title for the role - I like to preface custom roles with Custom - so they're easy to list later on
(Optional) Enter a description
Set Role launch stage to General Availability
Assign permissions:

* resourcemanager.projects.get
* storage.buckets.get
* storage.buckets.list
* storage.objects.create
* storage.objects.delete
* storage.objects.get
* storage.objects.list
* storage.objects.update

See the GCP documentation for Creating and managing custom roles for more details.

Service Account

A service account with the previously created rule will be used by the workflow to connect to Cloud Storage and upload files.

Create a new service account

Navigate to: IAM & Admin > Service Accounts > +Create Service Account
Enter a name for the account - I like to preface service accounts with sa -
Accept the recommended service account id
(Optional) Enter a description
Click Create
On the second page, select the custom role created in the previous step - this will assign permissions to the account
Select all other defaults to finish creating the account

See the GCP documentation page for Creating and managing service accounts for more details.

Create a service account key

Note: We will use the gcloud cli for this step even though the rest of the examples use the web console.
A key created in the console differs slightly from one created with gcloud - since our workflow uses gcloud, we need a correctly formatted key.

gcloud iam service-accounts keys create ~/key.json \
  --iam-account sa-name@project-id.iam.gserviceaccount.com

The key file will be created at ~/key.json.
Keep this file private - do no not store it in your repo - it contains everything needed to authenticate to your GCP project as the service account user.

See the GCP documentation page for Creating and managing service account keys for more details.

Secrets

The workflow uses the gsutil command line utility (included with gcloud) to interact to Cloud Storage.
gsutil needs the project, service account key, and Cloud Storage bucket supplied in order to connect and manage files.
We will store these as secrets in our GitHub repo and access them as variables in the workflow.

GCS_PROJECT

The name of the CGP project containing the Cloud Storage Bucket.

GCS_BUCKET

The name of the Cloud Storage bucket to deploy the site.

GCS_SA_KEY

The base64 encoded authentication key for the service account.

Base64 encode the key:

cat ~/key.json |base64

See the GitHub documentation page for Creating and storing encrypted secrets for more details.

Workflow

The workflow is created in your repo at .github/workflows/workflow.yml

Google provides some actions which can be imported to simplify your workflow, as well as some good examples in their github-actions repo.

Trigger the workflow

Since my workflow is for a personal website, I trigger when I push to master:

on:
  push:
    branches:
    - master

Set environment variables for the workflow runner

You can set environment variables for the workflow runner to use when it runs steps.
In this example we pull in the Cloud Storage bucket name from our GitHub secrets:

env:
  GCS_BUCKET: ${{ secrets.GCS_BUCKET }}

Install and configure gcloud (and gsutil)

Use the setup-gcloud action to install and configure the gcloud and gsutil command line tools.
Notice how we pull in the service account key and project id from our secrets:

steps:
- name: Checkout
    uses: actions/checkout@v2

- name: Setup - gcloud / gsutil
  uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
  with:
    service_account_key: ${{ secrets.GCS_SA_KEY }}
    project_id: ${{ secrets.GCS_PROJECT }}
    export_default_credentials: true

Upload to Cloud Storage

Use the gsutil command line to upload your files to the Cloud Storage bucket.
In this example, we are uploading the generated static website, stored in the content subdirectory - specify whatever source path you require.
Notice how we pull in the bucket name from the environment variable we created in the workflow:

- name: Deploy
  run: |-
    gsutil -m rsync -R output gs://$GCS_BUCKET

Run workflow and check status

The results and output of the workflow can be seen in the /actions url of your repo in GitHub.
There's also an Actions link at the top of the page (where you select between code, issues, pull requests, etc.).

This output can be helpful when troubleshooting a broken workflow.

Status badge

You can add a workflow status badge to your README.
Simply reference the workflow name (the name parameter in the workflow yml file) in the badge path:

![{{alt text}}](https://github.com/{{github user}}/{{github repo}}/workflows/{{workflow%20name}}/badge.svg)

See the GitHub documentation for Adding a workflow status badge to your repository for more details.

Final thoughts

That's all there is to it!
Despite all of the steps, it probably takes about 10 to 15 minutes to get this configured (plus any trial and error to get the workflow behaving).