DEV Community: Katie McLaughlin

Codegolf: Build a container in Cloud Build

Katie McLaughlin — Wed, 27 Mar 2024 01:19:28 +0000

Banner Photo by Viktor Kiryanov on Unsplash

I read Adam Ross's Modernizing cloudbuild.yaml for Container Builds and saw all his mentions of "character counts" and it got me thinking:

How codegolf can we get with this?

Context: like regular golf, codegolf asks you to implement something in the least amount of character possible.

Starting with Adam's finishing iteration:

steps:
- id: 'Build Container Image'
  name: 'gcr.io/cloud-builders/docker:latest'
  script: docker build . --tag "${_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${_REPO}/${_IMAGE}:latest"

images:
- "${_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${_REPO}/${_IMAGE}:latest"

options:
  automapSubstitutions: true

substitutions:
  _IMAGE: service
  _LOCATION: us
  _REPO: container

Initial character count: 358

It appears his character count doesn't inline new lines, which is effectively the same as the wc -c minus the wc -l. I'll be using the following for my character counts:
expr $(wc -c cloudbuild.yaml | cut -d' ' -f1) - $(wc -l cloudbuild.yaml | cut -d ' ' -f1)

He notes that the introduction of Artifact Registry make this a bit longer than it would be in Container Registry, but we'll sit with that since it's the supported registry.

We'll also keep with this last iteration's requirements: a project might have more than one registry, so we'll keep some parameterization.

Each iteration I'll make sure is a valid build. For testing, I'll ensure gcloud builds submit runs clean in a folder containing just this cloudbuild.yaml file and my go-to codegolf Dockerfile (that's also a valid Cloud Run service, just for fun).

What's in an ID?

To start, Cloud Build steps don't require an ID, so we can remove that line entirely. Without an ID, the Cloud Build build details in the Google Cloud console will use the image name as the step details, so there is a slight usability disadvantage to this change, but nothing that affects functionality.

-- id: 'Build Container Image'
-  name: 'gcr.io/cloud-builders/docker:latest'
+- name: 'gcr.io/cloud-builders/docker:latest'

Character count: 329.

Deduplicate with dynamics

We've also got the same image ID used multiple times. We want to keep the dynamic nature of this, but we also want to use substitutions in the value itself. We can use dynamicSubstitutions to allow use of substitutions within other substitutions, but adding this adds ~27 characters to our file (and we're removing 45x2 characters so this is a win!)

steps:
- name: 'gcr.io/cloud-builders/docker:latest'
  script: docker build . --tag "$_IMAGE_URI"

images:
- "$_IMAGE_URI"

options:
  automapSubstitutions: true
  dynamicSubstitutions: true

substitutions:
  _IMAGE_URI: ${_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${_REPO}/${_IMAGE}:latest
  _IMAGE: service
  _LOCATION: us
  _REPO: container

Character count: 326.

At this point we can also remove the superfluous quotation marks, saving 6 characters, bringing us down to 320.

Shortest values

Now we get to the part where we work out: What's the minimum value for the variables we're using?

The $PROJECT_ID is a default substitution, so we should keep that as is, and it will be dynamic for our project.

We're already using the shortest Artifact Registry region, the US multi-region (the longest Google Cloud region could be up to 24 characters, so 2 is great!)

But we can possibly do better with the Artifact Registry name and the Container Name.

d (for "Docker") is a valid Artifact Registry repository name, and i (for "image") are valid, so we can save ~15 characters there:

...
-  _IMAGE: service
+  _IMAGE: i
   _LOCATION: us
-  _REPO: container
+  _REPO: c

Character count: 306.

Minimal variables

Any of our underscore substitutions we can reduce down to two letters, the underscore and some useful character. (We're replacing "Image URL", "Image", "Repo", and "Location", respectively.)

steps:
- name: gcr.io/cloud-builders/docker:latest
  script: docker build . --tag $_U

images:
- $_U

options:
  automapSubstitutions: true
  dynamicSubstitutions: true

substitutions:
  _U: ${_L}-docker.pkg.dev/${PROJECT_ID}/${_R}/${_I}:latest
  _I: i
  _L: us
  _R: d

Character count: 254.

(We're now down below what Adam was before migrating to Artifact Registry! 😅)

Remove tags

Another reduction we can make is removing the :latest tags. "Latest" is used in Docker where there is a lack of tag, so this is implicit in our call (and it just so happens that the Cloud Builder we are referencing has a "latest" tag. (You can check this by going to gcr.io/cloud-builders/docker in the browser, which will show you information about this image in its registry. This image in particular will direct you to its Artifact Registry information, because this registry has been automatically redirected.)

steps:
- name: gcr.io/cloud-builders/docker
  script: docker build . --tag $_U

images:
- $_U

options:
  automapSubstitutions: true
  dynamicSubstitutions: true

substitutions:
  _U: ${_L}-docker.pkg.dev/${PROJECT_ID}/${_R}/${_I}
  _I: i
  _L: us
  _R: d

Character count: 240.

Putting the args back

The last step we can do to save space is to actually re-introduce the args parameter, rather than using script. In this case, we will be adding some characters for the separation of the arguments, but the biggest win is being able to remove the ~27 characters that the automapSubstitutions config setting adds.

steps:
- name: gcr.io/cloud-builders/docker
  args: ['build','.','--tag','$_U']

images:
- $_U

options:
  dynamicSubstitutions: true

substitutions:
  _U: ${_L}-docker.pkg.dev/${PROJECT_ID}/${_R}/${_I}
  _I: i
  _L: us
  _R: d

Character count: 213.

Tip: Make sure you check what your default entrypoint is for images! In this case, I have also removed docker, as my args are appended to the default entrypoint. The script needs the full command (I may have spent too long debugging this issue 😅)

Twist

Actually you only need 22 characters:

gcloud run deploy a
[ENTER]
[ENTER]

First time you run this you'll need 1 extra character (ENTER) to enable APIs, and also 1 extra character (ENTER) to create the Cloud Run Source Deploy image registry. (You can also use this method to automatically create the Artifact Registry for you, read more about this on my blog post: "Auto-provisioning Artifact Registry though Cloud Run Source Deploys (glasnt.com)").

Summary

You can make a very short Cloud Build configuration to build images, but you should consider the readability of your scripts as you go. Addressing any cruft and modernizing your scripts can be an interesting exercise for you and your infrastructure.

With thanks to Adam Ross for the review!

Migrating from Secret Manager API to built-in secrets

Katie McLaughlin — Sun, 19 Jun 2022 23:17:12 +0000

Secret Manager is directly integrated into a number of products, including Cloud Run and Cloud Build.

Many samples I've authored use the Secret Manager API by calling the API from within the code. This can be more secure, but variable leakage is something that needs to be considered in any deployment. Removing the direct API calls can help with portability, and reduce the amount of dependencies in the deployment, and the complexity of the code itself.

So let's look at how we can migrate to built-in secrets.

Integrating Secret Manager API

Screenshot: "Secret value: Input your secret value or import it directly from a file.", showing a file upload dialog, or a text area input field.

When you create secrets, you can create them from a file or from direct input. When you retrieve these secrets, you may presume they're a single value, or a file of content you have to process.

Once you create the secret, you can access it with the Secret Manager API using a client library for Secret Manager available in many different languages.

This post will use Python examples and packages, but the same patterns can be applied to other languages.

For example, this code snippet will retrieve the latest version of the secret mySecret in the current project:

import google.auth
from google.cloud import secretmanager as sm

# trick to detect current project
_, project = google.auth.default() 
client = sm.SecretManagerServiceClient()

secret = "mySecret"

name = f"projects/{project}/secrets/{secret}/versions/latest"
secret_value = client.access_secret_version(path).payload.data.decode("UTF-8")

The secret_value can then be used as required. This presumes that mySecret is a single value, for example an API key.

One way developers can include multiple secrets in one format is with .env files. These files contain one or more key/value pairs which mean you can store multiple secrets in one file, then use helper packages to parse these values into your code. This prevents you from having to define multiple separate secret values.

If mySecret was a .env file, I could load it using python-dotenv:

## pip install python-dotenv
import io
from dotenv import load_dotenv

load_dotenv(stream=io.StringIO(secret_value))

Or I could to similar with django-environ, for example:

## pip install django_environ
import io
import environ
env = environ.Env()
env.read_env(io.StringIO(secret_value))

In both of these examples, these packages presume that the values are in a .env file within the same directory as the running process. Using StringIO means that a string variable can be sent to these methods in a file-like object, so the values can be read as though they were a file.

When using the Secret Manager API, you can run the application in any place where Google Cloud authentication works. That is, any Google Cloud service, or your local machine if you have set up gcloud.

Consuming built-in secrets

If you have your secret configured as an environment variable, then you only have to retrieve it as an environment variable:

import os
secret_value = os.environ.get("MYSECRET", None)

This method prevents KeyErrors if the key is not found.

For products with built-in secrets, they may be made available in additional ways.

In Cloud Build you can connect secrets directly to environment variables. But Cloud Run additionally allows mounting the secret as a volume, which means it can be read as a file. Any time the file is read, if you specify latest, the current latest will be received! However, you must mount the file in a new volume so you can't mount it directly in the default .env.

If you want to adapt your code to handle a local .env file, or a separately mounted file, or an environment variable, you will have to ensure all methods are possible in your code.

You also need to consider which configuration takes priority, as by default both python-dotenv and django-environ accept the first declared value as the value they use. You can override this by using the --override or --overwrite respectively.

When developing applications, I might choose to say my local .env file takes priority, then any mounted secrets, and then any declared variables.

Another point to mention is that using python-dotenv, if a file is not found or a value is empty, it silently continues. This means you can include the method calls without having to explicitly handle errors:

import io
import os

from dotenv import load_dotenv

load_dotenv()
load_dotenv("/secrets/.env")
load_dotenv(stream=io.StringIO(os.environ.get("MYSECRET", None)))

The same code works for django-environ, where you can just import in the order of priority without having to worry about missing files.

import io
import os

import environ
env = environ.Env()

env.read_env()
env.read_env("/secrets/.env")
env.read_env(io.StringIO(os.environ.get("MYSECRET", None)))

Note that in these examples, I'm choosing /secrets/as my volume, and keeping the path the same name as the original file. You can choose any volume and path, as long as the volume is not already used by the application (for example, if you choose /app/ as your working directory, you cannot mount secrets there.)

Deploying built-in secrets

To run this code locally, you'd create a .env file with the contents MYSECRET=serkitValue.

If you're committing this code to git, ensure you're not committing the secret file! Make sure you add .env to your .gitignore file!

You can also choose to ignore any contents of your .gitignore file in your Google Cloud commands by adding .gitignore's contents to .gcloudignore:

.git
#!include:.gitignore

You can then create the secret from this file with gcloud:

gcloud secrets create mySecret --data-file .env

For Cloud Build, you will need to ensure the secret is available in the environment (which your script can then use):

steps: 
 - name: python:slim
   entrypoint: pip
   args: ['install', '-r', 'requirements.txt', '--user']

 - name: python:slim
   secretEnv: ['MYSECRET']
   entrypoint: python
   args: ['main.py']

availableSecrets:
  secretManager:
  - versionName: projects/$PROJECT_ID/secrets/mySecret/versions/latest
    env: 'MYSECRET'

You can also reference secrets in the args call itself, using bash variables.

For Cloud Run, you'll have to deploy the service specifying either a mount or an environment variable:

# for mounted volume
gcloud run deploy myservice --source .  \
  --update-secrets /secrets/.env=mySecret:latest

# for environment variable
gcloud run deploy myservice --source .  \
  --update-secrets MYSECRET=mySecret:latest

And never forget IAM!

Don't forget: you'll also need to make sure the service account you're using has permissions to access your secret!

Katie is a Developer Advocate for Google Cloud, focusing on Python and Serverless. She tweets @glasnt about clouds, crafts, and cats.

Using Workflows to dynamically update a Datasette website

Katie McLaughlin — Sun, 20 Feb 2022 22:04:32 +0000

A while ago I spoke with Simon Willison about the sorts of things he'd like to have in a cloud provider in order to deploy Datasette nicely and repeatedly. A few of the things mentioned were being able to attach to object storage, especially when databases were particularly large and it didn't make sense to mount them into the container image itself.

With the preview of the second generation execution environment with Cloud Run, one of the many new features is the ability to mount Cloud Storage as a network filesystem.

With this, you can attach any Cloud Storage bucket and have a Datasette web service serve data from that source.

This works very well when you have one database that you want to serve. However, one of the known limitations of Datasette, where if you serve a folder of content, the glob of files in that folder is loaded at server start time, and not refreshed unless the server is restarted.

There are few ways we can work around this issue. You could have the process within the container restart when the mounted filesystem changes, using something like inotify (fsevent on macOS). You could also consider periodically starting, but that may mean restarting when not required, or not often enough. For a sufficiently low traffic system, you'd end up automatically booting from cold after a time of inactivity, anyway.

The way I chose to solve this was to use the event of an object being uploaded to the bucket to trigger an update of the service, thus restarting Datasette (albeit dramatically).

And since I was now listening to the upload event, I thought, why not do more processing while I'm here?

I'd recently learnt from Lab 6 of the Pic-A-Daily Workshop how to use Workflows to handle events, so I thought I'd try developing one of my own.

The result is github.com/glasnt/dynamic-datasette, and once deployed it works like this:

you upload an new file to the upload bucket
the event of that file being uploaded triggers a function, that starts a Workflow
the workflow then follows through a number of steps:
- sends the file for processing:
  - if the file is already an SQLite database, then it sends it off to the serving bucket
  - if it's one of the types of files it knows about, it uses one of the many *-to-sqlite tools Simon has made. For example, csvs-to-sqlite.
  - Once converted, the resulting SQLite database file is uploaded to the serving bucket
- updates metadata for Datasette:
  - this step mostly just ensures that the Datasette service displays the datetime when this workflow was run, so users can know how new their data is that's being served
- restarts the web service
  - This step uses the replaceService API to force a 'restart', deploying a new revision of the Datasette service, ensuring that the latest information in the serving bucket is being relayed to users.

The Datasette container mounts the serving bucket, and metadata.json in that same bucket, then just calls "datasette serve" on the mounted folder.

You can try this out on your own project by deploying the code with Terraform, following the instructions in the README.md.

Try extending the functionality of the process-upload function to do more steps, or adding more metadata to the hosted service.

Photo by Victor on Unsplash

Open Source Credit Survey

Katie McLaughlin — Tue, 01 Feb 2022 21:31:08 +0000

The University of Vermont is conducting a survey to help understand how people receive credit for tasks in open source, and are seeking your participation!
Survey: https://qualtrics.uvm.edu/jfe/form/SV_1zUs19oVcZJ0SPA

Today's open source world has similarities to the world of open source when I first wrote about this problem in 2015. I wrote a tool that helps discover who is contributing to GitHub projects based on the contributions that aren't code. In 2022, we are still contributing countless hours of time towards communities to drive and improve open source across many industries, but we don't know how much of that work is credited beyond automated tools like this.

I'm a member of the OCEAN, a research team from the University of Vermont Complex Systems Center, and we're collaborating with industry experts from Google, OSI, IEEE, PSF, and OSSF, among others. You may have recently heard about our project on CHAOSSCast, or being shout-out on SustainOSS.

As we've written in our correspondence in Nature Computational Science, we believe that having a recognition model for the work done in any open source community can help more in just gaining credit for code, but providing clear signaling of the work done for the community.

As part of this work, the University of Vermont team is conducting a survey about how people receive credit for the tasks they do as part of open source projects.

The survey asks about your experiences with receiving credit on open source projects. The survey asks multiple choice questions about how often you received credit for the tasks you did, and written response questions about what did and did not go well on open source projects.

We hope that with this information, we can improve the equity of contributions across all of open source.

Click on this link to participate in the survey: https://qualtrics.uvm.edu/jfe/form/SV_1zUs19oVcZJ0SPA

OCEAN is a partnership between the Google Open Source Program Office and the Vermont Complex Systems Center. Google does not have any influence on the survey, and will be unable to see any individual responses.

When you're not around: trigger Cloud Run on a schedule

Katie McLaughlin — Fri, 22 Jan 2021 00:00:31 +0000

On this edition of Serverless Expeditions, we take a look at scheduling automated Cloud Run jobs with Cloud Scheduler.

Check out the video version of this blog post.

Source code for this blog post is available on GitHub under the "scheduled-cloud-run" folder.

GoogleCloudPlatform / serverless-expeditions

Cloud Run is designed to host a containerised web service -- any language you like, as long as the service listens on port 8080 -- but your code can run on events other than a user visiting your site.

An example of this design pattern would be a nightly billing job: you want to process billing data nightly at 1am, but you don't want to be the person to click a button in the middle of the night.

If you wanted to implement such a receiving service in NodeJS, you could implement it like in our example code: at the top of the code, create and initialize an Express app that listens for incoming HTTP calls. Note that Cloud Scheduler uses the “text/plain” content type, so you'll have to tell body-parser to parse more than the default “application/json” type. The POST handler parses the minimum balance that was sent to it from Cloud Scheduler, and then sends that off to the billing method later in the code.

Cloud Scheduler is a managed service offered on Google Cloud, where you can create jobs on a cron schedule that either makes a HTTPS call, or publishes a Pub/Sub message. In this example, you can setup a scheduled job that runs at 1am every night by entering the cron value 0 1 * * * in the Frequency field, and setting the HTTP Target to be the URL of your Cloud Run service you created earlier.

However, you need to make sure that this billing service is private, and not accessible by anyone on the internet. To convert a public service to private, you need to remove the allUsers member from the service -- that is, deny anyone from accessing it. To then allow Cloud Scheduler to access the service, create a new service account, assigning it the Cloud Run Invoker role. Then set that service account's email identifier in the Auth Header setting in Cloud Scheduler, under "Add OIDC token". OIDC, or OpenID Connect, is a small layer on top of OAuth that handles identity. This field tells Cloud Scheduler to use the new service account's identity for running the job.

The free tier of Cloud Scheduler allows you 3 jobs for free, no matter how many times the job is run. Each job after that is USD$0.10 each. You may also incur costs for excess Cloud Run execution time, database costs, etc.

By using Google Cloud managed services that connect to Cloud Run, you can build more complex architectures, and automate manual processes to make your developer live easier.

About Serverless Expeditions

Serverless Expeditions is a fun and cheeky video series that looks at what serverless means and how to build serverless apps with Google Cloud.

Follow these hosts on Twitter at @glasnt and @martinomander.

Running multiple versions of Python in Cloud Run

Katie McLaughlin — Fri, 11 Dec 2020 01:05:06 +0000

On this episode of Serverless Expeditions, we take a look at how to deploy multiple versions of Python in Cloud Run.

Check out the video version of this blog post.

Running multiple versions of Python in Cloud Run - Serverless Expeditions (YouTube.com)

As new versions of Python are released, trying to choose between the new and shiny or the old and stable can be an issue. But if you are deploying on Cloud Run, you don't have to choose between the two, you can run both.

Imagine this hypothetical scenario: you are an SRE for a company with a custom storefront, maintaining a fleet of virtual machines running various components, all with their own versions of Python. To improve reliability, you can upgrade all the components to use one version of Python, and move away from virtual machines, and into something more scalable, like serverless.

This upgrade won't happen overnight, and the storefront must keep running in the meantime. So your plan is multi-step: migrate from virtual machines to serverless first, then make the language upgrades. But before migrating the existing systems, a new service coming online could be deployed serverless first, skipping the virtual machine all together.

When choosing which serverless platform to upgrade to, limitations may exist on the languages available. For example, Cloud Functions only supports specific versions of Python.

However, the newest serverless offering from Google Cloud -- Cloud Run -- doesn't have this limitation. Cloud Run defines services as container images, which are defined by Dockerfiles, and can specify any base image. There is an entire repository of official Python images on Docker Hub which you can source from, by specifying the base image in the FROM line of your Dockerfile.

FROM python:2.7
...

FROM python:3.9
...

As you migrate each service from its original virtual machine, you can set the exact version of Python it was originally running. You can also upgrade each service in isolation, as your team has time and resources.

Once your dependencies have been upgraded and you want to test your service, you can deploy your new revision to only a limited number of users, using the Traffic Splitting feature of Cloud Run. You can set your new revision to serve, say, only 5% of requests, so if there is something wrong, only some of your users are affected, and you can quickly rollback if there are any issues.

By migrating each service in turn, you are able to control each service in isolation, allowing for a more incremental upgrade process, and a more reliable environment.

By being able to specify the exact version of Python being used in your project, you can run multiple services each with their own isolated Python environment, which allows you more time to upgrade each service to supported versions of Python.

It also allows you to try the newest versions of Python -- such as prereleases of Python 3.10 -- as soon as their images are available on Docker Hub.

About Serverless Expeditions

Serverless Expeditions is a fun and cheeky video series that looks at what serverless means and how to build serverless apps with Google Cloud.

Follow these hosts on Twitter at @glasnt and @martinomander.

"Point-and-click" continuous deployment with Cloud Run

Katie McLaughlin — Thu, 10 Dec 2020 20:36:47 +0000

On this edition of Serverless Expeditions, we take a look at how to deploy a GitHub repo on GitHub, though Cloud Run's new "point-and-click" integration, no Dockerfile required.

Check out the video version of this blog post.

Say you've created a website for a travel blog, and you want to host it online. You've got your code on GitHub, and you have a README of instructions of how a friend could deploy a copy of your website: they have to have your programming language installed on their machine, install some package dependencies, then run a command to start a web server.

If you want to host your website, you need a way that you can run these steps any time you want to update your code. This process is known as Continuous Deployment: when you commit a change to code a process is started that takes the new version of your code, performs the prerequisite tasks, and then starts a web server of your new code.

Cloud Run, a serverless platform from Google Cloud, allows you to to bring any container to serve your website, no matter the programming language used inside, as long as it follows the Runtime Container contract: a web server must be listening for requests on 0.0.0.0, on port 8080 (or the configured PORT environment variable).

So, if you have a website, how do you turn it into a container? You could write a Dockerfile, a set of instructions to build a container image based on your setup. But if you're using a common language like Python, Go, or Ruby, the instructions are going to be similar for many websites: copy your code into the container, install your dependencies, and start the web server.

You can setup your Cloud Run service through the Google Cloud Console using the new walkthrough integration with Cloud Build: you connect your GitHub repo with Cloud Run, and set up your website to be built any time your code updates.

However, instead of having to write a Dockerfile, you can now use Buildpacks, a set of common instructions for popular programming languages that help you build your code into a container image that Cloud Run can deploy.

If you're using a language like Go, Node.js, Python, Java or .NET Core, this will automatically be detected when building a container using Buildpacks. These common languages and their respective package installation processes are handled for you; all you need to do is provide the command to start your web server in a special file called Procfile.

In the example in the video, Martin is using Flask, a Python-based web framework. To start his application, he runs:

python3 -m flask run

So his Procfile will be:

web: python3 -m flask run

The "web" in this case means "web server", and is the special term Buildpack looks for in a Procfile to start the web server.

He should also configure his Flask app to listen on the expected IP and port. He can either add that to his app.run command, or in the Procfile itself as part of starting Flask:

web: python3 -m flask run --host 0.0.0.0 --port $PORT

As long as he adds this file to his codebase, he doesn't need to change anything else about his website to host it on Cloud Run.

Once configured, any commits made to the GitHub repo -- from the command line, VSCode, GitHub's website, or anywhere else -- are automatically detected and deployed.

To learn more about the concepts discussed in this video, read more:

You can try some of the sample buildpack apps at https://github.com/GoogleCloudPlatform/buildpack-samples

About Serverless Expeditions

Serverless Expeditions is a fun and cheeky video series that looks at what serverless means and how to build serverless apps with Google Cloud.

Follow these hosts on dev.to at @glasnt and @martinomander.

Deploying Django and Wagtail on Google Cloud

Katie McLaughlin — Thu, 01 Oct 2020 23:03:15 +0000

On this edition of Serverless Expeditions, we take a look at how to deploy more complex applications on Google Cloud, using Django as an example.

Check out the video version of this blog post.

When moving legacy applications into serverless infrastructure, there is going to be some sort of state involved: there will be a database, media, and other components. Traditionally these can all be run within the confines of a virtual machine, where all the services can talk to each other in that environment. But when moving to serverless, all these components need to be specifically provisioned and connected together.

While Cloud Run itself is stateless, it can be connected to other stateful services such as Cloud SQL for relational databases, or Cloud Storage for static media hosting.

In today's episode, we'll discuss containerizing and deploying a typical Django application using Cloud Run and other Google Cloud services. The steps described can be applied to any website whose CMS uses the Django web framework, such as Wagtail or Django CMS.

Django supports both PostgreSQL and MySQL, both of which are available in Cloud SQL, a managed database service. When connecting to a Cloud Run service, specifying --add-cloudsql-instances allows the service and the database to communicate over an automatically encrypted connection. The authentication from the application to the database is achieved by specifying a connection URL, stored securely in Secret Manager, loaded using the Python API. The database can then be populated by running ./manage.py migrate

Static and media assets can be migrated to use Cloud Storage, with help from the django-storages package. By creating a new storage bucket and specifying the name in the django settings.py, then running ./manage.py collectstatic, the images and assets are uploaded to the bucket.

However, those two manage.py commands -- migrate and collectstatic -- are typically run within an interactive terminal. Cloud Run has no such terminal, so these commands need to be run elsewhere. One method is to run these commands as part of the continuous integration process with Cloud Build. Adding a 'migrate' step using the app-engine-exec-wrapper between the typical build and deploy steps allows migration to happen as part of deployment.

If database migrations need to be manual, Cloud SQL proxy can be used to interact with a Cloud SQL database locally. Details of how to set this up with Docker Compose are available here.

With a managed database, highly-available static assets, automated deployment pipelines and robust containerisation solutions, you can migrate your Django applications to Google Cloud with confidence.

If you want to get started with these concepts, check out the Django Codelabs:

You can also find a sample implementation of these concepts at GoogleCloudPlatform/django-demo-app-unicodex.

About Serverless Expeditions

Serverless Expeditions is a fun and cheeky video series that looks at what serverless means and how to build serverless apps with Google Cloud.

Follow these hosts on Twitter at @glasnt and @martinomander.

Automating Python package releases by extending git

Katie McLaughlin — Mon, 10 Aug 2020 03:12:30 +0000

Like many automation workflows, you end up connecting different operations together that chain together.

In this post I'll describe how after initial setup, you can cut and release a new patch version of your GitHub hosted Python package by running:

git version-bump patch
git release

Link One

In Python's packaging ecosystem, versions are important. version must be declared in a package's setup.py. It doesn't have to be defined directly as a parameter of setup(), but it must be declared. You could setup a yourpackage/__version__.py, which allows you to at have the version stored outside of setup.py, but you still need to read from that file.

But why define the version in a file when you can dynamically query it from a git tag? How can you get setup.py to understand the version?

If you pip install versioneer and versioneer install it into your package, you can get versioneer to query git directly to get the current tag. This will work within the context of git. This is important later. Versioneer itself isn't a package dependency of your package, it's self-contained in a _versioneer.py file

Link Two

Git can execute external commands as if they were git commands. If you have an executable file in your PATH that starts with git-, e.g. git-thing, it also works as git thing (with a space rather than a hyphen). As of git v2.20.0, git help --all now lists these external commands. The implementation language of the executable doesn't matter; the interpreter used to run the executable is defined in the shebang.

Link Three

Git tags can be manually edited using git tag, but you can install a custom command to help automate the process of implementing say, semantic versioning, by allowing you to specify "bump patch" or "bump major" and the tag is updated based on the current version.

You can get this functionality as git version-bump if you gem install git-version-bump to automate semantic version git tag bumps. git-version-bump happens to be implemented in Ruby, but that doesn't matter as long as you have Ruby installed on your operating system.

Link Four

Git tags do appear in GitHub's web interface, but Releases are a richer experience. They are associated to a git tag, but allow attaching artefacts, and releases appear on the top level page of a repo.

You can automate publishing git tags as GitHub release by installing the github-release gem, which gives you the git release command. This uploads local tags to GitHub, using a custom API token setup for you within the command on first execution (including asking for your OTP). github-release is also written in Ruby.

Link Five

GitHub Actions are a now public feature which makes continuous deployment an integrated feature of GitHub. When you setup an Action on a Python repo, it's suggested that you could setup an action to publish to PyPI. This action is triggered when a GitHub release is created. Your PyPI username and password are stored as GitHub secrets. Within the scope of a running action, it is a git context. This was important from earlier.

Do you see where we're going with this?

By setting up versioneer to listen to git tags, and using git-version-bump to automate semver bumps of git tags, then github-release to convert git tags into GitHub Releases, which then trigger a GitHub Action to publish to PyPI, you can publish a new version of your package with just two git commands.

To get this setup on your existing package, install the local dependencies:

gem install git-version-bump
gem install github-release
pip install versioneer
versioneer install

Versioneer will guide you through the installation

And add a GitHub action for the publication of your package on GitHub release

Bonus Link One

If you setup versioneer and the GitHub Action, you can still trigger a new release by manually creating a GitHub Release from the website. Just be sure to git pull the new tags when you go back to the command line.

Bonus Link Two

You can follow the same release process for packages in other programming languages; you just need a way to query git tags into your package version declaration, and a workflow to perform publication. git-version-bump itself handles the git tag issue for ruby.

Bonus Link Three

If you're worried about installing random gems or ruby itself, you can first install rbenv (which pyenv was originally forked from), then rbenv install the latest version of ruby. Extending git only works if the executable is in your PATH, so you don't also need to install bundler just for this purpose.

Bonus Link Four (to be completed by the reader)

If you don't want worry about Ruby, you are more than free to create an executable shell scripts with the same functionality.

You could use a combination of git describe, semver-tool, and git tag to help with that part. It might even be a one-liner with enough pipes.

The creation of a GitHub release is a bit more complicated, but nothing you couldn't handle, I'm sure :)

Resources that helped me write this piece

Extending git, by Stefan Saasen, Atlassian.
github.com/git/git, a mirror of the git source code, and a URL that never stops being amazing.
My own post on this topic from 2014 from my Ruby days, which lead me to my own closed pull request on git-version-bump, wherein I was told the point "is to take this version information out of the filesystem", which lead me to find versioneer by searching "setup.py version from git tags" which sent me to this StackOverflow answer

This article was updated in August 2021 to ensure the line links to git source code were locked to a commit, and did not move over time.

Remember images in your cloudbuild.yaml!

Katie McLaughlin — Sun, 02 Aug 2020 23:43:02 +0000

If you've been using Cloud Build to automate Cloud Run service deployments for an amount of time, you might be familiar with the sort of configuration where you docker build, docker push, then gcloud run deploy your built container:

steps:
  - name: 'gcr.io/cloud-builders/docker'
    args: ['build', '-t', 'gcr.io/myproject/myservice', '.']

  - name: 'gcr.io/cloud-builders/docker'
    args: ['push', 'gcr.io/myproject/myservice']

  - name: 'gcr.io/cloud-builders/gcloud'
    args: ['run', 'deploy', 'myservice',
           '--platform', 'managed', '--region', 'us-central1',
           '--image',  'gcr.io/myproject/myservice']

The reason you have to docker push is because you want to make sure that the container repository you're using has the most recent version of your image. If you miss the push step on your first deployment, there's nothing in the registry to deploy!

Step #1: ERROR: (gcloud.run.deploy) 
Image 'gcr.io/myproject/myservice' not found.

And if you don't push each time, the version that's deployed won't be your most recent build! (Ask me how I know.)

So if you have to have a push step, what's the point of that images field?

images ↗

The images field in the build config file specifies one or more Docker images to be pushed by Cloud Build to Container Registry.

You're already pushing the image to the Container Registry, right?

Well, that's not all this configuration does.

If you go to your service in the Cloud Run part of the Cloud Console and navigate to the Revisions tab, you'll see information about that revision of your service:

Container image URL: the container the revision is using
Container build: (no build information available)
Source: (no source information available)

If you define images, even if it ends up pushing an image you've already pushed, the Container build field populates with a direct link to the Cloud Build job that built the image!

Even better, if you enable the Container Analysis API and you're running your builds based on build triggers, you get a link to the exact commit your container is based on in the Source field! (Container Analysis is free, and inspects container metadata. Not to be confused with Container Scanning, which has a cost, but also enacts vulnerability scanning)

First config, without images: config only shows Container URL.
Second config, with images: config shows a link to the Cloud Build job log.
Third config, with images and Container Analysis API: config shows additional link to the GitHub commit that triggered the build.

You can enable the Container Analysis API for your project here, or with gcloud:

gcloud services enable containeranalysis.googleapis.com

You can add images: to your cloudbuild.yaml by appending it to the end:

--- a/cloudbuild.yaml
+++ b/cloudbuild.yaml
@@ -11,6 +11,9 @@ steps:
            '--image', 'gcr.io/myproject/myservice']

+images:
+    - 'gcr.io/myproject/myservice'
+

Here's to a richer Cloud Run experience!

Learn more:

Generating a pseudorandom string: the what and the how

Katie McLaughlin — Fri, 10 Jul 2020 02:37:13 +0000

In a macOS/Linux environment, the following command will generate a randomish 30-character long string:

cat /dev/urandom | LC_ALL=C tr -dc '[:alpha:]'| fold -w 30 | head -n1

But just what is all this doing? And how is it doing it?

What is this command for?

No password is going to be 100% uncrackable, but there are many good reasons why your password should be long, complex, and not something easily remembered. This command uses these concepts and generates a string 30 characters long, using easily typeable characters that can be copy-pasted between programs, and will (in all likelihood) never return the same string twice. This string can be used for passwords, pass-phrases, etc.

For generating such strings without having to install specific software or utilities, this is a command that should just work.

Want a longer string? Change the 30 in the fold segment to something longer.

Want more strings at once? Change the 1 in the head segment to something larger.

How does this command work?

This command is many tools, working together.

One of the fundamental concepts of UNIX is "Do one thing, and do it well.". Linux borrows this philosophy, where each of the programs used in this command are from a suite of tools known as coreutils: literally, core utilities.

Each of these tools sends results to "standard output" (a.k.a. stdout; in this case, the terminal), which can be 'piped' from one program to another in the chain using the pipe character (| U+007C VERTICAL LINE, Shift + \).

`cat`

Another of the fundamental concepts of UNIX is "Everything is a file.". When you use cat, you con*cat*entate one or more files to stdout.

`/dev/urandom`

/dev/urandom is the preferred source of cryptographic randomness on UNIX-like systems.

Thomas Hühn, Myths about /dev/urandom

urandom is a special file that serve as pseudorandom number generators. This is an important term -- pseudorandom -- as this output isn't truly random, since it starts from known seed values. Hühn's article describes this in more detail.

However, for the purposes of generating a seemingly random sequences of characters, this is a good place to start.

However, do not just concatenate the contents of urandom to stdout. It is infinite, and full of unprintable characters. The output a pseudorandom sequence of bytes, only a subset of which are valid Unicode characters. Your terminal will interpret these in strange ways, with replacement characters and other not-very-pleasent-looking output. Which is where the next part comes in.

`tr`

tr allows the translation of characters. You can perform single character replacements by specifying two sets of characters, or you can specify one set to delete.

`tr -dc`

By specifying one set of characters with -d, you will delete all characters matching that set. However, by specifying -c, you specify the complement of the set.

`tr -dc '[:alpha:]'`

In this case, there is a special shortcut for all alphanumeric characters: [:alpha:]. That is, the combination of a through z, A through Z, and O through 9 (or a-zA-Z0-9).

Thus, piping the output of urandom through this tr command will return only letters and numbers.

`LC_ALL=C`

So far we've been dealing with programs from coreutils. However, these also exist in BSD flavours, but behave slightly differently. In the case of tr, performing the previous command on a macOS operating system (for example) will probably use the BSD version of tr, which returns the error Illegal byte sequence. This is due to the BSD version of tr being highly confused at the random bytes that we mentioned earlier.

On macOS if you run brew install coreutils, you can get homebrew to install the coreutils versions of these base programs that will be more familiar to the versions used on Ubuntu, Debian, etc. There are subtle differences between the BSD and coreutils versions of these utilities that expose themselves in unique ways; this is is one of them.

As an alternative to installing any additional programs, the LC_ALL=C shortcut is a workaround that forcing the "locale" of the command to use "C", being the C programming language, which uses the ASCII charset (256 characters rather than Unicode's 16 million). This will effectively ignore all the invalid bytes urandom is potentially outputting.

(You can read more on the background and details of these concepts on these StackOverflow replies: link one, link two)

`fold -w 30`

fold wraps -- or "folds" -- output to 80 columns. The column count is overridden with -w 30 to force the output to be 30 columns, or in our case, 30 ASCII characters, wide.

`head -n1`

As the name suggests, head returns the head -- or the first part -- of files, to a default of 10 lines. This default is overridden with -n1 to force the number of lines down to one.

Space, or no space?

In the fold and head examples, short options (-w, -n) where given parameters where there may or may not have had to have a space between the option and argument (30, 1). This is an artefact of a combination of POSIX and getopt. The effect: as a general rule short options (single hyphen, single character, e.g. -n) that are exactly one character long can be parsed by that length, as opposed to a space or an equals sign. Long options (double hyphen, longer names, e.g. --lines) can either have spaces or equals signs.

For example, the following invocations of head are identical:

head -n1
head -n 1
head --lines=1
head --lines 1

However, in this case -n=1 and --lines1 are invalid.

This is just a general guideline, as there are some options parsers that follow different standards (for instance, having long options denoted by single hyphens). Always check the manual!

Checking your version in the manual

You can see which version of these utilities your system is running by checking the manual, or man.

Checking the first and last line of man [program] will normally show information like the version number, the date of the release, the Unix family of tool, and often the name of the command followed by a number in brackets. This is the "section", which is useful when some names are overloaded; say, stand-alone system programs and also C libraries.

The manual is also useful for learning about what other options, long and short, are available to the utility, any example invocations, and much more.

All StackOverflow links in this article are shared CC BY-SA 3.0

Pure serverless Django with django-gcloud-connectors

Katie McLaughlin — Tue, 07 Jul 2020 03:28:02 +0000

Django officially supports a number of database backends -- including PostgreSQL, MySQL, and SQLite -- which means you can write your Django project once and deploy it to any of these database backends.

But, there are third-party packages to support other databases, even non-relational databases like Google Datastore.

In this project, I used django-gcloud-connectors, newly released on pypi as a 0.1.0 from Potato London, to make a Django deployment on Google Cloud completely serverless (and in theory, free!)

Sample project code → github.com/glasnt/mewgram

Sample project screenshot ⤵

I spoke at my first DjangoCon Europe back in 2016 in Budapest, where I saw an intriguing talk by Adam Alton: Building A Non-Relational Backend For The ORM (video). His talk centered around djangae, which allows vanilla Django sites deployed to App Engine to use Datastore, a highly scalable NoSQL database (currently referred to as "Firestore in Datastore Mode").

However, djangae -- as the name suggests with the GAE at the end -- requires Google App Engine. However, I'm a containers girl and I prefer Cloud Run, so I thought, how hard could it be to use the bits of djangae just to get the database backend into Datastore working?

Turns out, it's not hard when djangae itself uses django-gcloud-connectors.

By installing and integrating django-gcloud-connectors into a sufficiently simple Django app, and with some learning about how indexes in Datastore work, I was able to put together a proof of concept that deploys using the Cloud Run Button onto Google Cloud Run, allowing you to deploy this sample app and see it working today.

But why do you need a purpose-built backend connector?

You can use google-cloud-datastore to interact directly to Datastore with Python. However, if you want to take advantage of Django's ORM, you need to use a connection that maps Django's models to that specific database. The officially supported databases are officially supported for a reason -- the eccentricities of having a generic ORM interfacing to any relational database is not easy, and keeping them up to date for popular databases is important.

The folks at Potato have -- incredibly -- made Django (mostly) work on a non-relational database. I say mostly for a reason -- there are some limitations with Datastore itself which means some of the more complex functionality that is taken for granted in a relational database just isn't available for Datastore, and as such djangae and django-gcloud-connectors cannot support it. I came across issues where cross-join where filters were not supported when trying to migrate unicodex. Take a look at the NotSupportedError exceptions in parser/base.py for more examples.

Did you say free?

Well, yes, mostly. While you do need to have billing enabled to deploy mewgram, Datastore itself has a generous free tier, as do the other elements involved. I normally use Postgres for my Django demos, like unicodex, and the smallest Cloud SQL database has a non-zero monthly cost. In theory, unless your application gets super popular, you should have no ongoing costs running a Django project backed to Datastore. As always, consider the costs involved and setup billing alerting for long-running projects!

Cats?

Yup, the identicons are provided by robohash, a Python package that has a cat setting.

Why connector*s*?

The package is called django-gcloud-connector*s* -- plural -- the project README currently only supports Firestore in Datastore mode, but hopes to support Firestore (in Native mode) in the future. With Django 3.1 slated to introduce even more ASGI compatibility, I look forward to hopefully one day having a full asynchronous Django, with a realtime database to boot.

You can learn more about django-gcloud-connectors at gitlab.com/potato-oss/google-cloud/django-gcloud-connectors, and Potato London at https://p.ota.to/.