One of my recent projects involved an application that had to be hosted on Google Cloud App Engine. I didn't use App Engine before, but I've managed apps on Heroku and OpenShift and was interested to see what Google Cloud PaaS had to offer.
It was a fairly standard Node.js application with most of the configuration done with environment variables. Soon it became clear that this could be a problem – App Engine does not support configurable environment variables.
Intentional or not, App Engine has only one way of defining those variables – in
app.yaml configuration file. This file describes App Engine settings (runtime, url mappings etc.), including
env_variables section that instructs App Engine to set environment variables on deployment.
app.yaml file content:
runtime: nodejs10 handlers: - url: /api/.* script: auto secure: always - url: /.* static_files: index.html upload: index.html secure: always http_headers: X-Frame-Options: deny X-DNS-Prefetch-Control: off X-XSS-Protection: 1; mode=block X-Permitted-Cross-Domain-Policies: none env_variables: VAR1: 'VALUE1' VAR2: 'VALUE2'
Our deployment pipeline was already fully automated, so I needed to store
app.yaml file somewhere to supply it to build server before pushing code to Google Cloud. Ideally, it would be application's code repository. However, having environment variables in
app.yaml file caused an issue: we either needed to commit application configuration to the repository, or leave the whole file untracked. Neither of these options was suitable, so I started looking for any other ways of dealing with this App Engine limitation.
As a side note, Heroku and OpenShift (at least its previous incarnation) have an option to set environment variables from the web/command line interface, which simplified application configuration management.
My search brought me some disappointing results:
- Store application configuration in the Google Cloud Datastore and read from it on application startup (link).
- Encrypt configuration values with Cloud KMS and commit together with the rest of the
- Use separate
app.yamlfiles for different environments (and, I guess, commit them all to the repository?) (same link).
Option #1 assumed additional component in our infrastructure and vendor lock-in to Google Cloud Datastore database, which was far from ideal.
Option #2 solved the security part of the problem, but would mean hardcoding encrypted environment-specific values to the codebase. It would also require to update the code with each new environment added or any changes to the existing environment variables. Not ideal.
Option #3 didn't solve he problem at all – code would still store information about its environments and application secrets will be available right in the code repository...
Eventually, I've came up with an approach that involved compiling
app.yaml file from template file during the build process. At that moment we used Google Cloud Build as a build server for CI/CD, but quickly moved to GitLab CI since Cloud Build does not support environment variables either.
To be fair, I should mention that Cloud Build supports "substitutions", which are remotely similar to environment variables. Unfortunately, the only way to pass substitutions to the build job is through command line arguments, which means managing environment variables somewhere outside. And this brings us back to the original problem...
The application was already using EJS library, so I used it to compile the template:
# app.tpl.yaml runtime: nodejs10 env_variables: SSO_CLIENT_ID: <% SSO_CLIENT_ID %> SSO_SECRET: <% SSO_SECRET %>
with a script like that:
// bin/config-compile.js const fs = require('fs'); const ejs = require('ejs'); const template = fs.readFileSync('app.tpl.yaml').toString(); const content = ejs.render(template, process.env); fs.writeFileSync('app.yaml', content);
and GitLab CI step similar to this:
# .gitlab-ci.yml config-compile: stage: build image: node:10 script: - node bin/config-compile.js artifacts: paths: - app.yaml expire_in: 1 days when: always
This enabled us to manage application configuration in GitLab environment variables.
The approach can easily be adapted to any other templating library, programming language or build server, the code above is just an example.
I found it interesting that one of the oldest Google Cloud products doesn't support such a common functionality. However, I accept there could be valid security reasons for doing so, e.g. exposure to dependency vulnerability similar to the one discovered in rest-client or typo-squatting like in npm registry. This is especially relevant as App Engine does not provide options to limit/manage outgoing connections from the environment.
- "Secrets in Google App Engine" by Stuart Leitch
- "How to use Environment Variables in GCloud App Engine" by Gunar Gessner