I recently had a chance to work with Google Cloud App Engine. I haven't worked with it before, but I used to host a couple of apps on Heroku and OpenShift, so it was interesting to compare these platforms.
We were working on a fairly standard Node.js application with most of the configuration done with environment variables. So very quickly we found out that App Engine has a major difference with the most of other PaaS offerings – it does not support configurable environment variables.
Intentional or not, App Engine has only one way of defining those variables – in
app.yaml configuration file. This file describes App Engine settings (runtime, url mappings etc.), including
env_variables section that instructs App Engine to set environment variables on deployment.
Our deployment pipeline is fully automated, so we needed to store
app.yaml file somewhere to supply to build server before pushing code to Google Cloud. Ideally, it would be application code repository. However, having environment variables in
app.yaml file became a blocker. We either needed to commit application configuration to the repository, or leave the whole file unmanaged. Neither of these options worked for us, so I started looking for any other ways of dealing with this App Engine limitation.
As a side note, Heroku and OpenShift (at least its previous incarnation) have an option to set environment variables from the web interface, which simplified configuration management for applications like ours. It allows us to deploy the same codebase to different environments by changing only environment variables (e.g. test vs production API endpoints, application domains, cookie encryption keys, etc.).
My search brought me some disappointing results:
- Store application configuration in the Google Cloud Datastore and read from it on application startup (link).
- Encrypt configuration values with Cloud KMS and commit together with the rest of the
- Use separate
app.yamlfiles for different environments (and, I guess, commit them all to the repository?) (same link).
Option #1 assumes vendor lock-in to Google Cloud Datastore database, which we were trying to avoid with our application.
Option #2 solves the security part of the problem, but would mean hardcoding encrypted environment-specific values to the codebase. It would also require to update the code with each new environment we add to the project or if we change anything in any existing environments. Not ideal.
Option #3 does not solve anything – code would still store information about its environments and application secrets will be available right in the code repository...
We found a different approach: compile
app.yaml file from a template during the build process. At that moment we used Google Cloud Build as a build server for CI/CD, but quickly moved to GitLab CI since Cloud Build does not support environment variables either.
To be fair, I should mention that Cloud Build supports "substitutions", which are remotely similar to environment variables. Unfortunately, the only way to pass substitutions to the build job is through command line arguments, which means managing environment variables somewhere outside. And this brings us back to the original problem...
Our application is already using EJS library, so I used it to compile the template:
# app.tpl.yaml runtime: nodejs10 env_variables: SSO_CLIENT_ID: <% SSO_CLIENT_ID %> SSO_SECRET: <% SSO_SECRET %>
with a script like that:
// bin/config-compile.js const fs = require('fs'); const ejs = require('ejs'); const template = fs.readFileSync('app.tpl.yaml').toString(); const content = ejs.render(template, process.env); fs.writeFileSync('app.yaml', content);
and GitLab CI step similar to that:
# .gitlab-ci.yml config-compile: stage: build image: node:10 script: - node bin/config-compile.js artifacts: paths: - app.yaml expire_in: 1 days when: always
Now we can manage application configuration in GitLab environment variables.
This approach can easily be adapted to any other templating library, programming language or build server, the code above is just an example.
I found it interesting that one of the oldest Google Cloud products doesn't support such a common functionality. However, I accept there could be valid security reasons for doing so, e.g. exposure to dependency vulnerability similar to the one discovered in rest-client or typo-squatting like in npm registry.
- "Secrets in Google App Engine" by Stuart Leitch
- "How to use Environment Variables in GCloud App Engine" by Gunar Gessner