Configure Symfony Secrets with Hashicorp Vault

tl;dr: Integrating Vault and Symfony does not require any PHP code. Using vault-agent secrets variables can be dumped into .env file. Dynamic secrets can even be used as feature flags.

---

Prisma Media's websites and applications are mostly developed using Symfony framework. Many of them require secrets values: API keys, database credentials, private certificats… they needs to be treated carefully.

Why we choose Vault to store secrets

If you are there, you may know that storing secrets in your Git repository is a terrible practice that could lead to severe security issues.

A cryptographic "Vault" mecanism is proposed by Symfony. It uses encrypted secrets that can be spread in repositories and artifacts. This is a simple solution for basic needs. That has the advantage of being independent from any external system (simplicity, scalability).

We preferred HashiCorp Vault, a server solution that we deployed on our infrastructure. It is centralized, auditable and can handle dynamic secrets. This open-source product is not tied to a cloud provider. Easy to run locally for dev.

Vault works as a key-value store where secret variables are read and written using a REST API.

The need for frequent reload of secrets

Even if we generate the most secure password for the database, it will leak somewhere: logs, APM, error pages... To be safe, credentials needs to change, change all the time.

To get short-living secrets, Vault has a concept of Dynamic Secrets. Unlike key/value secrets where you had to put data into the store yourself, dynamic secrets are generated when they are accessed.

Not only we have to load the secrets from Vault, but also they have to be reloaded every time they change.

Dynamic config of Symfony with .env

Fabien Potencier

@fabpot

My yearly reminder that you should NOT use env vars to configure your #Symfony application. Use env vars ONLY if you want to be able to change values WITHOUT redeploying your application (aka flushing the cache). Parameters ARE the way to configure applications.

08:54 AM - 29 Dec 2020

Thank you Fabien, this is exactly what I need: changing the values without redeploying the application.

Unlike the YAML/PHP files in config/ directory, that are read only when the container cache is built, the .env file are read on every HTTP request. Updating this file while the application is running is a good way to update its configuration without impacting performance.

Read more: Configuring Symfony (slides by Nicolas Grekas)

Vault Agent can write secrets into a file

vault-agent is a small utility that can be used as a cache proxy for the Vault server; or as a schedule to write secrets into a file using a template (every 5 minutes, configurable).

Example of vault-agent configuration for an app running on AWS EC2 instance.

# vault.conf
vault {
  address = "https://vault.example.com"
  retry {
    num_retries = 5
  }
}

auto_auth {
  method "aws" {
    config = {
      type = "iam"
      role = "<iam role>"
      region = "eu-west-1"
      header_value = "vault.example.com"
    }
  }
}

template {
  source = "./.env.local.ctmpl"
  destination = "./.env.local"
}

The template maps Vault secrets to environment variables. The templating syntax allows some flexibility, but it looks very primitive for a developer with Twig practice.

# .env.local.ctmpl
# This will generate a regular .env file
APP_ENV=prod

{{ with secret "secret/example/app" }}
APP_SECRET={{ .Data.data.APP_SECRET }}
{{ end }}

# Real environment variables can be read
{{ $env := (env "ENVIRONMENT") }}
{{ with secret (printf "secret/example/%s/database" $env) }}
DB_HOST={{ .Data.data.DB_HOST }}
DB_NAME={{ .Data.data.DB_NAME }}
DB_USER={{ .Data.data.DB_USER }}
DB_PASSWORD={{ .Data.data.DB_PASSWORD }}
{{ end }}

Finally, vault agent can be launched with any process manager (supervisord or systemd) or scheduler (crond).

# Daemon for prod server
vault agent -config=vault.conf

# Single run for testing
vault agent -config=vault.conf -exit-after-aut

The last security recommendation is to not write secrets on disks. We can create a tmpfs volume and symlink from the project root to that volume.

For kubernetes, vault-agent runs in a sidecar container that renders Vault secrets to a shared memory volume.

Use dynamic configuration feature flag

Feature flags are a benefit of using Vault and supporting dynamic configuration. Even if they are not secrets, flags can be stored in Vault. With fine tuned policies, product managers could manage feature flags and being rejected from other secrets.

Example of feature flag to render a block in Twig

In this example, we create a feature flag in Vault, with is a boolean to show or hide a "sales" block on a page.

Create a variable in Vault KV:

# secret/example/app/
{
  "FEATURE_FLAG_SALES": "true"
}

Read this secret in the vault agent template

# .env.local.ctmpl
{{ with secret "secret/example/app" }}
FEATURE_FLAG_SALES={{ .Data.data.FEATURE_FLAG_SALES }}
{{ end }}

Share the value of the variable with Twig context.

# app/config/packages/twig.yaml
twig:
  variables:
    feature_flag_sales: "%env(bool:FEATURE_FLAG_SALES)%"

Use the variable to render the block conditionally:

{% if feature_flag_sales %}
    <div>My conditional sales block</div>
{% endif %}

Run the vault-agent.
When the variable updated, the block is shown or hidden after few minutes.

Read more...

• Tutorial: Read Secrets From Vault Using Vault Agent
• Article: Why Use the Vault Agent for Secrets Management?

Comments

[Alexander Frolov]

just saying. .env can be transformed into .env.local.php with composer dump-env prod and used with preload (or just with opcache) to eliminate the need of re-reading and re-evaluating env variables on every request

[Jérôme TAMARELLE]

After your remark, I found that vault-agent have an option to run a command to run composer dump-env prod when .env is actually modified.

command (string: "") - This is the optional command to run when the template is rendered. The command will only run if the resulting template changes. The command must return within 30s (configurable), and it must have a successful exit code. Vault Agent is not a replacement for a process monitor or init system.

template {
  source = "./.env.local.ctmpl"
  destination = "./.env.local"
  command = "composer dump-env prod"
}

[Jérôme TAMARELLE]

Good tip for performance, but this transformation needs to be done each time vault-agent updates the credentials. If the PHP file is loaded with preloading, php-fpm have to be reloaded to get the updated values.

The .env file reader is quite fast, but it could be optimised by using a native format like .ini

[Alexander Frolov]

Yeah. Sorry, I lost my touch with forums in slack/discord/etc era. I mostly was referring to the possibility that folks use not only .env

ini is not native format in any way. Parsing is quite simple and fast, but it still needs to be parsed. The only native format for PHP is array 😅

---

Let's say this .env file updating via vault-agent is atomic. How you handle change in credentials "mid-flight"? Just ignoring errors? Re-reading configuration mid-request?

[Jérôme TAMARELLE]

Good question. I run the described solution on production for months, with high HTTP request throughout, without issue. Nothing in logs sound like the app read a partial file. I'll make some deep tests to understand how that works.

---

PHP brings the function parse_ini_file that uses the C implementation. That's what I was meaning with "native". .env and YAML parsers are written in PHP and uses regex (inherently slower).

[Alexander Frolov]

First of all speed of regex (pcre2 library) may surprise your greatly) Second - to parse env on the most basic level you just need to split the line by = and process quotes on the right... but I digress

---

Nothing in logs sound like the app read a partial file

That's why I mentioned atomicity - this will guarantee that file always updates as a whole and instantaneously. Hypothetical scenario I'm talking about looks like this

• credentials rotation is triggered by timer
• credentials are changed in vault
• now there is some time between actual update in central vault and vault-agent reaction
• after that there is some time between vault-agent updating .env file and app reading it
• even after that there is a possibility when app read old .env file (in the beginning of the request) and 1ms later .env was updated

In all last three scenarios credentials that app has in memory are already obsolete. That will not matter for already established connections but should matter greatly for new ones.

The only way I see to mitigate it is to rotate "whole credentials" as user/password pair altogether and keep old ones active for a little while after update. In which case I am quite curious about how it's handled say with database of any kind where user/password is not enough, you also need provide the same level of permissions (say to database / tables / operations)

Ok. Nevermind, found my answer here learn.hashicorp.com/tutorials/vaul...

Sorry to bother😅

[Jeremy Moore]

Great write-up. I've been planing on integrating Vault in the future for our websites and CI/CD.