DEV Community: Mattias Fjellström

The Reluctant Software Developer: Contributing to HashiCorp Vault

Mattias Fjellström — Wed, 13 Dec 2023 21:10:31 +0000

First, a confession. I am not a software developer, or a programmer. I enjoy writing code, and I am confident in using Python, Go, and TypeScript. I am uncomfortable using Haskell, but I am eager to learn more.

What I am is a cloud architect. I work more in the interface between code and cloud services.

With that confession out of the way, let me move on. A few months ago I spent some intense time learning HashiCorp Vault, enough to feel confident in using Vault day-to-day and to later pass the associate certification.

Summarizing the Vault certification in four large parts we have:

Vault architecture
Vault CLI
Vault UI
Vault API

This is very simplified! It would be intimidating if this was the official list of things to learn to pass the exam.

The UI and API part of the exam is small, but the CLI part is huge. You need to know the commands. Luckily they are easy to learn.

During my learning journey I stumbled upon a missing feature in the CLI. In the Vault documentation for token accessors I read that you can do the following things with the accessor:

When tokens are created, a token accessor is also created and returned. This accessor is a value that acts as a reference to a token and can only be used to perform limited actions:

Look up a token's properties (not including the actual token ID)

Look up a token's capabilities on a path

Renew the token

Revoke the token

I tried to perform these actions using the CLI and discovered that it was not possible to list a token's capabilities on a path using its accessor (point 2 in the list above).

If it would have worked the command should have looked like this:

$ vault token capabilities -accessor <accessor value> /the/path

But it did not work! I wrote this discrepancy down in my list of notes for later.

Now it is "later"

I have not made any contributions to large collaborative projects like HashiCorp Vault before, so I was a bit hesitant to even get started.

As with any open-source¹ or source-available projects, it is a good idea to start reading the CONTRIBUTING.md file in the repository to see if there are any special guidelines to follow. For Vault (and I assume it is similar for other HashiCorp projects) there were a few things to keep in mind:

Connect your pull-request to an open issue

The first guideline to follow was to make sure you work on something that has an associated issue:

When submitting a PR you should reference an existing issue. If no issue already exists, please create one. This can be skipped for trivial PRs like fixing typos.

I searched the list of open issues for anything related to token capabilities. A few issues came up but nothing to do with what I wanted to implement concerning accessors. So I created my own issue (or feature request in the form of an issue).

To be honest, I waited with this part until I was done implementing the feature. I wanted to make sure I would be able to deliver a solution!

My issue: #24478.

Describe your work

I confess I cheated a bit with this guideline. This guideline asked you to describe your work in the pull-request:

Your pull request should have a description of what it accomplishes, how it does so, and why you chose the approach you did.

I took inspiration from other PRs where the description just included a link to the issue it solves. The issue itself contains enough details to describe what is going on. I guess DRY (don't repeat yourself) also applies to issues and pull-requests.

Include tests

As any good developer know you should test your code:

PRs should include unit tests that validate correctness and the existing tests must pass.

Of course you should add tests for the functionality you are adding. I took inspiration from the tests that were already written for the token capabilities command.

I added tests to verify that the correct number of arguments were provided to the command if the -accessor flag was added. I also added a test to verify that the correct capabilities were returned if a valid accessor was used.

Add a changelog entry

The last guideline to follow was to add a specific file describing the change:

Please include a file within your PR named changelog/#.txt, where # is your pull request ID. There are many examples under changelog, but the general format is
release-note:CATEGORY
COMPONENT: summary of change

After submitting the pull-request I copied the number it was given and created a new file named changelog/24479.txt with a summary of my change.

The change

So what did I change? Let me tell you!

I did not have to spend hours and hours understanding the structure of the code, it only took me a few minutes to find where I should implement my change. I wanted to update the token capabilities command, so I opened the command directory and found the token_capabilities.go file. It's almost like professional software developers have created this codebase.

First of all I had to add the accessor flag to the command. I did this in the TokenCapabilitiesCommand type:

type TokenCapabilitiesCommand struct {
    *BaseCommand

    // add this
    flagAccessor bool
}

I had to update the help text for the command to reflect the added flag. I got some feedback from a Vault developer here, read more about that in the next section.

Next up was to update the Flags() method on the TokenCapabilitiesCommand type. Before my change it looked like this:

func (c *TokenCapabilitiesCommand) Flags() *FlagSets {
    return c.flagSet(FlagSetHTTP | FlagSetOutputFormat)
}

I updated it to the following:

func (c *TokenCapabilitiesCommand) Flags() *FlagSets {
    set := c.flagSet(FlagSetHTTP | FlagSetOutputFormat)

    f := set.NewFlagSet("Command Options")

    f.BoolVar(&BoolVar{
        Name:       "accessor",
        Target:     &c.flagAccessor,
        Default:    false,
        EnvVar:     "",
        Completion: complete.PredictNothing,
        Usage:      "Treat the argument as an accessor instead of a token.",
    })

    return set
}

Here I specified details about the accessor flag and text that will appear when you use the help command.

Next up I needed to update the Run method on the TokenCapabilitiesCommand type. It was using a simple switch statement to check for the number of arguments provided, and taking different actions depending on how many. I needed to instead use a more complex switch statement:

switch {
case c.flagAccessor && len(args) < 2:
    c.UI.Error(fmt.Sprintf("Not enough arguments with -accessor (expected 2, got %d)", len(args)))
    return 1
case c.flagAccessor && len(args) > 2:
    c.UI.Error(fmt.Sprintf("Too many arguments with -accessor (expected 2, got %d)", len(args)))
    return 1
case len(args) == 0:
    c.UI.Error("Not enough arguments (expected 1-2, got 0)")
    return 1
case len(args) == 1:
    path = args[0]
case len(args) == 2:
    token, path = args[0], args[1]
default:
    c.UI.Error(fmt.Sprintf("Too many arguments (expected 1-2, got %d)", len(args)))
    return 1
}

Some of this code was present in the previous implementation, but to summarize my change I had to check if the -accessor flag was provided to the command, and if so I needed to check that the correct number of arguments was provided.

Further down in the Run method there was the following check:

if token == "" {
    capabilities, err = client.Sys().CapabilitiesSelf(path)
} else {
    capabilities, err = client.Sys().Capabilities(token, path)
}

This just checked if a token was provided or not. If it was not provided, it used the CapabilitiesSelf method to see the capabilities of the current token. If a token was provided it instead used the Capabilities method to check the capabilities for the provided token. With the -accessor flag in the mix I had to add a third case. I needed to update this from an if-else check to another switch statement:

switch {
case token == "":
    capabilities, err = client.Sys().CapabilitiesSelf(path)
case c.flagAccessor:
    capabilities, err = client.Sys().CapabilitiesAccessor(token, path)
default:
    capabilities, err = client.Sys().Capabilities(token, path)
}

This change included the case where a token was not provided but the -accessor flag was provided. Note that if the -accessor flag was included, there is a call to client.Sys().CapabilitiesAccessor(...). I had to implement this CapabilitiesAccessor method!

The home for that code was in api/sys_capabilities.go.

First of all I added the CapabilitiesAccessor method:

func (c *Sys) CapabilitiesAccessor(accessor, path string) ([]string, error) {
    return c.CapabilitiesAccessorWithContext(context.Background(), accessor, path)
}

This is just a wrapper for the CapabilitiesAccessorWithContext method, with the context argument added.

I also implement the CapabilitiesAccessorWithContext method:

func (c *Sys) CapabilitiesAccessorWithContext(ctx context.Context, accessor, path string) ([]string, error) {
    ctx, cancelFunc := c.c.withConfiguredTimeout(ctx)
    defer cancelFunc()

    body := map[string]string{
        "accessor": accessor,
        "path":     path,
    }

    reqPath := "/v1/sys/capabilities-accessor"

    r := c.c.NewRequest(http.MethodPost, reqPath)
    if err := r.SetJSONBody(body); err != nil {
        return nil, err
    }

    resp, err := c.c.rawRequestWithContext(ctx, r)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()

    secret, err := ParseSecret(resp.Body)
    if err != nil {
        return nil, err
    }
    if secret == nil || secret.Data == nil {
        return nil, errors.New("data from server response is empty")
    }

    var res []string
    err = mapstructure.Decode(secret.Data[path], &res)
    if err != nil {
        return nil, err
    }

    if len(res) == 0 {
        _, ok := secret.Data["capabilities"]
        if ok {
            err = mapstructure.Decode(secret.Data["capabilities"], &res)
            if err != nil {
                return nil, err
            }
        }
    }

    return res, nil
}

I won't go into details of the code, it is mostly a wrapper for calling the API which was already implemented (lucky me!) If it had not been implemented there would have been a lot more work, and probably not something I could have completed on my own.

Now I was done. I did add tests for this change as well, since it was one of the guidelines to follow. I won't cover the tests here.

Feedback from Vault developers

I got a wonderful comment from one Vault developer (you can read the conversation in the PR):

Looks good.

Then I got some feedback on what the help text for the command should be. I think the suggested changes was good, and accepted them without hesitation.

I am glad the code I wrote was acceptable!

The result

I want to summarize my work with a famous quote:

That's one small step for man, one giant leap for mankind.

Replace mankind with Mattias in this quote from Niel Armstrong and you have the feeling I am experiencing right now.

It was a small change. But a huge boost in confidence for me.

I have no plans to be a recurring contributor to Vault, or any other HashiCorp codebase. But this was a great experience and if I stumble upon something else I think is missing I will give it a go!

But, am I a software developer now?

Mentioning open source and HashiCorp in the same blog post will probably trigger some of you, but I know HashiCorp products are no longer open-source. It does not matter to me, but please let me know why it matters to you! ↩

Test permutations with Terraform and GitHub Actions

Mattias Fjellström — Thu, 23 Nov 2023 17:20:32 +0000

I have been exploring the new test framework for Terraform 1.6 extensively since HashiConf in October this year. I have already written two very long posts on the topic of testing and validation with Terraform that you can read here:

In this post I want to illustrate a pattern for scaling up your testing using GitHub Actions.

Scenario

You are part of a platform team developing a Terraform module that sets up an Azure storage account according to a specification appropriate for your organization.

Your module has dependency on a module created by a different team in your organization. That module sets up an Azure resource group where the storage account your module produces is placed.

The source code for your module is stored in a GitHub repository and you want to use GitHub Actions to perform testing before you publish new versions of your module.

The scenario is illustrated in the figure below:

You expect that new versions of your module should be compatible with the three most recent minor versions of the other team's resource group module that you depend on. You also expect that your module works as intended for each of the Azure regions that your organization is operating in. These regions are swedencentral, northeurope, and westeurope.

Your team has already figured out that there will be many tests to write to cover the permutation of the above criteria. To be precise, each individual test you write will need to be repeated 3x3=9 times (three locations, three versions).

The Terraform module your team is developing consists of a single main.tf file (to keep this scenario simple):

// main.tf
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.80.0"
    }
    random = {
      source  = "hashicorp/random"
      version = "3.5.1"
    }
  }
}

provider "azurerm" {
  features {}
}

variable "resource_group_name" {
  type = string
}

resource "random_id" "this" {
  keepers = {
    resource_group_name = var.resource_group_name
  }
  byte_length = 6
}

data "azurerm_resource_group" "this" {
  name = var.resource_group_name
}

resource "azurerm_storage_account" "this" {
  name                      = "st${random_id.this.dec}"
  access_tier               = "Hot"
  account_kind              = "StorageV2"
  account_replication_type  = "LRS"
  account_tier              = "Standard"
  resource_group_name       = data.azurerm_resource_group.this.name
  location                  = data.azurerm_resource_group.this.location
  enable_https_traffic_only = true
  tags                      = data.azurerm_resource_group.this.tags
}

The important parts to notice in this module is that it takes an input variable named resource_group_name:

variable "resource_group_name" {
  type = string
}

This variable is used in a data source for a resource group:

data "azurerm_resource_group" "this" {
  name = var.resource_group_name
}

And this data source is later referenced in arguments of the storage account resource:

resource "azurerm_storage_account" "this" {
  // ... 

  resource_group_name = data.azurerm_resource_group.this.name
  location            = data.azurerm_resource_group.this.location
  tags                = data.azurerm_resource_group.this.tags
}

For illustrative purposes your team is interested in running the following test as defined in tests/main.tftest.hcl:

// tests/main.tftest.hcl
run "setup" {
  variables {
    location    = "swedencentral"
    name_suffix = "tftest-swedencentral-1.1.0"
  }

  module {
    source  = "app.terraform.io/your-tf-org/resource-group-module/azurerm"
    version = "1.1.0"
  }
}

run "proper_tags_should_be_propagated" {
  variables {
    resource_group_name = run.setup.resource_group.name
  }

  command = apply

  assert {
    condition     = alltrue([
      contains(keys(azurerm_storage_account.this.tags), "source"),
      contains(keys(azurerm_storage_account.this.tags), "module")
    ])
    error_message = "Proper tags are not propagated to the storage account"
  }
}

There are two run blocks in this test file. The first run block named setup uses the module you depend on. In this case it specifically uses version 1.1.0 of the module. This run block defines a location variable that is set to swedencentral. This covers one of the nine cases we want to test.

The second run block is our actual test. It makes sure that appropriate tags are propagated to the storage account from the resource group. Specifically we require that two tags are set, source and module.

Now we turn to the solution to the issue of how to test all permutations of versions and locations.

Solution

There are a number of options for how to solve the permutation of tests you need to run in this scenario. In this post I present a simple solution using a strategy in GitHub Actions.

Before we look at the GitHub Actions workflow, let's look at a modified version of our test file, this one stored in templates/main.tftest.hcl.tpl:

// templates/main.tftest.hcl.tpl
run "setup" {
  variables {
    location    = "{{LOCATION}}"
    name_suffix = "tftest-{{LOCATION}}-{{VERSION}}"
  }

  module {
    source  = "app.terraform.io/your-tf-org/resource-group-module/azurerm"
    version = "{{VERSION}}"
  }
}

run "proper_tags_should_be_propagated" {
  variables {
    resource_group_name = run.setup.resource_group.name
  }

  command = apply

  assert {
    condition     = alltrue([
      contains(keys(azurerm_storage_account.this.tags), "source"),
      contains(keys(azurerm_storage_account.this.tags), "module")
    ])
    error_message = "Proper tags are not propagated to the storage account"
  }
}

The only difference to the test file shown before is that the explicit version is replaced by a placeholder value {{VERSION}}, and each explicit location is replaced by {{LOCATION}}.

The GitHub Actions workflow is located in the file .github/workflows/tftest.yaml in our repo. We start building our workflow like this:

on: workflow_dispatch

permissions:
  id-token: write
  contents: read

To start with we only want to trigger the workflow manually, that is why we have on: workflow_dispatch as the trigger. We add a few permissions that will be required for the Azure login action where we use a federated identity in Azure (see the documentation for details on how to set this up).

Next we start defining our job:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        version: ["1.1.0", "1.2.0", "1.3.0"]
        location: ["swedencentral", "westeurope", "northeurope"]

We have a single job named test. We will run the job on Ubuntu, using the latest version available. Next we define the strategy with the following configurations:

fail-fast: false, this is required to not have all tests cancelled if a single test fails. The default value for fail-fast is true.
matrix is used to configure a few settings that we want to vary between tests. In this case we vary version and location. There will be one run for each combination of version and location, for a total of nine runs.

The last part of the workflow consists of the steps we want to run for each test:

steps:
  - uses: azure/login@v1
    with:
      client-id: ${{ secrets.AZURE_CLIENT_ID }}
      tenant-id: ${{ secrets.AZURE_TENANT_ID }}
      subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
  - uses: actions/checkout@v4
  - run: |
      sed 's/{{VERSION}}/${{ matrix.version }}/g; s/{{LOCATION}}/${{ matrix.location }}/g' \
        templates/main.tftest.hcl.tpl > tests/main.tftest.hcl
  - uses: hashicorp/setup-terraform@v2
    with:
      terraform_wrapper: false
  - run: terraform init
    env:
      TF_TOKEN_app_terraform_io: ${{ secrets.TF_TOKEN }}
  - run: terraform test

The first step uses the azure/login@v1 action to sign in to Azure. This is required because we will be creating resources in Azure using Terraform (remember: the Terraform test framework runs actual plan and apply operations, creating actual resources!)

The second step of the workflow uses the actions/checkout@v4 action to check out the source code. If you come from an Azure DevOps background you might be surprised that you need to explicitly add this step. I prefer that GitHub Actions requires that you add this step if you intend to do something with the source code in the repository, I find the behind-the-scenes checkout in Azure DevOps to be confusing.

The third step requires some explanation:

- run: |
    sed 's/{{VERSION}}/${{ matrix.version }}/g; s/{{LOCATION}}/${{ matrix.location }}/g' \
      templates/main.tftest.hcl.tpl > tests/main.tftest.hcl

Here we use sed to:

Find each occurrence of {{VERSION}} in the file templates/main.tftest.hcl.tpl and replace it by ${{ matrix.version }} which in turn is replaced by GitHub Actions with a value from version in the matrix we defined above.
Find each occurrence of {{LOCATION}} in the file templates/main.tftest.hcl.tpl and replace it by ${{ matrix.location }} which in turn is replaced by GitHub Actions with a value from location in the matrix we defined above.

The result of these search-and-replace operations are stored in tests/main.tftest.hcl.

Here we opted for a simple sed search-and-replace, which is fine when the number of variables we need to replace are few. If we needed to replace many more variables than two we would probably look into using a templating tool for this instead. Use common sense here, if sed works for your purposes then there is no need to involve any other tool.

The last few steps sets up the Terraform CLI tool, initializes Terraform, and finally runs the tests:

- uses: hashicorp/setup-terraform@v2
  with:
    terraform_wrapper: false
- run: terraform init
  env:
    TF_TOKEN_app_terraform_io: ${{ secrets.TF_TOKEN }}
- run: terraform test

The terraform init step requires an environment variable TF_TOKEN_app_terraform_io with the value of a Terraform Cloud token. This is because the module we are using is located in a private Terraform registry (see the documentation for details on this).

The full workflow for reference:

on: workflow_dispatch

permissions:
  id-token: write
  contents: read

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        version: ["1.1.0", "1.2.0", "1.3.0"]
        location: ["swedencentral", "westeurope", "northeurope"]
    runs-on: ubuntu-latest
    steps:
      - name: Azure login
        uses: azure/login@v1
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
      - uses: actions/checkout@v4
      - run: |
          sed 's/{{VERSION}}/${{ matrix.version }}/g; s/{{LOCATION}}/${{ matrix.location }}/g' \
            templates/main.tftest.hcl.tpl > tests/main.tftest.hcl
      - uses: hashicorp/setup-terraform@v2
        with:
          terraform_wrapper: false
      - run: terraform init
        env:
          TF_TOKEN_app_terraform_io: ${{ secrets.TF_TOKEN }}
      - run: terraform test

When we trigger this single workflow we can see that nine individual jobs are started:

Jumping into Azure after a short while we can confirm that nine different resource groups have been created:

Finally, after a few minutes all jobs are finished¹:

The results are clearly indicating that a regression have been introduced in version 1.3.0 of the resource group module that we depend on. We can conclude that our module is not ready to be release as a new version, and it is time to talk to the team responsible for the resource group module to see what changes they have made.

When you use a matrix with many different variables (I use two) the number of billable minutes in GitHub Actions can quickly escalate. In the simple example I used 43 billable minutes. ↩

Testing Framework in Terraform 1.6: A deep-dive

Mattias Fjellström — Mon, 13 Nov 2023 21:11:43 +0000

In my previous blog post A Comprehensive Guide to Testing in Terraform: Keep your tests, validations, checks, and policies in order I went through all the options for testing and validation that are available to you when you write your Terraform configurations and modules. We saw check blocks, pre-conditions and post-conditions related to a resource's lifecycle, custom conditions for input variables and output values, and more. The latest and greatest topic I covered was the testing framework that arrived in Terraform 1.6.

In this post I want to focus more on the testing framework to uncover the possibilities that it brings.

Background on testing with Terraform

If this is the first time you hear about the new testing framework for Terraform I would like to give you a short introduction to what it is.

Terraform 1.6 brought a new testing framework to general availability after it had been available as an experimental feature for a period of time. The release notes for version 1.6 listed a single new feature¹:

terraform test: The terraform test command is now generally available. This comes with a significant change to how tests are written and executed, based on feedback from the experimental phase.

Terraform tests are written in .tftest.hcl files, containing a series of run blocks. Each run block executes a Terraform plan and optional apply against the Terraform configuration under test and can check conditions against the resulting plan and state.

What we learn from the release notes is that we can now write tests for our Terraform configurations by including one or more .tftest.hcl files that each contain a series of run blocks. We also learn that a run block runs terraform plan or terraform apply commands. This means that these tests could create real infrastructure from your configuration. For me this is a good thing. I am a strong believer in when it comes to testing infrastructure-as-code there is no way to be sure it will work unless you actually deploy it for real. Why is that? There are just too many things that could go wrong. There might be hidden dependencies that you have no idea of before you actually try to create new infrastructure.

What is not clear from the release notes is who this testing framework is intended for. Is it meant for all Terraform users? Should you immediately jump on the TDD-with-Terraform train and start writing tests for all Terraform configuration? This is not the case. At least this is not the primary intended case. The testing framework is designed for module producers.

Are you in charge of creating infrastructure modules internally for your organization or publicly for the Terraform community? Then you are a module producer and this testing framework is for you.

Are you consuming modules in order to create the infrastructure for your application? Then this testing framework is not primarily intended for you. However, there is nothing stopping you from using it if you think it makes sense for your situation.

Module producers write code that other users will consume. Users of your modules depend on the contract that your module exposes. What is part of the module contract? Generally this includes the following items:

The input variables your module expects.
The output values your module produces.
Any externally visible resources your module creates. This could include configuration files, application gateways in Azure, network loadbalancers in AWS, a Cloud Bigtable table in GCP, or it could be pretty much anything else.

That last point might make you wonder if there are any resources created in a module that is not part of the contract? There definitely could be. Some resources could be internal implementation details that are required in order to construct the rest of the infrastructure. If a resource could be swapped out with a different resource without module consumers noticing then it is an internal implementation detail and not part of the contact.

If you are a module producer and you make an update of your module where you unintentionally make a significant change to the contract your module exposes, then this mistake could end up causing trouble for your module consumers.

This is exactly the reasoning behind any other kind of test in software development, testing Terraform modules is no different!

One last point to make about the new testing framework is that you write your tests in HashiCorp Configuration Language (HCL). This means there is no need to learn a new language in order to write tests for your Terraform code. There is no need to install an additional test tool that you need to keep track of and update and scan for vulnerabilities and so on. There is no need to mix your Terraform configuration with a bunch of test-related framework files. Run your tests and deploy your infrastructure using one and the same Terraform binary.

With all that background out of the way, let's move on to seeing all the nitty-gritty details of what this testing framework can do.

Nomenclature

Sometimes I need to remind myself of the nomenclature of the Terraform HCL code. To make sure we are all on the same page I introduce the nomenclature I use here:

A block are containers of other content. A block have a block type and zero or more labels. The generic block looks like this:
```
<BLOCK TYPE> "<BLOCK LABEL 1>" "<BLOCK LABEL 2>" ... "<BLOCK LABEL N>" {
  # block content
}
```
In Terraform the number of labels is zero, one, or two. A block can contain other blocks. Some common block types are terraform, provider, resource, data, and run.
An expression represents a literal value such as a string or a number, or they could be referencing other values such as the name of a resource. An expression could also be more complex consisting of function calls, string interpolations, and references. Some examples of expressions:
```
"this is a string value"
12345
azurerm_storage_account.my_account.name
"rg-${var.name}-${var.location}"
```
An argument is the assignment of a value (from an expression) to a name. Arguments appear inside of blocks. Some examples of arguments:
```
resource_group_name = "rg-my-resource-group"
type = list(string)
count = length(var.number_of_regions)
```

Even if the details of the HCL were familiar from before the nomenclature might be unfamiliar.

Test framework patterns

The run block is what executes a given test when we run terraform test. This block is central to the testing framework, so this is a block you need to become familiar with.

In the following subsections I will go through three testing patterns² that you might see in Terraform.

Before we look at the patterns let's briefly look at a typical directory structure for Terraform tests.

First of all, remember that your test files should use the .tftest.hcl file ending. If not, the terraform test command will not execute the run blocks for your tests.

When you execute terraform test you should be located in the module root directory. This is the directory where your primary main.tf file exists. The Terraform binary will look for your test files in the root directory or in a directory named tests. I recommend you place your tests in the tests directory, and do not place test files in the module root directory.

A typical directory structure for a simple module with tests is this:
$ tree .
.
├── main.tf
├── outputs.tf
├── providers.tf
├── tests
│   └── main.tftest.hcl
└── variables.tf
If you place your test files somewhere else you need to add -test-directory=path/to/tests to the terraform test command. But once again I recommend that you keep your test files in the tests directory to avoid confusing future contributors to your module.

How many test files should you have? The simple answer is it depends! If you are building a large module with many moving parts you will probably need to have several test files divided up into coherent and related parts that test a certain part of your module. If your module have a small interface (variables and outputs) it might suffice with a single test file. Use common sense here, if it feels like a file has too many tests then it probable does.

Pattern 1: Assertions

The first pattern is simple, it consists of a run block with a nested assert block:

run "file_contents_should_be_valid_json" {
  command = plan

  assert {
    condition     = try(jsondecode(local_file.config.content), null) != null
    error_message = "The file is not valid JSON"
  }
}

Let's break down this run block:

The run block has one label. This label is the name of the test. You should give your test a self-explanatory name that describes what it does. If the test fails you should immediately know why it failed. In this example the name is file_contents_should_be_valid_json. If this test fails I know that the contents of a file was not valid JSON.
This run block executes a terraform plan command. You specify what command you would like the test to execute in the command = <command> argument. If you leave this out it will default to execute an apply command. Personally I think it is a good idea to be clear and always add the command = <command> argument to be explicit in what the test does.
The run block can contain zero or more nested assert blocks. Each assert block has a condition = <expression> argument where <expression> should evaluate to true or false to indicate if the test passes (true) or fails (false). If <expression> evaluates to false then the expression in error_message = <expression> will be displayed to the terminal (or in Terraform Cloud). In this case the error message is The file is not valid JSON.

Although this example showed a single run block containing a single assert block, remember that you could include multiple run blocks, each containing multiple assert blocks.

Pattern 2: Expecting failures

The second pattern concerns tests where we expect the test to fail, and we want the test to report success if it does. This is a common testing strategy. The following run block has a nested variable block and an expect_failures = [ ... ] argument:

run "bad_input_url_should_stop_deployment" {
  command = plan

  variables {
    config_url = "http://example.com"
  }

  expect_failures = [
    var.config_url
  ]
}

There are some new things to look at in this run block so let's break it down:

The variables block allows you to provide input values for any variables that your module expects. In this case the variables block is provided as a nested block to the run block, but it could also be provided as a standalone block outside of any run blocks. In that case the values would apply to all run blocks in the entire file. If you include a standalone variables block you can still include nested variables block inside of a run block to override the global values.
The expect_failures = [ ... ] specifies that we expect this test to fail, and we list the reasons for failure in the array expression of the argument. In this particular example I say that I expect this test to fail due to the variable named config_url. This basically mean that I validate the value provided for the config_url variable in my Terraform module, and the value provided in this test (http://example.com) should result in a failing validation. If the plan can proceed as normal without any failures, then this test would fail.

It is worth spending some time discussing expect_failures. The values in this array must be checkable objects with associated custom conditions. In my previous article I wrote a lot about custom conditions. Objects that can include custom conditions are variables, outputs, resources, data sources, and check blocks.

An important point about these custom conditions is that all of them except for the check block will actually cause Terraform to halt the execution of a plan or apply operation. What does this mean for your tests? It means that if you want to combine expect_failures with assert blocks you have to be careful in how you construct your module and your corresponding tests. If you include a variable in the expect_failures array of values and at the same time have an assert block that expects a plan to finish, then the assert block would never even be evaluated because the custom condition for the variable would halt the execution of the plan.

For this reason I suggest you keep your tests to either use one or more assert blocks, or use the expect_failures = [ ... ] argument, but not both unless you really know what you are doing.

Note that the array value to expect_failures could contain multiple values. But you most likely would not want to mix the type of checkable objects you include in this array because of the reason discussed above.

Pattern 3: Using helper modules

Sometimes it is necessary to create supporting infrastructure before you can test your module. This could be the case if your module creates resources in Azure and it expects that you use an existing resource group for all the resources. In order to test a module like that there must be an existing resource group you can use. A simple solution to this is to create a resource group up front and just let it sit there in your cloud environment for as long as required. A better solution is to create the resource group when you launch the terraform test command.

To illustrate what this looks like we have the following directory structure:

$ tree .
.
├── main.tf
├── outputs.tf
├── testing
│   └── setup
│       └── main.tf
├── tests
│   └── main.tftest.hcl
└── variables.tf

4 directories, 5 files

I have created a testing directory that contains a setup directory with a main.tf file. The contents of this file is:

// testing/setup/main.tf
variable "resource_group_name" {
  type = string
}

variable "location" {
  type = string
}

resource "azurerm_resource_group" "rg" {
  name     = var.resource_group_name
  location = var.location
}

It is a simple file that uses the azurerm provider to create a resource group. The module under test is:

// main.tf
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">= 3.0.0"
    }
  }
}

locals {
  resource_name_suffix = "${var.name_suffix}-${var.location}"
}

resource "azurerm_service_plan" "plan" {
  name                = "plan-${local.resource_name_suffix}"
  resource_group_name = var.resource_group_name
  location            = var.location
  os_type             = "Linux"
  sku_name            = var.appservice_plan_sku
}

resource "azurerm_linux_web_app" "app" {
  name                = "app-${local.resource_name_suffix}"
  service_plan_id     = azurerm_service_plan.plan.id
  resource_group_name = var.resource_group_name
  location            = var.location

  https_only = true

  site_config {
    always_on           = contains(["Free", "F1", "D1"], var.appservice_plan_sku) ? false : true
    http2_enabled       = true
    minimum_tls_version = "1.2"
  }
}

This module creates two resources. An App Service plan and a Linux Web App. The variables.tf file has the following content:

// variables.tf
variable "name_suffix" {
  type = string
}

variable "resource_group_name" {
  type = string
}

variable "location" {
  type = string
}

variable "appservice_plan_sku" {
  type = string

  validation {
    condition = contains([
      "B1", "B2", "B3", "D1", "F1", "S1", "S2", "S3"
    ], var.appservice_plan_sku)
    error_message = "Please provide a valid App Service Plan SKU"
  }
}

How do we create the resource group module before we run our tests? The test file main.tftest.hcl looks like this:

// tests/main.tftest.hcl
provider "azurerm" {
  features {}
}

variables {
  resource_group_name = "rg-app-service-test"
  location            = "swedencentral"
  appservice_plan_sku = "F1"
  name_suffix         = "apptest"
}

run "setup" {
  module {
    source = "./testing/setup"
  }
}

run "http_should_not_be_allowed" {
  command = plan

  assert {
    condition = azurerm_linux_web_app.app.https_only == true
    error_message = "Web App accepts HTTP traffic"
  }
}

run "confirm_always_on" {
  command = plan

  variables {
    name_suffix = "testingalwayson"
    appservice_plan_sku = "S1"
  }

  assert {
    condition = azurerm_linux_web_app.app.site_config[0].always_on == true
    error_message = "Always-On is off for S1 SKU"
  }
}

There are few new things to look at in this test file. Let's break it down.

First of all we configure the azurerm provider:

provider "azurerm" {
  features {}
}

This allows you to configure the provider in any way that fits your tests. In this case I use default settings (an empty features block is required). Note that you could also configure the provider to use a separate test subscription instead of any other default subscription you have configured.

The next piece in the test file defines global variables:

variables {
  resource_group_name = "rg-app-service-test"
  location            = "swedencentral"
  appservice_plan_sku = "F1"
  name_suffix         = "apptest"
}

These variables will be used for the setup module and all the following tests, unless the tests override these values.

Next we have the setup module:

run "setup" {
  module {
    source = "./testing/setup"
  }
}

A setup (or helper) module is created in its own run block. I set the label of this block to setup, but you can set it to whatever fits your context. The run block only contains a nested module block that specifies the source of the module to be my module located in the testing/setup directory. This run block is the first run block in the test file, so it will be run first (they are run in order). If I place the setup run block somewhere else in the file then the tests defined above the setup block would fail because the resource group would not exist.

The rest of the file contains two tests in two separate run blocks. The first block is similar to what we have seen before, but notice that we have a nested variables block in the other run block:

run "confirm_always_on" {
  command = plan

  variables {
    name_suffix = "testingalwayson"
    appservice_plan_sku = "S1"
  }

  assert {
    condition = azurerm_linux_web_app.app.site_config[0].always_on == true
    error_message = "Always-On is off for S1 SKU"
  }
}

This means that for this test we override the name_suffix and appservice_plan_sku variables.

I can run the tests with terraform test:

$ terraform test
tests/main.tftest.hcl... in progress
  run "setup"... pass
  run "http_should_not_be_allowed"... pass
  run "confirm_always_on"... pass
tests/main.tftest.hcl... tearing down
tests/main.tftest.hcl... pass

Success! 3 passed, 0 failed.

Notice that the output says 3 passed even though we only really had two tests. How come? This is because the setup module runs inside a run block, so it is considered to be a test from Terraform's point of view. I think this is a bit unfortunate, but for now we'll have to live with it.

My tests in this case used command = plan, so they are relatively fast to run. When you use command = apply you have to prepare for a potentially long test run, depending on what resources your module creates. Imagine if you have a module that creates multiple Kubernetes clusters and installs various components in these clusters, then an apply could take some time. Especially if you run multiple independent tests.

Terraform test state file and resource destruction

How does Terraform know what resources it should remove and in what order it should do it? If you are familiar with Terraform you know that there is usually a state file somewhere. When you run tests Terraform keeps the state files in memory, so you won't see any state files appearing in your module directory.

Terraform creates one state file for the main configuration under test, and one state file for each alternate module that you create through a run block. An example of an alternate module is the setup module we saw in an example above.

The state files are created in the order of the tests, and they are destroyed in the reverse order. An illustrative sample of what state files are created, updated, and destroyed and in what order, is shown below:

// first alternate module call
// creates state file for modules/setup-1.tf
run "setup-1" {
  module {
    source = "modules/setup-1.tf"
  }
}

// second alternate module call
// creates state file for modules/setup-2.tf
run "setup-2" {
  module {
    source = "modules/setup-2.tf"
  }
}

// test the main configuration
// creates the main statefile for the module's main.tf
run "test-1" {
  assert {
    ...
  }
}

// third alternate module call, once again to the setup-2 module
// updates the state file for modules/setup-2.tf
run "setup-2-again" {
  module {
    source = "modules/setup-2.tf"
  }
}

// second test for the main configuration
// updates the main statefile for the module's main.tf
run "test-2" {
  assert {
    ...
  }
}

// After everything is run clean-up starts:
// 1. The module's main.tf state file is destroyed
// 2. The alternate modules state files are destroyed in
//    reverse order from how they were created
//      - first the modules/setup-2 state file is destroyed
//      - then the modules/setup-1 state file is destroyed

A question that came to my mind when I first heard about the test framework was what happens if the test fails the destruction of resources? Let's see what happens!

I will run a test where an Azure App Service is created. I will use the setup module from before where I created a resource group. To make the test fail I will issue the following Azure CLI command in order to lock the resource group so that Terraform can't destroy it:

$ az lock create \
  --name failure \
  --resource-group rg-app-service-test \
  --lock-type ReadOnly

The test output is the following:

$ terraform test
tests/main.tftest.hcl... in progress
  run "setup"... pass
  .
  . (output truncated)
  .
Terraform left the following resources in state after executing
tests/main.tftest.hcl/http_should_not_be_allowed, and they need to
be cleaned up manually:
  - azurerm_linux_web_app.app
  - azurerm_service_plan.plan

There we go!

We are instructed that a number of resources need to be cleaned up manually. In Azure this is usually relatively simple if you have put all resources in the same resource group. However, if you are working with AWS you might be in for a tedious cleanup process if your module created a lot of resources!

I can see that this behavior could be an issue during development of your module and tests where you are not sure if everything works as intended. You will most likely end up with a few failing test cleanups.

Exploring the test command in the CLI

To cover everything we can about the test framework let's see what else we can do with the terraform test command:

$ terraform test -h
Usage: terraform [global options] test [options]

[ background description truncated for brevity ... ]

Options:

  -cloud-run=source     If specified, Terraform will execute this test run
                        remotely using Terraform Cloud. You must specify the
                        source of a module registered in a private module
                        registry as the argument to this flag. This allows
                        Terraform to associate the cloud run with the correct
                        Terraform Cloud module and organization.

  -filter=testfile      If specified, Terraform will only execute the test files
                        specified by this flag. You can use this option multiple
                        times to execute more than one test file.

  -json                 If specified, machine readable output will be printed in
                        JSON format

  -no-color             If specified, output won't contain any color.

  -test-directory=path  Set the Terraform test directory, defaults to "tests".

  -var 'foo=bar'        Set a value for one of the input variables in the root
                        module of the configuration. Use this option more than
                        once to set more than one variable.

  -var-file=filename    Load variable values from the given file, in addition
                        to the default files terraform.tfvars and *.auto.tfvars.
                        Use this option more than once to include more than one
                        variables file.

  -verbose              Print the plan or state for each test run block as it
                        executes.

There are a few interesting flags we can use. I want to highlight a few:

-cloud-run=source is useful if you have your module in a private registry in Terraform Cloud, and you want to trigger a test run in the cloud. I will cover testing in Terraform Cloud in a future post.
-filter is useful if you have a lot of test files and you would only want one or a few of the files to run. This is especially useful if you are testing a large module where your tests execute apply operations that take a long time.
-test-directory is useful if you place your test files somewhere else than in the tests directory. But as I mentioned earlier in this article you should probably stick to using the tests directory.

Summary

In this post we have looked at most of what the new testing framework for Terraform 1.6 has to offer. That is in fact not true. There are more we can say about the testing framework when it comes to Terraform Cloud. In a future post I will cover how we run tests in Terraform Cloud and some of the unique features that are available there.

The example patterns in this post have intentionally been left relatively simple. In reality creating good tests for your modules will require a lot of work. My purpose of this post has been to illustrate what we can do, what syntax is available, and a few of the behaviors we can expect from this framework.

I expect that there will be additional features added as HashiCorp receives feedback from users of this framework. We live in exciting times!

Apart from this single new feature there were enhancements and bug fixes included as well. ↩
I call them patterns here to use a familiar nomenclature. As with all patterns you will most likely not see them isolated in the real world. All patterns I present are most likely mixed and matched for real Terraform modules. The idea with patterns here is to introduce the testing framework piece by piece. ↩

The issue of recursive module calls in declarative infrastructure-as-code

Mattias Fjellström — Thu, 09 Nov 2023 14:15:39 +0000

A few years ago when I was working exclusively with the AWS platform I was early to jump on the Cloud Development Kit (CDK) train. I had been using AWS CloudFormation and HashiCorp Terraform for a few years for all my infrastructure-as-code needs up until then. However, I never got comfortable using the CDK and abandoned it long before it reached 1.0.

Why was the CDK making me uncomfortable? To me it just did not provide any benefits over the traditional declarative approach. In the end the CDK code looked like slightly more complex declarative code. It could be mapped one-to-one with an equivalent declarative template. There were a few things that were better than the declarative approach though, but keep in mind that I was comparing CDK primarily to CloudFormation at that time. Perhaps the best benefit with the CDK was that you could do any kind of string and array manipulation your chosen programming language offered. This was (is) a desperately needed feature in CloudFormation.

Fast-forward a few years and enter Azure Bicep! Bicep provides many things that CloudFormation does not have (ignoring platform-specific things in this comparison of course). There is not much feature-wise I am missing from Azure Bicep. We can discuss what I do miss in Bicep in another post.

However, I recently tried to achieve something involving recursive module calls in Bicep and quickly realized that it is not possible. The Bicep language server even warns you in your editor if you are trying to create a recursive loop of module calls. I turned to Terraform to see if HashiCorp has introduced support for recursive module calls. Initially the HashiCorp Developer AI actually told me that it should indeed be possible, so I was hopeful. It turns out that you can try to make recursive module calls, there is no immediate warning in your editor. However, once you run a terraform init you realize that Terraform is trying to dig an infinitely deep hole of module reference in module reference and it eventually errors out.

So what is it I am trying to do with recursive module calls?

A use-case for recursive module calls

The use-case I am trying to solve is that I want to define a structure of Azure management groups and subscriptions in a simple YAML file, or something similar. I picked YAML at first, but JSON would work too as well as defining the structure in the chosen declarative language (Bicep, Terraform).

For this post let's concentrate only on management groups. My idea was to define the structure like this:

id: mg-root
name: Tenant Root Group
children:
  - id: mg-contoso
    name: Contoso
    children:
      - id: mg-platform
        name: Platform
        children:
          - id: mg-identity
            name: Identity
          - id: mg-management
            name: Management
          - id: mg-connectivity
            name: Connectivity
      - id: mg-landing-zones
        name: Landing Zones
        children:
          - id: mg-sap
            name: SAP
          - id: mg-corp
            name: Corp
          - id: mg-online
            name: Online
      - id: mg-decommissioned
        name: Decommissioned
      - id: mg-sandbox
        name: Sandbox

The sample structure is fetched from the Azure Landing Zone documentation.

I would then like to read this file in Bicep (or Terraform) and through clever recursive module calls create this structure of management groups.

A proposed solution with Azure Bicep

To solve this with Bicep my approach was to have the following main.bicep file:

targetScope = 'tenant'

var data = loadYamlContent('data.yaml')

resource root 'Microsoft.Management/managementGroups@2023-04-01' existing = {
  name: data.id
}

module recursive 'modules/recursive.bicep' = [for (child, index) in data.children: {
  name: 'child-module-${index}'
  params: {
    name: child.name
    children: child.children
    id: child.id
    parentId: root.id
  }
}]

The recursive module file modules/recursive.bicep looks like this:

targetScope = 'tenant'

param parentId string
param id string
param name string
param children array = []

resource mg 'Microsoft.Management/managementGroups@2023-04-01' = {
  name: id
  properties: {
    details: {
      parent: {
        id: parentId
      }
    }
    displayName: name
  }
}

module childMgs 'recursive.bicep' = [for (child, index) in children: {
  name: 'child-module-${id}-${index}'
  params: {
    name: child.name
    children: child.children
    id: child.id
    parentId: mg.id
  }
}]

I thought it was a good idea, but Bicep did not agree. I have submitted a proposal to the Bicep team for how this can be allowed. Vote for this issue if you agree!

A proposed solution with HashiCorp Terraform

To solve this with Terraform my approach was to have the following main.tf file:

terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
    }
  }
}

provider "azurerm" {
  features {}
}

locals {
  data = jsondecode(file("data.json"))
}

module "children" {
  source   = "./modules/recursive"
  children = local.data.children
  name     = local.data.id
  parent   = local.data.parent
}

Terraform has no built-in support to read YAML so I converted the file to JSON and read it using jsondecode(...).

The recursive module file modules/recursive/main.tf looks like this:

terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
    }
  }
}

variable "name" {
  type = string
}

variable "parent" {
  type = string
}

variable "children" {
  type = list(object({
    id       = string
    name     = string
    children = list(any)
  }))
}

resource "azurerm_management_group" "this" {
  name                       = var.name
  parent_management_group_id = var.parent
}

module "recursive" {
  for_each = toset(var.children)
  source   = "./"
  name     = each.value.name
  parent   = azurerm_management_group.this.id
  children = each.value.children
}

To be honest I am not sure this would have worked even if recursive module calls were allowed, but at least my editor is not complaining at this point. When I run terraform init however:

$ terraform init
Initializing the backend...
Initializing modules...
╷
│ Error: Failed to remove local module cache
│
│ Terraform tried to remove
│ .terraform/modules/children.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive
│ in order to reinstall this module, but encountered an error: unlinkat
│ .terraform/modules/children.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive.recursive:
│ file name too long

Terraform does not know I make recursive module calls, but it does its best to find the end of the recursive calls but eventually ends up complaining about the length of a filename.

Solving the problem using the Cloud Development Kit for Terraform

In the introduction I spoke about the CDK. CDK is specifically for AWS infrastructure. However, a few years ago a new tool called Cloud Development Kit for Terraform (CDKTF) arrived. CDKTF follows the same structure as the CDK. I recommend that you read through the CDKTF documentation if you are interested to learn more, because I will not explain the details of CDKTF in this post.

I wrote my CDKTF code using TypeScript, but there are other alternatives available.

I defined the management group structure in TypeScript in managementGroups.ts:

export type ManagementGroupDefinition = {
    name: string
    parent?: string
    children?: ManagementGroupDefinition[]
  }

export const managementGroups: ManagementGroupDefinition = {
    name: "Pseudo Root Group",
    parent: "/providers/Microsoft.Management/managementGroups/<my tenant id>",
    children: [
        {
            name: "Contoso",
            children: [
                {
                    name: "Platform",
                    children: [
                        {
                            name: "Identity"
                        },
                        {
                            name: "Management"
                        },
                        {
                            name: "Connectivity"
                        }
                    ]
                },
                {
                    name: "Landing Zones",
                    children: [
                        {
                            name: "SAP"
                        },
                        {
                            name: "Corp"
                        },
                        {
                            name: "Online"
                        }
                    ]
                },
                {
                    name: "Decommissioned"
                },
                {
                    name: "Sandbox"
                }
            ]
        }
    ]
}

You could define the structure in YAML as before, but for simplicity I defined it as TypeScript. One thing to note is that I have not included the id field in this structure. This is because Terraform does not allow me to define a custom id for my management groups. This is unfortunately not ideal, but I'll let it slide for now. Another thing to note is that I have included a parent field in the root management group. The parent is my actual tenant root group, but I decided to create a pseudo root group instead of working directly in the actual root group.

Next I have my CDKTF application in main.ts:

import { Construct } from "constructs";
import { App, TerraformStack } from "cdktf";
import { AzurermProvider } from "@cdktf/provider-azurerm/lib/provider"
import { ManagementGroup } from "@cdktf/provider-azurerm/lib/management-group"
import { managementGroups, ManagementGroupDefinition } from "./managementGroups"

class Stack extends TerraformStack {
  constructor(scope: Construct, id: string) {
    super(scope, id);

    new AzurermProvider(this, "azurerm", {
      features: {}
    })

    this.makeLayer(managementGroups)
  }

  makeLayer(config: ManagementGroupDefinition) {
    const parent = new ManagementGroup(this, config.name, {
      displayName: config.name,
      parentManagementGroupId: config.parent ?? undefined
    })

    config.children?.forEach( (child) => {
      this.makeLayer({ ...child, parent: parent.id })
    })
  }
}

const app = new App();
new Stack(app, "cdktf");
app.synth();

The magic happens in the makeLayer method of my Stack class. This is where I create a new ManagementGroup with a given displayName and an optional parentManagementGroupId. Next I loop over each child to this management group and once again call makeLayer, and here we have the recursion!

CDKTF constructs a valid Terraform template from this main.ts file. No need for infinite recursive module calls because CDKTF knows the recursive loop has an end.

To be honest, the resulting Terraform configuration does not actually use modules. So the result is not a "solution" to the recursion problem. What CDKTF does here is generate a single configuration with all my management groups defined, it does not introduce a module and make recursive calls to it.

Summary

Currently neither Azure Bicep or HashiCorp Terraform supports recursive module calls out of the box. This is one use-case where an imperative approach to infrastructure-as-code wins. I am still not convinced the imperative approach is worth it in the long run, I prefer the clarity of the declarative approach.

Of course there is the middle ground, you could use an imperative approach to generate a declarative template which you can then deploy. And this is what happens under the hood with CDKTF anyway.

As with everything else it depends on what you want to do. So far in my career this was the first time I tried to do something in a declarative way that was just not supported.

Oh and by the way, I know one could argue for that I am trying to write imperative code using a declarative language when I do recursive module calls - but that is another discussion.

HashiCorp Developer AI

Mattias Fjellström — Wed, 25 Oct 2023 04:15:46 +0000

During the keynote at HashiConf 2023 Armon Dadgar, HashiCorp's Co-Founder and CTO, humorously quipped:

Nowadays there wouldn't be a keynote unless you mention AI at least once

To be fair, I don't remember the quote verbatim so take it with a grain of salt. However, the message he conveyed is captured in my rendition of the quote.

There were two AI-related announcements made during the keynote. The first was about automatic test generation for a module inside of Terraform Cloud. This was a part of the new test framework announcement, one that I covered in my previous post. The other announcement, which I will focus on in this post, was the Developer AI. The Developer AI is part of developer.hashicorp.com.

Developer AI introduces itself as:

Your AI-powered companion for finding reference materials, architectural guidance, and product examples from HashiCorp. Ask me anything to get started.

To get started using Developer AI you need to sign-up for the private beta[^beta]. Once you are in the private beta you can head on over to developer.hashicorp.com or specifically to your favorite product documentation. I am a huge fan of Terraform, so to try out Developer AI I pointed my browser at developer.hashicorp.com/terraform.

Developer AI hides in the search field in the upper right corner:

To open the search interface I can click on the search box, or press Command + K on my keyboard (since I am using a Mac, but you would press ctrl + K on a different kind of computer). The search interface contains a lot more than the Developer AI, but that is currently of no interest to me so instead I click on the Ask Developer button:

Finally I am ready to interact with Developer AI:

You are informed in the UI that Developer AI have the following features:

Personalized recommendations
Natural language conversations
Tap into HashiCorp knowledge base

The point about personalized recommendations will probably not work out-of-the-box since this is the first time I am using Developer AI. Perhaps if I use it a lot it will learn my personal preferences. As with any AI these days there is also a disclaimer in the UI:

Content generated by Developer AI may contain errors, inconsistencies, or outdated information. It is provided as-is without any warranties or guarantees of accuracy. Use of this feature and related content is governed by HashiCorp’s AI Policy.

As with other AIs this one might hallucinate, but unless you have lived under a rock the past year you are most likely aware of this. Always keep this in mind when communicating with an AI!

What do I want to ask Developer AI? Let's see if it can suggest any good way of parsing data stored in a CSV file for further use in Terraform! This is a problem I happened to encounter once[^csv]. I enter my question in the text box and press send:

Developer AI immediately generates the following reply for me:

To parse data stored in CSV files for use in Terraform, you can use the csvdecode function provided by Terraform. This function allows you to read and parse CSV files directly within your Terraform configuration.

Here's an example of how you can use the csvdecode function:

First, you need to read the CSV file using the file function and assign it to a variable:
locals {
  csv_data = file("${path.module}/data.csv")
}
Next, you can use the csvdecode function to parse the CSV data:
locals {
  parsed_data = csvdecode(local.csv_data)
}
Now, you can access the parsed data in your Terraform configuration. For example, if your CSV file has columns named "name" and "age", you can access the values like this:
resource "aws_instance" "example" {
  count = length(local.parsed_data)

  instance_type = local.parsed_data[count.index].name
  ami           = local.parsed_data[count.index].age
}
In this example, we assume that the CSV file has a header row with column names "name" and "age". The csvdecode function will parse the CSV data and create a list of objects, where each object represents a row in the CSV file. You can then access the values of each column using dot notation.

Remember to adjust the file path in the file function to match the location of your CSV file.

By using the csvdecode function, you can easily parse CSV data and use it in your Terraform configuration. This can be useful for dynamically generating resources or configuring variables based on data stored in CSV files.

For more information, you can refer to the Terraform documentation on the csvdecode function: Terraform csvdecode function

This might sound fabricated to increase the legitimacy of Developer AI, but I genuinely did not know the csvdecode function existed! I would call this response a great success! I also feel a little dumb because that function might have made my life easier back when I was in the situation where I needed it[^csv].

Since I am a responsible private beta participant I provide feedback to the response by clicking the thumbs-up button.

The next time I am performing complicated manipulations in Terraform, or doing anything that I don't know by heart, I will take Developer AI for another spin. This has been my first interaction and I am pleased with the results!

[^csv]: If you know, you know.

A Comprehensive Guide to Testing in Terraform: Keep your tests, validations, checks, and policies in order

Mattias Fjellström — Fri, 20 Oct 2023 14:53:04 +0000

This post discusses testing and validation for infrastructure-as-code (IaC) with HashiCorp Terraform. The insights and ideas presented here can surely be extended to IaC in general.

Introduction

With the release of the new testing framework for Terraform 1.6 presented at HashiConf 2023 in San Francisco¹, there is now the possibility to test your Terraform configurations using Terraform itself.

Testing IaC has always been a pain point. If you have been using IaC for any amount of time you may know that the only true test is to run terraform apply and see what happens. Test frameworks for IaC come and go, some are more successful than others. My belief is that there is no mock that will ever give you enough confidence that your IaC will behave as intended². For this reason I am glad the new test framework in Terraform does exactly that, it runs plan and apply commands for you, and it creates actual infrastructure.

But wait, there is more! The new testing framework is declarative, meaning that you write tests using the same language that you use for the rest of your infrastructure in Terraform. There is no need to install a third-party tool to keep track of, or to learn a new language.

The new testing framework is not your only option when it comes to testing and validating your infrastructure with Terraform. Terraform 1.5 introduced the concept of checks together with the new check block. Checks allow you to validate your infrastructure outside of the usual resource lifecycle. If you are using Terraform Cloud you can even set up continuous validations to regularly verify custom assertions. This post will not focus on Terraform Cloud, instead I will focus on what you can do with Terraform locally³.

Going even further back in time, Terraform introduced custom conditions for validating input variables, output values, and running validation checks at given points in a resource lifecycle. This is another way to test parts of your infrastructure directly in Terraform. Custom validation of input variables allow you to check that the provided input value fulfils given constraints. These constrains could be to limit the length of a supplied string to less than ten characters, or to restrict the allowed values of a variable to a predefined list of values. Resources and data sources have the lifecycle block where you can add precondition and postcondition blocks to validate things before, and after, running an apply, respectively. Similarly, you can add precondition blocks to your output blocks, to validate something about the output value.

With all the above mentioned options for testing and validation you can create robust and well-tested Terraform configurations and modules.

There is a related concept to testing that you should also consider using. I am talking about policies, and policy-as-code. It does not make sense to write tests for all the properties of every resource in your modules, because most properties have no designated wrong values that you need to test for. However, there are insecure or incompliant values for properties that your organization might not allow. Besides, maybe you allow insecure property values in your development environments but not in your production environments. Instead of having to write separate test files or custom validations depending on what environment you are testing, you should instead use policies.

In this blog post I will introduce each of the concepts discussed above through examples in Terraform. I will also discuss the important question of when you should use a given approach: a test, a check, a custom condition, or a policy.

All the source code from this post is available at my GitHub repo: github.com/mattias-fjellstrom/terraform-testing.

Using Terraform for testing and validation

This section will introduce each of the concepts discussed above, and in the following section I will go through two example Terraform configuration and include tests, checks, custom conditions, and policies, where appropriate.

CLI validation command

Let's start at the beginning.

When you design your test and validation strategy you can choose how deep into the rabbit hole you want to go.

If you are constrained on time but would like to include basic validation of your Terraform configuration, you can use the terraform validate command. This command captures trivial mistakes in your configuration, like typos in resource properties and erroneous syntax in your HashiCorp Configuration Language (HCL).

During development of your Terraform configuration it is a sensible step to run terraform validate often⁴. You might think that trivial mistakes like typos would be caught by your editor, but you would be surprised by how many mistakes can slip by. HashiCorp recently announced better support for editors like VS Code, so things are getting better in this area.

A good rule of thumb is that your deployment pipeline should never fail on the terraform validate command. You should catch these errors during development. However, your pipeline should definitely include the terraform validate command, because we all do make mistakes and forget things!

Note that terraform init also captures some basic mistakes in your configuration. This include things like using an invalid version number for a given provider.

Custom conditions

There are three kinds of custom conditions to discuss. The first custom condition is for variable validation.

We want to catch errors as soon as possible. If our modules include variable blocks, we would be naive to assume the user will only provide sensible values as input to these variables. Erroneous input values will most likely result in errors later in the plan or apply phases. The worst possible type of error is when a bad input value results in a failed apply 45 minutes into the operation.

How do we mitigate this?

The solution is to add validation blocks nested inside of our variable blocks:

variable "name_prefix" {
    type = string

    validation {
        condition     = length(var.name_prefix) <= 20
        error_message = "Variable length should be 20 characters or less"
    }
}

A validation block has no label, and it takes two arguments:

A condition which is a boolean expression that evaluates to true or false to indicate if the validation succeeded or failed.
An error_message that is presented to the user if the validation fails.

Now, let us move on to the second kind of custom condition.

We can include validations that run before or after changes are applied to a resource, or after a data source is read. We do this by including an appropriate precondition or postcondition block nested inside of a lifecycle block in our resource or data source.

An example of what this looks like:

// read an existing Azure resource group
data "azurerm_resource_group" "rg" {
  name = "rg-storage-resource-group"
}

// create a new Azure storage account in the resource group
resource "azurerm_storage_account" "stg" {
  name                     = "stsamplestorage"
  resource_group_name      = data.azurerm_resource_group.rg.name
  location                 = data.azurerm_resource_group.rg.location
  access_tier              = "Hot"
  account_replication_type = "LRS"
  account_tier             = "Standard"

  lifecycle {
    precondition {
      condition     = data.azurerm_resource_group.rg.managed_by == "terraform"
      error_message = "Resource group must be managed by Terraform!"
    }
  }
}

Similar to the validation block for variables, both the precondition and postcondition blocks takes two arguments: a condition and an error_message.

In the previous code snippet I declare a data source where I read an existing resource group from Azure (azurerm_resource_group). I then would like to create a new storage account in the resource group, but only if the precondition validation of the resource lifecycle passes.

In the precondition block I check that the managed_by property of the resource group is set to the value terraform. If the managed_by property has any other value, the condition will evaluate to false and the precondition fails. The operation will halt and the error_message is displayed to the user in the console.

The third, and last, custom condition to discuss is for output validation. We can add a precondition block nested inside of our output blocks. This allows us to verify something that the output depends on. This could be used to avoid outputting values from the output block into our state file and in our logs when the value would not be useful. Personally I think this custom condition has less obvious usecases, and I have not used it for anything myself.

The precondition block inside of the output block looks the same as for resources, it has one condition argument and one error_message argument.

An example of what a custom condition of outputs look like:

output "website_endpoint" {
  value = azurerm_storage_account.stg.primary_web_endpoint

  precondition {
    condition     = startswith(azurerm_storage_account.stg.primary_web_endpoint, "https://")
    error_message = "Website URL uses HTTP, you should instead configure HTTPS"
  }
}

This custom condition checks that a website deployed to an Azure storage account exposes a website URL starting with https://, and if this is not the case an error message is raised and the deployment fails.

Test framework

The Terraform test framework is new since Terraform 1.6. It was release a few days before HashiConf 2023 in San Francisco, and I believe it is a welcome change to Terraform.

Tests are written using HCL, so you do not need to learn a new language in order to test your IaC. Test files use the .tftest.hcl file ending.

By default, all test files stored in the same directory as your module, or stored in a tests directory, are run when you issue the new terraform test command. If you place your tests in a different directory you can add the -test-directory=path/to/my/test/directory flag to the command.

With all the necessary details out of the way, a typical setup of a simple module with tests could look like this:

$ tree .
.
├── main.tf
├── variables.tf
├── outputs.tf
└── tests
    ├── testfile1.tftest.hcl
    ├── testfile2.tftest.hcl
    └── testfile3.tftest.hcl

A test file consists of one or more run blocks. Each run block should be thought of as a test. A run block includes a number of arguments and blocks to configure what the test should do. Each test executes a command, which is either plan or apply. If you do not specify the command argument it defaults to apply.

An example of what a run block looks like:

// tests/main.tftest.hcl
run "file_contents_should_be_valid_json" {
  command = plan

  assert {
    condition     = try(jsondecode(local_file.config.content), null) != null
    error_message = "The file is not valid JSON"
  }
}

The run block label should use a descriptive name to explain what the test is about. In this case I have set the label to file_contents_should_be_valid_json to indicate that this tests validates that the content of a file that is produced by Terraform contains valid JSON content. The test runs a plan as indicated by the command argument. The meat of the test is in the assert statement. The condition argument evaluates to true or false, which results in the test passing or failing, respectively.

Note that you can add additional assert blocks to make several assertions in the same test. It is also possible to not use any assert block, but instead use the expect_failures argument where you can indicate if you expect the plan or apply to fail, and if so what will make it fail. Further down in this post I will show examples of what this looks like.

Checks

The concept of checks together with the check block was introduced in Terraform 1.5 and serve a purpose similar to custom validations for resources and data sources, but with one important difference. The check block is ideal for validations you want to make that should not stop a plan and apply from finishing. If a check fails the deployment will still continue. You will get a warning in the output indicating that the check failed.

A check block can include at most one scoped data block, and one or more assert blocks. Each assert block has a condition that evaluates to true or false, and an error_message that is displayed if the assertion fails. The purpose of the scoped data block is to read data about a resource, or perhaps an external website, that you want to include in the check.

Imagine that our Terraform configuration sets up a publicly available website (among other things), and we would like to check that the website responds to requests after it is set up. This is an ideal situation for a check block. An example of what this might look like:

// main.tf
module "website" {
  source = "./modules/web"

  // ... the rest of the arguments are left out for brevity
}

check "validate_website_response" {
  data "http" "static_website" {
    url = module.website.web_endpoint
  }

  assert {
    condition     = data.http.static_website.status_code == 200
    error_message = "The website returned an unhealthy status code"
  }
}

The main differences between check blocks and custom conditions for resources and data sources is that the latter is tied to the lifecycle (hence they are part of the lifecycle block) of the resource or data source, while check blocks are not. Also, check blocks do not fail an apply if the assert evaluates to false, while custom conditions for resources and data sources do.

This makes check blocks useful for continuous validation in Terraform Cloud. Continuous validation specifically runs the check blocks in your configuration on a schedule. This could be useful for basic health checks for your infrastructure.

Policies

Policies are closely related to tests, but they serve a different purpose.

There are two popular frameworks for policy-as-code that works well with Terraform, and they are both available in Terraform Cloud.

The first framework is the Open Policy Agent (OPA) together with its Rego language for policies, it has risen in popularity in recent years and is used for many types of policies. The second framework is HashiCorp Sentinel, which uses HCL. In this post I will use Sentinel, since it is most closely related to HashiCorp and Terraform and in my opinion it is also a little bit easier to learn.

The purpose of policies with Terraform is to validate that the changes you are introducing follow rules defined by you or your organization. These rules are most likely related to compliance or security.

An example of a Sentinel policy document is this:

import "tfplan/v2" as tfplan

// filter out all storage containers
containers = filter tfplan.resource_changes as _, c {
    c.type is "azurerm_storage_container"
}

// filter out containers that are created or updated
createdOrUpdated = filter containers as _, c {
    c.change.actions contains "create" or c.change.actions is ["update"]
}

// make sure each container has an access type set to private
main = rule {
    all createdOrUpdated as _, containers {
        containers.change.after.container_access_type is "private"
    }
}

The first time you see a Sentinel policy you might get puzzled at what it does. This post is not about explaining the syntax of Sentinel, the comments in the code snippet should be explanation enough. In essence, Sentinel takes the JSON-representation of a Terraform plan file from a terraform plan command, and searches for something specific in the plan, and finally evaluates the rule named main that is either true or false. If the rule evaluates to true the change is allowed by the policy.

In the policy example above I first apply a filter to find all the resources of type azurerm_storage_container, then I have another filter to find all the containers where the planned actions are create or update. Finally, I have my rule named main that checks that the container_access_type property is set to private for the containers that I have filtered out. If one or more containers has set container_access_type to something other than private, then this policy evaluation will fail, and I should not allow the change to be deployed.

The trick to learn how to write policies is to make sure you have a plan file in JSON format available, then you know what to look for and what all the fields are named. It also helps to start with an existing example and modify it to your needs.

You should run all your policies against each planned change to make sure you are not introducing anything that violates your compliance and security rules.

Applying testing and validation in the real world

In this section I want to go through two small sample Terraform configurations, and use the different options from above to test and validate the code. In the first sample I take on the role as a module developer, crafting a module for other developers to use. In the second sample I imagine I am part of a standalone team building infrastructure for our own applications.

Sample 1: Module producer

In this sample we are producers of a simple module that other teams in our organization will use in their own infrastructure. Many teams depend on our module, so we must make sure we test it thoroughly. The sample configuration might be a bit contrived, but bear with me!

The module we have produced takes three inputs, it creates a local file containing some JSON data, and it outputs the file content in base64 encoded format. This configuration file is part of a larger infrastructure where the file will be used to configure a separate system (not included in this sample). The system has constraints about the configuration file that must be respected. These constraints might be that the JSON file should be valid, it should include certain fields, and it must be at most a given size.

Our module has the following directory structure:

$ tree .
.
├── main.tf
├── outputs.tf
├── tests
│   └── main.tftest.hcl
└── variables.tf

1 directories, 4 files

The first file we will look at is variables.tf:

// variables.tf
variable "config_url" {
  type = string

  validation {
    condition     = startswith(var.config_url, "https://")
    error_message = "config_url should start with https://"
  }
}

variable "port" {
  type = number

  validation {
    condition     = var.port > 1023 && var.port <= 65535
    error_message = "Use an unprivileged port number (1024-65535)"
  }
}

variable "service_name" {
  type = string
}

There are three variables defined: config_url, port, and service_name.

Exactly what these variables mean in a larger context is out of scope but we imagine they are important! A brief description of what we do know:

config_url should be a valid URL. Currently the condition in the validation block makes sure that the URL starts with https://. We could change this condition to compare the provided value with a regex describing a valid URL if we want to. For simplicity I have left the validation as it is.
port should be a valid port number. The condition in the validation block checks that the user has provided un unprivileged port in the range 1024-65535.
service_name is the name of a service of some kind. Currently there is no validation of this variable, this means we have no known restrictions of the value the user can supply. This is a common situation in the real world, where we might not always know what to allow or not.

Next we take a look at main.tf:

// main.tf
terraform {
  required_providers {
    local = {
      source  = "hashicorp/local"
      version = "2.4.0"
    }
  }
}

resource "local_file" "config" {
  filename = "config.json"
  content  = <<EOF
{
    "endpoint": "${var.config_url}",
    "port": "${var.port}",
    "service": "${var.service_name}"
}
EOF
}

This simple Terraform configuration specifies one required provider, the local provider. It creates one resource of type local_file. This resource is the configuration file that we are creating. The content argument of the file is constructed as JSON, using the three supplied variables.

Next we look at outputs.tf:

// outputs.tf
output "file_content_base64" {
  value = base64encode(local_file.config.content)

  precondition {
    condition     = length(base64encode(local_file.config.content)) < 150
    error_message = "Config file is too large."
  }
}

A single output block is defined. The value argument is the base64-encoded value of the file content. There is a specific restriction on the length of this base64-encoded value, it can be at most 149 characters long. Don't ask me why, only the system knows. We use a precondition block to verify that the size restriction is fulfilled.

Could we have validated our variables in some way to avoid the precondition block for the output value? The problem is that the size of the JSON file depends on all three variables, and currently the service_name variable has no restrictions on its own. The config_url variable could also be almost any length. In this case it makes sense to check the size of the generated JSON as a precondition in the output instead.

When it comes to testing this module we should explicitly test the interface we are exposing. What is included in the interface of this module? It is the following:

The three variables config_url, port, and service_name.
The generated JSON-file config.json.
The output file_content_base64.

These are the things that a user of our module will see and interact with.

Below I will discuss one test at a time, all the tests I show are part of the file named tests/main.tftest.hcl. The first test I write is for the config_url variable:

// tests/main.tftest.hcl
variables {
  config_url   = "https://example.com"
  port         = 1414
  service_name = "my-service"
}

run "bad_input_url_should_fail" {
  command = plan

  variables {
    config_url = "http://example.com"
  }

  expect_failures = [
    var.config_url
  ]
}

The variables block defined in the root of the file provides default values for all the variables. These can be overridden in each test if desired. For this test I override the config_url variable inside of the run block with a URL starting with http://, which I expect should not be allowed. For this particular test I am not using an assert block to assert something about the test outcome, instead I use the expect_failures argument to tell this test that I expect it to fail, and I expect that the variable named config_url will be the reason for the failure.

The next test is a similar test but for the variable named port:

// ... continuation of tests/main.tftest.hcl

run "bad_input_port_should_fail" {
  command = plan

  variables {
    port = 80
  }

  expect_failures = [
    var.port
  ]
}

This test overrides the port variable with a value of 80. This is a privileged port, so I expect it will not be allowed. Once again I use the expect_failures argument to tell the test I expect it to fail and that the variable named port will be the reason for the failure. Note that to be thorough I should add a test that makes sure I can't provide a port number larger than 65535 but I will skip it for brevity.

As discussed above there is no wrong input value for the last variable named service_name. So there is no good test we can write for it. However, in the next test we will use the service_name variable in order to make the size of the JSON content too big. This is a test for our output validation.

// ... continuation of tests/main.tftest.hcl

run "too_large_output_file_should_fail_deployment" {
  command = plan

  variables {
    service_name = "this-service-name-creates-a-too-big-file"
  }

  expect_failures = [
    output.file_content_base64
  ]
}

This test overrides the service_name variable with a long name, and for the third time in a row I use the expect_failures argument to tell the test I expect it to fail on the output named file_content_base64.

The last test I have added for this module concerns the validity of the content of the JSON file. When you construct a JSON file inline of your Terraform configuration like this it is easy to make mistakes. A trailing comma or a missing colon would create a broken JSON file. A good test for this is:

// ... continuation of tests/main.tftest.hcl

run "file_contents_should_be_valid_json" {
  command = plan

  assert {
    condition     = try(jsondecode(local_file.config.content), null) != null
    error_message = "The file is not valid JSON"
  }
}

This test uses the assert block. The condition argument uses the try(...) function. This function will select the first valid value from its list of arguments. The first argument in this function is jsondecode(local_file.config.content), which encodes the string content of the file as JSON. If this encoding fails, the try(...) function will instead select its second argument: null. The condition argument checks that the result of try(...) is not equal to null. If it is, the assertion fails and so does the test.

To run the tests we issue the new terraform test command:

$ terraform test
tests/main.tftest.hcl... in progress
  run "bad_input_url_should_fail"... pass
  run "bad_input_port_should_fail"... pass
  run "too_large_output_file_should_fail_deployment"... pass
  run "file_contents_should_be_valid_json"... pass
tests/main.tftest.hcl... tearing down
tests/main.tftest.hcl... pass

Success! 4 passed, 0 failed.

To see what a failing test looks like, let's change the port number in the test named bad_input_port_should_fail to a valid number:

// ... temporary change in tests/main.tftest.hcl to trigger a failing test

run "bad_input_port_should_fail" {
  command = plan

  variables {
    port = 8080
  }

  expect_failures = [
    var.port
  ]
}

Rerunning the tests produces the following output:

$ terraform test
tests/main.tftest.hcl... in progress
  run "bad_input_url_should_fail"... pass
  run "bad_input_port_should_fail"... fail
╷
│ Error: Missing expected failure
│
│   on tests/main.tftest.hcl line 27, in run "bad_input_port_should_fail":
│   27:     var.port
│
│ The checkable object, var.port, was expected to report an error but did not.
╵
  run "too_large_output_file_should_fail_deployment"... skip
  run "file_contents_should_be_valid_json"... skip
tests/main.tftest.hcl... tearing down
tests/main.tftest.hcl... fail

Failure! 1 passed, 1 failed, 2 skipped.

There we have it. Four simple tests to validate that the interface of our module behaves as we intend it to do. During continued development of this module we would add additional tests when it makes sense.

This module included tests and validations, but no policies. Does it make sense to talk about policies during module development? Not really. As a module developer you are not sure in what context your module will be used, so it is unlikely the policies you add would make any sense to the user of your module.

Sample 2: Module consumer

In this sample we put ourselves in the shoes of a module consumer. In reality we might be both a module consumer and module producer at the same time, but in this case we imagine we are writing a Terraform configuration for our own application, and we are using a module provided for us by a platform team. To have a complete example to show I will include the web module as a local module, but note that this module could be hosted in an internal Terraform registry for our organization.

The file structure of this example configuration looks like the following:

$ tree .
.
├── index.html
├── main.tf
├── modules
│   └── web
│       ├── main.tf
│       ├── outputs.tf
│       └── variables.tf
└── policies
    ├── policy.sentinel
    └── sentinel.hcl

4 directories, 9 files

I will not go into the details of the web module itself, the source code is available at the GitHub repo for this post. Our main concern is the main.tf file:

// main.tf
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.76.0"
    }
  }
}

provider "azurerm" {
  features {}
}

module "website" {
  source = "./modules/web"
  providers = {
    azurerm = azurerm
  }

  location           = "swedencentral"
  name_prefix        = "web531"
  index_file_content = file("index.html")
}

check "validate_website_response" {
  data "http" "static_website" {
    url = module.website.web_endpoint
  }

  assert {
    condition     = data.http.static_website.status_code == 200
    error_message = "${data.http.static_website.url} returned an unhealthy status code"
  }

  assert {
    condition     = data.http.static_website.response_headers["Content-Type"] == "text/html"
    error_message = "${data.http.static_website.url} returned a wrong content-type header"
  }

  assert {
    condition     = strcontains(data.http.static_website.response_body, "Hello World")
    error_message = "${data.http.static_website.url} did not respond with expected content"
  }
}

This configuration lists azurerm as a required provider. This provider is configured according to our teams specification, and then provided to the module. The module block takes three variables: location, name_prefix, and index_file_content. The website content is read from a local file index.html.

Apart from the module there is one large check block with the label validate_website_response. Our team has included this check block in order to verify a few basic things about the website we are creating. In the check block there is one nested data block that uses the http provider in order to send a GET request to our websites endpoint. The endpoint is read from the module output named web_endpoint. Following the data source there are three assert blocks:

The first assert block checks that the website responds with a 200 OK response code. If it does not it indicates that the website is not functioning correctly.
The second assert block checks that the content-type of the returned page is text/html. If it is not then the website will not be displayed correctly in a browser.
The third assert block checks that the returned content contains the string Hello World. If it foes not then the right content has not been uploaded.

Appropriate error_message arguments are included for each assert block.

Notice that our directory structure does not include any tests. This is because tests are primarily for module producers, and in this case we are consuming a module. We hope that the producers of the module we are using have included tests though!

Also notice that our team have decided to use a check block instead of precondition or postcondition blocks. This is because our team would like to have the possibility to set up continuous checks via Terraform Cloud. The team also decided that if these simple checks fail we don't want to halt the terraform apply command, because in this case the website is not business critical. Remember that precondition and postcondition validations will halt the apply if they fail, but check blocks will only report a warning.

Looking back at the directory structure for this example we notice that we have included a policies directory. Currently there is a single policy document in this directory:

// policy.sentinel
import "tfplan/v2" as tfplan

// filter out storage accounts that are created or updated
storageAccounts = filter tfplan.resource_changes as _, resourceChange {
    resourceChange.type is "azurerm_storage_account" and
        (resourceChange.change.actions contains "create" or
            resourceChange.change.actions contains "update")
}

main = rule {
    all storageAccounts as _, storageAccount {
        storageAccount.change.after.public_network_access_enabled is false
    }
}

This policy filters out all Azure storage accounts that are about to be created or updated, it then checks that the public_network_access_enabled is set to false. If it is not, the policy will fail. This policy should be run as part of Terraform Cloud or whatever deployment pipeline the team will be using.

To run the policy locally we have to perform the following steps:

$ terraform plan -out=tfplan
$ terraform show -json tfplan | jq > policies/plan.json
$ cd policies
$ sentinel apply policy.sentinel
Pass - policy.sentinel

Where to learn more

If you want to learn more about the topics presented in this post I suggest you head over to the official documentation. For your convenience I have gathered the relevant links:

Terraform checks
Terraform tests
Sentinel language and framework for policy-as-code
Custom conditions

Summary

We have seen that there are many options for testing and validating Terraform configuration, so that we can build robust modules and configurations that will stand the test of time.

The new testing framework is intended for module producers. Add tests to verify the behavior of your module interface: the variables, outputs, and any other externally visible behavior such as files.

For module consumers and general Terraform configurations there are plenty of validation options such as variable validation, output validation, lifecycle conditions in resources and data sources, as well as check blocks.

Let us summarize the different available options:

We run terraform init and terraform validate to check that the HCL we are writing follows proper syntax.
We use tests with run blocks to verify that our contract to our users, our module interfaces, work as intended.
We use variable validation blocks to verify that the input provided by users of our modules and configurations adhere to allowed values.
We use precondition and postcondition blocks in our resources and data sources to verify conditions before or after applying changes.
We use check blocks to verify external behavior outside of resource lifecycles, and could even set up continuous validations in Terraform Cloud.
We use policies with Sentinel or OPA to make sure the changes we are applying follow our organizations compliance and security requirements.

It will be interesting to see how the new test framework for Terraform evolves over time, and to see what tests people will be writing for their modules.

I hope this blog post has helped you starting to think about how to use tests and validations, as well as introducing policy-as-code, for your Terraform configurations!

Actually, it was announced a few days before the conference. Why wait when you have something good to announce? ↩
I challenge you all to tell me how a mock, or any other means of not running a real apply (or whatever term your IaC tool use) gives you confidence that your IaC will work as intended. ↩
But I will mention Terraform Cloud a few times! ↩
While you are at it, also run terraform fmt often. If you, like me, are using format-on-save in VS Code, then terraform fmt is superfluous. ↩

Vault Associate Certification (Part 10): Explain encryption as a service

Mattias Fjellström — Fri, 15 Sep 2023 16:13:55 +0000

This is the last part of this series, thank you for reading!

We've already seen the transit secrets engine in one of the previous posts in this series, but we will revisit the topic in this post.

The transit secrets engine handles encryption on data in-transit. No data handled by this secrets engine is persisted to storage in Vault. Apart from encrypting and decrypting data the transit engine can also sign and verify data, generate random bytes, and more.

There are several key types (or encryption algorithms) supported by the transit secrets engine: aes128-gcm96, aes256-gcm96, chacha20-poly1305, ed25519, ecdsa-p256, ecdsa-p384, ecdsa-p521, rsa-2048, rsa-3072, rsa-4096, and hmac.

Explaining encryption as a service make up the tenth and last objective in the Vault certification journey. This objective covers the following sub-objectives:

Configure transit secret engine
Encrypt and decrypt secrets
Rotate the encryption key

Configure transit secret engine

To start off we should enable the transit secrets engine on our Vault server. We've seen how to enable secrets engines before:

$ vault secrets enable transit

Success! Enabled the transit secrets engine at: transit/

The main use-case for the transit secrets engine is to encrypt and decrypt data, thus we will create a key to do just that. To create a key we run the following command:

$ vault write -f transit/keys/demo-key \
    type=rsa-4096 \
    auto_rotate_period=1h

Key                       Value
---                       -----
allow_plaintext_backup    false
auto_rotate_period        1h
deletion_allowed          false
derived                   false
exportable                false
imported_key              false
keys                      map[1:map[creation_time:2023-09-15T17:47:46.020802+02:00 name:rsa-4096 public_key:-----BEGIN PUBLIC KEY-----
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAqF4TxwdIpmVcXTri+Y1a
MGL7fM2xUeGDpDMfxnR1gJwjhda2zFud07TZ9RwmeFgxnbj9VwYRu9ThL1tAZ+32
72nZjizb2rJWp0UUYSWk5BcgUCa1NPM7gTcrgp9INXidHeTnC5TxUC4N2alUoPsG
r5opxHEVrBWQZxRx6AuOnP6YzmQSJMS4pv+0cH+dP0HUj1X2Kso3OuFNFSplsSPm
I9bZxOazmPNz9x4MYRfp3a5EOp6FmE/EKd2kOZ6EYaD0R+oNsI1lg79lpsDckIHY
ISFZd9sM5xVZTrXhBng2MMfpWwjSqoaJVeycQ91RYr3KzkmUW5UrXN3ksq/ncLBM
SXOcfhFgkLAXUbsEaMQE08ug4J+hEJj5Jh9By9DlNS2UMASySAwuUTJ1c78SE3fs
1YGaxZkt88mEuLOY0wUD0/0gEMshn/Lh3pjJvlYxMtwsdtVMx/zUsdaa6ialAFFl
6aKMCnvaQHtwS3O3oPma6c/qX714AKGmceZswxp8IhErafG0Wca7/JHTrUQNasWZ
HzDFqS2S5F1bu7oKICl+JELnpHu570MjtYXAomw3EHqNvPvw72ihdZ5tBX/QH1l1
iwIv+a76bfM3F5myoPP5uzNk6fPt73u0zLCKYfVK4fvr/V8CtCML6foUFtXGn9OQ
0vS2bGk64K2fTNfcSbC+P6kCAwEAAQ==
-----END PUBLIC KEY-----
]]
latest_version            1
min_available_version     0
min_decryption_version    1
min_encryption_version    0
name                      demo-key
supports_decryption       true
supports_derivation       false
supports_encryption       true
supports_signing          true
type                      rsa-4096

In the previous command I specified that the key type should be rsa-4096 and I want the key to automatically rotate every hour. To see all the available configuration options when creating a key see the API documentation.

Encrypt and decrypt secrets

In this section I continue where I ended the last section. The status is that we have a key named demo-key. We could read information about our key using vault read transit/keys/demo-key.

Now I want to encrypt some data using my new key. Remember from earlier posts that the data I want to encrypt must be base64-encoded:

$ vault write transit/encrypt/demo-key plaintext=$(echo "s3cr3tm3ss4g3" | base64)

Key            Value
---            -----
ciphertext     vault:v1:N+bgVOhRl5xrwcZ7Aovlg2CF9I+sB9eHt8Yspxw0CrGnR/llTp+9KPZjrA8ou2pznNZ20RdVI291TB2e3jKNLVR36pPQ8iEwfm4HNv+TVS+KJl0CvCweBGa84RrDeIUauwSgHx4vWV2SSz6qJSV8qBrwXKRSq6GPWS4bAu6U9+H2Yq6Zpvr3/mpSX1oZz4HL1PP/zds0taYNWaRJBUr9TDbgRQVto57Swy408+x1Fn9tq670go/ghYbXBV5idL+8O5IndKVB5xukwXNVvrXI4fO/lMBBkSfnGsqWUiACqCzYRPABEN2MHI6Zgfjdfi3CEr2XNCMf/dqEyK01WC9S0ERJM064dT+5oCUQzsll8e0dOGSkbFrKfTGWZExMBHrSHDrGREUEf1h/fT4gNf0udg9V5SUt+dniVkKWXQt6JqUhwIMzEglkJmsmrPDOQxXt9nziiO3mUvI1f2d99nYDicdFgCBULP9jQ7EpLM8V0NC1F/J+Rqk663ZHHZrHotfK9hDk54wfWzTfqKj+XYtqmr5SneRePOyCCtB1bOSkulgrPJQpkvrWowMCGz5eiaTrbGZjPORnkYdX8nEqv3kGIrDR6ZiP+HANzU9/BQBalr05A+UWY8XMa/4P9nPCH7cnah/vZxDicvkEF8ROxoslVfZcPPhQwxSCeRwRdX7kDK4=
key_version    1

To get my data back again I can do so using the following command:

$ vault write transit/decrypt/demo-key \
    ciphertext=vault:v1:N+bgVOhRl5xrwcZ7Aovlg2CF9I+sB9eHt8Yspxw0CrGnR/llTp+9KPZjrA8ou2pznNZ20RdVI291TB2e3jKNLVR36pPQ8iEwfm4HNv+TVS+KJl0CvCweBGa84RrDeIUauwSgHx4vWV2SSz6qJSV8qBrwXKRSq6GPWS4bAu6U9+H2Yq6Zpvr3/mpSX1oZz4HL1PP/zds0taYNWaRJBUr9TDbgRQVto57Swy408+x1Fn9tq670go/ghYbXBV5idL+8O5IndKVB5xukwXNVvrXI4fO/lMBBkSfnGsqWUiACqCzYRPABEN2MHI6Zgfjdfi3CEr2XNCMf/dqEyK01WC9S0ERJM064dT+5oCUQzsll8e0dOGSkbFrKfTGWZExMBHrSHDrGREUEf1h/fT4gNf0udg9V5SUt+dniVkKWXQt6JqUhwIMzEglkJmsmrPDOQxXt9nziiO3mUvI1f2d99nYDicdFgCBULP9jQ7EpLM8V0NC1F/J+Rqk663ZHHZrHotfK9hDk54wfWzTfqKj+XYtqmr5SneRePOyCCtB1bOSkulgrPJQpkvrWowMCGz5eiaTrbGZjPORnkYdX8nEqv3kGIrDR6ZiP+HANzU9/BQBalr05A+UWY8XMa/4P9nPCH7cnah/vZxDicvkEF8ROxoslVfZcPPhQwxSCeRwRdX7kDK4=

Key          Value
---          -----
plaintext    czNjcjN0bTNzczRnMwo=

I get the base64-encoded value back, which I can decode to my secret:

$ echo "czNjcjN0bTNzczRnMwo=" | base64 -D

s3cr3tm3ss4g3

Rotate the encryption key

When we encrypted data in the previous section the resulting ciphertext was prepended with vault:v1. This tells us that the data has been encrypted by the transit secrets engine in Vault and the key version uses was v1. What happens if we rotate our key and get a new version of it?

Let's do just that. I can rotate my key using the following command:

$ vault write -f transit/keys/demo-key/rotate

Key                       Value
---                       -----
allow_plaintext_backup    false
auto_rotate_period        1h
deletion_allowed          false
derived                   false
exportable                false
imported_key              false
keys                      map[1:map[creation_time:2023-09-15T17:47:46.020802+02:00 name:rsa-4096 public_key:-----BEGIN PUBLIC KEY-----
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAqF4TxwdIpmVcXTri+Y1a
MGL7fM2xUeGDpDMfxnR1gJwjhda2zFud07TZ9RwmeFgxnbj9VwYRu9ThL1tAZ+32
72nZjizb2rJWp0UUYSWk5BcgUCa1NPM7gTcrgp9INXidHeTnC5TxUC4N2alUoPsG
r5opxHEVrBWQZxRx6AuOnP6YzmQSJMS4pv+0cH+dP0HUj1X2Kso3OuFNFSplsSPm
I9bZxOazmPNz9x4MYRfp3a5EOp6FmE/EKd2kOZ6EYaD0R+oNsI1lg79lpsDckIHY
ISFZd9sM5xVZTrXhBng2MMfpWwjSqoaJVeycQ91RYr3KzkmUW5UrXN3ksq/ncLBM
SXOcfhFgkLAXUbsEaMQE08ug4J+hEJj5Jh9By9DlNS2UMASySAwuUTJ1c78SE3fs
1YGaxZkt88mEuLOY0wUD0/0gEMshn/Lh3pjJvlYxMtwsdtVMx/zUsdaa6ialAFFl
6aKMCnvaQHtwS3O3oPma6c/qX714AKGmceZswxp8IhErafG0Wca7/JHTrUQNasWZ
HzDFqS2S5F1bu7oKICl+JELnpHu570MjtYXAomw3EHqNvPvw72ihdZ5tBX/QH1l1
iwIv+a76bfM3F5myoPP5uzNk6fPt73u0zLCKYfVK4fvr/V8CtCML6foUFtXGn9OQ
0vS2bGk64K2fTNfcSbC+P6kCAwEAAQ==
-----END PUBLIC KEY-----
] 2:map[creation_time:2023-09-15T18:00:01.621354+02:00 name:rsa-4096 public_key:-----BEGIN PUBLIC KEY-----
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAymFiLHw/G5jZN+xtEBlb
ILhBMmn8F0CchTbKjr/cm8n5vus1OonamkuWY9Zk23al9Rf9mOYg2ZP453fJ8W0z
ZrrUJQxGsEV0T2xmjab5Xk7nUh2cDwhHGZu1t27z0up3F5XKezknokyB+EPAH33H
UrUB/AHMDGRcR0Gb0qNum0k6N2w9Y4iUO9Hq9UjnkpqoqAqY6CFScdl+QKu+kYj3
AlucROc0wR4XhhxAvl6CX25g/0kmonUjLZFylilS6qzqVQMcg7jgHqC8JwJ0S9Ds
RZr65IV4YLw5xBzZhooLEUT8E/8pIZxwJ3UmrR5EZkjkewU+D8YVJ5qYM7S785Cu
kKWB1QwGZ4Bmoue4n3Nw7y5YPGpv4KUrTB8/Kusl420LcgCRA3zENIV9zg+QAoPb
b7JJ4TV3P18uxfbNYn6v/N/GKjaCWYtCMGEKAjhzNZkvtmbGK3zZoE+HhvKb3Z7y
2D5ARe1Hu2e+z9r0PL+LH8ELQT+9kExwVBoMrxcNJq1fPtWSj2szGcq2YJMip91g
G7HQ2p28MVMkkFHSPUsDLPnBcpVs0lOb8mjwU+zeHGTTigH79R3XXsxZ0gwbSPxH
VeFcv3tFAPcQ+suyM2kpwrfFzvGopJSZuL/oUoqMHKdzrSbQy/c2t4W5pVUoHXga
2iEE0ALm/6V1W/3eqP2AIAMCAwEAAQ==
-----END PUBLIC KEY-----
]]
latest_version            2
min_available_version     0
min_decryption_version    1
min_encryption_version    0
name                      demo-key
supports_decryption       true
supports_derivation       false
supports_encryption       true
supports_signing          true
type                      rsa-4096

The metadata in the output says that latest_version is now 2. However, min_decryption_version is 1 which means that even if some ciphertext is encrypted using version one of our key it will still be possible for us to decrypt the value. You can configure what the min_decryption_version should be. Perhaps you don't want to allow any other key than version two to be able to decrypt values, then you can configure this.

Something else you might want to do is to re-encrypt your data with the latest version of the key. You can do this without exposing the secret data at any stage. To do this run the following command together with the ciphertext from before:

$ vault write transit/rewrap/demo-key \
    ciphertext=vault:v1:N+bgVOhRl5xrwcZ7Aovlg2CF9I+sB9eHt8Yspxw0CrGnR/llTp+9KPZjrA8ou2pznNZ20RdVI291TB2e3jKNLVR36pPQ8iEwfm4HNv+TVS+KJl0CvCweBGa84RrDeIUauwSgHx4vWV2SSz6qJSV8qBrwXKRSq6GPWS4bAu6U9+H2Yq6Zpvr3/mpSX1oZz4HL1PP/zds0taYNWaRJBUr9TDbgRQVto57Swy408+x1Fn9tq670go/ghYbXBV5idL+8O5IndKVB5xukwXNVvrXI4fO/lMBBkSfnGsqWUiACqCzYRPABEN2MHI6Zgfjdfi3CEr2XNCMf/dqEyK01WC9S0ERJM064dT+5oCUQzsll8e0dOGSkbFrKfTGWZExMBHrSHDrGREUEf1h/fT4gNf0udg9V5SUt+dniVkKWXQt6JqUhwIMzEglkJmsmrPDOQxXt9nziiO3mUvI1f2d99nYDicdFgCBULP9jQ7EpLM8V0NC1F/J+Rqk663ZHHZrHotfK9hDk54wfWzTfqKj+XYtqmr5SneRePOyCCtB1bOSkulgrPJQpkvrWowMCGz5eiaTrbGZjPORnkYdX8nEqv3kGIrDR6ZiP+HANzU9/BQBalr05A+UWY8XMa/4P9nPCH7cnah/vZxDicvkEF8ROxoslVfZcPPhQwxSCeRwRdX7kDK4=

Key            Value
---            -----
ciphertext     vault:v2:aEu3sdXwj3Wp1gcu3YEwqnQ4iINzH8ZBqzpUZFuLFJjxIGHEs0W9nbeRFdn3ertys6M0yR6D+gOrl+te2tqXuwVmvmbskT9shQYFPzjVfI4L6DpjNWpOvGqXb2A+GS86UJIxwNK/5Y/S+WizSFnQ7BXpMunKvan6lIP0SPSfqfCYzWuqcWfKkftTnzLiddWYxKk8AJHd2B3z+ahSM9pkAtjmxm7I6+SCwJ+VcJRR1UMpz/4/Z9QDgzU2xEMV1Nk3VMdUppJseavXzdl98zkB6X4ulOj9zMDMLwJHpLCVTjnrENhrui4sLWqZAhzTFI5joydpqzSC3mGwHaxrMyGqJ/CQS5WivU05euiQC+QyyziPPoff1ltmgt+OkQoKzjZfgG9Vy/B2wz71NqmFY/zpF+XNrkUVs9G7TlUFCZXwwL0mqzCitUPmP8UllnWVx+s9XoZ3dtdvwHtMPoSlUmjMeftZmxUJXglA6YZXvsI1vhDY7jpornJmlwz2a2L0HsK0zVFGwJE7DKAFBeR85U7XQZmXl+CYS0WE/Mm3VycwQWdxfUpAz7niYz3bfKbSIGJ5b/fsMvHvDQc4nPszdVMqEe5NV6RPFlR0APfl/ys64Izl+GLagitrMzc89PJzj8Y/Z6umz/29hkBb8uom6YTf3cgHLIQEC4HTCV7KZ7wTGzA=
key_version    2

The new ciphertext is prepended with vault:v2 which indicates that it is now encrypted with the latest version of the key. Let's verify that we can still decrypt the data:

$ vault write transit/decrypt/demo-key \
    ciphertext=vault:v2:aEu3sdXwj3Wp1gcu3YEwqnQ4iINzH8ZBqzpUZFuLFJjxIGHEs0W9nbeRFdn3ertys6M0yR6D+gOrl+te2tqXuwVmvmbskT9shQYFPzjVfI4L6DpjNWpOvGqXb2A+GS86UJIxwNK/5Y/S+WizSFnQ7BXpMunKvan6lIP0SPSfqfCYzWuqcWfKkftTnzLiddWYxKk8AJHd2B3z+ahSM9pkAtjmxm7I6+SCwJ+VcJRR1UMpz/4/Z9QDgzU2xEMV1Nk3VMdUppJseavXzdl98zkB6X4ulOj9zMDMLwJHpLCVTjnrENhrui4sLWqZAhzTFI5joydpqzSC3mGwHaxrMyGqJ/CQS5WivU05euiQC+QyyziPPoff1ltmgt+OkQoKzjZfgG9Vy/B2wz71NqmFY/zpF+XNrkUVs9G7TlUFCZXwwL0mqzCitUPmP8UllnWVx+s9XoZ3dtdvwHtMPoSlUmjMeftZmxUJXglA6YZXvsI1vhDY7jpornJmlwz2a2L0HsK0zVFGwJE7DKAFBeR85U7XQZmXl+CYS0WE/Mm3VycwQWdxfUpAz7niYz3bfKbSIGJ5b/fsMvHvDQc4nPszdVMqEe5NV6RPFlR0APfl/ys64Izl+GLagitrMzc89PJzj8Y/Z6umz/29hkBb8uom6YTf3cgHLIQEC4HTCV7KZ7wTGzA=

Key          Value
---          -----
plaintext    czNjcjN0bTNzczRnMwo=

And we must remember to decode the base64-encoded value:

$ echo "czNjcjN0bTNzczRnMwo=" | base64 -D

s3cr3tm3ss4g3

Vault Associate Certification (Part 9): Explain Vault architecture

Mattias Fjellström — Fri, 15 Sep 2023 13:09:16 +0000

In this course I have demonstrated everything using the Vault development server. The development server is excellent for learning and for quick prototyping. It is not intended for production use. Apart from using Vault to configure auth methods, store secrets, write policies, etc, there is a lot to do when it comes to the Vault architecture. How to configure Vault in a production scenario, as well as how to actually start up a server or cluster, is a big topic. In this post we will look at a few things related to the architecture of Vault and a bit about how Vault functions under the hood.

Explain Vault architecture makes up the ninth objective in the Vault certification journey. This objective covers the following sub-objectives:

Describe Shamir secret sharing and unsealing
Describe seal/unseal
Describe storage backends
Describe the encryption of data stored by Vault
Describe cluster strategy
Be aware of replication
Describe the Vault agent
Describe secrets caching
Be aware of identities and groups
Explain response wrapping
Explain the value of short-lived, dynamically generated secrets

I have rearranged the order of the sub-objectives to make this post a more coherent story. I have also combined a few sub-objectives into one because they were basically covering the same thing.

Before I start to go through the sub-objectives I would like to briefly introduce how you start a Vault server that is not the development server, because we will see this at a few places in this post.

You write one or more Vault server configuration files in HashiCorp Configuration Language (HCL). Then you can start the server using a modified vault server command:

$ vault server -config=my-server-config.hcl

You could provide multiple -config flags to load multiple configuration files:

$ vault server -config=config01.hcl -config=config02.hcl -config=config03.hcl

What goes inside a configuration file? This depends on how you want to configure your Vault server, and we will not go into depth on all the various options. A sample configuration file is shown below:

ui            = true
cluster_addr  = "https://127.0.0.1:8201"
api_addr      = "https://127.0.0.1:8200"

storage "consul" {
  address = "127.0.0.1:8500"
  path    = "vault/"
}

listener "tcp" {
  address       = "127.0.0.1:8200"
  tls_cert_file = "/path/to/full-chain.pem"
  tls_key_file  = "/path/to/private-key.pem"
}

As in all HCL there are arguments and blocks¹. The configuration starts with a few arguments specifying that we would like to enable the UI (ui = true), and we say that our cluster address is https://127.0.0.1:8201 and API address is https://127.0.0.1:8200. Following the arguments we have two common blocks seen in most (all?) configuration files:

The storage block specifies what storage backend (more on storage backends later in the post) we want to use. In this case we use the consul storage backend. The arguments in this block depends on what type of storage backend we select.
The listener block specifies how we can reach this specific Vault server. In this case we want to use TLS so we need to provide details on where the TLS certificate and key is located.

This was a very short introduction to starting up a Vault server for real, but we'll see some more examples further down in this post.

Describe Shamir secret sharing and unsealing and Describe seal/unseal

When you initialize a new production Vault server it starts up in a sealed state. It is not possible to perform any actions in the Vault cluster when it is in the sealed state, except for one important operation: unsealing the Vault server. When the server is sealed it knows where its physical storage is and how to access it, but it does not know how to decrypt the data in this storage.

The data in the storage is encrypted by a key, the data encryption key (DEK). This key is in turn encrypted by the root key, or master key. Unsealing Vault is the process of obtaining the plaintext root key in order to decrypt the DEK and in turn to decrypt the data stored on disk.

How does the unseal process work? By default, Vault uses the Shamir's Secret Sharing algorithm in order to split the root key into a number of unseal keys. In order to reconstruct the root key you need a certain number of these unseal keys. The idea here is that you would give each unseal key to different people, and no single person would be able to get ahold of the root key. Here I would like to mention that there is a process known as auto-unseal which involves using a trusted cloud provider key store to automate the unseal process. I will not cover auto-unseal in this post, but you can read more about it here.

To demonstrate how the unseal process works we'll start up a Vault server. In order to do this we need a Vault server configuration file:

// config.hcl

// use in-memory storage, not intended for production!
storage "inmem" {}

// http listener on 127.0.0.1:8200
listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_disable = "true"
}

// we can communicate with Vault on 127.0.0.1:8200
api_addr = "http://127.0.0.1:8200"

// let's enable the UI
ui = true

Using my configuration file I can start up my Vault server locally:

$ vault server -config=config.hcl

Now I have a running Vault server, but it is sealed and I do not have any token to authenticate to Vault with. I can check the status of my server using the following command:

$ vault status

Key                Value
---                -----
Seal Type          shamir
Initialized        false
Sealed             true
Total Shares       0
Threshold          0
Unseal Progress    0/0
Unseal Nonce       n/a
Version            1.14.0
Build Date         2023-06-19T11:40:23Z
Storage Type       inmem
HA Enabled         false

The output tells me that my server is not initialized (Initialized=false), and that it is sealed (Sealed=true). To initialize our Vault server we run the following command:

$ vault operator init

Unseal Key 1: eu0+iP+Fs71au7YwZN2lo/UpAJDEGdolCLMCiwl0WKCX
Unseal Key 2: 1wYpo/9jyzQSelnXzK4SkSfhpFQe2K+jNSnRGrqQUE1u
Unseal Key 3: 7NXC2Gvded4qyPukMqcYKJRcsj5OVA5KSmdQ45vqcSg8
Unseal Key 4: U25S/3rHC9m/GoRqOs5MoAkAv023W4kcFemzfmfMYGlq
Unseal Key 5: XCrlRf1z3eDBoudSRUwxDPAvyM5dGvwwe7mHy1TjLHal

Initial Root Token: hvs.c3esiYi0AJNSzdn1YxYsJtjx

The output contains five unseal keys and one initial root token. If we check the status of our server again we see the following:

$ vault status

Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    0/3
Unseal Nonce       n/a
Version            1.14.0
Build Date         2023-06-19T11:40:23Z
Storage Type       inmem
HA Enabled         false

The server is now initialized, but it is still sealed. We see that Total Shares is set to 5 and Threshold is set to 3. The Unseal Progress is 0/3. What this information tells us is that to unseal Vault we must use three of the five available unseal keys. To start the unseal process, copy one of the unseal keys and run the following command:

$ vault operator unseal

Unseal Key (will be hidden): *****

Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       11a275e7-0e20-262c-7c65-30d4859c8357
Version            1.14.0
Build Date         2023-06-19T11:40:23Z
Storage Type       inmem
HA Enabled         false

The Unseal Progress changed to 1/3. We can continue the process by issuing a new vault operator unseal command:

$ vault operator unseal

Unseal Key (will be hidden): *****

Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       11a275e7-0e20-262c-7c65-30d4859c8357
Version            1.14.0
Build Date         2023-06-19T11:40:23Z
Storage Type       inmem
HA Enabled         false

Almost there, the Unseal Progress is now 2/3. We can run the last unseal command:

$ vault operator unseal

Unseal Key (will be hidden): *****

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.14.0
Build Date      2023-06-19T11:40:23Z
Storage Type    inmem
Cluster Name    vault-cluster-206551a5
Cluster ID      d15ae1cf-bff4-c53d-8131-c037cea78999
HA Enabled      false

Now Sealed reports false, and our Vault server is ready to go. We can set our root token in an environment variable and see if we can interact with Vault:

$ export VAULT_TOKEN=hvs.c3esiYi0AJNSzdn1YxYsJtjx
$ vault secrets list

Path          Type         Accessor              Description
----          ----         --------              -----------
cubbyhole/    cubbyhole    cubbyhole_0bce5c18    per-token private secret storage
identity/     identity     identity_bb8c3d8d     identity store
sys/          system       system_4afad89f       system endpoints

To recap, what we did was to use three unseal keys to reconstruct the root key in order to decrypt the DEK and finally be able to read the data and configuration stored in Vault.

Describe storage backends

As you might have realized Vault needs to store its configuration and data somewhere. This is the purpose of a storage backend. HashiCorp officially supports two storage backends: integrated storage (Raft) and HashiCorp Consul. For the record it is possible to use other storage backends. In this post I will only mention integrated storage and Consul.

When I have been using the development cluster for demonstrations all the configuration and data have been stored in-memory. This is a bad idea because as soon as I shut my development server down all the configuration and data is gone. This is the reason why we need a storage backend: we want to be able to persist all the important things we do in Vault so that a simple shutdown of a server doesn't wipe our Vault history clean.

The storage backend is untrusted storage. This might sound like a strange statement, but it simply means that the storage backend is external to Vault proper and thus not trusted. It is used purely for storing encrypted data.

If you use integrated storage with Raft the data is persisted on some volume attached to the machine where Vault is running. This could be a virtual hard-drive attached to a virtual machine running in the cloud, or a persistent volume attached to a pod in a Kubernetes cluster. To configure your Vault server to use integrated storage you would add a block similar to the following to your cluster configuration file:

// config.hcl
// ... the rest of the config omitted ...

cluster_addr = "http://localhost:8201"

storage "raft" {
  path    = "/opt/vault/data"
  node_id = "my-first-node"
}

Here we have specified that the data should be stored at the path /opt/vault/data. Note that this path is on the OS your Vault server is running, it is not a Vault path. We must also provide the cluster address in cluster_addr. This is because Raft is created for a distributed system, i.e. for running Vault as a cluster of nodes. The cluster_addr is the address where the nodes in the cluster can communicate with each other.

It is not clear from my example configuration file above what kind of storage media will be used. It is recommended to use a storage media with high IOPS to handle everything Vault throws at it. You will probably have to run tests to measure the performance of your storage media to determine how much IOPS your environment demands.

HashiCorp Consul is a separate tool offered by HashiCorp. The setup is more complex than integrated storage because here you will need to run a separate Consul cluster together with your Vault cluster. I will not go into any details of Consul². In fact, in the Vault documentation you can find many recommendations to use integrated storage with Raft instead of using Consul. This is simply because the setup with Consul will be more complex, with more moving parts that can break and cause problems in your environments. Also note that if you go with Consul, it is recommended to use a dedicated Consul cluster specifically as the storage backend for Vault. So if you are already running Consul for another purpose do not reuse the same cluster.

Describe the encryption of data stored by Vault

All data stored by Vault is encrypted using the Data Encryption Key (DEK). Vault stores data in a storage backend. The storage backend is not part of the core Vault architecture, it lives outside of Vault. Vault encrypts the data before it leaves Vault proper and ends up at the storage backend. This is because the storage backend is not trusted, it should never see any unencrypted data.

The concept of Vault not sending out data unencrypted outside its core is known as the cryptographic barrier. Data that passes the cryptographic barrier is encrypted using the DEK. Note however that this does not apply for data that we request when we do e.g. vault kv get kv/database/password to read a secret - because as we have seen we get the plaintext data back.

The point of the cryptographic barrier is to protect the data at rest. It would be bad if someone would get ahold of the storage media where we store all our Vault data! Thus the data must be encrypted.

Describe cluster strategy

This subsection will briefly outline the recommended cluster architecture for Vault in production. As mentioned in the previous section HashiCorp recommends that you use integrated storage with Raft as your storage backend instead of Consul. We'll use Raft in this subsection.

Today it is common to run Vault in a public cloud environment, so the following discussion applies to that situation.

A typical cloud provider (e.g. AWS, Azure, GCP) has a presence in several locations across the globe. These locations are called regions. Each region usually consists of three or more zones. A zone is a data center consisting of one or more buildings housing the networking, CPU, memory, and storage, that make up a cloud provider. The zones are separated by a certain distance from each other, far enough to guarantee that a natural disaster or infrastructure disturbance won't take out two or more zones at the same time. The purpose of zones is to allow you to design highly available and redundant architectures.

HashiCorp recommends that you run five Vault nodes in a cluster, spread out over at least three zones in a region. An example of what this looks like in Azure is shown in the following figure:

In this architecture we have five nodes (virtual machines) divided among three subnets in a virtual network (the virtual network itself is not shown in the figure). Each subnet is in a different zone of the region³. In front of the cluster we have a separate subnet containing an application gateway load balancer (AGW) with a web application firewall (WAF). The AGW is zone-redundant, meaning it is present in each zone of the region. Each subnet has its own network security group (NSG) to control what traffic is allowed to flow in and out of the subnet. The Vault nodes are not directly reachable from the outside, all traffic must go through the AGW/WAF. This is a good base design that follows HashiCorp recommendations but in a real environment this would probably be a small piece of a much larger infrastructure.

Why is this a recommended architecture? It allows for the loss of a whole zone, e.g. due to a power outage or a natural disaster that renders the data center unreachable. If we lose a zone we lose up to two Vault nodes, leaving us with at least three remaining nodes. This is important because part of the internal machinery of Raft requires at least three nodes to be able to elect a new leader node if the current leader node is lost. This architecture also allows for up to two simultaneous node failures to occur. These kinds of failures do happen, so you must be prepared for them.

What happens if the whole region would fail? Is this even something that could happen? It can indeed happen. The most likely scenario would be that the cloud provider make a bad deployment that affects a whole region at once. It is unlikely that all the zones would be affected by random events that take them offline at the same time. How should you prepare for a regional failure? The solution involves having multiple clusters, spread out in different regions. The details are beyond the scope of what we need to learn, but it involves setting up clusters and two or more regions and configuring replication between these clusters.

Be aware of replication

At the end of the last section I mentioned the word replication. There are two purposes of replication that you should be are of:

Disaster recovery replication
Performance replication

No matter what type of replication we consider the main principles are the same. The unit of replication is a cluster. There is one leader, or primary, cluster and any number of follower, or secondary, clusters. Most data is replicated asynchronously from the primary cluster to the secondary clusters.

In disaster recovery replication most data, including configuration, policies, tokens, leases are shared between the primary and the secondary clusters. This type of replication is meant to support continuous operation in case of failure in the primary cluster. Traffic could quickly be shifted over to the secondary cluster and clients would only detect a minor glitch before things start working again. A disaster recovery replication cluster should not receive any traffic during normal primary cluster operation.

In performance replication the underlying configuration, policies, and some secrets are shared among primary and secondary clusters. What is not shared however is tokens and secret leases. The secondary clusters can serve client requests. Each cluster in this setup keeps track of their own tokens and leases. This could at times lead to the need to reauthenticate to Vault in case traffic is shifted from one cluster to another for some reason.

In summary, I recreate the table from the Vault documentation that highlights the differences between the two types of replication:

Capability	Disaster Recovery	Performance Replication
Mirrors the configuration of a primary cluster	Yes	Yes
Mirrors the configuration of a primary cluster’s backends (i.e., auth methods, secrets engines, audit devices, etc.)	Yes	Yes
Mirrors the tokens and leases for applications and users interacting with the primary cluster	Yes	No. Secondaries keep track of their own tokens and leases. When the secondary is promoted, applications must reauthenticate and obtain new leases from the newly-promoted primary.
Allows the secondary cluster to handle client requests	No	Yes

Describe the Vault agent

The purpose of the Vault agent is to simplify the integration between your applications and Vault. The Vault agent can handle the authentication to Vault, the administration of token renewals, and the retrieval of secrets. This allows you to use Vault with your application without necessarily making any changes to your application code. The agent could run as a sidecar to your application, or as a separate process on a virtual machine.

The details of how to configure the Vault agent is beyond the scope of this post, and it will depend a lot on how you deploy and run your application. To get familiar with the Vault agent I recommend doing one of the ten available official tutorials on the subject. For the full list see the official page.

Describe secrets caching

This sub-objective discuss one important feature of the Vault agent: secrets caching.

If your application sends a large volume of requests to Vault to retrieve secrets and renew tokens, you might experience performance issues with Vault. This could show up as latency in your requests or the need to constantly scale up your Vault cluster to keep up with the demand.

To alleviate part of this problem the Vault agent can cache tokens and leased secrets. This minimizes the amount of traffic that goes directly to Vault, and instead keeps most of the traffic between your application and the Vault agent.

There is a specific official tutorial on how to configure secrets caching in Vault, I recommend you take a look at it here.

Be aware of identities and groups

I have mentioned the identity secrets engine in previous posts. In this sub-section I will repeat some of what I have said and we will see how to use the concept of entities in our policies.

The identity secrets engine is Vault's internal identity management solution. Our Vault server might have multiple auth methods enabled, and for a given user there might be more than one way to sign-in. Let's say we have a user named Bob who has a GitHub account bob-github and an account in an unspecified external OIDC system named bob-oidc. Our Vault server has both the GitHub and OIDC auth methods enabled. Bob could sign in using any of the two available auth methods.

We can assign Bob an entity in Vault, and we can create aliases for each of Bob's external accounts. So we can have one entity that represents Bob, and one alias connected to this entity for Bob's GitHub account bob-github, and another alias connected to this entity for Bob's OIDC account bob-oidc.

We could assign policies to Bob's entity. This means that no matter what auth method Bob uses to sign in, he would get a token with those policies attached. We could also assign different policies to each of Bob's aliases. This means when Bob signs in using GitHub, he would get some additional policies. When Bob signs in using OIDC he would get a different set of additional policies.

This is one of the strengths of the identity secrets engine, but there is more.

Imagine we would like to assign entities the permissions to administrate secrets under their own name. For instance, when Bob signs in we would like to provide him a policy that gives him full access to secrets under secrets/bob/*. Similarly when Alice signs in she should get a policy that gives her full access to secrets under secrets/alice/*. Do we need to create a policy for each entity in our system, and make sure to assign the correct policy to the correct entity? We could use the concept of identities to do this in a better way.

Take a look at the following policy:

// policy.hcl
path "secrets/data/{{identity.entity.name}}/*" {
    capabilities = ["read", "create", "update", "delete", "patch"]
}

path "secrets/metadata/{{identity.entity.name}}/*" {
    capabilities = ["read", "create", "update", "delete", "patch"]
}

Here we have used a special template syntax to make dynamic policies. When this policy is applied, the entity associated with the token is made available to the policy, and the corresponding values are replaced in the policy document. In this case, when Bob signs in {{identity.entity.name}} is replaced with bob. Similarly when Alice signs in {{identity.entity.name}} is replaced with alice.

The identity object contains a lot of useful information, see the documentation for a list of what is available.

Explain response wrapping

Let's say you want to share a secret from Vault with someone. You could retrieve the secret value and send it in plaintext to your recipient. This is usually a bad idea because most likely the secret is logged somewhere, at least in your chat or email history.

Vault has the concept of response wrapping, which allows you to wrap the response to a request. This could be a GET request to retrieve a secret value. The response wrapping hides the actual response to this request, and instead provides you with a single-use token that your recipient can use to retrieve the response. Why is the token single-use? This is a safety mechanism to allow only a single recipient to unwrap the response. If your intended recipient can't unwrap the response because the token has already been used, then you know some man-in-the-middle attack has taken place. You should also make the time-to-live (TTL) of the wrapped response short, preferably seconds or minutes at most.

To see this in action, let's start a Vault development server:

$ vault server -dev

Let's use the default key/value secrets engine mounted at secret/ and add a new secret:

$ vault kv put -mount=secret database/password password=s3cr3t

======== Secret Path ========
secret/data/database/password

======= Metadata =======
Key                Value
---                -----
created_time       2023-09-15T12:55:24.243192Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

To read this secret back I would run the following command:

$ vault kv get -mount=secret database/password

======== Secret Path ========
secret/data/database/password

======= Metadata =======
Key                Value
---                -----
created_time       2023-09-15T12:55:24.243192Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

====== Data ======
Key         Value
---         -----
password    s3cr3t

Let's run that command again, but this time we add the -wrap-ttl flag:

$ vault kv get -mount=secret -wrap-ttl=5m database/password

Key                              Value
---                              -----
wrapping_token:                  hvs.CAESIMuy<truncated>c3JWRml3TW1SOWdzMTk
wrapping_accessor:               Zr7aVEftkByAKPTl3KeZJR2P
wrapping_token_ttl:              5m
wrapping_token_creation_time:    2023-09-15 14:56:59.423872 +0200 CEST
wrapping_token_creation_path:    secret/data/database/password

We received a wrapping_token that is valid for a duration of wrapping_token_ttl (in this case 5m). How do we use this token? First of all we would probably send the wrapping token to the intended receiver, then that person/system would issue the following command:

$ vault unwrap hvs.CAESIMuy<truncated>c3JWRml3TW1SOWdzMTk

Key         Value
---         -----
data        map[password:s3cr3t]
metadata    map[created_time:2023-09-15T12:55:24.243192Z custom_metadata:<nil> deletion_time: destroyed:false version:1]

The values are shown as Go maps, but we can identify that the response contains the password. What happens if I try to use the token again? The following:

$ vault unwrap hvs.CAESIMuy<truncated>c3JWRml3TW1SOWdzMTk

Error unwrapping: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/wrapping/unwrap
Code: 400. Errors:

* wrapping token is not valid or does not exist

The token has expired, because it was only possible to use it once! This is how response wrapping works, and although I showed it with the key/value secrets engine you can use it at a lot of places apart from this one.

Explain the value of short-lived, dynamically generated secrets

This sub-objective was covered in part five of this series. To summarize that discussion briefly:

Dynamically generated secrets are created when requested, which means they don't exist all the time and get lost and spread around like long-lived static secrets. Dynamic secrets together with a short time-to-live means the secret only exists for a brief period of time when it is needed, and no longer. If the dynamic secret value gets highjacked there is only a small window of time when the attacker could cause harm using the secret.

It seems like the blocks are called stanzas in Vault. Not sure why this is! It would be better to use the same nomenclature in all HCL, but that is just my opinion. I will stick to using blocks in this course, because I am not sure it matters all that much what you call it. ↩
But I will probably write a blog series about the Consul certification in the future! ↩
In Azure you actually don't place a subnet in a given zone like you do in AWS, you would rather have to make sure the resources (e.g. virtual machines) that you place in your subnets are in a given zone. For simplicity I drew it as if it works similar to AWS. ↩

Vault Associate Certification (Part 8): Be aware of the Vault API

Mattias Fjellström — Mon, 11 Sep 2023 16:02:41 +0000

Being aware of the Vault API make up the eight objective in the Vault certification journey. This objective covers the following sub-objectives:

Authenticate to Vault via cURL
Access Vault secrets via cURL

As is clear from the number of sub-objectives in this part this is not a huge topic. It is important to be aware of a few general things related to using the API that is implicitly handled for you when using the Vault CLI. You are not expected to be able to perform every common action using the API.

Authenticate to Vault via cURL

This sub-objective covers how to authenticate to Vault when running any operation that requires a token.

The Vault token must be provided as a header in all requests to Vault that requires authentication. There are two options for providing the token:

As the X-Vault-Token HTTP header with a value of <token>.
As the Authorization header with a value of Bearer <token>.

How do you obtain a token to begin with using the API? This depends on the authentication method you want to use. As an example, using the userpass auth method for a user named bob:

$ curl \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data '{"password": "S3cr3t"}' \
    http://127.0.0.1:8200/v1/auth/userpass/login/bob

Note that all requests are sent to <server url>:8200/v1/<path>.

The response from this request looks like the following:

{
    "request_id": "efa4b077-5820-1682-6c52-d30e8b48cccb",
    "lease_id": "",
    "renewable": false,
    "lease_duration": 0,
    "data": null,
    "wrap_info": null,
    "warnings": null,
    "auth": {
        "client_token": "hvs.CAESIPi7<truncated>Qza1ZVQ29LUGs",
        "accessor": "Y13YI0FyHxNm3zXl6i0OtpFI",
        "policies": [
            "default"
        ],
        "token_policies": [
            "default"
        ],
        "metadata": {
            "username": "bob"
        },
        "lease_duration": 2764800,
        "renewable": true,
        "entity_id": "cdb98f68-38b8-ecbb-7979-2ad1b9818f76",
        "token_type": "service",
        "orphan": true,
        "mfa_requirement": null,
        "num_uses":
        0
    }
}

In this example we also saw how the X-Vault-Token header is provided in the request.

Access Vault secrets via cURL

For this sub-section I assume you have a development server running with the key/value secrets engine (version 2) enabled at kv/. The specifics of how to write and read secrets from a secrets engine depends on what secrets engine you are using, read the API documentation for your secrets engine of choice.

Let us first write a new secret to database/password with some data:

$ curl \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data '{"data":{"password": "S3cr3t"}}' \
    http://127.0.0.1:8200/v1/kv/data/database/password

Notice how I write this data to kv/data/<my path>. The response to this request is:

{
    "request_id": "b9891d16-779e-af48-7e37-02deaf786e4d",
    "lease_id": "",
    "renewable": false,
    "lease_duration": 0,
    "data": {
        "created_time": "2023-09-11T15:47:09.14348Z",
        "custom_metadata": null,
        "deletion_time": "",
        "destroyed": false,
        "version": 1
    },
    "wrap_info": null,
    "warnings": null,
    "auth": null
}

Now we can read this data back:

$ curl \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request GET \
    http://127.0.0.1:8200/v1/kv/data/database/password

The response is:

{
  "request_id": "af5770e1-3d10-f0be-5536-4b0ab25eafc2",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "data": {
      "password": "S3cr3t"
    },
    "metadata": {
      "created_time": "2023-09-11T15:47:09.14348Z",
      "custom_metadata": null,
      "deletion_time": "",
      "destroyed": false,
      "version": 1
    }
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

We see that the response contains details about the request, details about the lease, and all the data and metadata.

Summary

This was a short post, but it is unlikely to be a big topic on the exam. Remember the following things and you should be good:

The request URL has the format <server URL>:8200/v1/<path>.
The token is provided in a header:
- X-Vault-Token with the value of <token>, or
- Authorization with the value of Bearer <token>.

Vault Associate Certification (Part 7): Utilize Vault UI

Mattias Fjellström — Mon, 11 Sep 2023 13:30:09 +0000

Utilizing the Vault UI make up the seventh objective in the Vault certification journey. This objective covers the following sub-objectives:

Authenticate to Vault
Configure authentication methods
Configure Vault policies
Enable Secret engines
Access Vault secrets

When you start a Vault development server (vault server -dev) the UI is enabled by default. However, this is not the case for real Vault servers. We'll discuss this further in part nine of the series, for this part we'll stick to the development server. By default you can reach the UI on your development instance on http://127.0.0.1:8200/ui/.

Authenticate to Vault

When you first enter the Vault UI you meet the following sign-in page:

In the drop-down you select the authentication method you would like to use to sign in. Note that the values in the drop-down might show more auth methods than what have been enabled on your Vault server.

Unless you first enable a few auth methods using the CLI your only option right now is to use the root token to sign in. Copy the root token from the terminal window where you started your Vault development server and paste it into the token input field, then click Sign in:

Now you have successfully authenticated to Vault using the UI! The process is more or less the same for any of the other auth methods. You select the auth method you want to use, and you provide the required information to sign in. For some auth methods you will be re-directed to the auth methods own page and then re-directed back to Vault after successful authentication.

Configure authentication methods

To configure a new auth method click on Access in the main menu on the left. You come to a page showing all the currently enabled auth methods. If you are using a brand new development server you will only see the token auth method in the list. To enable a new auth method click on Enable new method:

You come to a page showing all the available auth methods that you can enable. For the purpose of this post we will enable the userpass auth method. Select userpass and then click Next:

Now you have the option to specify which path the auth method should be enabled on. If you expand the Method Options section you can also do advanced configuration of the auth method, this includes things such as specifying the default lease TTL, the max TTL, and more. We'll ignore the advanced configurations for now. Accept the default path and click on Enable Method:

Once the method is enabled, click on Go to method. You come to a page listing all the users that are configured in the userpass auth method. Currently we have no user. To add a new user, click on Create user:

Specify a username (I specify bob) and a password. Expand the Tokens section to specify advanced settings for this user, such as which policies to attach to the tokens for this user. For now, let's ignore the advanced settings and click on Save:

You can sign out of Vault and sign back in using the userpass method and your new user to verify that it works.

Configuring other auth methods follow the same basic steps, but the configuration for each method will differ. The best approach is to read the documentation for the auth method you want to enable to know what configuration options to provide.

Configure Vault policies

To configure a new policy click on Policies in the main menu on the left. You come to a page listing all the policies on the Vault server. If you are using a brand new development server this list will only contain the default and root policies. Note that you can't view the root policy, but you can view and even edit the default policy. Click on Create ACL policy to create a new policy:

On the Create ACL policy page enter a name for your policy (I use my-policy) and enter the policy itself, then click on Create policy:

Notice that you could upload a file instead of entering the policy directly on the page.

Now the policy is available for use in the rest of the Vault UI. If you need to edit or delete a policy go back to the policy page and open the policy you want to update/delete.

Enable Secret engines

To configure a new secrets engine click on Secrets engines in the main menu on the left. You come to a page listing all the currently enabled secrets engines. My list contains two secrets engines, cubbyhole mounted on cubbyhole/ and key/value mounted on secret/. To add a new secrets engine click on Enable new engine:

Select the secrets engine you want to enable. For this demonstration I will use the key/value secrets engine. Select the engine and then click on Next:

Specify the path where you want to enable the secrets engine and then click Enable engine:

Note that you can expand the Method Options section to reveal advanced settings for the secrets engine. For the key/value secrets engine part of these advanced settings look like the following image. This is where you could specify that you want version 1 of the key/value secrets engine instead of the (default) version 2:

Adding other secrets engines will follow the same pattern, but the various settings might differ between the different secrets engines.

Access Vault secrets

Now let's continue the last section by adding secrets to our secrets engine. Go to the secrets engine page by clicking on Secrets engines in the main menu on the left, then select the kv/ secrets engine in the list. Click on Create secret to add a new secret:

Provide a path for the secret and enter the secret data. I add my secret at the path of database/password and the secret data consists of password=<my secret password>. To create the secret click on Save:

Note that there was an option to add the secret data as JSON instead of adding them as key/value pairs in the UI.

If we want to read or update our secret we can find it by going back to the secrets engine page and click on database/:

Then click on password:

Finally we arrive at the page showing the secret:

We also see that we can create a new version of the secret, we can delete it, make a copy, view different versions, and more. Also note that there is a separate tab for Metadata.

Working with secrets using the other secrets engines follows similar patterns but will vary depending on the type of secrets engine.

Vault Associate Certification (Part 6): Utilize Vault CLI

Mattias Fjellström — Fri, 08 Sep 2023 15:17:15 +0000

In the previous parts of this series we have seen many examples of how to use the Vault CLI to perform various tasks in Vault. In fact, we have only used the CLI to interact with Vault. In the next part of this series we will see the first non-CLI way of interacting with Vault (spoiler: we will use the Vault UI).

Much of the content in this part will be repetition. But repetition is the mother of learning is what they say!

Using the CLI make up the sixth objective in the Vault certification journey. This objective covers the following sub-objectives:

Configure authentication methods
Authenticate to Vault
Configure Vault policies
Enable Secret engines
Access Vault secrets
Configure environment variables

Before we dig into the sub-objectives let's start with a short CLI-101.

Vault CLI 101

I begin with some basics around Vault CLI commands that is common for every command you might run.

Each (most) command follows this general form:

$ vault <command> [options] [path] [arguments]

Each command we'll see starts with vault, because that is the Vault CLI that we all know and love.
<command> might consist of one or two pieces that determines what operation we want to perform in Vault, e.g. secrets enable, auth enable, write, read, token revoke, etc.
The [options] include flags (i.e. -flag1=value, -flag2=value, etc). There are some global flags that are available for all commands, and some specific flags depending on the type of command.
The [path] specifies the path (remember that everything in Vault exists at a path) where you want to perform the action.

Using the help flag -h for the command you would like to run is one of the best way of getting information for a given command. There is also a path-help command that gives you specific help for a path, for example:

$ vault path-help /sys/capabilities

Request:        capabilities
Matching Route: ^capabilities$

Fetches the capabilities of the given token on the given path.

## PARAMETERS

    path (slice)

        (DEPRECATED) Use 'paths' instead.

    paths (slice)

        Paths on which capabilities are being queried.

    token (string)

        Token for which capabilities are being queried.

## DESCRIPTION

Returns the capabilities of the given token on the path.
The path will be searched for a path match in all the policies
associated with the token.

Configure authentication methods

To enable a new auth method, in this case the userpass auth method, run:

$ vault auth enable userpass

Success! Enabled userpass auth method at: userpass/

To see what auth methods you currently have enabled run:

$ vault auth list

Path         Type        Accessor                  Description                Version
----         ----        --------                  -----------                -------
token/       token       auth_token_5b8eda92       token based credentials    n/a
userpass/    userpass    auth_userpass_20319701    n/a                        n/a

Configuring default settings for an auth method is done using the tune subcommand. In the following example I change the default lease TTL for tokens coming from the userpass auth method from 32 days (the default value) to 12 hours:

$ vault auth tune -default-lease-ttl=12h userpass/

Success! Tuned the auth method at: userpass/

If you want to move an enabled auth method to another path you can do so using the move subcommand. In the following example I move the userpass auth method from userpass/ to up/

$ vault auth move auth/userpass/ auth/up/

Started moving auth method auth/userpass/ to auth/up/, with migration ID 3548ea1b-a039-99bc-85c9-94a2578d8d65
Success! Finished moving auth method auth/userpass/ to auth/up/, with migration ID 3548ea1b-a039-99bc-85c9-94a2578d8d65

Note that you must specify the full path of auth/userpass/ etc. This is a bit confusing, but all the auth methods are really mounted under auth/ and not directly at userpass/ and so on.

When you no longer need an auth method you can disable it:

$ vault auth disable up/

Success! Disabled the auth method (if it existed) at: up/

Authenticate to Vault

In the previous posts we've always had a root token set up with our local Vault CLI so we've not had to authenticate to Vault. If we're a user, or application, we must first authenticate using one of the enabled auth methods.

To illustrate how to authenticate to vault using the userpass auth method (to continue from the previous section) first enable the auth method:

$ vault auth enable userpass

Success! Enabled userpass auth method at: userpass/

Next create a user named alice:

$ vault write auth/userpass/users/alice \
    password=S3cr3t123 \
    policies=my-policy

Success! Data written to: auth/userpass/users/alice

To authenticate to Vault using the userpass auth method and the user named alice run the following command:

$ vault login -method=userpass username=alice

Password (will be hidden): ***

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  hvs.CAESIBZ7YFEM<truncated>PaXJwVFdaUDU
token_accessor         WuKRVGKKGMlcRE3oBdHfnyqP
token_duration         768h
token_renewable        true
token_policies         ["default" "my-policy"]
identity_policies      []
policies               ["default" "my-policy"]
token_meta_username    alice

Similar commands (vault login -method=<auth method>) would be used for the other available auth methods.

Configure Vault policies

Do you remember policies from back in part two of this course? Policies are used to allow capabilities (actions) on a path(s) for a token. Policies are written in HashiCorp Configuration Language (HCL) or JSON and consist of one or more path blocks.

An example of a basic policy:

// policy.hcl

path "kv/database/password" {
    capabilities = [ "read" ]
}

path "kv/api/key" {
    capabilities = [ "read" ]
}

This policy allows a given token to read from two paths, kv/database/password and kv/api/key.

Once we have written our policy document (policy.hcl in this case) we can create the actual policy in Vault:

$ vault policy write my-policy policy.hcl

Success! Uploaded policy: my-policy

We can list all our available policies using the following command:

$ vault policy list

default
my-policy
root

To view a specific policy we can use the following command:

$ vault policy read my-policy

path "kv/database/password" {
    capabilities = [ "read" ]
}

path "kv/api/key" {
    capabilities = [ "read" ]
}

If we no longer need to keep a policy we can delete it using the following command:

$ vault policy delete my-policy

Success! Deleted policy: my-policy

Apart from that there is not much more to say about working with policies using the Vault CLI. To see all available sub-commands and options use:

$ vault policy -h

Enable Secret engines

In the previous post we saw several examples of how to work with secrets engines using the Vault CLI. This section contains a summary of the common commands. I will illustrate the commands using the key/value (kv) secrets engine, both version 1 and 2.

To enable the kv secrets engine:

$ # enabling version 1 at the path kv/
$ vault secrets enable kv
$
$ # enabling version 1 at the path kvv1/
$ vault secrets enable -path=kvv1 kv
$
$ # enabling version 2 at the path kvv2/
$ vault secrets enable -version=2 -path=kvv2 kv

If you have enabled version 1 at the path kv/ and would like to upgrade it to version 2 you could run the following command:

$ vault kv enable-versioning kv

Success! Tuned the secrets engine at: kv/

To see all the secrets engines I have enabled I can run:

$ vault secrets list

Path          Type         Accessor              Description
----          ----         --------              -----------
cubbyhole/    cubbyhole    cubbyhole_e99e381f    per-token private secret storage
identity/     identity     identity_146bba4e     identity store
kv/           kv           kv_cd4c8f19           n/a
secret/       kv           kv_5260e2d2           key/value secret storage
sys/          system       system_55d6ae73       system endpoints

If I would like additional details I can add the -detailed flag:

$ vault secrets list -detailed

# (output not shown due to size limits)

I can move a secrets engine from one path to another. To move my kv secrets engine from kv/ to kv-new/ I run:

$ vault secrets move kv/ kv-new/

Started moving secrets engine kv/ to kv-new/, with migration ID a63e75dd-b047-f095-6b67-de1fe27bef88
Success! Finished moving secrets engine kv/ to kv-new/, with migration ID a63e75dd-b047-f095-6b67-de1fe27bef88

When I no longer need a given secrets engine I can disable it:

$ vault secrets disable kv-new

Success! Disabled the secrets engine (if it existed) at: kv-new/

Access Vault secrets

We saw many examples of how to access Vault secrets using the Vault CLI in the previous post, but we'll repeat that material a bit here. For these examples I will continue using the key/value secrets engine, version 2. To start I enable the secrets engine:

$ vault secrets enable -version=2 -path=secrets kv

Success! Enabled the kv secrets engine at: secrets/

Next I store a secret at the path database/passwords:

$ vault kv put secrets/database/password password=s3cr3tz

========= Secret Path =========
secrets/data/database/password

======= Metadata =======
Key                Value
---                -----
created_time       2023-09-08T15:05:57.185984Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Since I am using version 2 of the kv secrets engine I can update the password to a new version if I like:

$ vault kv put secrets/database/password password=s3cr3tzhello

========= Secret Path =========
secrets/data/database/password

======= Metadata =======
Key                Value
---                -----
created_time       2023-09-08T15:07:03.875497Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            2

We can see that version increased to 2. To read my secret back I can run:

$ vault kv get secrets/database/password

========= Secret Path =========
secrets/data/database/password

======= Metadata =======
Key                Value
---                -----
created_time       2023-09-08T20:07:03.875497Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            2

====== Data ======
Key         Value
---         -----
password    s3cr3tzhello

This output is not very convenient for parsing, so let's specify that we want JSON output instead:

$ vault kv get -format=json secrets/database/password

The returned data has this format:

{
  "request_id": "d39f28d7-14fc-2f40-5b8d-c7fac2c6cc9f",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": {
    "data": {
      "password": "s3cr3tzhello"
    },
    "metadata": {
      "created_time": "2023-09-08T20:07:03.875497Z",
      "custom_metadata": null,
      "deletion_time": "",
      "destroyed": false,
      "version": 2
    }
  },
  "warnings": null
}

To access a given version of the secret (i.e. an older version) I run:

$ vault kv get -version=1 secrets/database/password

========= Secret Path =========
secrets/data/database/password

======= Metadata =======
Key                Value
---                -----
created_time       2023-09-08T20:05:57.185984Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

====== Data ======
Key         Value
---         -----
password    s3cr3tz

If I would like to only access the metadata for a given secret I can do so:

$ vault kv metadata get secrets/database/password

========== Metadata Path ==========
secrets/metadata/database/password

========== Metadata ==========
Key                     Value
---                     -----
cas_required            false
created_time            2023-09-08T20:05:57.185984Z
current_version         2
custom_metadata         <nil>
delete_version_after    0s
max_versions            0
oldest_version          0
updated_time            2023-09-08T20:07:03.875497Z

====== Version 1 ======
Key              Value
---              -----
created_time     2023-09-08T20:05:57.185984Z
deletion_time    n/a
destroyed        false

====== Version 2 ======
Key              Value
---              -----
created_time     2023-09-08T20:07:03.875497Z
deletion_time    n/a
destroyed        false

When working with version 2 of the kv secrets engine we can delete versions of our secrets and we can destroy them. Deleting a secret version just places a delete marker on it, but it is recoverable. Destroying a secret version permanently removes it. In the following command I destroy both version 1 and 2 of my secret:

$ vault kv destroy -versions=1,2 secrets/database/password

Success! Data written to: secrets/destroy/database/password

Configure environment variables

There are a number of environment variables that Vault looks for. Most of them are not required at all but could be useful in certain situations. I'll explain a few of the most common environment variables below¹:

VAULT_TOKEN allows you to set the current Vault token which should be used for all the commands you execute.
VAULT_ADDR should contain the URL and port where Vault is running, e.g. https://my-vault-server.com:8200/.
VAULT_CLIENT_TIMEOUT is the timeout for commands issued to the Vault server, it defaults to 60 seconds.
VAULT_FORMAT allows you to select the output format for the CLI. The available values are json, yaml, and table.
VAULT_LOG_LEVEL allows you to control the verbosity of the logs that Vault reports. Available values are trace, debug, info, warn, and err. The default log level is info.

For a complete list of environment variables see https://developer.hashicorp.com/vault/docs/commands#environment-variables. ↩

Vault Associate Certification (Part 5): Compare and configure Vault secrets engines

Mattias Fjellström — Fri, 08 Sep 2023 08:45:37 +0000

Secrets engines make up the fifth objective in the Vault certification journey. This objective covers the following sub-objectives:

Define secrets engines
Choose a secret method based on use case
Contrast dynamic secrets vs. static secrets and their use cases
Define transit engine

Define secrets engines

Note: In the official list of sub-objectives for this certification objective the Define secrets engines sub-objective is listed as the last of four sub-objectives. I think it makes more sense to go through it first because all of the other sub-objectives build upon what secrets engines are.

Secrets engines allow us to work with secrets in a third-party system. I say third-party system, because that will most likely be the case. There are secrets engines that do not involve a third-party system too, but they are fewer in number. Generally, a secrets engine store, generate, or encrypts data.

There are a number of secrets engines that are enabled by default when you start up a Vault server. What secrets engines are available depends a bit on your server, but there are two which are always mounted by default and they are important to be aware about:

The cubbyhole secrets engine is a special secrets engine where each Vault token has its own secrets storage. A secret stored in a cubbyhole for one token can only be retrieved by that same token.
The identity secrets engine is another special secrets engine used for managing identities (or entities) in Vault. It is the internal identity management solution for Vault. If you have enabled multiple auth methods on your server your users might use any of them to sign in. An entity is tied to a specific user no matter what auth method that user used to sign in to Vault. The identity secrets engine is used to administer these entities.

Note that none of these two special secrets engines can be disabled.

To enable a new secrets engine using the CLI:

$ vault secrets enable <name>

Replace <name> with the secrets engine you want to enable. Currently the following official secrets engine are supported:

Active Directory
AliCloud
AWS
Azure
Consul
Cubbyhole
Databases (with plugins for e.g. Cassandra, Couchbase, Elasticsearch, MongoDB, MSSQL, Oracle, PostgreSQL, Redis, Snowflake, and more)
Google Cloud
Google Cloud KMS
Identity
Key Management (Azure Key Vault, AWS KMS, GCP Cloud KMS)
Key/Value (version 1 and 2)
KMIP
Kubernetes
MongoDB Atlas
Nomad
LDAP
PKI (certificates)
RabbitMQ
SSH
Terraform Cloud
TOTP
Transform
Transit
Venafi (certificates)

It is clear that this is a long list. To see which secrets engines you have currently enabled in your Vault server run the following command:

$ vault secrets list 

Path          Type         Accessor              Description
----          ----         --------              -----------
azure/        azure        azure_2f066eac        n/a
cubbyhole/    cubbyhole    cubbyhole_255a0b95    per-token private secret storage
identity/     identity     identity_1b99e3f9     identity store
kv/           kv           kv_35836191           n/a
kvv2/         kv           kv_5beecabf           n/a
secret/       kv           kv_cdd3836c           key/value secret storage
sys/          system       system_2d023c0b       system endpoints used for control, policy and debugging

I have enabled a few secrets engines as you can tell from the output.

Apart from the secrets engines listed above it is also possible to create your own plugins to extend the support in Vault to other systems. Of course I will not go through how to use each and every secrets engine in this post, but we'll see a few of them in use.

The lifecycle of a secrets engine generally follows these steps:

Enable the secrets engine in Vault.
Tune the global settings of the secrets engine. This step is not required if you accept the default settings.
Use the secrets engine to generate secrets, store secrets, and more, for users and applications.
Disable the secrets engine when no longer needed. This step might not happen very often, usually you keep a secrets engine for a forseeable future once you have started using it.

To get our hands dirty we will take a look at two common secrets engines that will most likely appear in the certification exam in one way or another: the key/value secrets engine. This secrets engine exists in two versions. The difference between the two versions is that version 2 supports versioned secrets. Let's begin by enabling version 1:

$ vault secrets enable kv

Success! Enabled the kv secrets engine at: kv/

The key/value secrets engine allows me to store static secrets as key/value pairs at paths below the kv/ path. Let's store some data at kv/database/password:

$ vault kv put kv/database/password password=p@ssw0rd

Success! Data written to: kv/database/password

Now let's read this data back:

$ vault kv get kv/database/password

====== Data ======
Key         Value
---         -----
password    p@ssw0rd

That was easy enough. Let's now perform the same steps using version 2 of the key/value secrets engine. I begin by enabling the secrets engine at the kvv2/ path:

$ vault secrets enable -version=2 -path=kvv2 kv

Success! Enabled the kv secrets engine at: kvv2/

Next let's store my database password:

$ vault kv put kvv2/database/password password=p@ssw0rd

======= Secret Path =======
kvv2/data/database/password

======= Metadata =======
Key                Value
---                -----
created_time       2023-09-08T20:19:25.450796Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

The output looks different than before. Notably we see that the output includes metadata about the secret, and among this metadata is the version. Remember that version 2 of the key/value secrets engine supports versioned secrets. What this means in practice is that I can store new versions of the same secret (i.e. rotating the secret) at the same path, but still retrieve old versions if needed. This is not possible with version 1 of the key/value secrets engine. If I write to a path that already exists version 1 of key/value secrets engine would just replace the value stored at that path.

Another thing to note is that although I wrote the data to kvv2/database/password the actual data is available at kvv2/data/database/password and the metadata is available at kvv2/metadata/database/password. This is a bit confusing but this is how this secrets engine works. It is important to remember this when writing policies for applications that must use this secrets engine. We could write policies that allow access to the metadata but not the data, or vice versa.

Now let's read this data back:

$ vault kv get kvv2/database/password

======= Secret Path =======
kvv2/data/database/password

======= Metadata =======
Key                Value
---                -----
created_time       2023-09-08T20:19:25.450796Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

====== Data ======
Key         Value
---         -----
password    p@ssw0rd

We see that I get both the metadata and data back when I read the secret path.

HashiCorp has great tutorials on the key/value secrets engine that you should go through in order to get some hands-on practice using them, it will be beneficial for the certification. See the following tutorials:

We have seen how to work with static secrets in Vault. In a later section we will see examples of dynamic secrets and how the concept of a lease exists for those kinds of secrets, similar to what we saw for tokens in the previous post.

Choose a secret method based on use case

Similarly to how we choose auth methods based on the use-case we have, or the available third-party systems that integrate with Vault, we also choose the secrets engines we want to use based on what our use-case is and which third-party systems we might already be using.

There are obvious use-cases where the choice of secrets engine is easy to make:

Let's say we want our application to be able to access blobs stored in Amazon S3. The AWS secrets engine seems apt in this case.
If I run my applications in Azure and I need Azure credentials in order to access databases, key vaults, storage, or anything else in Azure then the Azure secrets engine is the obvious choice.
If we are working with a given database system we will probably want to use database secrets engine with the correct plugin.
If I need to be able to generate certificates for my applications and clients I will probably use the PKI secrets engine.

We see that to choose a secrets engine is not that difficult. It depends on what types of secrets you want to work with and what third-party systems (e.g. AWS, Azure, GCP) you are already using.

Contrast dynamic secrets vs. static secrets and their use cases

Static secrets are secrets that doesn't change. At lease they do not change very often. There is a big issue with static secrets.

Dynamic secrets are generated on-the-fly when we ask for them, and they expire once the secret lease expires. If regular secrets would be known as Secrets 1.0, then dynamic secrets would be Secrets 2.0. To illustrate the power of dynamic secrets we'll go through an example.

I will use the Azure secrets engine to generate credentials for a service principal on-the-fly. I will not spend time on explaining the Azure-specific configuration in detail, we'll instead focus on what we need to configure in Vault. Let's begin by enabling the Azure secrets engine at the path azure/:

$ vault secrets enable azure

Success! Enabled the azure secrets engine at: azure/

We must allow Vault to administer Azure service principals for us, so we must configure the secrets engine with credentials to Azure. In Azure I configure a service principal that Vault can use, and I add this service principal as an owner of my subscription as well as give it permissions to create new service principals in my Azure AD tenant. I store some required values for this in environment variables AZURE_SUBSCRIPTION_ID, AZURE_TENANT_ID, AZURE_CLIENT_ID, and AZURE_CLIENT_SECRET. Now I am ready to configure the secrets engine:

$ vault write azure/config \
    subscription_id=$AZURE_SUBSCRIPTION_ID \
    tenant_id=$AZURE_TENANT_ID \
    client_id=$AZURE_CLIENT_ID \
    client_secret=$AZURE_CLIENT_SECRET

Success! Data written to: azure/config

At this point we technically know the password of this service principal that Vault will be using. That is not ideal. Most secrets engines allow us to rotate the credential that is used for administrative purposes, so that only Vault knows the secret value. To do this run the following command:

$ vault write -f azure/rotate-root

Success! Data written to: azure/rotate-root

Note that the old secret is actually not deleted, and you must go do that manually. The new secret is being used, and only Vault knows its value.

Next I must configure a role in the secrets engine. A role describes what permissions will be granted to the dynamic secrets that are generated. I will call my role demo-role and it will get the contributor role (here the role refers to the Azure RBAC role) on a specific resource group named rg-vault-demo in my Azure subscription:

$ vault write azure/roles/demo-role ttl=5m azure_roles=-<<EOF
    [
        {
            "role_name": "Contributor",
            "scope":  "/subscriptions/$AZURE_SUBSCRIPTION_ID/resourceGroups/rg-vault-demo"
        }
    ]
EOF

Success! Data written to: azure/roles/demo-role

I could have provided multiple Azure RBAC roles for various scopes if I wanted to. Note that I also configured that the time-to-live (TTL) of the dynamic secret should be five minutes (ttl=5m). To have Vault generate dynamic secrets I run the following command:

$ vault read azure/creds/demo-role

Key                Value
---                -----
lease_id           azure/creds/demo-role/HsUWYY9Ue37CXdI84bZ7e2UH
lease_duration     5m
lease_renewable    true
client_id          d4af1d42-d4c7-4c49-b8ec-9fb8920350e1
client_secret      wZP8Q<masked>NOadO

We are provided with client_id and client_secret that can be used to interact with Azure. Here we also see that I get a lease_id, a lease_duration, and a lease_renewable property. The lease ID is the handle for this secret that allows me to perform actions similar to what we saw for tokens in the previous post. I can lookup a given lease ID to get information about it:

$ vault lease lookup azure/creds/demo-role/HsUWYY9Ue37CXdI84bZ7e2UH

Key             Value
---             -----
expire_time     2023-09-08T21:20:38.971253+02:00
id              azure/creds/demo-role/HsUWYY9Ue37CXdI84bZ7e2UH
issue_time      2023-09-08T21:15:38.971252+02:00
last_renewal    <nil>
renewable       true
ttl             2m8s

I can renew the lease as long as it is renewable (i.e. it has not expired and renewable is set to true):

$ vault lease renew azure/creds/demo-role/HsUWYY9Ue37CXdI84bZ7e2UH

Key                Value
---                -----
lease_id           azure/creds/demo-role/HsUWYY9Ue37CXdI84bZ7e2UH
lease_duration     5m
lease_renewable    true

When I am done using the secret and want to revoke it early I can do so:

$ vault lease revoke azure/creds/demo-role/HsUWYY9Ue37CXdI84bZ7e2UH

All revocation operations queued successfully!

If I try to lookup my lease ID now I get an error:

$ vault lease lookup azure/creds/demo-role/HsUWYY9Ue37CXdI84bZ7e2UH

error looking up lease id azure/creds/demo-role/HsUWYY9Ue37CXdI84bZ7e2UH: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/leases/lookup
Code: 400. Errors:

* invalid lease

It is clear that working with leases for dynamic secrets seems a bit better than working with leases for tokens due to the fact that there is a separate lease_id for dynamic secrets. This means we could delegate the administration of the lease to a separate user or application, without giving away the actual secret value.

If the TTL expires by itself the secret is revoked, and the generated service principal in Azure is deleted. This is the power of dynamic secrets compared to static secrets with a possibly infinite lifetime.

So when should we use dynamic secrets? My answer is that use them whenever you can. Sometimes it might not be possible to do so, if the secrets engine you use does not support it (e.g. key/value secrets engine) or if you have a legacy system where it is difficult to implement the handling of dynamic secrets. Before you decide that it is too difficult to implement dynamic secrets in your legacy app, read up on the sidecar pattern and the Vault agent to see if they might help.

Define transit engine

The transit engine is used for cryptography-as-a-service (CaaS) or encryption-as-a-service (EaaS). It is used for various encryption services on data in-transit (thus the name). This secrets engine have a few different use-cases but the primary use-case concerns offloading cryptographic operations from applications, and allowing applications to encrypt data before it is stored in the application's primary data store. The transit secrets engine allows you to standardize how you handle encryption in all of your applications.

An important point to note is that the transit secrets engine does not itself store any data in Vault! It is your applications responsibility to persist the encrypted values to secure storage.

Enabling the transit secrets engine is done using the following command:

$ vault secrets enable transit

Success! Enabled the transit secrets engine at: transit/

I could also have enabled the transit secrets engine at a non-default path by adding the -path=<path> flag like so:

$ vault secrets enable -path=mytransit transit

Success! Enabled the transit secrets engine at: mytransit/

With the secrets engine enabled I can create an encryption-key named my-encryption-key with default settings:

$ vault write -f transit/keys/my-encryption-key

Key                       Value
---                       -----
allow_plaintext_backup    false
auto_rotate_period        0s
deletion_allowed          false
derived                   false
exportable                false
imported_key              false
keys                      map[1:1694077952]
latest_version            1
min_available_version     0
min_decryption_version    1
min_encryption_version    0
name                      my-encryption-key
supports_decryption       true
supports_derivation       true
supports_encryption       true
supports_signing          false
type                      aes256-gcm96

The default type of the encryption key is aes256-gcm96, if I wanted to I could provide a different type by adding the -type=<type> flag.

Finally I can use my new encryption-key to encrypt some data using the transit/encrypt/ path for my key. Note that the data we want to encrypt must be base64-encoded due to the fact that Vault accepts any binary data and not just text:

$ vault write transit/encrypt/my-encryption-key \
    plaintext=$(echo "s3cr3t p4ssw0rd" | base64)

Key            Value
---            -----
ciphertext     vault:v1:mMzuz5ZvqPmfupPUYplgKcwSWzu2XSpOoybXrgSqfYW9T3f8qBBH6XZbAW0=
key_version    1

We got back ciphertext which is the encrypted value. The ciphertext value starts with vault which indicates that Vault has handled the encryption of this value. Following this is v1 which indicates that the first version of my key (I only have one version at this time) was used to encrypt the data.

Once again, note that Vault has not stored this ciphertext value. It is our application's responsibility to store this value wherever it needs to be stored.

To get my plaintext data back I first decrypt the data using the transit/decrypt/ path for my key:

$ vault write transit/decrypt/my-encryption-key \ 
    ciphertext=vault:v1:mMzuz5ZvqPmfupPUYplgKcwSWzu2XSpOoybXrgSqfYW9T3f8qBBH6XZbAW0=

Key          Value
---          -----
plaintext    czNjcjN0IHA0c3N3MHJkCg==

And I decode the value from base64:

$ echo czNjcjN0IHA0c3N3MHJkCg== | base64 -d

s3cr3t p4ssw0rd

This was a short look at the transit secrets engine and what it is mainly used for. We will revisit this topic in part 10 of this course: Explain encryption as a service.