DEV Community: Ben Coleman

Contributing to the Azure Terraform Provider

Ben Coleman — Wed, 02 Dec 2020 17:21:11 +0000

Ok so you're using Terraform to deploy and manage your Azure resources? It's pretty neat, but at some point it's likely you will stumble over an Azure resource or configuration that's not available in Terraform. The Terraform Provider for Azure covers a huge number of resources in Azure, but there's always going to be gaps, given how quickly Azure evolves.

So what can you do? Well, you can find a workaround (like using Azure CLI or even ARM Templates), or raise an issue on the Azure Provider GitHub project and hope some kind soul picks it up, or <deep intake of breath> roll up your sleeves, open your IDE of choice (VS Code of course!) and start work on adding yourself.

In this post I'll talk through some of the planning, pre-work and other technical investigation you're going to need to do, before you go diving headfirst into the code base. This is a replay of my own journey of adding to the provider, and trust me this isn't something you want to start doing without more than a little planning.

Basics & Pre-reqs

In Dante's Inferno, he describes hell having nine different layers or circles. Thankfully we have just five to worry about, however the amount of suffering involved might be the same as a trip through Dante's vision of unending torment.

Our layers are a little different, but no less infernal 😉

The Azure REST API Spec - This repo contains OpenAPI specifications for all the Azure APIs. These specs are OpenAPI/Swagger definitions which describe shape of every API in Azure. Despite being at the top of this stack, most of the time you can pretty much ignore this, unless you think you've hit a bug with the way the API is behaving.
The Azure REST API - This is the main management interface to Azure and implements the above API spec. This is going to dictate & shape everything else we touch.
Azure SDK for Go - This wraps the above Azure API with a set of packages & client code making it "easier" than calling the API directly. Easier is matter of some debate.
Terraform Provider for Azure - Also called 'azurerm' (which is the provider name), this in turn wraps the Go SDK, with a set of CRUD operations that Terraform understands.
Terraform - The Azure provider is a plugin and extension to the core Terraform system

Needless you say you're going to need some Terraform experience, at least with the basics. It's also worth reading the docs on provider plugins to get familiar with the concepts and main interfaces around providers.

You're also going to need to be able to program in Go. You won't need to be a Go uber-coder, but if you've never touched it, you're going to have a very rough time, this is not the sort project in which I would advise you start learning Go with.

It also helps if you've some prior experience with the Azure REST APIs (also called the ARM APIs), such as how they are versioned, the sorts of operations you can perform and concepts such as authentication, resource providers, scopes etc.

So brave traveller, are you ready to descend?

The Azure API

First you need to identify the API or APIs in Azure that are going plug the gap you need. This might be something you can deploy or configure from the Azure portal, the CLI but not through Terraform. You're going to spend a lot of time in this section of the Azure docs https://docs.microsoft.com/en-us/rest/api/azure/

Let's assume you find what you need (if you didn't your journey has been a short one!)

Take plenty of time to research the API, try it out & experiment with it. My suggestion is to use the fantastic REST Client for VS Code. Build up a set of calls in a .http or .rest file which you can refer back to

I wrote a short post on using this extension to authenticate and call Azure APIs, and how easy it makes it - so it's worth referring to that

Calling Azure APIs with the REST Extension for VS Code

Ben Coleman for CSE Dev Crews ・ Nov 18 '20

#azure #rest #vscode #api

Test the API thoroughly, time spent here will time well spent later. Poke at all the edge cases, "what if that array is empty?", "what if I put an invalid string here?", "if I remove that value, what is the default?". The docs will be some help, but generally are pretty bare bones.

Once you're happy that you've got good a handle on calling the API directly, it's time to move on to the next layer...

So traveller, I see you are prepared to descend deeper, let us journey on?

Build Go Prototype / Code Spike

The temptation might be to now jump into the Terraform provider code, but I'd hold off. Next it's worth spending some time working with the Azure SDK for Go to call the API in question

Azure / azure-sdk-for-go

This repository is for active development of the Azure SDK for Go. For consumers of the SDK we recommend visiting our public developer docs at:

The Go SDK is auto-generated from the aforementioned Azure REST API Spec, this means it's not exactly the most friendly SDK to work with. And with extremely rigid adherence to having everything well typed, means it can be laborious work to even see what the API is returning.

I would suggest using a code spike approach and build a simple console Go app, and in doing so take a look at:

Which package in the SDK you need, as they are versioned to match the Azure API versions, but it can be a challenge to even find the right package in the complex tree within the SDK.
How the API client in that package is instantiated, normally with a subscription ID and possibly location, but each client is different.
Make calls to the create, update and get operations. It's important you know how to build the input struct for a create operation, which might not be as trivial as it sounds. Also how to extract data from the response, which can often involve some "walking" down through all the properties and casting/asserting as you go

This might seem like busy work and a waste of time, but you'll learn a lot in doing this, and it's much easier to test at this point before needing to go through an entire Terraform plan & apply loop. In addition a lot of this code you'll be able to copy and paste into the provider code.

Are you confident to push on, to our final infernal layer?

The Azure Terraform Provider

The final part of the "journey", but there's still a long way to go - as this part will by far take the longest. I'm not going to go super deep into the low level code & implementation specifics in this section, that could fill another post (or in fact several!), instead I want to focus on the design, approach & thinking you'll need to do.

Start with designing the shape of your resource, sketch out some pseudo code HCL. By now you've probably got a good idea what this will look like based on the body of your API requests, but think about if you can make it a little more user friendly. For example where it comes to field names, think what does the end user likely know this field/feature by? There can often be a disconnect between the "internal" names used in the API and the "external" names used in the Portal & CLI

One area to consider is where the API accepts an array of objects, this can be simplified in the HCL as a repeated block with the same name, this will automatically become an array. Therefor you will need to switch from a pluralized name to a singular. For example

Example API JSON payload

"rules": [
  {
    "name": "rule 1"
  },
  {
    "name": "rule 2"
  }
]

Example equivalent Terraform HCL

resource azurerm_amazing_new_thing "example" {
  rule {
    name = "rule 1"
  }
  rule {
    name = "rule 2"
  }
}

Moving onward finally to the provider code itself which is hosted here:

hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager

You're contributing to a public open source project on GitHub, ultimately ending with a pull request which (hopefully!) will be merged into in main codebase and released. There's a certain way of working when contributing to OSS projects, there's plenty of guides online to help with this, e.g. first-contributions and MarcDiethelm/contributing

Take plenty of time to explore the codebase, but thankfully you don't need to understand it all, focus on digging into the azurerm/internal/services tree. Take some time to look at some Azure resources you are familiar with, try to find something not too complex (e.g. App Service) but not so simple it's trivial.

The resources are grouped by their Azure Provider API, and under each group there will be some shared code common to all resources in that package. The key ones being:

registration.go - This should be a short file containing a SupportedResources() function which maps the Terraform resource names e.g. azurerm_new_thing to the Go function that provides that resource, e.g resourceArmNewThing(). You should be able to copy and paste an existing line in here.
client/client.go - This is where all the API clients from the SDK are created for the given API package, really it's another case of copy and pasting an existing one as there's a clear pattern to things.

Implementing your resource will be a matter of copying an existing one and making a LOT of changes. There will be a set of callback functions for the CRUD operations Create, Read, Update and Delete, implementing those functions is the heart of the task at hand, and detailed here. Also the Schema is defined in here, this is the implementation of the HCL design we discussed a moment ago. You can read the docs also take a good look at how other resources define their schemas, and learn by example

Testing your code will take two shapes, informal testing locally with some Terraform HCL is the first place to start. In order to load your locally built version of the provider there's some hoops to jump though, which have been made a lot more difficult since Terraform 0.13.

The script below I created helps with this, it builds the azurerm code, and then copies the resulting provider binary to where the Terraform CLI can find it. Runs terraform init with -plugin-dir set and then does a standard Terraform plan & apply

Build provider & load plugin test harness script:

https://github.com/benc-uk/terraform-sandbox/blob/master/provider-test-harness/run.sh

Note. In the script I pick a deliberately out of range version for the plugin as a belt and braces to make sure my copy is really being picked up, so the version of azurerm in your test Terraform would need to be version = ">=99.0.0"

You'll run this inner loop test many, MANY times until you get things working. The next set of tests are more formal acceptance tests. These live in the tests folder of the tree you are working in. All acceptance tests start with TestAcc this means they won't run in the standard unit test cycle, they all contain embedded inline HCL which defines the test resources to create & destroy.

These tests are only run when certain flags are specified, namely calling make acctest. Again I suggest learning by example, copy some existing tests and build on what you find. Note these tests will deploy real resources in Azure which means they can be very slow to run and incur costs!

The final stage before submitting your PR is to run though the test/check suite locally. This is done automatically in the CI pipeline of the project in GitHub Actions however there's no point submitting your PR if the CI validation is going to fail, so get it working locally first, the main checks to run locally are:

make tools, make test, make lint, make tflint, make depscheck

These scripts can take a little while to run and tend hit your machine quite hard, but once they are passing, you can finally submit your PR! Sit back and wait for the team to check it over, and get back to your with feedback. Be patient they are a very busy team!

Conclusion

PHEW! What a journey, but hopefully like Dante's Virgil you made it through unscathed! Undoubtedly you learned a lot in the process. Contributing to the Azure Terraform Provider isn't something you do lightly, but can be very rewarding.

Your addition can make Terraform just a little bit better for everyone using Terraform with Azure, and for that reason alone it's sometimes worth going on a bit of an adventure

Terraform Bootstrap & Backend State in Azure

Ben Coleman — Tue, 01 Dec 2020 16:54:32 +0000

Introduction

When starting a new project utilising Terraform to manage resources in Azure, there's usually a hurdle to overcome, where you need to bootstrap your environment with certain pre-reqs. The main one of those being a way hold shared state (what Terraform calls a backend)

To this end I'd like to share a set of scripts & Terraform to help you though this initial bootstrap & setup process

benc-uk / terraform-mgmt-bootstrap

Bootstrap core Azure state & resources using Terraform for use with Azure DevOps

The repo holds some reusable scripts and Terraform configuration to "bootstrap" a project in order to be able to start using Terraform with Azure. This paves the way for the set up of Azure DevOps Pipelines to deploy further resources. This aligns with the common "hub & spoke" style of Azure architecture. where one shared set of management resources are used to support the deployment and management of one or more spokes through automated CI/CD pipelines.

In this case Azure DevOps is the CI/CD system we will be using & configuring

These "spokes" could be separate subscriptions or simply multiple environments in different resource groups for hosting simultaneous instances of an app.

Note. There is no dependency or relation to the hub & spoke network topology

There's four main parts setup up by this process:

Bootstrap of backend state in Azure Storage for all Terraform to use.
Deployment (and redeployment) set of shared, management resources.
Creation of service principals with role assignments in Azure AD.
Initial configuration of Azure DevOps

All of the scripts in the repo are intended to be run manually and infrequently, and called from an administrators local machine or Azure Cloud Shell. There is no automation or CI/CD, this is by design - the purpose of this is to provide the bedrock to allow further CI/CD to happen.

Note. This article is not intended as an intro to using Azure and Terraform, so it assumes a fair degree of knowledge in this area.

Pre-reqs

Bash
Terraform 0.13+
Azure CLI
Authenticated connection to Azure, using Azure CLI, logged into the subscription you wish to configure
Azure DevOps organization and project
An Azure DevOps PAT token (full scope) for the relevant Azure DevOps organization

Configuration

Before running any of the scripts, the configuration and input variables need to be set. This is done in an .env file, and this file is read and parsed by scripts

Note. .tfvars file is not used, this is intentional. The dotenv format is easier to parse, meaning we can use the values in bash scripts and for other purposes

Copy the .env.sample file to .env and set values for all variables as follows:

TF_VAR_state_storage - The name of the storage account to hold Terraform state.
TF_VAR_mgmt_res_group - The shared resource group for all hub resources, including the storage account.
TF_VAR_state_container - Name of the blob container to hold Terraform state (default: tfstate).
TF_VAR_prefix - A prefix added to all resources, pick your project name or other prefix to give the resources unique names.
TF_VAR_region - Azure region to deploy all resources into.
TF_VAR_azdo_org_url - URL of Azure DevOps org to use, e.g. https://dev.azure.com/foobar
TF_VAR_azdo_project_name - Name of the Azure DevOps project in the above org.
TF_VAR_azdo_pat - Azure DevOps access token with rights to create variable groups and service connections.

Bootstrap of Backend State

As a principal we want all our resources defined in Terraform, including the storage account using by Terraform to hold backend state. This results in a chicken and egg problem.

To solve this a bootstrap script is used which creates the initial storage account and resource group using the Azure CLI. Then Terraform is initialized pointing at this storage account as a backend, and the storage account imported into state

See bootstrap.sh in the GitHub repo for details

This script should never need running a second time even if the other management resources are modified

Management Resource Deployment

The deployment of the rest of the shared management resources is done via Terraform, and the various .tf files in the root of the repo.

See deploy.sh in the GitHub repo for details

This Terraform creates & configures the following:

Resource Group (also in bootstrap).
Storage Account for holding Terraform state (also in bootstrap).
Azure Container Registry. Not used but intended for providing centralized access to containers.
Azure Log Analytics. Not used but intended for log aggregation
Key Vault* for holding credentials and secrets used by Azure DevOps Pipelines.
Service Principal, with RBAC role assignment to access the KeyVault "Key Vault Reader (preview)".
A second Service Principal (to be used for pipelines), with IAM role "Contributor" at the subscription level.
KeyVault access policy for above Service Principal to allow it to get & list secrets.
KeyVault access policy for the current user to be able to manage secrets.
Populates KeyVault with secrets, holding the credential details for the pipeline service principal

You may wish to modify the role assigned to the pipeline service principal

Azure DevOps

The deployment Terraform also sets up some initial configuration in Azure DevOps namely service connection called keyvault-access which will allow variable groups to be linked to the KeyVault.

The creation of a Azure DevOps variable group linked to KeyVault can not be done via Terraform or the Azure CLI. A work-around using cURL and REST API has been used. This is some what brittle but it serves the purpose well enough.

See azdo-var-group.sh in the GitHub repo for details

Running this script will create a variable group called shared-secrets in the Azure DevOps project and populate it with four variables

pipeline-sp-clientid
pipeline-sp-secret
azure-tenant-id
azure-sub-id

These variables can be used in subsequent Azure DevOps pipelines.

Next Steps & Example Environment

This repo is intended to lay the ground work for Azure DevOps pipelines to be set up to deploy further resources. The shared variable group is a key part of enabling this, but the configuration of those pipelines is something clearly project & environment specific, so is not covered further here.

A working CD pipeline with demo Terraform is given in the example/ directory. This environment is nothing more than a resource group & a storage account for illustrative purposes. The focus is the pipeline file - example/deploy.yaml this carries out the deployment as follows:

Uses the above shared-secrets variable group
Runs using the service principal details held in Key Vault
Keeps state in the management backend state storage account
Carries out standard Terraform init / plan / apply

Refer to the example/deploy.yaml file for further details

If you are using a mono-repo, the whole of this repo can be dropped in as a sub-folder, in order to keep the Terraform separate from any Terraform you wish to use in your other pipelines.

Calling Azure APIs with the REST Extension for VS Code

Ben Coleman — Wed, 18 Nov 2020 23:43:49 +0000

The fantastic REST Client for VS Code is a popular and valuable tool when doing any work with a REST API, be it your own or a 3rd party. Using this extension you can define a set of HTTP calls in a .http or .rest file. The list of features it supports is impressive, you can use variables, chain calls together and build up a reference set of API calls which you can reuse and refer back to, or distribute with your code

Get the extension here
https://marketplace.visualstudio.com/items?itemName=humao.rest-client

Huachao / vscode-restclient

REST Client Extension for Visual Studio Code

Set up & Pre-reqs

When calling any Azure API you clearly need to authenticate, to cut a VERY long story short, this means getting an access token.

We'll use the common authentication scenario using a registered client, which is a Azure service principal plus the 'non-interactive flow' to request a token

The REST extension can help us get this token. Yay.

First create a service principal and give it the permissions to the Azure subscription you want to use.

Create a .env file and place the service principal details into it, this lets us keep secrets out of source control (I mean, you have .env included in your .gitignore of course)

AZURE_SUBSCRIPTION_ID="__YOUR_SUBSCRIPTION_ID__"
AZURE_TENANT_ID="__YOUR_TENANT_ID_ID__"
AZURE_CLIENT_ID"=__YOUR_CLIENT_ID__"
AZURE_CLIENT_SECRET="__YOUR_CLIENT_SECRET__"

Requesting an Access Token

In VS Code, create a new file with any name, but it should have .http or .rest as the file extension (like azure-api.rest) this file extension is what activates the REST Extension for VS Code.

Paste in the following contents

### Get access token to call Azure ARM API
# @name getToken 
POST https://login.microsoftonline.com/{{$dotenv %AZURE_TENANT_ID}}/oauth2/token
Content-Type: application/x-www-form-urlencoded

grant_type=client_credentials
&resource=https://management.azure.com/
&client_id={{$dotenv %AZURE_CLIENT_ID}}
&client_secret={{$dotenv %AZURE_CLIENT_SECRET}}

### Capture access token from getToken request
@authToken = {{getToken.response.body.access_token}}

The {{$dotenv %FOO}} syntax is some of the magic in the REST extension to access variables from the .env file

A "Send Request" option should light up above the POST statement, click that to make the request and you should get a pane popup with the response containing the access token (and other details)

It will store the access_token from the HTTP result into the authToken variable, to be available to subsequent requests

Calling the Azure APIs

Now you're set up to add more requests to the file and use the access token to call any Azure API

For example to list all resource groups, paste the following after the code above. Note. The comment line starting with three hashes ### is important and what the REST extension uses to delimit requests. We plug the authToken variable value into the Authorization header to authorise the request

### List all storage accounts for subscription
GET https://management.azure.com/subscriptions/{{$dotenv %AZURE_SUBSCRIPTION_ID}}/providers/Microsoft.Storage/storageAccounts?api-version=2019-06-01
Authorization: Bearer {{authToken}}

One again, hit "Send Request" just above the GET and you should get a reply from Azure listing all your resource groups. Neat!

Summary

The REST Client for VS Code makes authentication to call Azure APIs a simple and reusable task. Then calling those APIs can be easily set up without ever needing to leave your IDE