DEV Community: CSE Dev Crews

Secure Azure as Code

Andrew Fryer — Thu, 10 Dec 2020 12:44:46 +0000

The Problem

Large organisations will typically have many, sometimes hundreds of Azure subscriptions. The challenge here is to secure and manage without slowing down the agility the cloud offers. What's need is a process to be able to stamp out subscriptions with security and monitoring baked in using devops pipelines.

Who we are

If you have an Azure problem, if no one else can help, and if you can engage them, maybe you can hire a dev crew (after the A-Team)

Our dev crew were recently tasked to make this a reality using our twin philosophies of "engineering excellence" and "code with". Code with means we work alongside our customers in a joint team pair programming against an agreed backlog. Engineering excellence is all about details:

use of pipelines to deploy any code
use PR with at least to reviewers
solid documentation
testing rigor in this case with Terratest and Go .. and many more!

Design Goals

We had the following design goals:

if the rules change we wanted the revised rules to be applied to all subscriptions not just new ones
Everything can be deployed from a pipeline
A mechanism to test the rules work by applying them to a test subscription
Default logging to be enabled for all resources

The Solution

A central management subscription with
- a key vault for tenant-based secrets
- a central container registry
- a central key vault for tenant based secrets about any of the managed subscriptions
- Security Center to show alerts across the tenant
Each managed subscription would just have a small set of default resources:
- key vault
- log storage
- custom roles
- a budget

We used several repos for this:

Tenant with the terraform to create the management subscription
Subscription with the terraform to create a subscription
Policies all policies were described in Terraform in a standard way and put in here.
Custom Roles. Azure Custom roles described in Terraform are stored here
Docs. We wanted to control the approval of designs and architecture so we used a PR process to control docs going into the main branch of this repo.

We used a number of pipelines in the project:

CI/CD to create the management subscription
CI to create a managed subscription for testing
CD to create a managed subscription in live which would be called from a Jira ticket via an Azure function
CI/CD for applying policy changes to the Management Group

Technology

We meet the customer where they are and use whatever tooling they are comfortable the only constraint being we only work on Azure projects. In this case we did our CI/CD in Azure devops but in such a way that all the flow control was handled in bash to make it as easy as possible to port pipelines to other tooling. The customer was already using Terraform for IaC.

The key to securing Azure is Policies. These can be grouped into Blueprints (if you're using ARM) or Initiatives which work for Terraform - for example we created one Policy initiative for the CIS benchmarks.Subscriptions can also be grouped into Management Groups both of which make all of this a lot simpler.

We also allowed users to create Policy Exceptions for example a contributor might want to create a VM without an encrypted disk, but if they do this then it'll be flagged in Security Center and they'll have to justify this to their security team.

Conclusion

We took 10 sprints to get this done and documented and then took two weeks out to share what we learnt more widely. For sharing we have contributed to the Terraform AzureRM provider, to the Cloud Adoption Framework and have written up the more interesting parts of this project:

Contributing to the Azure Terraform Provider

Ben Coleman — Wed, 02 Dec 2020 17:21:11 +0000

Ok so you're using Terraform to deploy and manage your Azure resources? It's pretty neat, but at some point it's likely you will stumble over an Azure resource or configuration that's not available in Terraform. The Terraform Provider for Azure covers a huge number of resources in Azure, but there's always going to be gaps, given how quickly Azure evolves.

So what can you do? Well, you can find a workaround (like using Azure CLI or even ARM Templates), or raise an issue on the Azure Provider GitHub project and hope some kind soul picks it up, or <deep intake of breath> roll up your sleeves, open your IDE of choice (VS Code of course!) and start work on adding yourself.

In this post I'll talk through some of the planning, pre-work and other technical investigation you're going to need to do, before you go diving headfirst into the code base. This is a replay of my own journey of adding to the provider, and trust me this isn't something you want to start doing without more than a little planning.

Basics & Pre-reqs

In Dante's Inferno, he describes hell having nine different layers or circles. Thankfully we have just five to worry about, however the amount of suffering involved might be the same as a trip through Dante's vision of unending torment.

Our layers are a little different, but no less infernal 😉

The Azure REST API Spec - This repo contains OpenAPI specifications for all the Azure APIs. These specs are OpenAPI/Swagger definitions which describe shape of every API in Azure. Despite being at the top of this stack, most of the time you can pretty much ignore this, unless you think you've hit a bug with the way the API is behaving.
The Azure REST API - This is the main management interface to Azure and implements the above API spec. This is going to dictate & shape everything else we touch.
Azure SDK for Go - This wraps the above Azure API with a set of packages & client code making it "easier" than calling the API directly. Easier is matter of some debate.
Terraform Provider for Azure - Also called 'azurerm' (which is the provider name), this in turn wraps the Go SDK, with a set of CRUD operations that Terraform understands.
Terraform - The Azure provider is a plugin and extension to the core Terraform system

Needless you say you're going to need some Terraform experience, at least with the basics. It's also worth reading the docs on provider plugins to get familiar with the concepts and main interfaces around providers.

You're also going to need to be able to program in Go. You won't need to be a Go uber-coder, but if you've never touched it, you're going to have a very rough time, this is not the sort project in which I would advise you start learning Go with.

It also helps if you've some prior experience with the Azure REST APIs (also called the ARM APIs), such as how they are versioned, the sorts of operations you can perform and concepts such as authentication, resource providers, scopes etc.

So brave traveller, are you ready to descend?

The Azure API

First you need to identify the API or APIs in Azure that are going plug the gap you need. This might be something you can deploy or configure from the Azure portal, the CLI but not through Terraform. You're going to spend a lot of time in this section of the Azure docs https://docs.microsoft.com/en-us/rest/api/azure/

Let's assume you find what you need (if you didn't your journey has been a short one!)

Take plenty of time to research the API, try it out & experiment with it. My suggestion is to use the fantastic REST Client for VS Code. Build up a set of calls in a .http or .rest file which you can refer back to

I wrote a short post on using this extension to authenticate and call Azure APIs, and how easy it makes it - so it's worth referring to that

Calling Azure APIs with the REST Extension for VS Code

Ben Coleman for CSE Dev Crews ・ Nov 18 '20

#azure #rest #vscode #api

Test the API thoroughly, time spent here will time well spent later. Poke at all the edge cases, "what if that array is empty?", "what if I put an invalid string here?", "if I remove that value, what is the default?". The docs will be some help, but generally are pretty bare bones.

Once you're happy that you've got good a handle on calling the API directly, it's time to move on to the next layer...

So traveller, I see you are prepared to descend deeper, let us journey on?

Build Go Prototype / Code Spike

The temptation might be to now jump into the Terraform provider code, but I'd hold off. Next it's worth spending some time working with the Azure SDK for Go to call the API in question

Azure / azure-sdk-for-go

This repository is for active development of the Azure SDK for Go. For consumers of the SDK we recommend visiting our public developer docs at:

The Go SDK is auto-generated from the aforementioned Azure REST API Spec, this means it's not exactly the most friendly SDK to work with. And with extremely rigid adherence to having everything well typed, means it can be laborious work to even see what the API is returning.

I would suggest using a code spike approach and build a simple console Go app, and in doing so take a look at:

Which package in the SDK you need, as they are versioned to match the Azure API versions, but it can be a challenge to even find the right package in the complex tree within the SDK.
How the API client in that package is instantiated, normally with a subscription ID and possibly location, but each client is different.
Make calls to the create, update and get operations. It's important you know how to build the input struct for a create operation, which might not be as trivial as it sounds. Also how to extract data from the response, which can often involve some "walking" down through all the properties and casting/asserting as you go

This might seem like busy work and a waste of time, but you'll learn a lot in doing this, and it's much easier to test at this point before needing to go through an entire Terraform plan & apply loop. In addition a lot of this code you'll be able to copy and paste into the provider code.

Are you confident to push on, to our final infernal layer?

The Azure Terraform Provider

The final part of the "journey", but there's still a long way to go - as this part will by far take the longest. I'm not going to go super deep into the low level code & implementation specifics in this section, that could fill another post (or in fact several!), instead I want to focus on the design, approach & thinking you'll need to do.

Start with designing the shape of your resource, sketch out some pseudo code HCL. By now you've probably got a good idea what this will look like based on the body of your API requests, but think about if you can make it a little more user friendly. For example where it comes to field names, think what does the end user likely know this field/feature by? There can often be a disconnect between the "internal" names used in the API and the "external" names used in the Portal & CLI

One area to consider is where the API accepts an array of objects, this can be simplified in the HCL as a repeated block with the same name, this will automatically become an array. Therefor you will need to switch from a pluralized name to a singular. For example

Example API JSON payload

"rules": [
  {
    "name": "rule 1"
  },
  {
    "name": "rule 2"
  }
]

Example equivalent Terraform HCL

resource azurerm_amazing_new_thing "example" {
  rule {
    name = "rule 1"
  }
  rule {
    name = "rule 2"
  }
}

Moving onward finally to the provider code itself which is hosted here:

hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager

You're contributing to a public open source project on GitHub, ultimately ending with a pull request which (hopefully!) will be merged into in main codebase and released. There's a certain way of working when contributing to OSS projects, there's plenty of guides online to help with this, e.g. first-contributions and MarcDiethelm/contributing

Take plenty of time to explore the codebase, but thankfully you don't need to understand it all, focus on digging into the azurerm/internal/services tree. Take some time to look at some Azure resources you are familiar with, try to find something not too complex (e.g. App Service) but not so simple it's trivial.

The resources are grouped by their Azure Provider API, and under each group there will be some shared code common to all resources in that package. The key ones being:

registration.go - This should be a short file containing a SupportedResources() function which maps the Terraform resource names e.g. azurerm_new_thing to the Go function that provides that resource, e.g resourceArmNewThing(). You should be able to copy and paste an existing line in here.
client/client.go - This is where all the API clients from the SDK are created for the given API package, really it's another case of copy and pasting an existing one as there's a clear pattern to things.

Implementing your resource will be a matter of copying an existing one and making a LOT of changes. There will be a set of callback functions for the CRUD operations Create, Read, Update and Delete, implementing those functions is the heart of the task at hand, and detailed here. Also the Schema is defined in here, this is the implementation of the HCL design we discussed a moment ago. You can read the docs also take a good look at how other resources define their schemas, and learn by example

Testing your code will take two shapes, informal testing locally with some Terraform HCL is the first place to start. In order to load your locally built version of the provider there's some hoops to jump though, which have been made a lot more difficult since Terraform 0.13.

The script below I created helps with this, it builds the azurerm code, and then copies the resulting provider binary to where the Terraform CLI can find it. Runs terraform init with -plugin-dir set and then does a standard Terraform plan & apply

Build provider & load plugin test harness script:

https://github.com/benc-uk/terraform-sandbox/blob/master/provider-test-harness/run.sh

Note. In the script I pick a deliberately out of range version for the plugin as a belt and braces to make sure my copy is really being picked up, so the version of azurerm in your test Terraform would need to be version = ">=99.0.0"

You'll run this inner loop test many, MANY times until you get things working. The next set of tests are more formal acceptance tests. These live in the tests folder of the tree you are working in. All acceptance tests start with TestAcc this means they won't run in the standard unit test cycle, they all contain embedded inline HCL which defines the test resources to create & destroy.

These tests are only run when certain flags are specified, namely calling make acctest. Again I suggest learning by example, copy some existing tests and build on what you find. Note these tests will deploy real resources in Azure which means they can be very slow to run and incur costs!

The final stage before submitting your PR is to run though the test/check suite locally. This is done automatically in the CI pipeline of the project in GitHub Actions however there's no point submitting your PR if the CI validation is going to fail, so get it working locally first, the main checks to run locally are:

make tools, make test, make lint, make tflint, make depscheck

These scripts can take a little while to run and tend hit your machine quite hard, but once they are passing, you can finally submit your PR! Sit back and wait for the team to check it over, and get back to your with feedback. Be patient they are a very busy team!

Conclusion

PHEW! What a journey, but hopefully like Dante's Virgil you made it through unscathed! Undoubtedly you learned a lot in the process. Contributing to the Azure Terraform Provider isn't something you do lightly, but can be very rewarding.

Your addition can make Terraform just a little bit better for everyone using Terraform with Azure, and for that reason alone it's sometimes worth going on a bit of an adventure

Test your Azure policies in parallel

Greg Oliver — Wed, 02 Dec 2020 15:46:50 +0000

This blog resulted from a customer development engagement. Want to read related blogs? Secure Azure as Code

Testing a policy is done with a few steps, each of which is a Terraform script:

Test the positive case, where the bad outcome the policy protects against is not challenged
Test the negative case, where the bad outcome is attempted (and hopefully audited or denied)
Both create and update use cases

If each test case requires creating a few Azure resources, the time to run these tests one after the other grows rapidly, especially in the case of testing policies. Having >100 policies to test is not unusual. Also, policies are normally defined at the management group level. This step can be done independently of the tests to set the context. Running the tests probably starts out by assigning policies to the test environment - a lower level management group or a subscription. Then the individual tests run on that test environment before it's reset with "terraform destroy".

Define policies and policy initiative in the management group

module "diagnostic_policies" {
  source                = "github.com/Nepomuceno/terraform-azurerm-monitoring-policies.git?ref=main"
  name                  = "DiagnosticsInitiative"
  management_group_name = var.definition_management_group
}

The Terraform script in the root folder of the github repo fully implements this step. It's done once before running any tests.

Run tests

In the tests folder of the github repo there are two approaches to this. One is implemented serially, the other in parallel. Both run the Terraform in the tests folder to do the policy assignment to the test environment. It also creates a couple of resources to use during tests that will follow.

Serial

func TestSerial(t *testing.T) {
    // setup the environment by assigning policy to the subscription
    options := &terraform.Options{
        TerraformDir: ".",
    }
    terraform.InitAndApply(t, options)
    defer terraform.Destroy(t, options)

    // run the first test
    t.Run("terraform_init_should_succeed", func(t *testing.T) {
        terraformTestOptions := &terraform.Options{
            TerraformDir: "./test-01",
    ... // and so on

Parallel

func TestParallel(t *testing.T) {
    // setup the environment by assigning policy to the subscription
    options := &terraform.Options{
        TerraformDir: ".",
    }
    terraform.InitAndApply(t, options)
    defer terraform.Destroy(t, options)

    t.Run("group", func(t *testing.T) {                 // <-- group the tests
        t.Run("terraform_init_should_succeed", func(t *testing.T) {
            t.Parallel()                                // <-- invoke Parallel()
            terraformTestOptions := &terraform.Options{
                TerraformDir: "./test-01",
    ... // and so on

Terraform module for custom Azure policies

Isabel Andrade — Wed, 02 Dec 2020 15:46:14 +0000

Terraform modules are used to create reusable components that include groups of resources meant to be deployed together. This is a natural fit for custom Azure policies and initiatives since it allows organizations to implement all those definitions in a centralized component.

This post goes through an example that shows how to implement and test a Terraform module that defines custom Azure policies and initiatives. The full version of the code can be found in the following repo:

beandrad / terraform-azurerm-policy-sample

Minimal Terraform module defining Azure policies and initiatives

Module structure

Hashicorp published a set of guidelines on module definitions stating some naming conventions and the general module structure.

According to those guidelines, module repository names follow the format terraform-<PROVIDER>-<NAME>, where PROVIDER in our case is AzureRM and NAME is a label describing the infrastructure provided. In our case, the Terraform module is called terraform-azurerm-policy-sample.

The module structure follows the standard practices recommended by Hashicorp; note, however, that some of the paths defined are specific to this module.

main.tf. Defines the configuration requirements. It may also configure other resources.
variables.tf. Declares the module input variables.
outputs.tf. Declares the module outputs; in this case, the initiative IDs.
/policies. (Module specific) Includes the definitions of custom Azure policies. Each policy has its own folder, where the file policy-rule.json has the definition and policy-parameters.json, the parameters if applicable.
<label>-initiative.tf. (Module specific) Configures the definition of policies and initiatives in Azure. Policies are grouped into initiatives based on the resources they affect and/or by industry-level standards (such as the CIS Hardening Guidelines).
initiative-parameters.tf. (Module specific) Declares the input parameters of all the initiatives defined in the module.
/tests. Implements the module acceptance tests.

Definition of policies and initiatives

This module defines custom policies and initiatives under a management group (definition_management_group in variables.tf) or current subscription if the management group name is not defined.

Custom policy definitions are created using the azurerm_policy_definition resource and built-in policies are imported using the azurerm_policy_definition data resource. Both resources are included in the corresponding initiatives Terraform configuration files; unless they are shared across initiatives, in which case they are defined in the main.tf file.

It is important to note that policy data resources should be imported using its policy name (as opposed to the displayName). The reason is that the displayName is not unique and it may change, whereas the name is unique and it remains the same until the policy is deleted. In the configuration, the displayName appears commented out as it describes the policy being imported.

Acceptance tests

The acceptance tests deploy resources to Azure to check whether the defined initiatives actually work: non-compliant resources cannot be deployed whereas compliant ones are allowed to be deployed.

The following conventions were followed when testing the policy module:

Tests check that the initiative is working as expected, as opposed to testing individual policies. The reason is that this module only outputs initiatives, all the policies are linked from an initiative.
Only the behavior of custom policies is tested; built-in policies are expected to work.

Tests are implemented using the Go testing framework together with the Terratest module. This configuration allows calling the Terraform configuration from the Go tests.

The lifecycle of the tests is as follows:

Setup: load the policy module to define policies and initiatives and assign initiatives in Azure (tests/terraform/main.tf).
Run: try to create compliant resources (for example, tests/terraform/resource-location-allow) and non-compliant resources (for example, tests/terraform/resource-location-audit).
Assert: check whether the policy has been correctly applied using the returned error from the Terraform apply for deny effect or the policy state for audit effect.
Teardown: delete test resources from Azure.

Running this kind of tests is slow, in particular, those checking effects other than deny. There are two main components that cause this delay: the first one is policy definition and assignment, and the second one is policy evaluation (which, as stated above, is required to check the audit effect).

In order to speed up the tests, test cases are run in parallel using the Parallel() function.

Finally, regarding the teardown, it is important to note that it is done in two different steps: one for the resources provisioned during the setup and the other for the resources deployed by each of the test cases. In the first case, the Cleanup function is used; defer wouldn't work since deferred functions are run before parallel subtests are executed. On the other hand, resources created by the test cases are destroyed in a deferred function.

Final thoughts

As stated in the beginning, the aim of this post is to provide a baseline example of an Azure policy module; however, it is important to point out that quite a few opinionated design decisions have been made and, therefore, this example shouldn't be taken as the only correct way of implementing an Azure policy module.

Having said that, I hope this post helps those devs out there looking for some guidance on how to implement Terraform modules, define Azure custom policies and initiatives, and test those definitions.

Auto-start collection of Azure diagnostic telemetry

Greg Oliver — Wed, 02 Dec 2020 13:20:43 +0000

This blog resulted from a customer development engagement. Want to read related blogs? Secure Azure as Code

Infrastructure and platform monitoring are the province of Azure Monitor. The documentation provides several ways to configure diagnostic profiles on your resources so that the telemetry flows, but doing it onesy-twosy is a tax. Using Azure Policy together with an open source Terraform module allows this task to fade into the background - done and dusted.

Define policies and policy initiative in the management group

module "diagnostic_policies" {
  source                = "github.com/Nepomuceno/terraform-azurerm-monitoring-policies.git?ref=main"
  name                  = "DiagnosticsInitiative"
  management_group_name = var.definition_management_group
}

The Terraform script in the root folder of the github repo fully implements this step.

Assign policy initiative to your subscription

resource "azurerm_policy_assignment" "diagnostics_initiative" {
  name                 = "diagnostics-initiative-assignment"
  scope                = data.azurerm_subscription.current.id
  policy_definition_id = data.azurerm_policy_set_definition.DiagnosticsInitiative.id
  description          = "Policy Assignment created via terraform"
  display_name         = "Unit test diagnostic Logs application"
  identity {
    type = "SystemAssigned"
  }
  location = "uksouth"

  metadata   = <<METADATA
    {
    "category": "Logs"
    }
METADATA
  parameters = <<PARAMETERS
{
  "workspaceId": {
    "value": "${azurerm_log_analytics_workspace.ws.id}"
  },
  "storageAccountName": {
    "value": "${azurerm_storage_account.storage.name}"
  }
}
PARAMETERS
}

Give the SystemAssigned identity defined above rights to remediate

resource "azurerm_role_assignment" "contributor" {
  scope                = data.azurerm_subscription.current.id
  role_definition_name = "contributor"
  principal_id         = azurerm_policy_assignment.diagnostics_initiative.identity[0].principal_id
}

Scenario 01 in the scenarios folder fully implements these steps. Test by using a subscription that already has resources in it that do not have diagnostic profile(s) defined, or create new resources after running the above. Scenarios 02 and 03 are provided for convenience. They are dependent on scenario 01, so run scenario 01 first, then 02 or 03, etc.

Explanation and Background

Azure Monitor diagnostic profiles tell the platform what information you want to collect and where to send it. "What information you want to collect" is different for each resource type and there is no "everything" button. And that's the rub - one must make decisions for each resource in the system architecture. Using the module makes it possible to encapsulate all of this into one easy step in the Terraform script.

A project frequently equates to an Azure subscription, and there are usually multiple projects within an organization. Managing the resources in these projects is made more convenient through the use of Management Group and Azure Policy. From this page:

Azure management groups provide a level of scope above subscriptions. You organize subscriptions into containers called "management groups" and apply your governance conditions to the management groups. All subscriptions within a management group automatically inherit the conditions applied to the management group. Management groups give you enterprise-grade management at a large scale no matter what type of subscriptions you might have. All subscriptions within a single management group must trust the same Azure Active Directory tenant.

Using the module, many Azure policies and a policy initiative are defined (usually) at the management group level. Because each project team monitors their own infrastructure and each diagnostic profile designates the destination of the telemetry, it's best to assign the policy initiative at the subscription level. This allows the telemetry collection point to be within that subscription. It is possible to create multiple diagnostic profiles per resource so that telemetry can be directed to a global management point if desired.

Each policy defined by the module targets a resource type, such as network security group, virtual machine, network interface card, and so on. If a resource does not meet the requirements of the policy, a remediation task creates the diagnostic profile for the resource. When the module is assigned to the subscription (via terraform apply) the results are not instantaneous - the Azure policy engine may take up to 15 minutes to scan resources and run remediation tasks.

Terraform Bootstrap & Backend State in Azure

Ben Coleman — Tue, 01 Dec 2020 16:54:32 +0000

Introduction

When starting a new project utilising Terraform to manage resources in Azure, there's usually a hurdle to overcome, where you need to bootstrap your environment with certain pre-reqs. The main one of those being a way hold shared state (what Terraform calls a backend)

To this end I'd like to share a set of scripts & Terraform to help you though this initial bootstrap & setup process

benc-uk / terraform-mgmt-bootstrap

Bootstrap core Azure state & resources using Terraform for use with Azure DevOps

The repo holds some reusable scripts and Terraform configuration to "bootstrap" a project in order to be able to start using Terraform with Azure. This paves the way for the set up of Azure DevOps Pipelines to deploy further resources. This aligns with the common "hub & spoke" style of Azure architecture. where one shared set of management resources are used to support the deployment and management of one or more spokes through automated CI/CD pipelines.

In this case Azure DevOps is the CI/CD system we will be using & configuring

These "spokes" could be separate subscriptions or simply multiple environments in different resource groups for hosting simultaneous instances of an app.

Note. There is no dependency or relation to the hub & spoke network topology

There's four main parts setup up by this process:

Bootstrap of backend state in Azure Storage for all Terraform to use.
Deployment (and redeployment) set of shared, management resources.
Creation of service principals with role assignments in Azure AD.
Initial configuration of Azure DevOps

All of the scripts in the repo are intended to be run manually and infrequently, and called from an administrators local machine or Azure Cloud Shell. There is no automation or CI/CD, this is by design - the purpose of this is to provide the bedrock to allow further CI/CD to happen.

Note. This article is not intended as an intro to using Azure and Terraform, so it assumes a fair degree of knowledge in this area.

Pre-reqs

Bash
Terraform 0.13+
Azure CLI
Authenticated connection to Azure, using Azure CLI, logged into the subscription you wish to configure
Azure DevOps organization and project
An Azure DevOps PAT token (full scope) for the relevant Azure DevOps organization

Configuration

Before running any of the scripts, the configuration and input variables need to be set. This is done in an .env file, and this file is read and parsed by scripts

Note. .tfvars file is not used, this is intentional. The dotenv format is easier to parse, meaning we can use the values in bash scripts and for other purposes

Copy the .env.sample file to .env and set values for all variables as follows:

TF_VAR_state_storage - The name of the storage account to hold Terraform state.
TF_VAR_mgmt_res_group - The shared resource group for all hub resources, including the storage account.
TF_VAR_state_container - Name of the blob container to hold Terraform state (default: tfstate).
TF_VAR_prefix - A prefix added to all resources, pick your project name or other prefix to give the resources unique names.
TF_VAR_region - Azure region to deploy all resources into.
TF_VAR_azdo_org_url - URL of Azure DevOps org to use, e.g. https://dev.azure.com/foobar
TF_VAR_azdo_project_name - Name of the Azure DevOps project in the above org.
TF_VAR_azdo_pat - Azure DevOps access token with rights to create variable groups and service connections.

Bootstrap of Backend State

As a principal we want all our resources defined in Terraform, including the storage account using by Terraform to hold backend state. This results in a chicken and egg problem.

To solve this a bootstrap script is used which creates the initial storage account and resource group using the Azure CLI. Then Terraform is initialized pointing at this storage account as a backend, and the storage account imported into state

See bootstrap.sh in the GitHub repo for details

This script should never need running a second time even if the other management resources are modified

Management Resource Deployment

The deployment of the rest of the shared management resources is done via Terraform, and the various .tf files in the root of the repo.

See deploy.sh in the GitHub repo for details

This Terraform creates & configures the following:

Resource Group (also in bootstrap).
Storage Account for holding Terraform state (also in bootstrap).
Azure Container Registry. Not used but intended for providing centralized access to containers.
Azure Log Analytics. Not used but intended for log aggregation
Key Vault* for holding credentials and secrets used by Azure DevOps Pipelines.
Service Principal, with RBAC role assignment to access the KeyVault "Key Vault Reader (preview)".
A second Service Principal (to be used for pipelines), with IAM role "Contributor" at the subscription level.
KeyVault access policy for above Service Principal to allow it to get & list secrets.
KeyVault access policy for the current user to be able to manage secrets.
Populates KeyVault with secrets, holding the credential details for the pipeline service principal

You may wish to modify the role assigned to the pipeline service principal

Azure DevOps

The deployment Terraform also sets up some initial configuration in Azure DevOps namely service connection called keyvault-access which will allow variable groups to be linked to the KeyVault.

The creation of a Azure DevOps variable group linked to KeyVault can not be done via Terraform or the Azure CLI. A work-around using cURL and REST API has been used. This is some what brittle but it serves the purpose well enough.

See azdo-var-group.sh in the GitHub repo for details

Running this script will create a variable group called shared-secrets in the Azure DevOps project and populate it with four variables

pipeline-sp-clientid
pipeline-sp-secret
azure-tenant-id
azure-sub-id

These variables can be used in subsequent Azure DevOps pipelines.

Next Steps & Example Environment

This repo is intended to lay the ground work for Azure DevOps pipelines to be set up to deploy further resources. The shared variable group is a key part of enabling this, but the configuration of those pipelines is something clearly project & environment specific, so is not covered further here.

A working CD pipeline with demo Terraform is given in the example/ directory. This environment is nothing more than a resource group & a storage account for illustrative purposes. The focus is the pipeline file - example/deploy.yaml this carries out the deployment as follows:

Uses the above shared-secrets variable group
Runs using the service principal details held in Key Vault
Keeps state in the management backend state storage account
Carries out standard Terraform init / plan / apply

Refer to the example/deploy.yaml file for further details

If you are using a mono-repo, the whole of this repo can be dropped in as a sub-folder, in order to keep the Terraform separate from any Terraform you wish to use in your other pipelines.

Standardize resource names in Terraform scripts

Greg Oliver — Tue, 01 Dec 2020 15:08:29 +0000

This blog resulted from a customer development engagement. Want to read related blogs? Secure Azure as Code

When starting a Terraform project for an Azure architecture, it's easy to come up with useful names for the resources in your architecture. They usually look like "my-resource-group', "my-public-ip", "my-vm", "mystorageaccount", and so on. When the architecture grows, or elements of it scale out, it becomes harder to design useful names that meet all requirements. This blog is about a tool that helps with this task.

If you agree with the above and just want a link to the tool, here it is: terraform-provider-azurecaf.

Usage samples are fully implemented in the scenarios folder of the github repo.

Everyone else, read on.

Requirements that must be met

for each resource type, rules vary
- length
- accepted characters
- accepted patterns (e.g. first character must be a lowercase alpha)
It must be possible to override generated names in special cases
It must be possible to generate globally unique names
Generated names must behave like any other resource in tfstate
- names persist across 'terraform apply' runs as long as name resource definition remains the same
- name resources can be destroyed, tainted, etc just like any other terraform resource

Additional nice-to-have features

a generated name should conform to a regular pattern that becomes familiar and instantly recognizable
when there are many resources in list, it should be possible to instantly recognize resource types from the name
clear and concise name generation code
when an architecture grows or resources scale out horizontally, name generation follows naturally
when testing permutations of resource properties, generating names is a very powerful enabling technique

Solution to the problem

The solution is a Terraform provider that generates resource names. Unsurprisingly, it meets all of the conditions above. Generated resource name configuration options include (in order of precedence):

name - overrides other options
slug - a few characters denoting the resource type
random - randomly generated chars
suffixes - an array of suffixes that are appended
prefixes - an array of prefixes that are pre-pended

All configuration options are defined in the provider repo.

The following examples are fully implemented in the scenarios folder of the github repo.

Generates and implements a resource group name similar to rg-xxxxxxxx (very simple example)

resource "azurecaf_name" "rg_name" {
  resource_type = "azurerm_resource_group"

  random_length = 8
}

resource "azurerm_resource_group" "rg" {
  name     = azurecaf_name.rg_name.result
  location = "uksouth"
}

Generates and implements a log analytics workspace name (example of name override)

resource "azurecaf_name" "ws_name" {
  resource_type = "azurerm_log_analytics_workspace"

  random_length = 8
  suffixes      = ["local"]
  name          = substr(data.azurerm_subscription.current.display_name, 0, 44)

  # desired outcome = log-<sub name>-xxxxxxxx-local
  # length = 3 + 1 + (44) + 1 + 8 + 1 + 5 = max of 63 characters for log analytics workspace name
  # 44 is the max # of chars to get from the subscription name in order to get everything else into the generated name
  # because the "name" parameter is an override, if more characters are used, other portions of the generated name will be truncated
}

resource "azurerm_log_analytics_workspace" "ws" {
  name                = azurecaf_name.ws_name.result

  ...
}

The resource name rules (such as the max length of 63 characters) for azurerm_log_analytics_workspace are in this file.

Generates multiple singleton names with a single azurecaf_name resource

resource "azurecaf_name" "names" {
  resource_type  = "azurerm_resource_group"
  resource_types = ["azurerm_lb", "azurerm_public_ip"]
  random_length  = 4
}

resource "azurerm_resource_group" "rg" {
  name     = azurecaf_name.names.result
  location = "uksouth"
}

resource "azurerm_public_ip" "pip" {
  name                = azurecaf_name.names.results["azurerm_public_ip"]

  ...
}

resource "azurerm_lb" "lb" {
  name                = azurecaf_name.names.results["azurerm_lb"]

  ...
}

Generates a set of names per vm instance of a cluster of vms

resource "azurecaf_name" "per_instance" {
  count = var.number_of_servers

  resource_type  = "azurerm_windows_virtual_machine"
  resource_types = ["azurerm_network_interface"]

  name          = "websvr"
  random_length = 4

}

resource "azurerm_network_interface" "nic" {
  count = var.number_of_servers

  name                = azurecaf_name.per_instance[count.index].results["azurerm_network_interface"]

  ...
}

resource "azurerm_network_interface_backend_address_pool_association" "example" {
  count = var.number_of_servers

  network_interface_id    = azurerm_network_interface.nic[count.index].id

  ...
}

resource "azurerm_windows_virtual_machine" "vm" {
  count = var.number_of_servers

  name                = azurecaf_name.per_instance[count.index].result
  network_interface_ids = [
    azurerm_network_interface.nic[count.index].id,
  ]

  ...
}

Bypassing policies in Azure

Isabel Andrade — Thu, 19 Nov 2020 09:13:24 +0000

Azure policies allow your Azure infrastructure to stay compliant by auditing and enforcing rules over the resources those policies are evaluated against.

In some instances, however, we need to be able to bypass a particular policy or initiative. One way of doing this is by using bypass tags.

Bypass tags can be defined in custom policies so that if a particular tag is included in a resource, the policy is ignored. This approach has two main shortcomings:

It only applies to custom policies; if a built-in policy wants to be bypassed, a new custom policy that also includes the corresponding tag needs to be defined;
auditing of bypass tags requires further configuration since it is not integrated into the Azure portal out of the box.



{
    "if": {
        "allOf": [
            {
                "not": {
                    "field": "location",
                    "in": "[parameters('allowedLocations')]"
                }
            },
            {
                "field": "tags['bypassLocationPolicy']",
                "exists": false
            }
        ]
    },
    "then": {
        "effect": "deny"
    }
}

However, there’s a better way of bypassing policies: Azure policy exemptions.

What is a policy exemption?

Policy exemptions allow bypassing policies in a particular scope so that the policies are not evaluated for the resources under this scope.

As opposed to bypass tags, Azure automatically audits policy exemptions, so that they can be easily monitored as shown in the picture below.

Another advantage is that policy exemptions can be applied to any policy including build-in policies, without the need of defining additional policies.

One of the issues with policy exemptions, as opposed to bypass tags, is that it is involved to exempt individual resources from policies with deny effect. The reason for that is that in order to be able to apply a policy exemption, the exemption scope, the resource itself in this case, needs to exist. One way of achieving this result is by assigning the exemption to a broader scope, for example, a resource group, create the resource, then assign the exemption to the individual resource and remove it from the broader scope.

Limiting the scope of exemptions

As mentioned above, policy exemptions are applied to a particular scope. This, in principle, would allow disabling exemptions for entire management groups or subscriptions, which may defeat the purpose of applying policies in the first place.

However, there’s one way we can limit the scope to which exemptions can be applied: through a custom policy!

The idea of this custom policy would be to block the creation of policy exemptions with particular scopes. There’s one minor issue, however, the policy exemption scope doesn’t exist as an alias, and therefore, it cannot be used as a filter in the policy definition. The solution to this is to use the policy exemption ID, which happens to embed the scope. In our case, we wanted to block policy exemptions with scopes broader than the resource group level, so our policy definition looks as follows.



{

    "if": {

        "allOf": [

            {

                "field": "type",

                "equals": "Microsoft.Authorization/policyExemptions"

            },

            {

                "field": "id",

                "notContains": "resourceGroup"

            }

        ]

    },

    "then": {

        "effect": "deny"

    }

}

Follow up

At the time of writing this post, policy exemptions are still a preview feature in Azure and therefore you may face some limitations when using them. Nevertheless, I invite you to try them out if you need to work in highly regulated environments (they have proven quite handy to us!). Please, find below some links that will help you get started!

Calling Azure APIs with the REST Extension for VS Code

Ben Coleman — Wed, 18 Nov 2020 23:43:49 +0000

The fantastic REST Client for VS Code is a popular and valuable tool when doing any work with a REST API, be it your own or a 3rd party. Using this extension you can define a set of HTTP calls in a .http or .rest file. The list of features it supports is impressive, you can use variables, chain calls together and build up a reference set of API calls which you can reuse and refer back to, or distribute with your code

Get the extension here
https://marketplace.visualstudio.com/items?itemName=humao.rest-client

Huachao / vscode-restclient

REST Client Extension for Visual Studio Code

Set up & Pre-reqs

When calling any Azure API you clearly need to authenticate, to cut a VERY long story short, this means getting an access token.

We'll use the common authentication scenario using a registered client, which is a Azure service principal plus the 'non-interactive flow' to request a token

The REST extension can help us get this token. Yay.

First create a service principal and give it the permissions to the Azure subscription you want to use.

Create a .env file and place the service principal details into it, this lets us keep secrets out of source control (I mean, you have .env included in your .gitignore of course)

AZURE_SUBSCRIPTION_ID="__YOUR_SUBSCRIPTION_ID__"
AZURE_TENANT_ID="__YOUR_TENANT_ID_ID__"
AZURE_CLIENT_ID"=__YOUR_CLIENT_ID__"
AZURE_CLIENT_SECRET="__YOUR_CLIENT_SECRET__"

Requesting an Access Token

In VS Code, create a new file with any name, but it should have .http or .rest as the file extension (like azure-api.rest) this file extension is what activates the REST Extension for VS Code.

Paste in the following contents

### Get access token to call Azure ARM API
# @name getToken 
POST https://login.microsoftonline.com/{{$dotenv %AZURE_TENANT_ID}}/oauth2/token
Content-Type: application/x-www-form-urlencoded

grant_type=client_credentials
&resource=https://management.azure.com/
&client_id={{$dotenv %AZURE_CLIENT_ID}}
&client_secret={{$dotenv %AZURE_CLIENT_SECRET}}

### Capture access token from getToken request
@authToken = {{getToken.response.body.access_token}}

The {{$dotenv %FOO}} syntax is some of the magic in the REST extension to access variables from the .env file

A "Send Request" option should light up above the POST statement, click that to make the request and you should get a pane popup with the response containing the access token (and other details)

It will store the access_token from the HTTP result into the authToken variable, to be available to subsequent requests

Calling the Azure APIs

Now you're set up to add more requests to the file and use the access token to call any Azure API

For example to list all resource groups, paste the following after the code above. Note. The comment line starting with three hashes ### is important and what the REST extension uses to delimit requests. We plug the authToken variable value into the Authorization header to authorise the request

### List all storage accounts for subscription
GET https://management.azure.com/subscriptions/{{$dotenv %AZURE_SUBSCRIPTION_ID}}/providers/Microsoft.Storage/storageAccounts?api-version=2019-06-01
Authorization: Bearer {{authToken}}

One again, hit "Send Request" just above the GET and you should get a reply from Azure listing all your resource groups. Neat!

Summary

The REST Client for VS Code makes authentication to call Azure APIs a simple and reusable task. Then calling those APIs can be easily set up without ever needing to leave your IDE