Terraform module for custom Azure policies

Isabel Andrade — Wed, 02 Dec 2020 15:46:14 +0000

Terraform modules are used to create reusable components that include groups of resources meant to be deployed together. This is a natural fit for custom Azure policies and initiatives since it allows organizations to implement all those definitions in a centralized component.

This post goes through an example that shows how to implement and test a Terraform module that defines custom Azure policies and initiatives. The full version of the code can be found in the following repo:

beandrad / terraform-azurerm-policy-sample

Minimal Terraform module defining Azure policies and initiatives

Module structure

Hashicorp published a set of guidelines on module definitions stating some naming conventions and the general module structure.

According to those guidelines, module repository names follow the format terraform-<PROVIDER>-<NAME>, where PROVIDER in our case is AzureRM and NAME is a label describing the infrastructure provided. In our case, the Terraform module is called terraform-azurerm-policy-sample.

The module structure follows the standard practices recommended by Hashicorp; note, however, that some of the paths defined are specific to this module.

main.tf. Defines the configuration requirements. It may also configure other resources.
variables.tf. Declares the module input variables.
outputs.tf. Declares the module outputs; in this case, the initiative IDs.
/policies. (Module specific) Includes the definitions of custom Azure policies. Each policy has its own folder, where the file policy-rule.json has the definition and policy-parameters.json, the parameters if applicable.
<label>-initiative.tf. (Module specific) Configures the definition of policies and initiatives in Azure. Policies are grouped into initiatives based on the resources they affect and/or by industry-level standards (such as the CIS Hardening Guidelines).
initiative-parameters.tf. (Module specific) Declares the input parameters of all the initiatives defined in the module.
/tests. Implements the module acceptance tests.

Definition of policies and initiatives

This module defines custom policies and initiatives under a management group (definition_management_group in variables.tf) or current subscription if the management group name is not defined.

Custom policy definitions are created using the azurerm_policy_definition resource and built-in policies are imported using the azurerm_policy_definition data resource. Both resources are included in the corresponding initiatives Terraform configuration files; unless they are shared across initiatives, in which case they are defined in the main.tf file.

It is important to note that policy data resources should be imported using its policy name (as opposed to the displayName). The reason is that the displayName is not unique and it may change, whereas the name is unique and it remains the same until the policy is deleted. In the configuration, the displayName appears commented out as it describes the policy being imported.

Acceptance tests

The acceptance tests deploy resources to Azure to check whether the defined initiatives actually work: non-compliant resources cannot be deployed whereas compliant ones are allowed to be deployed.

The following conventions were followed when testing the policy module:

Tests check that the initiative is working as expected, as opposed to testing individual policies. The reason is that this module only outputs initiatives, all the policies are linked from an initiative.
Only the behavior of custom policies is tested; built-in policies are expected to work.

Tests are implemented using the Go testing framework together with the Terratest module. This configuration allows calling the Terraform configuration from the Go tests.

The lifecycle of the tests is as follows:

Setup: load the policy module to define policies and initiatives and assign initiatives in Azure (tests/terraform/main.tf).
Run: try to create compliant resources (for example, tests/terraform/resource-location-allow) and non-compliant resources (for example, tests/terraform/resource-location-audit).
Assert: check whether the policy has been correctly applied using the returned error from the Terraform apply for deny effect or the policy state for audit effect.
Teardown: delete test resources from Azure.

Running this kind of tests is slow, in particular, those checking effects other than deny. There are two main components that cause this delay: the first one is policy definition and assignment, and the second one is policy evaluation (which, as stated above, is required to check the audit effect).

In order to speed up the tests, test cases are run in parallel using the Parallel() function.

Finally, regarding the teardown, it is important to note that it is done in two different steps: one for the resources provisioned during the setup and the other for the resources deployed by each of the test cases. In the first case, the Cleanup function is used; defer wouldn't work since deferred functions are run before parallel subtests are executed. On the other hand, resources created by the test cases are destroyed in a deferred function.

Final thoughts

As stated in the beginning, the aim of this post is to provide a baseline example of an Azure policy module; however, it is important to point out that quite a few opinionated design decisions have been made and, therefore, this example shouldn't be taken as the only correct way of implementing an Azure policy module.

Having said that, I hope this post helps those devs out there looking for some guidance on how to implement Terraform modules, define Azure custom policies and initiatives, and test those definitions.

Bypassing policies in Azure

Isabel Andrade — Thu, 19 Nov 2020 09:13:24 +0000

Azure policies allow your Azure infrastructure to stay compliant by auditing and enforcing rules over the resources those policies are evaluated against.

In some instances, however, we need to be able to bypass a particular policy or initiative. One way of doing this is by using bypass tags.

Bypass tags can be defined in custom policies so that if a particular tag is included in a resource, the policy is ignored. This approach has two main shortcomings:

It only applies to custom policies; if a built-in policy wants to be bypassed, a new custom policy that also includes the corresponding tag needs to be defined;
auditing of bypass tags requires further configuration since it is not integrated into the Azure portal out of the box.



{
    "if": {
        "allOf": [
            {
                "not": {
                    "field": "location",
                    "in": "[parameters('allowedLocations')]"
                }
            },
            {
                "field": "tags['bypassLocationPolicy']",
                "exists": false
            }
        ]
    },
    "then": {
        "effect": "deny"
    }
}

However, there’s a better way of bypassing policies: Azure policy exemptions.

What is a policy exemption?

Policy exemptions allow bypassing policies in a particular scope so that the policies are not evaluated for the resources under this scope.

As opposed to bypass tags, Azure automatically audits policy exemptions, so that they can be easily monitored as shown in the picture below.

Another advantage is that policy exemptions can be applied to any policy including build-in policies, without the need of defining additional policies.

One of the issues with policy exemptions, as opposed to bypass tags, is that it is involved to exempt individual resources from policies with deny effect. The reason for that is that in order to be able to apply a policy exemption, the exemption scope, the resource itself in this case, needs to exist. One way of achieving this result is by assigning the exemption to a broader scope, for example, a resource group, create the resource, then assign the exemption to the individual resource and remove it from the broader scope.

Limiting the scope of exemptions

As mentioned above, policy exemptions are applied to a particular scope. This, in principle, would allow disabling exemptions for entire management groups or subscriptions, which may defeat the purpose of applying policies in the first place.

However, there’s one way we can limit the scope to which exemptions can be applied: through a custom policy!

The idea of this custom policy would be to block the creation of policy exemptions with particular scopes. There’s one minor issue, however, the policy exemption scope doesn’t exist as an alias, and therefore, it cannot be used as a filter in the policy definition. The solution to this is to use the policy exemption ID, which happens to embed the scope. In our case, we wanted to block policy exemptions with scopes broader than the resource group level, so our policy definition looks as follows.



{

    "if": {

        "allOf": [

            {

                "field": "type",

                "equals": "Microsoft.Authorization/policyExemptions"

            },

            {

                "field": "id",

                "notContains": "resourceGroup"

            }

        ]

    },

    "then": {

        "effect": "deny"

    }

}

Follow up

At the time of writing this post, policy exemptions are still a preview feature in Azure and therefore you may face some limitations when using them. Nevertheless, I invite you to try them out if you need to work in highly regulated environments (they have proven quite handy to us!). Please, find below some links that will help you get started!

DEV Community: Isabel Andrade