DEV Community: Paul Delcogliano

How to Use Terraform Conditional Expressions

Paul Delcogliano — Tue, 14 Nov 2023 14:13:12 +0000

Conditional expressions are a key part of any programming language. Conditional expressions return a value based on whether an expression evaluates to true or false. In most modern languages, conditional expressions are represented by the if...else statement. Here is an example of a conditional expression: If this article is engaging, then people will continue reading it, else, no one will see it.

What is a conditional expression in terraform

Terraform doesn't offer the traditional if...else statement. Instead, it provides a ternary operator for conditional expressions. Conditional expressions in Terraform can be applied to myriad objects, including resources, data sources, outputs, and modules.

Conditional expressions provide flexibility and re-usability to Terraform configurations. They allow configurations to adapt to different environments, requirements, or scenarios.

What is the Terraform ternary operator

A ternary operator is one which operates on three operators. Syntactically, the ternary operator defines a Boolean condition, a value when the condition is true, and a value when the condition is false.

The ternary operator in Terraform looks like this:

condition ? true_part : false_part

The condition operand is any expression whose value resolves to a Boolean, like article == engaging. true_part is the value returned when the condition evaluates to true. false_part is the value when the condition evaluates to false.

Here is a basic example:

account_tier = var.environment == "dev" ? "Standard" : "Premium"

The previous ternary expression can be broken down like so:

Condition	?	true part	:	false part
If the `environment` variable is equal to "dev"	then	assign the value "Standard" to the account_tier attribute	else	assign "Premium"

The two result values, true_part, and false_part must both be the same data type, i.e., two strings. If the data types are different, terraform will attempt to convert them to a common type automatically. For example, terraform will automatically convert the result of the following expression to string since numbers can be converted to string:

count = var.allow_public == true ? 1 : "0"

While automatic data type conversion is a nice convenience, it should not be relied upon as it leads to configurations that are confusing and can be error-prone. Instead, explicitly convert data types to avoid automatic data type conversion:

count = var.allow_public == true ? 1 : tonumber("0")

The example illustrates the point but admittedly is a bit contrived.

Conditional expression use cases

A common use case for conditional expressions is to test for the existence of a variable's value and define a default value to replace invalid values:

var.environment == "" ? "dev" : var.environment

If the value of var.environment is an empty string then set its value to "dev", otherwise use the actual value of var.environment.

Conditional expressions are often used to configure settings differently based on certain conditions. In this example, a conditional expression configures an Azure storage account's access_tier attribute. If the var.environment value is "dev", the access tier will be set to "Cool". Otherwise, it will be "Hot".

resource "azurerm_storage_account" "my_storage" {
  name                            = "stmystorage"
  resource_group_name             = "rg-conditional-demo"
  location                        = "eastus"
  access_tier                     = var.environment == "dev" ? "Cool" : "Hot"
}

Create a resource using a conditional expression

By default, terraform creates one instance of a resource. Terraform's count meta-argument instructs terraform to create several similar objects without writing a separate block for each one. If a resource or module block includes a count argument with a whole number value, terraform creates that many instances of the resource. Setting count to zero results in no instances of the resource being created.

When combined with a conditional expression, count can be used to create powerful logic to control whether to create a resource. The following example evaluates the value of the add_storage_account Boolean variable. If it is true, count will be assigned 1. When this happens, an Azure storage account will be created. However, if add_storage_account is false, the count will be zero and no storage account will be created.

variable "add_storage_account" {
  description = "boolean to determine whether to create a storage account or not"
  type        = bool
}

resource "azurerm_storage_account" "my_storage_account" {
  count = var.add_storage_account ? 1 : 0

  resource_group_name      = "rg-conditional-demo"
  location                 = "eastus"
  account_tier             = "Standard"
  account_replication_type = "LRS"

  name = "stspacelift${count.index}${local.rand_suffix}"
}

Similar to count, terraform's for_each meta-argument is used to create many instances of the same resource. for_each works with a list of values to create resources with distinct arguments. The difference between the two meta-arguments is that count is best used when nearly identical resources need to be created. for_each is best for creating resources where some of the resources need distinct attribute values.

A typical use case for the for_each argument is to use a map of objects to assign multiple users to a group. A conditional expression can be added to filter out resources that should be added to a group based on their user type. This example shows one way to do that.

variable "users" {
  description = "A list of users to add"
  type = map(object({
    email = string,
    user_type = string
  }))
  default = {
    "member1" = {
      email = "member1@abc.com",
      user_type = "Member"
    },
    "member2" = {
      email = "member2@abc.com",
      user_type = "Member"
    },
    "guest1" = {
      email = "guest@abc.com",
      user_type = "Guest"
    }
  }
}

# Get the users from AAD
data "azuread_user" "my_users" {
  for_each = var.users


  user_principal_name = each.value.email
}

resource "azuread_group" "my_group" {
  display_name     = "mygroup"
  security_enabled = true
}

# Only add users who are members to the group
resource "azuread_group_member" "my_group_members" {
  for_each = { for key, val in data.azuread_user.my_users :
  key => val if val.user_type == "Member" }

  group_object_id  = azuread_group.my_group.id
  member_object_id = data.azuread_user.my_users[each.key].id
}

The users variable defines an object map, with each object having a property named "email". Three user objects are added to the map, two members and one guest. A data source is used to retrieve users from AAD. The for_each argument in the azuread_group_member resource loops through the users returned from AAD and uses a condition to apply a filter for users who are members. Each user in the filtered results will be added to the group named "my_group".

Conditional expressions on other Terraform objects

In addition to their application to resources, conditional expressions can be combined with count and for-each on the following terraform objects: module blocks, data sources, dynamic blocks, and local and/or output variables. The syntax is identical as shown for a resource block.

Object	Use Case
module block	control the creation and number of instances
data source	reduce number of records via filter
dynamic block	control the creation and number of instances
local variable	set variable values based on conditions
output variable	return values based on conditions

Here are some examples that use conditional expressions with count and for_each on a module block, a data source, a local variable, and an output variable.

# module examples
# determine if an account should be created
module "storage" {
  count = var.add_storage_account ? 1 : 0


  source = "./path to module tf file"
  ...
}

# filter list of users to add to a group
module "group_members" {
  for_each = { for key, val in data.azuread_user.my_users :
   key => val if val.user_type == "Member" }

  source = "./path to module tf file"
  ...
}

# data source example
# filter a data source using the `users` variable from above, looking for "members"
data "azuread_user" "my_users" {
  for_each = { for key, val in var.users :
   key => val if val.user_type == "Member" }

  user_principal_name = each.value.email
}

# local variable example
# uses a conditional expression to assign a value to the "rand_suffix" variable if the `add_storage_account` variable is true
locals {
    # "rand_suffix" can be appended to the storage account name.
    rand_suffix = var.add_storage_account ? ${random_string.random.result} : null
}

# output variable example
# return a storage account name, if an account was created. Empty string otherwise
output "storage_account_name" {
  value = var.add_storage_account ? azurerm_storage_account.my_storage_account[0].name : ""
}

Multiple Conditions

Complex logic can be created when conditional expressions are combined with Terraform's logical operators. Terraform provides the logical operators && (AND), || (OR), and ! (NOT).

This example combines two conditions using the and operator. In this case, if add_storage_account is true and environment equals "prod", two instances of the resource are created. Otherwise, none are created.

  count = var.add_storage_account && var.environment == "prod" ? 2 : 0

Conditional logic can also be nested. For instance, the true_part or false_part of the ternary operator could be another conditional expression. Converting the previous example, but replacing the logical and with nested logic would look like this:

  count = var.add_storage_account ? var.environment == "prod" ? 2 : 1 : 0

Here, the true_part is another condition, eg., does environment equal "prod". While the result is similar to the code using a logical and, the nested version is a bit harder to read and not as clean.

Wrapping it up

There are a few limitations to be aware of when using conditional expressions. While they can be applied to many object types, they cannot be applied to providers. count and for_each are mutually exclusive and cannot be used on the same object. Finally, while this won't affect many terraform implementations, it's important to note that module support was added for count and for_each in version 0.13. Both meta-arguments can be only applied to resource blocks in versions prior to 0.13.

As with all software development, conditional expressions have a few best practices to follow. Remember to avoid complex conditions. Nested conditions, while possible, add complexity to the configuration making it difficult to maintain and comprehend. Descriptive variable names facilitate the readability of the configuration. Also, be sure to test each conditional expression to ensure it works as intended.

Flexible configurations that adapt to different environments, requirements, and/or scenarios are possible with conditional expressions. Terraform's ternary operator is the main way to apply conditional logic. Ternary operators used on variables help set default and invalid values. Conditional expressions combined with count and for_each offer the ability to control whether a resource is created, and how many instances of a resource to create. They also allow for filtering data and configuring specific resource attributes.

Conditional expressions are easy to learn and implement and are another essential tool in any IaC toolbox.

Terraform vs. Pulumi : A Look at Both Tools

Paul Delcogliano — Sat, 08 Jul 2023 21:07:54 +0000

Cloud-native applications embrace the widely accepted practice of Infrastructure as Code, or IaC. IaC applies software development practices to the management of infrastructure, i.e., networks, virtual machines, and load balancers, etc.

Applying software development techniques to managing infrastructure provides several benefits, including versioning using a repository like GitHub, and streamlining provisioning and maintenance by creating repeatable processes. Code reviews and audits can be done to ensure changes to infrastructure are correct and won't introduce errors.

With IaC, infrastructure becomes reproducible in a consistent manner, and can be included in a CI/CD deployment pipeline. IaC is scalable and helps to eliminate human error.

Many IaC tools are available to facilitate the provisioning of cloud-native applications across all the major cloud platforms. Terraform and Pulumi are two such tools. In this article we'll take a look at both and provide a comparison between them.

Terraform

Terraform, is an open-source tool created by Hashicorp. It is probably the most well-known IaC infrastructure provisioning tool. Terraform is cloud-agnostic and allows automating infrastructure stacks from multiple cloud service providers simultaneously. Terraform uses its own domain-specific language called Hashicorp Configuration Language (HCL).

Terraform uses "plans" to validate the configuration and display exactly what elements are going to change before the changes are applied. For a detailed look at plans, check out the intro to Terraform plans article on spacelift.io.

Pulumi

Pulumi is an IaC tool that uses a declarative format to deploy infrastructure. Like Terraform, it is open source on GitHub and is free to use. It offers the ability to use any major development language in the creation of infrastructure template files. Pulumi, like Terraform, supports all of the major cloud providers.

Feature Comparison Between Terraform and Pulumi

Both tools are software that automate the management of cloud computing infrastructure using code and templates. They help businesses provision and configure servers, storage, databases, and networks without manual intervention or hardware controls.

They store the documented configurations in version control systems that ensure consistency and reduce turnaround time. They support DevOps and NetOps practices, enabling agility, scalability, and resilience. They can be cloud-specific or cloud-agnostic, depending on the provider and the environment.

The following table lists the major differences between each tool.

	Terraform	Pulumi
Language Support	Proprietary language - Hashicorp Configuration Language (HCL)	Pulumi SDK supports many languages including TypeScript, JavaScript, Go, C#, F#, Java, YAML. Also provides Pulumi YAML, which is a configuration language used to describe infrastructure.
IDE Features	Limited; through extensions	Code completion, strong typing, error indicators, etc
OSS License	Mozilla Public License 2.0	Apache License 2.0
Testability	Integration testing via third party tools like Terratest	Supports unit, property, and integration tests. Supports modern test frameworks
Cloud Providers	Multi-Provider Support	Multi-Provider Support
State Mgmt	Local state file by default; Cloud options available	Remote state by default managed via Pulumi cloud; Local file available
Secrets Mgmt	Unencrypted in state file	Encrypted in transit and state file
Infrastructure Reuse	Can reuse Terraform modules	Functions, classes, packages and Pulumi components can be reused

Let's take a deeper look into each of the differences listed above.

Language Support

Terraform's HCL is JSON-compatible and is used to create the configuration files that describe the infrastructure resources in the stack. It uses proprietary syntax for common programming constructs like conditionals and loops. Terraform recently released the beta of a development kit that allows you to use programming languages that compile to HCL.

Here's a look at a typical Terrafrom configuration file created using HCL to provision an Azure Resource Group.

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "=3.0.0"
    }
  }
}

variable "region" {
  description = "Azure location for the resource group"
  type        = string
}

provider "azurerm" {
  features { }
}

resource "azurerm_resource_group" "iac" {
  name     = "rg-iac-example-centralus"
  location = var.region
  tags = {name: "iac-test"}
}

Pulumi supports familiar programming languages like TypeScript, JavaScript, Python, Go, and C#, among others. This flexibility allows you write to your templates in the language you are most comfortable with. Pulumi will also convert HCL configurations into Pulumi templates.

This reference provides a document listing Terraform terminology and command equivalents in Pulumi. A Pulumi template file written in C# to create the same Resource Group in Azure looks like this:

using Pulumi;
using Azure = Pulumi.Azure;

class MyStack : Stack
{
    public MyStack()
    {
        var config = new Config();
        var region = config.Require("region");
        var iac = new Azure.Core.ResourceGroup("iac", new Azure.Core.ResourceGroupArgs
        {
            Location = region,
            Tags = 
            {
                { "name", "iac-test" },
            },
        });
    }
}

IDE Features

Plugins for Terraform are available for some IDEs like VS Code. IDE integration is limited to the capabilities of the plugin itself.

Pulumi's support for common programming languages means you get the rich IDE integration that comes with the language you choose. For example, when using C#, Visual Studio provides code completion, strong data types, intellisense, etc.

Open Source Licenses

Terraform uses the Mozilla Public License 2.0. Pulumi uses the Apache License 2.0. There are differences between each licensing model. Your use case will determine the license that is appropriate for you.

Testability

Pulumi's support for common programming languages like C# allows you to perform unit tests on your IaC templates using .Net test tools, like xUnit. Pulumi also provides the ability to perform tests that mock external calls.

Terraform supports integration testing. The plan command shows the proposed infrastructure changes before they are applied, allowing you to review the changes prior to their execution. There are also commands to validate, and format a configuration file. Robust integration testing can be performed using third party tools like Terratest. Terratest is a Go library which enables you to write automated tests for infrastructure code. It provides many helper functions and patterns for common infrastructure testing tasks such as unit testing, integration testing, end-to-end testing, dependency injection, testing parallelism, retry logic, error handling, and static analysis.

Cloud Provider Support

A resource provider handles communications with a cloud service to manage the infrastructure resources defined in IaC configurations.

Both tools support all the major cloud services. Terraform has a community of developers writing custom providers for a myriad of services. Navigating their provider registry you'll find provider support for services like Ansible and Atlassian. New cloud services aren’t always available to deploy using Terraform on day one.

Pulumi supports over 60 of the major cloud services. They also create "Native" providres for AWS, Azure, Google, and Kubernetes, which receive same-day support with every new release. The Pulumi registry provides detailed information for each provider they support. Also of note, Pulumi is able to adapt Terraform providers, so you can use the custom providers created by the Terraform community in your Pulumi templates.

State Management

IaC tools store metadata about your infrastructure so that it can manage your cloud resources. This metadata is called state. Both Terraform and Pulumi offer a desired state model where the code represents the desired infrastructure state and the tool compares this desired state with the stack’s current state to determine the resources to be created, updated, or deleted.

By default, Terraform stores state locally in a file named terraform.tfstate. Terraform state can be stored using a centrally located file, i.e. Azure blob container. This is the recommended configuration for the state file. Storing state in a centrally located file provides for backup and recovery scenarios, and allows larger teams to collaborate and manage infrastructure with shared state.

In contrast, Pulumi stores it state in the Pulumi Cloud by default. Pulumi's use of common development languages allows state to be compared as a "diff", like how code is compared and reviewed across versions.

Secrets Management

The management of secrets, e.g., database credentials, API keys, etc., in code can become a security risk. Terraform manages secrets through a separate product named Vault. Vault enables the management secrets without the need for developers to have direct access to the secrets. Vault encrypts the values it stores. It can be deployed as a SaaS or run locally as a service. Variables marked as "secret" will be excluded from an output unless the secrets command line arguement is provided.

terraform output secrets

It is important to note that Terraform state files store secrets in plain text. Because of this you should restrict access to the state file to only those with the appropriate level of authorization. A recommended best practice is to always encrypt your state file to prevent accidentally leaking sensitive information.

As mentioned above, Pulumi stores secrets in state in the Pulumi Cloud. You can also choose your own provider to store your secrets. Pulumi always stores and transmits secrets securely. It also supports encrypting sensitive data for extra protection. Configuration settings can be encrypted via the CLI command config set with the --secret flag. Secrets can also be set during runtime.

Infrastructure Reuse

A development best practice is to avoid duplicating code. The same practice applies to IaC. The ability to reuse configurations lowers the overall overhead involved with managing infrastructure.

Terraform provides a library of reusable modules. A module is comprised of a collection of .tf and/or .tf.json files stored in the same folder. Modules are the main way to package and reuse resource configurations with Terraform. Modules can be referenced from a public online registry or your own local module library.

Since Pulumi uses common programming languages, structures like classes, functions, and packages are reusable. Pulumi also provides a searchable registry where you can find packages that can be installed directly into your project. You can also create your own library of reusable packages.

Terraform or Pulumi: Which Should You Choose?

As in most decisions in software development, the answer is - it depends. Pulumi's native language support is its biggest advantage. By using native languages, you can embed IaC code directly within your application, perform unit testing, and reduce the learning curve by using a familiar language. This would be most important to developers who are new to DevOps. IT administrators with little coding experience will probably prefer the simplicity of HCL.

Bottom line, if the prospect of having to learn a proprietary language like HCL scares you, or if you think you'll need to integrate your IaC deployments within your application, choose Pulumi. Otherwise, go with Terraform. Either way, there is no wrong choice with Terraform or Pulumi. Both are more than capable of meeting your IaC requirements.

Conclusion

If you are looking to manage infrastructure as code, Spacelift is the way to go. It supports Git workflows, policy as code, programmatic configuration, context sharing, and many more great features. It currently works with Terraform, Pulumi, CloudFormation, and Kubernetes and has initial support for Ansible. You can test drive it by creating a free trial account.

How to Use Terraform `depends_on`

Paul Delcogliano — Thu, 27 Oct 2022 04:00:00 +0000

In this article you will learn why and how to use Terraform's depends_on meta-argument.

Resource Dependencies

Terraform uses resource dependencies to create and destroy resources in the correct order. Expression references create implicit dependencies between one resource and another resource. The following shows an implicit dependency created between two Azure resources using an expression reference.

resource "azurerm_resource_group" "rg" {
  name     = "rg-example-centralus"
  location = "centralus"
}

resource "azurerm_virtual_network" "vnet" {
  name                = "vnet-example-centralus"
  resource_group_name = azurerm_resource_group.rg.name
}

In this example, an implicit dependency is created because the VNet's resource_group_name attribute references the name of the resource group. The object with the dependency, the VNet, is referred to as the "object declaring the dependency", and the resource group is referred to as the "dependency object". Terraform completes all actions on the dependency object first, then performs actions on the object declaring the dependency.

What is Terraform `depends_on`

In the majority of scenarios, Terraform automatically determines dependencies between resources based on expressions within a resource block. There are rare scenarios, however, where Terraform cannot infer dependencies between resources. In these cases, you will need to create an explicit dependency. This is the purpose of the depends_on meta-argument. depends_on allows you to create an explicit dependency between two resources.

Dependencies are not limited to just resources. Dependencies can be created between modules. When the dependency is a module, depends_on affects the order in which Terraform processes all the resources and data sources associated with that module. Both implicit and explicit dependencies affect the order in which resources are destroyed as well as created. Explicit dependencies are only necessary when a resource or module relies on another resource's behavior but does not access any of that resource's data in its arguments.

There are some things you should consider when using depends_on:

Always include a comment to explain why using depends_on is necessary.
depends_on causes Terraform to create a more conservative plan. The plan may modify more resources than necessary. For example, there may be more values as unknown “(known after apply)”. This is more likely to occur when creating explicit dependencies between modules.
Adding explicit dependencies can increase the length of time it takes to build your infrastructure because Terraform must wait until the dependency object is created before continuing.
It is recommended to use expression references to create implicit dependencies whenever possible.

Use Cases for `depends_on`

Some scenarios that may require the use of an explicit dependency include running a SQL script after a DB is created, deploying containers only after the container registry is created, or you have an application running on a VM instance that expects to use a specific Azure Storage Container. This dependency is not visible to Terraform since the application’s architecture controls it. The following snippet demonstrates an explicit dependency between a VM and a storage container.

resource "azurerm_windows_virtual_machine" "vm" {
  resource_group_name      = "rg-demo"
  name                     = "vm-demo"
  location                 = "centralus"

  size                     = "Standard_F2"
  admin_username           = "adminuser"
  admin_password           = "P@$$w0rd1234!"
  network_interface_ids    = [
    azurerm_network_interface.nic.id,
  ]

  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Standard_LRS"
  }

  # all other relevant attributes

 depends_on = [
    azurerm_storage_account.st
 ]
}

resource "azurerm_storage_account" "st" {
  resource_group_name      = "rg-demo"
  name                     = "stdemo"
  location                 = "centralus"
  account_tier             = "Standard"
  account_replication_type = "LRS"
  # all other relevant attributes
}

Adding Multiple Dependency Objects

Multiple dependencies can be passed to the depends_on argument using a comma separated list of resources. Note: adding explicit dependencies can increase the length of time it takes to create your infrastructure because Terraform will wait to create the dependent resource until after the specified resource is created. The following snippet expands on the example from above and adds a second dependency, an Azure SQL Server, to the VM. This example is for illustration purposes and does not represent the best use case for creating an explicit dependency to multiple resources.

resource "azurerm_windows_virtual_machine" "vm" {
  # all other relevant attributes

 depends_on = [
    azurerm_storage_account.st,
    azurerm_mssql_server.sql
 ]
}

resource "azurerm_mssql_server" "sql" {
  resource_group_name           = "rg-demo"
  name                          = "stdemo"
  location                      = "centralus"
  administrator_login           = "adminuser"
  administrator_login_password  = "P@$$w0rd1234!"
  version                       = "12.0"
  # all other relevant attributes
}

How to use `depends_on` for modules in Terraform

As mentioned above, resources can have explicit dependencies on modules, and vice-versa. This next example uses a module from the Terraform registry to create a VM. The module will be dependent upon the successful creation of the storage account.

module "virtual-machine" {
  source = "Azure/network/azurerm"
  resource_group_name           = "rg-demo"
  # all other relevant attributes

  depends_on = [
      azurerm_storage_account.st
  ]  
}

resource "azurerm_storage_account" "st" {
  # all other relevant attributes
}

Here's an example of the same two resources with the dependency reversed. In this example the storage account resource is dependent upon the vm module. Note the use of the module keyword in the dependency resource.

module "virtual-machine" {
  source = "Azure/network/azurerm"
  resource_group_name           = "rg-demo"
  # all other relevant attributes
}

resource "azurerm_storage_account" "st" {
  # all other relevant attributes
  depends_on = [module.virtual-machine]
}

Understanding how Terraform uses implicit and explicit dependencies helps you to determine when and how to use the depends_on meta-argument. When necessary, depends_on instructs Terraform to create and destroy resources in the correct order. Misusing the argument can lead to potentially confusing, poor performing code.

Check Your IaC Deployment with the Terraform `plan` Command

Paul Delcogliano — Wed, 28 Sep 2022 15:53:42 +0000

Infrastructure as Code (IaC) is a key attribute of enabling best practices in any organization practicing a DevOps culture. The ability to build up and tear down infrastructure via code has many benefits, like reducing cost, increasing deployment speed, and lowered risk through consistency. There is, however, a dark side to IaC. If not used cautiously and carefully, it is way too easy to cause harm to an environment.

Applying infrastructure changes haphazardly can lead to unwanted changes to your environment. The Terraform plan command is available to bring visibility to your IaC deployments. The plan command reports on changes to infrastructure but it does not apply any of the proposed changes. Instead, it creates a reviewable execution plan, which you can use to confirm that the proposed changes are expected. The execution plan can be saved to a file, which can be applied later, or peer reviewed to enforce quality control. The ability to review the proposed changes prior to applying them can save you from making unexpected infrastructure changes. After you have confirmed the changes as expected, use the Terraform apply command to update the remote objects.

The plan command does three things:

Ensures the state is up to date by reading the current state of any already-existing remote infrastructure.
Determines the deltas between the current configuration and the prior state data.
Proposes a series of changes that will make the remote infrastructure match the current configuration.

If there aren't any deltas, the output will report that no actions need to be taken.

Usage Examples

Let's look at the output resulting from executing plan with a simple terraform configuration script. For this walkthrough, we'll use a new Azure subscription that does not contain any resources. The listing for the configuration script is below.



terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "=3.0.0"
    }
  }
}

provider "azurerm" {
  features { }
}

resource "azurerm_resource_group" "tf-plan" {
  name     = "rg-tf-plan-example-centralus"
  location = "centralus"
}

Run the following command on the command line to generate an execution plan. Before doing so, be sure to set environment variables for your Azure Subscription Id and Tenant Id, using the subscription id and tenant id from your Azure subscription, respectively. For more details on the different ways you can configure the AzureRM Terraform provider, see here.

terraform plan

The output contains a wealth of information, which is highlighted in the image above. Terraform proposed a create action, as indicated by the green plus sign. Changes and deletions appear as a yellow tilda and red minus sign, respectively. Terraform is also proposing to create a new resource group named "rg-tf-plan-example-centralus". The execution plan includes a summary count of each action in the proposal. In this case there is one add, zero change, and zero destroy, actions.

Apply the proposed changes by running the apply command (make sure you confirm the actions by typing yes when prompted):

terraform apply

In addition to creating the resource group, terraform also updated the local state data. Terraform uses state data to remember the remote infrastructure configurations and create deltas when configurations change.

Make the following change to the sample terraform script. Add the following line to the end of the script, just before the closing curly brace:

tags = {name: "test"}

The resource group resource should now look like this:



resource "azurerm_resource_group" "tf-plan" {
  name     = "rg-tf-plan-example-centralus"
  location = "centralus"
  tags = {name: "test"}
}

Re-run the plan command:

terraform plan

When the plan command executed, Terraform compared the stored state data for the rg-tf-plan-example-centralus resource to the changes made in the configuration script.

Looking at the output, you can see Terraform determined an update was necessary. The output also shows the proposed changes, including the actual change itself: the addition of the name tag.

Planning Modes

The plan command can be run in three different modes, normal, which is the default and the mode we've used so far, destroy, and refresh-only. The modes are mutually exclusive. You cannot use more than one mode at a time.

Destroy mode creates a plan which shows you all the remote objects that will be removed. Running the command in destroy mode does not actually perform any actions. A great use case for destroy mode is to delete a temporary development environment. You activate destroy mode by including the -destroy option on the command line:

terraform plan -destroy

There are times when you will intentionally change remote objects without the use of a Terraform script, i.e., directly via the Azure CLI. This can be necessary when troubleshooting a problem in the remote environment. In these cases, the state data will not match the remote object configuration and will need to be reconciled. Executing plan in refresh-only mode creates a plan with the goal of showing you the deltas between the state data and the remote objects. You can activate refresh-only mode by including the -refresh-only option on the command line. Note: the -refresh-only option is available only in Terraform v0.15.4 and later.

terraform plan -refresh-only

Planning Options

In addition to the alternate plan modes described above, there are a couple of other categories of options available, those affecting behavior and those affecting outputs. Options that modify the plan command's behavior are listed below.

Options Affecting How the Plan Command Behaves

Option	Description
-refresh=false	Ignores external changes to remote objects, i.e., those made via the CLI. This option will not sync the state data with the remote objects before checking for configuration changes. While this improves performance by making less remote API requests, ignoring external changes could result in incomplete or incorrect plans.
-replace=ADDRESS	Forces the replacement of the resource instead of an update action or no change at all. This will replace the resource at the given address. It can be applied multiple times to replace many objects at once.
-var 'NAME=VALUE'	Applies a value for an input variable. It can be applied multiple times, once per variable. Note: Errors will occur if a space appears before or after the equals sign (e.g., -var "region = centralus").
-var-file=FILENAME	Applies values for one or more input variables as defined in a "tfvars" file. It can be used multiple times, once for each "tfvars" file.
-lock=false	State data is unlocked during the operation. Note: This can lead to concurrency issues especially if others are running commands against the same workspace.
-lock-timeout=DURATION	Instructs Terraform to retry acquiring a lock for a period of time before returning an error. The duration value must be entered using the format nt where n is a number and t is a letter representing a unit of time, e.g., 3m is 3 minutes.

Let's turn the the resource group's location attribute into a variable. This adds flexibility and reusability to the script, allowing us to specify the location value at runtime. Modify the sample script as follows.

Add a variable to the sample script, named "region", just before the provider section and modify the azurerm_resource_group resource so the value for location attribute comes from a variable. Note: for simplicity, we are using one ".tf" file to define the resources and the variables. In practice, variables are declared in a separate file, typically named "variables.tf". The script below shows the result of making this change.



variable "region" {
  description = "Azure location for the resource group"
  type        = string
}

provider "azurerm" {
  features { }
}

resource "azurerm_resource_group" "tf-plan" {
  name     = "rg-tf-plan-example-centralus"
  location = var.region 
  tags = {name: "test"}
}

The "region" variable is declared as a string. Re-run the Terraform plan command. This time you will be prompted for the location in Azure where the resource group resides.

Run the plan command again. This time, using the -var option to provide a value for region on the command line, and notice that you are no longer prompted to supply a value for the variable.

terraform plan -var='region="centralus"'

Options Related to Formatting and Output

There will be use-cases where you will need to change the plan command's output. The following options are available for those use-cases:

Option	Description
-compact-warnings	Shows warnings as a summary unless the warnings include errors. If errors, the text will include useful information for troubleshooting.
-input=false	Disables prompts for input variables. that have not otherwise been assigned a value. This option is useful when automating Terraform scripts, like when run in a CI/CD pipeline.
-json	Returns output in JSON format.
-no-color	Removes color from the output.
-out=FILENAME	Saves the plan to a file that can be passed to the Terraform `apply` command to execute the planned changes. Any filename is allowed, but the recommended naming convention is to name it "tfplan". Do not use the ".tf" suffix as this will lead Terraform to interpret the file as a configuration script, and will fail. Note: the file contains all the values associated with planned changes, including any sensitive data, in cleartext in the plan file. For this reason, you must consider impacts to security when saving plans to a file.
-parallelism=n	By default, Terraform will run 10 operations concurrently, where possible. This option limits the number of concurrent operations to n.
-detailed-exitcode	Includes detailed exit codes which provide more granular information about the resulting plan. The list of codes is:

0 = Succeeded with empty diff (no changes)
1 = Error
2 = Succeeded with non-empty diff (changes present)

Resource Targeting

Resource targeting is like the -replace=ADDRESS option in that it tells Terraform to focus the plan only on resources that match the given address. Resource targeting includes all other objects that the selected resource(s) depend on, either directly or indirectly.

You can target resources using the -target=ADDRESS option. Note: It is not recommended to use -target under normal circumstances. Terraform enforces this point by including a warning message in the output. Run the following command to see the warning message embedded in the output:

terraform plan -var='region="centralus"' -target='azurerm_resource_group.tf-plan'

Instead of using -target, split large configurations into several smaller configurations that can each be applied independently. Doing so allows a complicated configuration to be separated into more manageable parts.

Conclusion

The plan command is very powerful despite not actually modifying any resources. Saving plans to output files enables automation and peer review. Supplying variable values on the command line, or in files, eliminates the need to hard code values, some of which may be sensitive. Using plan to show deltas between state data and remote configurations can help illuminate changes made directly to remote objects, which may need to be refreshed.

You should plan to include the plan command prior to applying your changes. This will help you avoid late night troubleshooting sessions for changes that have gone awry. Remember, failing to plan is planning to fail.

Practice DRY in Terraform Configurations Using a Dynamic Block

Paul Delcogliano — Fri, 15 Jul 2022 22:59:19 +0000

The Don't Repeat Yourself (DRY) principle of software development states that code should be written once and not repeated. An important goal of the DRY principle is to improve the maintainability of code.

See how you can keep your configuration DRY with Terragrunt on Spacelift.

Some Terraform resources include repeatable nested blocks in their arguments. These nested blocks represent separate resources that are related to the containing resource. For example, Azure VNets contain a subnet block attribute which repeats for each subnet within the VNet. This can lead to repeated configuration blocks in a Terraform script, which violates the DRY principle.

Dynamic blocks are a solution for applying DRY in terraform configuration scripts.

How to Use the Dynamic Blocks

Terraform provides the dynamic block to create repeatable nested blocks within a resource. A dynamic block is similar to the for expression. Where for creates repeatable top-level resources, like VNets, dynamic creates nested blocks within a top-level resource, like subnets within a VNet. A dynamic block iterates over a child resource and generates a nested block for each element of that resource.

Example

The following code shows the configuration of an Azure VNet and four subnets. In this example, the subnet blocks are written out explicitly, creating repeated code.

resource "azurerm_virtual_network" "dynamic_block" {
  name                = "vnet-dynamicblock-example-centralus"
  resource_group_name = azurerm_resource_group.dynamic_block.name
  location            = azurerm_resource_group.dynamic_block.location
  address_space       = ["10.10.0.0/16"]

  subnet {
    name           = "snet1"
    address_prefix = "10.10.1.0/24"
  }

  subnet {
    name           = "snet2"
    address_prefix = "10.10.2.0/24"
  }

  subnet {
    name           = "snet3"
    address_prefix = "10.10.3.0/24"
  }

  subnet {
    name           = "snet4"
    address_prefix = "10.10.4.0/24"
  }
}

That same configuration using a dynamic block is shown below. Replacing the four subnet blocks with a dynamic block removes repeated attributes, leading to cleaner code that is easier to maintain.

resource "azurerm_virtual_network" "dynamic_block" {
  name                = "vnet-dynamicblock-example-centralus"
  resource_group_name = azurerm_resource_group.dynamic_block.name
  location            = azurerm_resource_group.dynamic_block.location
  address_space       = ["10.10.0.0/16"]

  dynamic "subnet" {
    for_each = var.subnets
    iterator = item   #optional
    content {
      name           = item.value.name
      address_prefix = item.value.address_prefix
    }
  }
}

Here is the definition of the variable subnets:

variable "subnets" {
  description = "list of values to assign to subnets"
  type = list(object({
    name           = string
    address_prefix = string
  }))
}

The values for the subnets variable are defined in a tfvars file. Sample values are:

subnets = [
  { name = "snet1", address_prefix = "10.10.1.0/24" },
  { name = "snet2", address_prefix = "10.10.2.0/24" },
  { name = "snet3", address_prefix = "10.10.3.0/24" },
  { name = "snet4", address_prefix = "10.10.4.0/24" }
]

Dynamic Block Components

dynamic blocks are supported inside of resource, data, provider, and provisioner blocks. A dynamic block consists of the following components:

Component	Description
label	Specifies what kind of nested block to generate. In the above example, the label is "subnet". A subnet resource will be generated for each element in the `var.subnets` variable.
for_each	The complex value to iterate over.
iterator	(optional) Sets the name of a temporary variable that represents the current element. If not provided, the name of the variable defaults to the label of the dynamic block. An iterator has two attributes: key and value. Key is the element index. Value is the element value.
content	Defines the body of each generated block.

A dynamic block can only generate attributes that belong to the resource type being created. It isn't possible to generate meta-argument blocks like lifecycle. Some resource types have blocks with multiple levels of nesting. When working with multi-level nested blocks, the iterators for each block are important. It is recommended to use the optional iterator component to clearly define each level in the configuration, and to remove any ambiguities resulting from attributes with the same name as a parent block. The listing below illustrates a nested block example, with clearly defined iterators for each block.

  dynamic "origin_group" {
    for_each = var.load_balancer_origin_groups
    iterator = outer_block
    content {
      name = outer_block.key

      dynamic "origin" {
        for_each = outer_block.value.origins
        iterator = inner_block
        content {
          hostname = inner_block.value.hostname
        }
      }
    }
  }

Key Points

A dynamic block is a great way to apply the DRY principle in Terraform configuration scripts. Implementing a dynamic block where appropriate removes repeated code leading to configurations which are easier to read, maintain, and reuse - just as the DRY principle aims to do.

An Introduction to DevOps with Azure DevOps

Paul Delcogliano — Mon, 09 May 2022 12:22:38 +0000

Want to transform your company into an Agile powerhouse? To do that you'll need to start with a solid foundation built on the ideas and concepts of a DevOps culture. Once you have that foundation established you can look to technical solutions to convert your company's culture and technology.

This article introduces DevOps concepts and terminology, then introduces each of the features and services available from Microsoft's Azure DevOps suite of services and connects each service to the concepts of a DevOps culture.

Let's begin by defining "DevOps". DevOps is a culture whose goal is to change the way developers and IT staff collaborate to deliver software. The DevOps culture allows a software development team to better respond to business requirements, increase confidence in the software it delivers, and rapidly deploy high-quality software.

DevOps influences Application Lifecycle Management (ALM) through its planning, developing, delivering, and operating phases. Each phase is dependent upon its prior phase. Phases are not role specific. Collaboration is key in a true DevOps culture. To that end, each role is involved in all phases to some degree. The four phases are described below.

Planning

A software project starts at the planning phase. During this phase teams generate ideas, define the project, and describe the application and capabilities. Backlogs and sprints are used to manage the project. Progress can be tracked at differing levels of granularity, from single-product tasks to tasks that span many products. Scrum and Kanban boards are used to visualize progress using dashboards and reports.

Developing

During the development phase all aspects of application development take place, including code writing, testing, reviewing, and the management of source code. Development teams practicing a DevOps culture use Continuous Integration (CI) to achieve the goals of the developing phase. Continuous Integration is the practice of automating code merging and testing. CI helps a team to identify issues early in the development cycle, making them less expensive to resolve. The CI process typically includes automated tests which help to ensure code quality.

Delivery

The delivery phase includes the deployment of applications into various environments in a consistent and reliable way. Releases are controlled with manual approvals. Automation moves applications between environments all the way up to production. Teams implement Continuous Delivery (or Continuous Deployment) (CD) to meet the goals of the delivery phase.

Continuous Delivery is a process by which code is built, tested, and deployed to one or environments. Deploying and testing in multiple environments increases quality. The entire delivery process is scalable, repeatable, and controllable using automation. CD produces outputs which are deployable. Assets like executables, configuration files and even infrastructure can be included in the CD output. Teams use tools like Terraform, Bicep, or ARM templates to deploy infrastructure in a repeatable, reliable manner. These tools enable DevOps teams to create and tear down infrastructure via code, aka Infrastructure as Code (IaC). Monitoring and notification systems are constantly running to provide visibility into the whole CD process.

Operating

Teams adopting DevOps practices aim to ensure their systems are reliable, available, secure, and compliant. The operating phase involves maintaining, monitoring, and troubleshooting applications in production environments. DevOps teams look to identify issues in production before they affect the end-user's experience. They want to mitigate issues as soon as they occur. To meet these goals, teams require notifications and robust application logging.

What is Azure DevOps?

Once a company buys into the DevOps culture it needs a toolset to implement the Agile principles of delivering high-quality software. This is where Microsoft's Azure DevOps suite of services fits in. Azure DevOps provides tools for every phase of ALM including planning, development, delivery, and operations. You can choose how many of the Azure DevOps services to implement, from one to all. The services available within Azure DevOps are shown in the figure below.

Azure DevOps is available in two flavors, Azure DevOps Server and Azure DevOps Services. Azure DevOps Server is a self-hosted, on-premises installation of Azure DevOps. This version shares the same features as the cloud offering, Azure DevOps Services but makes you responsible for security, configuration, and all the other responsibilities of maintaining software in your data center. Organizations may choose Azure DevOps Server because compliance or security requirements necessitate the need to keep data within its data center. Integrating with other on-premises services like SQL Server Analysis Server would be another use case for choosing the on-prem version. Those use cases are exceptional. Most development shops will choose Azure DevOps Services.

Azure DevOps Services

Azure DevOps Services is a cloud-based, Software-as-a-Service (SaaS) offering. The SaaS version is part of the Microsoft Azure cloud platform. It reaps all the benefits of a SaaS offering, including 24x7 availability, automatic updates, and Azure AD security. Azure DevOps Services requires no extra configuration. Getting started is quick and easy, using your Microsoft account to sign up and create an organization (domain), a project, and add a new user.

The Azure DevOps Web portal provides access to all features You can use all the services included with Azure DevOps or choose just what you need to compliment your existing workflows.

Organization

Azure DevOps is arranged in a hierarchical structure. At the top of the hierarchy sits the organization. An Azure DevOps organization is a container of one or more related projects. Each Azure DevOps account can contain one or more organizations. A good rule of thumb is to start with one organization and add others as needed based on factors like security models, business units within the company, and the company's development teams. Organizations are isolated from one another. It is easier to partition an organization using projects than it is to combine different organizations. A good rule of thumb is to stick with one organization when possible.

Each organization has its own URL in the form of https://dev.azure.com/<Organization Name> and gets its own set of free tier resources (more to come on these resources in a bit) including:

Jobs for executing a CI/CD pipeline
Tools for tracking work items, defects using Kanban or Scrum
An unlimited amount of private Git repos
5 free users (basic)
2 GiB Artifact storage

Organizations can be created using an Azure Active Directory (Azure AD) or a Microsoft account. The account used influences the organization's security model. Use an MS account when you don't need to authenticate with an Azure AD. For stricter control, create an organization using an Azure AD account. This restricts access to the organization to only those AD users who are members in that directory. Removing a user from AD removes his/her access from the organization. This provides richer security because the Azure AD is managed by the administrator and can be controlled company wide. Users can be administered using Azure AD groups, further facilitating access control.

Projects

The next level down in the hierarchy is the Project. An Azure DevOps project is a grouping of services and features that are available through the Web portal or an IDE client, like Visual Studio. When you create your organization, a default project is created for you.

The screen shot below shows the options available during when creating a new project. The choices made during creation will influence the behavior and functionality of the tools within the project.

A project can be made available to the public or a private group. The Version control choice under the Advanced drop down determines the type of source control repository. There are two options, Git and Team Foundation Version Control. Most teams will choose Git. The Work item process drop down allows you to choose the project's process model. This link provides detailed information about process models. The table below summarizes each option available.

Process Model	Description
Basic	Provides the simplest model that tracks work through Issues, Tasks, and Epics
Agile	Supports Agile planning methods including Scrum, and tracks development and test activities separately. This process works great if you want to track user stories and (optionally) bugs on the Kanban board, or track bugs and tasks on the taskboard
Scrum	Tracks work using product backlog items (PBIs) and bugs on the Kanban board or viewed on a sprint taskboard. This process supports the Scrum methodology as defined by the Scrum organization
Capability Maturity Model Integration (CMMI)	Supports a framework for process improvement and an auditable record of decisions. With this process, you can track requirements, change requests, risks, and reviews. This process supports formal change management activities

An organization can have one or more projects. Much like the decisions to create multiple organizations, several factors should be considered when determining to create multiple projects. Unique security and access control requirements, staggered release schedules, disparate code bases, and diverse development teams are some reasons why you would create multiple projects. Single projects are great for aligning multiple teams along the same cadence, with common goals and release schedules. Be cautious using a single project for your entire enterprise. Poorly organized folder structures or bad naming conventions can make searching across a single project difficult. A best practice here is to create one project per source code repository.

Beneath the project hierarchy lies five child services, Boards, Repos, Pipelines, Test Plans, and Artifacts. An individual project has its own set of services which are isolated from the other projects. The list below describes each of the standalone services available within a project. These may differ depending upon the options you selected when creating the project:

Boards: Tools for planning and tracking work items, code defects and issues using Kanban and Scrum methods are available from the Azure Boards service.
Repos: source control is provided by the Azure Repos service.
Pipelines: Build pipeline for Continuous Integration and Continuous Deployment (CI/CD) functionality.
Test Plans: Tools for testing applications, both manual and automated.
Artifacts: The sharing of integrated packages like npm and NuGet is available from the Azure Artifacts service.

Boards

The set of tools available under the Azure DevOps Boards category are useful for managing a project during the ALM planning phase. The tools provide a rich set of capabilities including support for Agile, Scrum, and Kanban processes. Tools like workflow items, backlogs, sprints, and configurable dashboards are used for tracking and reporting a project's progress. These tools are designed to scale as the business grows and help the team to plan and manage a project throughout its lifecycle.

When starting a new project, teams will begin with setting up the "board". Everything begins with a work-item. The Work Items link is used for creating items that represent a task. Work-items contain details about the task, such as its type - it can be a bug, task, epic, or a user story, its priority, risk level, assignee, any related child or parent tasks, and various dates. The following screenshot shows a task I created for writing this post.

The DevOps team sets up all the work-items for a project. Views show the work items in various stages of development. The stage names are customizable by the team. The default stages are "New", "Active", "Resolved" and "Closed". The following screen shot shows the boards link in my sample project.

The Backlogs link provides a visual method for assigning work-items to a sprint. Using the backlog, work-items can be dragged to the Planning cards and dropped into an Iteration. An iteration is the default name for sprints in Azure DevOps. The iteration name is customizable. In the screen shot below, I moved my "Create Post Outline" task to the first sprint in the project.

Additional planning features are provided by the Sprints, Queries, and Delivery Plans links. Each tools provides functionality for managing the project. For example, items moved to a planning iteration are visible under the Sprints link. This view lets teams track the status of the work-items in a specific sprint.

Repos

Version control software like Git and Subversion enable teams to track changes made to code over time. As code is modified during the development phase, the version control system retains history of the changes. The version control system can also coordinate code changes across your team. A snapshot can be recalled later for comparison, restoration, and/or deployment. Regardless of project or team size, it is a good practice to use a version control solution.

Azure DevOps offers the Repos suite of version control tools. These tools allow DevOps teams to collaborate on development using free public and private repositories. As mentioned above, Azure Repos provides two version control systems to choose from: Git and TFVC. Primarily teams decide to use Git as it is a decentralized source control system and will receive most of the attention from Microsoft in the future. It also enables the greatest amount of flexibility and integrates with almost all tools in the developer ecosystem. A project can contain an unlimited amount of Git repos.

Azure DevOps Repos offers a tremendous amount of functionality. Highlighted below are some of the more relevant features.

Code can be shared using command-line, VS Code, or Visual Studio, among other IDEs.
Code reviews are performed with pull requests to ensure that changes build without errors and pass tests before being merged into the main branch.
Work-items can be linked to pull requests for project tracking.
Policies and permissions can be applied to individual branches.
Azure Functions can be used to create custom branch policies.

In projects where the Azure Repos service is enabled, clicking on the Repos link shows the following configuration options for initializing a repository.

These options initialize a project's repository. An empty repository can be created, or the repository can be initialized with existing code via upload. Options are also available for creating a readme file and a .gitignore file, which is a file that ensures certain files are not tracked by Git. Each development language has its own files to ignore. The setup options allow you to create a language specific .gitignore file.

DevOps teams with prior experience using Git will be familiar with the version control concepts Azure Repos implements. To get the most benefit from Repos, it is important to have a basic understanding of git version control concepts. The following table describes some of the relevant ones. You can read more about version control concepts here.

Concept	Description
Clone	Cloning from your local PC creates a complete working copy of a repository on your local workstation. Changes made to the local repository can be pushed to the remote repository in Azure DevOps.
Commit	A commit is a set of related code changes made to your local repository. These changes can be pushed or rolled back as a set.
Branch	Branches keep a history of commits and isolate code changes from the main branch. Branches are isolated, so committing changes to a branch doesn't affect other branches.
Pull Request	Pull requests allow the team to review code and provide feedback on changes before they are merged into a branch.
Fork	A fork is a repository copy of a repository. The copy includes all files, commits, and (optionally) branches from the remote. Files and branches in a forked repo are not shared between repositories. A pull request is required to merge forked files back into the main branch.

An Azure DevOps repository has a default branch named "main" or "master". The default branch should be protected from code that doesn't build properly. As a best practice, set up a branch policy that requires all merges into the default branch to be performed with a pull request (PR). This policy provides two benefits. First, it prevents changes to the main branch directly. This helps to ensure code quality by requiring a code review before changes are merged into the default branch. Second, the PR forces a clean build of the merged code before it is merged. Branching and release strategies are important concepts to understand. To learn more see "Adopt a Git branching strategy". Branches and PRs can be managed via the Branches and Pull requests links, respectively.

As mentioned above, a project can have multiple Git repositories. Multiple repositories are useful for projects that have independent deployment schedules. Multiple repositories will require separate security permissions, separate release schedules, and make sharing code across those repositories difficult. In most cases a single repository is the best approach. The overhead of maintaining multiple repositories isn't worth the effort vs. a single repository.

Forks are useful in specific scenarios such as when working with developer teams that should not have direct access to update the main repository. Forks can also be useful when used in an open-source project.

Pipeline

As discussed in the delivery phase, CI/CD plays an important role in a DevOps organization. The Pipelines service helps teams to set up and manage CI/CD for their projects. The ability to integrate code being worked on among many developers, and deploy releases through automation is a cornerstone of ensuring a quality product. Pipelines is flexible in that it works with many languages and project types. The pipeline itself is code, comprised of a series of stages, jobs, and steps, typically written using YAML. YAML is a human-readable, easy to understand language for creating configuration files. Azure pipelines integrates with Azure deployments, can build across platforms (Windows, Linux, or Mac), and can deploy to multiple environments. Azure pipelines is free for public projects. For private projects Azure provides 30 hours of pipeline jobs for free per month.

Creating a new pipeline is done via the pipelines link. This starts a wizard. The first step in the wizard is to provide the location of the source code repository. Choices are Azure Git Repos, GitHub, Bitbucket, Subversion, among others. Next, a repository from that location is chosen. The third step is to initialize the pipeline, using either a "starter pipeline" or an existing pipeline. Reviewing and saving the pipeline are the final steps in the wizard. The result is a file named "azure-pipelines.yml" which is automatically added to the project's repository. The following screenshot shows the azure-pipelines.yml file created by the starter pipeline wizard.

The starter pipeline mimics a typical "hello world" application. Running the pipeline outputs the string "hello world" as shown below.

There are two methods for creating pipelines, hand coding the azure-pipelines.yml file or choosing from a list of pre-defined tasks and configuring those tasks using a wizard. Azure provides a rich set of tasks to choose from. To learn more about the capabilities of a pipeline, spend some time reviewing the task list that is available in the portal.

It is essential to understand the basic terms and components of a pipeline. This knowledge is necessary to deliver code more efficiently and in a reliable manner. For brevity I will describe a some of the relevant pipeline concepts below. You can read more about pipeline components here.

Term / Component	Description
Agents	Agents effectively are virtual machines with installed agent software that run one job at a time.
Approvals	Approvals are a set of validations that are required to pass before a deployment runs.
Environment	A collection of resources, i.e., VMs, containers, or other services that are used to host the application. After a completed build, a pipeline may deploy changes to one or more environments.
Stage	A stage is a group of related jobs and steps. Stages run sequentially and can use conditional logic to determine if a stage should execute or be skipped.
Job	A job is a collection of steps that run together on the same agent.
Release	Branches keep a history of commits and isolate code changes from the main branch. Branches are isolated, so committing changes to a branch doesn't affect other branches.
Step	A step is the most granular task within a pipeline. It can be either a task or a script.
Deployment	A collection of steps that are executed sequentially.

Pipelines can include tasks for building out infrastructure, deploying code to cloud services, performing tests, and generating output assets. Outputs are referred to as "Artifacts" and can become input to downstream tasks or processes. Artifacts can be deployed with the application to an environment.

More information about pipelines can be found here.

Test Plans

An important part of the development process is the ability to test and validate the project deliverables. The Test Plans category offers a central location where the team can coordinate all of its manual test activities, track progress, and gain key insights. Using the Test Plans toolset, teams can create tests for user stories and run the tests directly on the Kanban board. Creating, configuring, and running tests involves many steps. Review the resources available at this link for a complete guide. For now, let's walk through the basics.

Test cases are created from work-items off the Board. This is a bit counterintuitive as you may be inclined to think tests would be created via the Test Plans link. To create a test, select a work-item from the Board and click on the three dots. This action opens a context menu. Choose Add Test from the menu as shown in the following screen shot.

Existing tests are available from the Test Plans link. The Test Plans dashboard lists all tests and provides a portal for running and managing tests.

Editing a test is accomplished via the context menu for each test case item. The form used to edit tests is shown below. The form provides places to enter test steps and to create parameters among other things.

The context menu also provides options for running tests and reviewing test results.

Robust reporting can be found under the Progress link. Details like the number of test plans, the percentage of tests that have passed, and the names of specific tests are useful for teams looking to ensure the highest quality software is delivered. Additional information and test executions are available under the Runs link.

Artifacts

Azure Artifacts provides a mechanism for developers to share and consume packages from different feeds and public registries. DevOps teams can consume packages from well-known package managers like NuGet, and npm, or can create and share their own packages. Sharing can be scoped to the same team, organization, or publicly.

Depending on usage, teams may incur a cost when using Artifacts. Artifacts is billed on a consumption basis and is free up until 2 GiB of storage. Once the 2GiB storage limit is reached, no new artifacts can be uploaded. For storage beyond 2GiB, a billing account must be setup.

Packages are organized into "Feeds" which allow the team to group packages and control sharing permissions as a cohesive unit. Public feeds enable packages to be shared publicly with anyone on the Internet. Literally anyone can access the packages, even if they don't have an Azure DevOps account or are not part of the company. Organization-scoped feeds are only viewable and accessible in the Azure Artifacts hub from projects within your organization. Packages that are shared via a feed are immutable.

Teams may decide to use Azure Artifacts to consume the artifact in a release, or as part of a deployment the build needs additional packages stored in Azure Artifacts, i.e., NuGet.

Azure DevOps Benefits/Advantages

The value proposition Azure DevOps brings is in its cohesive toolset. Everything a team needs to adopt a DevOps culture can be found within the Azure DevOps services. This is one of its biggest advantages over the alternatives. Other players offer similar features and services, but Azure DevOps brings them all together under one SKU.

Azure DevOps Alternatives

The list below includes some alternatives to Azure DevOps. This is not a comprehensive list. Many of these players are top alternatives, but none seem to provide the breadth of services that Azure DevOps offers.

Service Name	Description
GitHub	Provides tools for Git repositories, CI/CD, collaboration, and automation.
AWS	AWS offers ala carte services for DevOps teams. Included in their offerings is AWS CodePipeline. CodePipeline is a CI/CD service for application and infrastructure. AWS CodeDeploy CodeDeploy is a service which automates deployment to AWS or on-premises infrastructure. AWS CodeCommit CodeCommit is a source control solution.
BitBucket	Provides teams one place for source control repositories, project management and deployment.
Jenkins	An open source automation server which uses plugins to support building, deploying and automating any project.

Congratulations, you made it to the end of the article! Along the way you learned what is necessary for a software team to adopt a DevOps culture. Planning, Development, Delivery, and Operating are all key phases of that culture. Azure DevOps provides tools and services for teams to implement DevOps principles into their ALM software development process. In Azure DevOps it begins with the creation of an organization, and projects within organizations. This provides a wealth of tools for teams to use to track progress and report status, manage source control, set up a CI/CD pipeline, perform tests, and monitor the entire process. While alternatives exist, Azure DevOps is a best-in-class set of services and should be a serious consideration for any team looking to adopt a DevOps culture.

Automate EC2 Instance Shutdown with Auto Scaling Groups to Minimize AWS Costs

Paul Delcogliano — Mon, 18 Oct 2021 12:45:41 +0000

Shutting down EC2 instances when they are not in use is an effective way to manage cloud costs. In my previous post I described a method for starting and stopping EC2 instances on a schedule using CloudWatch Event Rules and Lambdas.

Launch Templates, in conjunction with Auto Scaling Groups (ASG), provide another solution which achieves the same result. This solution uses the ASG's ability to scale-out and scale-in on a schedule. More information about the use cases for each solution can be found in the previous post. This post focuses on setting up Launch Templates and ASGs. It assumes you have a general working knowledge of each of these AWS services.

Launch Templates and ASG

Employing this method to scale-out and scale-in EC2 instances on a schedule can be broken down into three steps:

Create a Launch Template from an EC2 instance
Create the Auto Scaling Group
Create schedules to scale the ASG

Step 1: Create a Launch Template from an EC2 instance

From the AWS documentation, a launch template

...specifies instance configuration information. It includes the ID of the Amazon Machine Image (AMI), the instance type, a key pair, security groups, and other parameters used to launch EC2 instances...

Launch Templates are used when scaling EC2 instances with ASG. Whenever an ASG needs to scale-out and create a new EC2 instance, the instance is configure using the settings from the launch template.

To create a launch template from an existing EC2 instance, navigate to the EC2 services console and click on the "Instances" link. Start the Launch Template wizard by selecting an existing instance and choosing Actions | Image and templates | Create template from instance menu options.

There are many settings in the wizard. I've listed the relevant settings by section below. Any settings not called out specifically can be left as the default.

Section	Setting Name	Value
Launch template name and description	Name	"schedule-launch-template"
	Auto Scaling guidance	false (unchecked)
Network Settings	Security Groups	none
Network Interfaces	Subnet	select "Don't include in launch template"
	Security Groups	none - remove any that are pre-selected
Advanced details	Request Spot Instances	false (unchecked)
	Shutdown behavior	select "Don't include in launch template"

After completing the form, click the "Create launch template" button to validate and save the template. A common source of invalid templates is the combination of selected security groups and subnets. If you have an invalid template, be sure to verify you have selected the proper values for those settings. Once the template is valid, you will see a success message similar to the one shown below.

This dialog presents three links. To start the wizard which guides the creation of the ASG, click on the "Create Auto Scaling group" link.

Step 2: Create the Auto Scaling Group

The AWS documentation describes an ASG as

...a collection of Amazon EC2 instances that are treated as a logical grouping for the purposes of automatic scaling and management...

ASGs provide functionality for scaling the number of running instances. It is this feature that makes ASGs ideal for this solution.

Like the launch template wizard, there are many settings in the ASG wizard. The ASG wizard walks through seven steps, four of which are optional. For brevity's sake, I focused on the values for the required steps and skipped over the optional ones. Any settings not called out specifically can be left as their default unless you have special needs for your unique configuration.

Step one asks for an ASG name and a Launch Template. Enter "schedule-instance-asg" as the name and select the "schedule-launch-template" Launch Template. Click the "Next" button to navigate to step two.

The first section in step two asks for the Instance Purchase options. Instance purchase options provide a way to specify how instances are to be distributed. Select the "Combine purchase options and instance types" option.

The settings in the Instance Distribution section allows further cost optimization by providing choices for how EC2 instances are paid. ASGs allow for two distribution options, On-demand and Spot. Spot instances are cheaper but have very specific use cases. You can learn more about EC2 pricing here.

Since this is a post about saving costs I decided to use spot instances instead of On-demand. You will have to decide what is best for your situation. To use spot instances provide the following values when filling out the form:

Setting Name	Value
On-Demand Instances	0
% On-Demand	0
% Spot	100
Spot allocation strategy per Availability Zone	select the recommended value
Capacity optimized Spot settings	select Capacity rebalance

The Instance Type section allows for provisioning the type of VM in the ASG. The instance type value will be pre-selected with the value specified in the launch template that was created in step one. You can modify this setting or leave it as the default. A nice benefit that comes from the scheduling of instances to shut down is that you can choose a beefier VM knowing that you are saving money when the instance shuts down. After selecting the instance type, move onto the Network section.

In this section, select the VPC and subnets where the instances are to be launched. The default VPC is pre-selected so be sure to change it if you are using a non-default VPC. That's all for wizard step two. Click the "Next" button until you reach step five where you can add notifications for ASG events.

SNS Topics are used to send notifications whenever the instances in an ASG are started or stopped. To setup notifications, select an existing topic from the SNS Topic dropdown, or click the "Create a topic" button to create a new SNS topic. Provide one or more email addresses in the With these recipients textbox and select one or more of the following ASG events: Launch, Terminate, Fail to launch, Fail to terminate, as shown below.

Click the "Skip to review" button to jump to the final step in the ASG wizard. Here you can review the settings and configurations chosen in all of the previous steps. After verifying everything, click the "Create Auto Scaling group" button to create the ASG.

Step 3: Create schedules to scale the ASG

The final step in this process is to setup the scale-out and scale-in schedules. Select the "schedule-instance-asg" ASG from the list, click on the "Automatic Scaling" link and scroll down to the Scheduled actions section. Click "Create scheduled action" to open the form shown below.

Use this form to create the schedule that will scale-out, i.e. startup the instance(s). Provide the values from the table below to scale-out to one instance at 7AM Est, Monday through Friday.

Setting Name	Value
Name	asg-scale-out
Desired Capacity	1
Min	1
Max	1
Recurrence	Cron
Cron expression	00 11 * * MON-FRI
Time zone	Etc/UTC

Click the "Create" button to save the schedule. Repeat the process to create the schedule to scale-in, i.e. shutdown, the instance(s). Provide the following values to run the schedule at 8PM Est, Tuesday through Saturday.

Setting Name	Value
Name	asg-scale-in
Desired Capacity	0
Min	0
Max	0
Recurrence	Cron
Cron expression	05 00 * * TUE-SAT
Time zone	Etc/UTC

Click the "Create" button to save the scale-in schedule.

Logging and Monitoring

One of the features of ASG is its inherent ability to log activity and monitor instances. You can view the logs by selecting the "schedule-instance-asg" ASG from the list and clicking the "Activity" link. The data shown includes the status and runtime of each ASG. It includes any deltas in the number of running instances among other useful information.

Instance monitoring is achieved by clicking on the "Monitoring" link. ASGs monitors a wealth of data including Minimum and Maximum Instance counts, the number of instances running and terminated, and resource usage of the EC2 instances in the ASG.

Conclusion

Controlling costs in the cloud can seem like a full time job (actually, it probably is). If left running, EC2 Instances quickly accrue fees and can lead to sticker shock when you get your monthly invoice. Automating the shutdown of EC2 Instances is one very effective method for controlling costs.

Part one of this series focused on using CloudWatch Event Rules and Lambdas to automate EC2 instance startup and shutdown. Here we looked at using Launch Templates and Auto Scaling Groups to achieve the same result. While each has its own pros and cons, both are extremely effective techniques for keeping costs under control. Drop me a note in the comments and let me know how you control costs in your cloud environment using these, or any other methods.

Automate EC2 Instance Shutdown with Lambdas to Minimize AWS Costs

Paul Delcogliano — Mon, 18 Oct 2021 12:44:43 +0000

In the cloud, cost control is a task unto itself. It takes diligent oversight over the services in your AWS cloud to ensure the monthly spend doesn't grow out of control. One method for lowering costs is to shutdown EC2 Instances when they are not in use. In this two-part series, I will show you how to reduce your monthly spend by scheduling the shut down of EC2 Instances using two different methods.

Controlling Costs

There are several factors which determine how an EC2 Instance is billed. The main factor is by usage, typically measured by the hour. As long as an EC2 Instance is in the running state, you are accruing fees, even if you are not actually using the VM. An effective method for controlling these costs is to shut down the VM when it is not needed. When the instance is in the stopped or terminated state you still pay for storage, but that is a fraction of the compute costs you pay while the instance is running.

Development, QA, or Test environments are all good use cases for automating the shutdown of an EC2 Instance. In these environments, the instances are most active during business hours. Those instances do not need to be running once everyone signs off at the end of the day.

I use two methods for shutting down EC2 instances. The first involves using CloudWatch Event Rules together with Lambdas. The second uses Launch Templates in conjunction with Auto Scaling Groups (ASG).

Each method has its own use cases. For simple startup/shutdown scheduling I like the CloudWatch/Lambda solution. This method is the easier of the two to setup. It works well when scheduling a set of EC2 instances. It is also appropriate for use cases involving EC2 instances with static public IP addresses (EIPs).

The relatively more complex Launch Template/ASG method provides additional functionality beyond scheduling like email notifications, and the ability to continually retry starting an instance if it fails to start. This is a good solution when no EIPs are involved, when all of your EC2 instances are the same instance type, or you want more robust monitoring and notifications.

I'll walk through both methods for controlling and minimizing cloud costs. This post focuses on using CloudWatch Event Rules and Lambdas to schedule instance startup and shutdown. The second post walks through using Launch Templates with ASG to achieve the same result.

CloudWatch Event Rules and Lambda

Among the many AWS services, CloudWatch Events and Lambdas can be combined to provide basic scheduling capabilities. The AWS documentation has in-depth instructions for using CloudWatch Events and Lambdas to shutdown and startup EC2 instances on a schedule.

There are three basic steps to using this technique. The steps below use the AWS Console to setup and configure the policy and role, the CloudWatch Events Rules and Lambda functions.

Create a custom Identity and Access Management (IAM) policy and execution role for the Lambda functions.
Create and test two Lambda functions; one to stop EC2 instances, and another to start them.
Create CloudWatch Event Rules that trigger the Lambda functions on a schedule.

Step 1 - Create IAM Policy and Role

Using the IAM service console, create an IAM policy by clicking on the "Create policy" button. This starts the policy creation wizard. On the JSON tab, enter the policy permissions from Listing 1 below. These permissions allow CloudWatch logging and EC2 startup and shutdown.

Listing 1

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:Start*",
                "ec2:Stop*"
            ],
            "Resource": "*"
        }
    ]
}

Click through the wizard until you get to the "Review" page. On the Review page, enter "lambda-policy-for-scheduling-ec2-instances" as the policy name. Click the "Create policy" button to save the policy. The following screenshot shows the policy review screen. Notice that the Summary section shows permissions being applied to CloudWatch Logs and EC2 services.

To setup the role, navigate to the Roles link in the IAM Console and click the "Create role" button. This starts the role creation wizard. On the first page, select the AWS service option and choose the "Lambda" use case. Click the "Next: Permissions" button. Search for the policy created earlier named "lambda-policy-for-scheduling-ec2-instances", and select it from the list to attach it to the role. Click through the wizard until you get to the Review Role step. The following screenshot shows the role review screen.

Enter "lambda-role-for-scheduling-ec2-instances" for the role name, then click the "Create role" button to save the role.

Step 2 - Create two Lambda functions

In the AWS console, open the Lambdas service console. Click the "Create Function" button to start the wizard. Select the Author from scratch option. Name the function "start-ec2-instances". Select "python 3.9" from the Runtime dropdown. Click on the "Change default execution role" link and select the Use an existing role option. Select the role created above, "lambda-role-for-scheduling-ec2-instances" from the drop down. Leave all other settings as their defaults, as shown below.

Click the "Create Function" button to save the function. The next step in the wizard is to provide code for the lambda. The python script in Listing 2 below is used to start your EC2 instances. Copy/paste the script into the lambda wizard's code tab.

Listing 2

import boto3
region = '<your region here>'
instances = ['<your 1st instance id>', '<your 2nd instance id>']
ec2 = boto3.client('ec2', region_name=region)

def lambda_handler(event, context):
    ec2.start_instances(InstanceIds=instances)
    print('started your instances: ' + str(instances))

The script contains two variables, region and instances. Replace <your region here> with the AWS region name where your EC2 instances are running, i.e. us-east-1. The instances variable is a comma separated list of EC2 Instance IDs. Provide one or more instances in the list. Each instance provided will be affected by the function. Navigate to the AWS EC2 console to find your Instance IDs.

After editing the script, click the "Test" button. This opens a dialog allowing the Lambda to be tested. Provide a value for the test's Event Name and click "Create". Click the "Test" button again to run the test. Review the output, looking for errors. The following screenshot shows the completed Test dialog form.

Of note; testing the Lambda function will execute the python script. Any instances listed in the startup function will be started. The same goes for the shutdown function. Testing that function will shutdown the instances.

Once any errors and warnings are resolved, repeat the process to create the "stop-ec2-instances" Lambda. The python script in Listing 3 provides the code for stopping EC2 instances. Provide the same values for region and instances here as you did for the "start-ec2-instances" script.

Listing 3

import boto3
region = '<your region here>'
instances = ['<your 1st instance id>', '<your 2nd instance id>']
ec2 = boto3.client('ec2', region_name=region)

def lambda_handler(event, context):
    ec2.stop_instances(InstanceIds=instances)
    print('stopped your instances: ' + str(instances))

Step 3 - Create CloudWatch Event Rules

The final step in this process is to setup the schedules that will stop and start the instances. Begin by navigating to the CloudWatch service in the AWS console. Click the Rules link and then the "Create rule" button. This will create a new rule which will be used to schedule instance startup.

The screenshot below shows the Event Source section. This is where the schedule is defined. Start by selecting the Schedule option and the Cron expression option. Enter the following cron expression to schedule instance startup to occur at 7 AM EST, Monday - Friday: 0 11 ? * MON-FRI *

Next, click the "Add Target" button. Select Lambda function from the drop down and select the "start-ec2-instances" function. Click the "Configure details" button and provide a name for the rule. Click the "Create rule" button to save the rule.

Repeat this process to configure a rule to shutdown the EC2 instances. The following cron expression schedules instance shutdown for 8:05 PM EST, Tuesday - Saturday: 05 00 ? * TUE-SAT *

Logging

You may have noticed earlier that the policy contained permissions for logging. Every time one of the Lambda functions executes, it logs its output to a CloudWatch Log group. To view the logs, navigate to the "Log groups" link under the CloudWatch console. In the list of log groups, you should see one named "aws/lambda/start-ec2-instances". This was created when you tested the lambda function. Click that log group and navigate to the log streams where you will find the results of the lambda's execution, including the output from any print statements in the python script.

Conclusion

Controlling costs in the cloud is a full time job. The combined power of CloudWatch Event Rules and Lambdas provides one method of controlling those costs by scheduling the startup and shutdown of EC2 instances. In the next post, we'll look at another approach to cost cutting using Launch Templates and Auto Scaling Groups. See you there.

AWS Subnet Tip: Using the Auto-Assign Attribute

Paul Delcogliano — Mon, 13 Sep 2021 13:19:14 +0000

Have you ever launched an EC2 instance into a subnet only to discover your instance doesn't have a public IPv4 address? Your subnet configuration may be the reason for this issue. In this quick tip, I will explain why this happens and show you how to control this behavior.

When launching an EC2 instance from the AWS portal, you need to specify how a public IP address gets assigned to the instance. There are a few options, but the one I want to focus on here allows AWS to auto-assign an IPv4 address. This option pulls an IP address from Amazon's public IP address pool and assigns it to your instance.

The auto-assign option is set via the launch wizard's auto-assign Public IP setting, as shown in the image below. There are three values to choose from, "Use subnet setting", "enable", or "disable".

The "enable" and "disable" values do exactly what you would expect; enable or disable the auto-assign functionality. Disabling the auto-assign property is useful when the EC2 instance shouldn't be publicly available or maybe you will assign an Elastic IP (EIP) address to the instance. Choosing the "Use subnet setting" can lead to an instance that isn't available publicly over the Internet.

It's All About the Subnet

Notice in the image below that subnets have a property named Enable auto-assign public IPv4 address. This property configures the subnet's auto-assign behavior.

An instance launched with the "Use subnet setting" value instructs AWS to apply the IP address assignment behavior as configured at the subnet level.

Subnet Behavior

Subnets created by AWS are called default subnets. These subnets have their auto-assign property set to true by default. Subnets you create, called non-default subnets, set the property's value to false by default. The one exception to this rule is a subnet created by the Instance Launch Wizard. The wizard sets the auto-assign property to true.

If you select the default subnet at the time of instance creation and choose the "Use subnet setting" option, the instance will have a public IPv4 address assigned. However, if you choose a non-default subnet, that instance may not get a public IP address. It all depends on how you configured your subnet to use the auto-assign functionality.

AWS provides an API to modify the subnet's auto-assign property. You can use the AWS CLI to enable or disable the property.

To enable auto-assign:



aws ec2 modify-subnet-attribute --subnet-id <your-subnet-id> --map-public-ip-on-launch

To disable auto-assign:



aws ec2 modify-subnet-attribute --subnet-id <your-subnet-id> --no-map-public-ip-on-launch

Changing the value does not affect existing instances. It only applies to future instances created within the subnet. The AWS portal can be used to modify this setting as well.

In this quick tip I wanted to point out the behavior of the auto-assign property, both at the instance, and more so at the subnet level. Not enabling the auto-assign option at the subnet level may lead to the creation of an EC2 instance that doesn't have a public IP address, forcing you to recreate the instance after you've modified the subnet properties.

Use Terraform Workspaces for Multi-Environment Deployments

Paul Delcogliano — Tue, 31 Aug 2021 15:12:02 +0000

Over the course of my brief time with Terraform, I've come to appreciate its power and relative ease of use over other Infrastructure as Code (IaC) tools. As an IaC noob, I have had to overcome many challenges that novices have stumbled over.

One such block I stumbled over involved applying the same configuration to different environments, i.e. dev, prod. Under certain conditions, when you apply the same configuration to multiple environments Terraform first destroys the previous resources before applying the changes to the new environment.

This can happen when you use variables as part of a resource name. Terraform sees changes to resource names as a reason to destroy a resource. These types of changes are destructive because you can't modify a resource's name in AWS. Instead, you have to delete and recreate the resource. Terraform is simply following the AWS rules.

In my case, I was naming my resources with the environment name appended to the resource name via a string interpolation template like user_${var.environment} This would give me names like "user_dev" or "user_prod". Terraform picked up the change in name and marked the IAM user resource for destruction when I ran the configuration for different environments.

I came to understand how Terraform stores this information in "state". I learned how Terraform Workspaces could be used to overcome this behavior. In this post I am going to describe what I learned and will demonstrate one method you can leverage using Workspaces to build your infrastructure in AWS for multiple environments.

A State of Destruction

Before diving into Workspaces, it's important to set some background about how Terraform saves configuration state.

Terraform stores data about a configuration in a local file named "terraform.tfstate". Terraform stores data like the values from outputs, dependencies, secrets, and AWS IDs of the resources it creates. As a side note, sensitive data is stored in plain text in the state file. You should protect those values at all costs. I'll do a future post talking about securing state data and state management.

Terraform uses state during subsequent apply operations to compare the previous plan to the current one and determine any deltas. Examining the deltas, Terraform knows whether or not it can update a resource, or if it must destroy the resource in order to apply the change.

When I was applying my configuration to multiple environments using variables, Terraform was capturing all changes to a single state file. So when the user name changed from user_dev to user_prod, Terraform saw that as a change to the user_dev resource and marked it for destruction. In actuality, what I wanted it do was create a new account named user_prod and not modify the existing user_dev account.

What I needed was a way to store state for each environment separately. This would allow me to apply my configuration to each environment independent of the others, and avoid the destruction of resources created in other environments. This is where Workspaces enter the picture.

Enter the Workspace

Terraform uses the concept of a Workspace for isolating state. When you initialize a configuration, Terraform creates a default Workspace named, "default". Configuration plans, and other data, are stored in the Workspace.

You can have multiple named Workspaces in Terraform. Each Workspace maintains its own state. This allows for multiple states to be associated with a single configuration. Knowing this, I set up one Workspace for each environment. Now I could apply the same configuration to different environments without affecting any other environments!

To create and manage a Workspace, use the terraform workspace set of commands as shown in the table below.

Command	Description
`terraform workspace new <workspace name>`	creates and switches to the newly created workspace
`terraform workspace select <workspace name>`	switches to the workspace
`terraform workspace list`	lists workspaces and highlights the active one

Let's use the following simple configuration to see Workspaces in action. The configuration below creates a read-only policy for all S3 buckets. It defines one variable named "environment", whose default value is "dev". The variable value is appended to the policy name to create a unique policy per environment.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.27"
    }
  }
  required_version = ">= 0.14.9"
}
provider "aws" {
  profile = "default"
  region  = "us-east-2"
}

variable "environment" {
  type    = string
  default = "dev"
}

resource "aws_iam_policy" "s3_policy" {
  name = "s3_policy_for_${var.environment}"

  policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Action" : "s3:GetObject",
        "Resource" : "*"
    }]
  })
}

Apply this configuration using the following command:

terraform apply

This creates the policy in AWS and names it "s3_policy_for_dev". Let's run it again, but this time provide a different value for the environment variable, like so:

terraform apply -var "environment=prod"

Our intention is to create a new policy named "s3_policy_for_prod", but as you can see from the command's output, Terraform sees the name change and decides it needs to replace the "s3_policy_for_dev" policy and create a new policy.

Let's resolve this by creating a Workspace for the dev environment. Run the following command to create a Workspace named "dev"

terraform workspace new dev

Reviewing the output you notice Terraform created and switched to a new Workspace named "dev". More importantly the output states, "...Terraform will not see any existing state for this configuration". Our changes to the dev environment will now be stored within the dev Workspace.

Remember what I said earlier about Workspaces having their own state? The dev Workspace is unaware of the previous apply command we executed. If you run the terraform plan command in the dev Workspace, the output will show you that Terraform is going to attempt to create a resource named "s3_policy_for_dev". This will fail upon execution because there is already a resource using that name in AWS.

Using the AWS portal, go ahead and delete the "s3_policy_for_dev" policy from AWS, then run the terraform apply command from the dev Workspace. This will synchronize the Workspace state with the resources in AWS.

Now, let's create the Workspace for our production configuration:

terraform workspace new prod

Create the policy for the production environment:

terraform apply -var "environment=prod"

This time, the Terraform output shows one new resource will be created and no other changes will be made. No resources will be destroyed because the separate state file doesn't know about the resources created in the dev Workspace.

Review your policies in AWS via the portal. You should see two policies, one named "s3_policy_for_dev" and the other named "s3_policy_for_prod".

In addition, if you review your local folder structure where your Terraform files are located, you'll see a folder named terraform.tfstate.d. Expand that folder to find subfolders, one for the dev Workspace and one for prod. Within each subfolder you will find the local state file associated with its parent Workspace.

Issue the terraform workspace list command to show all of the available Workspaces.

The Workspace marked with the asterisk is the currently selected Workspace. Issue the following command to switch back to the dev Workspace:

terraform workspace select dev

Conclusion

Terraform Workspaces help you to navigate issues that may arise from having a single state store. Isolating state within a Workspace provides additional options for multi-environment deployments. I recommend you use Workspaces in scenarios like these or perhaps segmenting your deployments by categories, i.e. networking, security, auditing, etc. Hit me up in the comments and let me know how you apply Workspaces to your configurations...and stay tuned for a post discussing state management.

Improve Your Web API with Swagger Documentation

Paul Delcogliano — Mon, 15 Mar 2021 17:13:30 +0000

This post is one in a series of posts I am using to further enhance my development skillset. In this post, I will describe how to improve the Mural API (my reference application used for educational purposes) using Swagger documentation. Swagger is a set of tools based upon the OpenAPI Specification (OAS) which are used for documenting Web APIs. You may be wondering why documenting your API is necessary. Well, good documentation contributes to the overall user experience and is one of the biggest factors for increased API growth and usage.

ASP.Net Core uses Swashbuckle, which is an open-source Swagger implementation used for generating API documentation. Through Swashbuckle you will generate living documentation every time you build your API, keeping the documentation in sync with the latest version of your API.

Basic Swagger Documentation

The first thing we need to do is install Swashbuckle into the API project. Swashbuckle is available as a NuGet package. I am using PowerShell to install the NuGet package.



Install-Package Swashbuckle.AspNetCore -Version 5.6.3

Swagger functionality is injected as middleware into the ASP.Net pipeline. Open the API project's Startup.cs file and edit the ConfigureServices() method. Add the line as shown below.



services.AddSwaggerGen(c =>
{
  c.SwaggerDoc("v1", new OpenApiInfo { Title = "Mural", Version = "v1" });
});

While still editing the Startup.cs file, locate the Configure() method and enable middleware for the Swagger UI by adding useSwagger and useSwaggerUI.



public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
  app.UseSwagger();
  app.UseSwaggerUI(c =>
  {
    c.SwaggerEndpoint("/swagger/v1/swagger.json", "Mural V1");
  });
  /// rest of method body

Save the Startup.cs and run the project. Swashbuckle includes an implementation of the Swagger UI, which is a Web page that provides interaction and visualization to an API's resources. Navigate to the Swagger UI page by going to localhost:44333/swagger/index.html. Note, your local port may vary. When you open this page you will see the basic documentation that comes out of the box from Swagger.

Nice! With just a few lines of code and a NuGet package, we have a documented API. But, we can enhance the documentation with additional information, like examples for calling the API methods and descriptions of the method parameters. In the next section, we'll enrich the documentation to create an even better user experience.

Providing Enhanced Documentation

We can enrich the API's documentation by including XML comments in the code. You place these comments directly before the code block about which you are commenting. The Swashbuckle tooling automatically includes XML comments in its documentation and makes them available to view via the Swagger UI Web page.

The first thing we need to do is to enable XML Comments in the project. XML comments can be enabled via several approaches. I chose to use the Project Properties Dialog as shown below.

Using the dialog, navigate to the "Build" tab and click on the checkbox next to "XML Documentation file". Provide a name for the file, such as "M-url.Api.xml". Of note, when you enable XML Documentation, your code will generate warnings for any method that does not contain XML comments. If you want to disable this compiler warning, include 1591 in the Suppress warnings textbox.

Next, we need to add code to the Startup.cs file to inform Swagger where it can find the XML comments file. Replace the call to AddSwaggerGen we added earlier with the code shown below:



services.AddSwaggerGen(c =>
  {
    c.SwaggerDoc("v1", new OpenApiInfo { Title = "Mural", Version = "v1" });

    // generate the XML docs that'll drive the swagger docs
    var xmlFile = $" 
      {Assembly.GetExecutingAssembly().GetName().Name}.xml";
    var xmlPath = Path.Combine(AppContext.BaseDirectory, xmlFile);
    c.IncludeXmlComments(xmlPath);
  });

Now we'll add comments to the code. We'll start identifying which content-types the API handles on requests and responses. You use the Produces and Consumes attributes on the controller class to specify the content types allowed by the API.



[Produces(MediaTypeNames.Application.Json, 
  MediaTypeNames.Application.Xml)]
[Consumes(MediaTypeNames.Application.Json, 
  MediaTypeNames.Application.Xml)]
public class SlugCollectionsController : ControllerBase
{ ... }

In the example above, the SlugsController accepts and returns both JSON and XML. This information will appear in the Swagger documentation.

Here I've added comments to the GetSlugCollection() method:



/// <summary>
/// Returns a collection of URLs
/// </summary>
/// <param name="slugs">list of slugs to retrieve</param>
/// <remarks>
/// Sample request:
///
///     Get /api/slugscollection/(d25tRx, fN5jpz)
///
/// </remarks>
/// <returns>IEnumerable of slugs</returns>
/// <response code="200">If all requested items are 
/// found</response>
/// <response code="400">If slugs parameter is missing</response>
/// <response code="404">If number of records found doesn't equal 
/// number of records requested</response>
[HttpGet("({slugs})", Name = "GetSlugCollection")]
[ProducesResponseType(StatusCodes.Status200OK)]
[ProducesResponseType(StatusCodes.Status400BadRequest)]
[ProducesResponseType(StatusCodes.Status404NotFound)]
public async Task<ActionResult> GetSlugCollection([ModelBinder(BinderType = typeof(ArrayModelBinder))] IEnumerable<string> slugs)
{ ... }

While many of the comments are self-explanatory, I do want to highlight a few of them. The following table details the interaction between each XML comment attribute and its effect on the Swagger documentation.

Comment Attribute	Swagger UI Action
summary	displays text next to the action method name
remarks	Supplements information specified in the element. Can consist of text, JSON, or XML
response	Renders as an example HTTP request in the Swagger documentation.

Looking at the code snippet above, you'll notice the additional attribute, ProducesResponseType has been added to the method. ProducesResponseType informs Swagger of the HTTP Status codes returned by the method. The response XML comment works in conjunction with the ProducesResponseType attribute to provide descriptive information for each HTTP status code the method returns.

Alternatively, in place of specifying each individual status code using ProducesResponseType, you can use the ApiConventionMethod attribute to specify 200, 400, and 404 status codes if your API produces a standard set of response codes. Here's an example:



[ApiConventionMethod(typeof(DefaultApiConventions))]

The final version of the GetSlugCollection method's documentation is below. You can see the additional descriptions next to the parameters, the HTTP status codes being returned, and an example of calling the API using a GET HTTP method.

There is one more change we can do to make the documentation complete. Back in the Startup.cs file, replace the call to AddSwaggerGen we modified earlier with the souped-up version shown below:



services.AddSwaggerGen(c =>
{
  c.SwaggerDoc("v1", new OpenApiInfo
  {
    Title = "Mural API",
    Description = "",
    Contact = new OpenApiContact
    {
      Name = "Paul Delcogliano",
      Email = "pdelco@gmail.com",
      Url = new Uri("https://github.com/pdelcogliano")
    },
    License = new OpenApiLicense
    {
      Name = "MIT License",
      Url = new Uri("https://opensource.org/licenses/MIT")
    }
  });

  // generate the XML docs that'll drive the swagger docs
  var xmlFile = $" 
    {Assembly.GetExecutingAssembly().GetName().Name}.xml";
  var xmlPath = Path.Combine(AppContext.BaseDirectory, xmlFile);
  c.IncludeXmlComments(xmlPath);
});

In this version, we are using the OpenApiInfo() method to provide additional data for the document header. We are also using the OpenApiLicense() method to provide information about the API license. Here's the final Swagger documentation, shown in all of its glory with enhanced header information, and enriched method descriptions:

Documenting an ASP.Net Core API using Swashbuckle and Swagger helps your API meet the goals of good documentation. Good documentation provides many benefits. It helps internal teams understand the API and agree on its attributes. It facilitates external clients' understanding of the API and the functionality it provides. More information about getting started with Swagger and Swashbuckle can be found here.

Building Your First API with Asp.Net Core

Paul Delcogliano — Mon, 15 Mar 2021 14:10:13 +0000

This is an introduction to a series of posts I am writing. The series describes how to build an API with ASP.Net Core. The series' purpose is to help me while I learn new technologies and to share what I learn with others.

I believe I have a unique perspective to share. You see, I've been a developer for a long time, but I haven't kept up with technology trends over the past several years. I am now starting a journey to refresh my skillset. In this series, I will apply all I've learned as a "maintenance developer" to today's newer frameworks and technologies, like ASP.Net Core, git, Azure, security, and many more.

My learning tool will be an API I created, and loving call, "Mural". You can find information about Mural here.

Mural is a simple API that creates short URLs from deep links. I intend to apply today's best practices to Mural to create a full feature, secure, scalable, and cloud-ready API. You can download Mural from my GitHub and follow along my journey with me.

DEV Community: Paul Delcogliano

How to Use Terraform Conditional Expressions

What is a conditional expression in terraform

What is the Terraform ternary operator

Conditional expression use cases

Create a resource using a conditional expression

Conditional expressions on other Terraform objects

Multiple Conditions

Wrapping it up

Terraform vs. Pulumi : A Look at Both Tools

Terraform

Pulumi

Feature Comparison Between Terraform and Pulumi

Language Support

IDE Features

Open Source Licenses

Testability

Cloud Provider Support

State Management

Secrets Management

Infrastructure Reuse

Terraform or Pulumi: Which Should You Choose?

Conclusion

How to Use Terraform `depends_on`

Resource Dependencies

What is Terraform depends_on

Use Cases for depends_on

Adding Multiple Dependency Objects

How to use depends_on for modules in Terraform

Check Your IaC Deployment with the Terraform `plan` Command

Usage Examples

Planning Modes

Planning Options

Options Affecting How the Plan Command Behaves

Options Related to Formatting and Output

Resource Targeting

Conclusion

Practice DRY in Terraform Configurations Using a Dynamic Block

How to Use the Dynamic Blocks

Example

Dynamic Block Components

Key Points

An Introduction to DevOps with Azure DevOps

Planning

Developing

Delivery

Operating

What is Azure DevOps?

Azure DevOps Services

Organization

Projects

Boards

Repos

Pipeline

Test Plans

Artifacts

Azure DevOps Benefits/Advantages

Azure DevOps Alternatives

Automate EC2 Instance Shutdown with Auto Scaling Groups to Minimize AWS Costs

Launch Templates and ASG

Step 1: Create a Launch Template from an EC2 instance

Step 2: Create the Auto Scaling Group

Step 3: Create schedules to scale the ASG

Logging and Monitoring

Conclusion

Automate EC2 Instance Shutdown with Lambdas to Minimize AWS Costs

Controlling Costs

CloudWatch Event Rules and Lambda

Step 1 - Create IAM Policy and Role

Listing 1

Step 2 - Create two Lambda functions

Listing 2

Listing 3

Step 3 - Create CloudWatch Event Rules

Logging

Conclusion

AWS Subnet Tip: Using the Auto-Assign Attribute

It's All About the Subnet

Subnet Behavior

Use Terraform Workspaces for Multi-Environment Deployments

What is Terraform `depends_on`

Use Cases for `depends_on`

How to use `depends_on` for modules in Terraform