DEV Community: Alexey Ryazhskikh

Using GitHub Copilot for Terraform code refactoring

Alexey Ryazhskikh — Sun, 23 Nov 2025 18:36:19 +0000

Terraform and OpenTofu are the only major IaC tools today that provide a fully deterministic, declarative, provider-agnostic, predictable execution plan across AWS + Azure + other cloud platforms.

However, the Terraform development experience is still close to what we had for JavaScript in 2008: we have syntax highlighting, basic static analysis, and basic code navigation. But the Terraform code refactoring still should be done manually. We still have no button "extract resources to the module" in JetBrains IDE or VS Code. Mainly because in the Terraform world, code refactoring should go together with state transformation.

By the end of 2024 in my team had written a lot of Terraform code. We have learned how to master infrastructure as code using Terraform, our expertise has grown up and the code written two years ago become ugly to us. But state transformations and the complexity of code change stopped us from starting to do refactoring on a large scale. For example, we have states with 6K resources.

We tried to use multiple ways to do state transformations:

Created Python scripts to generate Terraform moved blocks for the particular refactoring.
Tried to use Terraform file templates and construct object maps to pass old and new resource structure.

All these approaches worked, but required a lot of effort, especially for huge states and code base.
Everything changed when GitHub Copilot introduced Edits mode. We started actively using it for Terraform code refactoring, and even then, I realized that Copilot can solve our code quality issues faster.
GitHub Copilot Agents automated the refactoring now we use the following algorithms:

Ask Copilot for the Terraform code changes. Here is very important to limit the change scope and do refactoring fraction by fraction. Better to have multiple iterations for refactoring to not lose control on changes.
The plan for the small changes is much easier to review. The most frequent refactoring we do are to extract resources to the module and change the module structure.
Run plan and save output to the file terraform plan -no-color > plan.txt. We need the plan saved to add it to the context. In Agent mode Copilot also able to use shell commands like grep to analyze a plan.
Generate plan summary and validate if plan meet refactoring requirements. We use that step if we need to work with huge states. If the plan contains 500 changed resources, it is difficult to say if the change is valid. The plan summary allows for reviewing types of affected resources and correlating them with the initial change.
Generate moved.tf file with moved blocks to avoid any changes in the next plan. On this step, the agent can do multiple iterations for moving blocks generation and runing the Terraform plan. I would recommend to review moved.tf file because in complex cases LLM could do a wrong correlation between resources and generate move valid from Terraform point of view, but not from the purpose. For example:
```
moved {
  from = aws_iam_user[1].team_two_user
  to   = module["team1"].aws_iam_user.user
}
moved {
  from = aws_iam_user[2].team_one_user
  to   = module["team2"].aws_iam_user.user
}
```

Final Thoughts

In 2025, Terraform remains the strongest cloud‑agnostic Infrastructure‑as‑Code tool. Pulumi is easier to develop with, but it is not as deterministic as Terraform’s declarative model.
Modern AI assistants like GitHub Copilot and Cursor dramatically accelerate safe refactoring and reduce the fear of touching legacy state.
With the help of AI‑driven tooling, Terraform has become a more powerful IaC solution than at any previous point in its history.

Multi-environment infrastructure with terraform variables files

Alexey Ryazhskikh — Sat, 23 Nov 2024 18:36:19 +0000

In our company we have thousands of resources managed by Terraform. Which are deployed to multiple environments (dev, staging, production) and different regions.

The key principles we have for our Terraform codebase are:

Use the same Terraform codebase (.tf files) for all environments (dev, stage, prod).
All environment specific settings should be managed via Terraform variable files (.tfvars).

Below is an our typical Terraform codebase structure:

src/
├── environments/
│   ├── dev.tfvars
│   ├── stage.tfvars
│   └── prod.tfvars
├── variables.tf
├── db_server.tf
├── main.tf
├── terraform.tf
├── providers.tf
└── ...

variables.tf file contains the variables definitions:

variable "resource_group_name" {
  description = "The resource group name to deploy db server"
  type = string
}

variable "location" {
  description = "The db server location"
  type = string
}

variable "enable_replication" {
  description = "Enable DB replication of resources to other region"
  type = bool
}

src/environments/dev.tfvars contains the environment specific settings:

resource_group_name = "rg-dev"
location = "eastus"
enable_replication = false

Each environment has corresponding terraform state. So we need to specify the state file and .tfvars file to run terraform apply command for specific environment:

terraform apply -var-file="src/environments/dev.tfvars" -state="dev.tfstate"
terraform apply -var-file="src/environments/stage.tfvars" -state="stage.tfstate"
terraform apply -var-file="src/environments/prod.tfvars" -state="prod.tfstate"

If you use terraform cloud, you probably need to specify workspace name with TF_WORKSPACE environment variable instead of state file.

Feature flags

The Terraform variables file can also store feature flags together with terraform modules.

This approach allows to test the Terraform code in the dev and stage environment before it is applied to other environments. If staging and production environments have the same settings we can have the same code coverage for production.

Terraform has no if-else logic, so the only way to implement feature flags is to use for_each and count statements.

In the following example we create an azure resource group and role assignments if the enable_replication variable is true:

variable "enable_replication" {
  type = bool
}

resource "azurerm_resource_group" "replica_rg" {
  count = var.enable_replication ? 1 : 0
  name     = "replica-rg"
  location = var.location   
}

resource "azurerm_role_assignment" "role_assignments" {
  count = var.enable_replication ? 1 : 0
  scope                = azurerm_resource_group.replica_rg[0].id
  role_definition_name = "Contributor"
  principal_id         = "12345678-1234-1234-1234-123456789"  
}

This approach is not perfect, because count condition should be added to each dependent resource. Better to group dependent resources into the local module, to have a single count condition for entire module. For example:

variable "enable_replication" {
  type = bool
}

module "replica_rg" {
  source = "./modules/replica_rg"
  count = var.enable_replication ? 1 : 0
  rg_name     = "replica-rg"
  contributor_id = "12345678-1234-1234-1234-123456789"
}

Terraform variables as DSL

The terraform variables definitions becomes another layer of abstraction: instead of defining particular resources we define business entities and feature settings.
In fact, .tfvars files management becomes programming on DSL language defined by terraform variables blocks..
For example, the definition for CI/CD build agents:

variables.tf:

variable "build_agents" {
  description = "Build agents settings"
  type = map(object({
    number_of_vms = number
    vm_size = optional(string, "Standard_N2_v2")
    private_network_access = optional(bool, false) # limit access to agent vms
  }))
}

dev.tfvars:

build_agents = {    
    build_pool = {
        number_of_vms = 10
        vm_size = "Standard_D2_v2"
        private_network_access = false
    }
    deployment_pool = {
        number_of_vms = 5
        vm_size = "Standard_D2_v2"
        private_network_access = true
    }
}

Here we are not defining azure resources, but our infrastructure assets, which, in fact could be implemented differently.

Control interface

The .tfvars approach allows to define the control interface for infrastructure operators.
So the settings in .tfvars are working like control panel in pilot cockpit hiding the underlying resources and their dependencies.
For example, if ci/cd admin needs to increase number of agents he don't need to search for resources he needs to reconfigure in terraform codebase. He just changes the number_of_vms in .tfvars file.

Refactoring

Naming for the terraform variables and object properties is a challenge.
Time to time we need to do refactoring of variables to change objects structure, or introduce the new properties for all objects. Which also leads changes in all .tfvars files and states.

Normally, such refactoring has the following steps:

Modifying variables definitions in variables.tf file.
Modifying all .tfvars files.
Modifying terraform code to support the changes.
Generating moved blocks in terraform code.

For .tfvars modification and code generation you can use python libraries like
python-hcl2.
Unfortunately, hcl2 parsers are not available for many other languages, so previously I converted .tfvars to json and used json as an intermediate format.
I used this go application which is a wrapper over official Hashicorp hcl2 go library: https://github.com/musukvl/tfvars-parser

Recently I created my own C# dotnet library to work with .tfvars files: amba-tfvars.
The library focused on .tfvars file refactoring.
It can extract not only terraform variables data, but also code comments from .tfvars files, which could be very important to keep during the .tfvars files transformation.
Sometimes it is important to keep original formatting so the library collects information about original maps and lists code style: if they were one-liners, or each property has its own line.

Conclusion

I think the .tfvars files approach is a good way to manage multi-environment Terraform codebase for huge projects. It allows naturally to implement feature flags and truck based development for Infrastructure as Code.

The article repository: https://github.com/musukvl/article-terraform-tfvars-infro

zap log level codes

Alexey Ryazhskikh — Wed, 08 May 2024 13:29:07 +0000

Uber Zap logger level codes:

Name	Value
Debug	-1
Info	0
Warn	1
Error	2
DPanic	3
Panic	4

For details check: https://github.com/uber-go/zap/blob/cf2f580374cbb923a87d62449ce50d7712b75fa1/level.go#L30

Resource identification in Terraform

Alexey Ryazhskikh — Thu, 29 Jun 2023 14:00:48 +0000

Resource identification in Terraform

Basic concepts

Each resource created with Terraform present on three levels:

Resource has definition in .tf-file code as resource block.
Created resource has record in Terraform state file.
Resource is also actual resource created in a cloud.

On each level resource has its own identification.
You need clearly understand how resource identified on each level to avoid unexpected resource recreation and data loss.
Proper resource identifiers also make your code maintainable and readable.

The code examples in this article are based on Azure provider. But the same concepts are applicable for other providers.
The link to code examples: https://github.com/musukvl/article-terraform-resource-identification

Resource identifier in case of single resource

Let's check the simple example of azure storage account created with Terraform and track resource identification on each level:

Resource definition in .tf-file code level is:

resource "azurerm_storage_account" "application_storage" {}

It has resource type azurerm_storage_account and resource name application_storage. In code the storage account can be referenced as azurerm_storage_account.application_storage.

In the terraform state the storage account is identified with type and name fields:

"resources": [
  {
    "mode": "managed",
    "type": "azurerm_storage_account",
    "name": "application_storage",
    "provider": "provider[\"registry.terraform.io/hashicorp/azurerm\"]",
    "instances": [
      {
        "schema_version": 3,
        "attributes": {
            "id": "/subscriptions/xxxxx/resourceGroups/ary-app-rg/providers/Microsoft.Storage/storageAccounts/aryappsa",
            "name": "aryappsa",
            ...
        }  
      }
    ] 
  } 
]

Full example of state file is here.

Actual storage account has /subscriptions/xxxxx/resourceGroups/ary-app-rg/providers/Microsoft.Storage/storageAccounts/aryappsa id in Azure cloud.

Terraform matching resource on each level by corresponding identifier.

For example, if you change name of the resource in *.tf-file:

  resource "azurerm_storage_account" "application_storage_NEW" {}

During the terraform plan operation:

Terraform realizes that it has no matching code definition for resource in the state with type azurerm_storage_account and name application_storage. So Terraform will destroy resource in Azure and remove it from state.
Terraform realizes that it has no matching record in the state for resource defined in the code with identifier azurerm_storage_account.application_storage_NEW. So Terraform will create it in Azure and add record to the state.

Identifiers vs attributes

Keep in mind, that name attribute of azurerm_storage_account and name of Terraform resource block are different things.

The name attribute of azurerm_storage_account is not part of resource identifier.
Changing attributes could cause resource recreation in some cases and depends on resource provider, but changing identifier always cause resource recreation.

Resource identifier in case of `for_each`

If for_each is used in resource definition, multiple instances of resources will be created. Each instance can be addressed by index key.

For example, let's check the levels for azure storage account resource in for_each case example

Resource definition in .tf-file code is:

locals {
  application = {
    "document-parser-service" = {
    storage_account_name = "arydocparsesvc"
    }
    "email-sender-service" = {
    storage_account_name = "aryemailsendersvc"
    }
  }
}

resource "azurerm_storage_account" "application_storage" {
  for_each = local.application

It has resource type azurerm_storage_account and resource name application_storage. In code the storage account can be referenced with index key: azurerm_storage_account.application_storage["document-parser-service"].

In the state we will see two instances of storage account resource:

resources: [
{
  "mode": "managed",
  "type": "azurerm_storage_account",
  "name": "application_storage",
  "provider": "provider[\"registry.terraform.io/hashicorp/azurerm\"]",
  "instances": [
    {
      "index_key": "document-parser-service",
      "attributes": {
        "id": "/subscriptions/xxxxx/resourceGroups/ary-app-rg-document-parser-service/providers/Microsoft.Storage/storageAccounts/arydocparsesvc",
        "name": "arydocparsesvc",
        ...
      }
    },
    {
      "index_key": "email-sender-service",
      "attributes": {
        "id": "/subscriptions/xxxxx/resourceGroups/ary-app-rg-email-sender-service/providers/Microsoft.Storage/storageAccounts/aryemailsendersvc",
        "name": "aryemailsendersvc"            
        ...
      }
    }        
  ]
}
]

Full example of state file is here.

Actual storage accounts have /subscriptions/xxxxx/resourceGroups/ary-app-rg/providers/Microsoft.Storage/storageAccounts/aryappsa and /subscriptions/xxxxx/resourceGroups/ary-app-rg/providers/Microsoft.Storage/storageAccounts/aryemailsendersvc ids in Azure cloud.

Resource identifier in case of `count`

The count statement works pretty similar to for_each except resource idex is array index number instead of map key string.

Let's check count case example the same as for_each case example:

Resource definition in .tf-file code is:

locals {
  applications = [
    {
      name = "document-parser-service"
      storage_account_name = "arydocparsesvc"
    },
    {
      name = "email-sender-service"
      storage_account_name = "aryemailsendersvc"
    }
  ]
}

resource "azurerm_storage_account" "application_storage" {
  count = length(local.applications)

In code the storage account can be referenced with index key: azurerm_storage_account.application_storage[0].

In the state we will see two instances of storage account resource:

resources: [
{
  "mode": "managed",
  "type": "azurerm_storage_account",
  "name": "application_storage",
  "provider": "provider[\"registry.terraform.io/hashicorp/azurerm\"]",
  "instances": [
    {
      "index_key": 0,
      "attributes": {
        "id": "/subscriptions/xxxxx/resourceGroups/ary-app-rg-document-parser-service/providers/Microsoft.Storage/storageAccounts/arydocparsesvc",
        "name": "arydocparsesvc",
        ...
      }
    },
    {
      "index_key": 1,
      "attributes": {
        "id": "/subscriptions/xxxxx/resourceGroups/ary-app-rg-email-sender-service/providers/Microsoft.Storage/storageAccounts/aryemailsendersvc",
        "name": "aryemailsendersvc"            
        ...
      }
    }        
  ]
}
]

Full example of state file is here.

Storage accounts has the same IDs and names in azure as in for_each case example, but the difference identification in state file and in code.

Using int idex instead of string key is not very convenient, so for_each is preferred over count in most cases.

Conclusion

Understanding of resource identification is very important in case of shared terraform modules and for cases when resources can't be easily recreated for refactoring purposes.

Terraform resource dependency graph

Alexey Ryazhskikh — Mon, 03 Apr 2023 05:41:09 +0000

This article demonstrates how to manage the creation and updating of Terraform resources by properly configuring their dependencies, using practical examples.
The code examples, available in this GitHub repository utilize Azure provider resources, but the concepts discussed apply to any provider.

TL;DR

During planning phase terraform evaluates dependencies between resources and builds a dependency graph. So terraform ensuring the proper order and parallelism for resource change operations.

Use resource and module outputs to define dependencies between resources.
Use for_each for collecting outputs from created resources.
Use depends_on with caution, as it can lead to resource recreation due to minor changes in related resources.
Use the terraform graph command to review resource dependencies and refactor them.
Assign proper resource identifiers wisely to be able using it for lookup.

Using resource output to make dependency

The following example defines Azure resource group and storage account in it:

locals {
  location = "northeurope"
  resource_group_name = "ary-graph-example-rg"   
}

resource "azurerm_resource_group" "graph_example_rg" {
  name     = local.resource_group_name
  location = local.location
}

resource "azurerm_storage_account" "storage_account" {
  name                     = "arygrexstacc"
  resource_group_name      = local.resource_group_name
  location                 = local.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

(Check code in 001-local-for-name)

The terraform plan command for this example does not return any errors.
But if you run terraform apply you might see "Resource group 'ary-graph-example-rg' could not be found" error.

│ Error: creating Azure Storage Account "arygrexstacc": storage.AccountsClient#Create: Failure sending request: StatusCode=404 -- Original Error: Code="ResourceGroupNotFound" Message="Resource group 'ary-graph-example-rg' could not be found."    
│
│   with azurerm_storage_account.storage_account,
│   on main.tf line 11, in resource "azurerm_storage_account" "storage_account":
│   11: resource "azurerm_storage_account" "storage_account" {

The apply error happends because terraform creating resource group and storage account in parallel. (Default parallelism for terraform apply is 10).
Here is a work around for such problem: you can run terraform apply again and storage account will be created because resource group created before. It would work, but you might have problems with day zero provisioning when you need to recreate everything from scratch.

The right solution is to define relation between resource group and storage account.

It is possible to generate depenency graph with terraform graph command and draw image with Graphviz (DOT):

terraform graph | dot -Tpng > graph.png

The graph shows, that there is no dependency between resource group and storage account, but they both dependend on local.location variable.

The fix is use resource group output azurerm_resource_group.graph_example_rg.name instead of local variable:

locals {
  location = "northeurope"
  resource_group_name = "ary-graph-example-rg"   
}

resource "azurerm_resource_group" "graph_example_rg" {
  name     = local.resource_group_name
  location = local.location
  tags     = {
    "asset-owner" = "Mark"
  }
}

resource "azurerm_storage_account" "storage_account" {
  name                     = "arygrexstacc"
  resource_group_name      = azurerm_resource_group.graph_example_rg.name # <- resource output used for dependency
  location                 = local.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

(Check code in 002-output-dependency)

Now the graph shows that resource group and storage account are dependent on each other.

Using depends_on to make dependency

Using 'depends_on' is way to enforce dependency between resources. It is not recommended to use it, because it can lead to resource recreation because of minor change in related resource.

In the following example, resource group tag change, causes recreation of storage account:

locals {
  location = "northeurope"
  resource_group_name = "ary-graph-example-rg"   
}

resource "azurerm_resource_group" "graph_example_rg" {
  name     = local.resource_group_name
  location = local.location
  tags     = {
    "asset-owner" = "Mark"
  }
}

data "azurerm_resource_group" "graph_example_rg" {
  depends_on               = [azurerm_resource_group.graph_example_rg]
  name = local.resource_group_name
}

resource "azurerm_storage_account" "storage_account" {
  depends_on               = [data.azurerm_resource_group.graph_example_rg]
  name                     = "arygrexstacc"
  resource_group_name      = data.azurerm_resource_group.graph_example_rg.name
  location                 = data.azurerm_resource_group.graph_example_rg.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

(Check code in 003-depends_on)

For example in case of changing tag value from "Mark" to "Mark1" terraform will generate the following plan:

 # azurerm_resource_group.graph_example_rg will be updated in-place
  ~ resource "azurerm_resource_group" "graph_example_rg" {
        id       = "/subscriptions/5293af6a-eac6-493f-8d6f-e6358448a2ff/resourceGroups/ary-graph-example-rg"
        name     = "ary-graph-example-rg"
      ~ tags     = {
          ~ "asset-owner" = "Mark" -> "Mark1"
        }
        # (1 unchanged attribute hidden)
    }

  # azurerm_storage_account.storage_account must be replaced
-/+ resource "azurerm_storage_account" "storage_account" {
      ~ access_tier                       = "Hot" -> (known after apply)
      ~ id                                = "/subscriptions/5293af6a-eac6-493f-8d6f-e6358448a2ff/resourceGroups/ary-graph-example-rg/providers/Microsoft.Storage/storageAccounts/arygrexstacc" -> (known after apply)
      + large_file_share_enabled          = (known after apply)
      ~ location                          = "northeurope" # forces replacement -> (known after apply)
        name                              = "arygrexstacc"

Plan: 1 to add, 1 to change, 1 to destroy.

(Check code in 003-data-resource/update-plan.txt)

Terraform graph refactoring

Using terraform graph > graph.dot command is a good way to do dependency refactoring. You can generate graph file before refactoring and after and compare them. No changes in the graph means that you did not break anything.
You can visualize graph with Graphviz (DOT) tool: terraform graph > graph.dot

The previous example can be refactored to not use depends_on and having the same dependency graph by using resource output for dependency:

locals {
  location = "northeurope"
  resource_group_name = "ary-graph-example-rg"   
}

resource "azurerm_resource_group" "graph_example_rg" {
  name     = local.resource_group_name
  location = local.location
  tags     = {
    "asset-owner" = "Mark"
  }
}

data "azurerm_resource_group" "graph_example_rg" {  
  name = azurerm_resource_group.graph_example_rg.name   # <- resource output used for dependency
}

resource "azurerm_storage_account" "storage_account" {
  name                     = "arygrexstacc"
  resource_group_name      = data.azurerm_resource_group.graph_example_rg.name
  location                 = data.azurerm_resource_group.graph_example_rg.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

(Check code in 004-data-resource)

One to may relation example

In the following example, the resource group and storage account defined in the local.config variable.

locals {
  location = "northeurope"

  config = {
    "sample-rg1" = {
      name = "ary-graph-example-1-rg"
      storage_accounts = {
        "sa11" = {
          "name" = "arysa11grexstacc"        
        },
        "sa12" = {
          "name" = "arysa12grexstacc"        
        }
      }
    },
    "sample-rg2" = {
      name = "ary-graph-example-2-rg"
      storage_accounts = {
        "sa21" = {
          "name" = "arysa21grexstacc"        
        },
        "sa22" = {
          "name" = "arysa22grexstacc"        
        }
      }
    }
  }
}

resource "azurerm_resource_group" "graph_example_rg" {
  for_each = local.config
  name     = each.value.name  
  location = local.location
}

locals {
  storage_accounts_to_create = merge([
    for rg_key, rg in azurerm_resource_group.graph_example_rg : {
      for sa_key, sa in local.config[rg_key].storage_accounts : "${rg_key}_${sa_key}" => {
        resource_group = rg
        storage_account_name = sa.name
      }
    }
  ]...)
}

resource "azurerm_storage_account" "storage_account" {
  for_each = local.storage_accounts_to_create
  name                     = each.value.storage_account_name
  resource_group_name      = each.value.resource_group.name
  location                 = local.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

(Check code in 005-one-to-many-relation)

The essential of the example is using local variable to make dependency between created resources and local variable.
So local variable evaluated only when resource groups are created, and then storage accounts are created only when variable evaluated.

It is possible to use multiple local variables for different created resources and make join operations between them.
Using local variables for iterating created resources is also a good way to make code more readable and maintainable.

This example also demonstrates importance of resource identifiers management, in this case local.config map keys used as resource group resource identifiers, and concatenation of resource group key and storage account key used as storage account resource identifiers.

azurerm_resource_group.graph_example_rg["sample-rg1"]: Creating...
azurerm_storage_account.storage_account["sample-rg1_sa11"]: Still creating... [10s elapsed]
azurerm_storage_account.storage_account["sample-rg1_sa12"]: Still creating... [10s elapsed]

Using the same keys (rg_key) in the local.config map and as identifier of created resources allows to join created resource with the corresponding configuration in the local.config map.

for rg_key, rg in azurerm_resource_group.graph_example_rg : {
      for sa_key, sa in local.config[rg_key].storage_accounts

Conclusion

The article provides various examples to illustrate these concepts, such as using resource output to make dependencies, avoiding depends_on when possible, refactoring the Terraform graph, and managing one-to-many relationships between resources. These examples demonstrate the importance of managing resource identifiers and using local variables to make dependencies between created resources, making the code more readable and maintainable.

Using dotnet tool

Alexey Ryazhskikh — Sun, 18 Dec 2022 14:50:49 +0000

Dotnet tools - is a great technology to distribute command line tools.
For example, the following command installs EF Core CLI.

dotnet tool install --global dotnet-ef

Listing installed dotnet tools:

dotnet tool list --global

Uninstalling dotnet tool:

dotnet tool uninstall --global dotnet-ef

You can pack your own command line application as dotnet tool:

dotnet pack MyCli.csproj --configuration Release --output ./publish`
    -p:PackAsTool=true `
    -p:ToolCommandName=mytool `
    -p:Version=1.0.0

It generates MyCli.1.0.0.nupkg nuget package.
Built tool with be placed into tools\net7.0\any folder of the archive.
You can push the package into nuget repository as normal nuget package and install the tool from it.
But you can also install the tool from a local folder where nupkg file placed:

dotnet tool install MyCli --global --add-source ./publish

The dotnet tool is a part of .Net Core SDK, so you can't install your tool the environment where no dotnet SDK installed. But you can install dotnet tool to environment with SDK installed and copy it from there. Check the Andey Lock example for copying dotnet tool binaries to non-sdk docker image.

Access to GO modules in private Azure DevOps repositories

Alexey Ryazhskikh — Fri, 09 Dec 2022 08:39:47 +0000

If you tried using $(System.AccessToken) to authorize git downloading go module from you private Azure DevOps git repository, you could face with the following error:

fatal: unable to connect to dev.azure.com:

Official Microsoft documentation asks you to create PAT token for access git repoisitories:
https://learn.microsoft.com/en-us/azure/devops/repos/git/go-get?view=azure-devops#https

But you can use $(System.AccessToken) of pipeline identity instead. So you can avoid storing PAT token and renew it on expiration.

Go to Project Settings => Pipelines => Settings and make sure you have "Protect access to repositories in YAML pipelines" checkbox disabled. It will allow pipeline identities access other git repositories:

Be sure *GOPRIVATE * environment variable set to dev.azure.com

GOPRIVATE=dev.azure.com

After you added it you can execute the following commaind in your pipeline:

git config --global url."https://test:$(System.AccessToken)@dev.azure.com/<organization>/<project>/_git/<repo>".insteadOf "https://dev.azure.com/<organization>/<project>/<repo>"

Terraform basics

Alexey Ryazhskikh — Sun, 27 Nov 2022 15:07:02 +0000

The following article describes the basic terminology and concepts of Terraform and can be used as a first step in working with Terraform.

Terraform is a tool to manage your infrastructure. Basicaly it calls vendors API to create infrastructure resources.

Resources

Terraform resource is anything that can be managed via Terraform:) For example, it could be Azure resource group or AWS S3 bucket or GitHub repository, or anything else what supports terraform.

Provider

For creating Azure resources like Azure Key Vault or Azure SQL Database, Terraform needs to call Azure API. Terraform uses Azure provider for this purpose. Terraform provider is a plugin for Terraform. The provider incapsulates API Client for vendor API. You can use multipe providers in your project at the same time.

Providers are supplied by Hashicorp, API owners, or enthusiasts. You can find list of publicly available providers in Hashicorp Provider Registry.

HCL

You need to use Hashicorp configuration language (HCL) to declare resources you want to have. Instead of describing operations to create resource, you describe resources in declarative manner.
Terraform project is a set of .tf files. You can use Visual Studio Code or Jetbrains IDEA to work on it.
The following example declares Azure Resource Group:

resource "azurerm_resource_group" "resource_group" {
  location = "westeurope"
  name     = "demo-rg"
  tags     = {
    "asset-owner" = "Alexey Ryazhskikh"
  }
}

Terraform state

Terraform state is a database storing the information of existing resources. As soon as Terraform creates a resource, it writes information into the state database.
Then Terraform compares HCL declaration of resources and information about resources stored in the state.
Terraform keeps the state and the declaration synchronized. It creates missing resources, deletes resources if they were removed from declaration, etc.
The following example shows the state for the resource group described above. As you see, it stores not only resource id in Azure but also properties like name and tags.

You can trigger changes with Terraform CLI.

Terraform `plan` command

Terraform plan command calculates which resources needs to be created, updated or deleted. The plan command generates tfplan file describing all needed modifications.

Terraform `apply` command

Terraform apply command performs planned changes via Terraform providers and updates terraform state.

Terraform module

Terraform module is the way how you can declare multiple resources at once with some parameters. So you can think about the module as a function that declares a group of resources.
The following example shows a module usage example that creates an azure resource group and role assignments under the hood:

module "resource_group" {
  source   = "app.terraform.io/my-corp/resource_group/azurerm"
  version  = "1.0.0"
  location = var.location
  name     = "my-demo-rg"
  role_assignments = {
    "contributors" = {
      role_definition_name = "Contributor"
      principal_id         = "d3a2437e-15f1-4495-ba65-e10bf91578f2"
    }
  }
}

Modules allow the creation of abstraction layers in your terraform code. For example, you can create a database module to add Azure SQL database. After some time, you might need to implement backup logic for all databases you declared. You can extend your database module with storage account declaration, and all databases declared with your module will get the storage account after updating to the new version of the module.
Modules can be declared localy for each terraform project, or can be published to private or public module registry: https://registry.terraform.io/browse/modules

The Terraform downsides you should know

Alexey Ryazhskikh — Sun, 16 Oct 2022 19:20:55 +0000

Terraform is an excellent tool for infrastructure provisioning, you can find a lot of pros to starting using Terraform, but you also should know about the downside of Terraform.

Providers limitation

Terraform essentially is the way to create resources via providers. Under the hood provider calls some 3d party service API (for example, AWS API or Azure API).
So to use a new version of API, you need to have a new version of the Terraform provider. In reality, Terraform providers might not support the latest features of API. Or even some old features available via API might not be supported by the Terraform provider.
For example, Microsoft Azure DevOps API supports kubernetes resources creation, but it is not supported by terraform provider. The Pull request with this feature has not being approved for a long time.
Some companies develop Terraform providers for their API and maintain it. But some Terraform providers are created by enthusiasts who might abandon their project anytime.
Hashicorp knows about this problem, so they try to support many providers by themself until official providers are provided.
https://registry.terraform.io/search/providers

I recommend to check provider git history and activity before using terraform provider.

HCL limitations

HCL is Terraform language, it is a very limited.
For example, you need to use hacks to emulate "if" statements, like following:

Because of count usage terraform creates list of resources instead of single resource. So the path for created vm is: module.test_vm[0] just because we want to have conditional creation. StackOverflow question about it.

But Hashicorp improves HCL constantly, for example we just got Terraform 1.3 with optional object fields supported, so now we can have defaul values no only for entire variable, but also for object field.

HCL support in IDEs still far from perfect. Autocomplete is not very smart and reminds me early years of JavaScript. Jetbrains did a great job supporting HCL in their IDEA familiy.
Autocomplete doesn't support object definition in maps.

Apply-only errors

Even if Terraform plan is green, Terraform apply might be errored.

Terraform apply does real API calls, which might fail because of validation errors, eg. "resource name is too long" or network connectivity issues. For example, you might ask to create a database before the database server is created. Terraform requires you manually define resource dependencies with the depends_on property of the resource.

It would help if you had a stage environment with the same terraform state to test Terraform apply before doing the same change for production environment.

State is fragile

Terraform can manage resource only if it is tracked by Terraform state. So if a resource was deleted manually, you need manually delete the information about the resource from Terraform state. Terraform state might "drift" for various reasons, and fixing it might be tricky.
For example, I wrote script to get resources ids from REST API, and import it to terraform with terraform import command.
The drift detection is one of features Terraform Cloud, but not terraform itself.

Terraform Cloud

Terraform cloud has numer of cool features you might like to have. But it is also vendor lock and third-party dependency you can't control or diagnose.
The most irritating thing is "You have exceeded the API's rate limit." error in the middle of work you need to do.

Conclusion

Despite on all this issues, I think that Terraform is a great tool for infrastructure manament.

Docker concepts you should know

Alexey Ryazhskikh — Fri, 11 Feb 2022 12:21:35 +0000

Difference between image and container

You need to feel difference between docker "image" and "container".
Make sure you know understand the flow:
You have Dockerfile to build image you need to run containers.

Docker file build contexts

You need to understand the context of each line of dockerfile

what is the current build context (directory on agent)
what is the current working directory on image
what is the current user
what is the current stage (for multistage build)
how it affects layer cache

Agent is a machine where docker build command started. docker build command has required parameter "build context" which is folder on agent machine.
For example, the COPY <src> <dest> command <src> parameter is path relative to the current build context folder.
<dest> parameter is path relative to the current working directory on image file system.

For example this Dockerfile has the following file systems:

WORKDIR /src
COPY . .

Build context (agent)	Work directory (image)
- docker-concepts-you-should-know - src - WebApi - WebApi.csproj - Program.cs - Dockerfile	- src - src - WebApi - WebApi.csproj - Program.cs - Dockerfile

Build time and Runtime

You are able to run apps while docker build building image with RUN statement. And you can run your apps in running container with ENTRYPOINT or with docker exec. Normally you might need to run your development tools build time, but your service or utils (like database migration) in runtime.
Build time executions are focused on image content, for example you might want to build binaries from source code. Build time you don't have your private network and volumes, so you can't have access to database.

Layout caching

Each statement in docker file produces image cache layer on docker host machine.
Cache layers could help application build time, for example
dotnet restore result will be cached if files copied to the docker image are the same. That is why default Dockerfile generated by visual studio copies csproj files first, and other files only after dotnet restore.

WORKDIR /src
COPY ["src/webapi/webapi.csproj", "src/webapi/"]
RUN dotnet restore "src/webapi/webapi.csproj"
COPY . .

Andrew Lock wrote excellent article about this optimization.
https://andrewlock.net/optimising-asp-net-core-apps-in-docker-avoiding-manually-copying-csproj-files/

Docker-compose

Docker compose calls Docker under the hood. But docker-compose has it's own behavior and some commands could work differently.

Entrypoint

You can use shell script as Entrypoint. If you have windows developer environment you need to remember:

Set lr for .sh files in .gitattributes

*.sh text eol=lf

Set +x for .sh file in git metadata

git update-index --chmod=+x foo.sh

or you can use TortoiseGit:

Link between Azure VM Scale Set and Azure DevOps Agent Pool

Alexey Ryazhskikh — Thu, 30 Dec 2021 09:11:23 +0000

This article describes link between Azure VM Scale Set and Azure DevOps agent pool.

Azure DevOps allows to create Agent Pool based on Azure VM Scale Set. With this setup Azure DevOps Agent Pool manages Azure ScaleSet to scale up or down number of agents depend on how many agents required for pipelines in particular moment.

The scale set creates new instances of virtual machines with Azure Devops agent installed on it. Agent is able to interact with the pool after it is created.

There are two actions happen in time of Pool creation:

Azure DevOps adds tag __AzureDevOpsElasticPool="pool-name" to Azure Scale Set to identify scale set linked to it.
Azure DevOps adds Azure Pipelines Agent extension to virtual machine scale set and configures it with required PAT token. This extension is being called by scale set to install agent software to just created virtual machine. (Check Lifecycle of a Scale Set Agent manual) If you recreate vm scale set without __AzureDevOpsElasticPool tag or delete this tag you will see the following error in Diagnostics tab of your pool in Azure Devops UI: The Resource 'Microsoft.Compute/virtualMachineScaleSets/test-vmss' under resource group 'test-vmss-rg' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix If you recreate VM scale set with __AzureDevOpsElasticPool tag, but without the Azure Pipelines Agent extension, you will have no agents connected to the Pool, bu Azure DevOps will upscale the number of VM instances.

The only way I know to add the Agent extension to scale set is to recreate Pool.
But here is an issue: you can't create pool with existing scale set if this scale set already has __AzureDevOpsElasticPool tag on it.

Azure DevOps pipeline optimization for .net core projects

Alexey Ryazhskikh — Tue, 04 May 2021 21:36:17 +0000

I own a pipeline that builds, tests, and deploys many asp.net core services to k8s cluster. The pipeline is running on Azure DevOps hosted agents. They are scalable, but not robust, so I need to do some performance optimizations for my pipeline.

The pipeline has three main stages:

running various automated tests
building and pushing number of docker images for k8s deployment
deploying services by terraform

Running stages in parallel

My pipeline runs in two modes:

Pull request validation - it runs all tests and executes only terraform plan.
Deployment mode - it runs all tests and does actual deployment to the k8s cluster.

My pipeline doesn't change anything in PR validation mode, so I can validate terraform configuration, Dockerfiles, and run autotests in parallel.

Normal flow:

Pull Request flow:

By default stages are depend on each other, but you can change this behaviour by forcing dependsOn: [] in azure-pipelines.yml:

variables:
  isPullRequestValidation: ${{ eq(variables['Build.Reason'], 'PullRequest') }}

...

stages:
- stage: deploy_to_dev
  displayName: "Deploy to Dev"
  ${{ if eq(variables.isPullRequestValidation, true) }}:
    dependsOn: []

SQL Server startup optimization

For running integration tests, I use docker-compose with SQL Server and test container declarations.

version: "3.7"
services:

  mssql:
    image: "mcr.microsoft.com/mssql/server"
    environment:
      SA_PASSWORD: "Sa123321"
      ACCEPT_EULA: "Y"
    ports:
      - 5433:1433

  integrationTests:
    container_name: IntegrationTests
    image: my/my_integration_tests
    build:
      context: ..
      dockerfile: ./Integration.Tests/Dockerfile

There is a problem with running SqlServer in docker - it needs few minutes to start.
My idea is to run SQL Server before start building test containers.
For that, I split the docker-compose.yml into two ones: docker-compose.db.yml and docker-compose.tests.yml.
Docker-compose services running with the same --project-name share the same scope.
So my integration test job does the following steps:

docker-compose -f docker-compose.db.yml    --project-name test up -d
docker-compose -f docker-compose.tests.yml --project-name test build
docker-compose -f docker-compose.tests.yml --project-name test up -d
docker-compose -f docker-compose.tests.yml --project-name test exec integrationTests "/src/Integration.Tests/init-db.sh"
docker-compose -f docker-compose.tests.yml --project-name test exec integrationTests dotnet test

After the optimization Init test database step doesn't wait for SQL Server running, and I found no build speed degradation.

.Net Core services build optimization

Build Parallelism

Azure DevOps agents do not guarantee persistence for docker cache between jobs running. However, you can be lucky and get an agent with prepopulated docker cache.
Anyway, you surely can rely on the docker cache for layers built due to the current job.
It might be better to have many build tasks in one job, rather than having multiple parallel jobs.

Dockerfile code generation

My pipeline builds one .Net solution with C# project for each service. Each service has a similar Dockerfile.
I created a simple template tool for generating Dockerfiles for each service to keep them identical as much as possible.

Copy and restore .csproj

There is a recommendation to copy only *.csproj files for all projects and restore them before copying other source files. It allows creating a long-live caching image layer with restored dependencies.

# copy only *.csproj and restore
COPY ["MyService/MyService.csproj", "MyService/"]
COPY ["My.WebAPI/My.WebAPI.csproj", "My.WebAPI/"]
RUN dotnet restore "MyService/MyService.csproj"
RUN dotnet restore "My.WebAPI/My.WebAPI.csproj"

COPY . .
RUN dotnet build My.WebAPI/My.WebAPI.csproj"

That is not so important for Azure DevOps hosted agents, but useful for the local builds.
But you need to maintain a list of *.csproj in each Dockerfile of your project. So every new .csproj leads you to update every Dockerfile in your solution.
I think code-generation is the best way to automate it.

Do more, but once

Building single *.csproj with all dependencies takes about 1 minute for my case. Building an entire solution takes ~3 minutes. But thanks to image layer caching I need to build the solution only once.
Here is an example of a service Dockerfile I have:

FROM mcr.microsoft.com/dotnet/aspnet:5.0-buster-slim AS base
WORKDIR /app
FROM mcr.microsoft.com/dotnet/sdk:5.0-buster-slim AS build
WORKDIR /src

# creating common references cache
COPY ["Common/Common.csproj", "Common/"]
RUN dotnet restore "Common/Common.csproj"

COPY . .

# exclude test projects from restore/build
RUN dotnet sln ./My.sln remove **/*Tests.csproj
RUN dotnet restore ./My.sln

RUN dotnet build ./My.sln -c Release

WORKDIR "Services/My.WebAPI"
FROM build AS publish
RUN dotnet publish "My.WebAPI.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "My.WebAPI.dll"]

There is another way to improve solution build speed: exclude unnecessary projects from .sln file with dotnet sln command.

A piece of job execution that builds the services

As you see, building the entire solution took 2 minutes for the first service, other services used created cache and do only RUN dotnet publish for 30 seconds.

Conclusion

Job parallelism and utilizing docker layer cache are orthogonal technics, but you can find the balance for your project.
Docker-compose for integration tests could be split: so one set of services could be built while the other set is initializing.
Code generation can help to keep Dockerfiles similar and utilize the docker layer cache.

DEV Community: Alexey Ryazhskikh

Using GitHub Copilot for Terraform code refactoring

Final Thoughts

Multi-environment infrastructure with terraform variables files

Feature flags

Terraform variables as DSL

Control interface

Refactoring

Conclusion

zap log level codes

Resource identification in Terraform

Resource identification in Terraform

Basic concepts

Resource identifier in case of single resource

Identifiers vs attributes

Resource identifier in case of for_each

Resource identifier in case of count

Conclusion

Terraform resource dependency graph

TL;DR

Using resource output to make dependency

Using depends_on to make dependency

Terraform graph refactoring

One to may relation example

Conclusion

Using dotnet tool

Access to GO modules in private Azure DevOps repositories

Terraform basics

Resources

Provider

HCL

Terraform state

Terraform plan command

Terraform apply command

Terraform module

The Terraform downsides you should know

Providers limitation

HCL limitations

Apply-only errors

State is fragile

Terraform Cloud

Conclusion

Docker concepts you should know

Difference between image and container

Docker file build contexts

Build time and Runtime

Layout caching

Docker-compose

Entrypoint

Link between Azure VM Scale Set and Azure DevOps Agent Pool

Azure DevOps pipeline optimization for .net core projects

Running stages in parallel

SQL Server startup optimization

.Net Core services build optimization

Build Parallelism

Dockerfile code generation

Copy and restore .csproj

Do more, but once

Conclusion

Resource identifier in case of `for_each`

Resource identifier in case of `count`

Terraform `plan` command

Terraform `apply` command