Arindam Mitra

Posted on May 23, 2022 • Edited on Mar 30, 2023

Map Azure Devops runtime variables to Terraform input variables

#azure #devops #terraform #iac

Greetings my fellow Technology Advocates and Specialists.

In this Session, I will demonstrate -

How to Map Azure DevOps Runtime Variables to Terraform Input Variables.
If at all we need to put the values in variables.tf or in tfvars.

I had the Privilege to talk on this topic in ONE Azure Communities:-

NAME OF THE AZURE COMMUNITY	TYPE OF SPEAKER SESSION
Boston Azure User Group	Virtual

EVENT ANNOUNCEMENT:-

LIVE RECORDED SESSIONS:-
LIVE DEMO was Recorded as part of my Presentation in BOSTON AZURE USER GROUP Forum/Platform
Duration of My Demo = 41 Mins 07 Secs

REQUIREMENTS:-

Azure Subscription.
Azure DevOps Organisation and Project.
Service Principal with Delegated Graph API Rights and Required RBAC (Typically Contributor on Subscription or Resource Group)
Azure Resource Manager Service Connection in Azure DevOps.
Microsoft DevLabs Terraform Extension Installed in Azure DevOps.

CODE REPOSITORY:-

arindam0310018 / 23-May-2022-DevOps__Runtime-Variables-To-Terraform-Input-Variables

MAP AZURE DEVOPS RUNTIME VARIABLES TO TERRAFORM INPUT VARIABLES

Greetings my fellow Technology Advocates and Specialists.

In this Session, I will demonstrate -

How to Map Azure DevOps Runtime Variables to Terraform Input Variables.
If at all we need to put the values in variables.tf or in tfvars.

I had the Privilege to talk on this topic in ONE Azure Communities:-

NAME OF THE AZURE COMMUNITY	TYPE OF SPEAKER SESSION
Virtual Boston Azure	Virtual

EVENT ANNOUNCEMENT:-

LIVE RECORDED SESSIONS:-
LIVE DEMO was Recorded as part of my Presentation in BOSTON AZURE USER GROUP Forum/Platform
Duration of My Demo = 41 Mins 07 Secs

REQUIREMENTS:-

Azure Subscription.
Azure DevOps Organisation and Project.
Service Principal with Delegated Graph API Rights and Required RBAC (Typically Contributor on Subscription or Resource Group)
Azure Resource Manager Service Connection in Azure DevOps.
Microsoft DevLabs Terraform Extension Installed in Azure DevOps.

HOW DOES MY CODE PLACEHOLDER

…

View on GitHub

HOW DOES MY CODE PLACEHOLDER LOOKS LIKE:-

OBJECTIVE:-
Deploy a Resource Group and User Assigned Managed Identity from the values provided by user in the DevOps Runtime Variables Parameters and not providing it again in Terraform variables.tf or tfvars

PIPELINE CODE SNIPPET:-

AZURE DEVOPS YAML PIPELINE (azure-pipelines-usr-mid-v1.0.yml):-

trigger:
  none

######################
#DECLARE PARAMETERS:-
######################
parameters:
- name: SubscriptionID
  displayName: Subscription ID Details Follow Below:-
  default: 210e66cb-55cf-424e-8daa-6cad804ab604
  values:
  -  210e66cb-55cf-424e-8daa-6cad804ab604

- name: ServiceConnection
  displayName: Service Connection Name Follows Below:-
  default: amcloud-cicd-service-connection
  values:
  -  amcloud-cicd-service-connection

- name: RGNAME
  displayName: Please Provide the Resource Group Name:-
  type: object
  default: <Please provide the required Name>

- name: USRMIDNAME
  displayName: Please Provide the User Assigned Managed Identity Name:-
  type: object
  default: <Please provide the required Name>

######################
#DECLARE VARIABLES:-
######################
variables:
  TF_VAR_RG_NAME: ${{ parameters.RGNAME }}
  TF_VAR_USR_MID_NAME: ${{ parameters.USRMIDNAME }}
  ResourceGroup: tfpipeline-rg
  StorageAccount: tfpipelinesa
  Container: terraform
  TfstateFile: UMID/usrmid.tfstate
  BuildAgent: windows-latest
  WorkingDir: $(System.DefaultWorkingDirectory)/Usr-MID
  Target: $(build.artifactstagingdirectory)/AMTF
  Environment: NonProd
  Artifact: AM

#########################
# Declare Build Agents:-
#########################
pool:
  vmImage: $(BuildAgent)

###################
# Declare Stages:-
###################
stages:

- stage: PLAN
  jobs:
  - job: PLAN
    displayName: PLAN
    steps:
# Install Terraform Installer in the Build Agent:-
    - task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@0
      displayName: INSTALL TERRAFORM VERSION - LATEST
      inputs:
        terraformVersion: 'latest'
# Terraform Init:-
    - task: TerraformTaskV2@2
      displayName: TERRAFORM INIT
      inputs:
        provider: 'azurerm'
        command: 'init'
        workingDirectory: '$(workingDir)' # Az DevOps can find the required Terraform code
        backendServiceArm: '${{ parameters.ServiceConnection }}' 
        backendAzureRmResourceGroupName: '$(ResourceGroup)' 
        backendAzureRmStorageAccountName: '$(StorageAccount)'
        backendAzureRmContainerName: '$(Container)'
        backendAzureRmKey: '$(TfstateFile)'
# Terraform Validate:-
    - task: TerraformTaskV2@2
      displayName: TERRAFORM VALIDATE
      inputs:
        provider: 'azurerm'
        command: 'validate'
        workingDirectory: '$(workingDir)'
        environmentServiceNameAzureRM: '${{ parameters.ServiceConnection }}'
# Terraform Plan:-
    - task: TerraformTaskV2@2
      displayName: TERRAFORM PLAN
      inputs:
        provider: 'azurerm'
        command: 'plan'
        workingDirectory: '$(workingDir)'
        commandOptions: "--var-file=usrmid.tfvars --out=tfplan"
        environmentServiceNameAzureRM: '${{ parameters.ServiceConnection }}'

# Copy Files to Artifacts Staging Directory:-
    - task: CopyFiles@2
      displayName: COPY FILES ARTIFACTS STAGING DIRECTORY
      inputs:
        SourceFolder: '$(workingDir)'
        Contents: |
          **/*.tf
          **/*.tfvars
          **/*tfplan*
        TargetFolder: '$(Target)'
# Publish Artifacts:-
    - task: PublishBuildArtifacts@1
      displayName: PUBLISH ARTIFACTS
      inputs:
        targetPath: '$(Target)'
        artifactName: '$(Artifact)' 

- stage: DEPLOY
  condition: succeeded()
  dependsOn: PLAN
  jobs:
  - deployment: 
    displayName: Deploy
    environment: $(Environment)
    pool:
      vmImage: '$(BuildAgent)'
    strategy:
      runOnce:
        deploy:
          steps:
# Download Artifacts:-
          - task: DownloadBuildArtifacts@0
            displayName: DOWNLOAD ARTIFACTS
            inputs:
              buildType: 'current'
              downloadType: 'single'
              artifactName: '$(Artifact)'
              downloadPath: '$(System.ArtifactsDirectory)' 
# Install Terraform Installer in the Build Agent:-
          - task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@0
            displayName: INSTALL TERRAFORM VERSION - LATEST
            inputs:
              terraformVersion: 'latest'
# Terraform Init:-
          - task: TerraformTaskV2@2 
            displayName: TERRAFORM INIT
            inputs:
              provider: 'azurerm'
              command: 'init'
              workingDirectory: '$(System.ArtifactsDirectory)/$(Artifact)/AMTF/' # Az DevOps can find the required Terraform code
              backendServiceArm: '${{ parameters.ServiceConnection }}' 
              backendAzureRmResourceGroupName: '$(ResourceGroup)' 
              backendAzureRmStorageAccountName: '$(StorageAccount)'
              backendAzureRmContainerName: '$(Container)'
              backendAzureRmKey: '$(TfstateFile)'
# Terraform Apply:-
          - task: TerraformTaskV2@2
            displayName: TERRAFORM APPLY # The terraform Plan stored earlier is used here to apply only the changes.
            inputs:
              provider: 'azurerm'
              command: 'apply'
              workingDirectory: '$(System.ArtifactsDirectory)/$(Artifact)/AMTF'
              commandOptions: '--var-file=usrmid.tfvars'
              environmentServiceNameAzureRM: '${{ parameters.ServiceConnection }}'

Now, let me explain each part of YAML Pipeline for better understanding.

PART #1:-

BELOW FOLLOWS PIPELINE RUNTIME VARIABLES CODE SNIPPET:-

######################
#DECLARE PARAMETERS:-
######################
parameters:
- name: SubscriptionID
  displayName: Subscription ID Details Follow Below:-
  default: 210e66cb-55cf-424e-8daa-6cad804ab604
  values:
  -  210e66cb-55cf-424e-8daa-6cad804ab604

- name: ServiceConnection
  displayName: Service Connection Name Follows Below:-
  default: amcloud-cicd-service-connection
  values:
  -  amcloud-cicd-service-connection

- name: RGNAME
  displayName: Please Provide the Resource Group Name:-
  type: object
  default: <Please provide the required Name>

- name: USRMIDNAME
  displayName: Please Provide the User Assigned Managed Identity Name:-
  type: object
  default: <Please provide the required Name>

THIS IS HOW IT LOOKS WHEN YOU EXECUTE THE PIPELINE FROM AZURE DEVOPS:-

NOTE:-
Please Provide the Name of the Resource Group
For Example: AMTESTMIDRG
Please Provide the Name of the User Assigned Managed Identity
For Example: AMMID100

PART #2:-

BELOW FOLLOWS PIPELINE VARIABLES CODE SNIPPET:-

######################
#DECLARE VARIABLES:-
######################
variables:
  TF_VAR_RG_NAME: ${{ parameters.RGNAME }}
  TF_VAR_USR_MID_NAME: ${{ parameters.USRMIDNAME }}
  ResourceGroup: tfpipeline-rg
  StorageAccount: tfpipelinesa
  Container: terraform
  TfstateFile: UMID/usrmid.tfstate
  BuildAgent: windows-latest
  WorkingDir: $(System.DefaultWorkingDirectory)/Usr-MID
  Target: $(build.artifactstagingdirectory)/AMTF
  Environment: NonProd
  Artifact: AM

IMPORTANT TO NOTE:-
User Input Values from DevOps Runtime Parameters are referenced to DevOps Variables.
Notice the the variables TF_VAR_RG_NAME and TF_VAR_USR_MID_NAME.
Azure DevOps Variables gets automatically mapped to Environment Variables in Azure DevOps Build Agent.
Environment Variables which Starts with TF_VAR_ gets automatically mapped to Terraform Input Variables
Refer the link to find more: https://www.terraform.io/cli/config/environment-variables

GENERAL INFORMATION:-
Please feel free to change the values of the variables.
The entire YAML pipeline is build using Parameters and variables. No Values are Hardcoded.

PART #3:-

PIPELINE STAGE DETAILS FOLLOW BELOW:-

This is a Two Stage Pipeline with 4 Runtime Variables - 1) Subscription ID 2) Service Connection Name 3) Resource Group Name and 4) User Assigned Managed Identity Name
The Names of the Stages are - 1) PLAN and 2) DEPLOY

PIPELINE STAGE - PLAN:-

- stage: PLAN
  jobs:
  - job: PLAN
    displayName: PLAN
    steps:
# Install Terraform Installer in the Build Agent:-
    - task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@0
      displayName: INSTALL TERRAFORM VERSION - LATEST
      inputs:
        terraformVersion: 'latest'
# Terraform Init:-
    - task: TerraformTaskV2@2
      displayName: TERRAFORM INIT
      inputs:
        provider: 'azurerm'
        command: 'init'
        workingDirectory: '$(workingDir)' # Az DevOps can find the required Terraform code
        backendServiceArm: '${{ parameters.ServiceConnection }}' 
        backendAzureRmResourceGroupName: '$(ResourceGroup)' 
        backendAzureRmStorageAccountName: '$(StorageAccount)'
        backendAzureRmContainerName: '$(Container)'
        backendAzureRmKey: '$(TfstateFile)'
# Terraform Validate:-
    - task: TerraformTaskV2@2
      displayName: TERRAFORM VALIDATE
      inputs:
        provider: 'azurerm'
        command: 'validate'
        workingDirectory: '$(workingDir)'
        environmentServiceNameAzureRM: '${{ parameters.ServiceConnection }}'
# Terraform Plan:-
    - task: TerraformTaskV2@2
      displayName: TERRAFORM PLAN
      inputs:
        provider: 'azurerm'
        command: 'plan'
        workingDirectory: '$(workingDir)'
        commandOptions: "--var-file=usrmid.tfvars --out=tfplan"
        environmentServiceNameAzureRM: '${{ parameters.ServiceConnection }}'

# Copy Files to Artifacts Staging Directory:-
    - task: CopyFiles@2
      displayName: COPY FILES ARTIFACTS STAGING DIRECTORY
      inputs:
        SourceFolder: '$(workingDir)'
        Contents: |
          **/*.tf
          **/*.tfvars
          **/*tfplan*
        TargetFolder: '$(Target)'
# Publish Artifacts:-
    - task: PublishBuildArtifacts@1
      displayName: PUBLISH ARTIFACTS
      inputs:
        targetPath: '$(Target)'
        artifactName: '$(Artifact)'

PLAN STAGE PERFORMS BELOW:-

##	TASKS
1.	Terraform Installer installed in Azure DevOps Build Agent.
2.	Terraform Init
3.	Terraform Validate
4.	Terraform Plan
5.	Copy the Terraform files (Most Importantly Terraform Plan Output) to Artifacts Staging Directory.
6.	Publish Artifacts

PIPELINE STAGE - DEPLOY:-

- stage: DEPLOY
  condition: succeeded()
  dependsOn: PLAN
  jobs:
  - deployment: 
    displayName: Deploy
    environment: $(Environment)
    pool:
      vmImage: '$(BuildAgent)'
    strategy:
      runOnce:
        deploy:
          steps:
# Download Artifacts:-
          - task: DownloadBuildArtifacts@0
            displayName: DOWNLOAD ARTIFACTS
            inputs:
              buildType: 'current'
              downloadType: 'single'
              artifactName: '$(Artifact)'
              downloadPath: '$(System.ArtifactsDirectory)' 
# Install Terraform Installer in the Build Agent:-
          - task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@0
            displayName: INSTALL TERRAFORM VERSION - LATEST
            inputs:
              terraformVersion: 'latest'
# Terraform Init:-
          - task: TerraformTaskV2@2 
            displayName: TERRAFORM INIT
            inputs:
              provider: 'azurerm'
              command: 'init'
              workingDirectory: '$(System.ArtifactsDirectory)/$(Artifact)/AMTF/' # Az DevOps can find the required Terraform code
              backendServiceArm: '${{ parameters.ServiceConnection }}' 
              backendAzureRmResourceGroupName: '$(ResourceGroup)' 
              backendAzureRmStorageAccountName: '$(StorageAccount)'
              backendAzureRmContainerName: '$(Container)'
              backendAzureRmKey: '$(TfstateFile)'
# Terraform Apply:-
          - task: TerraformTaskV2@2
            displayName: TERRAFORM APPLY # The terraform Plan stored earlier is used here to apply only the changes.
            inputs:
              provider: 'azurerm'
              command: 'apply'
              workingDirectory: '$(System.ArtifactsDirectory)/$(Artifact)/AMTF'
              commandOptions: '--var-file=usrmid.tfvars'
              environmentServiceNameAzureRM: '${{ parameters.ServiceConnection }}'

DEPLOY STAGE PERFORMS BELOW:-

##	TASKS
1.	DEPLOY Stage will Execute only if PLAN Stage completed successfully. If not, DEPLOY Stage will get Skipped Automatically.
2.	DEPLOY Stage will Execute only after Approval. The Approval is integrated with Environment defined in the Pipeline Variable Section (Environment: NonProd) and applied in DEPLOY Stage Jobs (environment: $(Environment)).
3.	Download the Published Artifacts.
4.	Terraform Installer installed in Azure DevOps Build Agent.
5.	Terraform Init
6.	Terraform Apply

DETAILS AND ALL TERRAFORM CODE SNIPPETS FOLLOWS BELOW:-

TERRAFORM (main.tf):-

terraform {
  required_version = ">= 1.2.0"

   backend "azurerm" {
    resource_group_name  = "tfpipeline-rg"
    storage_account_name = "tfpipelinesa"
    container_name       = "terraform"
    key                  = "UMID/usrmid.tfstate"
  }
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.2"
    }   
  }
}
provider "azurerm" {
  features {}
  skip_provider_registration = true
}

TERRAFORM (usrmid.tf):-

## Azure Resource Group:-
resource "azurerm_resource_group" "rg" {
  name     = var.RG_NAME
  location = var.rg-location
}

## Azure User Assigned Managed Identities:-
resource "azurerm_user_assigned_identity" "az-usr-mid" {

  name                = var.USR_MID_NAME
  resource_group_name = azurerm_resource_group.rg.name
  location            = azurerm_resource_group.rg.location

  depends_on          = [azurerm_resource_group.rg]
  }

TERRAFORM (variables.tf):-

variable "RG_NAME" {
  type        = string
  description = "Name of the Resource Group"
}

variable "rg-location" {
  type        = string
  description = "Resource Group Location"
}

variable "USR_MID_NAME" {
  type        = string
  description = "Name of the User Assigned Managed Identity"
}

IMPORTANT TO NOTE:-
The Variable name of the Resource Group and User Assigned Managed Identities in usrmid.tf and variables.tf are in upper case.
This is because Azure DevOps Pipeline variables which automatically references to Build Agent Environment Variables gets converted to uppercase
If the variables are not defined as above, the Pipeline waits for Resource Group and User Assigned Managed Identity Name as Input.

The Pipeline is then cancelled manually

TERRAFORM (usrmid.tfvars):-

rg-location     = "West Europe"

IMPORTANT TO NOTE:-
There is No Resource Group and User Assigned Managed Identity Name Value provided in tfvars or in variables.tf

ITS TIME TO TEST:-
DESIRED RESULT: Stages - PLAN and DEPLOY should Complete Successfully. Resource Group and User Assigned Managed Identity Resources should get deployed. Remote State file gets created.
PIPELINE RUNTIME PARAMETERS WITH POPULATED VALUES:-

PIPELINE STAGE PLAN EXECUTED SUCCESSFULLY:-

PIPELINE STAGE DEPLOY WAITING APPROVAL:-


PIPELINE STAGE DEPLOY EXECUTED SUCCESSFULLY:-

PIPELINE OVERALL EXECUTION STATUS:-


VALIDATE RESOURCES DEPLOYED IN PORTAL:-

VALIDATE REMOTE TERRAFORM STATE FILE:-