DEV Community: Napoleon 'Ike' Jones

Deploying a .NET Core App to AWS Elastic Beanstalk via Azure DevOps

Napoleon 'Ike' Jones — Sun, 02 Feb 2020 18:52:43 +0000

Deploying a .NET Core App to AWS Elastic Beanstalk via Azure DevOps

Introduction

In another life and my two previous roles I evangalized Azure DevOps relentlessly. In one position we had nothing in terms of a tooling outside of a very outdated implementation of SubVersion. In the other postion, add outdated versions of Jira and Jenkins to the mix. At least in this instance, Jenkins and SubVersion were somewhat connected, however, Jira was left out in the cold.

Anytime I would demo Azure DevOps I faced the same questions and criticisms.

“But it won’t work with [insert non-Microsoft language], will it?”
“It’s Microsoft, it will never work outside of their ecosystem!”
“We want to host our applications with Amazon, this won’t work here.”

Those were the comments on the friendlier side of things. At times, they were just plain unprofessional (to put it nicely). But as @DonovanBrown, “This ain’t your daddy’s Microsoft.”

In this post I will show you how easy it is to deploy a .NET Core application to AWS Elastic Beanstalk.

Setting Up the AWS Environment

To begin, you will need to set up an AWS Elastic Beanstalk environment. Before writing this post, I had never touched AWS. The process is fairly straight forward.

Click the Create New Application button towards the upper right-hand corner. The below modal will pop up. Enter a name for your application in the Application name field and click the Create button.

This takes us to the Base configuration section. I selected .NET (Windows/IIS) for the Platform, however you could run .NET Core most of the Linux offerings. For Application code, choose Sample application as we will deploy our actual code with Azure DevOps. Once you have set the base configuration click the Create application button.

This will kickoff the creation of our Elastic Beanstalk environment.

Once the creation is complete it will redirect you to the Elastic Beanstalk dashboard shown below. You can view the application by clicking the URL at the top of the page.

Congratulations, you have just created a deault ASP.NET Web App.

### Creating an AWS IAM Account

Next, you will create an account that you can use to deploy your application from Azure DevOps. Click the dropdown menu associated with your account name and then select My Security Credentials.

A warning modal will popup, select Get Started with IAM Users.

Next, you will create a security group. Enter “AzureDevOps” for the Group Name. Then click Next Step.

This brings you to the Attach Policy page. Enter “AWSElasticBeanstalk” into the search box to filter through the policies. Then select AWSElasticBeanstalkFullAccess. Click Attach Policy to continue.

Review the settings and click Create Group.

This brings you to the Add user screen. Enter a User name. I used the name “AzureDevOps” for this example. For Acces type select Programmatic access. Then click Next: Permissions.

Next we need to set the permissions on the account. We do this by adding the users to groups with policies attached to them. We will place this account in the AzureDevOps group we previously created. Click Next: Tags.

This will bring us to the Tags page. In this example I did not add any tags. Click Next: Review.

Review your account settings and if they are correct click Create user.

Congratulations, you have successfully create a new user account. Next copy the Access key ID and Secret access key as we will be storing them in Azure Key Vault.

### Creating Azure DevOps Service Connection for AWS

The next step is to create a new Service Connection within Azure DevOps for AWS. This connection will use the account we created in the previous step.

First, go to the Projects Settings by clicking the Settings button towards the bottom of the left hand menu.

Then scroll down to Pipelines and select Service connections.

Next, click Create service connection

Select AWS and click Next

Fill in the Access Key ID and Secret Access Key from the IAM User you created in AWS.

Enter a Service connection name and check Grant access permission to all pipelines. Then click Save.

You have now created a new service connection.

### Creating the Application

In this post we will deploy a fairly simple .NET Core Application. Create a new .NET Core MVC Application and make the following changes.

In the Index.chtml file replace the existing code with the following.

@{
    ViewData["Title"] = "Home Page";
}

<div class="text-center">
    <h1 class="display-4">Welcome</h1>
    <p>This demo is running .NET Core on @ViewData["CloudEnv"].</p>
    <p>Learn about <a href="https://docs.microsoft.com/aspnet/core">building Web apps with ASP.NET Core</a>.</p>
</div>

Then in the HomeController add an IConfiguration property and add it to the constructor.

private readonly ILogger<HomeController> _logger;

private readonly IConfiguration _configuration;

public HomeController(ILogger<HomeController> logger, IConfiguration configuration)
{
     _logger = logger;
     _configuration = configuration;
 }

Replace the Index method of the HomeController with the following code.

```

public IActionResult Index()
{
ViewData["CloudEnv"] = _configuration["CloudEnv"];
return View();
}



Add the setting **CloudEnv** to appsettings.json. Not the prefix and postfix “\_\_” in the CloudEnv variable. We are tokenizing this variable so that the value can be replaced with a pipeline variable later on.



```<pre class="wp-block-preformatted">{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Warning",
      "Microsoft.Hosting.Lifetime": "Information"
    }
  },
  "AllowedHosts": "*",
  "CloudEnv":  "__CloudEnv__"
}

Afterwards, add the CloudEnv setting to appsettings.Development.json

```

{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft": "Warning",
"Microsoft.Hosting.Lifetime": "Information"
}
},
"CloudEnv": "my local machine"
}



That’s it for the application. Now on to the pipeline.

### Creating the Multi-Stage Pipeline

In this example we will create a multi-stage pipeline using YAML. The first stage will build the pipeline and the second will deploy it to AWS.

### Marketplace Extensions

The multi-stage pipeline we are creating requires two [Marketplace Extensions](https://docs.microsoft.com/en-us/azure/devops/marketplace-extensibility/?view=azure-devops). Install the following extensions into your project. Consult [this reference guide](https://docs.microsoft.com/en-us/azure/devops/marketplace/install-extension?view=azure-devops&tabs=browser) if you are having issues installing the extensions.

- [Replace Tokens](https://marketplace.visualstudio.com/items?itemName=qetza.replacetokens)
- [AWS Tools for Microsoft Visual Studio Team Services](https://marketplace.visualstudio.com/items?itemName=AmazonWebServices.aws-vsts-tools)

### Creating the Initial Pipeline

First, we will create the basis for the pipeline.

In Azure DevOps select the **“Rocket Ship”** from the lefthand menu, then select **Pipelines**.

![](https://i2.wp.com/ikethe.dev/wp-content/uploads/2020/01/01-Go-to-Pipelines.png?w=843&ssl=1)Next, select **New pipeline**

![](https://i2.wp.com/ikethe.dev/wp-content/uploads/2020/01/02-Select-New-Pipeline.png?fit=843%2C81&ssl=1)Then select where your code is located. For this example my code is stored in Azure Repos Git. However, your code could be in GitHub or any other source control tool.

![](https://i2.wp.com/ikethe.dev/wp-content/uploads/2020/01/03-Select-Code-Location.png?w=843&ssl=1)Afterwards, select your repository.

![](https://i2.wp.com/ikethe.dev/wp-content/uploads/2020/01/04-Select-Repository.png?w=843&ssl=1)Then select **Starter pipeline**. This will create a basic pipeline with a couple of tasks.

![](https://i1.wp.com/ikethe.dev/wp-content/uploads/2020/01/05-Select-Starter-Pipeline.png?w=843&ssl=1)### Modifying the YAML

Before we begin, I would like to warn you that YAML can be very fickle and whitespaces matter. One misaligned space can cause the entire pipeline to fail.

To learn more about YAML pipelines [this reference guide](https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema#pipeline-structure) is an excellent resource.

### Initial Pipeline

The initial portion of our pipeline defines the trigger, which is defined as the master branch of our repository. Anytime master changes, our pipeline will automatically run.



```<pre class="wp-block-preformatted">trigger:
- master

Stage 1 – Build

The first stage of our pipeline downloads any dependencies and then builds the solution. It then creates then publishes the artifact that will be used in the second stage.

```

stages:

stage: 'Build'
variables:
solution: '*/.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
displayName: 'Build'
jobs:
- job: 'Build' displayName: 'Build' pool: vmImage: 'windows-latest' steps:
- task: NuGetToolInstaller@1
- task: NuGetCommand@2 inputs: restoreSolution: '$(solution)'
- task: VSBuild@1 inputs: solution: '$(solution)' msbuildArgs: '/p:SkipInvalidConfigurations=true /p:DeployOnBuild=true /p:WebPublishMethod=FileSystem /p:publishUrl="$(build.artifactstagingdirectory)\" /p:DeployDefaultTarget=WebPublish' platform: '$(buildPlatform)' configuration: '$(buildConfiguration)'
- task: VSTest@2 inputs: platform: '$(buildPlatform)' configuration: '$(buildConfiguration)'
- task: PublishBuildArtifacts@1 inputs: PathtoPublish: '$(Build.ArtifactStagingDirectory)' ArtifactName: 'drop' publishLocation: 'Container' ```

Important: When deploying a .NET Core App to AWS Elastic Beanstalk you cannot use the default msbuildArgs. For some reason Elastic Beanstalk cannot find the web.config file when the artifact is published as a zip file. Please note the WebPublishMethod=FileSystem argument.

Stage 2 – Deploying to AWS

The second stage of the pipeline deploys the artifact from the first stage to AWS.

```

- stage: 'Deploy'
  displayName: 'Deploy'
  jobs:
  - deployment: DeployToAws
    displayName: 'Deploy To AWS Elastic Beanstalk'
    pool:
      vmImage: 'windows-latest'
    variables:
      CloudEnv: 'AWS Elastic Beanstalk'
    environment: 'Production'
    strategy:
      runOnce:
        deploy:
          steps:
          - download: current
            artifact: drop
          - task: replacetokens@3
            displayName: 'Replace Tokens'
            inputs:
              rootDirectory: '$(Pipeline.Workspace)/drop'
              targetFiles: '*/.json'
              tokenPrefix: ''
              tokenSuffix: ''

          - task: BeanstalkDeployApplication@1
            displayName: 'Deploy to Elastic Beanstalk'
            inputs:
              awsCredentials: 'AWS Deployment Service Connection'
              regionName: 'us-east-2'
              applicationName: DotNetCoreDemo
              environmentName: 'Dotnetcoredemo-env'
              applicationType: aspnetCoreWindows
              dotnetPublishPath: '$(Pipeline.Workspace)/drop'



Notice in the deployment job we are setting a job level variable **CloudEnv**. We will use variable in a later task to replace our tokenized appsetting from earlier.



```<pre class="wp-block-preformatted"> - deployment: DeployToAws
     displayName: 'Deploy To AWS Elastic Beanstalk'
     pool:
       vmImage: 'windows-latest'
     variables:
       CloudEnv: 'AWS Elastic Beanstalk'

Let’s break down the tasks in this stage of the pipeline.

The first task, download, downloads the artifact we created in the previous build stage.

```

- download: current
artifact: drop



The next task, **replacetokens@3**, is the first Marketplace Extension we installed earlier. This tasks looks for any string with a prefix and suffix of “\_\_” and attempts to replace them with a variable defined in the pipeline. In this example, “AWS Elastic Beanstalk” replaces the value for CloudEnv in appsettings.json



```<pre class="wp-block-preformatted"> - task: replacetokens@3
            displayName: 'Replace Tokens'
            inputs:
              rootDirectory: '$(Pipeline.Workspace)/drop'
              targetFiles: '**/*.json'
              tokenPrefix: '__'
              tokenSuffix: '__'

Important: The variable name must match the string without the prefix and suffix.

The final task, BeanstalkDeployApplication@1, is the second Marketplace Extension we installed earlier. This task deploys the artifact to AWS Elastic Beanstalk. Pay close attention to the inputs. The awsCredentials argument is the name of the service connection we created earlier. regionName, applicationName, and environmentName should match with the Elastic Beanstalk environment and application we originally created. The last setting, dotnetPublishPath, is the location of the artifact we downloaded.

```



### The Entire Pipeline

Below is the complete multi-stage pipeline.



```<pre class="wp-block-preformatted"> trigger:
- master

stages:
- stage: 'Build'
  variables:
    solution: '**/*.sln'
    buildPlatform: 'Any CPU'
    buildConfiguration: 'Release'
  displayName: 'Build'
  jobs:
  - job: 'Build'
    displayName: 'Build'
    pool:  
      vmImage: 'windows-latest' 
    steps:
    - task: NuGetToolInstaller@1

    - task: NuGetCommand@2
      inputs:
        restoreSolution: '$(solution)'

    - task: VSBuild@1
      inputs:
        solution: '$(solution)'
        msbuildArgs: '/p:SkipInvalidConfigurations=true /p:DeployOnBuild=true /p:WebPublishMethod=FileSystem /p:publishUrl="$(build.artifactstagingdirectory)\\" /p:DeployDefaultTarget=WebPublish'
        platform: '$(buildPlatform)'
        configuration: '$(buildConfiguration)'

    - task: VSTest@2
      inputs:
        platform: '$(buildPlatform)'
        configuration: '$(buildConfiguration)'

    - task: PublishBuildArtifacts@1
      inputs:
        PathtoPublish: '$(Build.ArtifactStagingDirectory)'
        ArtifactName: 'drop'
        publishLocation: 'Container'

- stage: 'Deploy'
  displayName: 'Deploy'
  jobs:
  - deployment: DeployToAws
    displayName: 'Deploy To AWS Elastic Beanstalk'
    pool:
      vmImage: 'windows-latest'
    variables:
      CloudEnv: 'AWS Elastic Beanstalk'
    environment: 'Production'
    strategy:
      runOnce:
        deploy:
          steps:
          - download: current
            artifact: drop
          - task: replacetokens@3
            displayName: 'Replace Tokens'
            inputs:
              rootDirectory: '$(Pipeline.Workspace)/drop'
              targetFiles: '**/*.json'
              tokenPrefix: '__'
              tokenSuffix: '__'

          - task: BeanstalkDeployApplication@1
            displayName: 'Deploy to Elastic Beanstalk'
            inputs:
              awsCredentials: 'AWS Deployment Service Connection'
              regionName: 'us-east-2'
              applicationName: DotNetCoreDemo
              environmentName: 'Dotnetcoredemo-env'
              applicationType: aspnetCoreWindows
              dotnetPublishPath: '$(Pipeline.Workspace)/drop'

At this point we can run the pipeline and it will deploy our application to our AWS Elastic Beanstalk environment.

Successful Deployment

As you can tell by the below screenshot the deployment was successful and the CloudEnv appsetting was replaced with the pipeline variable.

Reap What You Sow II

Napoleon 'Ike' Jones — Sat, 02 Nov 2019 03:56:04 +0000

Reap What You Sow II
IaC, Terraform, & Azure DevOps
Now With YAML

In this post I will take you through deploying your Infrastructure using Terraform and Azure DevOps. Instead of using Azure DevOps’ classic pipeline editor as I did in my previous post, Reap What You Sow, I will now use YAML.

Why Use YAML?

For one, we should be treating Everything as Code. Well at least everything we are able to. Treating things as code allows us to apply modern software engineering practices to them such as source control and a peer review process. Two, I like to learn new things. Three, I was challenged to do it so I’m required by law to do it.

Fear and Loathing in DevOps…

Prior to accepting this challenge, I had absolutely no experience with YAML. I had looked at some pipelines written in YAML at one point and I was honestly intimidated. How terrified was I? The day I accepted the challenge, I had been attending DevOps Days Indy and I explicitly told a few people about how I had been avoiding writing my pipelines in YAML. Little did I know I would have my first pipeline written in YAML up and running later that evening.

What is YAML?

YAML Ain’t Markup Language (YAML) is a data serialization language that is commonly used for configuration files. You can read more about YAML and its history here.

Let’s Get Started

In order to follow along this post you will need to take a look at my previous post. At a minimum complete the steps prior to “Back to Azure DevOps.” It is, however, highly recommend that you complete it entirely as you will be able to click the “View YAML” button if you get stuck.

### Enable Multi-stage Pipelines

Once you have completed the above steps, make sure you have the new multi-stage pipeline feature enabled. Click on your profile in the upper right corner, then select Preview features. Make sure the Multi-stage pipelines feature is turned on.

### Creating Your First YAML Pipeline

Next, click Pipelines in the left-hand menu. Afterwards, click New Pipeline in the upper right corner of the Pipelines section. Then select your repository. If you’ve followed this blog, select Azure Repos Git.

Next, select Starter pipeline.

This will create a basic YAML pipeline, add it to our repository and merge it into the master branch.

The above YAML is fairly straight forward. It is triggered by the master branch (lines 6 and 7). Line 9 and 10 tells it what type of agent to use. While line 12 begins to describe the steps in the pipeline. This pipeline runs two inline scripts.

You can learn more about Azure Pipelines YAML schema here.

Next we are going update the YAML with our Build tasks. Open VS Code or your editor of choice. If you are using VS Code, download this Azure Pipelines extension. Next, git pull on master. You should now see a new file, azure-pipelines.yml.

For the purposes of this post, I worked directly on master. Do not do this in the real world! In my first update of the pipeline, I removed the two existing steps. Then I made the following changes.

To begin with, change the vmImage to ‘vs2017-win2016’ .

Then add the CopyFile@2 task. TargetFolder is the only required input. Set it to ‘$(build.artifactstagingdirectory)/Terraform’.

We also set the Contents to ‘**’ this specifies that we want to copy all files in the root directory and all files in the sub-folders.

Reference documentation for the CopyFile@2 task is available here.

Next add the PublishBuildArtifacts@1 task. Set the PathtoPublish to ‘$(build.artifactstagingdirectory)/Terraform’, the ArtifactName to ‘drop’.

I’m not sure why but the “code” block in WordPress adds a space to the first line of each YAML code snippet.

# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

trigger:
- master

pool:
  vmImage: 'vs2017-win2016'

steps:
 - task: CopyFiles@2
   displayName: 'Copy Files'
   inputs:
     TargetFolder: '$(build.artifactstagingdirectory)/Terraform'
     Contents: '**'

 - task: PublishBuildArtifacts@1
   inputs:
   PathtoPublish: $(build.artifactstagingdirectory)/Terraform
   ArtifactName: drop

Now, git commit and push your code to master.

Congratulations you have a working build pipeline.

### Time to Refactor

Getting the build pipeline working was nice and all but what we really want is a full CI/CD pipeline.

The first step in this process is to refactor our existing pipeline into stages. Stages are logical boundaries in your pipeline such as Build, Deploy To Dev, Run Tests, Deploy to Prod, etc. You can read more about stages here.

The refactored pipeline looks like this

trigger:
- master
stages:
- stage: Build  
  jobs:
  - job: Build
    pool:
      vmImage: 'vs2017-win2016'
    steps:
      - task: CopyFiles@2
        displayName: 'Copy Files'
        inputs:
          TargetFolder: '$(build.artifactstagingdirectory)/Terraform'
          Contents: '**'

      - task: PublishBuildArtifacts@1
        inputs:
          PathtoPublish: $(build.artifactstagingdirectory)/Terraform
          ArtifactName: drop

I added a stages block and then added the Build stage. In that Build stage, we add a jobs block with a single job named Build. From there, the rest of our code looks exactly the same.

Adding the Deployment Functionality

Since the goal is to convert the entire pipeline to YAML, its time to add the deployment functionality.

- stage: Deploy
  jobs:
  - deployment: DeployInfrastructure
    pool:
      vmImage: 'vs2017-win2016'
    variables:
      functions_appinsights: puzzleplatesfunctionsappinsights
      functions_appservice: puzzleplatesfunctionsappservice
      functions_storage: puzzleplatesstorage
      location: centralus
      puzzleplates_functions: puzzleplatesfunctions
      resource_group: PuzzlePlates-ResourceGroup
      terraformstorageaccount: pzzlepltstfrmstrgaccnt
      terraformstoragerg: PuzzlePlatesStorage
    environment: 'Puzzel Plates Development Infrastructure'
    strategy:
      runOnce:
        deploy:

First, add a new stage named Deploy. Then add a jobs block and then add a deployment job. You can read more about deployment jobs here.

Set the vmImage to ‘ vs2017-win2016’ .

Then set your variables within the job. Variables can be set at either the job level or the root level.

From there, set your environment name. Then declare the deployment strategy. For now, only runOnce is available.

Next, define your deployment steps.

deploy:
  steps:
  - download: current
    artifact: drop

Continuing from deploy, the first step to define is downloading the build artifact. “-download” is short hand for the Download Pipeline Artifact task. You can read more about it here. All we are doing is telling it to download the most current artifact named “drop”.

- task: AzureCLI@1
  displayName: 'Azure CLI - Create Terraform State Storage'
  inputs:
    azureSubscription: 'Terraform_Azure_RM'
    scriptLocation: inlineScript
    inlineScript: |
      call az group create --location centralus --name $(terraformstoragerg)

      call az storage account create --name $(terraformstorageaccount) --resource-group $(terraformstoragerg) --location centralus --sku Standard_LRS

       call az storage container create --name terraform --account-name $(terraformstorageaccount)

- task: AzurePowerShell@3
  inputs:
    name: 'Azure PowerShell - Get Key'
    azureSubscription: 'Terraform_Azure_RM'
    azurePowerShellVersion: LatestVersion
    scriptType: InlineScript
    Inline: |
      $key=(Get-AzureRmStorageAccountKey -ResourceGroupName $(terraformstoragerg) -AccountName $(terraformstorageaccount)).Value[0]
                Write-Host "##vso[task.setvariable variable=storagekey]$key"

Next, add the AzureCLI task. Set the displayName, then set the inputs. The first input, azureSubscription is set to a service principal that can create resources in Azure. Deploying infrastructure into Azure from Azure DevOps requires an Azure Resource Manager service connection. Follow these instructions to create one. After setting the subscription set the scriptLocation to inlineScript. Then set the inlineScrpt to the code shown above. This code creates the Azure Blob Storage to store Terraform’s State. The pipe symbol signifies that any indented text that follows should be treated as multi-line scalar value. You can read more about it on stackoverflow.

Moving on, AzurePowerShell@3 task. Do not add the version 4 task as there are conflict with the inline script. Set the name. Then set the azureSubscription to the principle service we used in the previous task. Afterwards set the azurePowerShellVersion to LatestVersion and scriptTYpe to InlineScript. Then copy the above code. This code saves the storage key into the release pipeline.

Next we will be adding the replacetokens task. This task is a marketplace extension so you will need to install it into your Azure DevOps account. This task will replace all of the values with the “__” prefix and suffix with variables from the release pipeline. Click here to get the extension.

#Marketplace Extension
- task: replacetokens@3
  displayName: 'Replace Tokens'
  inputs:
    rootDirectory: '$(Pipeline.Workspace)/drop'
    targetFiles: '**/*.tf'
    tokenPrefix: '__'
    tokenSuffix: '__'

I had a little bit of trouble figuring out what the actual input arguments were named as they are slightly different that what is presented in the classic editor. I ended up having to dive into the extension’s code base on GitHub. Set the rootDirectory to the drop directory in pipeline workspace. Then set the targetFiles to scan all .tf files in the current directory including all sub-directories.

Next add the Terraform tasks. The Terraform task is another extension and you will need to install it into your Azure DevOps account. The extension can be found here.

#Terraform
- task: Terraform@2
  displayName: 'Terraform Init'
  inputs:
    TemplatePath: '$(Pipeline.Workspace)/drop'
    Arguments: init
    InstallTerraform: true
    UseAzureSub: true
    ConnectedServiceNameARM: 'Terraform_Azure_RM'

- task: Terraform@2
  displayName: 'Terraform Plan'
  inputs:
    TemplatePath: '$(Pipeline.Workspace)/drop'
    Arguments: plan
    InstallTerraform: true
    UseAzureSub: true
    ConnectedServiceNameARM: 'Terraform_Azure_RM'

- task: Terraform@2
  displayName: 'Terraform Apply'
  inputs:
    TemplatePath: '$(Pipeline.Workspace)/drop'
    Arguments: apply -auto-approve
    InstallTerraform: true
    UseAzureSub: true
    ConnectedServiceNameARM: 'Terraform_Azure_RM'

Terraform Init initializes a working directory containing Terraform configuration files on the agent. Set the TemplatePath to the artifact location, $(Pipline.Workspace)/drop. Next, set the Arguments to init. Then set InstallTerraform to true. Afterwards set UseAzureSub to true and set the ConnectedServiceNameARM to the service account you previously used.

Terraform Plan compares existing infrastructure to what is in the code you are deploying and determines if it needs to create, update, or destroy resources. The settings are exactly the same except you set the Arguments to plan.

Terraform Apply applies the plan to the infrastructure. The settings are exactly the same as the two previous Terraform tasks except you set the Arguments to apply -auto-approve. This final task will create, update, or destroy your infrastructure in Azure.

Congratulations you now have a working multi-stage Azure Pipeline written in YAML!!!

The Entire YAML Pipeline

trigger:
- master
stages:
- stage: Build  
  jobs:
  - job: Build
    pool:
      vmImage: 'vs2017-win2016'
    steps:
      - task: CopyFiles@2
        displayName: 'Copy Files'
        inputs:
          TargetFolder: '$(build.artifactstagingdirectory)/Terraform'
          Contents: '**'

      - task: PublishBuildArtifacts@1
        inputs:
          PathtoPublish: $(build.artifactstagingdirectory)/Terraform
          ArtifactName: drop

- stage: Deploy
  jobs:
  - deployment: DeployInfrastructure
    pool:
      vmImage: 'vs2017-win2016'
    variables:
      functions_appinsights: puzzleplatesfunctionsappinsights
      functions_appservice: puzzleplatesfunctionsappservice
      functions_storage: puzzleplatesstorage
      location: centralus
      puzzleplates_functions: puzzleplatesfunctions
      resource_group: PuzzlePlates-ResourceGroup
      terraformstorageaccount: pzzlepltstfrmstrgaccnt
      terraformstoragerg: PuzzlePlatesStorage
    environment: 'Puzzel Plates Development Infrastructure'
    strategy:
      runOnce:
        deploy:
          steps:
          - download: current
            artifact: drop

          - task: AzureCLI@1
            displayName: 'Azure CLI - Create Terraform State Storage'
            inputs:
              azureSubscription: 'Terraform_Azure_RM'
              scriptLocation: inlineScript
              inlineScript: |
                call az group create --location centralus --name $(terraformstoragerg)

                call az storage account create --name $(terraformstorageaccount) --resource-group $(terraformstoragerg) --location centralus --sku Standard_LRS

                call az storage container create --name terraform --account-name $(terraformstorageaccount)

          - task: AzurePowerShell@3
            inputs:
              name: 'Azure PowerShell - Get Key'
              azureSubscription: 'Terraform_Azure_RM'
              azurePowerShellVersion: LatestVersion
              scriptType: InlineScript
              Inline: |
                $key=(Get-AzureRmStorageAccountKey -ResourceGroupName $(terraformstoragerg) -AccountName $(terraformstorageaccount)).Value[0]
                Write-Host "##vso[task.setvariable variable=storagekey]$key"

          #Marketplace Extension
          - task: replacetokens@3
            displayName: 'Replace Tokens'
            inputs:
              rootDirectory: '$(Pipeline.Workspace)/drop'
              targetFiles: '**/*.tf'
              tokenPrefix: '__'
              tokenSuffix: '__'

          #Terraform
          - task: Terraform@2
            displayName: 'Terraform Init'
            inputs:
              TemplatePath: '$(Pipeline.Workspace)/drop'
              Arguments: init
              InstallTerraform: true
              UseAzureSub: true
              ConnectedServiceNameARM: 'Terraform_Azure_RM'

          - task: Terraform@2
            displayName: 'Terraform Plan'
            inputs:
              TemplatePath: '$(Pipeline.Workspace)/drop'
              Arguments: plan
              InstallTerraform: true
              UseAzureSub: true
              ConnectedServiceNameARM: 'Terraform_Azure_RM'

          - task: Terraform@2
            displayName: 'Terraform Apply'
            inputs:
              TemplatePath: '$(Pipeline.Workspace)/drop'
              Arguments: apply -auto-approve
              InstallTerraform: true
              UseAzureSub: true
              ConnectedServiceNameARM: 'Terraform_Azure_RM'

Conclusion

Though at first I was intimidated I was able to get my first pipeline up and running within a few hours.

I ran into some issues with spacing and indentation that took me longer than I care to admit. I also ran into an issue using the version 4 preview of the AzurePowershell task and downgraded to version 3.

My initial fear was completely unfounded (except for indenting/spacing nightmares, those are completely justified). Once I had the initial pipeline built I was hooked. I thoroughly enjoyed writing the pipeline and am looking forward to the new it will offer.

Reap What You Sow

Napoleon 'Ike' Jones — Sat, 02 Nov 2019 03:50:13 +0000

Reap What You Sow
IaC, Terraform, & Azure DevOps

The Waiting Game

In the past I have worked in environments that took ages to provision infrastructure. I’m not talking days here, I mean it took a minimum of six months – and that’s if you had a lot of political capital to spend. The majority of requests took about a year and required a minimum of twelve meetings and four, 30-page packets of paperwork that needed to be signed by multiple directors.

After everything was approved, the virtual machines were manually provisioned to the minimal specifications. This was true even for development servers. Once provisioned, it was not unusual for the servers to be configured incorrectly. I have seen everything fat-fingered from RAM settings to incorrect time zones.

Well, guess what? There is a much better way to provision infrastructure in modern environments.

Infrastructure as Code

Infrastructure as Code or IaC is the process of provisioning and managing infrastructure through descriptive code that defines exactly what should be created, updated, or destroyed instead of using traditional manual configuration tools.

By bringing current best practices of software engineering into the infrastruction world, IaC offers many benefits over the “old way” of doing things.

Reduced Risk
Consistency
Increased Speed
Decreased Cost

By removing the human factor from the provisioning process and putting the code in a repository, with a peer-review process, you begin to eliminate the mistakes that people are prone to make. Using the same code base and automated process to deploy your infrastructure – every time – increases consistency, while also increasing the speed of provisioning. This in turn frees up your engineers to focus on solving more complex problems, also known as reduced cost. Engineers aren’t cheap; so the less time they spend on menial work – the better.

Terraform

In this post I will be using Terraform to provision and manage my infrastructure in Azure. Terraform is an open source tool for creating, updating, and versioning infrastructure safely and efficiently. It supports Microsoft Azure, Amazon Web Services, Google Cloud Platform, IBM Cloud, VMware vSphere and many others.

Getting Started

Install the latest version of Terraform., here.
Install the latest version of the Azure CLI, here

Once those are installed, make your way over to Azure DevOps. This is where we will store our code and create our build/release pipeline. If you have never used Azure DevOps I highly recommend that you give it a try. I’m not going to go over Azure DevOps in detail as it is outside of the scope of this post.

Deploying infrastructure into Azure from Azure DevOps requires an Azure Resource Manager service connection. Follow these instructions to create one.

Create a new repository and select Terraform as the gitignore file:

Clone this repository to your computer and create a new file called main.tf. Write the following code:

terraform {

  required_version = ">= 0.11"

  backend "azurerm" {

    storage_account_name = "__terraformstorageaccount__"

    container_name = "terraform"

    key = "terraform.tfstate"

    access_key = "__storagekey__"

  }
}

This code tells Terraform that it must use a version greater than or equal to 0.11, it is provisioning resources in Microsoft Azure, and storing state in Azure Blob Storage. You can read more about storing Terraform State in Azure here. If you notice some of the values have “__” as prefixes and postfixes it is because we are using those to identify variables we will store in our pipeline.

Write the below code:

resource "azurerm_resource_group" "rg" {
  name     = "__resource_group__"
  location = "__location__"
}

This block of code tells Terraform to create a new resource group and we are defining the name and location via pipeline variables.

Write the below code:

resource "azurerm_storage_account" "functions_storage" {
  name                     = "__functions_storage__"
  resource_group_name      = "${azurerm_resource_group.rg.name}"
  location                 = "${azurerm_resource_group.rg.location}"
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

The above block tells Terraform to create a new storage account. If you look closely you will see that we are setting the resource_group_name and location a little differently. What we are doing is using configuration values from the previously created resource group to define configuration values on our storage account. We want everything in the same location to reduce network traffic and we want to use the resource group that we just created in the previous block of code.

The gist of this is pretty simple: you can access configuration values and output from created resource in other parts of your code.

Write the below block of code:

resource "azurerm_app_service_plan" "functions_appservice"{
  name                = "__functions_appservice__"
  location            = "${azurerm_resource_group.rg.location}"
  resource_group_name = "${azurerm_resource_group.rg.name}"
  kind                = "FunctionApp"

  sku {
    tier = "Dynamic"
    size = "Y1"
  }
}

We are now provisioning a new App Service Plan in Azure, which we are configuring with the sku for the consumption based plan since that is what Azure Functions requires to run.

Write the below block of code:

resource "azurerm_application_insights" "functions_appinsights"{
  name                = "__functions_appinsights__"
  location            = "${azurerm_resource_group.rg.location}"
  resource_group_name = "${azurerm_resource_group.rg.name}"
  application_type    = "Web"
}

Now we are provisioning application insights to provide metrics and error logging for our Azure Functions.

Write the below block of code:

resource "azurerm_function_app" "function_app" {
  name                      = "__puzzleplates_functions__"
  location                  = "${azurerm_resource_group.rg.location}"
  resource_group_name       = "${azurerm_resource_group.rg.name}"
  app_service_plan_id       = "${azurerm_app_service_plan.functions_appservice.id}"
  storage_connection_string = "${azurerm_storage_account.functions_storage.primary_connection_string}"

  app_settings = {
    "APPINSIGHTS_INSTRUMENTATIONKEY" = "${azurerm_application_insights.functions_appinsights.instrumentation_key}"
  }
}

This section creates the actual Azure Functions App and adds the instrumentation key for the previously created AppInsights.

Commit and Push your changes back to the remote repository.

Back To Azure DevOps

In Azure DevOps create a new build pipeline and select “use the classic editor” at the bottom.

Select your repository:

Select the Empty Job template at the top:

Add the Copy Files task to Agent job 1 with the same settings:

Add the Publish Artifact – drop task:

Go to Triggers and check Enable Continuous Integration. With CI enabled, every code changed published to the master branch will trigger this build:

Hit Save & Queue. This will save your pipeline and kickoff a build. Once the build is successful, hit the Release button in the upper right corner:

Select Empty job:

In Stage 1 select “1 job, 0 task” to edit the tasks in Stage 1 of the release pipeline:

Add the Azure CLI task with the below settings. For the Azure Subscription make sure you use the service connection you created earlier:

Add the following code to the inline script:

call az group create --location centralus --name $(terraformstoragerg)

call az storage account create --name $(terraformstorageaccount) --resource-group $(terraformstoragerg) --location centralus --sku Standard_LRS

call az storage container create --name terraform --account-name $(terraformstorageaccount)

This code creates the Azure Blob Storage to store Terraform’s State.

Afterwards add the Azure PowerShell task with the following settings:

Add the below code to the inline script:

$key=(Get-AzureRmStorageAccountKey -ResourceGroupName $(terraformstoragerg) -AccountName $(terraformstorageaccount)).Value[0]

Write-Host "##vso[task.setvariable variable=storagekey]$key"

This code saves the storage key into the release pipeline.

Add the Replace Tokens (installed from the Marketplace) to the pipeline. This task will replace all of the values with the “__” prefix and postfix with variables from the release pipeline:

Then add three Run Terraform tasks: one for Init, one for Plan, and one for Apply. Terraform Init initializes Terraform on the agent. Terraform Plan compares existing infrastructure to what is in the code you are deploying and determines if it needs to create, update, or destroy resources. Terraform Apply applies the plan to the infrastructure.

Terrraform Init:

Terraform Plan:

Terraform Apply:

Next you will have to create and set your release variables:

Afterwards hit Save in the upper right corner:

Click on the Continuous Deployment Trigger (Lightning Bolt) under Artifacts. Then enable continuous deployment. Now every time your build creates a new artifact a release will deploy your infrastructure into Azure:

Click Save again and then click Create release to the right of the save button:

This will begin your release.

If everything went according to plan, you should now have the following deployed in Azure:

Resource Group
Storage Account
App Service Plan
Application Insights
Azure Functions App