DEV Community: Martin

Azure Private DNS

Martin — Thu, 09 Nov 2023 00:06:48 +0000

Overview

DNS, DNS, DNS, It's all about DNS. I've circled around this topic for a while, and it's been a common discussion point with many of the clients I have worked with.

DNS is absolutely critical in any environment and if not done correctly it can cause a heap of issues. I am going to talk through some of the options for resolving DNS specifically for private endpoints in Azure and more importantly what Microsoft recommends and, in my opinion, works well.

Private Endpoints

Firstly, let's start with private endpoints. What are they? A private endpoint in Azure is simply a network interface card (NIC) with a private IP address in your virtual network that you can attach to common Azure PaaS services like Azure Storage or Azure SQL that allows you to communicate with that service privately and securely over an internal RFC1918 IP address. Prior to private endpoints these services typically used a service endpoint where in order to connect, your traffic traversed the Microsoft backbone network instead of your virtual network which of course many customers deemed not secure as your traffic is leaving your private network. Service endpoints are still available today. They are by no means unsecure but don't meet the security controls in some heavily regulated environments.

Azure Private DNS

Azure Private DNS is simply a DNS service in Azure that cannot be resolved publicly. It allows you to use your own custom DNS names rather than the DNS names provided by Azure which are public. Azure Private DNS is the fundamental building block of Private Endpoints. Private Endpoints heavily rely on a private DNS zone to resolve a DNS name against the private IP address of the Network Interface Card (NIC) I mentioned previously. For example, if you want to resolve a private endpoint for Azure blob storage then you would create a zone named privatelink.blob.core.windows.net. In order to resolve DNS, the zone must be linked to a virtual network where the originating DNS request is coming from.

Azure Private DNS Resolution Options

Getting into the meat here. There are several options to resolve private DNS in Azure and from on-premises. I will list the most common:

On-Premises to Azure

DNS Virtual Machine in Azure with a forwarder configured on the on-premises DNS Server (sends all DNS to Azure)
DNS Virtual Machine in Azure with a conditional forwarder configured on the on-premises DNS Server (sends only specific zones to Azure)
Azure DNS Private Resolver (DNS Forwarder PaaS Service)

Azure Only & Hybrid Workloads

Azure Private DNS Zone hosted in your hub virtual network and linked to any spoke virtual networks (Recommended for Azure only workloads)
Azure Private DNS Zone hosted in your spoke network and linked to your spoke virtual network (Not Recommended)
Azure DNS Private Resolver Centralised (Recommended for Hybrid Workloads)
Azure DNS Private Resolver Distributed (Recommended for Hybrid Workloads)

A common approach I see which is not recommended is where a DNS zone is deployed for each spoke or application landing zone deployment which means you might end up with tens or hundreds of zones just for one service for example privatelink.blob.core.windows.net. This is hard to manage and not necessary. You want to manage your DNS centrally similar to managing a hub firewall. You wouldn't deploy an NVA for each spoke network so why do it for DNS.

Azure DNS Private Resolver

Before I dive into the architecture options, DNS Private Resolver is a fairly recent service introduced by Microsoft. It's essentially a DNS Forwarder offered as a PaaS service. This eliminates the need for deploying a DNS forwarded virtual machine. One less IaaS service to patch & maintain. The networking enthusiasts were pretty happy to see this introduced.

I thought I would include a cost comparison. The IaaS option includes a 2 VM high availability setup. The DNS resolver comes out slightly more expensive however this doesn't factor in the operational overhead with managing a virtual machines, backup costs etc. It's fully a PaaS service so you really don't have to do any maintenance apart from updating your forwarding rulesets.

Distributed vs Centralised

With DNS Private Resolver, there are 2 architectures that can be adopted.

Centralised DNS has the Azure DNS Private Resolver as the custom DNS server on all spoke networks meaning all DNS is resolved within the Hub network. In this scenario the DNS Resolver Forwarding Ruleset is linked to the hub virtual network only where the rules are defined.

Distributed DNS uses the default Azure DNS server for all spoke networks with the DNS Resolver Forwarding Ruleset linked to the spoke networks to allow for the resolution to private DNS zones. By default, all DNS is resolved in the spoke virtual network unless it's a private DNS zone.

The first option has the Private DNS Zones linked to the hub only. The second option you can opt in for either. You can link the DNS Zones to each spoke network or link the DNS zone only to the hub but ensure you have forwarding rules in your linked ruleset to resolve private DNS zones. Either approach works. Microsoft references both in 2 different articles with no definitive answer on which is best. Both seem to have the same operational overhead.

In my example I have opted for a direct DNS zone link to the spoke virtual network.

Distributed DNS Architecture

In a typical scenario where you might have a hub network, you can deploy a single zone and link it each spoke virtual network. Now your only managing 1 zone for that service across your whole Azure estate.

Let's break this down visually:

(1, 2 & 3) All on-premises DNS queries that need to resolve an Azure Private DNS name (i.e. Private Endpoints) are forwarded to the Azure Private Resolver using conditional forwarding on Active Directory DNS.
(4) Any DNS queries that need to resolve Azure Private DNS from the spoke network are resolved through the private DNS zone link using the default Azure DNS on the virtual network.
(5,6) Any DNS queries from the spoke virtual network that need to resolve DNS on-premises are forwarded to the on-premises DNS server through a forwarding rule set on the Private DNS resolver that uses the outbound endpoint. This is done through a DNS forwarding virtual network link. Essentially a conditional forwarder for any on-prem DNS queries.

This configuration is referred to as a distributedDNS architecture.

Centralised DNS Architecture

In a centralised DNS architecture, the spoke virtual networks use the Private DNS Resolver inbound endpoint as a custom DNS server.

The key difference here is that the Private DNS Zone is linked to the hub instead of the spokes and the forwarding ruleset is linked to the hub virtual network and not the spoke, so the forwarding rules are managed centrally.

Here is the different visually:

(1, 2 & 3) All on-premises DNS queries that need to resolve an Azure Private DNS name (i.e. Private Endpoints) are forwarded to the Azure Private Resolver using conditional forwarding on Active Directory DNS.
(4) Any DNS queries that need to resolve Azure Private DNS from the spoke network are resolved through the Private DNS resolver in the Hub which queries Azure DNS to resolve the Private DNS zone linked to the hub.
(5) Any DNS queries from the spoke virtual network that need to resolve DNS on-premises are forwarded to the on-premises DNS server through a forwarding rule set on the Private DNS resolver that uses the outbound endpoint. This is done through a DNS forwarding virtual network link. Essentially a conditional forwarder for any on-prem DNS queries.

There is no right or wrong here with both architectures. It depends on how you want to manage this. Some organisations have to inspect all traffic centrally through an NVA so this centralised option would be best suited.

It's worth noting that if you have Azure Firewall, it can act as a DNS proxy if you adopt the centralised DNS architecture:

In this scenario you can forward your DNS requests to the Azure firewall IP. Now you get the advantage of traffic inspection and DNS caching. More on the topic here

If your workloads are only in Azure and you don't have the need for a DNS forwarder then create your private DNS zones in the Hub and link them to all of your spoke networks.

Key takeaway is not to duplicate DNS zones by keeping them centralised.

Hope this was useful and let me know in the comments if you would like to see more on this topic. I am looking to get this setup automated using Terraform as a reference architecture.

Running Azure Functions on Azure Container Apps

Martin — Mon, 07 Aug 2023 05:41:52 +0000

Service Overview

Microsoft recently announced the ability to run Azure Functions on Azure Container Apps (in preview). This is a big improvement over the current serverless option for Azure Functions as it allows us to leverage the benefits of microservices and the features of Azure Container Apps like DAPR and KEDA.

Behind the scenes, KEDA is used as the event driven auto scaler for the Functions. You still need to spin up an Azure Function for managing your functions however they execute within the containers (compute environment). This is also a lot more cost efficient as you can use the Azure Container Apps consumption plan and only pay for what you use.

These are the triggers currently supported whilst in preview that allow dynamic scaling from zero instances:

Http
Azure Storage Queue
Azure Service Bus
Azure Event Hubs
Kafka Trigger

I'm certain that more triggers will be available down the line as KEDA itself has a huge library of various trigger types.

Demo

I will run through a quick demo which is also available on the Microsoft docs. It's a PowerShell Function that responds to an http request (API, Get) and returns a response if the request is successful. Pretty simple but you can develop whatever function you desire in all the languages currently supported by Azure Functions (.net, Java etc). I also ran some load Tests using Azure Load Testing (JMeter) to see how the scaling performs in real life so you can check that out at the end of the post.

Architecture

This is what we will be getting up and running today. The exception being that we won't be using virtual network integration or private endpoints just to speed things up but in a production environment you definitely want to enforce all the network traffic to remain within a virtual network in Azure.

Pre-Requisites

You will need the following if you want to run this yourself

Azure Functions Core Tools version 4.x.
Install the .NET 6 SDK.
Azure CLI version 2.4 or a later version.
An Azure Container Registry to push the container images to.

Set up local development environment

Before we deploy anything to Azure, let's get our Function set up locally and testing it using Postman (you can also just use a web browser).

1: Initiate the Functions project



func init --worker-runtime powershell --docker

The docker feature flag will generate a .Dockerfile for the project which we will use later to wrap up the code into a Docker image and push it up to the container registry.

2: Let's add a Function to our project.



func new --name HttpTriggerContainerApp --template "HTTP trigger" --authlevel anonymous

3: Once that's done, we need to test it locally to make sure our Function executes properly before we containerise it. To start the Function locally run:



func start

You should see the URL for the Function in your terminal: http://localhost:7071/api/HttpTriggerContainerApp

4: Now let's call our API using by suppliying a query parameter in the Http Get request. If you haven't got Postman you can just call pass the parameter via your web browser:

http://localhost:7071/api/HttpTriggerContainerApp?name=Functions

Now we should have a response if the call was successful:

Now we know the Function executes using the Http trigger locally we can move on to build our image and getting the same Function executing in Azure Container Apps.

Building our Docker image

Our Docker image was automatically generated when we created our Functions project so all we need to do is build it, tag it and push it up to our container registry.

You will need Docker to do this step. I am using Podman which is an alternative for building OCI images but the commands are the same. Just replace podman with dockerin the command line. Make sure you cd into the root directory of the project.

You will need to replace the acr_name with your registry in Azure. Make sure you have the admin credentials enabled on the registry. You can do this via the CLI az acr update -n <acr_name> --admin-enabled true

1: Build the image:



podman build --tag <acr_name>.azurecr.io/azurefunctionsimage:v1.0.0 .

2: Run the image locally and test it again via podman or your web browser:



podman run -p 8080:80 -it <acr_name>.azurecr.io/azurefunctionsimage:v1.0.0

3: Our container is now listening on port 8080 so here is the new URL to test with: http://localhost:8080/api/HttpTriggerContainerApp?name=Functions

Now we know our image works when running in a container we can push it our container registry.

4: Log in to your Azure Container Registry:



podman login <acr_name>.azurecr.io -u <acr_username> -p <acr_password)

5: Push the image to the container registry by running the following:



podman push <acr_name>.azurecr.io/azurefunctionsimage:v1.0.0

Now that our image is ready we can proceed to setting up our Azure environment and deploying our Function App on to Azure Contaiener Apps.

Deploy Function on to Azure Container Apps

1: Run the below commands to register the required namespaces and to upgrade the Container Apps CLI extension:



az extension add --name containerapp --upgrade -y
az provider register --namespace Microsoft.Web 
az provider register --namespace Microsoft.App 
az provider register --namespace Microsoft.OperationalInsights

2: Create the resource group for the resources in Azure:



az group create --name AzureFunctionsDemo-rg \
--location australiaeast

3: Create the Container App environment:



az containerapp env create --name azure-functions-demo-cae \
--resource-group AzureFunctionsDemo-rg \
--location australiaeast

4: Create the storage account required for the Function Apps. This is used to store all of the Function code used in the runtime:



az storage account create --name auefunctionsdemo001 \
--location australiaeast \
--resource-group AzureFunctionsDemo-rg \
--sku Standard_LRS

4: Create the Function App. You will notice here we need to provide the container registry details. This is what allows Azure Function to deploy our Function on to the Container App Environment:



az functionapp create --name azure-functions-demo-func \
--storage-account auefunctionsdemo001 \
--environment azure-functions-demo-cae \
--resource-group AzureFunctionsDemo-rg  \
--functions-version 4 --runtime powershell \
--image <acr_name>.azurecr.io/azurefunctionsimage:v1.0.0 \
--registry-server <acr_name>.azurecr.io \
--registry-username <acr_username> \
--registry-password <acr_password>

In a production scenario you would use a managed identity to authenticate to the container registry.

5: Let's retrieve the endpoint of our function so we can call this and test that we get the expected response from the API:



az functionapp function show \
--resource-group AzureFunctionsDemo-rg \
--name azure-functions-demo-func \
--function-name HttpTriggerContainerApp \
--query invokeUrlTemplate

Now that we have the endpoint, we can test it the same way we did when we deployed it locally. I am using Postman to do this.

We can now see that we get the exact same response with the Function running on Azure container apps!

This was a relatively simple setup that shows the concept of running Functions on Container Apps. The benefits are that we can use the feature of Container Apps on our Functions like Keda and DAPR. Additionally, it seems it's cheaper to run Azure Container Apps with vNet integration that running a standard App Service Plan or Azure Functions premium which are required in order to integrate into your own virtual network.

P.S You can configure your Container App Environment in your own virtual network allowing your Function app to communicate privately over RFC1918 to any other Azure services you have or private endpoints.

New Azure Service Retirement Workbook from Microsoft

Martin — Wed, 14 Jun 2023 12:37:50 +0000

Microsoft regularly releases new features and services in Azure. This is the evergreen nature of the cloud however they also retire services which is where infrastructure lifecycle management comes into play. Some new releases never make it out of preview and some services become legacy and get retired. A good example is a lot of the Azure classic services which were introduced when Azure was called Windows Azure.

Microsoft has released a neat little workbook that provides a central view of service retirements across Azure and specifically which resources in your subscriptions may be impacted to make it easier for customers to track these retirements. Previously we had to rely on Azure Advisor and checking the Microsoft update blog.

Here is how you can view this in your Azure tenant.

Step 1 - Navigate to the Azure workbooks Gallery and select the Service Retirement workbook

Now you should be able to see a list of services that may be impacted across your subscriptions. I have also enabled auto refresh and set the frequency to daily.

In the list above you can see I only have one resource that is affected and this is my public IP addresses with a basic SKU. Something Microsoft is retiring so only standard SKU will be available going forward.

You can also filter the view to show all retirements rather than a list of just retirements where resources in your subscriptions may be impacted.

Additionally, you can open a up a link for each retirement which will take you to the Microsoft Docs so you can find more details and remediation guidance.

Do note that the workbook is in public preview so not all services that are retiring are listed. Microsoft will be expanding on these in the future and hopefully this will make it to GA.

Pretty cool. Hopefully something you can leverage in your environment.

Convert ARM Template to Bicep using VS Code Extension

Martin — Tue, 11 Apr 2023 23:49:25 +0000

Introduction

Microsoft has just announced a new enhancement to their Bicep extension for Visual Studio code and I thought I would share this with the wider community.

Many of us have been working with ARM templates for a long time and it is evident that Microsoft is now pushing for Bicep adoption which is leaving a lot of engineers having to refactor existing ARM template deployments to Bicep.

Microsoft has now made this a lot easier with the new feature on the Bicep Visual Studio Code extension that allows you to paste ARM templates (JSON) into the Bicep format.

Pre-Requisites

Make sure you have the Bicep extension installed in Visual Studio Code.

Some JSON code that you want to convert. We will be using an Azure Container Registry for this demo.

Converting your template.

In this example we will convert the following template for an Azure Container Registry:



{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "registries_example_name": {
            "defaultValue": "example",
            "type": "String"
        }
    },
    "variables": {},
    "resources": [
        {
            "type": "Microsoft.ContainerRegistry/registries",
            "apiVersion": "2023-01-01-preview",
            "name": "[parameters('registries_example_name')]",
            "location": "australiaeast",
            "sku": {
                "name": "Basic",
                "tier": "Basic"
            },
            "properties": {
                "adminUserEnabled": true,
                "policies": {
                    "quarantinePolicy": {
                        "status": "disabled"
                    },
                    "trustPolicy": {
                        "type": "Notary",
                        "status": "disabled"
                    },
                    "retentionPolicy": {
                        "days": 7,
                        "status": "disabled"
                    },
                    "exportPolicy": {
                        "status": "enabled"
                    },
                    "azureADAuthenticationAsArmPolicy": {
                        "status": "enabled"
                    },
                    "softDeletePolicy": {
                        "retentionDays": 7,
                        "status": "disabled"
                    }
                },
                "encryption": {
                    "status": "disabled"
                },
                "dataEndpointEnabled": false,
                "publicNetworkAccess": "Enabled",
                "networkRuleBypassOptions": "AzureServices",
                "zoneRedundancy": "Disabled",
                "anonymousPullEnabled": false
            }
        }
    ]
}

1: Create a new file with the Bicep extension e.g. test.bicep

2: Right click on the file in Visual Studio Code and select "Paste JSON as Bicep"

3: Your template should now be converted to Bicep!

Here is the Bicep code:




param registries_example_name string = 'example'

resource registries_example_name_resource 'Microsoft.ContainerRegistry/registries@2023-01-01-preview' = {
  name: registries_example_name
  location: 'australiaeast'
  sku: {
    name: 'Basic'
    tier: 'Basic'
  }
  properties: {
    adminUserEnabled: true
    policies: {
      quarantinePolicy: {
        status: 'disabled'
      }
      trustPolicy: {
        type: 'Notary'
        status: 'disabled'
      }
      retentionPolicy: {
        days: 7
        status: 'disabled'
      }
      exportPolicy: {
        status: 'enabled'
      }
      azureADAuthenticationAsArmPolicy: {
        status: 'enabled'
      }
      softDeletePolicy: {
        retentionDays: 7
        status: 'disabled'
      }
    }
    encryption: {
      status: 'disabled'
    }
    dataEndpointEnabled: false
    publicNetworkAccess: 'Enabled'
    networkRuleBypassOptions: 'AzureServices'
    zoneRedundancy: 'Disabled'
    anonymousPullEnabled: false
  }
}

As you would expect some manual enhancements may be required at times, but this seems to make the process much easier.

You can also paste single lines or parts of code from JSON and it will also automatically convert them which is a nice feature.

Adding linting, validation and what-if to Bicep templates in Azure DevOps

Martin — Sun, 26 Feb 2023 22:10:57 +0000

We are picking up from my last post where I walked through automating Bicep template deployments through Azure DevOps so if you haven't caught up here is a link

The pipeline we set up in the previous post ran a single job that deployed our Azure resource using Bicep. However, in a production scenario we want to run some validation checks to ensure our new code won't breaking anything. Stages in Azure DevOps pipelines comprise of jobs and tasks e.g. running a script and you can think of them as containers for grouping different jobs.

Let's look at the new stages that we will be adding to our pipeline.

Let's understand each of the stages:

Lint: Use the Bicep linter to verify that the Bicep file is well formed and doesn't contain any obvious errors.
Validate: Use the Azure Resource Manager preflight validation process to check for problems that might occur when you deploy.
Preview: Use the what-if command to validate the list of changes that will be applied against your Azure environment. This allows people to review and approve the changes before deployment.
Deploy: Submit your deployment to Resource Manager and wait for it to finish pending the approval gate.

Converting our pipeline to a multistage pipeline

Since we want to run multiple stages we need to add some additional code to our YAML template. This is going to look something like this:



stages:

- stage: Lint
  jobs:
  - job: LintCode

- stage: Validate
  dependsOn: Lint
  jobs: 
  - job: ValidateBicepTemplate

- stage: Preview
  dependsOn: Validate
  jobs: 
  - job: Preview

- stage: Deploy
  dependsOn: Preview
  jobs: 
  - job: Deploy

By default stages run in parallel if you have enough agents however, we want to run these stages sequentially so I have added a dependsOn: to achieve this meaning that each stage can only run once the previous stage completes successfully.

This is the basic structure and in the next few sections we will build up each stage of the pipeline.

Adding Linting

Linting validation will check for things like unused parameters, unused variables, interpolation, secure parameters and more. The first thing we need to do is add the lint stage to our pipeline:



trigger:
- master

name: Deploy Bicep files


variables:
  vmImageName: 'ubuntu-latest'

  azureServiceConnection: 'devops-poc3-deploy-spi'
  location: 'australiaeast'
  templateFile: 'bicep/azure_resource_group/template.bicep'
  templateParameterFile: 'bicep/azure_resource_group/parameters.json'

pool:
  vmImage: $(vmImageName)

stages:  

- stage: Lint
  jobs:
  - job: LintCode
    displayName: Lint code
    steps:
      - script: |
          az bicep build --file $(Build.SourcesDirectory)/$(templateFile)
        name: LintBicepCode
        displayName: Run Bicep linter

Azure pipeline doesn't treat lint warnings as problems, so we need to add a configuration file to the repository where our bicep file is located. Copy and paste the below and name the file bicepconfig.json:



{
  "analyzers": {
    "core": {
      "enabled": true,
      "verbose": true,
      "rules": {
        "adminusername-should-not-be-literal": {
          "level": "error"
        },
        "max-outputs": {
          "level": "error"
        },
        "max-params": {
          "level": "error"
        },
        "max-resources": {
          "level": "error"
        },
        "max-variables": {
          "level": "error"
        },
        "no-hardcoded-env-urls": {
          "level": "error"
        },
        "no-unnecessary-dependson": {
          "level": "error"
        },
        "no-unused-params": {
          "level": "error"
        },
        "no-unused-vars": {
          "level": "error"
        },
        "outputs-should-not-contain-secrets": {
          "level": "error"
        },
        "prefer-interpolation": {
          "level": "error"
        },
        "secure-parameter-default": {
          "level": "error"
        },
        "simplify-interpolation": {
          "level": "error"
        },
        "protect-commandtoexecute-secrets": {
          "level": "error"
        },
        "use-stable-vm-image": {
          "level": "error"
        }
      }
    }
  }
}

Our pipeline will now fail if any of the linting flags warnings.

Adding Bicep template validation

We can use the Azure Resource Manager Deployment Task from the previous post to also run a validation against the ARM API to check for anything that may cause the deployment to fail e.g. reserved name for a storage account.

Let's get this stage added. Copy and paste the below straight after the lint stage (careful of the YAML indentation):



- stage: Validate
  dependsOn: Lint
  jobs: 
  - job: ValidateBicepTemplate
    steps:
    - task: AzureResourceManagerTemplateDeployment@3
      inputs:
        deploymentScope: 'Subscription'
        azureResourceManagerConnection: '$(azureServiceConnection)'
        location: '$(location)'
        templateLocation: 'Linked artifact'
        csmFile: '$(Build.SourcesDirectory)/$(templateFile)'
        csmParametersFile: '$(Build.SourcesDirectory)/$(templateParameterFile)'
        deploymentMode: 'Validation'

Adding What-If (Preview) analysis.

Before deploying the changes, we can run a what-if operation that will give you a preview of the template will change. This is a really cool feature and very similar to Terraform Plan where you can see exactly the number of changes the Bicep template will perform (if any).

Let's get this stage added. Copy and paste the below straight after the validation stage:



- stage: Preview
  dependsOn: Validate
  jobs: 
  - job: Preview
    steps:
    - task: AzureCLI@2
      inputs:
        azureSubscription: '$(azureServiceConnection)'
        scriptType: 'bash'
        scriptLocation: 'inlineScript'
        inlineScript: |
          az deployment sub what-if \
            --template-file $(Build.SourcesDirectory)/$(templateFile) \
            --parameters '$(Build.SourcesDirectory)/$(templateParameterFile)' \
            --location '$(location)'

Run the pipeline and select the preview stage to look at the results:

We can see that the what-if is showing that one resource will be created and in this instance it's the resource group we defined in the previous post.

Add approval gates

The final step is to add an approval gate to our deploy stage which will deploy the Bicep template with the desired infrastructure.

Paste the following code after the preview stage:



- stage: Deploy
  jobs:
    - deployment: Deploy
      environment: BicepEnvironment
      strategy:
        runOnce:
          deploy:
            steps:
            - checkout: self

            - task: AzureResourceManagerTemplateDeployment@3
              name: DeployBicepTemplate
              inputs:
                deploymentScope: 'Subscription'
                azureResourceManagerConnection: '$(azureServiceConnection)'
                action: 'Create Or Update Resource Group'
                location: '$(location)'
                templateLocation: 'Linked artifact'
                csmFile: '$(Build.SourcesDirectory)/$(templateFile)'
                csmParametersFile: '$(Build.SourcesDirectory)/$(templateParameterFile)'
                deploymentMode: 'Incremental'
                deploymentName: 'DeployPipelineTemplate'

You will notice that in order to use an environment for approvals we have added a deployment which is structured slightly differently from a regular job. An environment is simply a place where you deploy your solution and allows you to add approval gates and more.

Let's run the pipeline and see the approval in action:

Because we are using an environment, we need to allow our pipeline permissions to deploy using the environment:

Once we allow our pipeline access to the environment in Azure DevOps we can go ahead and approve the changes:

Here is our Resource Group deployed in Azure:

Here is the complete multi-stage pipeline. You can also find the code on my GitHub page here.



trigger:
- master

name: Deploy Bicep files


variables:
  vmImageName: 'ubuntu-latest'

  azureServiceConnection: 'devops-poc3-deploy-spi'
  location: 'australiaeast'
  templateFile: 'bicep/azure_resource_group/template.bicep'
  templateParameterFile: 'bicep/azure_resource_group/parameters.json'

pool:
  vmImage: $(vmImageName)

stages:  

- stage: Lint
  jobs:
  - job: LintCode
    displayName: Lint code
    steps:
      - script: |
          az bicep build --file $(Build.SourcesDirectory)/$(templateFile)
        name: LintBicepCode
        displayName: Run Bicep linter

- stage: Validate
  dependsOn: Lint
  jobs: 
  - job: ValidateBicepTemplate
    steps:
    - task: AzureResourceManagerTemplateDeployment@3
      inputs:
        deploymentScope: 'Subscription'
        azureResourceManagerConnection: '$(azureServiceConnection)'
        location: '$(location)'
        templateLocation: 'Linked artifact'
        csmFile: '$(Build.SourcesDirectory)/$(templateFile)'
        csmParametersFile: '$(Build.SourcesDirectory)/$(templateParameterFile)'
        deploymentMode: 'Validation'        

- stage: Preview
  dependsOn: Validate
  jobs: 
  - job: Preview
    steps:
    - task: AzureCLI@2
      inputs:
        azureSubscription: '$(azureServiceConnection)'
        scriptType: 'bash'
        scriptLocation: 'inlineScript'
        inlineScript: |
          az deployment sub what-if \
            --template-file $(Build.SourcesDirectory)/$(templateFile) \
            --parameters '$(Build.SourcesDirectory)/$(templateParameterFile)' \
            --location '$(location)'

- stage: Deploy
  jobs:
    - deployment: Deploy
      environment: BicepEnvironment
      strategy:
        runOnce:
          deploy:
            steps:
            - checkout: self

            - task: AzureResourceManagerTemplateDeployment@3
              name: DeployBicepTemplate
              inputs:
                deploymentScope: 'Subscription'
                azureResourceManagerConnection: '$(azureServiceConnection)'
                action: 'Create Or Update Resource Group'
                location: '$(location)'
                templateLocation: 'Linked artifact'
                csmFile: '$(Build.SourcesDirectory)/$(templateFile)'
                csmParametersFile: '$(Build.SourcesDirectory)/$(templateParameterFile)'
                deploymentMode: 'Incremental'
                deploymentName: 'DeployPipelineTemplate'

In a production environment you would also want to add some smoke testing or pester testing which can be easily achieved with PowerShell.

Hope you enjoyed the read.

Deploying Bicep Template using Azure DevOps

Martin — Thu, 23 Feb 2023 02:39:05 +0000

In this short post we are going to look at automating the deployment of a Bicep template using Azure DevOps. Additionally, we are going to look at how you can convert an ARM template to Bicep.

You can find all the files used in this post on my GitHub here.

Azure Bicep is almost an abstraction of ARM (JSON) templates specifically developed for Azure and hence they are not cloud agnostic. The main difference is that Bicep is a lot more human readable and generally you need fewer lines of code to achieve the same result through the use of simple syntax.

You can read more about Bicep here

Some pre-requisites before we get started:

Azure CLI - Download here https://learn.microsoft.com/en-us/cli/azure/install-azure-cli
Azure DevOps Access
Azure Subscription

Converting an ARM Template to Bicep

With the Azure CLI Microsoft provides functionality for you to decompile an ARM/JSON template to Bicep. This is extremely useful if you have existing deployments, and you want to start managing them with Bicep.

For this tutorial we are going to perform a simple resource group deployment. Let's look at our ARM template. It's split into a template.json file which has the definition of what we want to deploy and a parameters.json where we provide specific named variables required by the template.json.

It's worth noting that the parameters file does not need to be converted. This remains in a JSON format.

Template File

{
  "$schema": "https://schema.management.azure.com/schemas/2018-05-01/subscriptionDeploymentTemplate.json#",
  "contentVersion": "1.0.0.1",
  "parameters": {
      "rgName": {
          "type": "string"
      },
      "rgLocation": {
          "type": "string"
      },
      "tags": {
          "type": "object",
          "defaultValue": {}
      }
  },
  "variables": {},
  "resources": [
      {
          "type": "Microsoft.Resources/resourceGroups",
          "apiVersion": "2018-05-01",
          "location": "[parameters('rgLocation')]",
          "name": "[parameters('rgName')]",
          "properties": {},
          "tags": "[parameters('tags')]"
      }
  ],
  "outputs": {}
}

Parameter File

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "rgName": {
            "value": "arm-template-test-rg"
        },
        "rgLocation": {
            "value": "australiaeast"
        },
        "tags": {
            "value": {}
        }
    }
}

Let's convert our template to Bicep

1: Make sure you have the Azure CLI installed on your local machine. Navigate to the folder containing your ARM template and run the following command:

az bicep decompile --file template.json

Once the process completes, you will notice another file has been added to the directory called template.bicep. This is our converted template.

2: Let's review the code for our new bicep file:

targetScope = 'subscription'
param rgName string
param rgLocation string
param tags object = {
}

resource rg 'Microsoft.Resources/resourceGroups@2018-05-01' = {
  location: rgLocation
  name: rgName
  properties: {
  }
  tags: tags
}

Here you can notice straight away how the template is much shorter and more concise. We went from 28 lines of code to 13. It's worth noting that the conversion for bigger files can have some issue with naming formats so you may need to do some polishing before it's done. We can use our existing JSON parameter file to supply the values required.

Deploy the template manually

Let's deploy the template manually to make sure it works using the Azure CLI.

1: Run the following Azure CLI command. Make sure to replace the template and parameter file name if you used something different in the previous step.

az deployment sub create `
  --name TestDeployment `
  --location australiaeast `
  --template-file template.bicep `
  --parameters parameters.json

We can now see the RG is ready in the Azure portal.

Let's delete the RG and deploy the same templates using Azure DevOps for full automation.

Deploy the template using Azure DevOps

1: Create a new pipeline from Azure DevOps by navigating to Pipelinesand then New pipelines

2: Select Azure Repos Git for your code repository

3: Select Starter pipeline

4: Paste the following code:

trigger:
- master

name: Deploy Bicep files


variables:
  vmImageName: 'ubuntu-latest'

  azureServiceConnection: 'devops-poc3-deploy-spi'
  location: 'australiaeast'
  templateFile: 'bicep/azure_resource_group/template.bicep'
  templateParameterFile: 'bicep/azure_resource_group/parameters.json'
pool:
  vmImage: $(vmImageName)

steps:
- task: AzureResourceManagerTemplateDeployment@3
  inputs:
    deploymentScope: 'Subscription'
    azureResourceManagerConnection: '$(azureServiceConnection)'
    action: 'Create Or Update Resource Group'
    location: '$(location)'
    templateLocation: 'Linked artifact'
    csmFile: '$(Build.SourcesDirectory)/$(templateFile)'
    csmParametersFile: '$(Build.SourcesDirectory)/$(templateParameterFile)'
    deploymentMode: 'Incremental'
    deploymentName: 'DeployPipelineTemplate'

In this scenario I am providing the template and parameter file as a linked artifact in Azure DevOps by specifying the location in the following parameters csmFile & csmParametersFile. The pipeline also has a trigger for CD so every merge to the main/master branch will re-run it. Since we are using incremental deployments this is pretty safe to do.

5: Save the pipeline and run it.

We have now deployed the resource group through Azure DevOps using Bicep.

In the next post we are going to look at how to perform some automated tests on our Bicep deployment using the same pipelines including code linting, deployment validation, what-if checks and approval gates.

Microsoft Defender for DevOps (Preview)

Martin — Mon, 07 Nov 2022 15:02:25 +0000

Microsoft recently announced the new Defender for DevOps which is an extension built in directly into Microsoft Defender for Cloud now allowing you to monitor and manage security across a number of pipeline environments including Github and Azure DevOps.

Some of the features include:

Vulnerability management in open source code.
Exposed secrets in code that have been hard coded.
Container image scanning
Pull Request annotations

This has been long awaited and finally it's here. In this short post we will go through how to on-board the feature using Defender for Cloud in Azure.

The Microsoft Security DevOps extension currently supports the following open source tools:

Name	Language	License
Bandit	Python	Apache License 2.0
BinSkim	Binary--Windows, ELF	MIT License
ESlint	JavaScript	MIT License
Credscan	Credential Scanner (also known as CredScan) is a tool developed and maintained by Microsoft to identify credential leaks such as those in source code and configuration files common types: default passwords, SQL connection strings, Certificates with private keys	Not Open Source
Template Analyzer	ARM template, Bicep file	MIT License
Terrascan	Terraform (HCL2), Kubernetes (JSON/YAML), Helm v3, Kustomize, Dockerfiles, Cloud Formation	Apache License 2.0
Trivy	container images, file systems, git repositories	Apache License 2.0

You can customise the extension and only run a selection of these tools which we will look at later in this post.

Adding the Azure DevOps connector in Microsoft Defender

At the time of writing, Defender for DevOps is currently in preview and only available in the Central US Azure region.

You will need the following permissions to get this configure:

Organisation administrator in Azure DevOps
Security Administrator role in Defender for Cloud.
Contributor role on the subscription you are creating the connector in.
In Azure DevOps, configure: Third-party applications gain access via OAuth, which must be set to On . Learn more about OAuth

1: Open up Microsoft Defender for Cloud in the Azure Portal and navigate to DevOps Security

2: Select "Add Connector" and choose Azure DevOps

3: Choose a Resource Group and give your connector a name (globally unique).

4: Select the plan shown. Currently this is free as the service is in preview

5: On the next section select "Authorize". This will authorise your account and give Microsoft Defender permissions to your Azure DevOps organisation. You will see a pop up appear which will prompt for sign in. Make sure you sign in with the right account and have the right organisation selected if you are part of multiple. This caught me out on the first attempt and the creation failed.

6: Next on the same screen you will need to select which Azure DevOps projects and repositories you want to grant access to. You can either select specific ones or use auto discovery which will on-board everything and any future ones that are created.

7: Review the configuration and select "create".

8: You should then see the environment successfully connected in Microsoft Defender in the Azure DevOps security blade:

Next we will look at the installing the extensions in Azure DevOps and setting up the pipeline required to perform the scans.

Configuring the extensions and pipeline in Azure DevOps

1: Navigate to Azure DevOps and select manage extensions at the top right of your screen:

2: Select "Browse marketplace" and install the following extensions:

Microsoft Security DevOps
SARIF SAST Scans Tab (Add a tab to your build to show the scan results)

Configure the security scan pipeline

In order to scan our code we need to configure the pipeline with the required tooling from Microsoft that runs the extension.

1: Create a new starter pipeline in your repository and paste the following code (more detailed instructions provide by MS here:

# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
trigger: none
pool:
  vmImage: 'windows-latest'
steps:
- task: UseDotNet@2
  displayName: 'Use dotnet'
  inputs:
    version: 3.1.x
- task: UseDotNet@2
  displayName: 'Use dotnet'
  inputs:
    version: 5.0.x
- task: UseDotNet@2
  displayName: 'Use dotnet'
  inputs:
    version: 6.0.x
- task: MicrosoftSecurityDevOps@1
  displayName: 'Microsoft Security DevOps' 
  inputs:
    categories: 'IaC,secrets'
    tools: 'terrascan,credscan'

I mainly store Terraform code in my selected repository and so I have specified that I want to run the terrascan and 'credscan" tool and I want to specifically scan for Infrastructure As Code misconfiguration and exposed secrets in my code.

2: Run the pipeline and wait for the job to complete. Once this is done you can view the results in the scan tab:

3: You can now view the results in the scan tab. You can see I have quite a few areas in TF where adjustments are needed:

You can also customise the sensitivity of the scans. More information on how to configure the analysers on Github

Additionally the scan results are also published as an artifact:

4: Next you can navigate back to Microsoft Defender for Cloud in Azure and view the results and security posture directly there giving you a unified single pane of glass:

Enable pull request annotations

Defender for DevOps allows you to expose the above security findings as annotations in pull requests so developers know where issue need to be resolved before the code is merged into the main branch. This prevents issues before any code reaches production.

Enable build validation on your main branch in Azure DevOps

1: Sign in to Azure DevOps and navigate to project settings and then repositories

2: Select your repository and then navigate to policies and then select your main branch.

3: Navigate to the build validation section and enable the build validation with the default settings. Select the pipeline we created earlier in the build pipeline dropdown. Give it an appropriate name:

Enable pull request annotation in Defender for Cloud

1: Login to the Azure portal and navigate to Defender for Cloud > DevOps Security

2: Select the relevant repository to enable the pull request annotation and select configure:

3: Enable the annotations and select a category. Currently only secrets are supported whilst in preview.

Now we are all set. I have intentionally hard coded a secret in my repository and created a pull request to show the annotations working. The pull request triggered the build validation we set up earlier which performs a new scan.

This was just a crash course on this preview feature and there is defiantly a lot more left to explore however it's great to be able to perform these scans with a single extension and view them directly in Microsoft Defender now providing the capability not only to view your security posture in Azure but also in Azure DevOps.

Create Azure AD groups in bulk using PowerShell with a CSV input.

Martin — Mon, 26 Sep 2022 12:48:14 +0000

I recently had to replicate an Azure AD group structure from one tenant to another and wrote a super simple script to speed things up.

Set up PowerShell

Firstly you will need to install the Azure AD PowerShell module which allows you to intreact with AAD.

Install-Module AzureADPreview

Authenticate to your Azure Active Directory tenant by running

Connect-AzureAD -TenantId "<insert tenant id here>"

Retrieve AD Groups

I used a simple command to export the AD groups from the tenant I want to copy from to a csv file

Get-AzureADMSGroup | select DisplayName | Export-Csv c:\development\azureadusers.csv -NoTypeInformation

You can modify the export path as needed.

Create the AD Groups

Now that you have the AD groups exported you can run the script. Make sure you re-authenticate to the new Tenant where you want o set up the groups. Here is an example of what my CV looks like:

Note that I am only filtering the Display Names of the AD groups without the descriptions and owners. I haven't had a chance to script this up but it's fairly straight forward to do.

Run the script to set up the groups:

$Groups = Import-Csv -Path "C:\development\azureadusers.csv"

foreach($Group in $Groups)

{
New-AzureAdGroup -DisplayName $Group.DisplayName -MailEnabled $False -SecurityEnabled $True -MailNickName "NotSet"
}

The AD groups are now created!

There are some limitations with the Azure AD module. It doesn't support all parameters so for example you can't enable groups to be assignable to roles. You can achieve the same by using the Az.Resources module

You can also expand the script to add descriptions to the AD groups and replicate/assign owners.

Hope this helps.

Containerising a simple JavaScript application using Docker - Part 1

Martin — Wed, 31 Aug 2022 21:51:40 +0000

Introduction

In this tutorial we are going to be taking a simple Javascript application (source code taken from Docker) and running it in a container using Docker.

This is the first of a series of posts on containers. Here is a breakdown of what's to come.

Part 1 will look at putting a simple app into a Docker container

2: Part 2 will look at building some persistent storage for our application using Dockeer.

3: Part 3 will look at taking our application we containerised using Docker and putting it into a local Kubernetes deployment using minikube.

4: Part 4 will look at taking our application and deploying it to an Azure Kubernetes Cluster and exposing it publicly.

5: Part 5 will look at adding an ingress controller and WAF in front of our application using Azure Application Gateway.

Docker has grown in adoption since it was first introduced in 2013. Docker engine is by no means the only containerisation technology out in the market. You can use open source tools such as Buildah or Podman to achieve the same results. Docker actually consists of a number of products however it is most commonly known for Docker engine.

The images we are going to be building are of file type .Dockerfile however these are actually OCI images. OCI or open container initiative is an open source governance structure for creating open industry standards for container formats and runtimes. What this means is that you can any OCI formatted image is generally transportable to any container runtime supporting OCI and not just Docker so you could actually run your image using the containerd runtime or a runtime that supports OCI.

Containers vs Virtualisation

Before we dive into the tutorial here is a comparison of containers vs virtualisation. In simple terms, containers allow you to run multiple applications on a single operating system on a single server whereas virtualisation allows you to run multiple operating systems on a single physical server. This means containers are portable and lightweight as they do not carry the burden of the operating system. You can run containers either on bare metal servers or more commonly on virtual machines so typically containers run on top of OS virtualisation.

Prerequisites

There are some tools required in order to put our sample application into Docker as listed below.

Docker Desktop
WSL2 if you are using windows. This allows you to run a Linux subsystem if your are using a windows machine. If you are using Mac OS x then this is not required.
Git

Putting our app in a container

The application we are going to containerise is a simple to do list built using Javascript. I took the sample code from the Docker docs any take no credit for building it.

1: Start of by creating a fork of my repository located on Github.

2: Clone the repository to your local machine by running Git clone

3: Navigate to the docker/app directory where you will see the Dockerfile for the application and the source code which we will compile when building the image.

Before we build the image let's breakdown the instructions in the Dockerfile. A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image. These are what's used to build our image.



FROM node:12-alpine
RUN apk add --no-cache python2 g++ make
WORKDIR /app
COPY . .
RUN yarn install --production
CMD ["node", "src/index.js"]
EXPOSE 3000

From - Here we are specifying an existing image from the public Docker repository to use as a base.
Run - RUN instruction is used to executes any commands on top of the current image and this will create a new layer.
WORKDIR -
COPY - COPY instruction is used to copy files, directories and remote URL files to the destination within the filesystem of the Docker Images.
CMD - CMD instruction is used to set a command to be executed when running a container. There must be only one CMD in a Dockerfile.
EXPOSE - EXPOSE instruction is used to inform about the network ports that the container listens on runtime

Let's build the image

4: Run Docker Build -t todo-list

The image will take a couple minutes to build. Once it's complete we can check the image exists by listing our docker images:



Docker image ls

We can see our image was built successfully and is tagged "latest". We can optionally use semantic versioning.

5: Let's run the application locally:



docker run -dp 3000:3000 todo-list

You should now be able to access the to-do list app on http://localhost:3000/ in your browser.

The -dp flag is telling Docker to detach the container and map the application port of 3000 to a local port on our machine and in this it's 3000. You can map the port the application is listening to on any port available on your machine.

Next steps

Currently any items added to the to-list will be removed once the container is stopped. In the next post we will attach some persistent storage to our application that will allow us to persist the items added to the list in local storage when the container is stopped so we don't loose the data.

Azure Load Testing (PaaS) using Apache JMeter

Martin — Mon, 27 Jun 2022 22:04:13 +0000

Overview

Azure Load Testing is a new managed load testing service from Microsoft (currently in preview) that lets you simulate volumes of traffic to your website regardless of where you are hosting your application. It uses Apache JMeter to both define and run the loads tests.

This can be specifically useful if you have a website that needs to scale on demand to meet peak performance requirements. Microsoft seem to be adding more abstraction layers in Azure which is nice to see and previously you would have to run these JMeter tests on your own compute environment or virtual machines.

In this article we are going to look at how you can create a sample load testing in the portal. The service is currently in preview and it seems like only manual deployment through the portal is available. There is also a comprehensive guide from MS on how you can can automate the load testing through CI/CD using Github actions so I won't be repeating something that is already well documented.

Create the Load Testing Service

Step 1: Navigate to the Azure marketplace and search for "Azure Load Testing". You can also try the following link after you have logged into the portal.

Step 2: Fill out the required details. In my case I set up a new resource group for the service and click "create".

Step 3: Navigate to the service once it's deployed. It should spin up in a few minutes.

Create the Load Test and View the Results

For the purposes of this demo we are going to set up one of the out the box tests which will hit an endpoint we specify. You can optionally upload your own JMeter scripts.

Step 1: Navigate to the "tests" blade in the left pane

Step 2: Input the required parameters. Here you can specify how many users you would like to simulate the load for, the test duration and the ramp up time for the virtual users. For the purposes of this demo we are going to load www.google.co.uk.

Step 3: The test will start running and it should complete based on the duration specified in the previous step:

Step 4: Once the test completes you can navigate over and analyse the results:

You can optionally export the results. For this test we defined no pass criteria so the test result will display as "not applicable".

Conclusion

This is certainly a neat new service to see from Microsoft. The big issue with load testing is having to spin up compute to execute your tests and then integrating this into your CI/CD from scratch. A lot of this friction is removed with this service.

This is only a fraction of the functionality so be sure to check out the Microsoft docs for more info.

Microsoft also provide full CI/CD integration for Azure load testing using Github actions meaning you can automatically run these tests when creating a PR (or any other trigger) and define pass/fail criteria which embeds load testing into the core of the software development lifecycle. I can also see a use for this when deploying infrastructure changes through IaC where changing the sizing of your compute may impact the performance when under high load.

Deploy Azure DevOps Self-Hosted Build Agents on Kubernetes (AKS) and scale them using KEDA

Martin — Wed, 22 Jun 2022 14:07:17 +0000

Overview

KEDA is an event driven autoscaler for Kubernetes that allows you to scale containers based on events.

It is a lightweight single purpose component that can be added to any Kubernetes cluster. Keda can also work alongside the Kubernetes horizontal autoscaler.

The diagram below (taken from the Keda docs) shows how Keda integrates with the horizontal pod autoscalers, external events and the Kubernetes API:

More information on Keda can be found in the official docs

In this tutorial will be going over how you can use deploy self-hosted build agents on to an Azure Kubernetes Cluster and scale them using KEDA (scaled jobs) based on the number of jobs in a build queue.

As a side note, Azure Container Apps also supports KEDA and scaling using the pipelines trigger however KEDA kills containers half through a job and scaling jobs are not yet supported in Container Apps.

All of the code for this project can be found on my Github page here

Tools Required

We will be using the following tools so make sure you have them installed on your local machine.

Helm (Kubernetes Package Manager) - Install Guide
Azure CLI - Install Guide
Kubernetes Command Line - Install Guide
Terraform (Infrastructure as Code) - Install Guide
Docker Desktop - Install Guide

Deploy Azure resourcse

Before we get started we need to deploy the Azure components that will host our solution. These are the following:

Azure Resource Group
AKS Cluster
Container Registry

The Kubernetes cluster will be basic with a single pool and single virtual machine. No advanced networking will be used for the purposes of proving the concept.

We will be deploying this through Terraform which is an infrastructure as code deployment tool. The Terraform deployment file can be found in my Git repository here.

Step 1: Fork the repository and create a local clone on your machine

Step 2: Navigate to the folder that contains the main.tf file and run:



Terraform Init

This will initialise Terraform and create the local state file.

Step 3: Now we are going to run Terraform plan which will list the resources that will be deployed. It should be 4 in total as listed above (1 resource is a role assignment to link AKS to ACR).



Terraform Plan

Step 4: Once we are happy with what Terraform is going to deploy we can run the apply stage which will deploy the resources into Azure.



Terraform Apply

You should now see the resources in the Azure portal:

Build the docker images

Next we need to build the Docker image for the Azure DevOps self-hosted agents. Microsoft have documented this quite for Docker here. I have modified the image slightly to include PowerShell and we are using run.sh rather than docker in the start.sh script as ContainerD is the new container runtime in AKS version 1.19 and higher. You can re-use my images for these next steps from my Github repo.

Step 1: Start up docker Desktop

Step 2: Navigate to the repository you cloned earlier and navigate to the folder that contains the docker image.

Step 3: Let's build the image and tag it:



Docker build -f <docker-image-path> -t <tag>

You can test the image by running the container locally which will register it in DevOps. Just supply the environment variables.



docker run -e AZP_URL= -e AZP_TOKEN= -e AZP_AGENT_NAME= -e AZP_POOL= <image>

Step 4: Let's login to the Azure Container Registry.



Docker login <login-server> -u <username> -p <password>

You can get the above parameters from the container registry in the portal:

Step 5: Let's push the image to the Azure Container Registry



Docker push <imagename:tag>

The image should now be in the container registry which allow the containers running the agents to pull it.

Install KEDA on to the AKS cluster

KEDA runs in a container on the Kubernetes cluster and it's not built in so we need to install it. The KEDA pod handled all of the event driven scaling. We are going to use Helm to the install however you can apply the manifests directly.

Make sure you are authenticated to thee AKS cluster before running these next steps.

Step 1: Add the helm repo



helm repo add kedacore https://kedacore.github.io/charts

Step 2: Update the helm repo



helm repo update

Step 3: Create a new namespace and install the KEDA helm chart



kubectl create namespace keda
helm install keda kedacore/keda --namespace keda

You should now see the KEDA pods running in the keda namespace:

We are now ready to start using KEDA for scaling our containers.

Deploying the agents

Set up a new Agent Pool in Azure DevOps

Before we apply the Kubernetes manifests we need to set up a new agent pool in DevOps.

Step 1: Set up a self hosted pool on the organisation level. Take a note of the pool id. This can be found in the URL when you select the pool:

https://dev.azure.com/organisation/_settings/agentpools?poolId=16&view=jobs

In this case it's 16.

Step 2: You will also need to generate a PAT token with Agent Pools read & manage permissions.

Step 3: You will need to encode the token to Base64. You can do this either through Bash or through this website.

Apply the manifests

Because we are going to be using scaling jobs, we cannot specify idle agents. This creates a problem as you can't queue an Azure pipelines job on an empty agent pool so a workaround is to deploy a static agent and turn it offline.

Apply the deployment.yaml from the cloned repository by running:



kubectl apply -f ./deployment.yaml

Make sure you replace the variables in the YAML for the agent pool name, organisation URL, image and PAT token. See examples below:

image: = <acr-name>.azurecr.io/<repository-name>:<tag>
AZP_URL = https://dev.azure.com/<organisation-name>
AZP_TOKEN = Base64 encoded token that we created earlier.
AZP_POOL = Name of the self-hosted agent pool in Azure DevOps

Once you have applied the manifest you should see the pod running in the default namespace:

We can now see the agent running in DevOps:

Disable the agent and leave it running.

Apply the KEDA Scale Job Manifest

The final step is to apply the scaled job object through our keda-scaled-jobs.yaml manifest. You will need to replace the values for the image, organisation, pool, token, and pool ID.



kubectl apply -f ./keda-scaled-jobs.yaml

Now that we have applied the scale object, KEDA will be listening to the build queue every 10 seconds (can be customised). I have included a load testing pipeline for Azure DevOps that triggers 10 jobs which run some PowerShell loops. Let's run some jobs and see the scaling in action:

Pods are spinning up:

Agents are coming online:

We have now configured our self-hosted agents to run in Docker using Kubernetes as the orchestrator and KEDA as the scaling engine!

You can customise KEDA on how often it monitors the job queue, maximum number of replicas and more.

Limitations

I have discovered some limitations with Keda and scaling jobs which I'm working on resolving:

If you run some jobs in DevOps and cancel them the containers keep running.
KEDA does not remove any pods in a "completed" state. This can be resolved with a custom clean up shell script running as a cron on the cluster however it may be possible to do it with KEDA.
Offline agents are not removed from the DevOps pool. I have a cron job running in Azure DevOps that cleans this up.

Next Steps

All of the above steps can of course be automated through a simple build pipeline either through Github actions or Azure DevOps pipelines.

I will be building all of this functionality into a Helm chart that will also support Github runners so keep an eye out!

Import your existing Azure infrastructure into Terraform using Azure Terrafy

Martin — Mon, 13 Jun 2022 21:31:50 +0000

Overview

Terraform is a great tool for defining your infrastructure using code and getting it deployed to Azure or your chosen cloud provider. One of the many challenges you may face is importing existing infrastructure into Terraform so that you can then manage it through code along with the benefits that come along with that.

You will often hear the term state file. Terraform must store state about your managed infrastructure and configuration. Thee state is used by Terraform to map real world resources in the cloud back to your configuration and to keep track of metadata.

This state is stored by default in a local file named terraform.tfstate however it can also be stored remotely and more commontly in an Azure storage account as a blog which is often what's chosen by large scale enterprises to avoid having local state files.

There are usually 3 steps that typically need to be completed in order to import your cloud resource into Terraform using the process provided by Hashicorp:

Define your infrastructure using Terraform.
Run Terraform import to update your state file.
Run Terraform plan to verify the import is successful.

This method is well defined in the Terraform docs.

In this tutorial we will be talking about a new tool called Azure Terrafy developed by Microsoft which aims to automate the import process by generating both the configuration and state file for your existing infrastructure making it much easier to get started with Terraform.

Installation

You can install Azure Terrafy using the Go Tool chain by running:

go install github.com/Azure/aztfy@latest

If you haven't got Go then you can install it by following the instructions on the official Go docs.

You can also manually download the latest Binary from the Azure Terrafy Github page.

Just make sure to update your environment path so you can run Terrafy from your chosen shell.

This Stack Overflow article contains instructions for setting the PATH on Windows through the user interface.

For Mac OS you can run echo $PATH and place the executable/binary into one of the listed paths.

Verify you have it correctly installed by running aztfy.

We are now ready to start importing our infrastructure into Terraform .

Importing your Infrastructure

Before we start the import we need to get the environment ready. Let's deploy an Azure Resource Group and a static web app resource using the Azure CLI and try to import it into Terraform using Terrafy to prove the concept.

Create Azure Resource Group

az group create --location "uksouth" --name "terafform-import-rg" --subscription "7309f068-5a47-4a28-851c-09979529cd8e"

You should get an output in the CLI once the RG provisioning is successful:

Create an Azure Static Web App Resource

Let's create an Azure Static Web App in the RG we created:

az staticwebapp create \
-n terraform-import-webapp \ 
-g "terafform-import-rg" \

You should get an output in the CLI that the provisioning was successful. Validate the Static Web App is deployed by navigating to the Resource Group in the Azure Portal:

Set up a new directory

Set up a new directory/folder where you want to Terrafy to store the Terraform files. By default it uses the root directory however you can also override this with a custom path.

1: Run mkdir <folder-name> to create a new folder.
2: Navigate into the directory above cd <path>

Import the Infrastructure using Terrafy

Make sure you have logged into Azure using by running az login and set the subscription that contains the RG and web app created earlier by running az account set -s <subscription_id>

1: Run the import command

aztfy terafform-import-rg

Terrafy will begin the import process by scanning the RG and presenting you with a list of resources to import. Navigation instructions are given at the bottom of your shell window.

2: Select the resource you want to import. If you want to import all of them just hit W on your keyboard.

In this case we want to import all our resources (static web app and RG) so we are going to hit W on our keyboard.

Terrafy will begin the import process and the Terraform files will be stored in the root directory we created earlier. You will be prompted once import is complete:

3: Verify the files have been generated:

We can see we now have our state and Terraform files. Viewing the main.tf file shows the definition for our static web app and resource group:

NOTE
You should note that currently Azure Terrafy does not support all resource types so if you see "skip" listed next to the resource this means it cannot be imported.

Validate the Terraform state

The final step is to validate that the Terraform state has been imported successfully by running Terraform.

1: Run terraform init
2: Run terraform plan

We can see that Terraform is reporting that no changes are required and that our infrastructure in Azure matches our configuration or desired state in Terraform.

You can also specify a remote backend before running aztfy:

aztfy --backend-type=azurerm --backend-config=resource_group_name=<resource group name> --backend-config=storage_account_name=<account name> --backend-config=container_name=<container name> --backend-config=key=terraform.tfstate <importing resource group name>

We have successfully imported our infrastructure into Terraform using Azure Terrafy!

Limitations

Currently the configuration is imported into a single main.tf files. Modules should be considered.
Dependencies. Aztfy uses many of the dependencies in your resource group to map those out in the templates (see the 'depends_on' line declared in the main.tf above), much of that can be cleaned up once naming conventions and modules are implemented.
Currently you can only import at the resource group level however support will be added for subscriptions and management groups.
Not all resources are currently supported for the import process.

This is certainly a great start to fully automating the TF import process and the tool will develop and grow making it much easier for enterprises to get started in managing their existing infrastructure with Terraform.

The above import process can be fully automated by running all the tasks in a DevOps pipeline and specifying only the RG name. Terrafy supports non-interactive "batch" mode which is CI/CD friendly.