Kohei Kawata

Posted on Aug 29, 2022

Virtual Network architecture 2 - Deployment pipelines

Summary

This article is Part.2 of virtual network architecture series. I will share the details of the entire pipeline in the sample template api-management-vnet.

Overview

The sample template api-management-vnet includes two scenario - 1) Base architecture without virtual network, 2) Virtual network architecture. Both architectures use the same Azure services below.

Used in both architectures

Azure Pipelines: Deploy infrastructure, build and deploy codes, build and deploy self-hosted agent, execute integration tests by sending REST API requests.
Azure API Management: Platform to manage APIs behind which are running on Azure App Service in this template.
Azure App Service: Hosting service of web APIs.
Azure Functions: Serverless hosting service that runs codes.
Azure SQL: SQL Server on the Azure cloud.
Azure Key Vault: Secret management service that stores keys and certificates accessible from other Azure services.
Azure Service Bus: Message broker in the Azure cloud with message queue and pub-sub topics.

Only in virtual network architecture

Azure Application Gateway: A Web traffic load balancer that works as a gateway of virtual network architecture in this template.

Architecture

Both architectures have the same Web API, Functions, Database, and Service Bus topic. Azure Pipelines conducts the same integration tests that sends API requests to the Web API and Service Bus topic. There are two important differences in the virtual network architecture from the base architecture.

Application Gateway resides in front of the API Management. API Management and App Service cannot be accessed from the internet but only through the Application Gateway. So, the API endpoint is different.
Self-hosted agent is built in the first pipeline and deployed inside the virtual network subnet. The second pipeline runs on this self-hosted agent so it can access the private endpoints of App Service and Service Bus.

Base architecture

Virtual network architecture

Pipelines

You can run the two pipelines by following getting-started documentation. In this article, I am going to explain the details of pipeline steps.

Two pipelines overview

Base architecture has only pipeline-base.yml but Virtual network architecture includes pipeline-vnet1.yml and pipeline-vnet2.yml. The first pipeline of virtual network architecture deploys Azure resources without private endtpoints and builds a self-hosted agent, and the second pipeline runs on the self-hosted agent and deploys and sets up private points for resources. In the end of the second pipeline, it executes integration tests inside virtual network.

pipeline-base.yml

Step 1. In keyvault-secret.yml, the pipeline creates Key Vault if it does not exist and pass secrets input from Azure Pipelines variables so you do not need to keep those secrets in the Azure Pipeline variables but after the second time the pipeline automatically detects pipeline variable and download secrets from existing Key Vault. This technique is described in Azure Pipelines secret management with Key Vault.

Step 2. azure-resource.yml deploys Azure resources with main.bicep.

You need to specify the secret variables handled in the previous Key Vault step. If you have more secret variables for improvement, you need to add variables here.

- job: AzureResourceDeployment
    variables:
    - template: ./variables.yml
    - name: SqlPass
      value: $[ stageDependencies.KeyVault.SQL_PASS.outputs['SQL_PASS.SQL_PASS'] ]
    - name: AadSecretClient
      value: $[ stageDependencies.KeyVault.AAD_SECRET_CLIENT.outputs['AAD_SECRET_CLIENT.AAD_SECRET_CLIENT'] ]

Azure API Management takes around one hour to deploy. In order to avoid a time out error, it specifies timeout: 0. However, I found errors sometimes after one hour. If you face a timeout error, you can just re-run the pipeline, and you will see everything goes smoothly after that.

- job: AzureResourceDeployment
    timeoutInMinutes: 0

Step 3. sql.yml deploys SQL Database schema with .dacpac file built by WeatherDB SQL project.

Step 4. func.yml deploys the .NET C# code of ServicebusListener to Azure Functions.

Step 5. webapi.yml deploys the .NET C# code of WeatherAPI project to Azure App Service. Furthermore, it defines the API in the API Management and generates swagger.json and deploys it to the API Management.

- task: DotNetCoreCLI@2
  displayName: dotnet new tool-manifest
  inputs:
    command: custom
    custom: new
    arguments: tool-manifest
    workingDirectory: ${{ parameters.apiProjectDirectory }}
- task: DotNetCoreCLI@2
  displayName: dotnet tool install
  inputs:
    command: custom
    custom: tool
    arguments: install Swashbuckle.AspNetCore.Cli --version $(SWASHBUCKLE_VERSION)
    workingDirectory: ${{ parameters.apiProjectDirectory }}
- task: DotNetCoreCLI@2
  displayName: dotnet swagger tofile
  inputs:
    command: custom
    custom: swagger
    arguments: tofile --output ${{ parameters.swaggerPath }} ${{ parameters.apiDllPath }} $(SWAGGER_VERSION)
    workingDirectory: ${{ parameters.apiProjectDirectory }}
- task: AzureCLI@2
  displayName: Deploy API to API Management
  inputs:
    azureSubscription: ${{ parameters.azureSvcName }}
    scriptType: bash
    scriptLocation: inlineScript
    inlineScript: |
      az apim api import -g $(RESOURCE_GROUP_NAME) \
        --service-name $(API_MANAGEMENT_NAME) \
        --api-id $(API_NAME) \
        --path /$(API_NAME) \
        --specification-format OpenApiJson \
        --specification-path ${{ parameters.swaggerPath }} \
        --service-url $(API_MANAGEMENT_SERVICE_URL)

Step 6. integration-test-powershell.yml executes integration tests for deployed Web API and Functions.

With the powershell script below, it extracts OAuth2.0 token from Azure Active Directory so it can attach the token later when sending a REST request to the API Management. Because the Azure Pipelines agent sends a request with automation, it follows Client Credential Flow of OAuth2.0. This OAuth2.0 flow is explained in Azure AD app configuration for Web API.

$clientSecret = Get-AzKeyVaultSecret -VaultName $env:KEYVAULT_NAME -Name $env:KVSECRET_NAME_AADCLIENT -AsPlainText
$authorizeUri = "https://login.microsoftonline.com/${{ parameters.aadTenantId }}/oauth2/v2.0/token"
$body = 'grant_type=client_credentials' + `
'&client_id=${{ parameters.aadAppidClient }}' + `
'&client_secret=' + $clientSecret + `
'&scope=api://${{ parameters.aadAppidBackend }}/.default'
$token = (Invoke-RestMethod -Method Post -Uri $authorizeUri -Body $body).access_token

The base architecture does not have Application Gateway and the endpoint is at the API Management. The route /Weatherforecast is to GET and POST the data to Azure SQL Databse and /Weatherforecast/message to send a message to Service Bus topic, which triggers Azure Functions and inserts that record in SQL Database.

parameters:
  url: https://$(API_MANAGEMENT_NAME).azure-api.net

$Uri = "${{ parameters.url }}/$env:API_NAME/Weatherforecast"
$UriMessage = "${{ parameters.url }}/$env:API_NAME/Weatherforecast/message"

pipeline-vnet1.yml

pipeline-vnet1.yml includes steps below.

Store secrets from Pipeline variables to Key Vault
Generate SSL certificate for Application Gateway backend pool
Deploy Azure services
Build a self-hosted agent
Deploy projects to App Service, Functions, and SQL Database
Execute integration tests

After this pipeline is executed, some resources such as App Service, Key Vault, SQL Database, Functions, and Service Bus are not protected within the virtual network. Only Application Gateway and API Management are connected to and located in the virtual network. The self-hosted agent is built and deployed to Azure Container Instances and ready for usage.

Step 1. cert-appgateway.yml generates and sets in Key Vault a SSL certificate for the backend pool of API Management. The Key Vault requires to grant the access right for Azure Pipeliens Microsoft hosted agent.

Set-AzKeyVaultAccessPolicy -VaultName $env:KEYVAULT_NAME -PermissionsToCertificates get,create -ObjectId ${{ parameters.aadObjectidSvc }}

Step 2. keyvault-secret.yml is the same template as used in the Base architecture. This time it adds AZP_TOKEN secret variable that is needed to deploy a self-hosted agent. How to get the personal access token is described in Getting-started.

Step 3. azure-resource.yml deploys Azure resources with main1.bicep. All the virtual networks and subnets are deployed this time with VirtualNetwork.bicep. All the virtual network IP address ranges are defined in main.parameters.json.

Step 4. restart-appgateway.yml restarts Application Gateway. I faced errors without restarting Application Gateway. According to the Microsoft documentation Updates to the DNS entries of the backend pool, when you change DNS entries for the backend pool, Application Gateway must be restarted. In the Azure resource deployment with PrivateDns1.bicep, it creates the DNS A record for the API Management. Then you need to restart the Application Gateway.

resource PrivateDnsAApim 'Microsoft.Network/privateDnsZones/A@2020-06-01' = {
  name: '${PrivateDnsApim.name}/${ApiManagementName}'
  properties: {
    ttl: 3600
    aRecords: [
      {
        ipv4Address: ApiManagementPrivateIPAddress
      }
    ]
  }
}

Step 5. self-hosted-agent.yml builds and deploys a Linux based self-hosted agent.

As of August 2022, you can run a Windows based self-hosted agent in the docker container but Windows based Azure Container Instance cannot be deployed on the virtual network, according to the Microsoft documentation Deploy container instances into an Azure virtual network. Windows based self-hosted agent is needed to deploy SQL Database projects, but for now the Windows self-hosted agent build part is commented out.
The self-hosted agent container application is defined in Dockerfile.
After the container image is pushed into Azure Container Registry, the deployment to Azure Container Instance is implemented in linux-agent.bicep.

Step 6. sql.yml is the same as used in the Base pipeline and deploys SQL Database schema with .dacpac file built by WeatherDB SQL project.

Step 7. func.yml is the same as used in the Base pipeline and deploys the .NET C# code of ServicebusListener to Azure Functions.

Step 8. webapi.yml is the same as used in the Base pipeline and deploys the .NET C# code of WeatherAPI project to Azure App Service and defines the API in the API Management and generates swagger.json and deploys it to the API Management.

Step 9. integration-test-powershell.yml is the same as used in the Base pipeline and executes integration tests for deployed Web API and Functions.

pipeline-vnet2.yml

pipeline-vnet2.yml includes steps below.

Execute integration tests before setting private endpoints
Deploy and set private endpoints
Execute integration tests for private endpoints
Deploy projects to App Service and Functions with private endpoints by the self-hosted agent
Execute integration tests for code deployment through private endpoints.

This pipeline deploys and set private endpoints so all Azure services are protected by the virtual network. You cannot deploy new software to the App Service and Functions any more as they are not allowed to access through the Internet. The self-hosted agent running on Azure Container Instance is located inside the virtual network and capable of building and deploying the new software.

You can see this pipeline use the self-hosted agent by specifying the pool like below. All steps below can be executed on a Linux based agent.

pool:
  name: Default

Step 1. integration-test-bash.yml executes integration tests before private endpoints are deployed and set up. The self-hosted agent is Linux based, and this integration test is written in bash scripts. Obtaining Azure Active Directory OAuth2.0 token by bash script is written in the way below.

clientSecret=$(az keyvault secret show --vault-name $(KEYVAULT_NAME) --name $(KVSECRET_NAME_AADCLIENT) --query value -o tsv)
authorizeUri="https://login.microsoftonline.com/${{ parameters.aadTenantId }}/oauth2/v2.0/token"
token=$(curl -X POST $authorizeUri -d "client_id=${{ parameters.aadAppidClient }}&grant_type=client_credentials&scope=api://${{ parameters.aadAppidBackend }}/.default&client_secret=$clientSecret" | jq ".access_token" | tr -d \")

Step 2. private-endpoints.yml deploys Azure resources related to private endpoints with main2.bicep. All the virtual network IP address ranges are defined in main.parameters.json.

Step 3. restart-appgateway.yml restarts Application Gateway. This should be executed for the same reason as the V-net pipeline 1 because private DNS A records for Key Vault, App Service, SQL Database, Funtions, and Service Bus are added.

Step 4. integration-test-bash.yml executes integration tests after private endpoints are set. This step can make sure all the configurations of private endpoints are correctly deployed.

Step 5. func.yml is the same as used in the Base pipeline but this time is executed on the self-hosted agent. Functions do not have accessible public IP any more, and it should be connected through the private IP inside the virtual network.

Step 6. webapi.yml is the same as used in the Base pipeline but this time is executed on the self-hosted agent. The App Service does not have accessible public IP any more, and it should be connected through the private IP inside the virtual network.

Step 7. integration-test-bash.yml executes integration tests to make sure the code deployment to App Service and Funtions are correctly done through the self-hosted agent.