DEV Community: Kohei Kawata

Virtual Network architecture 6 - Service Bus Private Endpoint

Kohei Kawata — Mon, 29 Aug 2022 14:39:00 +0000

Summary

This article is Part.6 of virtual network architecture series that includes the details of the private endpoint Azure Service Bus in api-management-vnet.

Private Endpoint configuration

Private Endpoint: Deploy the private endpoint and connect it to Service Bus and its subnet in PrivateEndpoint.bicep.

ServiceBusId:existingSb.id
VirtualNetwork3SubnetIdSb:existingVnet3.properties.subnets[0].id

resource PrivateEndpointSb 'Microsoft.Network/privateEndpoints@2021-03-01' = {
  properties: {
    privateLinkServiceConnections: [
      {
        properties: {
          privateLinkServiceId: ServiceBusId
          groupIds: [
            'namespace'
          ]
        }
      }
    ]
    subnet: {
      id: VirtualNetwork3SubnetIdSb
      properties: {
        privateEndpointNetworkPolicies: 'Enabled'
      }
    }
  }
}

Private DNS: Deploy Private DNS with the DNS name privatelink.servicebus.windows.net in PrivateDns2.bicep.

var pdns_name_sb = 'privatelink.servicebus.windows.net'

resource PrivateDnsSb 'Microsoft.Network/privateDnsZones@2020-06-01' = {
  name: pdns_name_sb
  location: 'global'
}

Virtual network link: Link the deployed Private DNS to the virtual network 3 where the Service Bus resides and the virtual network 2 so Azure Functions can access the Service Bus topic in PrivateDns2.bicep. The virtual network 2 and 3 are connected with Virtual Network Peering.

VirtualNetwork2Id:existingVnet2.id
VirtualNetwork3Id:existingVnet3.id

resource VnetLinkSb2 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2020-06-01' = {
  name: '${PrivateDnsSb.name}/${PrivateDnsSb.name}-link2'
  location: 'global'
  properties: {
    registrationEnabled: false
    virtualNetwork: {
      id: VirtualNetwork2Id
    }
  }
}

resource VnetLinkSb3 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2020-06-01' = {
  name: '${PrivateDnsSb.name}/${PrivateDnsSb.name}-link3'
  location: 'global'
  properties: {
    registrationEnabled: false
    virtualNetwork: {
      id: VirtualNetwork3Id
    }
  }
}

Private DNS A record: Create a DNS A record and set the IP address from the deployed private endpoint in PrivateDns2.bicep.

output PrivateEndpointSbIpAddress string = PrivateEndpointSb.properties.customDnsConfigs[0].ipAddresses[0]

resource PrivateDnsASb 'Microsoft.Network/privateDnsZones/A@2020-06-01' = {
  name: '${PrivateDnsSb.name}/${ServiceBusName}'
  properties: {
    ttl: 3600
    aRecords: [
      {
        ipv4Address: PrivateEndpointSbIpAddress
      }
    ]
  }
}

Public network access

Public network access is disabled in ServiceBus2.bicep so all access except through the Private Endpoint are denied.

resource ServiceBus 'Microsoft.ServiceBus/namespaces@2022-01-01-preview' = {
  sku: {
    name: 'Premium'
  }
  properties: {
    publicNetworkAccess: 'Disabled'
  }
}

Support tier

According to the Microsoft documentation Azure Service Bus Private Endpoint, only Premium tier supports Azure Service Bus Private Endpoint feature.

Virtual Network architecture 7 - Self-hosted agent

Kohei Kawata — Mon, 29 Aug 2022 14:39:00 +0000

Summary

This article is Part.7 of virtual network architecture series. I will share how Azure Pipelines self-hosted agent in the docker works in the sample template api-management-vnet.

Azure Container Instance

Before I tried the self-hosted agent by myself, I thought the self-hosted agent running on Azure Container Instance is totally isolated from the Internet. And I was wondering how the self-hosted agent can access Azure Repos to get the updated software codes to deploy to services inside the virtual network. However, now I know Azure Container Instance is not isolated from the Internet at all. Below is some tips I found through my experience.

Only outbound traffic of Azure Container Instance should be considered. And then the self-hosted agent does not need to have Private Endpoint, because Private Endpoint is only for inbound traffic.
Similar with other Azure PaaS resources, a self-hosted agent in the docker running on Azure Container Instance has the public IP address. You have to protect Azure Container Instance in some ways, for example, described in Configure a single public IP address for outbound and inbound traffic to a container group.
You can find what runtime the self-hosted agent installs when seeing start.ps1 on Create and build the Dockerfile for Windows, for example. The runtime controls the outbound traffic from the Azure Container Instance, and you do not need to take care of it.

Virtual Network architecture 5 - App Service Private Endpoint

Kohei Kawata — Mon, 29 Aug 2022 14:39:00 +0000

Summary

This article is Part.5 of virtual network architecture series and includes the details of the private endpoint with Azure App Service and Azure Functions App in api-management-vnet.

Private Endpoint configuration

Private Endpoint: Deploy the private endpoint and connect it to App Service and Functions and their subnets in PrivateEndpoint.bicep. Both App Service and Functions need two subnets for each for the purpose of V-net integration and Private Endpoint.
- This sample template takes Regional V-net integration and it requires delegated subnet. The integration subnet has to be delegated to Microsoft.Web/serverFarms.
- The Private Endpoint for App Service and Function App has to have one subnet that is not the same subnet as the one for V-net integration, according to the Microsoft documentation Private Endpoint for App Service.

AppServiceId:existingAppService.id
FuncId:existingFunc.id
VirtualNetwork2SubnetIdAsePe:existingVnet2.properties.subnets[1].id
VirtualNetwork2SubnetIdFuncPe:existingVnet2.properties.subnets[5].id

resource VirtualNetwork2 'Microsoft.Network/virtualNetworks@2021-03-01' = {
  properties: {
    subnets: [
      {
        name: snet_name_2_ase_vi
        properties: {
          addressPrefix: snet_prefix_2_ase_vi
          networkSecurityGroup: {
            id: Nsg2AseViId
          }
          delegations: [
            {
              name: 'delegation'
              properties: {
                serviceName: 'Microsoft.Web/serverfarms'
              }
            }
          ]
          privateEndpointNetworkPolicies: 'Disabled'
        }
      }
      {
        name: snet_name_2_ase_pe
        properties: {
          addressPrefix: snet_prefix_2_ase_pe
          networkSecurityGroup: {
            id: Nsg2AsePeId
          }
          privateEndpointNetworkPolicies: 'Enabled'
        }
      }
      {
        name: snet_name_2_func_vi
        properties: {
          addressPrefix: snet_prefix_2_func_vi
          networkSecurityGroup: {
            id: Nsg2FuncViId
          }
          delegations: [
            {
              name: 'delegation'
              properties: {
                serviceName: 'Microsoft.Web/serverfarms'
              }
            }
          ]
          privateEndpointNetworkPolicies: 'Disabled'
        }
      }
      {
        name: snet_name_2_func_pe
        properties: {
          addressPrefix: snet_prefix_2_func_pe
          networkSecurityGroup: {
            id: Nsg2FuncPeId
          }
          privateEndpointNetworkPolicies: 'Enabled'
        }
      }
    ]
  }
}

Private DNS: Deploy Private DNS with the DNS name privatelink.azurewebsites.net in PrivateDns2.bicep. App Service and Functions can share one Private DNS Zone.

var pdns_name_app = 'privatelink.azurewebsites.net'

resource PrivateDnsApp 'Microsoft.Network/privateDnsZones@2020-06-01' = {
  name: pdns_name_app
  location: 'global'
}

Virtual network link: Link the deployed Private DNS to three of the virtual network in PrivateDns2.bicep. This is because App Service and Functions reach out to all three virtual networks. Also, the virtual network 1 and 2 are peered, and the virtual network 2 and 3, too.

VirtualNetwork1Id:existingVnet1.id
VirtualNetwork2Id:existingVnet2.id
VirtualNetwork3Id:existingVnet3.id

resource VnetLinkApp1 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2020-06-01' = {
  name: '${PrivateDnsApp.name}/${PrivateDnsApp.name}-link1'
  location: 'global'
  properties: {
    registrationEnabled: false
    virtualNetwork: {
      id: VirtualNetwork1Id
    }
  }
}

resource VnetLinkApp2 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2020-06-01' = {
  name: '${PrivateDnsApp.name}/${PrivateDnsApp.name}-link2'
  location: 'global'
  properties: {
    registrationEnabled: false
    virtualNetwork: {
      id: VirtualNetwork2Id
    }
  }
}

resource VnetLinkApp3 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2020-06-01' = {
  name: '${PrivateDnsApp.name}/${PrivateDnsApp.name}-link3'
  location: 'global'
  properties: {
    registrationEnabled: false
    virtualNetwork: {
      id: VirtualNetwork3Id
    }
  }
}

Private DNS A record: Create DNS A records and set the IP address from the deployed private endpoints in PrivateDns2.bicep. You have to create two A records each for App Service and Functions because both have another endpoint of SCM(Source Control Manager) site. You can see the documentation of Kudu service about it. Both private endpoints need the same private IP address configurations.

output PrivateEndpointAseIpAddress string = PrivateEndpointAse.properties.customDnsConfigs[0].ipAddresses[0]
output PrivateEndpointFuncIpAddress string = PrivateEndpointFunc.properties.customDnsConfigs[0].ipAddresses[0]

resource PrivateDnsAAse 'Microsoft.Network/privateDnsZones/A@2020-06-01' = {
  name: '${PrivateDnsApp.name}/${AppServiceName}'
  properties: {
    ttl: 3600
    aRecords: [
      {
        ipv4Address: PrivateEndpointAseIpAddress
      }
    ]
  }
}

resource PrivateDnsAAseScm 'Microsoft.Network/privateDnsZones/A@2020-06-01' = {
  name: '${PrivateDnsApp.name}/${AppServiceName}.scm'
  properties: {
    ttl: 3600
    aRecords: [
      {
        ipv4Address: PrivateEndpointAseIpAddress
      }
    ]
  }
}

resource PrivateDnsAFunc 'Microsoft.Network/privateDnsZones/A@2020-06-01' = {
  name: '${PrivateDnsApp.name}/${FuncName}'
  properties: {
    ttl: 3600
    aRecords: [
      {
        ipv4Address: PrivateEndpointFuncIpAddress
      }
    ]
  }
}

resource PrivateDnsAFuncScm 'Microsoft.Network/privateDnsZones/A@2020-06-01' = {
  name: '${PrivateDnsApp.name}/${FuncName}.scm'
  properties: {
    ttl: 3600
    aRecords: [
      {
        ipv4Address: PrivateEndpointFuncIpAddress
      }
    ]
  }
}

Inbound and Outbound

App Service and Functions have different network features of inbound and outbound. Inbound works with Private Endpoint and outbound with V-net integration, and both need to have two different subnets for each.

Access restrictions

According to the Microsoft documentation of Private Endpoints for Azure Web App, by default, enabling Private Endpoints to your Web App disables all public access. You do not need to configure Access restrictions.

Support tier

Microsoft documentation of Private Endpoints for Azure Web App describes only Basic, Standard, PremiumV2, PremiumV3, IsolatedV2, Functions Premium support Private Endpoint.

Virtual Network architecture 4 - SQL Database Private Endpoit

Kohei Kawata — Mon, 29 Aug 2022 14:38:00 +0000

Summary

This article is Part.4 of virtual network architecture series. I explain the details of the private endpoint with Azure SQL Database in api-management-vnet.

Private Endpoint configuration

Private Endpoint: Deploy the private endpoint and connect it to SQL Database and its subnet in PrivateEndpoint.bicep.

SqlServerId:existingSql.id
VirtualNetwork2SubnetIdSql:existingVnet2.properties.subnets[2].id

resource PrivateEndpointSql 'Microsoft.Network/privateEndpoints@2021-03-01' = {
  properties: {
    privateLinkServiceConnections: [
      {
        properties: {
          privateLinkServiceId: SqlServerId
          groupIds: [
            'sqlServer'
          ]
        }
      }
    ]
    subnet: {
      id: VirtualNetwork2SubnetIdSql
      properties: {
        privateEndpointNetworkPolicies: 'Enabled'
      }
    }
  }
}

Private DNS: Deploy Private DNS with the DNS name privatelink.database.windows.net in PrivateDns2.bicep.

var pdns_name_sql = 'privatelink${environment().suffixes.sqlServerHostname}'

resource PrivateDnsSql 'Microsoft.Network/privateDnsZones@2020-06-01' = {
  name: pdns_name_sql
  location: 'global'
}

Virtual network link: Link the deployed Private DNS to the virtual network where the SQL Server subnet exists in PrivateDns2.bicep.

VirtualNetwork2Id:existingVnet2.id

resource VnetLinkSql 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2020-06-01' = {
  name: '${PrivateDnsSql.name}/${PrivateDnsSql.name}-link'
  location: 'global'
  properties: {
    registrationEnabled: false
    virtualNetwork: {
      id: VirtualNetwork2Id
    }
  }
}

Private DNS A record: Create a DNS A record and set the IP address from the deployed private endpoint in PrivateDns2.bicep.

output PrivateEndpointSqlIpAddress string = PrivateEndpointSql.properties.customDnsConfigs[0].ipAddresses[0]

resource PrivateDnsASql 'Microsoft.Network/privateDnsZones/A@2020-06-01' = {
  name: '${PrivateDnsSql.name}/${SqlServerName}'
  properties: {
    ttl: 3600
    aRecords: [
      {
        ipv4Address: PrivateEndpointSqlIpAddress
      }
    ]
  }
}

Access to SQL Database

App Service and Functions need to access the SQL Database to insert and extract the data records. Both can access through V-net integration and Private Endpoint. In SqlDatabase2.bicep, public network access is disabled so no one can access through the public IP.

resource SqlServer 'Microsoft.Sql/servers@2021-02-01-preview' = {
  properties: {
    publicNetworkAccess: 'Disabled'
  }
}

SQL project update

The biggest problem is when you want to update SQL projects including data schema on the Azure SQL Database. In this sample template, you cannot update the SQL project because a Linux based self-hosted agent does not support build and deployment of SQL Database projects. On the other hand, a Windows based self-hosted agent in the docker container on Azure Container Instance cannot be deployed on the virtual network as I mentioned in the previous article Virtual Network architecture 2 - Deployment pipelines. Possible workaround is to create a Windows based self-hosted agent on a Virtual Machine, to change IP filtering rules when in code deployment, or to wait for the Windows docker container supporting the virtual network deployment.

Virtual Network architecture 1 - Do I need virtual network?

Kohei Kawata — Mon, 29 Aug 2022 14:38:00 +0000

Summary

This is a part of virtual network architecture series. In this article, I discuss problems with software development for cloud native architectures and pros and cons of virtual network architecture.

Background

As many of enterprise cloud users are moving on to cloud native technologies, so that they can meet continuously changing requirements and keep up with the velocity of fast application development. I often see those cloud users wondering where their security responsibilities lie. According to shared responsibility model, IaaS systems require fully to take care of secure infrastructures (including identity and directory infrastructure, applications, network controls, and operating system), while PaaS systems allow to offload some of those security responsibilities to the cloud provider. So I got a lof of questions from enterprise clients about in what areas they have to take security responsibilities and how to build the cloud native architecture strongly enough against potential attacks coming through the Internet. Zero trust approach is one of the important security concepts as PaaS resources are usually exposed over public endpoints by default and the traditional firewall-based security does not work. However, even with authentication and authorization mechanisms, the system is still threatened by network attacks and unauthorized traffic from the Internet.

In order to achieve both dynamic modern software architecture and enterprise production-ready security, Microsoft Azure offers Azure Private Link technology that enables access to Azure PaaS Services over a private endpoint located in a virtual network. Most of Azure PaaS Services such as Azure App Service have independent public endpoints while Azure IaaS Services such as Azure Virtual Machines are deployed on a virtual network. Private Link adds a private endpoint to Azure PaaS Services and enables access only from the virtual network through the private endpoint. What I want to emphasize here is that PaaS Services such as SQL Database is an independent entity on the internet and still has its public IP address to which someone on the internet can connect. Data access should be through Private Endpoint and private IP address, and the public IP is protected by IP filtering.

Problems

There would always be pros and cons when you adopt a new technology. Designing and developing a cloud native architecture with Azure Private Link costs you more. The pricing itself is usually not a big problem as one private endpoint costs around $2.4 per day. Instead, in reality the costs on developers become high.

1. Virtual network design and development workload

Compared with architecture without virtual network, it requires more workload to design and develop. For example, in my past work with an enterprise cloud developers I designed the network architecture with four virutal networks, eleven subnets, eight network security group (NSG) configurations, two virtual network peerings, five private DNS zones, five private endpoints, and one service endpoint. And then I designed and implemented a Infrastructure as Code (IaC) piepeline so the enterprise clients can automate those virtual network infrastructure deployment and manage resources by code. It actually took two months by our team.

2. Local development

With Azure Private Link technology, the cloud user can protect their Azure PaaS Services agaist any attacks and unauthorized access from the Internet, but at the same time the enterprise develpers cannot access those services either. For instance, without the virtual network protection the developers can test their code on their own local machines reading and writing data on a Azure SQL Database. When the virtual network envirnment limits access only from private endpoints, the developers' local machines cannot access the database on Azure. That would happen to other PaaS services (such as Azure App Service, Azure Key Vault, and Azure Service Bus).

3. DevOps agent access

In my work, I usually use Azure Pipelines agents (in Azure DevOps) to build, test, and deploy code to Azure PaaS Services. For example, I deploy a .NET application to Azure App Service and Azure Functions, OpenAPI specifications to Azure API Management, and data-tier applications (DACPAC) to Azure SQL Database. If the virtual network protects access from the Internet, even the Azure Pipelines agent cannot access the PaaS services for deployment.

How to deal with those problems

How we handle those problems depends on project contexts. Some enterprise companies have rigid company network restrictions, some have strong network design skills and experiences. It depends on the projects and development environments, but I would like to share how we usually deal with those network access problems.

1. Virtual network design and development workload

Our team is proud of contributing to open source software assets and Microsoft platform that are broadly available. In every project, we create reusable and sharable software assets that can be widely applicable with the agreement of the enterprise clients. Our team practices growth mindset by trying new things and learning from others, and then reuse the learnings and create shared software assets. One examle is Azure-Samples/MLOpsManufacturing created with learnings from multiple projects. As we work more engagements with more clients, more and more other developers can reuse the assets and do not need to spend months designing network security architectures.

2. Local development

One workaround for local development problems is to create similar environments in your local machine. For example, you can create a database on a local SQL Server with the same configurations as the Azure SQL Database and set up your applications, so that developers can switch between local and Azure databases by changing a database connection string. Another workaround is to have multiple environments on Azure. For instance, you can create dev/stg/prod environments. And you can have the dev environment without virtual networks, but the stg/prod environments with virtual networks and private endpoints, so developers can easily test PaaS resources in the dev environment, but make sure of strong protection for the stage and prod environments. The other workaround is to use Azure Bastion with virtual machines where developers can work on codes inside the virtual network. This workaround still has a bit problem because developers usually prefer to work with their own machine they are used to.

3. DevOps agent access

I see two options to solve Azure Pipelines agent access problems in the previous projects. The first option is to manage a Self-hosted agent where you build, test, and deploy your code to Azure PaaS resources in Azure Pipelines. This self-hosted agenet is located inside the virtual network, so it can access all of customers' Azure PaaS resources. You can set up your agent on Linux, macOS, Windows, and also on a Docker container. The downside of this options is you need to manage a IaaS virtual machine in your PaaS architecture, which causes you to maintain the compute infrastructure including update of OS, middleware, dependencies and tools. The other options is to create and remove firewall rules only when Azure Pipelines agent accesses to Azure PaaS resouces. For example, Azure SQL Database Deployment task has IpDetectionMethod option that adds agent IP address range for allowed IP Address rules and DeleteFirewallRule option that deletes the allowed IP address rule after the deployment. (Important notice is that this does not work for Deny public network access to the Azure SQL Server.)

Virtual Network architecture 2 - Deployment pipelines

Kohei Kawata — Mon, 29 Aug 2022 14:38:00 +0000

Summary

This article is Part.2 of virtual network architecture series. I will share the details of the entire pipeline in the sample template api-management-vnet.

Overview

The sample template api-management-vnet includes two scenario - 1) Base architecture without virtual network, 2) Virtual network architecture. Both architectures use the same Azure services below.

Used in both architectures

Azure Pipelines: Deploy infrastructure, build and deploy codes, build and deploy self-hosted agent, execute integration tests by sending REST API requests.
Azure API Management: Platform to manage APIs behind which are running on Azure App Service in this template.
Azure App Service: Hosting service of web APIs.
Azure Functions: Serverless hosting service that runs codes.
Azure SQL: SQL Server on the Azure cloud.
Azure Key Vault: Secret management service that stores keys and certificates accessible from other Azure services.
Azure Service Bus: Message broker in the Azure cloud with message queue and pub-sub topics.

Only in virtual network architecture

Azure Application Gateway: A Web traffic load balancer that works as a gateway of virtual network architecture in this template.

Architecture

Both architectures have the same Web API, Functions, Database, and Service Bus topic. Azure Pipelines conducts the same integration tests that sends API requests to the Web API and Service Bus topic. There are two important differences in the virtual network architecture from the base architecture.

Application Gateway resides in front of the API Management. API Management and App Service cannot be accessed from the internet but only through the Application Gateway. So, the API endpoint is different.
Self-hosted agent is built in the first pipeline and deployed inside the virtual network subnet. The second pipeline runs on this self-hosted agent so it can access the private endpoints of App Service and Service Bus.

Base architecture

Virtual network architecture

Pipelines

You can run the two pipelines by following getting-started documentation. In this article, I am going to explain the details of pipeline steps.

Two pipelines overview

Base architecture has only pipeline-base.yml but Virtual network architecture includes pipeline-vnet1.yml and pipeline-vnet2.yml. The first pipeline of virtual network architecture deploys Azure resources without private endtpoints and builds a self-hosted agent, and the second pipeline runs on the self-hosted agent and deploys and sets up private points for resources. In the end of the second pipeline, it executes integration tests inside virtual network.

pipeline-base.yml

Step 1. In keyvault-secret.yml, the pipeline creates Key Vault if it does not exist and pass secrets input from Azure Pipelines variables so you do not need to keep those secrets in the Azure Pipeline variables but after the second time the pipeline automatically detects pipeline variable and download secrets from existing Key Vault. This technique is described in Azure Pipelines secret management with Key Vault.

Step 2. azure-resource.yml deploys Azure resources with main.bicep.

You need to specify the secret variables handled in the previous Key Vault step. If you have more secret variables for improvement, you need to add variables here.

- job: AzureResourceDeployment
    variables:
    - template: ./variables.yml
    - name: SqlPass
      value: $[ stageDependencies.KeyVault.SQL_PASS.outputs['SQL_PASS.SQL_PASS'] ]
    - name: AadSecretClient
      value: $[ stageDependencies.KeyVault.AAD_SECRET_CLIENT.outputs['AAD_SECRET_CLIENT.AAD_SECRET_CLIENT'] ]

Azure API Management takes around one hour to deploy. In order to avoid a time out error, it specifies timeout: 0. However, I found errors sometimes after one hour. If you face a timeout error, you can just re-run the pipeline, and you will see everything goes smoothly after that.

- job: AzureResourceDeployment
    timeoutInMinutes: 0

Step 3. sql.yml deploys SQL Database schema with .dacpac file built by WeatherDB SQL project.

Step 4. func.yml deploys the .NET C# code of ServicebusListener to Azure Functions.

Step 5. webapi.yml deploys the .NET C# code of WeatherAPI project to Azure App Service. Furthermore, it defines the API in the API Management and generates swagger.json and deploys it to the API Management.

- task: DotNetCoreCLI@2
  displayName: dotnet new tool-manifest
  inputs:
    command: custom
    custom: new
    arguments: tool-manifest
    workingDirectory: ${{ parameters.apiProjectDirectory }}
- task: DotNetCoreCLI@2
  displayName: dotnet tool install
  inputs:
    command: custom
    custom: tool
    arguments: install Swashbuckle.AspNetCore.Cli --version $(SWASHBUCKLE_VERSION)
    workingDirectory: ${{ parameters.apiProjectDirectory }}
- task: DotNetCoreCLI@2
  displayName: dotnet swagger tofile
  inputs:
    command: custom
    custom: swagger
    arguments: tofile --output ${{ parameters.swaggerPath }} ${{ parameters.apiDllPath }} $(SWAGGER_VERSION)
    workingDirectory: ${{ parameters.apiProjectDirectory }}
- task: AzureCLI@2
  displayName: Deploy API to API Management
  inputs:
    azureSubscription: ${{ parameters.azureSvcName }}
    scriptType: bash
    scriptLocation: inlineScript
    inlineScript: |
      az apim api import -g $(RESOURCE_GROUP_NAME) \
        --service-name $(API_MANAGEMENT_NAME) \
        --api-id $(API_NAME) \
        --path /$(API_NAME) \
        --specification-format OpenApiJson \
        --specification-path ${{ parameters.swaggerPath }} \
        --service-url $(API_MANAGEMENT_SERVICE_URL)

Step 6. integration-test-powershell.yml executes integration tests for deployed Web API and Functions.

With the powershell script below, it extracts OAuth2.0 token from Azure Active Directory so it can attach the token later when sending a REST request to the API Management. Because the Azure Pipelines agent sends a request with automation, it follows Client Credential Flow of OAuth2.0. This OAuth2.0 flow is explained in Azure AD app configuration for Web API.

$clientSecret = Get-AzKeyVaultSecret -VaultName $env:KEYVAULT_NAME -Name $env:KVSECRET_NAME_AADCLIENT -AsPlainText
$authorizeUri = "https://login.microsoftonline.com/${{ parameters.aadTenantId }}/oauth2/v2.0/token"
$body = 'grant_type=client_credentials' + `
'&client_id=${{ parameters.aadAppidClient }}' + `
'&client_secret=' + $clientSecret + `
'&scope=api://${{ parameters.aadAppidBackend }}/.default'
$token = (Invoke-RestMethod -Method Post -Uri $authorizeUri -Body $body).access_token

The base architecture does not have Application Gateway and the endpoint is at the API Management. The route /Weatherforecast is to GET and POST the data to Azure SQL Databse and /Weatherforecast/message to send a message to Service Bus topic, which triggers Azure Functions and inserts that record in SQL Database.

parameters:
  url: https://$(API_MANAGEMENT_NAME).azure-api.net

$Uri = "${{ parameters.url }}/$env:API_NAME/Weatherforecast"
$UriMessage = "${{ parameters.url }}/$env:API_NAME/Weatherforecast/message"

pipeline-vnet1.yml

pipeline-vnet1.yml includes steps below.

Store secrets from Pipeline variables to Key Vault
Generate SSL certificate for Application Gateway backend pool
Deploy Azure services
Build a self-hosted agent
Deploy projects to App Service, Functions, and SQL Database
Execute integration tests

After this pipeline is executed, some resources such as App Service, Key Vault, SQL Database, Functions, and Service Bus are not protected within the virtual network. Only Application Gateway and API Management are connected to and located in the virtual network. The self-hosted agent is built and deployed to Azure Container Instances and ready for usage.

Step 1. cert-appgateway.yml generates and sets in Key Vault a SSL certificate for the backend pool of API Management. The Key Vault requires to grant the access right for Azure Pipeliens Microsoft hosted agent.

Set-AzKeyVaultAccessPolicy -VaultName $env:KEYVAULT_NAME -PermissionsToCertificates get,create -ObjectId ${{ parameters.aadObjectidSvc }}

Step 2. keyvault-secret.yml is the same template as used in the Base architecture. This time it adds AZP_TOKEN secret variable that is needed to deploy a self-hosted agent. How to get the personal access token is described in Getting-started.

Step 3. azure-resource.yml deploys Azure resources with main1.bicep. All the virtual networks and subnets are deployed this time with VirtualNetwork.bicep. All the virtual network IP address ranges are defined in main.parameters.json.

Step 4. restart-appgateway.yml restarts Application Gateway. I faced errors without restarting Application Gateway. According to the Microsoft documentation Updates to the DNS entries of the backend pool, when you change DNS entries for the backend pool, Application Gateway must be restarted. In the Azure resource deployment with PrivateDns1.bicep, it creates the DNS A record for the API Management. Then you need to restart the Application Gateway.

resource PrivateDnsAApim 'Microsoft.Network/privateDnsZones/A@2020-06-01' = {
  name: '${PrivateDnsApim.name}/${ApiManagementName}'
  properties: {
    ttl: 3600
    aRecords: [
      {
        ipv4Address: ApiManagementPrivateIPAddress
      }
    ]
  }
}

Step 5. self-hosted-agent.yml builds and deploys a Linux based self-hosted agent.

As of August 2022, you can run a Windows based self-hosted agent in the docker container but Windows based Azure Container Instance cannot be deployed on the virtual network, according to the Microsoft documentation Deploy container instances into an Azure virtual network. Windows based self-hosted agent is needed to deploy SQL Database projects, but for now the Windows self-hosted agent build part is commented out.
The self-hosted agent container application is defined in Dockerfile.
After the container image is pushed into Azure Container Registry, the deployment to Azure Container Instance is implemented in linux-agent.bicep.

Step 6. sql.yml is the same as used in the Base pipeline and deploys SQL Database schema with .dacpac file built by WeatherDB SQL project.

Step 7. func.yml is the same as used in the Base pipeline and deploys the .NET C# code of ServicebusListener to Azure Functions.

Step 8. webapi.yml is the same as used in the Base pipeline and deploys the .NET C# code of WeatherAPI project to Azure App Service and defines the API in the API Management and generates swagger.json and deploys it to the API Management.

Step 9. integration-test-powershell.yml is the same as used in the Base pipeline and executes integration tests for deployed Web API and Functions.

pipeline-vnet2.yml

pipeline-vnet2.yml includes steps below.

Execute integration tests before setting private endpoints
Deploy and set private endpoints
Execute integration tests for private endpoints
Deploy projects to App Service and Functions with private endpoints by the self-hosted agent
Execute integration tests for code deployment through private endpoints.

This pipeline deploys and set private endpoints so all Azure services are protected by the virtual network. You cannot deploy new software to the App Service and Functions any more as they are not allowed to access through the Internet. The self-hosted agent running on Azure Container Instance is located inside the virtual network and capable of building and deploying the new software.

You can see this pipeline use the self-hosted agent by specifying the pool like below. All steps below can be executed on a Linux based agent.

pool:
  name: Default

Step 1. integration-test-bash.yml executes integration tests before private endpoints are deployed and set up. The self-hosted agent is Linux based, and this integration test is written in bash scripts. Obtaining Azure Active Directory OAuth2.0 token by bash script is written in the way below.

clientSecret=$(az keyvault secret show --vault-name $(KEYVAULT_NAME) --name $(KVSECRET_NAME_AADCLIENT) --query value -o tsv)
authorizeUri="https://login.microsoftonline.com/${{ parameters.aadTenantId }}/oauth2/v2.0/token"
token=$(curl -X POST $authorizeUri -d "client_id=${{ parameters.aadAppidClient }}&grant_type=client_credentials&scope=api://${{ parameters.aadAppidBackend }}/.default&client_secret=$clientSecret" | jq ".access_token" | tr -d \")

Step 2. private-endpoints.yml deploys Azure resources related to private endpoints with main2.bicep. All the virtual network IP address ranges are defined in main.parameters.json.

Step 3. restart-appgateway.yml restarts Application Gateway. This should be executed for the same reason as the V-net pipeline 1 because private DNS A records for Key Vault, App Service, SQL Database, Funtions, and Service Bus are added.

Step 4. integration-test-bash.yml executes integration tests after private endpoints are set. This step can make sure all the configurations of private endpoints are correctly deployed.

Step 5. func.yml is the same as used in the Base pipeline but this time is executed on the self-hosted agent. Functions do not have accessible public IP any more, and it should be connected through the private IP inside the virtual network.

Step 6. webapi.yml is the same as used in the Base pipeline but this time is executed on the self-hosted agent. The App Service does not have accessible public IP any more, and it should be connected through the private IP inside the virtual network.

Step 7. integration-test-bash.yml executes integration tests to make sure the code deployment to App Service and Funtions are correctly done through the self-hosted agent.

Virtual Network architecture 3 - Key Vault Private Endpoint

Kohei Kawata — Mon, 29 Aug 2022 14:38:00 +0000

Summary

This article is Part.3 of virtual network architecture series. I will share the details of the private endpoint with Key Vault in api-management-vnet.

Private Endpoint configuration

Private Endpoint: Deploy the private endpoint and connect it to Key Vault and Key Vault's subnet in PrivateEndpoint.bicep.

KeyVaultId:existingKv.id
VirtualNetwork2SubnetIdKv:existingVnet2.properties.subnets[3].id

properties: {
  privateLinkServiceConnections: [
    {
      name: pe_name_kv
      properties: {
        privateLinkServiceId: KeyVaultId
        groupIds: [
          'vault'
        ]
      }
    }
  ]
  subnet: {
    id: VirtualNetwork2SubnetIdKv
    properties: {
      privateEndpointNetworkPolicies: 'Enabled'
    }
  }
}

Private DNS: Deploy Private DNS with the DNS name privatelink.vaultcore.azure.net in PrivateDns2.bicep.

var pdns_name_kv = 'privatelink.vaultcore.azure.net'

resource PrivateDnsKv 'Microsoft.Network/privateDnsZones@2020-06-01' = {
  name: pdns_name_kv
  location: 'global'
}

Virtual network link: Link the deployed Private DNS to the virtual network where the Key Vault subnet exists in PrivateDns2.bicep.

VirtualNetwork2Id:existingVnet2.id

resource VnetLinkKv 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2020-06-01' = {
  name: '${PrivateDnsKv.name}/${PrivateDnsKv.name}-linkc'
  location: 'global'
  properties: {
    registrationEnabled: false
    virtualNetwork: {
      id: VirtualNetwork2Id
    }
  }
}

Private DNS A record: Create a DNS A record and set the IP address from the deployed private endpoint in PrivateDns2.bicep.

output PrivateEndpointKvIpAddress string = PrivateEndpointKv.properties.customDnsConfigs[0].ipAddresses[0]

resource PrivateDnsAKv 'Microsoft.Network/privateDnsZones/A@2020-06-01' = {
  name: '${PrivateDnsKv.name}/${KeyVaultName}'
  properties: {
    ttl: 3600
    aRecords: [
      {
        ipv4Address: PrivateEndpointKvIpAddress
      }
    ]
  }
}

Access to Key Vault

App Service extracts credentials from the Key Vault with Key Vault reference through the Private Endpoint. The outbound from App Service is through the v-net integration. The Key Vault reference is defined in AppService.bicep.

resource AppServiceConfig 'Microsoft.Web/sites/config@2021-02-01' = {
  properties: {
    'Sql:ConnectionString': '@Microsoft.KeyVault(VaultName=${kv_name};SecretName=${kvsecret_name_sqlcs})'
    'Servicebus:ConnectionString': '@Microsoft.KeyVault(VaultName=${kv_name};SecretName=${kvsecret_name_sbcs})'
  }
}

Functions extracts credentials from the Key Vault in the same way as the App Service.The Key Vault reference is defined in FunctionApp.bicep.

resource FuncConfig 'Microsoft.Web/sites/config@2021-02-01' = {
  properties: {
    'ServiceBusConnectionString': '@Microsoft.KeyVault(VaultName=${kv_name};SecretName=${kvsecret_name_sbcs})'
    'SqlConnectionString': '@Microsoft.KeyVault(VaultName=${kv_name};SecretName=${kvsecret_name_sqlcs})'
  }
}

Container Instance running the self-hosted agent extracts Azure Active Directory client app secret from the Key Vault to get the Azure Active Directory OAuth2.0 token for the integration test in integration-test-bash.yml.

clientSecret=$(az keyvault secret show --vault-name $(KEYVAULT_NAME) --name $(KVSECRET_NAME_AADCLIENT) --query value -o tsv)

Application Gateway implements TLS termination for the backend pool, which means the API Management, with the certificate stored in the Key Vault. Application Gateway is not capable of accessing Key Vault through Private Endpoint but through Service Endpoint.

var agw_kv_secret_id = 'https://${kv_name}${environment().suffixes.keyvaultDns}/secrets/${kvcert_name_agw}'

sslCertificates: [
  {
    name: agw_ssl_certificate_name
    properties: {
      keyVaultSecretId: agw_kv_secret_id
    }
  }
]

httpListeners: [
  {
    properties: {
      sslCertificate: {
        id: resourceId('Microsoft.Network/applicationGateways/sslCertificates', agw_name, agw_ssl_certificate_name)
      }
    }
  }
]

Application Gateway SSL certificate

The problem is that Application Gateway cannot access Key Vault through Private Endpoint to extract SSL certificate in the configuration above, as of August 2022. This is discussed in the forum Application Gateway: Integration with Key Vault does not work #33157. In sum, Application Gateway has to access Key Vault's public IP through Service Endpoint, not through private IP.

Service Endpoint: To enable Service Endpoint for this case, you have to add the configuration when deploying the virtual network subnet for the Key Vault. You can see VirtualNetwork.bicep and find the configuration below. The Service Endpoint configuration should be implemented on the virtual network subnet where the service that wants to access Key Vault, which means Application Gateway subnet.

resource VirtualNetwork1 'Microsoft.Network/virtualNetworks@2021-03-01' = {
  properties: {
    subnets: [
      {
        name: snet_name_1_agw
        properties: {
          addressPrefix: snet_prefix_1_agw
          networkSecurityGroup: {
            id: Nsg1AgwId
          }
          serviceEndpoints: [
            {
              service: 'Microsoft.KeyVault'
              locations: [
                '*'
              ]
            }
          ]
        }
      }
    ]
  }
}

Public network access should be enabled because Application Gateway reaches out to the Key Vault through the public IP, but you can limit the access only from the Application Gateway subnet in KeyVault2.bicep. bypass: 'AzureServices' enables all Azure services to access the Key Vault through its public IP, virtualNetworkRules: [{id: VirtualNetwork1SubnetIdAgw}] allows access from Application Gateway subnet, and defaultAction: 'Deny' shut down all the access except Azure services and the Application Gateway virtual network subnet. All the access other than Application Gateway SSL certificate are through the Private Endpoint.

resource KeyVault 'Microsoft.KeyVault/vaults@2021-10-01' = {
  properties: {
    publicNetworkAccess: 'Enabled'
    networkAcls: {
      bypass: 'AzureServices'
      defaultAction: 'Deny'
      virtualNetworkRules: [
        {
          id: VirtualNetwork1SubnetIdAgw
        }
      ]
    }
  }
}

Azure IoT Edge Integration Test template - Part.1

Kohei Kawata — Sun, 28 Aug 2022 10:31:00 +0000

Summary

This is a part of article series introducing Azure IoT Edge integration template - azure-iot-edge-integration-test-template. In this Part.1, I will explain the entire pipeline including Infrastructure as Code and integration test. Part.2 would provide the information about each IoT Edge module, and Part.3 share about the IoT Edge manifest generator.

Overview
Architecture
Test steps
Infrastructure deployment
- Network Security Group - inbound Port 22
- VM domain name label
- IoT Hub consumer group
Setup and installation
- VM Service Connection
- Blob Contributor Role
Code deployment and test execution
- edge-module.yml
- test-prep.yml
- test-execution.yml
- test-cleanup.yml

Overview

When you have multiple Azure IoT Edge modules on an edge device and want to update codes of one of those modules, you can make sure the code quality by implementing linter and unit tests, but it is difficult to validate communications among modules. That is why executing integration tests every time you update the software.

In this template, you can execute integration tests of Azure IoT Edge modules on Test Environment on Azure Virtual Machine. The test procedure is all automated by Azure Pipelines. By taking advantage of this template, you deploy and execute integration tests before deploying your code to the edge device.

Architecture

This template includes six IoT Edge sample modules - FileGenerator, FileUpdater, FileUploader, IothubConnector, WeatherObserver, and LocalBlobStorage. Details of IoT Edge modules are explained in Azure IoT Edge Integration Test template - Part.2.

Test steps

You need to execute three steps, 1) Infrastructure deployment, 2) Setup and installation, and 3) Code deployment and test execution. Everything to know to run this template is described in Getting-started, but I am going instruct details and important points for those pipelines.

Infrastructure deployment

Azure resources needed for this template are defined in main.bicep and you can run the IaC(Infrastructure as Code) pipeline of iac.yml on Azure Pipelines.

Network Security Group - inbound Port 22

main.bicep defines a Network Security Group inbound rule that opens Port 22. This is necessary for you to access the VM through SSH when installing Azure IoT Edge in the next step and for Azure Pipelines agent to set up VM's directory and grant directory access permissions in test-prep.yml.

properties: {
  securityRules: [
    {
      name: 'SSH'
      properties: {
        protocol: 'Tcp'
        sourcePortRange: '*'
        destinationPortRange: '22'
        sourceAddressPrefix: '*'
        destinationAddressPrefix: '*'
        access: 'Allow'
        priority: 100
        direction: 'Inbound'
      }
    }
  ]
}

VM domain name label

Setting up the domain name label for the VM is important because IP Address is dynamically allocated and Azure Pipelines agent accesses with VM Service Connection descrited in Create SSH Service Connection.

Host name should be edge-{BASE_NAME}.{LOCATION}.cloudapp.azure.com, which is defined in main.bicep.

var dns_label = 'edge-${base_name}'

resource PublicIp 'Microsoft.Network/publicIPAddresses@2021-05-01' = {
  name: public_ip_name
  location: location
  sku: {
    name: 'Basic'
  }
  properties: {
    publicIPAllocationMethod: 'Dynamic'
    publicIPAddressVersion: 'IPv4'
    dnsSettings: {
      domainNameLabel: dns_label
    }
    idleTimeoutInMinutes: 4
  }
}

IoT Hub consumer group

It is better to create consumer groups of IoT Hub. If you want to consume messages on the IoT Hub through another tool such as Azure IoT Hub Explorer that would cause errors of the integration test. main.bicep deploys the consumer group that dedicates to Azure Pipelines agent executing integration tests.

param iothub_cg_name string
resource IoTHubConsumerGroup 'Microsoft.Devices/IotHubs/eventHubEndpoints/ConsumerGroups@2021-07-02' = {
  name: '${IoTHub.name}/events/${iothub_cg_name}'
  properties:{
    name: iothub_cg_name
  }
}

Setup and installation

VM Service Connection

In this template, it uses SSH service connection. It is possible you use bash commands like ssh testuser@edge-{BASE_NAME}.{LOCATION}.cloudapp.azure.com. By combining retryCountOnTaskFailure of Azure Pipelines task, you can handle unstable connection errors of SSH.

Blob Contributor Role

You need to set up manually Storage Blob Data Contributor of Azure built-in roles of Azure RBAC, which is tied to your Azure Subscription so Azure Pipelines agent can generate SAS token of Blob Storage. You need Azure Subscription Owner or User Access Administrator role.

az role assignment create `
    --role "Storage Blob Data Contributor" `
    --assignee {Object ID of Azure Service Connection} `
    --scope "/subscriptions/{Azure Subscription ID}/resourceGroups/rg-{BASE_NAME}/providers/Microsoft.Storage/storageAccounts/st{BASE_NAME}"

Code deployment and test execution

edge-module.yml

Call this template with each IoT Edge module. This builds and pushes container images to Azure Container Registry.
Use not Azure Pipeline Docker task but docker commands because it does not need to create Docker Registry service connection manually. The process that Azure Pipelines extracts Azure Container Registry key, buil, and push is all automated.

acrkey=$(az acr credential show --name $(ACR_NAME) --query passwords[0].value -o tsv)
cd ${{ parameters.dockerfileDirectory }}
docker login -u $(ACR_NAME) -p $acrkey $(ACR_NAME).azurecr.io
docker build --rm -f Dockerfile -t $(ACR_NAME).azurecr.io/${{ parameters.repositoryName }}:$(Build.BuildNumber) .
docker push $(ACR_NAME).azurecr.io/${{ parameters.repositoryName }}:$(Build.BuildNumber)

test-prep.yml

Call this template with IoT Edge modules as parameters. The parameters are used for iterative tasks that check if module images exist in Azure Container Registry and each module is running on IoT Edge runtime.

- template: ./templates/test-prep.yml
  parameters:
    azureSvcName: $(AZURE_SVC_NAME)
    vmSshSvcName: $(VM_SVC_NAME)
    EdgeImages:
      module1:
        name: IothubConnector
        repository: iothub-connector
        tag: $(Build.BuildNumber)
      module2:
        name: WeatherObserver
        repository: weather-observer
        tag: $(Build.BuildNumber)
      module3:
        name: FileGenerator
        repository: file-generator
        tag: $(Build.BuildNumber)
      module4:
        name: FileUploader
        repository: file-uploader
        tag: $(Build.BuildNumber)
      module5:
        name: FileUpdater
        repository: file-updater
        tag: $(Build.BuildNumber)

SSH task
- Remove /edge directory to refresh the leftover of past test executions.
- Grant read, write, and execute permissions to the host machine directory. Azure IoT Edge Hub with UID 1000 and Azure IoT Edge local blob storage with user ID 11000 and user group ID 11000.
- Host system permissions
- Granting directory access to container user on Linux
- Install the tree command so it can show the host machine directory in the log.
- Restart Azure IoT Edge runtime because you deleted the directory that the current modules implement bind mount. To refresh the connection between modules and directory, you need to restart the runtime, otherwise it would cause errors.

if [ -d "/edge" ]
then
  sudo rm -r /edge
fi

sudo mkdir -p $(FILE_UPLOADER_DIR)
sudo chown -R 1000 $(FILE_UPLOADER_DIR)
sudo chmod -R 700 $(FILE_UPLOADER_DIR)

sudo mkdir -p $(FILE_UPDATER_DIR)
sudo chown -R 1000 $(FILE_UPDATER_DIR)
sudo chmod -R 700 $(FILE_UPDATER_DIR)

sudo mkdir -p $(LOCAL_BLOB_STORAGE_DIR)
sudo chown -R 11000:11000 $(LOCAL_BLOB_STORAGE_DIR)
sudo chmod -R 700 $(LOCAL_BLOB_STORAGE_DIR)

sudo apt-get install tree
tree /edge

sudo iotedge system restart

test-execution.yml

Use acynchronous bash method & so the Azure Pipelines agent sends a direct method request to IoT Hub and at the same time listens to messages on IoT Hub sent from IoT Edge modules. --timeout is currently set 30 sec. Sometimes the response from IoT Edge is slow and they do not respond and cause errors. 30 seconds is probably is good time to wait. If it is longer than 30 seconds, something is going wrong on Edge modules.

az iot hub invoke-module-method --hub-name $(IOTHUB_NAME) --device-id $(IOTHUB_DEVICE_ID) --module-id IothubConnector --method-name request_weather_report --method-payload '{"city": "Tokyo"}' &
testResult=$(az iot hub monitor-events --hub-name $(IOTHUB_NAME) --device-id $(IOTHUB_DEVICE_ID) --module-id IothubConnector --cg $(IOTHUB_CONSUMER_GROUP) --timeout 30 -y)

test-cleanup.yml

Remove all directory but keep the blob container weather

az storage blob directory delete --account-name $(STORAGE_ACCOUNT_NAME) --container-name $(BLOB_CONTAINER_NAME) --directory-path $(TEST_ORGANIZATION_NAME) --recursive

Azure IoT Edge Integration Test template - Part.2

Kohei Kawata — Sun, 28 Aug 2022 10:31:00 +0000

Summary

This article is following Part.1 and sharing important tips for IoT Edge modules in azure-iot-edge-integration-test-template.

Overview

This sample template has six IoT Edge modules and two integration test data flows. One is Azure Pipelines agent sending direct method and receiving a weather report and weather report files uploaded to Azure Blob Storage. Another integration test flow is the direct method request asks to download the archive weather files.

IothubConnector

This module depends both on IoTHubModuleClient and rclypy(ROS Client Library for Python). You cannot run this module locally without ROS2 environment setup or iotedgehubdev.
There some options to keep the application running and waiting for callback. IothubConnector uses the edge signal handler using threading and signal. On the other hand, FileGenerator uses rclpy.spin().

def module_termination_handler(signal, frame):
    print ("IoTHubClient sample stopped")
    stop_event.set()

stop_event = threading.Event()
signal.signal(signal.SIGTERM, module_termination_handler)
try:
    stop_event.wait()
except Exception as e:
    print("Unexpected error %s " % e)
    raise
finally:
    print("Shutting down client")
    module_client.shutdown()

Direct method callback from IoT Hub is implemented with method_request_handler(). On receiving a direct method request, it sends a request message to WeatherReporter through Azure Iot Edge route communication.

async def method_request_handler(method_request):
    print('Direct Method: ', method_request.name, method_request.payload)
    if method_request.name in request_method_list:
        response_payload, response_status = await request_method_list[method_request.name](method_request.payload, module_client)
    else:
        response_payload = {"Response": "Direct method {} not defined".format(method_request.name)}
        response_status = 404

    method_response = MethodResponse.create_from_method_request(method_request, response_status, response_payload)
    await module_client.send_method_response(method_response)

module_client.on_method_request_received = method_request_handler

async def request_weather_report(payload: dict, module_client: IoTHubModuleClient):
    try:
        json_string = json.dumps(payload)
        message = Message(json_string)
        await module_client.send_message_to_output(message, 'reportRequest')
        return {"Response": "Send weather report request for {}".format(payload['city'])}, 200
    except Exception as e:
        print(e)
        return {"Response": "Invalid parameter"}, 400

Azure IoT Edge route communication callback is implemented with receive_message_handler(). It receives messages from WeatherReport module and sends requests to FileGenerator module through ROS2 topic communication.

async def receive_message_handler(message_received):
    if message_received.input_name == 'reportResponse':
        message_telemetry = Message(
            data=message_received.data.decode('utf-8'),
            content_encoding='utf-8',
            content_type='application/json',
        )
        await module_client.send_message_to_output(message_telemetry, 'telemetry')
        print("Weather report sent to IoT Hub")

        ros2_publisher_client.ros2_publisher(message_received.data.decode('utf-8'))
        print("ROS2 message sent to FileGenerator")

    elif  message_received.input_name == 'updateResponse':
        message_telemetry = Message(
            data=message_received.data.decode('utf-8'),
            content_encoding='utf-8',
            content_type='application/json',
        )
        await module_client.send_message_to_output(message_telemetry, 'telemetry')
        print("Download status sent to IoT Hub")
    else:
        print("Message received on unknown input: {}".format(message_received.input_name))

module_client.on_message_received = receive_message_handler

WeatherObserver

WeatherObserver is written with .NET6 console app in C#. This is simple and independent from Azure IoT Edge or ROS2 and you can run it locally for debugging purpose.
The callback that processes requests from IothubConnector is implemented with ioTHubModuleClient.SetInputMessageHandlerAsync().

await ioTHubModuleClient.SetInputMessageHandlerAsync(routeC2W, ParseRequests, ioTHubModuleClient).ConfigureAwait(false);

FileGenerator

FileGenerator is a module written in Python that receives a request from IothubConnector and generates weather report files in the edge host machine.
This module uses rclpy.spin() to keep the application itself running and waiting for the requests.

rclpy.init(args=args)
file_generator = FileGenerator()
rclpy.spin(file_generator)
file_generator.destroy_node()
rclpy.shutdown()

The callback for ROS2 requests is implemented by create_subscription() of ROS2 topic subscription.

def __init__(self):
    super().__init__('file_generator')
    self.subscription = self.create_subscription(
        String,
        os.getenv('ROS_TOPIC_NAME'),
        self.listener_callback,
        10)
    self.subscription

def listener_callback(self, msg):

FileUploader

FileUploader is a module written in .NET6 C# that extracts files generated by FileGenerator and sends them to LocalBlobStorage.

FileUpdater

FileUpdater modules is writtein in .NET6 C# and downloads archive files from Azure Blob Storage.
It uses SAS token that allows the module only to access a specific directory so this system can work for multi-tenant and multi edge device environments. This SAS token is generated by ManifestGenerator which is explained in Part.3.

LocalBlobStorage

This sample template uses LocalBlobStorage that is built by Microsoft and used from mcr.microsoft.com/azure-blob-storage:latest. The reason for using the built-in local blob is that Azure IoT Edge runtime and default modules including EdgeHub and EdgeAgent do not support file uploading features.

Azure IoT Edge Integration Test template - Part.3

Kohei Kawata — Sun, 28 Aug 2022 10:31:00 +0000

Summary

This article is a part of series of article about azure-iot-edge-integration-test-template. Following Part.2, I am going to focus on ManifestGenerator in this article.

Deployment manifest

In this sample template, the .NET app ManifestGenerator running on Azure Pipelines agent generates Azure IoT Edge Deployment Manifest and deploys it to Azure IoT Hub. IoT Edge modules connected to the IoT Hub are going to pull container images pointed on the manifest.

What ManifestGenerator does

The main purpose of ManifestGenerator app is to generate manifest.json. It includes the information of all IoT Edge modules including system default and custom.

Default module: edgeAgent and edgeHub are default modules that manage modules and their communications.
Custom module: You can specify what modules you want to deploy to the IoT Edge environment.
IoT Edge communication route: You have to specify route name here. This is very important that those names are synchronized with routes defined in each module apps. For example, the route specified in the manifest is like FROM /messages/modules/IothubConnector/outputs/reportRequest INTO BrokeredEndpoint("/modules/WeatherObserver/inputs/reportRequest"). IothubConnector module needs to specify reportRequest route name on its send message method, and WeatherObserver to specify reportRequest route name on its callback method.
Image URL, Environment variables, Bind mount directory: You have to specify image URL, environment variables, bind mount directory, and other configurations like below.

You can see a sample here - manifest.example.json

"FileGenerator": {
  "version": "1.0",
  "type": "docker",
  "status": "running",
  "restartPolicy": "always",
  "settings": {
    "image": "crsample1.azurecr.io/file-generator:20220818.2",
    "createOptions": "{\"HostConfig\":{\"Binds\":[\"/edge/upload/reports:/genroot\"]}}"
  },
  "env": {
    "OUTPUT_DIRECTORY_PATH": {
      "value": "/genroot"
    },
    "ROS_TOPIC_NAME": {
      "value": "ros2_topic_download"
    }
  }
},

Configurations

In this section, I am going to describe how to set configurations for manifest.json.

Extract credentials

manifest.json needs some credentials including IoT Hub connection string, Container Registry key, Storage Account key, local blob storage key. The first three can be extracted through Azure CLI command as Azure Pipelines agent has the access right with Azure Service Connection. For the local blob storage key, it could be anything of base64 string with length in 16 bytes. The Azure Pipeline agent generates the one randomly and sets it as environment variables.

az extension add --name azure-iot
iothubcs=$(az iot hub connection-string show --hub-name $(IOTHUB_NAME) -o tsv)
echo "##vso[task.setvariable variable=iotHubCs]$iothubcs"
acrkey=$(az acr credential show --name $(ACR_NAME) --query passwords[0].value -o tsv)
echo "##vso[task.setvariable variable=AcrKey]$acrkey"
storagekey=$(az storage account keys list --resource-group $(RESOURCE_GROUP_NAME) --account-name $(STORAGE_ACCOUNT_NAME) --query [0].value -o tsv)
echo "##vso[task.setvariable variable=storageAccountKey]$storagekey"
localstoragekey=$(openssl rand -base64 16)
echo "##vso[task.setvariable variable=LocalStorageKey]$localstoragekey"

Environment variables

With Azure Pipelines .NET Core CLI task, you can pass neccesary variables as .NET environment variables to ManifestGenerator app when running it. Those variables include credentials Azure Pipelines agent generated in the previous step.

- task: DotNetCoreCLI@2
  displayName: Generate/deploy IoT Edge manifest
  inputs:
    command: run
    projects: $(Build.SourcesDirectory)/src/apps/ManifestGenerator/ManifestGenerator/ManifestGenerator.csproj
    arguments: --configuration Release
  env:
    STORAGE_ACCOUNT_NAME: $(STORAGE_ACCOUNT_NAME)
    STORAGE_ACCOUNT_KEY: $(storageAccountKey)
    ACR_NAME: $(ACR_NAME)
    ACR_PASS: $(AcrKey)
    IOTHUB_CONNECTOR_IMAGE: $(ACR_NAME).azurecr.io/${{ parameters.EdgeImages.module1.repository }}:${{ parameters.EdgeImages.module1.tag }}
    WEATHER_OBSERVER_IMAGE: $(ACR_NAME).azurecr.io/${{ parameters.EdgeImages.module2.repository }}:${{ parameters.EdgeImages.module2.tag }}
    FILE_GENERATOR_IMAGE: $(ACR_NAME).azurecr.io/${{ parameters.EdgeImages.module3.repository }}:${{ parameters.EdgeImages.module3.tag }}
    FILE_UPLOADER_IMAGE: $(ACR_NAME).azurecr.io/${{ parameters.EdgeImages.module4.repository }}:${{ parameters.EdgeImages.module4.tag }}
    FILE_UPDATER_IMAGE: $(ACR_NAME).azurecr.io/${{ parameters.EdgeImages.module5.repository }}:${{ parameters.EdgeImages.module5.tag }}
    IOTHUB_DEVICE_ID: $(IOTHUB_DEVICE_ID)
    IOTHUB_CONNECTION_STRING: $(iotHubCs)
    LOCAL_STORAGE_KEY: $(LocalStorageKey)
    ORGANIZATION_NAME: $(TEST_ORGANIZATION_NAME)

appsettings.json

ManifestGenerator app retains appsettings.json that specifies design values. You do not need to change it unless you change the design such as bind mount paths or SAS token expiration period.

{
  "RouteTelemetry": "telemetry",
  "RouteReportRequest": "reportRequest",
  "RouteReportResponse": "reportResponse",
  "RouteUpdateRequest": "updateRequest",
  "RouteUpdateResponse": "updateResponse",
  "Ros2Topic": "ros2_topic_download",
  "FileGeneratorContainerBind": "/edge/upload/reports:/genroot",
  "FileUploaderContainerBind": "/edge/upload:/uploadroot",
  "FileUpdaterContainerBind": "/edge/download:/downloadroot",
  "LocalBlobStorageBind": "/edge/localblob:/blobroot",
  "CloudBlobContainerName": "weather",
  "LocalBlobContainerName": "weather",
  "LocalBlobAccountName": "stlocal",
  "LocalBlobEndpoint": "http://LocalBlobStorage:11002",
  "FileGeneratorWorkdir": "/genroot",
  "FileUploaderWorkdir": "/uploadroot",
  "FileUpdaterWorkdir": "/downloadroot",
  "SasExpirationMonths": 6
}

IoTEdgeObjectModel NuGet package

This sample template uses IoTEdgeObjectModel NuGet package. This package helps you reduce lines of code you write for manifest.json. You do not need to specify system module configuration or other default configurations. In my experience in the last project, I wrote a manifest.json all by myself from scratch with C# dictionary instances. By using IoTEdgeObjectModel package, you can reduce roughly 50% of your codes.

The three main classes of this package are EdgeAgentDesiredProperties, EdgeHubDesiredProperties, ModuleSpecificationDesiredProperties.

You specifies edgeAgent properties like below with EdgeModuleSpecification class.

EdgeAgentDesiredProperties edgeAgentDesiredProperties = new ()
{
    SystemModuleVersion = "1.3",
    RegistryCredentials = new List<RegistryCredential>()
    {
        new RegistryCredential(acrName, $"{acrName}.azurecr.io", acrName, acrPass),
    },
    EdgeModuleSpecifications = new List<EdgeModuleSpecification>()
    {
        new EdgeModuleSpecification(name:"IothubConnector", image:iothubConnectorImage, environmentVariables:iothubConnectorEnv),
        new EdgeModuleSpecification(name:"WeatherObserver", image:weatherObserverImage),
        new EdgeModuleSpecification(name:"FileGenerator", image:fileGeneratorImage, createOptions:fileGeneratorCreateOptions, environmentVariables:fileGeneratorEnv),
        new EdgeModuleSpecification(name:"FileUploader", image:fileUploaderImage, createOptions:fileUploaderCreateOptions, environmentVariables:fileUploaderEnv),
        new EdgeModuleSpecification(name:"FileUpdater", image:fileUpdaterImage, createOptions:fileUpdaterCreateOptions, environmentVariables:fileUpdaterEnv),
        new EdgeModuleSpecification(name:"LocalBlobStorage", image:localBlobStorageImage, createOptions:localBlobStorageCreateOptions, environmentVariables:localBlobStorageEnv),
    },
};

EdgeHubDesiredProperties mainly specifies Azure IoT Edge route communication.

EdgeHubDesiredProperties edgeHubConfig = new ()
{
    Routes = new List<Route>()
    {
        new Route("route_telemetry", route_telemetry),
        new Route("route_c2w", route_c2w),
        new Route("route_w2c", route_w2c),
        new Route("route_w2u", route_w2u),
        new Route("route_u2w", route_u2w),
    },
};

ModuleSpecificationDesiredProperties specifies custom modules and their module twin desired properties.

ModuleSpecificationDesiredProperties localBlobStorage = new ()
{
    Name = "LocalBlobStorage",
    DesiredProperties = new Dictionary<string, object>
    {
        ["deviceAutoDeleteProperties"] = new Dictionary<string, object>
        {
            ["deleteOn"] = true,
            ["deleteAfterMinutes"] = 5,
            ["retainWhileUploading"] = true,
        },
        ["deviceToCloudUploadProperties"] = new Dictionary<string, object>
        {
            ["uploadOn"] = true,
            ["uploadOrder"] = "NewestFirst",
            ["deleteAfterUpload"] = true,
            ["cloudStorageConnectionString"] = cloudStorageSasConnectionString,
            ["storageContainersForUpload"] = new Dictionary<string, object>
            {
                [localBlobContainerName] = new Dictionary<string, object>
                {
                    ["target"] = iotHubDeviceId,
                }
            },
        },
    },
};

SAS token

ManifestGenerator has a service class SasService.cs that generates a SAS(Shared Access Signature) token. In this way, you can have one Azure Blob Storage for multiple edge devices from different entities. This SAS token makes edge devices follow the security boundary so edge devices cannot access data of different entities.

It is important to convert a SAS token generated to a connection string. This connection string is set as an environment variable of LocalBlobStorage. For FileUpdater, you can use AzureSasCredential to convert the SAS token into the one readable for BlobClient.

string[] sasContents = weatherFileInfo.BlobSasUrl.Split('?');
AzureSasCredential azureSasCredential = new (sasContents[1]);
Uri blobUri = new (sasContents[0]);
BlobClient blobClient = new (blobUri, azureSasCredential, null);
await blobClient.DownloadToAsync(zipFilePath).ConfigureAwait(false);

However, LocalBlobClient by default needs a blob connection string, not SAS token. So you need to convert the SAS token into a blob connection string in ManfiestGenerator Program.cs

string sasUri = directoryClient.GenerateSasUri(sasBuilder).ToString();
string[] sasContents = sasUri.Split('?');
string sasConnectionString = $"BlobEndpoint=https://{this.dataLakeServiceClient.AccountName}.blob.core.windows.net/{blobContainerName}/{sasDirectory};SharedAccessSignature={sasContents[1]}";

devcontainer for ROS2 project

Kohei Kawata — Fri, 26 Aug 2022 14:42:01 +0000

Summary

In this article, I want to share how to create your local development environment for ROS2 projects. Recent code development like the one with ROS2 framework has become very complex and heavily depending on OSS packages. VS Code devcontainer makes it much easier to create your local development environment.

Overview

devcontainer is one of Visual Studio Code Remote Development features that helps create local develop environment. Personally I love Windows OS and Visual Studio when developing .NET C# applications, but when in development with some other languages like Python, Typescript etc., VS Code devcontainer is really powerful. Here is my personal opinion of whe it is so useful.

Independent of your machine's operating system: According to statistica.com, Windows has around 75 percent of the global market share of operating system in 2022. Combined with WSL(Windows Subsystem for Linux) technology, VS Code devcontainer enables you to run Linux development environment in your Windows machine.
Package verion combinations: In general, you want to divide development environments depending on projects. For example, you can use pyenv, virtualenv, conda or whatever virtual environment you prefer for Python. However, when you want to combine the Python environment with other framework, for example ROS2, it is going to be more complex. VS Code devcontainer solves this problem by creating a just in time docker container for each development environment.
No need to install all dependencies from scratch: To create your just in time development environment, you do not need to install all dependencies from scratch. You can utilize publicly available images from the well-known container registry. For example, I am using ros:foxy-ros-base-focal image for my ROS2 developenvironment - Dockerfile
Debugger: Visual Studio Code supports debuggers for different languages. For this exmaple - devcontainer.json, it uses ms-python VS Code extension with which you can run a Python debugger in your local machine.
VS Code support and extensions: VS Code has built-in support and so many extensions for different languages. You can use those support even in the docker container development environment of devcontainer.

.devcontainer/devcontainer.json

The sample devcontainer.json looks like below.

dockerfile: This points to Dockerfile located in the same folder as devcontainer.json under .devcontainer directory.
workspaceFolder: The directory in the docker container where .devcontainer directory exists.
extensions: VS Code extensions you want to use in the docker contaienr development environment.

{
  "name": "ROS devcontainer",
  "dockerFile": "Dockerfile",
  "workspaceFolder": "/workspaces/EdgeIntegrationTest/src/apps/FileGenerator",
  "settings": {},
  "extensions": [
        "ms-python.python"
    ]
}

.devcontainer/Dockerfile

The sample Dockerfile under .devcontainer directory looks like below.

FROM ros:foxy-ros-base-focal: In this sample, it uses that ros image as the base image. It bases on ros:foxy-ros-core-focal, which bases on ubuntu:focal, which is the same as ubuntu:20.04.
COPY requirements.txt ./: Copy requirements.txt to /app in the docker container so the docker container can run pip install to set up necessary Python dependencies.
RUN echo 'source /opt/ros/foxy/setup.bash' >> /root/.bashrc: According to ROS2 documentation Configuring environment, in order to set up the docker container to access ROS2 commands, you need to source the set up files in the docker container - /opt/ros/foxy/setup.bash. To avoid running the source command every time, it adds the source command to shell startup script.

FROM ros:foxy-ros-base-focal

SHELL ["/bin/bash", "-c"]

WORKDIR /app

COPY requirements.txt ./
RUN apt update && apt install -y \
    python3-pip \
    python3-colcon-common-extensions
RUN pip install -r requirements.txt

RUN echo 'source /opt/ros/foxy/setup.bash' >> /root/.bashrc

The location of setup.bash file in the docker container

/opt
└── ros
    └── foxy
        ├── _local_setup_util.py
        ├── bin
        ├── cmake
        ├── include
        ├── lib
        ├── local_setup.bash
        ├── local_setup.sh
        ├── local_setup.zsh
        ├── opt
        ├── setup.bash
        ├── setup.sh
        ├── setup.zsh
        ├── share
        ├── src
        └── tools

Directory structure

How to run the docker container development environment

The step to open the development environment is like below.

Go to your terminal and change your directory to where .devcontainer folder exists.
Run code . in your terminal, and you will see VS Code is coming up.
Go to Open a Remote Window and select Reopen in Container, and you will see VS Code is reopened with the one running in the docker container.

Where is the project directory?

In this sample - .devcontainer, when the docker container VS Code is open, you will see your directory in the docker container is /workspaces/EdgeIntegrationTest/src/apps/FileGenerator. You opened the devcontainer at FileGenerator in your local machine directory. However, the docker container copied EdgeIntegrationTest directory in your local machine to /workspaces in the docker container.

EdgeIntegrationTest
└── src
    └── apps
        └── FileGenerator
              ├── .devcontainer
              |   ├── Dockerfile
              |   ├── devcontainer.json
              |   └── requirements.txt
              ├── Dockerfile
              ├── main.py
              └── requirements.txt

Is this because your articulate "workspaceFolder": "/workspaces/EdgeIntegrationTest/src/apps/FileGenerator" in devcontainer.json? No. If you change workspaceFolder in devcontainer.json to "workspaceFolder": "/workspaces/FileGenerator", you will see EdgeIntegrationTest directory is copied under /workspaces in the docker container and see the error below.

It seems like the VS Code Remote Development extension copies the directory where one .git exists over your current directory in which you Reopen in Container. I tried three experiments below.

Experiment 1

Reopen in Container at FileGenerator in your local machine

C:.
└── FileGenerator
    ├── .devcontainer
    |   ├── Dockerfile
    |   ├── devcontainer.json
    |   └── requirements.txt
    ├── Dockerfile
    ├── main.py
    └── requirements.txt

Your directory in the docker container

root@xxxxxxx:/workspaces/FileGenerator#

Experiment 2

Reopen in Container at FileGenerator in your local machine

C:.
└── ProjectFolder1
    ├── .git
    └── FileGenerator
        ├── .devcontainer
        |   ├── Dockerfile
        |   ├── devcontainer.json
        |   └── requirements.txt
        ├── Dockerfile
        ├── main.py
        └── requirements.txt

Your directory in the docker container

root@xxxxxxx:/workspaces/ProjectFolder1/FileGenerator#

Experiment 3

Reopen in Container at FileGenerator in your local machine

C:.
└── ProjectFolder2
    ├── .git
    └── ProjectFolder1
        ├── .git
        └── FileGenerator
            ├── .devcontainer
            |   ├── Dockerfile
            |   ├── devcontainer.json
            |   └── requirements.txt
            ├── Dockerfile
            ├── main.py
            └── requirements.txt

Your directory in the docker container

root@xxxxxxx:/workspaces/ProjectFolder1/FileGenerator#

C# Class design for cloud storage management application

Kohei Kawata — Tue, 03 May 2022 13:10:41 +0000

Summary

I want to share a C# class design pattern I recently did. I know there are a lot of different design patterns and there is no single best way, but learning different design patterns helps myself develop extensible and scalable applications.

Sample code: blob-console-app

Architecture

Operators place files which they want to upload to Azure Blob Storage.
The .NET console app checks the pointed directory in a certain interval and if those files are valid, it uploads to the Azure Blob Storage.
The app deletes those files from the local file system once the files are uploaded.
The app extracts metadata and attaches the blob on cloud.
Currently the file types are logs and events which are located different directory in the local file system.

Class design

I decided to use Interface, Abstract, and sub classes because:

Currently file types are only Logs and Events, but the system will extend. For example, DeviceInfo and Requests will be added in the future.
It is better for future extended file types to divide the base features and ones with different requirements for each file type so the system reuses the base class features in the future.
In order to avoid changing the base class when adding new file types, an Interface class and abstract class defines base properties, and sub classes define metadata properties.
FolderName is a base property but it points to the directory for each file type. It is defined as Abstract property because Sub classes must override it and define for their own.
JudgeValidFile() and GetMetadata() have to be implemented in sub classes, and then they are defined as Abstract. UploadBlobAsync(), UploadMetadataAsync(), and DeleteFiles() are defined as Abstract because they are implemented in the base class but can be overriden in sub classes accordingly.

Codes

List file type instance

Currently the system has only two file types, Logs and Events. If you want to add another file type, only thing to do is to implement a new class for the file type and add a new instance here.

Program.cs

List<IUploadData> uploadDataList = new ()
{
    new Logs(),
    new Events(),
};

for instead of foreach

for (int i = 0; i < uploadDataList.Count; ++i) processes differently depending on class instance added to the list.
It uses for instead of foreach because the class method GetMetadata() gets metadata and put into its own property

Program.cs

uploadDataList[i] = uploadDataList[i].GetMetadata();

Dependency Injection

It does not use dependency injection but assigns the values to properties. This is because the value changes for different files found and file types. For example, BlobClient needs a file name and directory to instantiate for different files and different file types. It cannot create an instance in constructor.

Program.cs

List<IUploadData> uploadDataList = new()
{
    new Logs(),
    new Events(),
};

while (true)
{
    for (int i = 0; i < uploadDataList.Count; ++i)
    {
        uploadDataList[i].FileFullPaths = Directory.GetFiles(Path.Combine(localPath, uploadDataList[i].FolderName), "*", SearchOption.AllDirectories);

        foreach (string fileFullPath in uploadDataList[i].FileFullPaths)
        {
            uploadDataList[i].FileFullPath = fileFullPath;
            uploadDataList[i].FileName = Path.GetFileName(fileFullPath);
            uploadDataList[i].BlobClient = containerClient.GetBlobClient($"/{uploadDataList[i].FolderName}/{DateTime.Now.Year}/{DateTime.Now.Month}/{uploadDataList[i].FileName}");

Upload file validation

If a file found in the directory is not valid to upload, it stops the for loop and goes to a next file.
How to valid files is different depending on file types. That is why JudgeValidFile() is overriden in each file type class.

Program.cs

uploadDataList[i].JudgeValidFile();
if (uploadDataList[i].IsValidFile is false) continue;

Logs.cs

this.IsValidFile = (Path.GetExtension(this.FileName) == ".csv" && this.FileFullPaths.Contains(this.FileFullPath + ".json"));

Events.cs

this.IsValidFile = (Path.GetExtension(this.FileName) == ".json");

Generic class

I wanted to use a generic class for UploadMetadataAsync() because it has to retrieve the properties dynamically depending on the sub class. I thought UploadMetadataAsync<uploadDataList[i].GetType()>() works but actually not. After investigating sometime, I gave up using Generic class here.

Generic class

public virtual async Task UploadMetadataAsync<T>() where T : IUploadData
{
    PropertyInfo[] propertyCollection = typeof(T).GetProperties();
    foreach (PropertyInfo property in propertyCollection)
    {
        blobMetadata[property.Name.ToString()] = property.GetValue(this).ToString();
    }
}

Caller

await uploadDataList[i].UploadMetadataAsync<uploadDataList[i].GetType()>().ConfigureAwait(true);

Metadata into dictionary

I realized I cannot use Generic class, and I was thinking how to insert only metadata properties into the dictionary for uploading. What I did is to see the difference of property between Abstract class and sub class. For example, the Abstract class UploadData has only the base properties FolderName, IsValidFile, FileFullPaths, FileFullPath, FileName, BlobClient, but the sub class Logs has the metadata properties BeginTime, EndTime, Temperature, Humidity, Location in addition to the base properties.

UploadData.cs

this.blobMetadata.Clear();
PropertyInfo[] propertiesIUploadData = typeof(IUploadData).GetProperties();
IEnumerable<PropertyInfo> propertiesThis = this.GetType().GetRuntimeProperties();

foreach (PropertyInfo propertyThis in propertiesThis)
{
    int count = 0;
    foreach (PropertyInfo propertyIUploadData in propertiesIUploadData)
    {
        if (propertyThis.Name == propertyIUploadData.Name) ++count;
    }
    if (count == 0)
    {
        this.blobMetadata[propertyThis.Name.ToString()] = propertyThis.GetValue(this).ToString();
    }
}
await this.BlobClient.SetMetadataAsync(this.blobMetadata).ConfigureAwait(false);

Json deserialize

It is elegant to deserialize the value from a json file because I do not need to change the code JsonSerializer.Deserialize<Logs>(jsonString)!; for any model property. However, the problem was that the sub class has both the base property and metadata property, and when in deserialization, it sets the metadata properties with the value from a json file and also the base properties with null.
I do not like this, but I did not come up with an elegant workaround, and what I implemented is to keep the value to local variables and then set them back after deserialization.

json file example

{
  "BeginTime": "2022-03-07T12:21:55Z",
  "EndTime": "2022-03-07T14:01:36Z",
  "Temperature": 25,
  "Humidity": 63,
  "Location": "Tokyo"
}

Logs class property

public string FolderName { get; }
public bool IsValidFile { get; set; }
public string[] FileFullPaths { get; set; }
public string FileFullPath { get; set; }
public string FileName { get; set; }
public BlobClient BlobClient { get; set; }
public DateTime BeginTime { get; set; }
public DateTime EndTime { get; set; }
public int Temperature { get; set; }
public int Humidity { get; set; }
public string Location { get; set; }

Logs.cs

public override IUploadData GetMetadata()
{
    bool IsValidFile = this.IsValidFile;
    string[] FileFullPaths = this.FileFullPaths;
    string FileFullPath = this.FileFullPath;
    string FileName = this.FileName;
    BlobClient BlobClient = this.BlobClient;

    string jsonString = File.ReadAllText(this.FileFullPath + ".json");
    Logs log = JsonSerializer.Deserialize<Logs>(jsonString)!;

    log.IsValidFile = IsValidFile;
    log.FileFullPaths = FileFullPaths;
    log.FileFullPath = FileFullPath;
    log.FileName = FileName;
    log.BlobClient = BlobClient;

    return log;
}

DEV Community: Kohei Kawata

Virtual Network architecture 6 - Service Bus Private Endpoint

Summary

TOC

Private Endpoint configuration

Public network access

Support tier

Virtual Network architecture 7 - Self-hosted agent

Summary

TOC

Azure Container Instance

Virtual Network architecture 5 - App Service Private Endpoint

Summary

TOC

Private Endpoint configuration

Inbound and Outbound

Access restrictions

Support tier

Virtual Network architecture 4 - SQL Database Private Endpoit

Summary

TOC

Private Endpoint configuration

Access to SQL Database

SQL project update

Virtual Network architecture 1 - Do I need virtual network?

Summary

TOC

Background

Problems

How to deal with those problems

Virtual Network architecture 2 - Deployment pipelines

Summary

TOC

Overview

Architecture

Pipelines

Two pipelines overview

pipeline-base.yml

pipeline-vnet1.yml

pipeline-vnet2.yml

Virtual Network architecture 3 - Key Vault Private Endpoint

Summary

TOC

Private Endpoint configuration

Access to Key Vault

Application Gateway SSL certificate

Azure IoT Edge Integration Test template - Part.1

Summary

TOC

Overview

Architecture

Test steps

Infrastructure deployment

Network Security Group - inbound Port 22

VM domain name label

IoT Hub consumer group

Setup and installation

VM Service Connection

Blob Contributor Role

Code deployment and test execution

edge-module.yml

test-prep.yml

test-execution.yml

test-cleanup.yml

Azure IoT Edge Integration Test template - Part.2

Summary

TOC

Overview

IothubConnector

WeatherObserver

FileGenerator

FileUploader

FileUpdater

LocalBlobStorage

Azure IoT Edge Integration Test template - Part.3

Summary

TOC

Deployment manifest

What ManifestGenerator does

Configurations