Virtual Network architecture 3 - Key Vault Private Endpoint

Summary

This article is Part.3 of virtual network architecture series. I will share the details of the private endpoint with Key Vault in api-management-vnet.

Private Endpoint configuration

Private Endpoint: Deploy the private endpoint and connect it to Key Vault and Key Vault's subnet in PrivateEndpoint.bicep.

KeyVaultId:existingKv.id
VirtualNetwork2SubnetIdKv:existingVnet2.properties.subnets[3].id

properties: {
  privateLinkServiceConnections: [
    {
      name: pe_name_kv
      properties: {
        privateLinkServiceId: KeyVaultId
        groupIds: [
          'vault'
        ]
      }
    }
  ]
  subnet: {
    id: VirtualNetwork2SubnetIdKv
    properties: {
      privateEndpointNetworkPolicies: 'Enabled'
    }
  }
}

Private DNS: Deploy Private DNS with the DNS name privatelink.vaultcore.azure.net in PrivateDns2.bicep.

var pdns_name_kv = 'privatelink.vaultcore.azure.net'

resource PrivateDnsKv 'Microsoft.Network/privateDnsZones@2020-06-01' = {
  name: pdns_name_kv
  location: 'global'
}

Virtual network link: Link the deployed Private DNS to the virtual network where the Key Vault subnet exists in PrivateDns2.bicep.

VirtualNetwork2Id:existingVnet2.id

resource VnetLinkKv 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2020-06-01' = {
  name: '${PrivateDnsKv.name}/${PrivateDnsKv.name}-linkc'
  location: 'global'
  properties: {
    registrationEnabled: false
    virtualNetwork: {
      id: VirtualNetwork2Id
    }
  }
}

Private DNS A record: Create a DNS A record and set the IP address from the deployed private endpoint in PrivateDns2.bicep.

output PrivateEndpointKvIpAddress string = PrivateEndpointKv.properties.customDnsConfigs[0].ipAddresses[0]

resource PrivateDnsAKv 'Microsoft.Network/privateDnsZones/A@2020-06-01' = {
  name: '${PrivateDnsKv.name}/${KeyVaultName}'
  properties: {
    ttl: 3600
    aRecords: [
      {
        ipv4Address: PrivateEndpointKvIpAddress
      }
    ]
  }
}

Access to Key Vault

App Service extracts credentials from the Key Vault with Key Vault reference through the Private Endpoint. The outbound from App Service is through the v-net integration. The Key Vault reference is defined in AppService.bicep.

resource AppServiceConfig 'Microsoft.Web/sites/config@2021-02-01' = {
  properties: {
    'Sql:ConnectionString': '@Microsoft.KeyVault(VaultName=${kv_name};SecretName=${kvsecret_name_sqlcs})'
    'Servicebus:ConnectionString': '@Microsoft.KeyVault(VaultName=${kv_name};SecretName=${kvsecret_name_sbcs})'
  }
}

Functions extracts credentials from the Key Vault in the same way as the App Service.The Key Vault reference is defined in FunctionApp.bicep.

resource FuncConfig 'Microsoft.Web/sites/config@2021-02-01' = {
  properties: {
    'ServiceBusConnectionString': '@Microsoft.KeyVault(VaultName=${kv_name};SecretName=${kvsecret_name_sbcs})'
    'SqlConnectionString': '@Microsoft.KeyVault(VaultName=${kv_name};SecretName=${kvsecret_name_sqlcs})'
  }
}

Container Instance running the self-hosted agent extracts Azure Active Directory client app secret from the Key Vault to get the Azure Active Directory OAuth2.0 token for the integration test in integration-test-bash.yml.

clientSecret=$(az keyvault secret show --vault-name $(KEYVAULT_NAME) --name $(KVSECRET_NAME_AADCLIENT) --query value -o tsv)

Application Gateway implements TLS termination for the backend pool, which means the API Management, with the certificate stored in the Key Vault. Application Gateway is not capable of accessing Key Vault through Private Endpoint but through Service Endpoint.

var agw_kv_secret_id = 'https://${kv_name}${environment().suffixes.keyvaultDns}/secrets/${kvcert_name_agw}'

sslCertificates: [
  {
    name: agw_ssl_certificate_name
    properties: {
      keyVaultSecretId: agw_kv_secret_id
    }
  }
]

httpListeners: [
  {
    properties: {
      sslCertificate: {
        id: resourceId('Microsoft.Network/applicationGateways/sslCertificates', agw_name, agw_ssl_certificate_name)
      }
    }
  }
]

Application Gateway SSL certificate

The problem is that Application Gateway cannot access Key Vault through Private Endpoint to extract SSL certificate in the configuration above, as of August 2022. This is discussed in the forum Application Gateway: Integration with Key Vault does not work #33157. In sum, Application Gateway has to access Key Vault's public IP through Service Endpoint, not through private IP.

Service Endpoint: To enable Service Endpoint for this case, you have to add the configuration when deploying the virtual network subnet for the Key Vault. You can see VirtualNetwork.bicep and find the configuration below. The Service Endpoint configuration should be implemented on the virtual network subnet where the service that wants to access Key Vault, which means Application Gateway subnet.

resource VirtualNetwork1 'Microsoft.Network/virtualNetworks@2021-03-01' = {
  properties: {
    subnets: [
      {
        name: snet_name_1_agw
        properties: {
          addressPrefix: snet_prefix_1_agw
          networkSecurityGroup: {
            id: Nsg1AgwId
          }
          serviceEndpoints: [
            {
              service: 'Microsoft.KeyVault'
              locations: [
                '*'
              ]
            }
          ]
        }
      }
    ]
  }
}

Public network access should be enabled because Application Gateway reaches out to the Key Vault through the public IP, but you can limit the access only from the Application Gateway subnet in KeyVault2.bicep. bypass: 'AzureServices' enables all Azure services to access the Key Vault through its public IP, virtualNetworkRules: [{id: VirtualNetwork1SubnetIdAgw}] allows access from Application Gateway subnet, and defaultAction: 'Deny' shut down all the access except Azure services and the Application Gateway virtual network subnet. All the access other than Application Gateway SSL certificate are through the Private Endpoint.

resource KeyVault 'Microsoft.KeyVault/vaults@2021-10-01' = {
  properties: {
    publicNetworkAccess: 'Enabled'
    networkAcls: {
      bypass: 'AzureServices'
      defaultAction: 'Deny'
      virtualNetworkRules: [
        {
          id: VirtualNetwork1SubnetIdAgw
        }
      ]
    }
  }
}