Abdul

Posted on Nov 7

Azure Storage Private Endpoints with Bicep: The DNS Bits Everyone Gets Wrong

#azure #bicep #devops #cloud

You turn off public access on a storage account, add a private endpoint, deploy... and your app gets a 403 "This request is not authorized to perform this operation".

The networking is almost never the problem. It's DNS.

This post walks through a Bicep pattern that works, explains why each piece is there, and covers the hybrid DNS gotchas that bite once on-premises networks get involved.

Why it breaks

When you create a private endpoint for mystorage.blob.core.windows.net, Azure changes the public DNS record into a CNAME:

mystorage.blob.core.windows.net
  -> CNAME mystorage.privatelink.blob.core.windows.net

If your client can resolve privatelink.blob.core.windows.net from a private DNS zone, it gets the endpoint's private IP (say 10.10.0.4) and everything works.

If it can't, the lookup falls through to public DNS, the client gets the account's public IP, and because public network access is disabled, the request is refused.

So the job is: make sure every client resolves the privatelink name privately.

The five pieces

A storage account with public network access disabled
A private DNS zone named privatelink.blob.<storage suffix>
A link from that zone to your VNet
The private endpoint itself, targeting the blob sub-resource
A DNS zone group on the endpoint, so Azure manages the A record for you

Step 5 is the one most examples skip. Without it you end up creating A records by hand, and they silently go stale if the endpoint's IP ever changes.

The Bicep

1. Storage account, locked down

resource storage 'Microsoft.Storage/storageAccounts@2023-01-01' = {
  name: storageName
  location: location
  kind: 'StorageV2'
  sku: {
    name: 'Standard_LRS'
  }
  properties: {
    minimumTlsVersion: 'TLS1_2'
    supportsHttpsTrafficOnly: true
    allowBlobPublicAccess: false
    allowSharedKeyAccess: false
    defaultToOAuthAuthentication: true
    publicNetworkAccess: 'Disabled'
    networkAcls: {
      defaultAction: 'Deny'
      bypass: 'AzureServices'
    }
  }
}

publicNetworkAccess: 'Disabled' is stronger than firewall rules: nothing outside your network can connect, whatever the ACLs say. Turning off shared keys pushes everyone onto Entra ID and RBAC, which is logged and revocable.

2. The private DNS zone

resource blobDnsZone 'Microsoft.Network/privateDnsZones@2020-06-01' = {
  name: 'privatelink.blob.${environment().suffixes.storage}'
  location: 'global'
}

Using environment().suffixes.storage instead of hard-coding core.windows.net keeps the template working in sovereign clouds, and keeps the Bicep linter happy.

3. Link the zone to the VNet

resource blobDnsLink 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2020-06-01' = {
  parent: blobDnsZone
  name: '${vnet.name}-link'
  location: 'global'
  properties: {
    registrationEnabled: false
    virtualNetwork: {
      id: vnet.id
    }
  }
}

A private DNS zone does nothing until it's linked to the VNet your clients use for DNS.

4. The private endpoint

resource blobPe 'Microsoft.Network/privateEndpoints@2023-05-01' = {
  name: 'pe-${storageName}-blob'
  location: location
  properties: {
    subnet: {
      id: privateEndpointSubnetId
    }
    privateLinkServiceConnections: [
      {
        name: 'blob'
        properties: {
          privateLinkServiceId: storage.id
          groupIds: [
            'blob'
          ]
        }
      }
    ]
  }
}

5. The DNS zone group (don't skip this)

resource blobPeDns 'Microsoft.Network/privateEndpoints/privateDnsZoneGroups@2023-05-01' = {
  parent: blobPe
  name: 'default'
  properties: {
    privateDnsZoneConfigs: [
      {
        name: 'blob'
        properties: {
          privateDnsZoneId: blobDnsZone.id
        }
      }
    ]
  }
}

This tells Azure to create and maintain the A record in the zone for you.

Other storage services need their own zones

Each storage sub-resource has its own endpoint and its own zone. If you use more than blob, repeat steps 2 to 5 for each:

Sub-resource (`groupIds`)	Private DNS zone
`blob`	`privatelink.blob.core.windows.net`
`file`	`privatelink.file.core.windows.net`
`queue`	`privatelink.queue.core.windows.net`
`table`	`privatelink.table.core.windows.net`
`dfs` (Data Lake Gen2)	`privatelink.dfs.core.windows.net`

Hybrid DNS: the part that bites later

Everything above works for resources inside Azure. On-premises clients are where it gets interesting. (Thanks to the r/AZURE folks who added these in the comments on my original post.)

Forward the public zone, not the privatelink one. On your on-prem DNS servers, create the conditional forwarder for blob.core.windows.net, not privatelink.blob.core.windows.net. If you only forward the privatelink zone, an upstream recursive resolver can answer the initial public name before your server ever gets to forward the CNAME target.

Forward to something inside Azure that can see the zone. Azure DNS Private Resolver (inbound endpoint) is the clean option. If you already run domain controllers or DNS servers in Azure, they can do the job and save you the resolver's cost.

Link the zone to the resolver's VNet too. The resolver can only answer from zones linked to the VNet it lives in. In a hub-and-spoke design, that usually means linking every privatelink zone to the hub.

Consider fallback to internet. If some clients need to reach storage accounts that don't have private endpoints, enable the fallback on the VNet link so names missing from the private zone resolve publicly (this property needs a recent API version for virtualNetworkLinks, such as 2024-06-01):

properties: {
  registrationEnabled: false
  resolutionPolicy: 'NxDomainRedirect'
  virtualNetwork: {
    id: vnet.id
  }
}

At landing-zone scale: let Azure Policy do it

The Bicep above suits a single workload, or learning how the pieces fit together. Across many subscriptions, Microsoft's Cloud Adoption Framework recommends a different split:

The platform team creates every privatelink.* zone once, in a central connectivity subscription, and links them to the hub.
DeployIfNotExists policies watch for new private endpoints and automatically add the privateDnsZoneGroups pointing at the central zones.
App teams deploy only the endpoint itself and never touch DNS.

Microsoft documents this in its guide Private Link and DNS integration at scale. If you run a landing zone, it's the approach to use.

Don't forget disaster recovery

Private endpoints are tied to a region. If you fail storage over to a secondary region, your existing endpoints and DNS records won't follow it automatically. Plan for this before you need it: create endpoints in the secondary region ahead of time, or have a tested runbook to update DNS, and include it in your DR tests.

"It works, but I can't see my data in the portal"

That's expected. With public access disabled, the portal's storage browser and az storage commands from your laptop get a 403 too. Work from a VM or service inside (or peered to) the VNet, or temporarily allow your IP while you need it.

Quick checklist

[ ] publicNetworkAccess: 'Disabled' on the account
[ ] One private DNS zone per sub-resource you use
[ ] Each zone linked to the VNet(s) your clients resolve from, including the resolver's
[ ] Private endpoint per sub-resource
[ ] DNS zone group on every endpoint
[ ] On-prem conditional forwarders for the public zone names
[ ] RBAC data roles (e.g. Storage Blob Data Contributor) for your apps' managed identities
[ ] nslookup from each network segment (spoke, hub, on-prem) returns the private IP
[ ] At scale: central zones plus DeployIfNotExists policies instead of per-app DNS
[ ] A DR plan for endpoints and DNS in the secondary region

If you'd like this as a complete, deployable template, I've packaged it with two others (a passwordless Web App + Azure SQL setup and a hub-and-spoke network) plus a PDF guide: Azure Bicep Starter Pack.

DEV Community