Solved: Private endpoints yes or not?

#devops #programming #tutorial #cloud

🚀 Executive Summary

TL;DR: Azure Private Endpoints frequently cause connectivity issues because Virtual Machines, by default, resolve public IP addresses for PaaS services, leading to application timeouts due to firewall blocks. The fundamental solution involves correctly configuring DNS, primarily through Azure Private DNS Zones, to ensure hostnames resolve to their private IP addresses within the Virtual Network.

🎯 Key Takeaways

Azure Private Endpoints assign private IP addresses to PaaS services within your VNet, but default DNS resolvers still provide public IPs, leading to connection failures.
Azure Private DNS Zones are the recommended best practice for Private Endpoint DNS resolution, automatically creating ‘A’ records and linking to VNets for seamless private IP resolution.
For complex hybrid environments, implementing hybrid DNS forwarding with a dedicated DNS server in Azure and conditional forwarding to Azure’s magical DNS IP (168.63.129.16) unifies DNS resolution across on-premises and cloud networks.

Struggling with Azure Private Endpoints and the confusing DNS issues they create? This is a breakdown of why your VMs can’t connect and three real-world fixes, from the emergency hack to the proper architectural solution.

Azure Private Endpoints: To Use Them or Not? That’s Not The Real Question.

I remember it vividly. It was a Tuesday, just after lunch, when the PagerDuty alert screamed. A new deployment for our inventory service couldn’t connect to its brand-new Azure SQL database, inv-prod-sql-01.database.windows.net. A junior engineer, let’s call him Alex, was pulling his hair out. “Darian, I don’t get it! I can ping the private IP address of the database from the app server, the Network Security Groups are wide open between them, but the application refuses to connect. It keeps timing out.” He was right. The network path was there. But the application wasn’t using that path. This, my friends, is the classic Private Endpoint “gotcha” that separates a smooth deployment from a day of frantic troubleshooting. The real question isn’t if you should use them, it’s how you handle the DNS fallout when you do.

So, What’s Actually Going On? The DNS Lie.

When you create a Private Endpoint for a PaaS service like Azure SQL or a Storage Account, Azure does something brilliant and confusing at the same time. It carves out a private IP address from your own Virtual Network (VNet) and assigns it to that service. Your resource is now, for all intents and purposes, part of your private network. No more traversing the public internet. Awesome.

Here’s the problem: your Virtual Machine, app-prod-vm-01, doesn’t know that. By default, it’s still configured to use Azure DNS or some other public resolver. When your application tries to connect to inv-prod-sql-01.database.windows.net, the DNS server happily replies with the public IP address. Your VM then tries to connect to that public IP, which is promptly blocked by the PaaS firewall you (hopefully) configured to deny all public access. The ping to the private IP worked because Alex was using the IP directly, but the application uses the hostname. DNS is lying to your application, and that’s the root of all this pain.

Pro Tip: When troubleshooting connectivity, always, always, always check what IP a hostname resolves to from the source machine itself. A simple nslookup your-db-name.database.windows.net will save you hours of pain.

The Fixes: From “Get It Working NOW” to “Do It Right”

Let’s walk through the options, from the battlefield triage fix to the proper architectural pattern.

1. The Quick Fix: The `/etc/hosts` Band-Aid

This is the “It’s 2 AM and I need the site back up” solution. It’s dirty, it’s manual, and it’s a form of technical debt, but it will get you out of a jam. You manually tell the specific VM’s operating system to override DNS for that one hostname.

On your Linux VM (app-prod-vm-01), you’d SSH in and edit the hosts file:

sudo nano /etc/hosts

Then, add a line at the bottom mapping the public hostname to the private IP address of your endpoint:

# TEMPORARY FIX for Private Endpoint DNS - DEVOPS-1234
10.20.1.5   inv-prod-sql-01.database.windows.net

The moment you save that file, your application will be able to connect. You’re a hero.

WARNING: Do not leave this in place. This is a landmine. When you fail over the database, or if that private IP ever changes, this machine will be broken, and no one will remember to check a hardcoded hosts file. Use this to restore service, then immediately create a ticket to implement the permanent fix.

2. The Permanent Fix: Azure Private DNS Zones

This is the way you should be doing it 90% of the time. You create a special DNS zone in Azure that only works for your virtual networks. It hijacks the normal DNS process for a specific domain and provides the private IP instead.

Here’s the process:

Create a Private DNS Zone: In Azure, create a new Private DNS Zone. The name has to be specific to the service. For Azure SQL, it’s privatelink.database.windows.net. For a storage account, it would be privatelink.blob.core.windows.net.
Link the VNet: In the settings for your new Private DNS Zone, go to “Virtual network links” and link it to the VNet where your app servers live. This tells VMs in that VNet to use this zone for lookups.
Let Azure Do the Work: Now, when you create the Private Endpoint for your SQL database, Azure will see that the VNet is linked to the correct private zone and it will automatically create the ‘A’ record (e.g., inv-prod-sql-01 -> 10.20.1.5) for you. If you already created the endpoint, you might need to create this ‘A’ record manually.

Now, any VM in that VNet that does an nslookup will get the correct private IP. No manual hacks, and it scales perfectly as you add more app servers or databases.

3. The ‘Nuclear’ Option: Hybrid DNS Forwarding

What if you have a complex network? What if you have an on-premises monitoring server that needs to connect to the database over an ExpressRoute or VPN? Your on-prem DNS servers have no idea what an Azure Private DNS Zone is. This is where you bring out the heavy machinery.

The goal is to teach your entire network (on-prem included) how to resolve these privatelink addresses. The most common pattern is:

Deploy a DNS Server in Azure: This could be a pair of Windows Server VMs with the DNS role, or a Linux VM running BIND.
Configure Conditional Forwarding: On this DNS server, you set up a conditional forwarder. You tell it: “Any request for *.database.windows.net should be forwarded to Azure’s magical DNS IP, which is 168.63.129.16.”
Update VNet & On-Prem DNS: You point your Azure VNet’s DNS settings to use your new DNS server VM. On-prem, you configure your main DNS servers to forward requests for that zone to your Azure DNS server as well.

This creates a chain: On-prem App -> On-prem DNS -> Azure DNS VM -> Azure DNS Resolver -> Private IP. It’s more complex to set up but unifies DNS resolution across your entire hybrid environment.

Comparison at a Glance


Solution	Complexity	Best For	Scalability
`/etc/hosts` File	Very Low	Emergency fixes; single-server tests.	Terrible. Do not use for production.
Azure Private DNS Zone	Low	Most Azure-native workloads.	Excellent. The standard best practice.
Hybrid DNS Forwarding	High	Complex hybrid networks with on-prem needs.	Excellent, but requires more infrastructure.

So, back to Alex’s problem. We deleted the entry from his /etc/hosts file, created the privatelink.database.windows.net zone, linked the VNet, and like magic, everything just worked. He learned a valuable lesson that day: with Private Endpoints, networking is only half the battle. The other half is always, always DNS.