Virtual Network architecture 1 - Do I need virtual network?

Summary

This is a part of virtual network architecture series. In this article, I discuss problems with software development for cloud native architectures and pros and cons of virtual network architecture.

Background

As many of enterprise cloud users are moving on to cloud native technologies, so that they can meet continuously changing requirements and keep up with the velocity of fast application development. I often see those cloud users wondering where their security responsibilities lie. According to shared responsibility model, IaaS systems require fully to take care of secure infrastructures (including identity and directory infrastructure, applications, network controls, and operating system), while PaaS systems allow to offload some of those security responsibilities to the cloud provider. So I got a lof of questions from enterprise clients about in what areas they have to take security responsibilities and how to build the cloud native architecture strongly enough against potential attacks coming through the Internet. Zero trust approach is one of the important security concepts as PaaS resources are usually exposed over public endpoints by default and the traditional firewall-based security does not work. However, even with authentication and authorization mechanisms, the system is still threatened by network attacks and unauthorized traffic from the Internet.

In order to achieve both dynamic modern software architecture and enterprise production-ready security, Microsoft Azure offers Azure Private Link technology that enables access to Azure PaaS Services over a private endpoint located in a virtual network. Most of Azure PaaS Services such as Azure App Service have independent public endpoints while Azure IaaS Services such as Azure Virtual Machines are deployed on a virtual network. Private Link adds a private endpoint to Azure PaaS Services and enables access only from the virtual network through the private endpoint. What I want to emphasize here is that PaaS Services such as SQL Database is an independent entity on the internet and still has its public IP address to which someone on the internet can connect. Data access should be through Private Endpoint and private IP address, and the public IP is protected by IP filtering.

Problems

There would always be pros and cons when you adopt a new technology. Designing and developing a cloud native architecture with Azure Private Link costs you more. The pricing itself is usually not a big problem as one private endpoint costs around $2.4 per day. Instead, in reality the costs on developers become high.

1. Virtual network design and development workload

Compared with architecture without virtual network, it requires more workload to design and develop. For example, in my past work with an enterprise cloud developers I designed the network architecture with four virutal networks, eleven subnets, eight network security group (NSG) configurations, two virtual network peerings, five private DNS zones, five private endpoints, and one service endpoint. And then I designed and implemented a Infrastructure as Code (IaC) piepeline so the enterprise clients can automate those virtual network infrastructure deployment and manage resources by code. It actually took two months by our team.

2. Local development

With Azure Private Link technology, the cloud user can protect their Azure PaaS Services agaist any attacks and unauthorized access from the Internet, but at the same time the enterprise develpers cannot access those services either. For instance, without the virtual network protection the developers can test their code on their own local machines reading and writing data on a Azure SQL Database. When the virtual network envirnment limits access only from private endpoints, the developers' local machines cannot access the database on Azure. That would happen to other PaaS services (such as Azure App Service, Azure Key Vault, and Azure Service Bus).

3. DevOps agent access

In my work, I usually use Azure Pipelines agents (in Azure DevOps) to build, test, and deploy code to Azure PaaS Services. For example, I deploy a .NET application to Azure App Service and Azure Functions, OpenAPI specifications to Azure API Management, and data-tier applications (DACPAC) to Azure SQL Database. If the virtual network protects access from the Internet, even the Azure Pipelines agent cannot access the PaaS services for deployment.

How to deal with those problems

How we handle those problems depends on project contexts. Some enterprise companies have rigid company network restrictions, some have strong network design skills and experiences. It depends on the projects and development environments, but I would like to share how we usually deal with those network access problems.

1. Virtual network design and development workload

Our team is proud of contributing to open source software assets and Microsoft platform that are broadly available. In every project, we create reusable and sharable software assets that can be widely applicable with the agreement of the enterprise clients. Our team practices growth mindset by trying new things and learning from others, and then reuse the learnings and create shared software assets. One examle is Azure-Samples/MLOpsManufacturing created with learnings from multiple projects. As we work more engagements with more clients, more and more other developers can reuse the assets and do not need to spend months designing network security architectures.

2. Local development

One workaround for local development problems is to create similar environments in your local machine. For example, you can create a database on a local SQL Server with the same configurations as the Azure SQL Database and set up your applications, so that developers can switch between local and Azure databases by changing a database connection string. Another workaround is to have multiple environments on Azure. For instance, you can create dev/stg/prod environments. And you can have the dev environment without virtual networks, but the stg/prod environments with virtual networks and private endpoints, so developers can easily test PaaS resources in the dev environment, but make sure of strong protection for the stage and prod environments. The other workaround is to use Azure Bastion with virtual machines where developers can work on codes inside the virtual network. This workaround still has a bit problem because developers usually prefer to work with their own machine they are used to.

3. DevOps agent access

I see two options to solve Azure Pipelines agent access problems in the previous projects. The first option is to manage a Self-hosted agent where you build, test, and deploy your code to Azure PaaS resources in Azure Pipelines. This self-hosted agenet is located inside the virtual network, so it can access all of customers' Azure PaaS resources. You can set up your agent on Linux, macOS, Windows, and also on a Docker container. The downside of this options is you need to manage a IaaS virtual machine in your PaaS architecture, which causes you to maintain the compute infrastructure including update of OS, middleware, dependencies and tools. The other options is to create and remove firewall rules only when Azure Pipelines agent accesses to Azure PaaS resouces. For example, Azure SQL Database Deployment task has IpDetectionMethod option that adds agent IP address range for allowed IP Address rules and DeleteFirewallRule option that deletes the allowed IP address rule after the deployment. (Important notice is that this does not work for Deny public network access to the Azure SQL Server.)