...art of the practical
Networking between managed OpenShift clusters has become practical through Azure's Virtual WAN. When deploying CockroachDB on top, you become the owner of a production-ready, globally available distributed database. Let's create the key components that facilitate secure multi-region networking and architect this database solution.
what is this?
This blog features a single logical CockroachDB database that’s deployed on ARO (Azure Red Hat OpenShift) clusters which are provisioned across multiple regions.
This article is my second instalment into OpenShift networking at global scale. Here we leverage Microsoft Azure, and this complements my original post CockroachDB: Multi-Region ROSA using Cloud WAN that focuses on AWS equivalent infrastructure.
the technical catalog
- Virtual Networks are the fundamental building blocks for private networks in Azure. This foundation provides the ability for cloud services to interconnect and securely communicate with each other.
- Subnets are the specific IP addresses range constructs where resources are grouped and interconnected within a virtual network.
- ARO is the highly available, fully managed service for deploying Red Hat OpenShift instances that are monitored and operated jointly by Microsoft and Red Hat. OpenShift is Kubernetes running on Red Hat Enterprise Linux (RHEL) with built-in management UI services for convenience.
- Virtual WAN is a networking service that brings many networking, security, and globally scaled routing functionalities together, managed by a single operational interface. This is the key offering that lets us manage and connect local and regional virtual networks together.
- Virtual HUB is the service that lets you connect virtual networks together across sites and regions. The Virtual WAN architecture is a hub and spoke model so at least 1 hub is always required to establish a network-service mesh for your resources across the Microsoft network backbone.
- CockroachDB (aka CRDB) is the distributed database that is deployed on the worker nodes across the OpenShift clusters.
the typical ARO cluster
It is important to have a background into the relevant components of a managed Red Hat OpenShift cluster to understand how they can be connected together. When creating an ARO cluster, the resources and services are pre-configured and automatically deployed based on the sizing criteria that is specified during the creation process. Virtual Network Subnets are the required critical networking aspect of ARO clusters that compartmentalize the master and worker node pools. These are used by the Cloud WAN service and Hubs to establish the network routing rules.
Virtual WAN: facilitator of OpenShift cluster networking
This service defines the networking infrastructure and connectivity rules for the complete solution. The Virtual WAN defines the core network-framework that manages hubs, VPN sites, EspressRoute circuits, and virtual networks. My searches online revealed many great examples of single and multi-hub networks that handle groups of networks, including how to establish links to on-prem environments or other cloud resources (eg: multi cloud solutions).
In our solution, we will use a single HUB to define our CockroachDB network, and a distinct Virtual Network connections that’s associated with every managed OpenShift cluster in our solution.
The most important caveat is to define unique subnet ranges for each OpenShift environment, as this is how we’ll differentiate and access the actual network IPs across each node in each cluster.
the network service mesh
- In the following diagram, all master nodes are hidden for simplicity. They are typically hosted on the 172.x.0.x/16 subnet, while all workers are hosted on the 172.x.1.x/16 subnet.
- Worker nodes within each cluster are inherently connected; links have been omitted for simplicity in this chart.
- Every node has network visibility to every other node across clusters and regions. This example only illustrates Node 2 links for simplicity.
let’s create the Azure resources
The following steps will establish the necessary cluster infrastructure based on the included screenshots, but your settings may vary, such as IP CIDR ranges or regional settings.
Step 1: Create your Virtual Networks
For each OpenShift cluster, you need a corresponding virtual network with 2 subnets. One subnet is for OpenShift workers, the other is for OpenShift masters. This is required for associating an ARO cluster with a valid virtual network during the creation process.
Below is my new virtual network, named zlamal-aro-canada-01 with the 2 subnets.
Step 2: Create your OpenShift Clusters
We will focus on the networking aspects of the Azure ARO Creation Wizard.
After specifying the cluster name, region and node sizing, the Virtual Networks list will be filtered to the same region of the cluster region. In my example I chose Canada Central for my Cluster & networking.
I’ve chosen to make the endpoints visible for convenience and to avoid the challenges of establishing a VPN framework for this exercise.
Notice: This screenshot indicates 172.18.x.x virtual network subnets while the prior screenshot shows 172.23.x.x. This is because during this blog write-up I’ve created yet another virtual network for an upcoming OpenShift cluster, but normally these CIDR ranges would align with each other.
Step 3: Create a Virtual WAN instance
In this diagram, the first step is to specify the management region of the Cloud WAN instance. Actual networking at this phase is agnostic of locality. Specify your name and ensure the Type is Standard.
Step 4: Create the Hub instance
This hub resides in the Virtual WAN, and will connect all the OpenShift clusters together in the subsequent step. The hub is also assigned a region, but ultimately accepts connections from any Azure network-resource provisioned in any region.
In this example it’s named canada-east-hub, along with a special-defined address space used internally by Microsoft to manage/operate the hub instance. I’ve chosen a non-overlapping CIDR of 172.100.0.0/16 and this will not interfere with my clusters or any future additions to this network. To save on costs I kept the hub capacity at the minimal possible throughput and connection capability.
Step 5: Add the connections to the Hub
Every OpenShift cluster resides on a unique Virtual Network. This step establishes the connectivity on all the virtual networks. The hub manages and operates this pool of connections.
As you scale-out, create, or delete OpenShift clusters, here is where the interfaces are established or removed. This means creation of a Virtual WAN along with the Hub is a one-time process.
Step 6: Establish CockroachDB nodes
Every OpenShift is interconnected at this point. This means that if you open a terminal window to any running pod, you can curl to any worker on any of the clusters.
From this point you can create a unique OpenShift deployment instance for each worker-node affinity to ensure a 1:1 relationship between workers and CRDB nodes.
Catalog the IP addresses of the worker nodes in your primary OpenShift cluster that will serve as the starting point for CockroachDB. Once this primary cluster of 3+ nodes is initialized, all the other nodes can use these 3+ IP addresses, along with corresponding service NodePorts to connect to every other node on the cluster.
Load balancers can be created per-OpenShift-cluster, providing additional redundancy if a cluster becomes unavailable. This also clearly defines multiple access-points for clients to use and prioritize based on latency and geographic location.
This YAML fragment illustrates the locality and network parameters for a single node in the cluster. The --join statement is typically static for all nodes, while the sql-addr, listen-addr, and advertise-addr will be unique for each node.
...
...
apiVersion: apps/v1
kind: Deployment
metadata:
name: crdb-deployment-zlamal-172-23-1-4
labels:
crdb-cluster: crdb-zlamal
spec:
selector:
matchLabels:
crdb-pod-name: crdb-pod-zlamal-172-23-1-4
replicas: 1
template:
metadata:
labels:
crdb-pod-name: crdb-pod-zlamal-172-23-1-4
crdb-cluster: crdb-zlamal
...
spec:
...
containers:
command:
- /bin/bash
- '-ecx'
- >-
exec /cockroach/cockroach
start
--cache=.25
--logtostderr
--certs-dir=/cockroach/cockroach-certs
--locality=country=CA,region=CA-East,zone=Zone-1
--sql-addr=:26257
--listen-addr=:26357
--advertise-addr=172.23.1.4:31951
--join 172.23.1.6:31950,172.23.1.4:31951,172.23.1.5:31952
...
...
Each CRDB node will require its own NodePort service to establish the internal database network mesh, while a common service can internally load-balance connections for co-located applications.
This example service aligns with the above fragment to define unique NodePort port numbers that must not overlap with other services in the current OpenShift cluster.
kind: Service
apiVersion: v1
metadata:
name: crdb-service-zlamal-172-23-1-4
spec:
ports:
- name: sql-access
protocol: TCP
port: 26257
targetPort: 26257
nodePort: 31851
- name: node-comms
protocol: TCP
port: 26357
targetPort: 26357
nodePort: 31951
- name: console-ui
protocol: TCP
port: 8080
targetPort: 8080
type: NodePort
selector:
crdb-pod-name: crdb-pod-zlamal-172-23-1-4
A generic service would have the selector not to a pod, but rather the common label of the deployments. In this case set the selector to crdb-zlamal for the load balancers to properly interface with all the active pods.
For example a load balancer service definition handing the local cluster will have a selector as follows:
apiVersion: v1
kind: Service
metadata:
name: crdb-zlamal-load-balancer
labels:
crdb-cluster: crdb-zlamal
spec:
ports:
- name: console-ui
port: 8080
protocol: TCP
targetPort: 8080
- name: sql-access
port: 26257
protocol: TCP
targetPort: 26257
type: LoadBalancer
selector:
crdb-cluster: crdb-zlamal
conclusion
These Azure network services abstract away the complexities of infrastructure, security, and resource-management, offering a view of the entire architecture from a single pane of glass. I can't cover every detail into these technologies, but Azure also provides service & network monitoring, security controls, and visual representations of the connected ecosystem.
For a starter-pack of Kubernetes templates to help with deploying CockroachDB, go to the Cockroach/cloud/kubernetes GitHub repo.
references
Cockroach Labs
CockroachDB: Multi-Region ROSA (AWS) using Cloud WAN
What is Azure Virtual WAN?
Azure ARO
Top comments (0)