DEV Community: Alex

Tailscale vs WireGuard

Alex — Mon, 07 Aug 2023 18:03:25 +0000

Introduction

WireGuard is a next-gen, open source VPN protocol. It is easy to implement, provides extremely fast speeds, and has modern cryptography.

Tailscale is a VPN service built on top of the WireGuard protocol. It provides secure networking for teams and individuals, allowing them to create a network amongst their devices across various platforms.

In this article we’ll compare and contrast Tailscale and WireGuard, and introduce Netmaker, another VPN service built on WireGuard. Or, scroll to the bottom and get a side-by-side comparison of all three.

Why WireGuard?

For decades, businesses have relied on old, slow, but reliable VPN implementations like OpenVPN, IPsec, SSTP, and others. VPNs have tended to be clunky, and as such, businesses have started to migrate to new patterns like zero trust and SASE, that eliminate the VPN altogether.

When WireGuard was released in 2020, it changed the value proposition of VPNs. It is so fast that it can be used for data-intensive workloads with minimal impact on performance. It is so efficient, it requires minimal processing power. It is so simple that a basic VPN between two machines can be set up in minutes. And it is an extremely secure implementation, using state-of-the art cryptography and symmetric key encryption. Not to mention, it is open source.

This is why, for most IT administrators implementing a low-level VPN from scratch, WireGuard is really the only choice.

Why Tailscale?

With all these advantages, WireGuard is still a low-level protocol. This is by design. Jason Donenfeld has stated that he wants to keep WireGuard small and simple, and prefers to let others build more complex tools and platforms on top of the protocol.

Because of this, WireGuard lacks many of the features that users of more “standard” VPNs have become accustomed to. Things like user authentication, access controls, and a central server to manage the VPN. Enter Tailscale.

Tailscale takes the WireGuard protocol and wraps it in their own client application. Users register with Tailscale, set up an account, and can enroll clients in a private network. Connections between devices are done using WireGuard, but Tailscale manages more advanced aspects like users and device discovery. Using Tailscale’s UI, users can set up advanced access controls for a whole organization, and shape their network accordingly.

Advantages of Tailscale over WireGuard

Device Discovery: With Tailscale, you simply use the Tailscale client to authenticate with your network, and then you are immediately given access to all the other devices in the network. With WireGuard, this would be a manual effort. You would need to modify all of the WireGuard clients in the network to account for the new machine.

Network Management: Tailscale gives you a nice UI to log into, which allows you to view all of the devices in your network, and make configurations as necessary, which are automatically applied to the network. With WireGuard alone, this is again a manual effort. Any change to the network must be applied to each device individually.

User Management: Tailscale allows you to match devices to users, and use SSO to join a network. With WireGuard alone, there is no concept of “users”, just devices.

Advanced Features: There are many standard things a VPN administrator might like to do, which again are a manual process with pure WireGuard. Things like routing to an external network, private DNS, and access controls. All of these are included in Tailscale.

Advantages of WireGuard over Tailscale

Cost: While Tailscale offers a substantial free tier for personal use, using Tailscale in a business setting will not be free. Based on the setup, it could cost you $18 per user per month.

Data Ownership: Tailscale is a SaaS platform, meaning your devices are registered to a 3rd party. While theoretically your data is fully encrypted and invisible to Tailscale, it can still be concerning to have your sensitive info on a 3rd party platform. Additionally, traffic will regularly route through Tailscale’s relay servers, rather than directly between your devices.

Low Level Configuration: While WireGuard is fully customizable, there is no simple way to integrate Tailscale with a regular WireGuard network, or to customize a Tailscale interface as if it were a WireGuard interface. If you would like to create your own WireGuard interfaces, or manage a Tailscale device as if it were WireGuard, you’re probably out of luck. This will be frustrating to some users who wish to do some low-level tweaking of their device settings.

Speed: Lastly, Tailscale is slower than a pure WireGuard approach. Tailscale defaults to a non-kernel implementation of WireGuard, which has security benefits, but the tradeoff is speed. Also, your traffic is often routed through Tailscale’s relay servers, and some users have found this adds significant latency to their setups.

Netmaker

Netmaker is a third option which combines some of the more powerful features of Tailscale with a more native WireGuard approach. Like Tailscale, Netmaker offers user management, device management, advanced features, and a central UI / control plane for administrators. But Netmaker also has some additional advantages, for those who prefer pure WireGuard.

Advantages of Netmaker over Tailscale

Cost: Netmaker has a completely free community version, and a paid SaaS version, which starts at just $1 per device per month.

Data Ownership: In addition to a SaaS offering, Netmaker allows you to “self host” the control plane and relay servers, giving you complete data ownership.

Low Level Configuration: Netmaker has a “Client Gateway” feature, which allows you to generate and “hook in” pure WireGuard config files, meaning you can customize a WireGuard interface however you want, and integrate it into your network.

Speed: Netmaker defaults to kernel WireGuard, taking full advantage of WireGuard’s native speed benefits. You can see some speed tests here and here.

Limitations

User Management: Netmaker’s user authentication and authorization is simple, and does not currently offer the level of control or integration that Tailscale offers.

Client Application: Netmaker’s client is available for Windows, Mac, Linux, and FreeBSD, but the Windows and Mac experience is much less polished than Tailscale. There is also currently no iOS or Android application, and users must use the standard WireGuard client to access a Netmaker network from their mobile devices.

The Verdict

Tailscale and WireGuard are both excellent solutions but cater to different needs. If you're looking for a simple, secure, and easy-to-use VPN for larger networks or teams, Tailscale is a fantastic choice. It removes much of the hassle associated with setting up and managing a VPN.

On the other hand, if you need to perform low-level customizations, want complete control of your implementation, and do not want to pay, WireGuard is the way to go. It is also probably the best approach if you only have a few devices in your network, which will remain relatively static.

Netmaker offers a healthy mix of both options, making it an excellent choice for users who want full WireGuard speed, data control, and a degree of customization, while still getting a management platform and client application to simplify their implementation.

All of these options are secure, efficient, and modern VPN solutions. Your choice will depend on your specific requirements, technical expertise, and budget. By understanding the strengths and weaknesses of each, you can choose the right tool that aligns with your network needs.

Tailscale vs ZeroTier

Alex — Tue, 01 Aug 2023 12:18:16 +0000

Businesses have migrated towards a “remote first” and “cloud first” posture. In this world, traditional corporate firewalls with physical routers and switches are going away. Additionally, the rise of IoT and Edge Computing has made networking more complex, with corporate devices often deployed in public networks.

Companies have had to adapt. Traditional VPNs just don’t getting the job done, and new patterns like “zero trust” and SASE have taken their place. One alternative is the “mesh VPN” (Tailscale’s term) or “SD-WAN” (ZeroTier’s term), which bring the best of VPN’s and modern networking together to provide fast, point-to-point networks coupled with fine-grained access controls, giving businesses the best of both worlds.

In this article we’ll compare and contrast Tailscale and ZeroTier, and introduce Netmaker, another comparable platform. Or, scroll to the bottom and get a side-by-side comparison of all three.

Overview

Tailscale is a VPN service built on the WireGuard protocol. It provides secure networking for teams and individuals, allowing them to create a network amongst their devices across various platforms. Tailscale’s core advantage is its simplicity — A single user can hit the ground running in just minutes.

ZeroTier is a VPN platform that uses their own custom protocol to connect devices securely across the internet. ZeroTier’s advanced network virtualization capabilities allow you to “treat the entire planet like one data center.”

Similarities

At their core, ZeroTier and Tailscale are both very similar. Both are platforms that allow you to enroll and manage devices in a secure, peer-to-peer VPN network.

Endpoint Management

‍Both Tailscale and ZeroTier give every device in your network a private IPv4 address, which is reachable from anywhere, simplifying network management and remote access to devices. Both platforms also let you edit and make changes to each device in your network.

Peer-to-Peer Networks

‍ Both platforms also give you a peer-to-peer network by default, meaning every device has a direct connection to every other device, which is great for speed.

Access Controls

‍Both Tailscale and ZeroTier allow you to define policies that control the reachability of devices and the flow of traffic across your network.

Differences

While ZeroTier and Tailscale are very similar, and can often be used to accomplish the same use cases, they do have some differences, both technical and non-technical, which could make the difference for some users.

Protocol

‍Tailscale uses a userspace WireGuard protocol for its clients, while ZeroTier uses their own in-house protocol. Both are secure and performant, but there are advantages to using an “industry standard” like WireGuard.

Self-Hosting

Tailscale is a pure SaaS and you cannot “self-host” their control plane. Additionally, data will pass through their relay (DERP) servers fairly regularly. As a workaround, some users use Headscale, a project that lets you use Tailscale clients with a self-hosted server. ZeroTier has some self-hosting options, but you cannot use their UI if you do this, making it somewhat inconvenient, but still possible, to self-host.

Pricing

‍Both offer substantial free tiers, but ZeroTier prices by “node pack”, meaning you are buying the ability to deploy 25 nodes at a time, which is pretty simple to calculate. Tailscale meanwhile offers several “tiers” with different features, and charges per-user, with each user getting 10 or 20 devices included. Trying to figure out what you will end up paying is much more difficult for a Tailscale setup.

Configurability and Ease-of-Use

‍ZeroTier allows you to manage multiple networks, CIDR’s, IP’s, multicasting, and more. As mentioned above, you can also self-host portions of their platform which you cannot with Tailscale.

Tailscale, meanwhile, is focused on a simple user experience. You don’t have as many options as ZeroTier, but it is much easier to use, especially for users with limited networking knowledge.

User Management

‍Tailscale has a much more well-defined user management schema, and is more suited to the standard “corporate VPN” use case of end-user devices.

Device Management

‍ZeroTier has more low-level configuration options, which make it better at integrating devices like servers and VM’s.

Netmaker

Netmaker is a third option which combines some of the more powerful features of ZeroTier with the ability to run kernel WireGuard.

Advantages

Protocol: Similar to Tailscale, Netmaker is based on WireGuard, which makes it cryptographically modern, standard, secure, and fast.

Speed: Netmaker has shown to be faster than ZeroTier and Tailscale, due to its use of kernel WireGuard (Tailscale uses userspace WireGuard, which is slower). You can view two different speed tests here and here.

Flexibility: Netmaker is highly configurable. You can create your own relays, egress gateways, WireGuard gateways, and set up access controls to create many different types of networks besides the standard “mesh VPN.” You can also fully self-host Netmaker, unlike both Tailscale and ZeroTier.

Price: Netmaker has a substantial free tier, and unlike both ZeroTier and Tailscale, who’s pricing can be confusing, the paid tier starts at just $1 per device, making it an easy usage-based calculation.

Drawbacks

User Auth and Management: Netmaker’s user authentication and authorization is simple, and does not currently offer many of the integrations that some businesses may be looking for, such as session expiration and LDAP integration. Netmaker’s access controls are also on the device level, rather than the user level.

Client Application: Netmaker’s client is available for Windows, Mac, Linux, and FreeBSD, but the Windows and Mac experience is less polished than many user-centric VPN’s. There is also currently no iOS or Android application, and users must use the standard WireGuard client to access a Netmaker network from their mobile devices.

The Verdict

Tailscale and ZeroTier are excellent solutions but cater to different needs. If you’re looking for a simple, secure, and easy-to-use VPN for smaller networks or teams, Tailscale is a fantastic choice. It removes much of the hassle associated with setting up and managing a VPN.

On the other hand, if you’re dealing with a more complex, device-centric network infrastructure, and WireGuard is not a priority, ZeroTier is your go-to option.

Netmaker can offer additional configurability over both Tailscale and ZeroTier, as well as faster speeds, while using WireGuard under the hood, making it a great choice for users with device-centric networks needing WireGuard integration, or for users who need to host their own control plane.

Remote Access VPN to Azure with WireGuard and Netmaker

Alex — Thu, 20 Jul 2023 16:12:25 +0000

Intro

If you use Azure, you may deploy resources that you don’t want to expose publicly. For instnace, a Windows Server. In these scenarios, the resource will be contained in an Azure Virtual Network and Subnet. But how do you access these resources remotely from your devices?

Azure has their own remote access VPN solution called “Azure VPN Gateway”. However, this can be unnecessarily expensive. With several users and endpoints, you can easily spend hundreds of dollars per month.

Luckily, it is pretty easy to build an alternative to Azure VPN Gateway using WireGuard® and Netmaker for free. Follow these steps, and you should be up and running in about 30 minutes.

By the end of this tutorial, you will have a gateway device running on Azure, which you can use to access your private Azure resources using a WireGuard VPN client.

Scenario

In our example scenario, we have Windows Server 2019 Datacenter running on Azure, which is only accessible via RDP over the Virtual Network subnet address (10.0.0.4). We want RDP access to the server using this address.

For your setup, this could be any private IPs or subnets on Azure, as long they are accessible from the gateway device, which we will set up next.

Step 1: Deploy the Gateway Device

Deploy a device in Azure to act as your VPN gateway. We recommend using the latest Ubuntu on the smallest possible instance type, since it is not resource intensive. However, any linux distro or instance type should work.

This device must have access to the target devices or subnets, so add it to the same subnet. Reminder, make sure the Inbound Port rules on the target device or subnet will allow traffic from the gateway device.

Lastly, the device must be accessible over the WireGuard port, which by default for Netmaker is 51821, so open 51821/udp from Any source in the Inbound Port Rules (we’re doing 51821–51830/udp to be on the safe side), and make sure it has a Public IP Address.

Gateway Device Requirements:

Device Type: VM or Container (VM recommended)
OS: Linux (Ubuntu 22.04 recommended)
Size: any
Network Settings: Must have a public IP, be a part of the virtual network, and expose 51821/udp publicly (as well as port 22 for SSH access)

Step 2: Add Gateway Device to Netmaker

Now that you’ve deployed a suitable gateway device, you must add this device to Netmaker. You can self-host Netmaker, but to get started quickly (and for free), simply sign up at https://app.netmaker.io.

By default, your account will have a virtual network named “netmaker” and an access key, also named “netmaker”. You should use these for the remainder of the tutorial, but note that in our example and screenshots these are named “azure-gw”.

Click on the network, click on “hosts”, and then click the “Add a new host” button:

Follow the steps to add the gateway device to Netmaker: SSH to the device, download and installing the netclient, and joining the network.

Once the device is visible in your “hosts” lists, you can continue to configure the device as a Gateway.

Step 3: Configure Egress

Click on “Egress” and then “Create Egress”. We will set the gateway device as an egress to the virtual network subnet range in Azure. However, we could just as easily set it to a single IP or list of IPs within the subnet.

Step 4: Configure Client Gateway

The last step is to provide remote access via a “Client Gateway”. The Client Gateway simply allows you to generate WireGuard config files, which are routed through the gateway device and into the network. So, after configuring, a user will be able to reach the Egress range via the Client Gateway.

Our gateway device on Azure will act as both an “Egress Gateway”, to forward traffic to the private subnet, and a “Client Gateway”, so that it can accept traffic from WireGuard.

Click on “Clients” and then “Create Client”. Since you do not have a Client Gateway yet, it will prompt you to select a device to act as the gateway, and will generate your first client (WireGuard config file) on top of this gateway.

You can now download this config file, and run it using any standard WireGuard client.

(Click on the Client ID to view details)

If everything has gone correctly, you should now be able to RDP to the device using the private IP address:

You can generate additional clients as necessary, so that your gateway provides access for a whole team.

Summary

In this tutorial, we:

Configured Azure for a remote access gateway
Configured an Azure VM instance to act as the remote access gateway
Generated and ran a WireGuard config file locally, to access a private Windows server via the gateway

There is much more you can do with Netmaker and WireGuard, so I hope this was a good first experience.

If you have any questions or feedback, let me know in the comments!

Build your own Remote Access VPN to AWS with WireGuard and Netmaker

Alex — Wed, 19 Jul 2023 15:18:31 +0000

Introduction

An AWS account typically consists of multiple VPC’s and private subnets. You may wish to provide remote access to private subnets or endpoints on AWS without exposing them publicly.

AWS has their own remote access VPN solution called “AWS Client VPN”. However, this can be unnecessarily expensive. With several users and endpoints, you can easily spend hundreds of dollars per month.

Luckily, it is pretty easy to build your own solution using WireGuard® and Netmaker for free. Follow these steps, and you should be up and running in about 30 minutes.

By the end of this tutorial, you will have a gateway device running on AWS, on which you can easily attach WireGuard clients to access private AWS resources.

The Problem

In our example scenario, we have Rocket Chat running on AWS, which is only accessible over the VPC address (172.31.95.26). We want a developer to be able to log into Rocket Chat using this address.

For your setup, this can be any private IPs or subnets on AWS, as long as the addresses are accessible from the gateway device (EC2 instance).

Part 1: Deploy the Gateway Instance

Select a device in AWS to act as your VPN gateway. This can be a container or EC2 instance, but must be linux-based. You can use an existing instance, but if deploying a new instance, we recommend using the latest Ubuntu (22.04 as of this writing). You can use t2.micro, as it is not resource intensive.

This device must have access to the target devices or subnets, so make sure it is deployed in the correct availability zone, and that the target devices’ security settings allow traffic from the gateway device.

Lastly, the device must be accessible publicly over the WireGuard port, which by default for Netmaker is 51821, so open 51821/udp to 0.0.0.0/0 in the Security settings, and make sure it has a publicly reachable IP (e.g. Elastic IP address).

Gateway Requirements:

Device Type: EC2 Instance or Container (EC2 Instance recommended)
OS: Linux (Ubuntu 22.04 recommended)
Size: any (t2.micro recommended)
Network Settings: Must have a public endpoint, and expose 51821/udp publicly

Part 2: Setup the Gateway with Netmaker

Now that you’ve configured a suitable gateway device, you must add this device to Netmaker. You can self-host Netmaker, but to get started quickly (and for free), simply sign up at https://app.netmaker.io.

Click on the network, click on “hosts”, and then click the “Add a new host” button:

Follow the steps to add the gateway device to Netmaker, by downloading and installing the netclient, and joining the network.

Once the device is visible in your “hosts” lists, you can continue to configure the device as a Gateway.

Part 3: Configure Egress Gateway

Click on “Egress” and then “Create Egress”. We will set the gateway device as an egress to the target IP address in AWS. In our example this is 172.31.95.26/32, but modify this as appropriate, providing multiple ranges if necessary.

The device is now prepared to serve traffic to the target destination.

Part 4: Configure the WireGuard Client Gateway

Our device on AWS will act as both an “Egress Gateway” and a “Client Gateway”, so that it can accept traffic from WireGuard, and forward it to the private subnet.

You can now download this config file, and run it using any standard WireGuard client.

If everything has gone correctly, the private address should now be accessible from the local device:

Accessing the private Rocket Chat instance in the browser

You can generate additional clients as necessary, so that your gateway provides access for a whole team.

Conclusion

In this tutorial, we:

Configured AWS for a remote access gateway
Configured an EC2 instance to act as the remote access gateway
Generated and ran a WireGuard config file locally, to access AWS via the gateway

There is much more you can do with Netmaker and WireGuard, so I hope this was a good first experience. The above steps are also available as a click-through tutorial at the following link: https://www.netmaker.io/tutorials#remote-access-gateway

If you have any questions or feedback, let me know in the comments!

Netmaker - WireGuard Made Easy

Alex — Tue, 18 Jul 2023 12:15:46 +0000

If you've worked much in networking, you've probably heard of WireGuard®. If not, allow me to introduce it.

WireGuard is a VPN protocol developed several years ago which is exremely fast, lightweight, and uses state-of-the art cryptography. Its performance blows legacy VPN's out of the water.

A lot of people stumble upon WireGuard when they're looking to:
a) Build their own VPN
b) Set up remote access
c) Build a site-to-site network

The thing is, WireGuard can also be a little complicated to manage at scale. When you set up your network, you may find packets aren't going through, and you don't know why.

It gets more complicated the more you have to do. Route traffic to a private network? Set up roaming? Get around a NAT or Firewall? Your number of open tabs starts to increase exponentially.

Enter Netmaker.

Netmaker is a VPN platform built on WireGuard®, that lets you create any sort of network you might be looking to set up. Remote access? Sure thing. Inter-VPC overlay network? No problem.

Netmaker manages the hard parts of setting up a WireGuard network, including peer discovery, routing, key distribution, endpoint changes, and more.

Netmaker is easy to get started with. You can sign up here for free, or, if you're into self-hosting, check out the GitHub.

Then you're off to the races. Here's a tutorial to help get you started.

In summary:

How to Create Four Types of VPN’s Quickly with Wireguard® and Netmaker

Alex — Tue, 02 Nov 2021 16:26:14 +0000

A VPN can be a lot of things. You might think you know what a VPN is…but do you really? A virtual private network just sends private traffic over a public network. But where is that traffic going, and for what purpose? After all, networks come in all shapes and sizes.

In this two-part tutorial, we will discuss four “types” of VPN’s, which could just be considered use cases, and set them up with WireGuard® and Netmaker, a free networking platform, in under 10 minutes:

Personal (Private Browsing)
Remote Access
Site-to-Site
Mesh (virtual LAN/WAN)

WireGuard® is a new and very fast VPN technology. Usually, VPN’s make your internet slower. With WireGuard®, the difference is negligible. Netmaker uses WireGuard® under the hood, and can simplify setting up environments from homelab to enterprise-scale.

Let’s start by describing the 4 types of VPN’s. Then, we’ll set up a Netmaker server with a single command. Finally, we’ll go through each type of VPN and walk through a quick example setup. As a note, these tutorials expect some basic Linux skills. If you haven’t touched a terminal before, you may want to skip these.

In part one, we’ll cover Personal and Remote Access VPN’s for home use. These use cases are pretty easy to set up. In part two, we’ll discuss Site-to-Site and Mesh VPN’s, which are much more complicated, and geared towards business use cases.

The Four Types of VPN

First, a disclaimer. A VPN can have many, many topologies. Here, we break it down into four rough “categories”, but keep in mind as we move through the tutorials that in reality, a VPN could take on many arbitrary designs. These are just some helpful and common ways to break it down.

#1 - The Personal VPN (Private Browsing)

Most people are familiar with this type of VPN. If you’ve ever used a paid VPN service for your personal computer, it was probably this. This type of VPN is also actually a “Remote Access” VPN like in #2, but has a different purpose.

How it works is, before any of your network traffic hits the internet, it is first encrypted and then routed through some company’s servers. This way, no one knows the source of the traffic, giving you some anonymity.

#2 - The Remote Access VPN

Have you ever had to use a corporate VPN? If you’re working remote, like many people are in 2021, the answer is probably yes.

This type of VPN is actually very similar to the “personal” VPN, with one key difference: rather than routing your traffic to the internet, the VPN “server” can route your traffic to the corporate network. However, it is very often also routing your traffic for the general internet! That way, they can stop you from playing video games and browsing Reddit all day.

By accessing the corporate network via the VPN, you can access company applications and services (HR, payroll, etc) that aren’t on the public internet.

#3 - The Site-to-Site VPN

Unless you work in network engineering, you’ve probably never encountered this one. Still, you’ve probably experienced it without even knowing. Let’s say you have two offices, A and B. A and B need to talk to each other and share services, but they’re on different networks! Rather than expose all these services to the internet, they can set up a “Site-to-Site” VPN that bridges the two networks with a private link.

#4 - The Mesh VPN (Virtual LAN/WAN)

This is currently one of the least common VPN’s because of its complexity, but it is growing in popularity. You will most likely encounter this if you work in advanced computing topics like multi-cloud, Kubernetes, or IoT. In a mesh VPN, every single device has a direct VPN connection to every other device. This can be useful for creating “flat” networks, where you don’t have to route traffic through other devices, which would increase latency or decrease security.

Now that we’ve described the four use cases, let’s demonstrate them with four examples, using Netmaker and WireGuard®. In each example, we will set up a different VPN using the same Netmaker server, which can create and manage many networks:

Personal: **We will set up a VPN, and use it to access **the internet from our phone.
Remote Access: **We will set up a VPN, and use it to access **our **home network **from our phone.
Site-to-Site: We will set up a VPN, and use it to make a private subnet accessible from a cloud environment and vice-versa.
Mesh: We will set up a VPN, and connect VM’s in different clouds together so they can coordinate.

In part one, we will cover the first two examples, which are more relevant to individuals, and then in part two we’ll move on to site-to-site and mesh VPN’s, which are more relevant to organizations.

Getting Started: Deploy Netmaker

A Netmaker server will manage our various networks. Netmaker is a platform for deploying and automating WireGuard-based networks. We’re going to be running the quick install. The instructions can also be found in the README on GitHub.

To deploy Netmaker, you will need a cloud VM, or really any machine with a public IP. I’m using a $5/mo Digital Ocean Droplet (You can get $100 in free credit here using our referral code).

We recommend deploying a Ubuntu 20.04 instance. It is most important that you remember to **open the firewall **for your VM, and configure security groups if necessary. The following ports must be open and reachable on the machine:

443 (TCP and UDP)
53 (TCP and UDP)
51821–51830 (UDP)

If you are using DigitalOcean, those ports will already be open. Once you’ve created a VM and opened the ports, deploying Netmaker is a breeze. Just run the following command:

ssh root@yourhost
sudo wget -qO - https://raw.githubusercontent.com/gravitl/netmaker/develop/scripts/nm-quick.sh | bash -s -- -v true -c 10

This command will install WireGuard® and some other local dependencies, deploy Netmaker, and configure Netmaker with **two networks: **A default (peer-to-peer) network, and a “VPN” network (this is what the options -v true -c 10 add).

Do not clear the output, which contains valuable information for configuring the network. You may want to save this in a text editor. Example output:

checking for root permissions...

setting flags...

checking for root permissions...

checking dependencies...

setting caddyfile...

setting docker-compose...

starting containers...visit dashboard.nm.148-188-172-168.nip.io to log in

creating default network (10.101.0.0/16)

creating default key

configuring netmaker server as ingress gateway

finished configuring server and network. You can now add clients.

**For Linux and Mac clients, install with the following command:**

curl -sfL https://raw.githubusercontent.com/gravitl/netmaker/develop/scripts/netclient-install.sh | sudo KEY=eyaJjbb3JcaiaaefI6IjE0N5y4xOa6aDeIeuMTcyaaLjaE2OCIsImFwaWNviJhcGkubm0uMTQ3LTE4Maefi0xNzItMTY4Lm5pcC5pbaDMiLCJhcGloba3NaefiYXBpLm5tLjEa0Ny0xODItMTsrgaBjd2dwb3J0IjoiIiwiZ3JwY3dncHViasrgiIiwiZ3JwaY3dnZW5kxfahiOiIaifQ== sh -

**For Windows clients, perform the following from powershell, as administrator:**

1. Make sure WireGuardNT is installed - https://download.wireguard.com/windows-client/wireguard-installer.exe

2. Download netclient.exe - wget https://github.com/gravitl/netmaker/releases/download/latest/netclient.exe

3. Install Netclient - powershell.exe .\netclient.exe join -t eyaJjbb3JcaiaaefI6IjE0N5y4xOa6aDeIeuMTcyaaLjaE2OCIsImFwaWNviJhcGkubm0uMTQ3LTE4Maefi0xNzItMTY4Lm5pcC5pbaDMiLCJhcGloba3NaefiYXBpLm5tLjEa0Ny0xODItMTsrgaBjd2dwb3J0IjoiIiwiZ3JwY3dncHViasrgiIiwiZ3JwaY3dnZW5kxfahiOiIaifQ==

4. Whitelist C:\ProgramData\Netclient in Windows Defender

**For Android and iOS clients, perform the following steps:**

1. Log into UI at dashboard.nm.147-182-172-168.nip.io

2. Navigate to "EXTERNAL CLIENTS" tab

3. Select the gateway and create clients

4. Scan the QR Code from WireGuard app in iOS or Android

Netmaker setup is now complete. You are ready to begin using Netmaker.

creating vpn network (10.201.0.0/16)

configuring netmaker server as vpn inlet...

configuring netmaker server vpn gateway...

creating client configs...

finished configuring vpn server.

**To configure clients, perform the following steps:**

**1. log into dashboard.nm.148-188-172-168.nip.io**

**2. Navigate to "EXTERNAL CLIENTS" tab**

**3. Download or scan a client config (vpnclient-x) to the appropriate device**

**4. Follow the steps for your system to configure WireGuard on the appropriate device**

**5. Create and delete clients as necessary. Changes to netmaker server settings require regenerating ext clients.**

**Netmaker setup is now complete. You are ready to begin using Netmaker.**

This last part (highlighted) is what we will use for our first VPN.

Personal (Private Browsing) VPN

As we discussed earlier, a personal VPN allows you to browse the web with some anonymity. Typically, you pay a company to host the VPN for you. Here, we have just deployed our own VPN server, so we will use that instead.

By running the Netmaker install script with the optional “-v true” command, Netmaker set up a Personal (Private Browsing) VPN for us. All we need to do is follow those last steps of output.

**To configure clients, perform the following steps:**

**1. log into dashboard.nm.148-188-172-168.nip.io**

**2. Navigate to "EXTERNAL CLIENTS" tab**

**3. Download or scan a client config (vpnclient-x) to the appropriate device**

**4. Follow the steps for your system to configure WireGuard on the appropriate device**

**5. Create and delete clients as necessary. Changes to netmaker server settings require regenerating ext clients.**

1. Navigate to the dashboard and create an admin user.

2. Log in with your new admin user.

3. You are taken to the home screen. Click on EXTERNAL CLIENTS in the upper right.

4. Configure Client:

Netmaker configured 10 VPN clients, which can be used to connect with WireGuard®. At this point, you need to decide which device(s) will use the VPN. These devices need WireGuard installed. For our tutorial, I will be connecting from Mac, which means installing from the Apple store.

Once you have the app, open it, and click “Add Tunnel”. You can then click “Create from QR Code”, and scan the QR code of one of the clients (e.g., vpnclient-1.

Activate the tunnel and visit whatsmyip.org. You will see that your IP address is now the IP of the server. Congrats! You have set up your own personal VPN.

Now, are you ready to take it to the next level?

Remote Access VPN

Remember what we said about “Remote Access” VPN’s? Typically they are used to access a work environment. However, you probably don’t have access to configure your corporation’s network access controls. For this tutorial, let’s just assume you want to access your home network. It actually works quite similarly.

For this, we need a “gateway” into our home network. That could be a router, or it could be a device on the network. Netmaker does not offer “official” support for routers yet (though it can be run on, for instance, OpenWRT), so we’re going to use a device on the network, namely, my personal computer.

To create a gateway with Netmaker, we need the address range, and the local network interface to use. First, from my computer, I’ll check what networks I have access to:

afeiszli@alex-sys76:~$ ip route

default via 192.168.40.1 dev wlo1 proto dhcp metric 600

**192.168.40.0/24 dev wlo1 proto kernel scope link src 192.168.40.75 metric 600**

Okay, so based on this, my local network is 192.168.40.0/24, and I access this network over the wlo1 interface. Now, time to install the netclient.

If you recall from the install, two networks were created, default and VPN. We’re going to use the **default **network for this setup. In the output from the install, there was a section with instructions for installing on various devices:

**For Linux and Mac clients, install with the following command:**

curl -sfL https://raw.githubusercontent.com/gravitl/netmaker/develop/scripts/netclient-install.sh | sudo KEY=eyaJjbb3JcaiaaefI6IjE0N5y4xOa6aDeIeuMTcyaaLjaE2OCIsImFwaWNviJhcGkubm0uMTQ3LTE4Maefi0xNzItMTY4Lm5pcC5pbaDMiLCJhcGloba3NaefiYXBpLm5tLjEa0Ny0xODItMTsrgaBjd2dwb3J0IjoiIiwiZ3JwY3dncHViasrgiIiwiZ3JwaY3dnZW5kxfahiOiIaifQ== sh -

**For Windows clients, perform the following from powershell, as administrator:**

1. Make sure WireGuardNT is installed - https://download.wireguard.com/windows-client/wireguard-installer.exe

2. Download netclient.exe - wget https://github.com/gravitl/netmaker/releases/download/latest/netclient.exe

3. Install Netclient - powershell.exe .\netclient.exe join -t eyaJjbb3JcaiaaefI6IjE0N5y4xOa6aDeIeuMTcyaaLjaE2OCIsImFwaWNviJhcGkubm0uMTQ3LTE4Maefi0xNzItMTY4Lm5pcC5pbaDMiLCJhcGloba3NaefiYXBpLm5tLjEa0Ny0xODItMTsrgaBjd2dwb3J0IjoiIiwiZ3JwY3dncHViasrgiIiwiZ3JwaY3dnZW5kxfahiOiIaifQ==

Keep in mind, a gateway can only be installed on Linux devices currently. Luckily, my computer runs Linux! So we just have to run that first command.

curl -sfL https://raw.githubusercontent.com/gravitl/netmaker/develop/scripts/netclient-install.sh | sudo KEY=xxxxxx sh -

checking dependencies...

wireguard

wireguard is installed

no $VERSION provided, fallback to latest

OS Version = Linux

Netclient Version = latest

Binary = netclient

2021/10/31 11:42:06 [netclient] node created on remote server...updating configs

2021/10/31 11:42:06 [netclient] retrieving peers

2021/10/31 11:42:07 [netclient] starting wireguard

2021/10/31 11:42:08 [netclient] joined default

Great! We’re now in the network. Going to the dashboard, we can now turn this machine into a gateway.

A small icon indicates that the machine is now a gateway:

Now, if I add a new machine that is outside of my home network, it will be able to access anything on that network. For instance, I’ve deployed a Wordpress site that lives on 192.168.40.30:

Now, how can I access this site from the road? Let’s use an External Client, just like we did for the personal VPN:

We click the little plus button and confirm, and presto!

Now, connect from your phone just like before (using the WireGuard® app), and…

Nice! I now have home network access from the road.

One More Thing

We could have easily created the “home gateway” node on our “VPN” network, and have both in one! Secure access to our home network, and secure access to the internet. That’d be pretty great, right? If interested, I’m going to challenge you to set that up on your own.

Conclusion / Part 2

This concludes Part 1 of our tutorial on the four types of VPN’s. Subscribe to stay notified of part two, where we’ll cover how to configure a Site-to-Site virtual network, and a mesh VPN network. That really gets into some heavy networking stuff, and is meant for more advanced users. In the meantime, feel free to check out more examples for running Netmaker on our Resources Page.

**Disclaimer:* WireGuard is a registered trademark of Jason A. Donenfeld.*

3 Reasons to Choose Wide k8s Clusters vs. Multi-Cluster

Alex — Thu, 21 Oct 2021 18:57:21 +0000

At this year's KubeCon, the buzz was all about distributed Kubernetes: Edge, Hybrid Cloud, and Multi-Cloud.

For each of these topics there's an endless number of solutions: KubeEdge, OpenShift Edge, Akri, Baetyl, Kubermatic, Lens, Rancher, KubeFed, KubeSphere, Red Hat ACM, Liqo, Skupper, Linkerd, Fleet…

…the list goes on and on and on and on and…you get the point.

One thing not really discussed? You can run a single cluster across servers in different locations. You can skip the new tools and run a cluster like you normally would, but extended to new environments. Sounds like a no-brainer, right?

This is called a "wide cluster" or "stretched cluster", and it's an alternative to the "multi-cluster" model that has been coming into popularity. Before we discuss why you might want to implement a wide cluster vs. a multi-cluster architecture, let's discuss some common concerns.

Latency

Etcd (the Kubernetes "brain") is latency intolerant, so if your control plane nodes are too far apart, a wide cluster just plain wont work. You also don't usually want to introduce latency between worker nodes either because, well, application performance.

However, both problems can be easily solved in by:

Co-locating Etcd nodes, or using a non-Ectd alternative (dqlite, anyone?)
Applying node labels for locations and using node selectors for apps

Security

If you're running a single cluster between clouds, that likely means running inter-node traffic over a public network, which is scary. It can also be challenging, since you need to provide direct connectivity between all nodes.

The solution here is a mesh VPN (no, not a service mesh).

The VPN will encrypt all your traffic and provide a flexible subnet where all your nodes can communicate directly and securely.

You may also have special considerations around access controls. There are ways to manage access from a single cluster, but maybe you'd rather just have multiple clusters as your way of managing access. And that's ok, I won't judge you.

Cloud Costs

The number one reason you might need to avoid running a wide cluster is the cloud costs. Some certain, giant, cloud providers will charge you through the nose for data egress. If you're running worker nodes between such clouds and your data center, you're gonna pay for it. Still, as we'll discuss below, you may actually end up saving money with a wide cluster.

Cloud Support

Most cloud-hosted k8s options can't be easily extended to other locations, which makes sense, since cloud providers have every incentive to keep you in their cloud. If you're stuck running certain distributions, you may just be stuck.

So Why Run a Wide Cluster?

We've discussed the concerns, some of which have easy answers and some of which are harder. With all that, here's why you might want to think about running a wide Kubernetes cluster as an alternative to a multi-cluster architecture.

Reason #1: No complicated tooling

As discussed, there are a thousand tools for running multi-cluster, hybrid cloud, and edge computing with Kubernetes.

A large portion of these tools and platforms require a whole new framework for app deployments which must be adopted across all of your clusters. That's a lot of learning, and a lot of dependency on a new tool. The solution might also require relying on a single k8s distribution or cloud provider.

Alternatively, with the single, multi-cloud cluster approach, you can run your operations exactly as you would with a standard cluster. No new tools, and vastly simplified operations.

Reason #2 : No extra overhead

If you're running large, complex clusters, you probably have a lot of redundant components across all of them. Components to handle storage, networking, metrics, logging, images, pipelines, and more may need to be replicated across each and every cluster. That overhead adds up.

In addition, each cluster needs its own control plane, and assuming they're all HA, that's 3+ additional nodes per cluster. Compute gets expensive.
Compare this to a single, wide cluster, where there is one control plane and one set of services to support nodes in different clusters. You can add in special tooling as needed for particular environments, but you don't have to, which is a key distinction.

This is why if you're hesitant because of egress data charges in your cloud, you might still want to weigh this against the cost of a multi-cluster infrastructure.

Reason #3: Ultimate Flexibility

It's almost impossible to describe how flexible a cluster on a mesh VPN becomes. You have given it an extensible networking base from which to grow into new environments. A normal cluster will always be just that, a normal cluster occupying a subnet in a data center.

Sure, you can put some tools on top to make it handle some more fancy operations, but the cluster itself is fundamentally limited to that location in the data center.

On the other hand, a k8s cluster built on a mesh VPN can grow. It can expand to new locations. Its nodes can live in any location it needs to be. You can cloud burst into a new provider and remove all those nodes when they're no longer needed. The cluster can shift from place to place. The underlying infrastructure becomes incredibly malleable.

This is why, even if you're running just a single cluster on a single cloud, you may still want to deploy it on a mesh VPN just in case that ever changes.

Wrapping Up

This concludes my Ted Talk. We've discussed the pros and cons of a wide cluster on a mesh VPN vs. the standard multi-cluster architecture. I hope this at least sparks some ideas for you when planning your next Kubernetes deployment.

How to Deploy a Cross-Cloud Kubernetes Cluster with Built In Disaster Recovery

Alex — Sun, 18 Jul 2021 12:49:08 +0000

This article explains how to run a single Kubernetes cluster that spans a hybrid cloud environment, making it resilient to failure. It will explain why this is necessary, and how to implement this architecture using MicroK8s, WireGuard, and Netmaker. Okay, ready?

Kubernetes is hard, but you know what’s even harder? Multi-cloud, multi-cluster Kubernetes, which is inevitably what you end up dealing with when running Kubernetes in production.

Typically you will deploy two clusters at a bare minimum for any production setup: one as the live environment and one for failover (for disaster recovery). This may lead you to wonder:

“Why do I need two clusters to handle disaster recovery? I thought Kubernetes had a distributed architecture?”

Correct! Kubernetes is distributed! It’s distributed inside of a single data center/region. Outside of that…not so much.

To have a truly “high available” infrastructure, you’re going to need two clusters (or more), and on top of that you’re going to need some automation tools to move and copy applications between the clusters, along with some sort of mechanism to handle failover when a cluster goes down. Sounds like fun, right?

Stick with me, and we’ll walk through a less painful way of handling disaster recovery (and hybrid workloads, for that matter), this time with a single cluster.

The Distributed Cluster — MicroK8s and Netmaker

There are three limitations that typically prevent using a single cluster across environments:

Etcd: It is the brain of your cluster and is not latency tolerant. Running it across geographically separated environments is problematic.
Networking: Cluster nodes need to be able to talk to each other directly and securely.
Latency: High latency is unacceptable for enterprise applications. If a microservice-based application spans multiple environments, you might end up with sub-optimal performance.

We can solve all three problems with MicroK8s and Netmaker:

Etcd: Etcd is the default datastore for Kubernetes, but it’s not the only option. MicroK8s runs Dqlite by default. Dqlite is latency tolerant, allowing you to run master nodes that are far apart without breaking your cluster.
Networking: Netmaker is easy to integrate with Kubernetes and creates flat, secure networks over WireGuard for nodes to talk over.
Latency: Netmaker is one of the fastest virtual networking platforms available because it uses Kernel WireGuard, creating a negligible decrease in network performance (unlike options such as OpenVPN). In addition, we can use Kubernetes’ built-in placement policies to group applications together onto nodes in the same data center, eliminating the cross-cloud latency issue.

So now we have our answer. By running MicroK8s and Netmaker, you can eliminate complex, traditional, multi-cluster deployments.

Enough talking, let’s put this into action!

Setting up our environment

We’re going to use three environments. This ensures that if any one environment goes down, our masters can still form consensus.

You’ll notice that we don’t differentiate between masters and workers going forward. That’s because in MicroK8s, every node has a copy of the control plane, so there really isn’t a distinction.

Our Cluster Layout

Data Center: **2 Nodes** (datacenter1, datacenter2)

DigitalOcean (region 1): **1 Node** (do1)

DigitalOcean (region 2): **1 Node** (do2)

We have two data center nodes and two cloud nodes which can be used in case of failover.

We’re using DigitalOcean for our cloud nodes because they have the lowest bandwidth costs. Data transfer costs can add up real fast depending on your cloud provider. DigitalOcean’s bandwidth pricing is very reasonable and you should be able to run your cluster without incurring excess costs at all.

All of our nodes are running Ubuntu 20.04, and every node should have WireGuard installed before running this tutorial.

apt install wireguard wireguard-tools

Node 1: The Seed

SSH to your first node, which will act as the “seed” for your cluster, because it will setup Netmaker and establish the network which will run on the other nodes. This node should be publicly accessible (We’re using do1):

ssh root@do1
snap install microk8s --classic
microk8s enable dns ingress storage

You’ll note we’re using the built-in MicroK8s storage. For a production setup you will likely want something more robust like openebs, another MicroK8s plugin.

Next, make sure you have wildcard DNS set up to point towards this machine. for instance, in Route53 you can create a record for *.kube.mydomain.com pointing to the public IP of this machine.

After this is done, let’s set up some certs:

microk8s kubectl create namespace cert-manager

microk8s kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.14.2/cert-manager.yaml

Then create and apply the following clusterissuer.yaml, replacing the EMAIL_ADDRESS placeholder with your email:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: cert-manager
spec:
  acme:
    # The ACME server URL
    server: [https://acme-v02.api.letsencrypt.org/directory](https://acme-v02.api.letsencrypt.org/directory)
    # Email address used for ACME registration
    email: EMAIL_ADDRESS
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
          class: public

It will take a few minutes for the cert manager to become available, so this command will not succeed immediately after the above steps:

microk8s kubectl apply -f clusterissuer.yaml

Now, we’re ready to deploy Netmaker. First, wget the template:

wget https://raw.githubusercontent.com/gravitl/netmaker/develop/kube/netmaker-template.yaml

Next, insert the domain you would like Netmaker to have. This must be a subdomain of the DNS you set up above, for instance, if your external load balancer is pointing to *.kube.mydomain.com, you might choose nm.kube.mydomain.com. The template will then add the following subdomains on top: dashboard.nm.kube.mydomain.com, api.nm.kube.mydomain.com, and grpc.nm.kube.mydomain.com.

sed -i 's/NETMAKER_BASE_DOMAIN/<your base domain>/g' netmaker-template.yaml

And now, install Netmaker!

microk8s kubectl create ns nm

microk8s kubectl config set-context --current --namespace=nm

microk8s kubectl apply -f netmaker-template.yaml -n nm

It will take about 3 minutes for all the pods to come up. Once they are up, go to the dashboard (“microk8s kubectl get ingress” to find the domain):

microk8s kubectl get ingress nm-ui-ingress-nginx

Create a user and log in. You will see a default network:

Delete this (go to edit → delete) and create a new one, which we’ll call microk8s. Make sure the IP range does not overlap with microk8s. We’re giving ours 10.101.0.0/16:

Now we need a key for our nodes to securely connect into this network. Click on “Access Keys,” generate a new key (give it a high number of uses, e.g. 1000), and click create. Make sure to copy and save the value under Your Access Token. This will appear only once.

This is all we need to do inside the Netmaker server. Now we can set up our nodes with the netclient, the agent that handles the networking on each machine.

But first, a MicroK8s caveat! MicroK8s requires node hostnames to be reachable from each other in order to function properly. For instance, if you log into your machine and see root@mymachine, mymachine should be a resolvable address from the other machines. Netmaker will handle this, provided we set our hostnames correctly.

Node hostnames should be of the format nodename.networkname. Since our nodes are on the microk8s network, every hostname must be of the format .microk8s.

On our “seed node”, let’s set the hostname to do1.microk8s. That way we know it’s a Digital Ocean node.

hostnamectl set-hostname do1.microk8s

Okay, now we’re ready to deploy the netclient:

wget https://github.com/gravitl/netmaker/releases/download/v0.5.11/netclient && chmod +x netclient

./netclient join -t <YOUR_TOKEN> --dns off --daemon off --name $(hostname | sed -e s/.microk8s//)

You should now have the nm-microk8s interface:

wg show

#example output
#interface: nm-microk8s
#  public key: AQViVk8J7JZkjlzsV/xFZKqmrQfNGkUygnJ/lU=
#  private key: (hidden)
#  listening port: 51821

We could have deployed the netclient as a systemd daemon, but instead, we’ll use a cluster daemonset to manage our netclient. This allows us to handle network changes and upgrades using Kubernetes.

wget https://raw.githubusercontent.com/gravitl/netmaker/develop/kube/netclient-template.yaml

sed -i 's/ACCESS_TOKEN_VALUE/< your access token value>/g' netclient-template.yaml

microk8s kubectl apply -f netclient-template.yaml

This daemonset takes over management of the netclient and performs “check ins”.

If everything has gone well, you should see logs somewhat like the following:

root@do1:~# microk8s kubectl logs netclient-<id>
2021/07/13 17:11:16 attempting to join microk8s at grpc.nm.k8s.gravitl.com:443
2021/07/13 17:11:16 node created on remote server...updating configs
2021/07/13 17:11:16 retrieving remote peers
2021/07/13 17:11:16 starting wireguard
2021/07/13 17:11:16 joined microk8s
Checking into server at grpc.nm.k8s.gravitl.com:443
Checking to see if public addresses have changed
Local Address has changed from  to 210.97.150.30
Updating address
2021/07/13 17:11:16 using SSL
Authenticating with GRPC Server
Authenticated
Checking In.
Checked in.

The node should also now be visible in the UI.

Nodes 2 through X

For all subsequent nodes our task is straightforward. Run through these steps on each node (one at a time, not in parallel), and be patient: you don’t want to rush through the steps before any previous steps have had time to finish processing.

0. Change the Hostname

On the joining node, run:

hostnamectl set-hostname <nodename>.microk8s

Join the Network

Use the same commands and key you used on the seed node to install the netclient and join the network:

wget https://github.com/gravitl/netmaker/releases/download/v0.5.11/netclient && chmod +x netclient

./netclient join -t <YOUR_TOKEN> --daemon off --dns off --name $(hostname | sed -e s/.microk8s//)

Confirm the node has joined the network with wg show:

root@datacenter2:~# wg show
interface: nm-microk8s
  public key: 2xUDmCohypHcCD5dZukhhA8r6BGWN879J8vIhrcwSHg=
  private key: (hidden)
  listening port: 51821

peer: lrZkcSzWdgasgegaimEYnrr5CgopcEAIP8m3Q1M7+hiM=
  endpoint: 192.168.88.151:51821
  allowed ips: 10.101.0.3/32
  latest handshake: 41 seconds ago
  transfer: 736 B received, 2.53 KiB sent
  persistent keepalive: every 20 seconds

peer: IUobp84wipq44aFGP0SLuRhdSsDWvcxvBFefeRCE=
  endpoint: 210.97.150.30:51821
  allowed ips: 10.101.0.1/32
  latest handshake: 57 seconds ago
  transfer: 128.45 MiB received, 9.03 MiB sent
  persistent keepalive: every 20 seconds

2. Generate Join command

On the “seed” node, run **microk8s add-node. **Copy the command containing the WireGuard IP Address created by Netmaker:

root@do1:~# microk8s add-node
From the node you wish to join to this cluster, run the following:
microk8s join 209.97.147.27:25000/14e3a77f1584cb42323f39ce8ece0852/be5e4c7be0c6

If the node you are adding is not reachable through the default interface you can use one of the following:**
 microk8s join 210.97.150.27:25000/14e3a77f1584bc42323f39ce8ece0852/be5e4c7eb0c
microk8s join 10.17.0.5:25000/14e3a77f1584bc42323f39ce8ece0852/be5e4c7eb0c6
 microk8s join 10.108.0.2:25000/14e3a77f1584bc42323f39ce8ece0852/be5e4c7eb0c6
microk8s join 10.101.0.1:25000/14e3a77f1584bc42323f39ce8ece0852/be5e4c7eb0c6

3. Join the Cluster

On the joining node:

microk8s join 10.101.0.1:25000/14e3a77f1584bc42323f39ce8ece0852/be5e4c7eb0c6

Wait for the node to join the network. Here are a few commands to run that will help you determine if the node is healthy:

microk8s kubectl get nodes: should show node in Ready state

microk8s kubectl get pods -A: all pods should be running

microk8s logs netclient-: get the logs of the netclient on this node

Repeat steps 0 through 3 for each node you are adding to the cluster. Be patient. Wait for each node to join the network and cluster before moving on to the next step/machine.

At the end of this process, your cluster and your Netmaker instance should look similar to this:

root@do1:~/kube# microk8s kubectl get nodes -o wide
NAME                   STATUS    VERSION   INTERNAL-IP   EXTERNAL-IP 
do2.microk8s           Ready     v1.21.1-3+ba   10.101.0.2   <none>        
datacenter1.microk8s   Ready     v1.21.1-3+ba   10.101.0.3    <none>        
do1.microk8s           Ready     v1.21.1-3+ba   10.101.0.1    <none>        
datacenter2.microk8s   Ready     v1.21.1-3+ba   10.101.0.4    <none>

Congratulations! You have a cross-cloud Kubernetes cluster, one that will transfer workloads if an environment fails.

Testing for DR

now that we’ve got our cluster set up, we can test out a DR scenario to see how it plays out. Let’s setup an application that runs in the data center. First, add some node labels so we know which node is which:

microk8s kubectl label nodes do1.microk8s do2.microk8s location=cloud

microk8s kubectl label nodes datacenter1.microk8s datacenter2.microk8s location=onprem

Now, we’ll deploy an Nginx application which lives in our data center:

wget https://raw.githubusercontent.com/gravitl/netmaker/develop/kube/example/nginx-example.yaml

#BASE_DOMAIN should be your wildcard, ex: app.example.com

#template will add a subdomain, ex: nginx.app.example.com

sed -i 's/BASE_DOMAIN/<your base domain>/g' nginx-example.yaml

microk8s kubectl apply -f nginx-example.yaml

Run a “get pods” to see that all instances are running in the data center. This is due to the “node affinity” label in the deployment. It has an affinity for nodes with the label location=onprem.

root@do1:~# k get po -o wide | grep nginx
nginx-deployment-cb796dbc7-h72s8    1/1     Running   0          2m53s   10.1.99.68       datacenter1.microk8s   <none>           <none>
nginx-deployment-cb796dbc7-p5bhr    1/1     Running   0          2m53s   10.1.99.67       datacenter1.microk8s   <none>           <none>
nginx-deployment-cb796dbc7-pxpvw    1/1     Running   0          2m53s   10.1.247.3       datacenter2.microk8s   <none>           <none>
nginx-deployment-cb796dbc7-7vbwz    1/1     Running   0          2m53s   10.1.247.4       datacenter2.microk8s   <none>           <none>
nginx-deployment-cb796dbc7-x862w    1/1     Running   0          2m53s   10.1.247.5       datacenter2.microk8s   <none>           <none>

Going to the domain of the ingress, you should see the Nginx welcome screen:

Simulating Failure

Simulating failure is pretty easy in this scenario. Let’s just turn off the data center nodes:

root@datacenter2:~# microk8s stop

root@datacenter1:~# microk8s stop

It will take a little while for the cluster to realize the nodes are missing. By default, Kubernetes will wait 5 minutes after a node is in “NotReady” state before it begins to reschedule the pods.

For scenarios where uptime is ultra-critical, you can change the parameters to make this happen much faster.

Check in on the node status:

root@do2:~# k get nodes
NAME                   STATUS     ROLES    AGE    VERSION
do2.microk8s           Ready      <none>   77m    v1.21.1-3+ba118484dd39df
do1.microk8s           Ready      <none>   106m   v1.21.1-3+ba118484dd39df
datacenter1.microk8s   NotReady   <none>   62m    v1.21.1-3+ba118484dd39df
datacenter2.microk8s   NotReady   <none>   40m    v1.21.1-3+ba118484dd39df

Eventually, you should see the old pods terminating and new pods scheduling on the cloud nodes:

And just like before, our webpage is intact:

So, without multiple clusters or custom automation, we successfully set up a single cluster that will handle DR for us!

Conclusion

What did we learn here?

Scenarios such as DR historically required a multi-cluster deployment
A multi-cluster model is not absolutely necessary
You can enable multi-cluster patterns with a single cluster
Enabling these patterns requires tools like MicroK8s and Netmaker

There are many other use cases this approach enables. For instance, you can burst an application into the cloud, deploy nodes to the edge, or access resources in a cloud environment using a single node. We lay out some of those patterns here.

There are also many pitfalls to consider when deploying this sort of system. The networking can get complicated, and we did not cover what can go wrong, how to fix it, or how to optimize this system.

If you are interested in learning more, check out some of the resources below, or email info@gravitl.com.

Resources

gravitl/netmaker

MicroK8s

https://gravitl.com/netmaker

https://www.wireguard.com