Rahul Kiran Gaddam

Posted on Sep 22, 2022

Netmaker - Multi Cloud Networking

#netmaker #wiregaurd #cloud #multicloud

Philosophy

Now Multi cloud is the new reality, because not everything is available with every cloud provider.
With this being new reality, we need a mechanism to connect cloud environments and create a internal and secured network.

Overview

Here was my last post on installing kubernetes using Oracle OCI cloud.
Instead of creating a single node, can we create a cluster? can we create this cluster between multiple regions? how will they communicate?

Network Communication Is a PROBLEM >>>>

Answer for all these is VPN or we can try Netmaker.io

Installation

Netmaker works on Client - Server architecture like any other network connectivity resolution applications.
We will have a Netmaker Server installed on a server and Netmaker Agent/Client installed on the others.
Below installation is done on OCI Arm64 - Oracle Linux, which could not be achieved by following steps here
Create 3 Node where 2 Node will act as Agents/Clients and 1 Server
- All nodes should have Public IP
- Allow firewall for Ports 443 (tcp): for Traefik & 51821-518XX (udp): for WireGuard
- Register below domains freenom
  - base-domain.extenssion
  - broker-base-domain.extenssion
  - api-base-domain.extenssion
  - dashboard-base-domain.extenssion

Server Installation

I created below folders
Host

#!/bin/bash

PUBLIC_IP=$(curl -s ifconfig.me)
PRIVATE_IP=$(hostname -I | cut -f 1 -d " ")
DOMAIN_NAME="base-domain.extenssion"
MATCHES_IN_HOSTS="$(grep -n $PUBLIC_IP /etc/hosts)"

hostnamectl set-hostname $DOMAIN_NAME

if [ ! -z "$MATCHES_IN_HOSTS" ] 
then
    echo "Host Details Already Exisits"
else 
    echo "Registering Host Details";
    echo "===================================";
    cat /dev/null > /etc/hosts;
    echo "127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4" >> /etc/hosts;
    echo "::1         localhost localhost.localdomain localhost6 localhost6.localdomain6" >> /etc/hosts;
    echo "$PRIVATE_IP $PUBLIC_IP $DOMAIN_NAME" >> /etc/hosts;
    echo "========Updated Details============";
    cat /etc/hosts
    echo "===================================";
fi

Firewall

#!/bin/bash

# -- Disabling firewalld
systemctl disable firewalld

# -- Enabling iptables
yum install iptables-services -y
systemctl start iptables
systemctl enable iptables

# -- Flushing iptables
iptables -F

# -- Allowing everthing
iptables -A FORWARD -j ACCEPT
iptables -A INPUT -j ACCEPT
iptables -A OUTPUT -j ACCEPT

# -- Saving
service iptables save
systemctl restart iptables

# -- Display Settings
iptables -L -n

Docker

#!/bin/bash

# -- Enable kernel modules
modprobe overlay
modprobe br_netfilter
cat <<EOF |  tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF
cat <<EOF |  tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system

# -- Disabling Swap Memory
swapoff -a
sed -i '/ swap / s/^/#/' /etc/fstab
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

## Podman is by default provided, K8 can run on Podman
## I was unable to install using Podman and need to move to docker
# -- Remove Podman
yum remove podman buildah  -y

# -- Install Docker
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install -y docker-ce docker-ce-cli containerd.io

# -- Configure Docker
systemctl  stop docker
/usr/sbin/usermod -a -G docker opc
/usr/sbin/sysctl net.ipv4.conf.all.forwarding=1
systemctl  start docker
chmod 777 /var/run/docker.sock
tee /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

# -- Start and enable Services
sudo systemctl daemon-reload 
sudo systemctl restart docker
sudo systemctl enable docker

Netmaker

# docker-compose.yml
version: "3.4"

services:
  netmaker:
    container_name: netmaker
    image: gravitl/netmaker:v0.15.2@sha256:1e4cb5ca0907eea83eb84b850fe5e242e481dd4c1be59b60f96d5e577c67f5a9
    cap_add: 
      - NET_ADMIN
      - NET_RAW
      - SYS_MODULE
    sysctls:
      - net.ipv4.ip_forward=1
      - net.ipv4.conf.all.src_valid_mark=1
      - net.ipv6.conf.all.disable_ipv6=0
      - net.ipv6.conf.all.forwarding=1
    restart: always
    volumes:
      - /root/dnsconfig:/root/config/dnsconfig
      - /root/sqldata:/root/data
      - /root/shared_certs:/etc/netmaker
    environment:
      SERVER_NAME: "broker-base-domain.extenssion"
      SERVER_HOST: "SERVER_PUBLIC_IP"
      SERVER_API_CONN_STRING: "api-base-domain.extenssion:443"
      COREDNS_ADDR: "SERVER_PUBLIC_IP"
      DNS_MODE: "on"
      SERVER_HTTP_HOST: "api-base-domain.extenssion"
      API_PORT: "8081"
      CLIENT_MODE: "on"
      MASTER_KEY: "REPLACE_MASTER_KEY"
      CORS_ALLOWED_ORIGIN: "*"
      DISPLAY_KEYS: "on"
      DATABASE: "sqlite"
      NODE_ID: "netmaker-server-1"
      MQ_HOST: "mq"
      MQ_PORT: "443"
      MQ_SERVER_PORT: "1883"
      HOST_NETWORK: "off"
      VERBOSITY: "1"
      MANAGE_IPTABLES: "on"
      PORT_FORWARD_SERVICES: "dns"
    ports:
      - "51821-51830:51821-51830/udp"
    expose:
      - "8081"
    labels:
      - traefik.enable=true
      - traefik.http.routers.netmaker-api.entrypoints=websecure
      - traefik.http.routers.netmaker-api.rule=Host(`api-base-domain.extenssion`)
      - traefik.http.routers.netmaker-api.service=netmaker-api
      - traefik.http.services.netmaker-api.loadbalancer.server.port=8081
  netmaker-ui:
    container_name: netmaker-ui
    image: gravitl/netmaker-ui:v0.15.2@sha256:11fe0092e8a8e8a7a6a07e6aa50d448ed2de24ee6d2eb045e3956b3c6c24af50
    depends_on:
      - netmaker
    links:
      - "netmaker:api"
    restart: always
    environment:
      BACKEND_URL: "https://api-base-domain.extenssion"
    expose:
      - "80"
    labels:
      - traefik.enable=true
      - traefik.http.middlewares.nmui-security.headers.accessControlAllowOriginList=*-base-domain.extenssion
      - traefik.http.middlewares.nmui-security.headers.stsSeconds=31536000
      - traefik.http.middlewares.nmui-security.headers.browserXssFilter=true
      - traefik.http.middlewares.nmui-security.headers.customFrameOptionsValue=SAMEORIGIN
      - traefik.http.middlewares.nmui-security.headers.customResponseHeaders.X-Robots-Tag=none
      - traefik.http.middlewares.nmui-security.headers.customResponseHeaders.Server= # Remove the server name
      - traefik.http.routers.netmaker-ui.entrypoints=websecure
      - traefik.http.routers.netmaker-ui.middlewares=nmui-security@docker
      - traefik.http.routers.netmaker-ui.rule=Host(`dashboard-base-domain.extenssion`)
      - traefik.http.routers.netmaker-ui.service=netmaker-ui
      - traefik.http.services.netmaker-ui.loadbalancer.server.port=80
  coredns:
    container_name: coredns
    image: coredns/coredns-arm64@sha256:224c4ecc9d9eea3765d0beee0e624e6cf837230c370440bd38a7d9901dd04dc4
    command: -conf /root/dnsconfig/Corefile
    depends_on:
      - netmaker
    restart: always
    volumes:
      - /root/dnsconfig:/root/dnsconfig
  traefik:
    image: traefik:v2.6@sha256:9aecceb73e3b24b6547d401c95eea6cdf475a99ddfd0b86464c5413925e062da
    container_name: traefik
    command:
      - "--certificatesresolvers.http.acme.email=YOUR_EMAIL"
      - "--certificatesresolvers.http.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.http.acme.tlschallenge=true"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.websecure.http.tls=true"
      - "--entrypoints.websecure.http.tls.certResolver=http"
      - "--log.level=INFO"
      - "--providers.docker=true"
      - "--providers.docker.exposedByDefault=false"
      - "--serverstransport.insecureskipverify=true"
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /root/traefik_certs:/letsencrypt
    ports:
      - "443:443"
  mq:
    container_name: mq
    image: eclipse-mosquitto:2.0.11-openssl@sha256:459f8503a3a248156855501a8fa718b92783bf02fc5b1ea414fae07ca1d1396d
    depends_on:
      - netmaker
    restart: unless-stopped
    volumes:
      - /root/mosquitto.conf:/mosquitto/config/mosquitto.conf
      - /root/mosquitto_data:/mosquitto/data
      - /root/mosquitto_logs:/mosquitto/log
      - /root/shared_certs:/mosquitto/certs
    expose:
      - "8883"
    labels:
      - traefik.enable=true
      - traefik.tcp.routers.mqtts.rule=HostSNI(`broker-base-domain.extenssion`)
      - traefik.tcp.routers.mqtts.tls.passthrough=true
      - traefik.tcp.services.mqtts-svc.loadbalancer.server.port=8883
      - traefik.tcp.routers.mqtts.service=mqtts-svc
      - traefik.tcp.routers.mqtts.entrypoints=websecure

# Create Corefile by removing this comment
coredns.io {
 log stdout
 file /root/dnsconfig/coredns.dat
}

#!/bin/bash

# -- Prerequists
# yum install -y wireguard-tools net-tools jq
# modprobe ip_tables
# echo 'ip_tables' >> /etc/modules

# -- Install Docker Compose
# curl -L --fail https://github.com/docker/compose/releases/download/v2.11.0/docker-compose-linux-aarch64 -o /usr/sbin/docker-compose
# chmod +x /usr/sbin/docker-compose

# -- Setting env variables
NETMAKER_BASE_DOMAIN=$DOMAIN_NAME
COREDNS_IP=$(ip route get 1 | sed -n 's/^.*src \([0-9.]*\) .*$/\1/p')
SERVER_PUBLIC_IP=$(curl -s ifconfig.me)
MASTER_KEY=$(tr -dc A-Za-z0-9 </dev/urandom | head -c 30 ; echo '')
EMAIL="gaddam.rahul@email.com"
MESH_SETUP="true"
VPN_SETUP="false"
NUM_CLIENTS=5

echo "   ----------------------------"
echo "                SETUP ARGUMENTS"
echo "   ----------------------------"
echo "        domain: $NETMAKER_BASE_DOMAIN"
echo "         email: $EMAIL"
echo "    coredns ip: $COREDNS_IP"
echo "     public ip: $SERVER_PUBLIC_IP"
echo "    master key: $MASTER_KEY"
echo "   setup mesh?: $MESH_SETUP"
echo "    setup vpn?: $VPN_SETUP"
if [ "${VPN_SETUP}" == "true" ]; then
echo "     # clients: $NUM_CLIENTS"
fi
echo "   ----------------------------"

sleep 5

# -- Installation
echo "setting mosquitto.conf..."
wget -q -O /root/mosquitto.conf https://raw.githubusercontent.com/gravitl/netmaker/master/docker/mosquitto.conf

echo "setting docker-compose..."
mkdir -p /root/dnsconfig
cp /home/opc/setup/netmaker/Corefile /root/dnsconfig/
mkdir -p /root/traefik_certs
mkdir -p /root/shared_certs
mkdir -p /root/sqldata
mkdir -p /root/mosquitto_data
mkdir -p /root/mosquitto_logs

cp /home/opc/setup/netmaker/docker-compose.yml /root/docker-compose.yml
sed -i "s/SERVER_PUBLIC_IP/$SERVER_PUBLIC_IP/g" /root/docker-compose.yml
sed -i "s/COREDNS_IP/$COREDNS_IP/g" /root/docker-compose.yml
sed -i "s/REPLACE_MASTER_KEY/$MASTER_KEY/g" /root/docker-compose.yml
sed -i "s/YOUR_EMAIL/$EMAIL/g" /root/docker-compose.yml

echo "starting containers..."
docker-compose -f /root/docker-compose.yml up -d

test_connection() {

echo "testing Traefik setup (please be patient, this may take 1-2 minutes)"
for i in 1 2 3 4 5 6
do
curlresponse=$(curl -vIs https://api-base-domain.extenssion 2>&1)

if [[ "$i" == 6 ]]; then
  echo "    Traefik is having an issue setting up certificates, please investigate (docker logs traefik)"
  echo "    exiting..."
  exit 1
elif [[ "$curlresponse" == *"failed to verify the legitimacy of the server"* ]]; then
  echo "    certificates not yet configured, retrying..."

elif [[ "$curlresponse" == *"left intact"* ]]; then
  echo "    certificates ok"
  break
else
  secs=$(($i*5+10))
  echo "    issue establishing connection...retrying in $secs seconds..."       
fi
sleep $secs
done
}

set +e
test_connection


cat << "EOF"


 __   __     ______     ______   __    __     ______     __  __     ______     ______    
/\ "-.\ \   /\  ___\   /\__  _\ /\ "-./  \   /\  __ \   /\ \/ /    /\  ___\   /\  == \   
\ \ \-.  \  \ \  __\   \/_/\ \/ \ \ \-./\ \  \ \  __ \  \ \  _"-.  \ \  __\   \ \  __<   
 \ \_\\"\_\  \ \_____\    \ \_\  \ \_\ \ \_\  \ \_\ \_\  \ \_\ \_\  \ \_____\  \ \_\ \_\ 
  \/_/ \/_/   \/_____/     \/_/   \/_/  \/_/   \/_/\/_/   \/_/\/_/   \/_____/   \/_/ /_/ 


EOF


echo "visit https://dashboard-base-domain.extenssion to log in"
sleep 7

setup_mesh() {( set -e
echo "creating netmaker network (10.101.0.0/16)"

curl -s -o /dev/null -d '{"addressrange":"10.101.0.0/16","netid":"netmaker"}' -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' https://api-base-domain.extenssion/api/networks

sleep 5

echo "creating netmaker access key"

curlresponse=$(curl -s -d '{"uses":99999,"name":"netmaker-key"}' -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' https://api-base-domain.extenssion/api/networks/netmaker/keys)
ACCESS_TOKEN=$(jq -r '.accessstring' <<< ${curlresponse})

sleep 5

echo "configuring netmaker server as ingress gateway"

curlresponse=$(curl -s -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' https://api-base-domain.extenssion/api/nodes/netmaker)
SERVER_ID=$(jq -r '.[0].id' <<< ${curlresponse})

curl -o /dev/null -s -X POST -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' https://api-base-domain.extenssion/api/nodes/netmaker/$SERVER_ID/createingress

sleep 5

echo "finished configuring server and network. You can now add clients."
echo ""
echo "For Linux, Mac, Windows, and FreeBSD:"
echo "        1. Install the netclient: https://docs.netmaker.org/netclient.html#installation"
echo "        2. Join the network: netclient join -t $ACCESS_TOKEN"
echo ""
echo "For Android and iOS clients, perform the following steps:"
echo "        1. Log into UI at dashboard-base-domain.extenssion"
echo "        2. Navigate to \"EXTERNAL CLIENTS\" tab"
echo "        3. Select the gateway and create clients"
echo "        4. Scan the QR Code from WireGuard app in iOS or Android"
echo ""
echo "Netmaker setup is now complete. You are ready to begin using Netmaker."
)}

setup_vpn() {( set -e
echo "creating vpn network (10.201.0.0/16)"

curl -s -o /dev/null -d '{"addressrange":"10.201.0.0/16","netid":"vpn","defaultextclientdns":"10.201.255.254"}' -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' https://api-base-domain.extenssion/api/networks

sleep 5

echo "configuring netmaker server as vpn inlet..."

curlresponse=$(curl -s -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' https://api-base-domain.extenssion/api/nodes/vpn)
SERVER_ID=$(jq -r '.[0].id' <<< ${curlresponse})

curl -s -o /dev/null -X POST -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' https://api-base-domain.extenssion/api/nodes/vpn/$SERVER_ID/createingress

echo "waiting 5 seconds for server to apply configuration..."

sleep 5


echo "configuring netmaker server vpn gateway..."

[ -z "$GATEWAY_IFACE" ] && GATEWAY_IFACE=$(ip -4 route ls | grep default | grep -Po '(?<=dev )(\S+)')

echo "gateway iface: $GATEWAY_IFACE"

curlresponse=$(curl -s -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' https://api-base-domain.extenssion/api/nodes/vpn)
SERVER_ID=$(jq -r '.[0].id' <<< ${curlresponse})

EGRESS_JSON=$( jq -n \
                  --arg gw "$GATEWAY_IFACE" \
                  '{ranges: ["0.0.0.0/0"], interface: $gw}' )

echo "egress json: $EGRESS_JSON"
curl -s -o /dev/null -X POST -d "$EGRESS_JSON" -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' https://api-base-domain.extenssion/api/nodes/vpn/$SERVER_ID/creategateway

sleep 3

echo "creating client configs..."

for ((a=1; a <= $NUM_CLIENTS; a++))
do
        CLIENT_JSON=$( jq -n \
                  --arg clientid "vpnclient-$a" \
                  '{clientid: $clientid}' )

        curl -s -o /dev/null -d "$CLIENT_JSON" -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' https://api-base-domain.extenssion/api/extclients/vpn/$SERVER_ID
        sleep 2
done

echo "finished configuring vpn server."
echo ""
echo "To configure clients, perform the following steps:"
echo "        1. log into dashboard-base-domain.extenssion"
echo "        2. Navigate to \"EXTERNAL CLIENTS\" tab"
echo "        3. Download or scan a client config (vpnclient-x) to the appropriate device"
echo "        4. Follow the steps for your system to configure WireGuard on the appropriate device"
echo "        5. Create and delete clients as necessary. Changes to netmaker server settings require regenerating ext clients."

)}

if [ "${MESH_SETUP}" != "false" ]; then
        setup_mesh
fi

if [ "${VPN_SETUP}" == "true" ]; then
        setup_vpn
fi

echo ""
echo "Netmaker setup is now complete. You are ready to begin using Netmaker."

On successful completion, user will be requested to login to https://dashboard-base-domain.extenssion/

Agent Installation

Agent will have same files expect the Netmaker
Netmaker

#!/bin/bash

# -- Prerequists
yum install -y wireguard-tools net-tools jq
modprobe ip_tables
echo 'ip_tables' >> /etc/modules

# -- Install Netmaker Client
wget https://github.com/gravitl/netmaker/releases/download/v0.15.2/netclient-arm64 -O /usr/sbin/netclient
chmod +x /usr/sbin/netclient
netclient daemon &

# -- Joining Netmaker Master
netclient join -t <token>

Verification

On successful installation, User will be able to view new network on ifconfig
Ping to different systems will be accessible.
Dashboard will show all nodes connected

References

This site is built on Heroku

Join the ranks of developers at Salesforce, Airbase, DEV, and more who deploy their mission critical applications on Heroku. Sign up today and launch your first app!

Get Started

DEV Community

Netmaker - Multi Cloud Networking

Philosophy

Overview

Network Communication Is a PROBLEM >>>>

Installation

Server Installation

Agent Installation

Verification

References

This site is built on Heroku

Top comments (0)

See why 4M developers consider Sentry, “not bad.”

Read next

Exploring the Power and Community Behind Apache Flink

Exploring the Journey of Apache Cassandra: Innovation, Funding, and Community Impact

Exploring Spring Cloud: Innovation, Funding, and Community Empowerment

Robot Control Breakthrough: AI System Generates Safe, Diverse Navigation Paths Using Temporal Logic

Okay