Robert Wang

Posted on Apr 14 • Edited on Apr 22

Build k8s Cluster on WSL2

#docker #k8s #devops #wsl2

The main difference between WSL1 and WSL2 that later is hyper-v VM based Microsoft Linux version, it sounds like excited news at least it is running full Linux kernel but there are too many issues if using as devops box.

Here are some outstanding pain points:

default systemd is not enabled, which needed to install packages
default network is NAT, always assign dynamic ip after reboot, will give trouble for app like k8s api server

Lab env

win10 host 32GB RAM
WSL2 Ubuntu 22.04
Docker 25.0.4 on WSL2
k8s 1.29.3 on WSL2

Install WSL2 Ubuntu

Install WSL2 from CMD or PS on win host, following official doc,

https://docs.microsoft.com/en-us/windows/wsl/install

wsl -l -o
The following is a list of valid distributions that can be installed.
Install using 'wsl.exe --install <Distro>'.

NAME                                   FRIENDLY NAME
Ubuntu                                 Ubuntu
Debian                                 Debian GNU/Linux
kali-linux                             Kali Linux Rolling
Ubuntu-18.04                           Ubuntu 18.04 LTS
Ubuntu-20.04                           Ubuntu 20.04 LTS
Ubuntu-22.04                           Ubuntu 22.04 LTS
OracleLinux_7_9                        Oracle Linux 7.9
OracleLinux_8_7                        Oracle Linux 8.7
OracleLinux_9_1                        Oracle Linux 9.1
openSUSE-Leap-15.5                     openSUSE Leap 15.5
SUSE-Linux-Enterprise-Server-15-SP4    SUSE Linux Enterprise Server 15 SP4
SUSE-Linux-Enterprise-15-SP5           SUSE Linux Enterprise 15 SP5
openSUSE-Tumbleweed                    openSUSE Tumbleweed

wsl --install Ubuntu-22.04

Launch WSL2 instance from CMD or PS shell,

wsl -d Ubuntu-22.04

Launch WSL2 instance from SecureCRT client

create wsl2.bat under C:\tools, including below line

wsl -d Ubuntu-22.04

SecureCRT->Session->Connection/Protocol:'Local Shell', Connection->Local Shell/Shell path: C:\tools\wsl2.bat

WSL2 config

There are two kind of config for WSL2:

global config for all WSL2 instance on win host, C:\Users\oldhorse\.wslconfig

[wsl2]
memory=16GB
processors=4
swap=0

config per one WSL2 instance inside vm, /etc/wsl.conf

[boot]
systemd=true
[network]
hostname = wsl2
generateHosts = false
generateResolvConf = false

This will enable systemd, setup hostname and don't generate /etc/resolv.conf.

Remedy for "static" ip

After WSL2 instance launched, will see NAT ip as below,

oldhorse@wsl2:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc mq state UP group default qlen 1000
    link/ether 00:15:5d:ea:e8:22 brd ff:ff:ff:ff:ff:ff
    inet 172.30.73.22/20 brd 172.30.79.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::215:5dff:feea:e822/64 scope link 
       valid_lft forever preferred_lft forever

Assume we want to keep static ip 192.168.80.2 for WSL2 Ubuntu, then need to manually assign ip to WSL2 eth0 inside WSL2 VM, and recreate NAT network adaptor for WSL on win host end, it is in fact the network type of Host-only + NAT.

Run below batch script as Administrator on windows host, you can type wsl2ip.bat from CMD, PS or place shortcut icon on desktop.

wsl2ip.bat

date /t
time /t

wsl -d Ubuntu-22.04 -u root ip addr del $(ip addr show eth0 ^| grep 'inet\b' ^| awk '{print $2}' ^| head -n 1) dev eth0
wsl -d Ubuntu-22.04 -u root ip addr add 192.168.80.2/24 broadcast 192.168.80.255 dev eth0
wsl -d Ubuntu-22.04 -u root ip route add 0.0.0.0/0 via 192.168.80.1 dev eth0
wsl -d Ubuntu-22.04 -u root echo nameserver 8.8.8.8 ^> /etc/resolv.conf
wsl -d Ubuntu-22.04 -u root chattr +i /etc/resolv.conf

powershell -c "Get-NetAdapter 'vEthernet (WSL)' | Get-NetIPAddress | Remove-NetIPAddress -Confirm:$False; New-NetIPAddress -IPAddress 192.168.80.1 -PrefixLength 24 -InterfaceAlias 'vEthernet (WSL)'; Get-NetNat | ? Name -Eq WSLNat | Remove-NetNat -Confirm:$False; New-NetNat -Name WSLNat -InternalIPInterfaceAddressPrefix 192.168.80.0/24;"

date /t
time /t

pause

The default generated /etc/resolv.conf won't work for DNS for Internet access, so that is why we disable generateResolvConf in /etc/wsl.conf, add 8.8.8.8 to it, and changed it to read-only file.

After run above script, you can access from host to WSL2 due to it is host-only static ip now.

PS C:\Users\oldhorse> Get-NetAdapter 'vEthernet (WSL)' | Get-NetIPAddress

IPAddress         : 192.168.80.1
InterfaceIndex    : 24
InterfaceAlias    : vEthernet (WSL)
AddressFamily     : IPv4
Type              : Unicast
PrefixLength      : 24
PrefixOrigin      : Manual
SuffixOrigin      : Manual
AddressState      : Preferred
ValidLifetime     : Infinite ([TimeSpan]::MaxValue)
PreferredLifetime : Infinite ([TimeSpan]::MaxValue)
SkipAsSource      : False
PolicyStore       : ActiveStore

PS C:\Users\oldhorse> Get-NetNat 'WSLNat'

Name                             : WSLNat
ExternalIPInterfaceAddressPrefix :
InternalIPInterfaceAddressPrefix : 192.168.80.0/24
IcmpQueryTimeout                 : 30
TcpEstablishedConnectionTimeout  : 1800
TcpTransientConnectionTimeout    : 120
TcpFilteringBehavior             : AddressDependentFiltering
UdpFilteringBehavior             : AddressDependentFiltering
UdpIdleSessionTimeout            : 120
UdpInboundRefresh                : False
Store                            : Local
Active                           : True

PS C:\Users\oldhorse> ipconfig

Ethernet adapter vEthernet (WSL):

   Connection-specific DNS Suffix  . :
   IPv4 Address. . . . . . . . . . . : 192.168.80.1
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :

PS C:\Users\oldhorse> ping 192.168.80.2

Pinging 192.168.80.2 with 32 bytes of data:
Reply from 192.168.80.2: bytes=32 time=1ms TTL=64
Reply from 192.168.80.2: bytes=32 time=1ms TTL=64

also you can access from WSL2 to host, Internet like google as well,

oldhorse@wsl2:~$ ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc mq state UP group default qlen 1000
    link/ether 00:15:5d:ea:e8:22 brd ff:ff:ff:ff:ff:ff
    inet 192.168.80.2/24 brd 192.168.80.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::215:5dff:feea:e822/64 scope link
       valid_lft forever preferred_lft forever

oldhorse@wsl2:~$ ip r
default via 192.168.80.1 dev eth0 
192.168.80.0/24 dev eth0 proto kernel scope link src 192.168.80.2

oldhorse@wsl2:~$ ping google.ca
PING google.ca (142.251.40.99) 56(84) bytes of data.
64 bytes from lga25s79-in-f3.1e100.net (142.251.40.99): icmp_seq=1 ttl=105 time=69.4 ms
64 bytes from lga25s79-in-f3.1e100.net (142.251.40.99): icmp_seq=2 ttl=105 time=60.5 ms

oldhorse@wsl2:~$ cat /etc/resolv.conf 
nameserver 8.8.8.8

Common issue for WSL2 NAT

Since we changed WSL2 NAT manually, sometimes it hanging on wsl cli, as remedy run below reset batch file as admin from win host.
wslreset.bat

taskkill /f /im wslservice.exe

wsl -l 

pause

From now on the WSL2 with static ip and Internet access ready for k8s installation.

k8s installation automation

It is possible to install k8s installation in one shot using my handy script toolkit.
First of all, download it via git clone, or manually download from github.

git clone git@github.com:robertluwang/hands-on-nativecloud.git
cd ./hands-on-nativecloud/src/k8s-cri-dockerd

Here is complete step to install a single node k8s cluster on WSL2.

bash docker-server.sh

exit and enter to WSL2 instance, to make sure non-root user to access docker.

bash cri-dockerd.sh
bash k8s-install.sh
bash k8s-init.sh 192.168.80.2

Next I will explain more details for each step as below.

Linux native Docker server on WSL2

The systemd is ready so just following the docker doc to install docker on Ubuntu,

https://docs.docker.com/engine/install/

I make it as handy script, docker-install.sh.

docker-server.sh

# docker-server.sh
# handy script to install docker on ubuntu 
# run on k8s cluster node (master/worker)
# By Robert Wang @github.com/robertluwang
# Nov 21, 2022

echo === $(date) Provisioning - docker-server.sh by $(whoami) start

sudo apt-get update -y
sudo apt-get install -y ca-certificates curl gnupg lsb-release
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update -y
sudo apt-get install -y docker-ce docker-ce-cli containerd.io 

sudo groupadd docker
sudo usermod -aG docker $USER

# turn off swap
sudo swapoff -a
sudo sed -i '/swap/d' /etc/fstab

sudo mkdir /etc/docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF
sudo systemctl enable docker
sudo systemctl daemon-reload
sudo systemctl restart docker
sleep 30 
sudo systemctl restart docker

echo === $(date) Provisioning - docker-server.sh by $(whoami) end

Let's run it,

bash docker-server.sh

re-login to WSL2, docker working for non-root user, in my case it is oldhorse.

docker info 
docker ps

verify docker daemon,

oldhorse@wsl2:~$ systemctl status docker
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2024-04-14 12:07:34 EDT; 1h 10min ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 253 (dockerd)
      Tasks: 104
     Memory: 208.5M
     CGroup: /system.slice/docker.service
             ├─253 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

docker bridge docker0 ready,

oldhorse@wsl2:~$ ip addr show docker0
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:a7:dc:0d:c9 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

cri-dockerd setup for k8s + docker on WSL2

As we known the docker support removed since k8s 1.24+, so need cri-dockerd installed to have cri interface with Docker Engine API for k8s cluster. More details please see https://github.com/Mirantis/cri-dockerd.

You might wonder why we want to still keep docker for k8s setup, because I would like get benefit from pure docker test and k8s test in same local lab.

I also make handy script for setup - cri-dockerd.sh

cri-dockerd.sh

# cri-dockerd.sh
# handy script to install cri-dockerd on ubuntu 
# run on k8s cluster node (master/worker)
# By Robert Wang @github.com/robertluwang
# Nov 26, 2022

echo === $(date) Provisioning - cri-dockerd by $(whoami) start

# cri-dockerd 
VER=$(curl -s https://api.github.com/repos/Mirantis/cri-dockerd/releases/latest|grep tag_name | cut -d '"' -f 4|sed 's/v//g')
wget https://github.com/Mirantis/cri-dockerd/releases/download/v${VER}/cri-dockerd-${VER}.amd64.tgz
tar xvf cri-dockerd-${VER}.amd64.tgz
sudo mv cri-dockerd/cri-dockerd /usr/local/bin/
sudo chmod +x /usr/local/bin/cri-dockerd

wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.service
wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.socket

sudo mv cri-docker.socket cri-docker.service /etc/systemd/system/
sudo chown root:root /etc/systemd/system/cri-docker.*
sudo sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service

sudo systemctl daemon-reload
sudo systemctl enable cri-docker.service
sudo systemctl enable --now cri-docker.socket
sleep 30
sudo systemctl restart cri-docker.service

echo === $(date) Provisioning - cri-dockerd by $(whoami) end

Let's run it,

bash cri-dockerd.sh

Verify it is active running,

oldhorse@wsl2:~$ systemctl status cri-docker
● cri-docker.service - CRI Interface for Docker Application Container Engine
     Loaded: loaded (/etc/systemd/system/cri-docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2024-04-14 12:07:35 EDT; 1h 25min ago
TriggeredBy: ● cri-docker.socket
       Docs: https://docs.mirantis.com
   Main PID: 880 (cri-dockerd)
      Tasks: 17
     Memory: 100.8M
     CGroup: /system.slice/cri-docker.service
             └─880 /usr/local/bin/cri-dockerd --container-runtime-endpoint fd://

k8s packages installation on WSL2

It is time to install k8s following up official k8s doc.

Install k8s packages using below handy script k8s-install.sh,

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

k8s-install.sh

# k8s-install.sh
# handy script to install k8s on ubuntu 
# run on all k8s cluster node (master/worker)
# By Robert Wang @github.com/robertluwang
# Oct 30, 2021

echo === $(date) Provisioning - k8s-install.sh by $(whoami) start

sudo apt-get update -y
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
sudo curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update -y
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

# cli completion
sudo apt-get install bash-completion -y
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc
echo 'alias k=kubectl' >>~/.bashrc
echo 'complete -F __start_kubectl k' >>~/.bashrc
echo "export do='--dry-run=client -o yaml'" >>~/.bashrc

echo === $(date) Provisioning - k8s-install.sh by $(whoami) end

Let's run it,

bash k8s-install.sh

Verify installed package version,

oldhorse@wsl2:~/dev$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"29", GitVersion:"v1.29.3", GitCommit:"6813625b7cd706db5bc7388921be03071e1a492d", GitTreeState:"clean", BuildDate:"2024-03-15T00:06:16Z", GoVersion:"
go1.21.8", Compiler:"gc", Platform:"linux/amd64"}
o
ldhorse@wsl2:~/dev$ kubectl version 
Client Version: v1.29.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.3

oldhorse@wsl2:~/dev$ kubelet --version
Kubernetes v1.29.3

k8s cluster init on WSL2

Run handy script k8s-init.sh for first time setup k8s cluster on WSL2,

k8s-init.sh

# k8s-init.sh
# handy script to init k8s cluster on ubuntu 
# run on k8s master node only
# By Robert Wang @github.com/robertluwang
# Nov 21, 2022
# $1 - master/api server ip

echo === $(date) Provisioning - k8s-init.sh $1 by $(whoami) start

if [ -z "$1" ];then
    sudo kubeadm init --pod-network-cidr=192.168.0.0/16 --ignore-preflight-errors=NumCPU --ignore-preflight-errors=Mem --cri-socket unix:///var/run/cri-dockerd.sock | tee /var/tmp/kubeadm.log
else
    sudo kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=$1 --ignore-preflight-errors=NumCPU --ignore-preflight-errors=Mem --cri-socket unix:///var/run/cri-dockerd.sock | tee /var/tmp/kubeadm.log
fi

# allow normal user to run kubectl
if [ -d $HOME/.kube ]; then
  rm -r $HOME/.kube
fi
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# install calico network addon
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.24.5/manifests/tigera-operator.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.24.5/manifests/custom-resources.yaml

# allow run on master
kubectl taint nodes --all node-role.kubernetes.io/control-plane-

echo === $(date) Provisioning - k8s-init.sh $1 by $(whoami) end

We need to give eth0 static ip 192.168.80.2 as parameter to k8s-init.sh, let the k8s api server pointing to eth0 ip.

bash k8s-init.sh 192.168.80.2

k8s cluster reset on WSL2

Run handy script k8s-reset.sh to quickly rebuild k8s cluster on WSL2 if anything mess up,

k8s-reset.sh

# k8s-reset.sh
# handy script to reset k8s cluster on ubuntu 
# run on k8s cluster node (master/worker)
# By Robert Wang @github.com/robertluwang
# Nov 21, 2022

echo === $(date) Provisioning - k8s-reset.sh by $(whoami) start

sudo kubeadm reset -f --cri-socket unix:///var/run/cri-dockerd.sock 

echo === $(date) Provisioning - k8s-reset.sh by $(whoami) end

For example,

bash k8s-reset.sh

then following k8s-init.sh with eth0 static ip,

bash k8s-init.sh 192.168.80.2

k8s cluster test on WSL2

oldhorse@wsl2:~$ k get node
NAME   STATUS   ROLES           AGE    VERSION
wsl2   Ready    control-plane   7d3h   v1.29.3
oldhorse@wsl2:~$ k get pod -A
NAMESPACE          NAME                                       READY   STATUS    RESTARTS        AGE
calico-apiserver   calico-apiserver-99d8c8bc6-5csh6           1/1     Running   9 (104m ago)    7d3h
calico-apiserver   calico-apiserver-99d8c8bc6-lgdh4           1/1     Running   9 (104m ago)    7d3h
calico-system      calico-kube-controllers-78788579b8-jdhj4   1/1     Running   9 (104m ago)    7d3h
calico-system      calico-node-vzhwj                          1/1     Running   9 (104m ago)    7d3h
calico-system      calico-typha-7647b5ff5b-dkxks              1/1     Running   15 (40m ago)    7d3h
kube-system        coredns-76f75df574-4dg8d                   1/1     Running   9 (104m ago)    7d3h
kube-system        coredns-76f75df574-5cmgf                   1/1     Running   9 (104m ago)    7d3h
kube-system        etcd-wsl2                                  1/1     Running   32 (46m ago)    7d3h
kube-system        kube-apiserver-wsl2                        1/1     Running   30 (46m ago)    7d3h
kube-system        kube-controller-manager-wsl2               1/1     Running   10 (104m ago)   7d3h
kube-system        kube-proxy-x7mxt                           1/1     Running   9 (104m ago)    7d3h
kube-system        kube-scheduler-wsl2                        1/1     Running   10 (104m ago)   7d3h
tigera-operator    tigera-operator-6fbc4f6f8d-d5fgp           1/1     Running   16 (40m ago)    7d3h

Do dry run on new pod,

oldhorse@wsl2:~$ k run test --image=nginx $do
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: test
  name: test
spec:
  containers:
  - image: nginx
    name: test
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

Let's launch test pod if everything looks good,

oldhorse@wsl2:~$ k run test --image=nginx 
pod/test created
oldhorse@wsl2:~$ k get pod 
NAME   READY   STATUS              RESTARTS   AGE
test   0/1     ContainerCreating   0          

oldhorse@wsl2:~$ k get pod -o wide
NAME   READY   STATUS    RESTARTS   AGE   IP              NODE   NOMINATED NODE   READINESS GATES
test   1/1     Running   0          48s   192.168.9.254   wsl2   <none>           <none>

oldhorse@wsl2:~$ curl 192.168.9.254
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Conclusion

So far the one single node k8s cluster lab is ready on WSL2:

k8s cluster has static ip, can be accessed from inside and outside of WSL2
it is docker based so same lab env for any docker related test
it is ready for local LLM AI test on both docker and k8s

About Me

Hey! I am Robert Wang, live in Montreal.

More simple and more efficient.

GitHub: robertluwang
Twitter: robertluwang
LinkedIn: robertluwang
Medium: robertluwang
Dev.to: robertluwang
Web: dreamcloud.artark.ca

DEV Community

Build k8s Cluster on WSL2

Lab env

Install WSL2 Ubuntu

WSL2 config

Remedy for "static" ip

Common issue for WSL2 NAT

k8s installation automation

Linux native Docker server on WSL2

cri-dockerd setup for k8s + docker on WSL2

k8s packages installation on WSL2

k8s cluster init on WSL2

k8s cluster reset on WSL2

k8s cluster test on WSL2

Conclusion

About Me

Top comments (0)

Read next

Authorizing endpoints of external apps in k8s

Deploy and Manage GitLab Runners on Amazon EC2

My First Full-Stack Deployment with Docker and NGINX as Load Balancer

Why Monorepo Projects Sucks: Performance Considerations with Nx