The latest articles on DEV Community by upinder sujlana (@upindersujlana). https://dev.to/upindersujlana https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F325346%2F4134bf8a-a36a-4db4-884d-1197370cf8c4.png https://dev.to/upindersujlana en upinder sujlana Sat, 11 Dec 2021 21:07:24 +0000 https://dev.to/upindersujlana/k8s-baremetal-loadbalancer-and-ingress-setup-and-usage-demo-294n https://dev.to/upindersujlana/k8s-baremetal-loadbalancer-and-ingress-setup-and-usage-demo-294n <p>Some notes related to my testing loadbalancer, ingress &amp; ingress controller (nginx) in my baremetal kubernetes cluster.</p> <p><a href="https://github.com/upinder-sujlana/k8s-baremetal-loadbancer-ingress-demo/tree/main">https://github.com/upinder-sujlana/k8s-baremetal-loadbancer-ingress-demo/tree/main</a></p> kubernetes nginx metallb upinder sujlana Fri, 22 Oct 2021 20:26:53 +0000 https://dev.to/upindersujlana/prometheus-and-grafana-install-on-a-kubernetes-cluster-using-helm-37b7 https://dev.to/upindersujlana/prometheus-and-grafana-install-on-a-kubernetes-cluster-using-helm-37b7 <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>Below are some quick notes on how I setup helm, prometheus and grafana on a Kubernetes cluster using helm. [+] Have a K8S cluster already. kmaster2@kmaster2:~$ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kmaster2 Ready control-plane,master 9d v1.22.2 192.168.1.86 &lt;none&gt; Ubuntu 18.04 LTS 4.15.0-161-generic docker://20.10.9 knode3 Ready &lt;none&gt; 9d v1.22.2 192.168.1.87 &lt;none&gt; Ubuntu 18.04 LTS 4.15.0-161-generic docker://20.10.9 knode4 Ready &lt;none&gt; 9d v1.22.2 192.168.1.88 &lt;none&gt; Ubuntu 18.04 LTS 4.15.0-161-generic docker://20.10.9 kmaster2@kmaster2:~$ All three nodes have below OS details:- kmaster2@kmaster2:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04 LTS Release: 18.04 Codename: bionic kmaster2@kmaster2:~$ [+] Install helm on kmaster2 ( https://helm.sh/docs/intro/install/ ) I preffered using apt-get sudo curl https://baltocdn.com/helm/signing.asc | sudo apt-key add - sudo apt-get install apt-transport-https --yes echo "deb https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list sudo apt-get update sudo apt-get install helm [+] Then install prometheus-community/kube-prometheus-stack sudo helm repo add prometheus-community https://prometheus-community.github.io/helm-charts sudo helm repo update [+] Create a namespace for keeping the charts in its own namespace kubectl create ns prometheus [+] Install prometheus-community/kube-prometheus-stack sudo helm install prometheus prometheus-community/kube-prometheus-stack -n prometheus kmaster2@kmaster2:~$ helm list -n prometheus NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION prometheus prometheus 1 2021-10-22 10:07:13.399835228 -0700 PDT deployed kube-prometheus-stack-19.2.2 0.50.0 kmaster2@kmaster2:~$ https://helm.sh/docs/helm/helm_list/ [+] Check all the objects created: kmaster2@kmaster2:~/$ kubectl get all -n prometheus NAME READY STATUS RESTARTS AGE pod/alertmanager-prometheus-kube-prometheus-alertmanager-0 2/2 Running 0 44s pod/prometheus-grafana-b8cd4d67-4t9wb 2/2 Running 0 3m39s pod/prometheus-kube-prometheus-operator-bcdfdbc79-cf8cc 1/1 Running 0 3m39s pod/prometheus-kube-state-metrics-58c5cd6ddb-9xtmt 1/1 Running 0 3m39s pod/prometheus-prometheus-kube-prometheus-prometheus-0 2/2 Running 0 44s pod/prometheus-prometheus-node-exporter-46f6g 1/1 Running 0 3m41s pod/prometheus-prometheus-node-exporter-sc6c7 1/1 Running 0 3m41s pod/prometheus-prometheus-node-exporter-zzq2q 1/1 Running 0 3m41s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/alertmanager-operated ClusterIP None &lt;none&gt; 9093/TCP,9094/TCP,9094/UDP 3m14s service/prometheus-grafana ClusterIP 10.96.216.191 &lt;none&gt; 80/TCP 3m44s service/prometheus-kube-prometheus-alertmanager ClusterIP 10.109.69.174 &lt;none&gt; 9093/TCP 3m45s service/prometheus-kube-prometheus-operator ClusterIP 10.97.223.67 &lt;none&gt; 443/TCP 3m48s service/prometheus-kube-prometheus-prometheus ClusterIP 10.110.169.144 &lt;none&gt; 9090/TCP 3m50s service/prometheus-kube-state-metrics ClusterIP 10.109.193.189 &lt;none&gt; 8080/TCP 3m50s service/prometheus-operated ClusterIP None &lt;none&gt; 9090/TCP 3m11s service/prometheus-prometheus-node-exporter ClusterIP 10.104.193.94 &lt;none&gt; 9100/TCP 3m47s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/prometheus-prometheus-node-exporter 3 3 3 3 3 &lt;none&gt; 3m44s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/prometheus-grafana 1/1 1 1 3m43s deployment.apps/prometheus-kube-prometheus-operator 1/1 1 1 3m43s deployment.apps/prometheus-kube-state-metrics 1/1 1 1 3m43s NAME DESIRED CURRENT READY AGE replicaset.apps/prometheus-grafana-b8cd4d67 1 1 1 3m42s replicaset.apps/prometheus-kube-prometheus-operator-bcdfdbc79 1 1 1 3m42s replicaset.apps/prometheus-kube-state-metrics-58c5cd6ddb 1 1 1 3m42s NAME READY AGE statefulset.apps/alertmanager-prometheus-kube-prometheus-alertmanager 1/1 3m13s statefulset.apps/prometheus-prometheus-kube-prometheus-prometheus 1/1 3m11s kmaster2@kmaster2:~/$ [+] Get the dashboard port information and default user created kmaster2@kmaster2:~/$ kubectl get pods -o=custom-columns=NameSpace:.metadata.namespace,NAME:.metadata.name,CONTAINERS:.spec.containers[*].name -n prometheus NameSpace NAME CONTAINERS prometheus alertmanager-prometheus-kube-prometheus-alertmanager-0 alertmanager,config-reloader prometheus prometheus-grafana-b8cd4d67-4t9wb grafana-sc-dashboard,grafana prometheus prometheus-kube-prometheus-operator-bcdfdbc79-cf8cc kube-prometheus-stack prometheus prometheus-kube-state-metrics-58c5cd6ddb-9xtmt kube-state-metrics prometheus prometheus-prometheus-kube-prometheus-prometheus-0 prometheus,config-reloader prometheus prometheus-prometheus-node-exporter-46f6g node-exporter prometheus prometheus-prometheus-node-exporter-sc6c7 node-exporter prometheus prometheus-prometheus-node-exporter-zzq2q node-exporter kmaster2@kmaster2:~/$ Note from above the POD of interest "prometheus-grafana-b8cd4d67-4t9wb" and the container of interest is "grafana". [+] Get the HTTP port number and user info: kmaster2@kmaster2:~/$ kubectl logs prometheus-grafana-b8cd4d67-4t9wb -c grafana -n prometheus | grep -E "Listen|default admin" t=2021-10-22T17:09:45+0000 lvl=info msg="Created default admin" logger=sqlstore user=admin t=2021-10-22T17:09:46+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=http subUrl= socket= kmaster2@kmaster2:~/$ [+] Password for grafana is "prom-operator" lookup from here: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml [+] Review Grafana dashboard by just using the POD (port-forward) kmaster2@kmaster2:~$ kubectl port-forward -n prometheus pod/prometheus-grafana-b8cd4d67-4t9wb 3000 Forwarding from 127.0.0.1:3000 -&gt; 3000 Forwarding from [::1]:3000 -&gt; 3000 go to http://127.0.0.1:3000 ( admin / prom-operator ) [+] Review prometheus dashboard by just using the POD &amp; container logs (port-forward) kmaster2@kmaster2:~$ kubectl logs prometheus-prometheus-kube-prometheus-prometheus-0 -n prometheus -c prometheus | grep -i 9090 level=info ts=2021-10-22T17:38:39.008Z caller=web.go:541 component=web msg="Start listening for connections" address=0.0.0.0:9090 kmaster2@kmaster2:~$ kmaster2@kmaster2:~$ kubectl port-forward -n prometheus prometheus-prometheus-kube-prometheus-prometheus-0 9090 Forwarding from 127.0.0.1:9090 -&gt; 9090 Forwarding from [::1]:9090 -&gt; 9090 [+] Create a quick SVC to just use the Grafana deployment on a nodeport kmaster2@kmaster2:~$ kubectl get pod -n prometheus -l app.kubernetes.io/name=grafana NAME READY STATUS RESTARTS AGE prometheus-grafana-b8cd4d67-4t9wb 2/2 Running 2 (54m ago) 80m kmaster2@kmaster2:~$ kmaster2@kmaster2:~$ kubectl get deployment -n prometheus -l app.kubernetes.io/name=grafana NAME READY UP-TO-DATE AVAILABLE AGE prometheus-grafana 1/1 1 1 80m kmaster2@kmaster2:~$ kmaster2@kmaster2:~$ kmaster2@kmaster2:~$ kubectl expose deployment prometheus-grafana -n prometheus --name=prometheus-svc --port=3000 --type=NodePort service/prometheus-svc exposed kmaster2@kmaster2:~$ kmaster2@kmaster2:~$ kubectl get svc -n prometheus | grep -i prometheus-svc prometheus-svc NodePort 10.109.152.21 &lt;none&gt; 3000:30371/TCP 73s kmaster2@kmaster2:~$ Now in a browser go to any of the cluster nodes IP and port 30371 to get into the grafana dashboard In my cluster I went to : http://192.168.1.86:30371/login ( admin / prom-operator ) [+] For a constant service you can do: kmaster2@kmaster2:~$ kubectl expose deployment prometheus-grafana -n prometheus --name=prometheus-svc --port=3000 --type=NodePort --dry-run=client -o yaml &gt; grafana.yaml kmaster2@kmaster2:~$ Edit the YAML file to add "nodePort: 30000" kmaster2@kmaster2:~$ cat grafana.yaml apiVersion: v1 kind: Service metadata: creationTimestamp: null labels: app.kubernetes.io/instance: prometheus app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: grafana app.kubernetes.io/version: 8.2.1 helm.sh/chart: grafana-6.17.2 name: prometheus-svc namespace: prometheus spec: ports: - port: 3000 nodePort: 30000 protocol: TCP targetPort: 3000 selector: app.kubernetes.io/instance: prometheus app.kubernetes.io/name: grafana type: NodePort status: loadBalancer: {} kmaster2@kmaster2:~$ kmaster2@kmaster2:~$ kubectl apply -f grafana.yaml service/prometheus-svc created kmaster2@kmaster2:~$ kmaster2@kmaster2:~$ kubectl get svc -n prometheus | grep -i prometheus-svc prometheus-svc NodePort 10.111.152.221 &lt;none&gt; 3000:30000/TCP 20s kmaster2@kmaster2:~$ kmaster2@kmaster2:~$ kubectl describe svc prometheus-grafana -n prometheus Name: prometheus-grafana Namespace: prometheus Labels: app.kubernetes.io/instance=prometheus app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=grafana app.kubernetes.io/version=8.2.1 helm.sh/chart=grafana-6.17.2 Annotations: meta.helm.sh/release-name: prometheus meta.helm.sh/release-namespace: prometheus Selector: app.kubernetes.io/instance=prometheus,app.kubernetes.io/name=grafana Type: ClusterIP IP Family Policy: SingleStack IP Families: IPv4 IP: 10.96.216.191 IPs: 10.96.216.191 Port: service 80/TCP TargetPort: 3000/TCP Endpoints: 10.44.0.1:3000 Session Affinity: None Events: &lt;none&gt; kmaster2@kmaster2:~$ Now in a browser go to any of the cluster nodes IP and port 30000 to get into the grafana dashboard In my cluster I went to : http://192.168.1.86:30000/login ( admin / prom-operator ) and it works :) </code></pre> </div> upinder sujlana Mon, 23 Nov 2020 17:57:11 +0000 https://dev.to/upindersujlana/upgrade-kubernetes-cluster-to-1-19-4-using-kubeadm-3ien https://dev.to/upindersujlana/upgrade-kubernetes-cluster-to-1-19-4-using-kubeadm-3ien <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code> The primary intention of the article is to note down the steps I tool to upgrade my three node cluster from 1.18.1 to 1.19.4. I could not find any notes/blog out there that had these steps from a user perspective. The documentation for upgrading kubernetes cluster using kubeadm is very good, but as I shall not be upgrading my home cluster often I wanted to catalogue the steps for posterity. FYI also review the release notes for any red flags. Documentation: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/ My Cluster ------------ kmaster@kmaster:~$ sudo kubectl get nodes [sudo] password for kmaster: NAME STATUS ROLES AGE VERSION kmaster Ready master 157d v1.18.1 knode1 Ready &lt;none&gt; 157d v1.18.1 knode2 Ready &lt;none&gt; 157d v1.18.1 kmaster@kmaster:~$ ################################# # Master node steps only # ################################# kmaster@kmaster:~$ sudo apt update kmaster@kmaster:~$ sudo apt-cache madison kubeadm [+] The last command will show you the options, I am going for 1.19.4-00. Note the version is important as you shall see later on. kubeadm | 1.19.4-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages &lt;&lt;&lt;&lt;&lt; Going for this one kubeadm | 1.19.3-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.2-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.1-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.0-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kmaster@kmaster:~$ sudo apt-mark unhold kubeadm &amp;&amp; \ &gt; sudo apt-get update &amp;&amp; sudo apt-get install -y kubeadm=1.19.4-00 &amp;&amp; \ &gt; sudo apt-mark hold kubeadm &lt;snip&gt; Preparing to unpack .../kubeadm_1.19.4-00_amd64.deb ... Unpacking kubeadm (1.19.4-00) over (1.19.0-00) ... Setting up kubeadm (1.19.4-00) ... kubeadm set on hold. kmaster@kmaster:~$ Verify ------- kmaster@kmaster:~$ kubeadm version kubeadm version: &amp;version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-11T13:15:05Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"} kmaster@kmaster:~$ Cordon the Master ------------------------ kmaster@kmaster:~$ sudo kubectl drain kmaster --ignore-daemonsets --delete-local-data node/kmaster cordoned WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-flannel-ds-amd64-djm58, kube-system/kube-proxy-9sbdw, lens-metrics/node-exporter-b4v4z evicting pod kube-system/coredns-66bff467f8-xw52j evicting pod lens-metrics/kube-state-metrics-767bd96f84-b5s8j evicting pod kube-system/coredns-66bff467f8-bmjcm evicting pod kubernetes-dashboard/kubernetes-dashboard-7b544877d5-x5pbn evicting pod monitoring/prometheus-deployment-54686956bd-22xdk pod/kube-state-metrics-767bd96f84-b5s8j evicted pod/kubernetes-dashboard-7b544877d5-x5pbn evicted pod/prometheus-deployment-54686956bd-22xdk evicted pod/coredns-66bff467f8-bmjcm evicted pod/coredns-66bff467f8-xw52j evicted node/kmaster evicted kmaster@kmaster:~$ Now run the upgrade plan --------------------------- kmaster@kmaster:~$ sudo kubeadm upgrade plan [upgrade/config] Making sure the configuration is correct: [upgrade/config] Reading configuration from the cluster... [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [preflight] Running pre-flight checks. [upgrade] Running cluster health checks [upgrade] Fetching available versions to upgrade to [upgrade/versions] Cluster version: v1.18.4 [upgrade/versions] kubeadm version: v1.19.4 [upgrade/versions] Latest stable version: v1.19.4 [upgrade/versions] Latest stable version: v1.19.4 [upgrade/versions] Latest version in the v1.18 series: v1.18.12 [upgrade/versions] Latest version in the v1.18 series: v1.18.12 Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT CURRENT AVAILABLE kubelet 3 x v1.18.1 v1.18.12 Upgrade to the latest version in the v1.18 series: COMPONENT CURRENT AVAILABLE kube-apiserver v1.18.4 v1.18.12 kube-controller-manager v1.18.4 v1.18.12 kube-scheduler v1.18.4 v1.18.12 kube-proxy v1.18.4 v1.18.12 CoreDNS 1.6.7 1.7.0 etcd 3.4.3-0 3.4.3-0 You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.18.12 _____________________________________________________________________ Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT CURRENT AVAILABLE kubelet 3 x v1.18.1 v1.19.4 Upgrade to the latest stable version: COMPONENT CURRENT AVAILABLE kube-apiserver v1.18.4 v1.19.4 kube-controller-manager v1.18.4 v1.19.4 kube-scheduler v1.18.4 v1.19.4 kube-proxy v1.18.4 v1.19.4 CoreDNS 1.6.7 1.7.0 etcd 3.4.3-0 3.4.13-0 You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.19.4 _____________________________________________________________________ The table below shows the current state of component configs as understood by this version of kubeadm. Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually upgrade to is denoted in the "PREFERRED VERSION" column. API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED kubeproxy.config.k8s.io v1alpha1 v1alpha1 no kubelet.config.k8s.io v1beta1 v1beta1 no _____________________________________________________________________ kmaster@kmaster:~$ [+] Using the command printed out in the previous command output. kmaster@kmaster:~$ sudo kubeadm upgrade apply v1.19.4 [upgrade/config] Making sure the configuration is correct: [upgrade/config] Reading configuration from the cluster... [upgrade/config] FYI: You can look at this &lt;snip&gt; [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.19.4". Enjoy! [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so. kmaster@kmaster:~$ Upgrade Kubelet ----------------- kmaster@kmaster:~$ sudo apt-mark unhold kubelet kubectl &amp;&amp; \ &gt; sudo apt-get update &amp;&amp; sudo apt-get install -y kubelet=1.19.4-00 kubectl=1.19.4-00 &amp;&amp; \ &gt; sudo apt-mark hold kubelet kubectl &lt;snip&gt; Unpacking kubelet (1.19.4-00) over (1.18.1-00) ... Setting up kubelet (1.19.4-00) ... Setting up kubectl (1.19.4-00) ... kubelet set on hold. kubectl set on hold. kmaster@kmaster:~$ kmaster@kmaster:~$ sudo systemctl daemon-reload kmaster@kmaster:~$ sudo systemctl restart kubelet kmaster@kmaster:~$ kmaster@kmaster:~$ sudo kubectl get nodes NAME STATUS ROLES AGE VERSION kmaster Ready,SchedulingDisabled master 157d v1.19.4 knode1 Ready &lt;none&gt; 157d v1.18.1 knode2 Ready &lt;none&gt; 157d v1.18.1 kmaster@kmaster:~$ kmaster@kmaster:~$ kmaster@kmaster:~$ sudo kubectl uncordon kmaster node/kmaster uncordoned kmaster@kmaster:~$ kmaster@kmaster:~$ sudo kubectl get nodes NAME STATUS ROLES AGE VERSION kmaster Ready master 157d v1.19.4 knode1 Ready &lt;none&gt; 157d v1.18.1 knode2 Ready &lt;none&gt; 157d v1.18.1 kmaster@kmaster:~$ ################################# # Worker node steps only # ################################# [+] Run below steps ONLY one node at a time. knode1@knode1:~$ sudo apt-mark unhold kubeadm &amp;&amp; \ &gt; sudo apt-get update &amp;&amp; sudo apt-get install -y kubeadm=1.19.4-00 &amp;&amp; \ &gt; sudo apt-mark hold kubeadm [sudo] password for knode1: [+] NOTE : From kmaster do for knode1 for starters kmaster@kmaster:~$ sudo kubectl drain knode1 --ignore-daemonsets --delete-local-data knode1@knode1:~$ sudo kubeadm upgrade node [upgrade] Reading configuration from the cluster... [upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' knode1@knode1:~$ sudo apt-mark unhold kubelet kubectl &amp;&amp; \ &gt; sudo apt-get update &amp;&amp; sudo apt-get install -y kubelet=1.19.4-00 kubectl=1.19.4-00 &amp;&amp; \ &gt; sudo apt-mark hold kubelet kubectl knode1@knode1:~$ sudo systemctl daemon-reload knode1@knode1:~$ sudo systemctl restart kubelet knode1@knode1:~$ kmaster@kmaster:~$ sudo kubectl uncordon knode1 node/knode1 uncordoned kmaster@kmaster:~$ [+] Now move onto the knode2 and do te above steps to complete the upgrade. Finally after upgrade (testing) --------------------------------- kmaster@kmaster:~$ sudo kubectl get nodes NAME STATUS ROLES AGE VERSION kmaster Ready master 157d v1.19.4 knode1 Ready &lt;none&gt; 157d v1.19.4 knode2 Ready &lt;none&gt; 157d v1.19.4 kmaster@kmaster:~$ root@kmaster:~# kubectl create deployment multitool --image=praqma/network-multitool --replicas=1 deployment.apps/multitool created root@kmaster:~# root@kmaster:~# kubectl get pods NAME READY STATUS RESTARTS AGE multitool-74477484b8-gh4ct 1/1 Running 0 28s root@kmaster:~# root@kmaster:~# kubectl exec -it multitool-74477484b8-gh4ct -- /bin/sh / # / # ping www.google.com PING www.google.com (172.217.164.100) 56(84) bytes of data. 64 bytes from sfo03s18-in-f4.1e100.net (172.217.164.100): icmp_seq=1 ttl=116 time=14.2 ms ^C --- www.google.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 14.154/14.154/14.154/0.000 ms / # </code></pre> </div> upinder sujlana Mon, 27 Jan 2020 22:54:40 +0000 https://dev.to/upindersujlana/using-kubernetes-with-nfs-storage-3iel https://dev.to/upindersujlana/using-kubernetes-with-nfs-storage-3iel <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>In this article, I want to show how Kubernetes cluster can use an external NFS server for storage. The code for this article is here https://github.com/upinder-sujlana/K8S-Volumes/blob/master/README.md Topology -------------- kmaster - 192.168.1.80 knode1 - 192.168.1.81 knode2 - 192.168.1.82 The three form a K8S cluster:- kmaster@kmaster:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION kmaster Ready master 233d v1.14.2 knode1 Ready &lt;none&gt; 233d v1.14.2 knode2 Ready &lt;none&gt; 233d v1.14.2 kmaster@kmaster:~$ The 3-nodes are OS details are the same:- kmaster@kmaster:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04 LTS Release: 18.04 Codename: bionic kmaster@kmaster:~$ Additionally, I have a 4th node outside the cluster, but in the same LAN that I am using as a NFS Server :- minikube - 192.168.1.85 (NFS Server running here) On the NFS Server, I have exposed three directories to the above cluster (permit all) directory names are gold, silver, bronze. Time to create a Persistent volume. kmaster@kmaster:~/dockerimagemaker/NFS$ cat nfs-pv.yml apiVersion: v1 kind: PersistentVolume metadata: # any PV name name: nfs-pv labels: volume: nfs-pv-volume spec: capacity: # storage size storage: 5Gi accessModes: # ReadWriteMany(RW from multi nodes), ReadWriteOnce(RW from a node), ReadOnlyMany(R from multi nodes) - ReadWriteMany persistentVolumeReclaimPolicy: # retain even if pods terminate Retain nfs: # NFS server's definition path: /home/minikube/NFSShare/gold server: 192.168.1.85 readOnly: false kmaster@kmaster:~/dockerimagemaker/NFS$ kmaster@kmaster:~/dockerimagemaker/NFS$ kubectl create -f nfs-pv.yml persistentvolume/nfs-pv created kmaster@kmaster:~/dockerimagemaker/NFS$ kmaster@kmaster:~/dockerimagemaker/NFS$ kubectl get pv --show-labels -o wide NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE LABELS nfs-pv 5Gi RWX Retain Available 100s volume=nfs-pv-volume kmaster@kmaster:~/dockerimagemaker/NFS$ kmaster@kmaster:~/dockerimagemaker/NFS$ kubectl describe pv nfs-pv Name: nfs-pv Labels: volume=nfs-pv-volume Annotations: &lt;none&gt; Finalizers: [kubernetes.io/pv-protection] StorageClass: Status: Available Claim: Reclaim Policy: Retain Access Modes: RWX VolumeMode: Filesystem Capacity: 5Gi Node Affinity: &lt;none&gt; Message: Source: Type: NFS (an NFS mount that lasts the lifetime of a pod) Server: 192.168.1.85 Path: /home/minikube/NFSShare/gold ReadOnly: false Events: &lt;none&gt; kmaster@kmaster:~/dockerimagemaker/NFS$ Creating a persistent volume claim. kmaster@kmaster:~/dockerimagemaker/NFS$ cat nfs-pvc.yml apiVersion: v1 kind: PersistentVolumeClaim metadata: # any PVC name name: nfs-pvc spec: selector: matchLabels: volume: nfs-pv-volume accessModes: # ReadWriteMany(RW from multi nodes), ReadWriteOnce(RW from a node), ReadOnlyMany(R from multi nodes) - ReadWriteMany resources: requests: # storage size to use storage: 1Gi kmaster@kmaster:~/dockerimagemaker/NFS$ kmaster@kmaster:~/dockerimagemaker/NFS$ kubectl create -f nfs-pvc.yml persistentvolumeclaim/nfs-pvc created kmaster@kmaster:~/dockerimagemaker/NFS$ kmaster@kmaster:~/dockerimagemaker/NFS$ kubectl get pvc --show-labels NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE LABELS nfs-pvc Bound nfs-pv 5Gi RWX 32s &lt;none&gt; kmaster@kmaster:~/dockerimagemaker/NFS$ kmaster@kmaster:~/dockerimagemaker/NFS$ kubectl describe pvc nfs-pvc Name: nfs-pvc Namespace: default StorageClass: Status: Bound Volume: nfs-pv Labels: &lt;none&gt; Annotations: pv.kubernetes.io/bind-completed: yes pv.kubernetes.io/bound-by-controller: yes Finalizers: [kubernetes.io/pvc-protection] Capacity: 5Gi Access Modes: RWX VolumeMode: Filesystem Events: &lt;none&gt; Mounted By: &lt;none&gt; kmaster@kmaster:~/dockerimagemaker/NFS$ Time to test, going to create a test deployment (busybox) and see if it will work. The Pod shall mount the gold directory to its /tmp folder and I shall just send output of date command to the folder. kmaster@kmaster:~/dockerimagemaker/NFS$ cat nfstester.yml apiVersion: apps/v1 kind: Deployment metadata: name: nfstester labels: type: nfstester spec: replicas: 1 selector: matchLabels: type: nfstester strategy: type: Recreate template: metadata: labels: type: nfstester spec: volumes: - name: nfstester persistentVolumeClaim: claimName: nfs-pvc containers: - name: nfstester image: busybox command: [ 'sh', '-c', 'while true; do date;sleep 10; done &gt;&gt; /tmp/hellopod.txt'] volumeMounts: - name: nfstester mountPath: /tmp kmaster@kmaster:~/dockerimagemaker/NFS$ All this test pod does is every 10 sec wakes up and dumps the "date" to the mounted NFS share. kmaster@kmaster:~$ kubectl create -f nfstester.yml deployment.apps/nfstester created kmaster@kmaster:~$ Went to the NFS server directory and started the tail on the newly created file: minikube@ubuntu:~/NFSShare/gold$ tail -f hellopod.txt Wed Jan 15 20:50:56 UTC 2020 Wed Jan 15 20:51:06 UTC 2020 Wed Jan 15 20:51:16 UTC 2020 Wed Jan 15 20:51:26 UTC 2020 kmaster@kmaster:~/dockerimagemaker/NFS$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE nfs-pvc Bound nfs-pv 5Gi RWX manualstorageclass 24d kmaster@kmaster:~/dockerimagemaker/NFS$ kmaster@kmaster:~/dockerimagemaker/NFS$ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE nfs-pv 5Gi RWX Retain Bound default/nfs-pvc manualstorageclass 24d kmaster@kmaster:~/dockerimagemaker/NFS$ </code></pre> </div>