Another method to access AKS Linux Nodes

Radu Andra — Fri, 12 Feb 2021 18:17:49 +0000

Deploy a demonset with privileged containers to access each node

cat <<EOF > sshNode.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: privileged
spec:
  selector:
    matchLabels:
      name: privileged-container
  template:
    metadata:
      labels:
        name: privileged-container
    spec:
      containers:
      - name: busybox
        image: busybox
        resources:
          limits:
            cpu: 200m
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 50Mi
        stdin: true
        securityContext:
          privileged: true
        volumeMounts:
        - name: host-root-volume
          mountPath: /host
          readOnly: false
      volumes:
      - name: host-root-volume
        hostPath:
          path: /
      hostNetwork: true
      hostPID: true
      restartPolicy: Always
EOF

Bellow, I list the nodes and the pods, I see that I have 3 nodes and 3 privileged pods each one related to one specific node.
I exec into one created pod and I enter directly on the node from there I access the kubelet logs.
Exit the node by entering this command: exit

kubectl get nodes -owide
kubectl get pods -owide                                            
kubectl exec -it privileged-dr5mf chroot /host

Configure AKS node pools with HugePage support.

Radu Andra — Fri, 12 Feb 2021 18:08:17 +0000

In kubernetes to enable HugePages you add in kubelet conf --feature-gates=HugePages=true

In OpenShift you use node tunning operator

But in AKS seems that there is no direct solution.

The only official Azure Hugepage doc is about transparentHugePage and it's in preview (https://docs.microsoft.com/en-us/azure/aks/custom-node-configuration).
As far as I know, Transparent HugePages and HugePages are two different things.

If you describe the node after you enabled the Transparent HugePages, by the way, on most Linux system it’s already enabled, the HugePages value is still 0, that means that even with Transparent activated the HugePages are still not enabled:

In this case if you will create a pod asking :

    resources:
     limits:
        hugepages-2Mi: 100Mi

This pod will fail with Insufficient HugePages because we have hugepages 0 on the node (means not enabled)

Manual temporary workaround is to:

Please take note that this procedure will last as long as the node is kept in its original state. You also need to repeat this on all newly added nodes and after node update ore reimage

Enter on the node, once inside the node I applied the fallowing commands:

mkdir -p /mnt/huge**                                                             mount -t hugetlbfs nodev /mnt/huge                                               echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages                                                               cat  /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages

Then you will have:

Or add below value in /etc/sysctl.conf and reload configuration by issuing sysctl -p command

To automate things and not worry about node scaling or node updates use daemonsets:

cat <<EOF > sshNode.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: privileged
spec:
  selector:
    matchLabels:
      name: privileged-container
  template:
    metadata:
      labels:
        name: privileged-container
    spec:
      containers:
      - name: busybox
        image: busybox
        command: ["/bin/sh","-c"]
        args: ["mkdir -p host/mnt/huge; mount -t hugetlbfs nodev host/mnt/huge; echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages"]
        resources:
          limits:
            cpu: 200m
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 50Mi
        stdin: true
        securityContext:
          privileged: true
        volumeMounts:
        - name: host-root-volume
          mountPath: /host
          readOnly: false
      volumes:
      - name: host-root-volume
        hostPath:
          path: /
      hostNetwork: true
      hostPID: true
      restartPolicy: Always
EOF

To finally enable the HugePages on the nodes enter on each node and do:

systemctl daemon-reload

systemctl restart kubelet

You need to restart the nodes via CLI or from the portal, for the HugePages to be enabled on the nodes.

Inspiration Articles:

HugePages on Kubernetes - DEV Community 👩‍💻👨‍💻

DEV Community: Radu Andra

Another method to access AKS Linux Nodes

Deploy a demonset with privileged containers to access each node

Configure AKS node pools with HugePage support.