Kubernetes hack

NOTE: this is an updated copy of my post in medium, where I'm not writing anymore.

Have you lost ssh access to one of your Kubernetes nodes? Why do you even need ssh access to nodes in the first place? Well, maybe something is stuck, or you need to see a config with your own eyes… I don’t know and I don’t care, they are your servers, not mine…

I’m assuming you have admin level into kubernetes API.

Talk is cheap, show me the code®:

$ NODE_NAME=master-0
$ kubectl create -n kube-system -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: root-shell-$NODE_NAME
  namespace: kube-system
spec:
  nodeName: $NODE_NAME
  containers:
  - command:
    - /bin/cat
    image: alpine:3
    name: root-shell
    securityContext:
      #privileged: true
      allowPrivilegeEscalation: true
      readOnlyRootFilesystem: true
      runAsGroup: 0
      runAsUser: 0
      capabilities:
        drop:
        - ALL
      seLinuxOptions:
        level: s0
        user: system_u
    tty: true
    stdin: true
    volumeMounts:
    - mountPath: /host
      name: hostroot
  hostNetwork: true
  hostPID: true
  hostIPC: true
  tolerations:
  - effect: NoSchedule
    operator: Exists
  - effect: NoExecute
    operator: Exists
  volumes:
  - hostPath:
      path: /
    name: hostroot
EOF

This pod will create a privileged POD into the node master-0 (change it to your node name) running /bin/cat forever. Now you simply exec into it and change the host’s root to pod’s root:

$ kubectl -n kube-system exec -it root-shell-$NODE_NAME chroot /host /bin/bash
[root@master-0 /]# id
uid=0(root) gid=0(root) groups=0(root)

Profit!

PS: Here is a DaemonSet for the lazy like me.

$ kubectl create serviceaccount -n kube-system root-shell
### For OKD/Openshift clusters only:
$ oc adm add-scc-to-user privileged -n kube-system -z root-shell
$ kubectl create -n kube-system -f - <<EOF
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: root-shell
  namespace: kube-system
spec:
  revisionHistoryLimit: 0
  selector:
    matchLabels:
      app: root-shell
  template:
    metadata:
      labels:
        app: root-shell
    spec:
      terminationGracePeriodSeconds: 0
      containers:
      - command:
        - /bin/cat
        image: alpine:3
        name: root-shell
        tty: true
        stdin: true
        volumeMounts:
        - mountPath: /host
          name: hostroot
        #securityContext:
        #  privileged: true
        allowPrivilegeEscalation: true
        readOnlyRootFilesystem: true
        runAsGroup: 0
        runAsUser: 0
        capabilities:
          drop:
          - ALL
        seLinuxOptions:
          level: s0
          user: system_u
      hostNetwork: true
      hostPID: true
      hostIPC: true
      serviceAccountName: root-shell
      hostNetwork: true
      tolerations:
      - effect: NoSchedule
        operator: Exists
      - effect: NoExecute
        operator: Exists
      volumes:
      - hostPath:
          path: /
        name: hostroot
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 100%
    type: RollingUpdate
EOF

AWS Bottlerock

Since now bottlerocket is a really nice OS alternative in EKS clusters, lets use control/admin containers to gain root-level access to the host.

As you known, Bottlerocket is a Read-only, container oriented operating system. This gives us a lot of benefits, but you can't simply kubect exec + chroot into it as we did in the stone age. Turns out there is two containers in the system: control and admin. We whant admin, since control is for adminstrative tasks (like upgrades and reboots).

First, we need a plain container to have access to host's apiclient binary:

$ kubectl create -n kube-system -f - <<EOF
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: apiclient
  namespace: kube-system
spec:
  revisionHistoryLimit: 0
  selector:
    matchLabels:
      app: apiclient
  template:
    metadata:
      labels:
        app: apiclient
    spec:
      containers:
      - command:
        - sleep
        - infinity
        image: fedora
        imagePullPolicy: Always
        name: regain-access
        securityContext:
          seLinuxOptions:
            level: s0
            role: system_r
            type: control_t
            user: system_u
        volumeMounts:
        - mountPath: /usr/bin/apiclient
          name: apiclient
          readOnly: true
        - mountPath: /run/api.sock
          name: apiserver-socket
      restartPolicy: Always
      terminationGracePeriodSeconds: 0
      volumes:
      - hostPath:
          path: /usr/bin/apiclient
          type: File
        name: apiclient
      - hostPath:
          path: /run/api.sock
          type: Socket
        name: apiserver-socket
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 100%
    type: RollingUpdate
EOF

With that in place, let's use apiclient exec subcommand to activate and enter admin container:

$ kubectl exec -i -t -n kube-system apiclient-xtwxh -- apiclient exec -t control enter-admin-container
Confirming admin container is enabled...
Waiting for admin container to start...
Entering admin container
          Welcome to Bottlerocket's admin container!
    ╱╲
   ╱┄┄╲   This container provides access to the Bottlerocket host
   │▗▖│   filesystems (see /.bottlerocket/rootfs) and contains common
  ╱│  │╲  tools for inspection and troubleshooting.  It is based on
  │╰╮╭╯│  Amazon Linux 2, and most things are in the same places you
    ╹╹    would find them on an AL2 host.

To permit more intrusive troubleshooting, including actions that mutate the
running state of the Bottlerocket host, we provide a tool called "sheltie"
(`sudo sheltie`).  When run, this tool drops you into a root shell in the
Bottlerocket host's root filesystem.
[root@admin]#

At the time of this writing, all you have to do now is execute sudo sheltie and voilà!

[root@admin]# sudo sheltie
bash-5.1# ps 1
    PID TTY      STAT   TIME COMMAND
      1 ?        Ss    14:25 /sbin/init systemd.log_target=journal-or-kmsg systemd.log_color=0 systemd.show_status=true
bash-5.1#