Day 63: Two Tiers and a Root Key

Day 63 of DevOps, Day 13 of Azure. Both tasks today had something already written before I arrived. Kubernetes filled in resource requests I never typed, which changed the pod's QoS class. And cloud-init had already put a line in root's authorized_keys on the Azure VM, which is why adding my key to it did not work on its own.

One Kubernetes task, one Azure task. Deploy a frontend and a database into their own namespace, then install a root SSH key on a VM with no SSH access to start from. The tasks come from the KodeKloud Engineer platform.

Task 1: A two-tier app in its own namespace

The task: a namespace, iron-namespace-nautilus, holding a frontend Deployment, a MariaDB Deployment, a ClusterIP Service for the database on 3306, and a NodePort Service for the frontend on 32678.

Step 1: Create the namespace

kubectl create ns iron-namespace-nautilus
kubectl get ns

NAME                      STATUS   AGE
default                   Active   96m
iron-namespace-nautilus   Active   23m
kube-node-lease           Active   96m
kube-public               Active   96m
kube-system               Active   96m

The docs: "namespaces provide a mechanism for isolating groups of resources within a single cluster", and "Names of resources need to be unique within a namespace, but not across namespaces."

Step 2: The frontend Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: iron-gallery-deployment-nautilus
  labels:
    run: iron-gallery
  namespace: iron-namespace-nautilus
spec:
  replicas: 1
  selector:
    matchLabels:
      run: iron-gallery
  template:
    metadata:
      labels:
        run: iron-gallery
    spec:
      volumes:
      - name: config
        emptyDir: {}
      - name: images
        emptyDir: {}
      containers:
      - name: iron-gallery-container-nautilus
        image: kodekloud/irongallery:2.0
        resources:
          limits:
           memory: "100Mi"
           cpu: "50m"
        volumeMounts:
        - name: config
          mountPath: /usr/share/nginx/html/data
        - name: images
          mountPath: /usr/share/nginx/html/uploads

Every object needs namespace: in its metadata, or it lands in default. Every kubectl call afterwards needs -n. That is the cost of the isolation, and it is easy to forget halfway through a task.

Step 3: The database Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: iron-db-deployment-nautilus
  labels:
    db: mariadb
  namespace: iron-namespace-nautilus
spec:
  replicas: 1
  selector:
    matchLabels:
      db: mariadb
  template:
    metadata:
      labels:
        db: mariadb
    spec:
      volumes:
      - name: db
        emptyDir: {}
      containers:
      - name: iron-db-container-nautilus
        image: kodekloud/irondb:2.0
        volumeMounts:
        - name: db
          mountPath: /var/lib/mysql
        env:
        - name: MYSQL_DATABASE
          value: database_apache
        - name: MYSQL_ROOT_PASSWORD
          value: testroot123
        - name: MYSQL_PASSWORD
          value: test123
        - name: MYSQL_USER
          value: nileena

Those are the task's credentials, in the task's plain-text form. They are also the anti-pattern, and Day 66 does the same job with Secrets instead.

Step 4: Two Services, two types

apiVersion: v1
kind: Service
metadata:
  name: iron-db-service-nautilus
  namespace: iron-namespace-nautilus
spec:
  selector:
    db: mariadb
  ports:
    - protocol: TCP
      port: 3306
      targetPort: 3306
---
apiVersion: v1
kind: Service
metadata:
  name: iron-gallery-service-nautilus
  namespace: iron-namespace-nautilus
spec:
  type: NodePort
  selector:
    run: iron-gallery
  ports:
    - port: 80
      targetPort: 80
      nodePort: 32678

The database Service has no type, so it is ClusterIP, which the docs describe as making the Service "only reachable from within the cluster". That is the whole security model for the database here: it has no route in from outside.

Step 5: Apply, then check the endpoints

kubectl apply -f deploy-1.yaml
kubectl apply -f deploy-2.yaml
kubectl apply -f svc-1.yaml
kubectl apply -f svc-2.yaml

kubectl get po -n iron-namespace-nautilus
kubectl describe svc -n iron-namespace-nautilus

NAME                                               READY   STATUS    RESTARTS   AGE
iron-db-deployment-nautilus-6cbdbfbb4d-tgvkz       1/1     Running   0          30s
iron-gallery-deployment-nautilus-87cf5796b-9m2l2   1/1     Running   0          3m26s

Endpoints:                10.22.0.10:3306
Endpoints:                10.22.0.9:80

The Endpoints lines are the real check. Neither Service names a Deployment anywhere; they match on db: mariadb and run: iron-gallery. Two pod IPs in the Endpoints means the labels lined up. An empty list would mean they did not, and the Service would still look perfectly healthy.

The QoS class I did not ask for

Limits:
  cpu:     50m
  memory:  100Mi
Requests:
  cpu:        50m
  memory:     100Mi
QoS Class:    Guaranteed

I wrote limits. I did not write requests. Kubernetes wrote them for me, and the docs say so: when a container specifies a limit and no request, "Kubernetes copies the limit you specified and uses it as the requested value for the resource."

Equal requests and limits on every resource in every container is the definition of Guaranteed. So the shortest path to the strongest QoS class is to specify less, not more, which is the reverse of what Day 50 led me to expect.

The database pod in the same namespace is BestEffort, because its manifest has no resources block at all. One application, two tiers, two QoS classes, and the difference is four lines of YAML.

What the docs recommend

Give the database real storage. Both deployments use emptyDir, and the docs are unambiguous about what that means: "When a Pod is removed from a node for any reason, the data in the emptyDir is deleted permanently." The schema, the user and every row go with the pod. That is acceptable for a task that only asks whether the page loads, and it is the thing to change first.

The docs also point at the workload type. StatefulSets exist for applications needing "Stable, unique network identifiers" and "Stable, persistent storage", which is a database in two bullets.

Pick the node port deliberately. 32678 falls in the dynamic band of the default range, 30086 to 32767, which is where Kubernetes allocates automatic ports from first. The static band below it is where a hand-picked number is least likely to collide with one the cluster assigned to somebody else.

Do not assume a third-party image honours the usual variables. kodekloud/irondb:2.0 has no entry in the official image documentation. The MYSQL_* variables show up in kubectl describe pod because I set them, which proves they were delivered to the container and nothing more. Even the official mariadb image only documents MARIADB_DATABASE among these and defers the rest elsewhere.

One honest note about my own transcript. kubectl get svc printed 80/TCP for the database Service while kubectl describe svc printed 3306/TCP for the port, the target port and the endpoint. The manifest says 3306 and three of the four agree, so the get line is almost certainly a copy error on my part rather than a real state. I did not re-run it, so I am flagging it rather than quietly picking the convenient one.

Task 2: Install a root SSH key without SSH

The task: put the client host's root public key into nautilus-vm's root authorized_keys, and prove password-less root login works.

Step 1: Get a shell without having a shell

The obvious route needs credentials for azureuser, which the task did not give me. Azure has a way round that:

az vm run-command invoke -g $RG -n nautilus-vm --command-id RunShellScript --scripts "..."

Microsoft: "The Run Command feature uses the virtual machine (VM) agent to run shell scripts within an Azure Linux VM", and "Scripts run by default as an elevated user on Linux." No SSH, no key, no open port. It runs over the Azure control plane through the agent already inside the VM.

That is the shape of the whole task: use a channel that does not depend on SSH to set up SSH.

Step 2: Look before writing

The script printed root's existing authorized_keys first:

BEFORE:
no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="echo 'Please login as the user \"azureuser\"

That line is cloud-init's doing. It copies the admin user's key into root's file behind a forced command, so a root login authenticates successfully, prints the message, and disconnects.

It is identical on AWS and Azure, because it comes from cloud-init rather than from either cloud. Day 35 on the AWS track hit exactly this.

Step 3: Filter rather than append

grep -v 'Please login as' /root/.ssh/authorized_keys \
  | grep -vF '$KEYBODY' > /tmp/ak
echo '$PUBKEY' >> /tmp/ak
mv /tmp/ak /root/.ssh/authorized_keys

The reason not to simply append is that the banner line probably contains this same key already. The lab likely built the VM using /root/.ssh/id_rsa.pub from the client, and cloud-init copied it across with the forced command in front.

I had been carrying an explanation for why that blocks a clean duplicate: sshd uses the first matching line. When I went to check it, the sshd manual does not say that. Its AUTHORIZED_KEYS FILE FORMAT section describes the line format and says a forced command "is executed whenever this key is used for authentication", with no statement anywhere about ordering or precedence between two lines carrying the same key.

So the forced command certainly applies when its line is the one that authenticates. Which line wins is undocumented, and I did not test it. The script above sidesteps the question: drop every banner line, drop any existing copy of this key, append it once. That is correct either way, and it is closer to what adding a key ought to mean than overwriting the file.

-rw------- 1 root root  399 Sep 22 07:33 authorized_keys

Step 4: Fix the permissions before testing, not after

chmod 700 /root/.ssh
chmod 600 /root/.ssh/authorized_keys
chown -R root:root /root/.ssh

StrictModes is on by default and "Specifies whether sshd(8) should check file modes and ownership of the user's files and home directory before accepting login."

When that check fails, sshd tells the client nothing. The login quietly falls through to the next authentication method, and the reason goes to the server's auth log, which you cannot read if you cannot log in. Wrong permissions look exactly like a wrong key from the outside.

Step 5: Ask sshd what it will actually do

sshd -T | grep -i permitrootlogin

permitrootlogin without-password

sshd -T is "Extended test mode. Check the validity of the configuration file, output the effective configuration to stdout and then exit." It resolves every include and default, which reading /etc/ssh/sshd_config by hand does not, and cloud images put their overrides in /etc/ssh/sshd_config.d/.

without-password is the legacy spelling. The manual: root may log in with a key "If this option is set to prohibit-password (or its deprecated alias, without-password)", with password authentication disabled for root. Had this printed no, the key would have been irrelevant.

Step 6: Prove it was the key

ssh -o BatchMode=yes -o StrictHostKeyChecking=no -o ConnectTimeout=10 root@$VM_IP \
  'whoami; hostname; stat -c "%a %U %n" /root/.ssh /root/.ssh/authorized_keys'

root
nautilus-vm
700 root /root/.ssh
600 root /root/.ssh/authorized_keys

BatchMode is what turns this from "login worked" into "login worked without a password". The manual: "If set to yes, user interaction such as password prompts and host key confirmation requests will be disabled."

Worth reading that precisely. It names password prompts and host key confirmation. It says nothing about passphrase prompting for a local key, so it is narrower than "disables all prompting".

What the docs recommend

Do not enable root SSH. The cloud-init banner exists to stop exactly what this task asked for, and a compromised root key is a compromised machine with no sudo trail in between. Microsoft's alternatives, in rough order: log in as azureuser and use sudo; use Microsoft Entra ID login for Linux VMs, which lets you "use Microsoft Entra ID as a core authentication platform and a certificate authority to SSH into a Linux VM" with no keys to distribute; or use Azure Bastion, after which "your virtual machines don't need a public IP address, agent, or special client software".

Know what the convenient command does besides the obvious. az vm user update --ssh-key-value is the other way to install a key, and it "appends the new public key text to the ~/.ssh/authorized_keys file for the admin user on the VM. This does not replace or remove any existing SSH keys." But the VMAccess extension behind it also "grants sudo permission to account specified in the username field", and it "changes ChallengeResponseAuthentication to no and PasswordAuthentication to yes". Turning on password authentication is an odd side effect of a command you reached for to install a key.

Check both NSG levels, and remember the rule is not symmetric. For inbound, "both NSG1 and NSG2 must include a rule that allows port 80 from the internet". For outbound, "either one of or both of the network security groups must have a rule that denies port 80". Microsoft's current advice is to sidestep it: "avoid associating network security groups with both a subnet and its network interfaces at the same time."

What I learned

Both tasks had state in them before I started. Kubernetes supplied requests to match my limits, which quietly upgraded the pod's QoS class. cloud-init had written a line into root's authorized_keys that would have beaten anything I appended after it.

The second one is where I had to change my mind about something I thought I knew. The behaviour I was relying on turns out not to be documented anywhere in the sshd manual, and the fix was to write the script so the answer does not matter.

So here is the Day 63 question. What is already configured in your environment that you have never read, because it was put there before you got the machine?

Day 63 down. Thirty-seven to go.