Day 49: A Deployment Keeps Pods Alive, and Routable Is Not Reachable

Today was layers. A Deployment that does not run pods itself, but owns a ReplicaSet that does. A peering connection that makes a subnet routable, a security group that decides whether it is reachable, and an SSH hop in between that did not carry the key I handed it.

One Kubernetes task, one AWS task. Run an application as a Deployment, then move a log file from a private VPC to S3 through a peered public one. The tasks come from the KodeKloud Engineer platform.

The layer you forget is there

kubectl create deployment httpd --image=httpd:latest
kubectl get deployments
kubectl get replicasets
kubectl get pods

Kubernetes describes a Deployment as providing declarative updates for Pods and ReplicaSets, and the ReplicaSet is the part that answers yesterday's question. A ReplicaSet replaces pods that are deleted or terminated for any reason. Delete one of this Deployment's pods and a replacement appears within seconds, with a different name.

The names give the structure away. A Deployment's pods are called something like httpd-<hash>-<suffix>, where the hash identifies the ReplicaSet that made them. The name changes every time a pod is replaced, so nothing should ever depend on it.

What the Deployment actually uses to find its pods is a label. kubectl create deployment gives them app=httpd, and kubectl get pods -l app=httpd lists exactly this Deployment's pods and nothing else.

Two facts worth having before Day 51. Scaling is not an update: Kubernetes is explicit that a rollout is triggered only when the pod template changes, so kubectl scale just changes a count. And old ReplicaSets are kept after an update, ten by default, which is what makes rolling back possible at all.

Four things have to say yes

The AWS task had an instance with no internet access writing a log file, which had to end up in S3. The route was across a VPC peering connection to an instance that did have internet access, and from there to the bucket.

The private instance had no public IP, no NAT and no internet route. The only way to reach it at all was through the public one, over the peering connection, so the networking had to be finished before a single line of configuration could reach the private side. The plumbing was not a step, it was the prerequisite.

And peering turned out to need four separate things, each documented, each capable of failing on its own:

1. CIDRs that do not overlap. AWS will not peer VPCs with matching or overlapping ranges, and there is no translation option. The private VPC was 10.10.0.0/16, so the new one became 10.20.0.0/16.
2. Acceptance. Even within one account the connection sits in pending-acceptance until someone accepts it. You can create and accept it yourself, but you do have to accept it.
3. Routes on both sides. AWS requires a route in the route tables for both instances' subnets. One direction only gives you packets that arrive and replies that cannot get home, and the symptom is a timeout identical to having no route at all.
4. Security groups. The one people miss after the routes are right. AWS's own guidance is to update the security group rules so traffic to and from the peer VPC is not restricted.

The fourth is the title. Peering made the private instance routable from the public VPC. Without a security group rule allowing 10.20.0.0/16, SSH would have been dropped at the instance with the routing working perfectly. Routes and security groups are separate layers and both have to agree.

It is also worth knowing that peering does not chain. AWS says plainly that it does not support transitive peering relationships, so a third VPC peered to the public one would still have no path to the private one.

The key that did not make the hop

This is the one that actually cost time:

ssh -i /root/.ssh/datacenter-key.pem -J ubuntu@$PUB_IP ubuntu@10.10.1.114 'hostname'

ubuntu@32.196.116.111: Permission denied (publickey).

Seconds earlier, the same key had logged into that same public instance directly. The error names the jump host, which is the only clue that the private instance was never reached.

The ssh manual has the answer, in a note under -J: configuration directives supplied on the command line generally apply to the destination host and not to any jump hosts, and you should use ~/.ssh/config to configure jump hosts. So -i went to the private instance. The public one was only offered the default key files and anything in an ssh agent, and none of those was right.

The fix is to stop passing it on the command line:

Host pubec2
    HostName 32.196.116.111
    User ubuntu
    IdentityFile /root/.ssh/datacenter-key.pem

Host privec2
    HostName 10.10.1.114
    User ubuntu
    IdentityFile /root/.ssh/datacenter-key.pem
    ProxyJump pubec2

Every hop carries its own identity, and ssh privec2 works as though the private instance were directly reachable.

One more silent failure

The log shipping ran from /etc/cron.d, and that directory has a trap:

* * * * * root scp ... /var/log/boots.log ubuntu@10.20.1.134:/home/ubuntu/boots.log

Six fields before the command, not five. The crontab manual describes five time fields followed by a username in the system crontab, and says jobs in cron.d are system jobs that need that username. A crontab -e entry has no user field because the user is whoever owns the crontab. Put a five-field line in /etc/cron.d, and cron reads scp as a username, logs a failure to syslog, and never runs anything.

Checking each stage

The file landed. I checked both halves separately, the copy on the public instance and the object in S3, because on a two-stage pipeline that one extra ls halves the search when something breaks. Nothing on the public instance points at the peering, the security group or the first cron job. A file there but not in S3 points at the role or the second.

I should also name what the task required and what I would not repeat. Both instances used the same key pair, so the private instance ended up holding a private key that also opens the public one. Compromise either and you have both. In anything real, this would be SSM Session Manager and CloudWatch Logs, and no instance would hold another's credentials.

Each layer has its own no

The Deployment works because a layer I rarely look at does the replacing. The peering works because four independent layers each agreed. And the SSH hop failed because an option I thought was global only applied to one layer of the connection.

So here is the Day 49 question. When something is unreachable, how many separate layers are you checking before you decide which one is wrong?

Day 49 down. Fifty-one to go.