Day 36: A Running Container Is Not a Published One, and an ALB Needs a Listener

Both of today's tasks end with something that is definitely running and definitely unreachable, for two different reasons. A container serving nginx that no packet on the host can reach. A load balancer with a working DNS name that refuses every connection.

Neither is broken. Both are missing a connector nobody told you to create.

One Docker task, one AWS task. Run an nginx container on App Server 2, then front an EC2 instance with an Application Load Balancer. The tasks come from the KodeKloud Engineer platform.

docker run, and the flag that is not there

docker run -d --name nginx_2 nginx:alpine
docker ps -a

Three operations in one. docker run pulls the image if it is not local, creates a container from it, and starts that container. docker create plus docker start is the long form, worth knowing because it lets you configure a container before it ever runs.

-d detaches. Without it, the container's output takes over your terminal and Ctrl+C stops the container, which is a memorable way to discover what the flag does.

--name is not cosmetic. Skip it and Docker invents something like dazzling_kepler, and every command afterwards references an ID you have to go and look up. Names are also unique, so reusing one needs a docker rm first.

What is not in that command is -p. The task did not ask for it, so I did not add it, and the result is a container where nginx is genuinely listening on port 80 inside its own network namespace, and nothing on the host can reach it. Publishing is -p <host>:<container>, and it is the only thing that opens a host port. An EXPOSE line in an image does not do it either; that is documentation.

Two smaller things worth carrying. docker ps lists running containers only, and docker ps -a includes the ones that exited. A container that started and died instantly is invisible to the first and obvious in the second, which is the first thing to check when docker run appears to have done nothing at all. And alpine variants are the same software on a much smaller base: faster to pull, fewer packages to have vulnerabilities in, and noticeably barer when you exec in and discover there is no bash and no curl.

The ALB task is a security group task wearing a costume

Five resources: a security group, an instance running nginx from user data, a target group, a load balancer, and a listener. All five were straightforward. The task actually lives in two lines that look like boilerplate.

Internet  --80-->  default SG (on the ALB)  --80-->  devops-sg (on the EC2)

aws ec2 authorize-security-group-ingress \
  --group-id $DEVOPS_SG --protocol tcp --port 80 --source-group $DEF_SG

aws ec2 authorize-security-group-ingress \
  --group-id $DEF_SG --protocol tcp --port 80 --cidr 0.0.0.0/0

The first is the stated instruction: open port 80 for the default security group. --source-group rather than a CIDR, so the instance accepts traffic from anything wearing the default SG and from nothing else on the internet. That is the whole point of putting a load balancer in front of something.

The second comes from the throwaway clause "make appropriate changes in the default security group if necessary". It is necessary. The ALB carries the default SG, and a default security group only permits inbound traffic from itself. Without that rule the ALB resolves in DNS and then nothing happens. Security groups drop disallowed packets rather than rejecting them, so the SYN vanishes and curl sits on connect until it times out instead of coming back refused. Every resource you can inspect looks correct.

Reading a task for the rule it implies rather than the rule it states is most of the skill.

The listener is the only thing joining two halves

aws elbv2 create-listener --load-balancer-arn $ALB_ARN \
  --protocol HTTP --port 80 \
  --default-actions Type=forward,TargetGroupArn=$TG_ARN

A load balancer and a target group created separately have no relationship at all. The listener is what connects them, and there is no default. Skip it and describe-load-balancers looks healthy, describe-target-health looks healthy, and nothing works.

The console creates the listener as part of the ALB form, which is exactly why it is easy to miss from the CLI.

Two more ALB facts worth having in advance. It requires subnets in at least two availability zones, because it is a distributed service and will not create with one. And registering a target does not wait for the target to be ready:

aws elbv2 register-targets --target-group-arn $TG_ARN --targets Id=$EC2_ID

That succeeded seconds after run-instances, while the instance was still pending. Registration is a membership record, not a readiness check. The target sits initial with reason Elb.RegistrationInProgress, moves to unhealthy with Target.Timeout once checks start failing, and only reaches healthy when they pass. So register immediately and let the health checker do its job rather than adding dead time.

While we are here, the health check path deserves more thought than it usually gets:

--health-check-protocol HTTP --health-check-path /

/ works because nginx ships a default index page. Deploy an application that returns 404 at / and every target goes unhealthy while the application runs perfectly. The health check asks a different question than "is the process up".

Two habits from the finish

First, chain the waiters so the terminal is unattended until there is a result:

aws elbv2 wait load-balancer-available --load-balancer-arns $ALB_ARN \
  && aws elbv2 wait target-in-service --target-group-arn $TG_ARN --targets Id=$EC2_ID \
  && curl -s http://$ALB_DNS/ | head -5

target-in-service is the one that matters. It polls until the target reads healthy, which is the first moment the entire path is proven rather than assumed.

Second, when it times out, read the reason rather than guessing:

aws elbv2 describe-target-health --target-group-arn $TG_ARN \
  --query 'TargetHealthDescriptions[].{State:TargetHealth.State,Reason:TargetHealth.Reason}'

Target.Timeout means the security group chain is broken or nginx is not listening. Target.ResponseCodeMismatch means it answered with the wrong status. Elb.RegistrationInProgress means wait. Three very different problems that all present as "not working yet" if you only look at the state.

One last detail I have started applying everywhere. Resolve the AMI rather than hardcoding it:

aws ssm get-parameters --names \
  /aws/service/canonical/ubuntu/server/22.04/stable/current/amd64/hvm/ebs-gp2/ami-id \
  --query 'Parameters[0].Value' --output text

Canonical publishes current Ubuntu AMI IDs as public SSM parameters. AMI IDs differ per region and change with every image release, so a hardcoded one is wrong somewhere, or eventually.

Running is not reachable

The container was running and unpublished. The ALB was provisioned and unlistened. In both cases every component existed and the thing joining it to the outside did not.

The tell is the same in both: a status check that looks fine. docker ps says Up. describe-load-balancers says active. Neither is a claim about whether a packet can arrive.

So here is the Day 36 question. For the last thing you deployed, what did you check at the end, and was it a status field or an actual request?

Day 36 down. Sixty-four to go.

Comments

[Vinh Nguyen]

The --source-group $DEF_SG rule says what you want it to say only while the default SG stays small. The second rule opens that same SG to 0.0.0.0/0 on 80, and every instance launched in the VPC without an explicit SG lands in the default SG by definition, so "accepts traffic from anything wearing the default SG" is a larger set than the load balancer. The guarantee that survives is "nothing else on the internet reaches it directly", which is a real guarantee and not the one the sentence reads as. Cheapest split is giving the ALB a security group of its own, and the reason the task nudges the other way is that it says "make changes in the default SG if necessary" rather than "give the ALB its own".