Today was the first troubleshooting task of the run rather than a build task, and it needs the opposite instinct. Build tasks reward moving. Repair tasks punish it. Change nothing until you know the cause.
What made it satisfying is that every system involved told me exactly what was wrong, in its own vocabulary, before I touched anything.
One Docker task, one AWS task. Configure Apache inside a running container with docker exec, then find out why an EC2 instance running nginx is unreachable. The tasks come from the KodeKloud Engineer platform.
docker exec, and why your shell leaving does not kill anything
docker exec -it kkloud bash
exec starts an additional process inside a container that is already running. It does not restart the container and it does not disturb PID 1. A stopped container has nothing to exec into, so docker start comes first.
-i and -t do different jobs and you need both. -i keeps stdin connected so the process can read what you type; -t allocates a pseudo-terminal so prompts and line editing behave. docker exec kkloud bash without them exits immediately with no output, which looks like a broken container and is not.
Inside, the first surprise for anyone used to a normal machine:
service apache2 restart # works
systemctl restart apache2 # System has not been booted with systemd as init system
Most images have no init system and nothing resembling systemd as PID 1. The SysV service wrapper still works because it calls the init script directly. That error message is precise and still manages to look like a broken installation.
Then the thing that surprises people in the other direction. Typing exit ends your shell and the container keeps running:
exit
docker ps # still there
A container's life is tied to PID 1, not to your session. This is the difference from docker attach, which connects you to PID 1 itself. Docker documents Ctrl+C there as terminating the container when it was started without -i and -t. exec gives you a second process; attach gives you the first one.
Two config files had to agree, which is a theme that comes back on Day 41. ports.conf decides what Apache listens on, and the virtual host block decides what it answers for. Change one and not the other and you get a server bound correctly that serves nothing.
And everything done this way lives in the container's writable layer. It vanishes with docker rm, and a second container from the same image starts without any of it. That is the argument for a Dockerfile, which is where the track goes next.
000 is a diagnosis
The AWS task: xfusion-ec2 runs nginx, the security group allows 80, the instance is running with a public IP, nothing reaches it.
The first command was not an AWS call.
curl -s -m 8 -o /dev/null -w "http_code=%{http_code}\n" http://98.82.179.65/
http_code=000
000 from curl means no HTTP response at all: the TCP connection never established. That single number rules out an entire category of causes before touching the AWS API.
A 403 or a 502 would mean packets arrive and something answers, which makes it an nginx or application problem. 000 means nothing is listening, or nothing can reach what is listening. Network path. Eight seconds well spent.
Sweep everything, then change one thing
An inbound packet to an EC2 instance crosses five checkpoints, and any one of them breaks it identically from outside: an internet gateway attached to the VPC, a default route pointing at it, the subnet associated with that route table, a NACL allowing the port, and a security group allowing the port. Plus a public IP, plus a process actually listening.
I checked all of them before changing anything. That discipline is the whole lesson, and it is not about being thorough for its own sake. Guess-and-check on a VPC means making changes to things that were never broken, and by the time it works you cannot say what fixed it or whether you left something worse behind.
The sweep found this:
{ "Dest": "0.0.0.0/0", "GW": "igw-0cf9ecd79f552b081", "State": "blackhole" }
blackhole is the most informative word in the entire output. It means the route exists and its target does not resolve: the gateway is deleted, detached, or otherwise unusable. AWS is telling you precisely what is wrong, in a field most people scroll past because the route itself looks correct.
Everything else was clean. The security group allowed 80 from anywhere. The NACL was the default open configuration. The instance was running with a public IP. The subnet was associated with the right route table. One thing was broken, and it announced itself.
Detached and deleted look identical
aws ec2 describe-internet-gateways --internet-gateway-ids igw-0cf9ecd79f552b081
# { "Attachments": [] }
The gateway existed with no attachments. Detached, not deleted. So the fix was one command:
aws ec2 attach-internet-gateway --internet-gateway-id igw-0cf9ecd79f552b081 --vpc-id $VPC
And the route went from blackhole to active on its own. No route edit, no re-association. The route was always valid; its target simply became reachable again.
Had the gateway been deleted, this would have been three commands, because a new IGW gets a new ID and the existing route would still point at the dead one. Same symptom, same first diagnosis, different remedy, and one describe call to tell them apart.
Why creating the missing thing would have failed
This is the part worth the post. The obvious move on "make it accessible" is to create an internet gateway and attach it, since the VPC visibly had none. That would have succeeded as a sequence of API calls and left the site broken.
The existing route points at igw-0cf9ecd79f552b081 specifically. A different gateway does not change what that route targets, so it would have stayed blackhole. You would end up with a working IGW, an orphaned one, a still-broken site and no idea why. A VPC also permits only one attached internet gateway, so the second attach fails once the first succeeds.
Creating the missing resource is not the same as fixing the broken reference.
There was a red herring too, worth naming because it is exactly the kind of thing that pulls you off course:
MapPublicIpOnLaunch: False
That looks wrong for a public subnet. It is not the problem. It only controls whether instances launched into the subnet later get an automatic public IP, and this instance already had one. Changing it would have been a no-op, and fixing incidental things during an incident makes it harder to say afterwards what actually resolved it.
Then close with the command you opened with:
curl -s -m 10 http://98.82.179.65/
# <title>Welcome to nginx!</title>
Same command, different answer. That symmetry is the point: the closing check should be the opening check, so the fix is demonstrated rather than assumed.
Everything told me
System has not been booted with systemd names its own cause. http_code=000 says the connection never opened. blackhole says the route's target does not exist. Three different systems, three precise messages, all of them easy to skim past in favour of a guess.
There is one thing I still want to know that the task did not ask for. An internet gateway does not detach itself, so somebody or something detached it, and that is a CloudTrail question. Fixing the symptom without finding out means it happens again.
So here is the Day 40 question. The last outage you resolved, can you name what caused it, or only what you changed?
Day 40 down. Sixty to go.
Top comments (1)
Wow 40 days complete ✅