I failed today's AWS task on the first attempt, with every resource created correctly and every status field green. That is the more useful half of this post, so it gets the space.
The Docker half sets it up neatly, because both are about the gap between a label and the thing it labels.
One Docker task, one AWS task. Pull an image and give it a second tag, then build and run a container on ECS Fargate. The tasks come from the KodeKloud Engineer platform.
docker tag copies nothing
docker pull <image>:<tag>
docker tag <image>:<tag> <image>:<new-tag>
docker images
Run that and docker images shows two rows, two different tags, and one identical IMAGE ID. Disk usage does not move. docker tag did not duplicate anything; it added a second name pointing at the same image.
Once that lands, two other behaviours stop being surprising. docker rmi on one of two tags prints Untagged: rather than Deleted:, because removing a name is not removing an image, and the layers survive while anything else references them. And an image reference is really [registry/][namespace/]repository[:tag], so docker pull ubuntu silently means docker.io/library/ubuntu:latest.
That last one is the bridge to the AWS half, and it is worth stating plainly: the registry is part of the name. Not a setting, not a flag, part of the string.
The other thing worth carrying: a tag is a mutable pointer. The same tag can point at a different image next month, and docker pull on an unchanged tag can return different layers. image@sha256:... pins a digest and is the only reference that is genuinely immutable.
Every resource correct, task failed
Six resources: an ECR repository, an image built and pushed, a Fargate cluster, a task definition, a service, and a running task.
All six came up clean. The service read ACTIVE. runningCount matched desiredCount. The task read RUNNING with no stoppedReason. The container had pulled from ECR without complaint. I checked all of that, concluded it worked, and submitted.
Application is not accessible
Nothing had opened port 80. The service used the default security group, which permits inbound traffic only from itself. assignPublicIp=ENABLED had given the task a public IP, so it could reach out to ECR to pull the image, but security groups are stateful and directional: outbound worked, inbound never did. nginx was listening on a routable address that no packet could arrive at.
One rule, and it belongs before the service is created rather than as a repair afterwards:
aws ec2 authorize-security-group-ingress \
--group-id $DEF_SG --protocol tcp --port 80 --cidr 0.0.0.0/0
This is the same rule I wrote about two days ago as the load-bearing part of the ALB task. Recognising it in one task and not applying it in the next is the actual failure here. Nothing about it is ECS-specific.
What RUNNING actually claims
RUNNING is a statement about a container's lifecycle. The image pulled, the process started, it has not exited. It says nothing whatsoever about whether anything can reach it. Two independent facts, and only one of them was what the task asked for.
The rule I took from it: the final check has to exercise the thing the task promises, not the thing that is easiest to query. A status field is easy to query. It is not the deliverable.
For Fargate that check is slightly awkward, which is probably why it gets skipped. A task does not carry its IP; it gets its own network interface, and the address is two lookups away:
ENI=$(aws ecs describe-tasks --cluster datacenter-cluster --tasks $TASK_ARN \
--query "tasks[0].attachments[0].details[?name=='networkInterfaceId'].value" --output text)
TASK_IP=$(aws ec2 describe-network-interfaces --network-interface-ids $ENI \
--query 'NetworkInterfaces[0].Association.PublicIp' --output text)
curl -s -m 10 http://$TASK_IP/
With EC2 the public IP is one field on the instance. Here it is task to ENI, ENI to address. Worth building the habit anyway, because without a load balancer in front it is the only way to prove the deployment works.
Three ECS specifics that cost real time
The registry is the address. docker push datacenter-ecr:latest is not a misconfigured push to ECR, it is a push to Docker Hub under a repository of that name, and it fails on authorization moments after you watched ECR say Login Succeeded. Re-tag with the full URI first. Same fact as the Docker half, in a place where it bites.
Two roles, and they are not the same job. executionRoleArn is used by the Fargate infrastructure before your container exists: it pulls the image and creates log streams. taskRoleArn is used by your application code once it is running. The trust principal for both is ecs-tasks.amazonaws.com, not ecs.amazonaws.com. Both principals exist and mean different things, and naming the wrong one produces a role that looks correct in the console and can never be assumed by a task.
CannotPullContainerError has two unrelated causes. A missing or wrong execution role produces it. So does a task with no route to ECR, because Fargate pulls the image over the network like any other client. The distinguishing detail is the wording underneath: an authorization failure means the role, a timeout means networking. Reading only the headline sends you to the wrong half of the problem.
And one piece of pure trivia that is not trivia when it fails. awsvpcConfiguration={subnets=[...]} takes a comma-separated list. --subnets on create-load-balancer takes space-separated values. Two conventions in the same CLI, and mixing them gives you a parse error rather than anything that explains itself. I found that one by getting it wrong rather than by reading it anywhere.
The label is not the thing
A tag names an image, and adding one copies nothing. RUNNING names a lifecycle state, and it promises nothing about reachability. In both cases the label is accurate and answers a narrower question than the one being asked of it.
The cost of the first is a moment of confusion in docker images. The cost of the second was a failed submission on work that was otherwise complete.
So here is the Day 38 question. When you call something done, is it because the deliverable responded, or because a field said the word you were hoping to see?
Day 38 down. Sixty-two to go.
Top comments (0)