DEV Community: Nnamdi Felix Ibe

Day 38: A Tag Is a Name, Not a Copy, and RUNNING Is Not Reachable

Nnamdi Felix Ibe — Thu, 10 Sep 2026 17:00:00 +0000

I failed today's AWS task on the first attempt, with every resource created correctly and every status field green. That is the more useful half of this post, so it gets the space.

The Docker half sets it up neatly, because both are about the gap between a label and the thing it labels.

One Docker task, one AWS task. Pull an image and give it a second tag, then build and run a container on ECS Fargate. The tasks come from the KodeKloud Engineer platform.

docker tag copies nothing

docker pull <image>:<tag>
docker tag <image>:<tag> <image>:<new-tag>
docker images

Run that and docker images shows two rows, two different tags, and one identical IMAGE ID. Disk usage does not move. docker tag did not duplicate anything; it added a second name pointing at the same image.

Once that lands, two other behaviours stop being surprising. docker rmi on one of two tags prints Untagged: rather than Deleted:, because removing a name is not removing an image, and the layers survive while anything else references them. And an image reference is really [registry/][namespace/]repository[:tag], so docker pull ubuntu silently means docker.io/library/ubuntu:latest.

That last one is the bridge to the AWS half, and it is worth stating plainly: the registry is part of the name. Not a setting, not a flag, part of the string.

The other thing worth carrying: a tag is a mutable pointer. The same tag can point at a different image next month, and docker pull on an unchanged tag can return different layers. image@sha256:... pins a digest and is the only reference that is genuinely immutable.

Every resource correct, task failed

Six resources: an ECR repository, an image built and pushed, a Fargate cluster, a task definition, a service, and a running task.

All six came up clean. The service read ACTIVE. runningCount matched desiredCount. The task read RUNNING with no stoppedReason. The container had pulled from ECR without complaint. I checked all of that, concluded it worked, and submitted.

Application is not accessible

Nothing had opened port 80. The service used the default security group, which permits inbound traffic only from itself. assignPublicIp=ENABLED had given the task a public IP, so it could reach out to ECR to pull the image, but security groups are stateful and directional: outbound worked, inbound never did. nginx was listening on a routable address that no packet could arrive at.

One rule, and it belongs before the service is created rather than as a repair afterwards:

aws ec2 authorize-security-group-ingress \
  --group-id $DEF_SG --protocol tcp --port 80 --cidr 0.0.0.0/0

This is the same rule I wrote about two days ago as the load-bearing part of the ALB task. Recognising it in one task and not applying it in the next is the actual failure here. Nothing about it is ECS-specific.

What RUNNING actually claims

RUNNING is a statement about a container's lifecycle. The image pulled, the process started, it has not exited. It says nothing whatsoever about whether anything can reach it. Two independent facts, and only one of them was what the task asked for.

The rule I took from it: the final check has to exercise the thing the task promises, not the thing that is easiest to query. A status field is easy to query. It is not the deliverable.

For Fargate that check is slightly awkward, which is probably why it gets skipped. A task does not carry its IP; it gets its own network interface, and the address is two lookups away:

ENI=$(aws ecs describe-tasks --cluster datacenter-cluster --tasks $TASK_ARN \
  --query "tasks[0].attachments[0].details[?name=='networkInterfaceId'].value" --output text)

TASK_IP=$(aws ec2 describe-network-interfaces --network-interface-ids $ENI \
  --query 'NetworkInterfaces[0].Association.PublicIp' --output text)

curl -s -m 10 http://$TASK_IP/

With EC2 the public IP is one field on the instance. Here it is task to ENI, ENI to address. Worth building the habit anyway, because without a load balancer in front it is the only way to prove the deployment works.

Three ECS specifics that cost real time

The registry is the address. docker push datacenter-ecr:latest is not a misconfigured push to ECR, it is a push to Docker Hub under a repository of that name, and it fails on authorization moments after you watched ECR say Login Succeeded. Re-tag with the full URI first. Same fact as the Docker half, in a place where it bites.

Two roles, and they are not the same job. executionRoleArn is used by the Fargate infrastructure before your container exists: it pulls the image and creates log streams. taskRoleArn is used by your application code once it is running. The trust principal for both is ecs-tasks.amazonaws.com, not ecs.amazonaws.com. Both principals exist and mean different things, and naming the wrong one produces a role that looks correct in the console and can never be assumed by a task.

CannotPullContainerError has two unrelated causes. A missing or wrong execution role produces it. So does a task with no route to ECR, because Fargate pulls the image over the network like any other client. The distinguishing detail is the wording underneath: an authorization failure means the role, a timeout means networking. Reading only the headline sends you to the wrong half of the problem.

And one piece of pure trivia that is not trivia when it fails. awsvpcConfiguration={subnets=[...]} takes a comma-separated list. --subnets on create-load-balancer takes space-separated values. Two conventions in the same CLI, and mixing them gives you a parse error rather than anything that explains itself. I found that one by getting it wrong rather than by reading it anywhere.

The label is not the thing

A tag names an image, and adding one copies nothing. RUNNING names a lifecycle state, and it promises nothing about reachability. In both cases the label is accurate and answers a narrower question than the one being asked of it.

The cost of the first is a moment of confusion in docker images. The cost of the second was a failed submission on work that was otherwise complete.

So here is the Day 38 question. When you call something done, is it because the deliverable responded, or because a field said the word you were hoping to see?

Day 38 down. Sixty-two to go.

Day 37: The Role Is Not What You Attach, and the Bucket Is Not the Objects

Nnamdi Felix Ibe — Tue, 08 Sep 2026 17:00:00 +0000

Two things today turned out to be one thing wrapped in another. You do not attach an IAM role to an EC2 instance, you attach an instance profile that holds the role. And an S3 bucket and the objects inside it are separate resources with separate ARNs, which is why a policy that looks complete can only do half the work.

One Docker task, one AWS task. Copy a file into a running container, then give an EC2 instance permission to use an S3 bucket without putting a single credential on disk. The tasks come from the KodeKloud Engineer platform.

docker cp, and where the file actually goes

docker cp /tmp/<file> ubuntu_latest:/opt/

There is no direction flag. The colon decides it: docker cp SRC container:DEST copies in, docker cp container:SRC DEST copies out. That is the entire syntax.

Two behaviours worth knowing before you rely on it. Docker does not create parent directories for the destination, so /opt has to already exist inside the container, which is why checking first is worth a command. And ownership is not preserved: files copied into a container are created owned by root, and files copied out are owned by whoever ran the command. docker cp -a preserves the source's uid and gid instead. An application running as a non-root user inside the container can find the file present and unreadable, which is a confusing five minutes if you did not expect it.

Then the verification, which is the habit rather than the command:

sha256sum /tmp/<file>
docker exec ubuntu_latest sha256sum /opt/<file>

Comparing ls -l on both sides proves the sizes match. It does not prove the bytes match. One extra command turns "it looks like it worked" into "it worked".

One structural point that leads into the next two days. The copied file lands in the container's writable layer. It is not part of the image, it does not survive docker rm, and a second container from the same image starts without it. Making a file part of an image means COPY in a Dockerfile, or committing the container. For a file that has to outlive the container and be shared with the host, the answer is a volume at run time, not a copy afterwards.

The wrapper nobody mentions

The AWS task was to let an EC2 instance read and write to one S3 bucket, via a role, with no access keys anywhere.

The console makes this feel like one step. Create a role, pick EC2 as the trusted entity, attach it to the instance. On the CLI, it is three resources, because you cannot attach a role to an instance at all:

aws iam create-role --role-name xfusion-role --assume-role-policy-document '...'
aws iam create-instance-profile --instance-profile-name xfusion-role
aws iam add-role-to-instance-profile --instance-profile-name xfusion-role --role-name xfusion-role

aws ec2 associate-iam-instance-profile \
  --instance-id $EC2_ID --iam-instance-profile Name=xfusion-role

An instance profile is a container that holds exactly one role, and it is the thing EC2 actually accepts. The console creates one silently and gives it the same name as the role, which is why almost nobody knows it exists until the CLI says:

Invalid IAM Instance Profile name

That error reads like a typo. It means the resource does not exist.

Two ARNs, because they are two resources

Here is the policy, and the shape of it is the lesson:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject"],
      "Resource": "arn:aws:s3:::xfusion-s3-458492027197/*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::xfusion-s3-458492027197"
    }
  ]
}

Two statements for three actions, because those actions operate on two different resource types. GetObject and PutObject act on objects, so the ARN carries /*. ListBucket acts on the bucket itself, so the ARN does not.

Collapse them into one statement, and exactly half of it works. With only the /* ARN, aws s3 cp succeeds and aws s3 ls returns AccessDenied. With only the bare ARN, the reverse. Both failures look like a broken policy in general rather than a resource-granularity mistake, which is what makes this worth committing to memory.

Note what is not in there: no s3:DeleteObject, no s3:*, no wildcard bucket. Put, get and list in one bucket.

It goes on as an inline policy rather than a managed one, because it is scoped to a single bucket and has no reason to be reusable:

aws iam put-role-policy --role-name xfusion-role \
  --policy-name xfusion-s3-access --policy-document file://policy.json

Ask who you are before you ask what you can do

The instinct after attaching a role is to try the thing you wanted to do. Better to ask a smaller question first:

ssh root@$EC2_IP 'aws sts get-caller-identity'

{ "Arn": "arn:aws:sts::458492027197:assumed-role/xfusion-role/i-04e16a83d51bd745a" }

assumed-role/xfusion-role/<instance-id> is the confirmation. The instance pulled temporary credentials from the metadata service, and the session name is its own instance ID.

Splitting the check this way separates two failure modes that produce an identical AccessDenied: the role is not attached or has not propagated, so there is no identity at all; or the role is attached, and the policy is wrong. One extra command turns an ambiguous error into a specific one.

Then the actual proof:

upload: ./testfile.txt to s3://xfusion-s3-458492027197/testfile.txt
2026-08-28 04:25:41         23 testfile.txt

There is no ~/.aws/credentials on that instance, and aws configure was never run. The credential chain works through a list of sources in precedence order, with environment variables near the top, the shared credentials and config files in the middle, and EC2 instance metadata last of all. Everything above IMDS is empty here, so it falls through to the metadata service and gets temporary credentials that rotate on their own.

That is the real point of the task. A long-lived access key on an instance can leak through a backup, an AMI, a log, or a compromised process, and it stays valid until somebody notices. Role credentials expire by themselves and are scoped to one instance.

One regional oddity while we are in S3, since it costs people time:

aws s3api create-bucket --region us-east-1 --bucket $BUCKET

No --create-bucket-configuration. us-east-1 is the API default and passing a location constraint for it returns InvalidLocationConstraint. Every other region requires it.

Granularity is the whole game

The instance profile exists because EC2 needs a wrapper around a role. The two ARNs exist because a bucket and its contents are genuinely different things. In both cases, the abstraction that felt like one object is two, and the error message when you get it wrong describes the symptom rather than the structure.

So here is the Day 37 question. In the last IAM policy you wrote, do you know which statements act on a container and which act on its contents, or did it work the first time and you moved on?

Day 37 down. Sixty-three to go.

I asked Gemini what skill a charity was missing. It got it wrong

Nnamdi Felix Ibe — Sun, 06 Sep 2026 21:39:38 +0000

This is a submission for Weekend Challenge: Generosity Edition

What I Built

Handover takes a plain description of what is going wrong inside a small charity and works out the role that would fix it.

Not the role they asked for. The one they actually need.

You type something like "our books are a mess, and we have missed two filing deadlines". It comes back with a full trustee role: the diagnosis, what the person would do, a deliberately short list of essential skills, an honest time commitment, and an advert you can paste straight into your newsletter.

Then a volunteer pastes their CV, badly, and gets scored against every open role with a reason and an honest note on where the fit is thin.

Why

A couple of days ago I got an email saying Reach Volunteering is closing after 45 years. It genuinely hurt to read.

Reach connected small UK charities with people who wanted to give them professional skills. Last year it placed 5,996 volunteers and trustees across 2,440 organisations. The people it placed contributed around £60 million in expertise. Ninety-six per cent of those organisations ran on under £1 million a year, and nearly half on under £50,000.

It is not closing because the work stopped mattering. It is closing because funding for the infrastructure that helps small charities build capacity has dried up. Reach was the largest single source of trustees in the sector, and it is shutting at the peak of its impact.

I volunteer as a digital navigator, which mostly means sitting with people who have been handed a system that assumes a confidence nobody ever gave them. You watch someone decide they are the problem, when the thing in front of them was just badly built. Reach existed to stop small charities from feeling like that about their own gaps, and now it is shutting down.

I cannot rebuild 45 years of relationships in a weekend. So I picked the one piece of what Reach did that was pure expertise rather than headcount, and rebuilt that.

The thing everyone gets wrong

Every volunteering board opens with the same question. What role are you advertising?

Small charities cannot answer it. That is the actual problem.

A trustee who says "our books are a mess" is describing a symptom. The missing capability is financial oversight. Until somebody names that, the charity advertises for a bookkeeper, gets a bookkeeper, and still cannot answer a funder's question about restricted money. So it stops applying for the grants it could win, and nobody ever connects the two things.

Reach's real value was having someone experienced enough to hear the symptom and name the gap. That is the part I wanted to keep working.

So Handover refuses to ask what role you need. It asks what is going wrong, and reasons upward.

Demo

Live app: https://handover-nd-codes.vercel.app

The live site works with no API key at all. Six example organisations are already on the board, spread across the four nations, and you can read any of them aloud straight away.

Every canned result is labelled "demo output" on screen. A tool asking you to trust a diagnosis cannot pass off fixtures as one, so the labelling is not optional.

To see the real thing, paste a Gemini key into the Settings tab and describe a problem of your own. Keys stay in your browser and go straight to Google. There is no backend in this project, so there is nowhere else for them to go.

If you want the fastest honest test of whether this works, give it this and see what comes back:

We run a lunch club for older people twice a week out of a church hall. There are six of us, all volunteers, all over seventy. Our secretary died in March, and she was the one who knew how everything worked, including the booking with the church and where the insurance documents are. We are still running the lunches, but I do not think anyone could take over if I stopped.

Code

ND-codes / handover

Handover

Small charities rarely know which skill they are missing. They know what hurts.

Handover reads the symptom, names the capability underneath it, and writes the trustee or volunteer role that closes the gap.

Built for the DEV Weekend Challenge: Generosity Edition, September 2026.

Why this exists

On the week I started this, Reach Volunteering announced it is closing after 45 years.

Reach connected small UK charities with people who wanted to give them their professional skills. In 2025 alone it placed 5,996 volunteers and trustees, worked with 2,440 organisations, and the people it placed contributed around £60 million in expertise. Ninety-six per cent of those organisations had income under £1 million. Nearly half were under £50,000.

It is closing at the peak of its impact, because funding for infrastructure that helps small charities build capacity has dried up.

Handover is not a replacement for Reach. A weekend project…

View on GitHub

React 19, TypeScript and Vite, with no backend at all. Prompts and response schemas live in one module that both the app and the test suite import, for reasons I will get to.

How I Built It

Three Gemini calls, each with its own response schema. Diagnose a need, profile a volunteer, match one against the other.

Google AI: the first version did not work

I built the whole app against demo fixtures because I had no key yet. Scaffolding, matching UI, tiered results, all of it looking great on canned data. When I finally ran a real call, I gave it that lunch club above.

Gemini answered "Secretary".

It handed back the job title of the person who died. Which is exactly the reflex the entire project exists to interrupt. Refill the chair, leave the organisation just as fragile, wait for it to happen again.

Three things were wrong, and two of them were mine.

No worked examples. Nothing anchored the reasoning from symptom to gap, so the model did the obvious thing.

Most of the behaviour lives in the schema, not the prompt. The field descriptions are where the actual instructions are:

```ts
essentialSkills: {
  type: "ARRAY",
  items: { type: "STRING" },
  description:
    "At most three. Only what someone genuinely cannot do the role without. " +
    "Every extra item here shrinks the pool.",
},
```

I added two worked examples showing the lazy answer next to the good one, plus a hard rule against ever returning a title the charity already used.

A voice instruction bleeding across fields. My house style said, "write as the charity, using we". That was meant for the advert. It leaked into the diagnosis, which came back as "We lack documented records". That is the charity talking to itself, not an adviser telling a board something it had missed. Voice is scoped per field.

A spec stated in the wrong place. The advert length lived only in a schema description, and the model returned 55 words against a 150 to 220 word target. Restating it in the prompt body fixed it.

It now answers "Governance and Continuity Trustee" and gets there properly:

The missing capability is documented operational continuity and compliance oversight, not administrative secretarial support.

That is the difference between a tool and a toy, and I only caught it because I tested with a real key before writing this post.

Testing something that has no right answer

You cannot unit test a diagnosis. But you can test whether it avoids the obvious wrong one.

There is a smoke script with 13 assertions running against live Gemini. The one that matters feeds in the bereaved lunch club and asserts the answer is not "Secretary".

One check defeated regular expressions entirely. I wanted to confirm the diagnosis names the continuity risk, but the model phrases the same correct idea as "institutional memory", then "unwritten personal memory", then "one key resignation away". Keyword matching either missed good answers or had to be widened until it passed anything at all.

So the check became another model call, given both the shallow answer and the correct one:

```js
const judge = await ai.models.generateContent({
  model: MODEL,
  contents: [
    'A small charity said this:', `"${NEED}"`,
    'A tool was asked to name the capability they are actually missing,',
    'rather than the vacancy they described. It answered:',
    `Title: "${pack.title}"`,
    `Diagnosis: "${pack.diagnosis}"`,
    '',
    "The shallow answer here is 'they need a new secretary to do the admin'.",
    'The correct insight is that the organisation has no continuity:',
    'knowledge lived in one person, nothing is written down, and it is one',
    'more departure away from collapsing.',
    '',
    'Does the answer above reach the correct insight, rather than the',
    'shallow one? Judge the substance, not the wording.',
  ].join('\n'),
  config: {
    responseMimeType: 'application/json',
    responseSchema: {
      type: 'OBJECT',
      properties: {
        reachesInsight: { type: 'BOOLEAN' },
        reason: { type: 'STRING' },
      },
      required: ['reachesInsight', 'reason'],
    },
    temperature: 0,
  },
});
```

So that one check is a second Gemini call acting as a judge, given the shallow answer and the correct insight and asked which one the output reached. Slower and non-deterministic, but it tests the thing I actually care about instead of the words that happened to appear.

That is also why the prompts sit in a module both the app and the test import. A test that drifts from the prompt is worse than no test, because it reports on something nobody runs.

ElevenLabs: reading it aloud is the accessibility path

The role pack can be read aloud, and this is not decoration.

The people running £20,000 charities are often not confident readers of formal documents. A trustee advert is precisely the kind of text that makes somebody decide the role is not for them before they finish it. Hearing it spoken plainly moves that line.

I picked Alice, a British voice, over the usual default. This is a UK tool and the listener is often somebody who already feels talked down to by officialdom. An American narrator would be one more small signal that this was not built for them.

Testing with a real key caught a second bug. My hardcoded default was a library voice, and ElevenLabs returns 402 "Free users cannot use library voices" on a free plan. So the whole voice feature would have failed for anybody setting it up fresh, including any judge who tries it.

It now recovers instead of failing. On a 400, 402 or 404 it asks the account which voices it genuinely has, prefers a British one, retries, and remembers the answer. A 401 is a bad key and is not worth retrying.

There is also a Web Speech API fallback, so the accessibility feature is never the thing behind the paywall. That fallback had its own bug: it reported success without waiting for voices to load, which meant it could show "reading aloud" over complete silence. On the accessibility path that is the worst possible way to fail.

Decisions I will defend

Essential skills are capped at three. Every unnecessary requirement excludes somebody who could have done the job. The pool Reach built was over 40% under 40 and over 40% from ethnic minorities, well ahead of national trends, and bloated requirement lists are one of the mechanisms that quietly narrows a pool like that.

Match scores are forced to spread. A matcher returning everything at 78 has not judged anything. Real scores come back between 92 and 18, which is why the results split into three tiers instead of one ranked list.

Every match carries a "where it is thin" note. A volunteer who finds out in month three that the fit was wrong does not stay, and that costs the charity more than never recruiting them.

The weak matches stay visible. Collapsed, not hidden. Handing somebody six roles they cannot do is asking them to do the triage the tool was supposed to do for them, but hiding the rejections entirely is just flattery.

The example organisations are invented, and the app says so. Attaching made-up recruitment needs to real registered charities would put words in their mouths. The names are fictional. The shapes follow the real distribution Reach reported.

What it is not

The diagnosis is a starting point for a board conversation, not governance or legal advice. The model can be confidently wrong about which capability is missing, so the role pack is built to be edited rather than accepted.

Matching runs against roles held in one browser. This demonstrates the mechanism, it is not a live marketplace.

And it is not a replacement for Reach. Not affiliated with them either. It is one piece of what they did, kept working.

Prize Categories

Best Use of Google AI. Three schema-constrained Gemini calls carry the whole product, and the diagnosis prompt is the product. Most of the quality lives in the response schema field descriptions and two worked examples that stop the model from restating the charity's problem back at it. Tested by 13 live assertions, one of which uses Gemini itself as a judge.

Best Use of ElevenLabs. Text to speech as an accessibility route rather than a feature, aimed at people who find formal documents off-putting, with a British voice chosen deliberately, graceful recovery when an account cannot use a given voice, and a Web Speech API fallback so the accessible path is never the paid one.

The bit that bothers me

Reach closed because nobody would fund the unglamorous infrastructure underneath the sector. Matching software does not fix that. A funder deciding capacity building is worth paying for fixes that.

But the need did not disappear when the charity did. There are still thousands of small organisations who know exactly what hurts and cannot name the skill that would stop it hurting.

If you have run a budget, chaired anything, written a policy, or built a system other people depended on, a grassroots charity somewhere needs precisely that and has no way left to find you.

So which is it? Are you going to paste your CV into something like this and find out where you would actually be useful, or is that a job for somebody with more time than you?

Day 36: A Running Container Is Not a Published One, and an ALB Needs a Listener

Nnamdi Felix Ibe — Sun, 06 Sep 2026 04:57:57 +0000

Both of today's tasks end with something that is definitely running and definitely unreachable, for two different reasons. A container serving nginx that no packet on the host can reach. A load balancer with a working DNS name that refuses every connection.

Neither is broken. Both are missing a connector nobody told you to create.

One Docker task, one AWS task. Run an nginx container on App Server 2, then front an EC2 instance with an Application Load Balancer. The tasks come from the KodeKloud Engineer platform.

docker run, and the flag that is not there

docker run -d --name nginx_2 nginx:alpine
docker ps -a

Three operations in one. docker run pulls the image if it is not local, creates a container from it, and starts that container. docker create plus docker start is the long form, worth knowing because it lets you configure a container before it ever runs.

-d detaches. Without it, the container's output takes over your terminal and Ctrl+C stops the container, which is a memorable way to discover what the flag does.

--name is not cosmetic. Skip it and Docker invents something like dazzling_kepler, and every command afterwards references an ID you have to go and look up. Names are also unique, so reusing one needs a docker rm first.

What is not in that command is -p. The task did not ask for it, so I did not add it, and the result is a container where nginx is genuinely listening on port 80 inside its own network namespace, and nothing on the host can reach it. Publishing is -p <host>:<container>, and it is the only thing that opens a host port. An EXPOSE line in an image does not do it either; that is documentation.

Two smaller things worth carrying. docker ps lists running containers only, and docker ps -a includes the ones that exited. A container that started and died instantly is invisible to the first and obvious in the second, which is the first thing to check when docker run appears to have done nothing at all. And alpine variants are the same software on a much smaller base: faster to pull, fewer packages to have vulnerabilities in, and noticeably barer when you exec in and discover there is no bash and no curl.

The ALB task is a security group task wearing a costume

Five resources: a security group, an instance running nginx from user data, a target group, a load balancer, and a listener. All five were straightforward. The task actually lives in two lines that look like boilerplate.

Internet  --80-->  default SG (on the ALB)  --80-->  devops-sg (on the EC2)

aws ec2 authorize-security-group-ingress \
  --group-id $DEVOPS_SG --protocol tcp --port 80 --source-group $DEF_SG

aws ec2 authorize-security-group-ingress \
  --group-id $DEF_SG --protocol tcp --port 80 --cidr 0.0.0.0/0

The first is the stated instruction: open port 80 for the default security group. --source-group rather than a CIDR, so nothing on the internet reaches the instance directly. That much is real, and it's most of the point of putting a load balancer in front of something.

It's worth being precise about what the rule actually permits, though. --source-group names a set, not the load balancer. An instance launched without a specified security group goes into the VPC default group automatically, so that set grows by omission rather than by decision, and the second rule below makes every member of it reachable from the internet on 80. The cleaner shape is a dedicated security group for the ALB, which costs one extra command and makes --source-group refer to exactly one thing. The task nudges you away from it by saying "make appropriate changes in the default security group if necessary" rather than "give the ALB its own. Corrected after a good catch from @vinhnguyenthanhdn in the comment section.

The second comes from the throwaway clause "make appropriate changes in the default security group if necessary". It is necessary. The ALB carries the default SG, and a default security group only permits inbound traffic from itself. Without that rule the ALB resolves in DNS and then nothing happens. Security groups drop disallowed packets rather than rejecting them, so the SYN vanishes and curl sits on connect until it times out instead of coming back refused. Every resource you can inspect looks correct.

Reading a task for the rule it implies rather than the rule it states is most of the skill.

The listener is the only thing joining two halves

aws elbv2 create-listener --load-balancer-arn $ALB_ARN \
  --protocol HTTP --port 80 \
  --default-actions Type=forward,TargetGroupArn=$TG_ARN

A load balancer and a target group created separately have no relationship at all. The listener is what connects them, and there is no default. Skip it and describe-load-balancers looks healthy, describe-target-health looks healthy, and nothing works.

The console creates the listener as part of the ALB form, which is exactly why it is easy to miss from the CLI.

Two more ALB facts worth having in advance. It requires subnets in at least two availability zones, because it is a distributed service and will not create with one. And registering a target does not wait for the target to be ready:

aws elbv2 register-targets --target-group-arn $TG_ARN --targets Id=$EC2_ID

That succeeded seconds after run-instances, while the instance was still pending. Registration is a membership record, not a readiness check. The target sits initial with reason Elb.RegistrationInProgress, moves to unhealthy with Target.Timeout once checks start failing, and only reaches healthy when they pass. So register immediately and let the health checker do its job rather than adding dead time.

While we are here, the health check path deserves more thought than it usually gets:

--health-check-protocol HTTP --health-check-path /

/ works because nginx ships a default index page. Deploy an application that returns 404 at / and every target goes unhealthy while the application runs perfectly. The health check asks a different question than "is the process up".

Two habits from the finish

First, chain the waiters so the terminal is unattended until there is a result:

aws elbv2 wait load-balancer-available --load-balancer-arns $ALB_ARN \
  && aws elbv2 wait target-in-service --target-group-arn $TG_ARN --targets Id=$EC2_ID \
  && curl -s http://$ALB_DNS/ | head -5

target-in-service is the one that matters. It polls until the target reads healthy, which is the first moment the entire path is proven rather than assumed.

Second, when it times out, read the reason rather than guessing:

aws elbv2 describe-target-health --target-group-arn $TG_ARN \
  --query 'TargetHealthDescriptions[].{State:TargetHealth.State,Reason:TargetHealth.Reason}'

Target.Timeout means the security group chain is broken or nginx is not listening. Target.ResponseCodeMismatch means it answered with the wrong status. Elb.RegistrationInProgress means wait. Three very different problems that all present as "not working yet" if you only look at the state.

One last detail I have started applying everywhere. Resolve the AMI rather than hardcoding it:

aws ssm get-parameters --names \
  /aws/service/canonical/ubuntu/server/22.04/stable/current/amd64/hvm/ebs-gp2/ami-id \
  --query 'Parameters[0].Value' --output text

Canonical publishes current Ubuntu AMI IDs as public SSM parameters. AMI IDs differ per region and change with every image release, so a hardcoded one is wrong somewhere, or eventually.

Running is not reachable

The container was running and unpublished. The ALB was provisioned and unlistened. In both cases every component existed and the thing joining it to the outside did not.

The tell is the same in both: a status check that looks fine. docker ps says Up. describe-load-balancers says active. Neither is a claim about whether a packet can arrive.

So here is the Day 36 question. For the last thing you deployed, what did you check at the end, and was it a status field or an actual request?

Day 36 down. Sixty-four to go.

Day 35: Start and Enable Are Different Promises, and the Slow Thing Goes First

Nnamdi Felix Ibe — Fri, 04 Sep 2026 04:34:11 +0000

Docker starts today. After fourteen days of Git, the track moves to containers, and it opens with the least glamorous task available: install the packages and start the service.

Both halves of today turned out to be about time. When something starts, how long it takes, and what still holds after a reboot.

One Docker task, one AWS task. Install docker-ce and Compose on App Server 2, then build an EC2 and private RDS application stack that actually serves a page. The tasks come from the KodeKloud Engineer platform.

Install Docker, and the two verbs people conflate

sudo yum install -y yum-utils && \
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo && \
sudo yum install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

docker-ce is not in the base RHEL or CentOS repositories, so the repo has to be added first, which is why those three commands are chained rather than pasted separately. Run the install first, and you get "No match for argument: docker-ce", which sends you hunting for a typo instead of a missing repository.

Four packages, four jobs. docker-ce is the daemon, docker-ce-cli is the client you type at, containerd.io is the runtime the daemon delegates to, and docker-compose-plugin is what makes docker compose a subcommand. That last one matters: docker compose with a space is Compose v2 shipped as a plugin, and docker-compose with a hyphen is the old standalone binary. Different programs. Any guide using the hyphenated form is telling you about v1.

Then:

sudo systemctl start docker && sudo systemctl enable docker

These are two different promises. start says the daemon is running now. enable says it comes back after a reboot. A task that says "start the service" almost always means both, and systemctl enable --now docker does the pair in one call.

Worth verifying each claim with the command that actually tests it:

docker -v                # the client is installed
docker compose version   # the plugin is installed
systemctl status docker  # the daemon is running

docker -v answering does not mean the daemon is up. The client is a separate binary and never talks to it. Two different failures get conflated here: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? means the service is not started, while permission denied while trying to connect to the Docker daemon socket means it is running and your user is not in the docker group. Same symptom, opposite fixes.

An hour and eight minutes of it provisioning

The AWS half was four things that only count if all four work together: a private MySQL RDS instance, security groups joining it to an EC2, password-less root SSH, and an index.php that renders Connected successfully in a browser.

The lab window was an hour, and RDS takes five to ten minutes to provision. So the first command was the RDS create, and everything else happened while it built. Roughly eight minutes of provisioning cost about thirty seconds of actual waiting, because the SSH and Apache work filled the gap.

That generalises further than it looks: in any time-boxed environment, find the slowest asynchronous resource and start it before you start thinking about anything else.

One flag saved a whole step. The task lists "create a database named devops_db" separately, which reads like connecting a MySQL client to an endpoint you cannot yet reach:

aws rds create-db-instance ... --db-name devops_db

RDS creates the schema during provisioning. Note the naming trap while you are there: --db-instance-identifier is the RDS instance (devops-rds, hyphens) and --db-name is the MySQL schema (devops_db, underscore). Both read as "the database name" in English.

For the security group, reference the group rather than an address:

aws ec2 authorize-security-group-ingress \
  --group-id $RDS_SG --protocol tcp --port 3306 --source-group $EC2_SG

--source-group means anything wearing that security group can connect. It survives the instance being replaced or its private IP changing, and grants nothing to anything else in the VPC. A hardcoded /32 works today and breaks the first time the instance is recycled.

Sixty seconds

The task says to connect from the console, which means the EC2 Instance Connect button. Its CLI equivalent has a property the console hides:

aws ec2-instance-connect send-ssh-public-key --region $REGION \
  --instance-id $EC2_ID --instance-os-user ubuntu \
  --ssh-public-key file:///root/.ssh/id_rsa.pub

That injects your public key into the user's authorized_keys for sixty seconds. It is a temporary grant, not an install, so the SSH has to be chained onto the same command with &&. Paste them as two commands, and you will usually miss the window. The pattern is to spend that one login installing the key permanently.

Also, do not guess the username:

aws ec2 describe-images --image-ids $AMI --query 'Images[0].Name' --output text
# ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20240701

Ubuntu, so ubuntu. My instinct was ec2-user, which is Amazon Linux, and it would have failed with Permission denied (publickey), the identical error you get from a genuinely broken key. Two unrelated problems, one message, one describe-images call to tell them apart.

And when installing the root key, overwrite rather than append. Cloud Ubuntu images ship /root/.ssh/authorized_keys pre-populated with a forced-command entry that prints "Please login as the user ubuntu" and hangs up. Appending leaves it in place and it fires first.

The one that gets everyone

index.php was in /var/www/html/, correct, and the browser kept showing the old default page.

DirectoryIndex index.html index.cgi index.pl index.php ...

index.html is listed first, and it already existed. Apache served it and never looked at the PHP file.

mv /var/www/html/index.html /var/www/html/index.html.bak

No reload needed. DirectoryIndex is evaluated per request against what is on disk, so the next request picks up index.php immediately.

The symptom to file away: a PHP file that is definitely present, definitely correct, and definitely not being served is a DirectoryIndex problem, not a PHP problem.

Finish from the shell rather than the browser:

aws rds wait db-instance-available --db-instance-identifier devops-rds && curl -s http://$EC2_IP/

Connected successfully<br />

That one line proves Apache is serving, PHP is executing, php-mysql is loaded, the security group is open, RDS is reachable, the credentials are valid and devops_db exists. A browser proves the same thing while adding caching and your own network as extra variables.

Everything here was a clock

start versus enable is a question about time. Firing the RDS create first is a question about time. The sixty-second key grant is a question about time. Even DirectoryIndex is ordering, just in a config file rather than a schedule.

So here is the Day 35 question. In the thing you deployed most recently, which parts survive a reboot, and do you know that because you checked or because it has not been rebooted yet?

Day 35 down. Sixty-five to go.

Day 34: A Hook That Cannot Fail, and an Error Three Commands From Its Cause

Nnamdi Felix Ibe — Tue, 01 Sep 2026 21:44:08 +0000

Today's theme is failures that report themselves somewhere other than where they happened. A Git hook with the wrong permissions does nothing and says nothing. A heredoc with a trailing space produces an error about a missing zip file, three commands downstream.

One Git task, one AWS task. Install a post-update hook that tags every push to master, then deploy a Python Lambda from the CLI. The tasks come from the KodeKloud Engineer platform.

This is also the last Git task in the track. Days 21 to 34 covered the lot: repository setup, forks, branches, merges, remotes, revert, cherry-pick, pull requests, hard reset, stash, rebase, conflicts, and now hooks.

The hook that runs after it is too late to object

The task was to make every push to master create a release tag automatically. That work belongs on the server, in the bare repository:

cd /opt/apps.git/hooks
vi post-update

#!/bin/bash

for ref in "$@"
do
    if [ "$ref" = "refs/heads/master" ]; then
        TODAY=$(date +%F)
        COMMIT=$(git rev-parse refs/heads/master)
        git tag "release-$TODAY" "$COMMIT" 2>/dev/null
    fi
done

chmod +x post-update

post-update receives the name of every ref that was just updated, one per argument, which is what "$@" is walking. Two details in that script are worth more than they look. git rev-parse needs no repository path because GIT_DIR is already set when a hook runs. Redirecting the tag error to /dev/null keeps the hook quiet on the second push of the day, but it isn't idempotent. git tag refuses to move a name that already exists, so the tag keeps pointing at the first commit of the day while later pushes land untagged. The redirect suppresses the only message that would have said so. The fix is a name that can't collide.

Then the part that decides whether any of this is a good idea. Git's documentation says post-update is meant primarily for notification and cannot affect the outcome of git receive-pack. It runs after the refs have already moved. Exit non-zero, and the push still succeeds.

So if you want a hook that rejects bad pushes, this is the wrong hook. pre-receive and update run before refs move and can fail the push. post-receive and post-update run afterwards and cannot. Pick the wrong one, and you have written a validation rule that logs its objection into the void while the push lands anyway.

And the quiet one:

chmod +x post-update
ls -la | grep post-update

A hook file with perfect contents and no execute bit is skipped in silence. Nothing in the push output mentions it. This is also why Git ships its samples with a .sample suffix, and why hooks never travel: they live in the repository directory, and neither clone nor push carries them. A hook that has to run on the server has to be put on the server.

One last thing, because it looks like the hook failed when it did not:

git fetch --tags
git tag
# release-2026-08-23

The tag was created on the server. Your clone does not know about it until you ask.

The trailing space

The AWS half was Day 33's Lambda again, with a different name and a pre-existing role.

Two useful things before the mistake. aws iam get-role before create-role, because this lab pre-provisions the role and create-role is not idempotent, so a blind create returns EntityAlreadyExists and aborts a chained script. Read before write when a resource might already exist. Worth knowing that attach-role-policy is idempotent, so re-attaching a policy is safe.

Now the mistake:

cat > /root/lambda-build/lambda_function.py <<'EOF'
def lambda_handler(event, context):
    ...
EOF

There is a space after that final EOF. The shell never recognises its terminator, keeps reading, and drops the prompt to >. Ctrl+C out and the file was never written.

Three commands later:

Unable to load paramfile fileb:///root/function.zip: [Errno 2] No such file or directory

The zip step had been skipped because the source did not exist, so the zip did not exist, so create-function failed. The error names a missing archive. The cause is an invisible character in a command that appeared to succeed three steps earlier.

The fix I have adopted since is to stop pasting heredocs for short files:

printf '%s\n' \
  'def lambda_handler(event, context):' \
  '    return {' \
  '        "statusCode": 200,' \
  '        "body": "Welcome to KKE AWS Labs!"' \
  '    }' \
  > /root/lambda-build/lambda_function.py

Each argument becomes a line, single quotes prevent expansion, and there is no terminator to get wrong. Then cat the file rather than assuming.

The same principle applies to the artifact:

cd /root/lambda-build && zip -q /root/function.zip lambda_function.py && unzip -l /root/function.zip

  Length      Date    Time    Name
      125  2026-08-25 07:44   lambda_function.py

The name column has no directory prefix, which is the only thing that matters. lambda-build/lambda_function.py inside the archive would deploy fine and fail at invoke time with an import error. Check the artifact, not the exit code of the tool that made it.

And one habit worth stealing outright:

aws lambda wait function-active --region us-east-1 --function-name datacenter-lambda-cli

create-function returns with "State": "Pending" and invoking a pending function fails. There is a waiter for that, and there are waiters for far more things than most people expect: ec2 wait instance-running, rds wait db-instance-available, cloudformation wait stack-create-complete. Any sleep 30 in a script is worth one search.

Look upstream of the error

The hook and the heredoc are the same problem wearing different clothes. In both cases, the thing that broke was silent, and the thing that spoke up was downstream of it and blameless.

The habit that helps is separating "this command exited zero" from "this command did what I wanted", and checking the second one deliberately: ls -la on the hook, unzip -l on the archive, git fetch --tags before concluding the tag was never made.

So here is the Day 34 question. When something in your pipeline fails, do you check the step that reported the error, or the last step that anyone actually verified?

Day 34 down. Sixty-six to go.

16 Posts, Started Commenting, then got the Shield

Nnamdi Felix Ibe — Fri, 28 Aug 2026 20:53:31 +0000

I joined DEV in June and started doing what I have been avoiding for a long time. Now and then, I hit the publish button and closed the DEV window tab.

Sixteen posts went out that way. They averaged 1.6 reactions. Several of them got none at all. I told myself the work was good and the readers would find it, which was the comfortable thing to believe while nothing was happening.

Fast forward to 24th July, I started actually reading other people's posts and leaving my thoughts in their comment sections.

The twelve posts since then have averaged an increase in reactions and followers.

Then this week, DEV granted me Trustee Member status. I did not ask. I researched what this actually means. I found out their guide lists three ways it happens, and two of them are things I have never done. The one left I suspect is that somebody noticed.

They did not notice my post; I guess sixteen of those had gone out to almost nobody.

Maybe they noticed how I turned up in other people's threads.

That is the thing I would tell my June self. The writing was never the bottleneck. Showing up was.

Thank you to @xulingfeng, whose Stratagems series I have followed comment by comment. To @technogamerz, who leaves every thread warmer than she found it. And to @vinhnguyenthanhdn, who answered a question at the end of one of my posts with a better debugging story than the post itself.

None of you was a strategy. I have found real family here in this community. Thank you all @fellowdevteam , for the trust. I intend to use the thumbs down about as rarely as that guide implies I should.

Full guide, if you want the detail: https://dev.to/trusted-member

Day 33: A Conflict Is Git Refusing to Guess, and Lambda's 200 Isn't the One That Matters

Nnamdi Felix Ibe — Fri, 28 Aug 2026 19:26:23 +0000

Today was a lesson in reading what a system is actually telling you. Git stopped and said it could not decide, which is the most honest thing a tool can do. Lambda said 200, which sounds like success and is not quite a claim about success at all.

One Git task, one AWS task. Resolve a merge conflict after a rejected push, then deploy a Python Lambda function under a purpose-built IAM role. The tasks come from the KodeKloud Engineer platform.

The rejected push is the warning, not the problem

The setup was ordinary. Log in as a different user this time, finish a half-done edit in a shared blog repository, commit it, and push.

ssh max@ststor01
cd /home/max/story-blog

vi story-index.txt
git status
git add .
git commit -m "Add the fourth story to the index"
git push
# ! [rejected]  master -> master (fetch first)

Another developer had already pushed to master. My push was not a fast-forward, so Git refused it.

It is worth sitting with that refusal for a second, because the instinct is to treat it as an obstacle. It is not. Git is telling you that accepting this push would drop commits that already exist on the remote. Forcing here would delete a colleague's work. Fetching and merging first is the only correct response.

git pull
# Auto-merging story-index.txt
# CONFLICT (content): Merge conflict in story-index.txt
# Automatic merge failed; fix conflicts and then commit the result.

We had both edited the same line. Git merges changes to different parts of the same file without any drama. A conflict means two people had an opinion about the same region, and Git will not pick one.

<<<<<<< HEAD
4. The Lion and the Mooshika : Ankit
=======
4. Mowgli and the Wolves : Sarah
>>>>>>> 8f2c1a9b3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f89

Three marker lines, not two. <<<<<<< HEAD opens your side, ======= divides, >>>>>>> closes theirs. All three have to be deleted. A stray ======= left behind is one of the most common ways a conflict resolution gets committed broken, because the file still parses fine to a human skimming it.

Resolving means writing the text you actually want. It is not a choice between two buttons. Sometimes it is your line, sometimes theirs, and often it is both kept or a rewrite of the two.

vi story-index.txt      # edit and remove all three markers

git add story-index.txt
git status              # the file leaves "Unmerged paths"
git commit -m "Resolve merge conflict in story-index.txt"
git push

git add on a conflicted file is how you tell Git you are done with it. That is the only signal it has, which is why git status during a conflict is worth reading properly: "Unmerged paths" is your remaining work, and the list shrinks as you stage.

Two things worth having ready before you need them. git merge --abort puts everything back to the pre-merge state, and it is the right move the moment a conflict looks bigger than you expected. And git config --global rerere.enabled true makes Git record how you resolved a conflict and replay that resolution when the same one reappears, which it will on any long-lived branch.

The cheapest fix, of course, sits upstream of all of it. Pull before you start work, not after you finish it.

Lambda: four parts, and two status codes that mean different things

The AWS task was a Python function returning a fixed greeting, running under a role called lambda_execution_role. The task said to use the console; the CLI produces identical resources and is repeatable, so I did it there.

A Lambda deployment from nothing is always four parts, in order, and missing any one of them fails: a role Lambda is allowed to assume, a permissions policy on that role, zipped code, and the function wired to both.

The first two get conflated constantly. The trust policy answers who is allowed to become this role, and the answer has to be the Lambda service principal:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "lambda.amazonaws.com" },
    "Action": "sts:AssumeRole"
  }]
}

It grants no permissions at all. Those come separately, and AWSLambdaBasicExecutionRole is the AWS-managed minimum: create a log group, create a log stream, put log events, nothing else.

aws iam attach-role-policy --role-name lambda_execution_role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

Note the service-role/ in that ARN. Dropping it produces a "policy does not exist" error that sends you looking in entirely the wrong place.

Then the packaging details, both of which fail quietly rather than loudly:

(cd /root/nautilus-lambda && zip -q /root/nautilus-lambda.zip lambda_function.py)

aws lambda create-function --region us-east-1 \
  --function-name nautilus-lambda \
  --runtime python3.12 \
  --role "$ROLE_ARN" \
  --handler lambda_function.lambda_handler \
  --zip-file fileb:///root/nautilus-lambda.zip

The .py has to be at the root of the archive. Zip the parent directory and you get nautilus-lambda/lambda_function.py inside, which Lambda cannot import. The handler string is module.function with no extension, so lambda_function.lambda_handler means call lambda_handler inside lambda_function.py; get it wrong and the deploy succeeds and the invoke fails. And it is fileb://, not file://, because the zip is binary and reading it as text corrupts it.

There is also a race worth knowing. Call create-function immediately after create-role and you get "The role defined for the function cannot be assumed by Lambda". The role exists, IAM is eventually consistent, and Lambda cannot see it yet. It clears in five to fifteen seconds. A retry loop beats a guessed sleep, and the same race turns up any time you create an IAM role and immediately hand it to ECS, CodeBuild or EC2.

Now the part I actually want to flag:

aws lambda invoke --region us-east-1 --function-name nautilus-lambda \
  /root/lambda-out.json --query 'StatusCode' --output text
cat /root/lambda-out.json

200
{"statusCode": 200, "body": "Welcome to KKE AWS Labs!"}

Two 200s, and they are not the same claim.

The outer one is the Lambda service saying it received the request and ran the function. AWS states it plainly in the Invoke API reference: the status code in the API response does not reflect function errors, and error codes are reserved for problems that stop the function executing at all, such as permissions or configuration. A function that throws an unhandled exception still returns 200 there.

The inner statusCode is your own code's return value. That is the one that proves the function did what you wrote.

If the function had failed, the response would also carry a FunctionError field and the payload would hold a stack trace. Which means checking only the outer status code is a false positive waiting to happen, and it is exactly the check a hurried health script tends to make.

One small trap alongside it: invoke writes the payload to the file argument, it does not print to stdout. Leave the path off and you get a usage error rather than your output.

Read what it said, not what you expected

A merge conflict and a 200 look like opposites. One is a stop, the other is the number you were hoping for. They are the same lesson from two directions.

Git refused to guess and told you exactly which lines it could not decide about, which is more information than a silent success would have given you. Lambda answered a narrower question than the one you were asking, accurately, and it is on you to know which question that was.

So here is the Day 33 question. The green check in your pipeline: do you know precisely which claim it is making, and which one it is not?

Day 33 down. Sixty-seven to go.

Day 32: Rebase Replays Your Commits, and a Restore Inherits Everything You Don't Override

Nnamdi Felix Ibe — Wed, 26 Aug 2026 21:21:02 +0000

Today's two tasks are both about a new base. A feature branch that needs to sit on top of a master that has moved. A database instance that needs to come back from a snapshot taken when things were fine. In each case, the interesting question is the same: what carries over, and what do you have to say out loud?

One Git task, one AWS task. Rebase a feature branch onto master without creating a merge commit, then snapshot an RDS instance and restore it into a new one. The tasks come from the KodeKloud Engineer platform.

Rebase: not moving commits, replaying them

The requirement was specific, and the specificity is the lesson. A developer's feature branch was behind master. Bring it up to date without losing any feature work, and without a merge commit.

That second clause rules out git merge master. Merge joins two histories and records the join, which is the merge commit. Rebase does something else entirely.

cd /usr/src/kodekloudrepos/media

git branch
git log --oneline --graph --all --decorate

git checkout feature
git rebase master

git log --oneline --graph --decorate

Git's own documentation describes what happens under git rebase master: it lists the commits on your branch that are not on master, checks out master, and then replays each of your commits on top of it, one at a time, in a way it compares to running git cherry-pick for each one.

Replays. Not moves. Every commit that comes out the other side has a new hash, because a commit's identity includes its parent, and the parent is different now. Your work is preserved, the commits carrying it are not the same objects they were.

That is exactly why there is no merge commit. Rebase does not join two histories, it rewrites yours so it looks like it was always based on master's current tip. You get a straight line, at the cost of a history that is no longer a record of what actually happened.

Two things I had to be deliberate about.

Direction. Rebase applies to the branch you are standing on and takes the branch you name as the new base. git checkout feature then git rebase master means replay feature onto master. Run it the other way round, and you have rewritten master, which is a much worse afternoon.

Then the push:

git push --force-with-lease

History was rewritten, so a normal push is refused. --force-with-lease requires the remote ref to still match your remote-tracking branch and refuses if it does not, so it will not silently flatten something a colleague pushed while you were rebasing.

Worth reading the caveat in Git's own docs, though, because it is not the guarantee people assume. The lease is checked against your remote-tracking ref, so anything that quietly runs git fetch in the background, a cron job or an IDE, updates that ref and defeats the protection. --force-if-includes, which additionally requires the remote tip to be reachable from your branch's reflog, closes that gap.

The conflict experience also differs from merge in a way nobody warns you about. A merge conflicts once, with both complete sides in front of you. A rebase can conflict once per replayed commit, each time showing you one commit's worth of change against a base that has moved. More stops, smaller pieces, and git rebase --abort puts everything back exactly as it was if it stops being worth it.

And the rule that matters most, straight from the documentation: rebasing a branch that others have based work on forces everyone downstream to fix their history by hand. Rebase your own unshared branches freely. Leave shared ones alone.

The restore: you only type what you want to be different

The AWS task was four steps: wait for the source instance, snapshot it, restore the snapshot into a new instance, and force the new one to db.t3.micro.

aws rds wait db-instance-available --region us-east-1 \
  --db-instance-identifier datacenter-rds

aws rds create-db-snapshot --region us-east-1 \
  --db-instance-identifier datacenter-rds \
  --db-snapshot-identifier datacenter-snapshot

aws rds wait db-snapshot-available --region us-east-1 \
  --db-snapshot-identifier datacenter-snapshot

The leading waiter is not padding. create-db-snapshot refuses outright if the source instance is in any state other than available, mid-modification or still backing up included, and the waiter absorbs that instead of making you retry by hand. Note also that there are two different waiters in that block. db-instance-available watches an instance, db-snapshot-available watches a snapshot, and using the wrong one silently watches the wrong resource while you assume it is working.

Then the restore, which is a single call:

aws rds restore-db-instance-from-db-snapshot --region us-east-1 \
  --db-instance-identifier datacenter-snapshot-restore \
  --db-snapshot-identifier datacenter-snapshot \
  --db-instance-class db.t3.micro

One override. That is the whole thing, and it is worth understanding why.

A restore is not an empty instance that happens to receive data. It clones the source's configuration. Engine and version, allocated storage, storage type, VPC and subnet group, parameter group, character set, master username and the data all come across without being asked for. The restored instance in this lab came out as MySQL 8.4.5 on 5 GiB of gp2, and none of those three were typed anywhere.

What you can change at restore time is a specific list: instance class, subnet group, availability zone, port, multi-AZ, public accessibility, storage type, tags, and auto-minor-version-upgrade. The task wanted one of those, so the command has one flag.

The master password is on neither list. It comes from the snapshot and cannot be set during the restore. Changing it is a modify-db-instance --master-user-password afterwards.

One distinction that matters more than it looks. create-db-snapshot makes a manual snapshot, and manual snapshots live until you delete them. Automated snapshots age out with the backup retention window and disappear when the instance is deleted. For a pre-upgrade safety net you want the one that outlives the thing it is protecting. The flip side is that manual snapshots keep billing for storage after the instance is long gone, and they are exactly the thing people forget to clean up.

What comes along for free

Rebase and restore both start from someone else's state and put your intent on top of it. Rebase inherits master's history and replays your commits onto it. A restore inherits the snapshot's entire configuration and takes only the overrides you name.

The failure mode is the same in both directions too. Re-specifying engine and storage on a restore either duplicates what the snapshot already carries or quietly diverges from it. Rebasing a branch other people are standing on rewrites history they already have.

So here is the Day 32 question. When you rebuild something from a known-good starting point, do you know which of its properties you inherited and which you actually chose?

Day 32 down. Sixty-eight to go.

Day 31: Stash Keeps What Status Won't Show, and Private Is a Flag, Not a Subnet

Nnamdi Felix Ibe — Mon, 24 Aug 2026 20:12:31 +0000

Both of today's tasks involved something real that the interface never mentions. Git keeps work in a place that git status will not tell you about. The RDS console wizard builds a resource on your behalf and never says it did. Neither is a bug, and both cost you time the first time you meet them.

One Git task, one AWS task. Park uncommitted work with stash and bring a specific entry back, then create a private MySQL RDS instance on the free tier. The tasks come from the KodeKloud Engineer platform.

Stash: the drawer that nothing points you to

Stash is for the moment when you are halfway through something and have to be somewhere else. A hotfix, a colleague's branch, a review. You do not want a commit that says "wip", so you park it.

cd /usr/src/kodekloudrepos/<repo>

git stash list
# stash@{0}: WIP on master: 3a1f2b9 earlier commit
# stash@{1}: WIP on master: 3a1f2b9 earlier commit

git stash show -p 'stash@{1}'
git stash apply 'stash@{1}'

stash@{0} is the newest and stash@{1} is the one before it, and the bare integer works too, so git stash apply 1 is the same call. The braces need quoting in zsh, which reads them as globbing characters.

Then the step that catches people, including me:

git status
git add .
git commit -m "changes of stash@{1}"
git push origin master

apply restores the working tree, not the index. Your files come back modified, not staged, so a bare git commit -m has nothing to commit. git stash apply --index restores the staged state as well, if that is what you parked.

Two more things worth knowing before you rely on it.

apply and pop are not interchangeable. pop restores the entry and deletes it. apply restores it and leaves it in the list. Use apply when you are not yet certain the restore is clean, then git stash drop once you are. And dropping renumbers everything below it, so never drop entries in a loop by fixed index.

Untracked files are not stashed at all by default. A new file that seems to have vanished into a stash was never in it. git stash push -u includes untracked files, and -a includes ignored ones too.

The part that actually matters, though, is that none of this is visible. A stash is a real commit under refs/stash, but it is not in your branch, not in your log, not in git status, and not pushed anywhere. It is local, and it stays local. Work parked in a stash on a server you lose access to is work you have lost. Treat stash as a few hours of parking, never as storage.

The private RDS instance: the flag, and the thing the wizard did for you

The task read like a console walkthrough. Free tier template, full configuration, MySQL 8.4.x, db.t3.micro, 20 GiB of gp2, autoscaling capped at 22 GiB, private. Doing it on the CLI means translating each of those, and two of them do not translate at all.

"Full configuration" is a console-only creation method. The CLI is always full configuration. "Free tier" is a console preset that means db.t3.micro plus single-AZ, which you pin with --no-multi-az. Storage autoscaling and its maximum threshold collapse into one flag, --max-allocated-storage 22, and passing it is what turns autoscaling on.

And "private" is the interesting one. It is not about which subnet the instance lands in. It is an attribute on the instance itself:

--no-publicly-accessible

That flag is what decides whether RDS gives the instance a public DNS name that resolves to a routable address. Put a database in a private subnet with --publicly-accessible, and you have a private-subnet database that AWS is still advertising publicly.

Then the CLI stopped and told me something the console never would have:

aws rds describe-db-subnet-groups --region us-east-1 --output table

Nothing. There was no DB subnet group in the account, and RDS will not create an instance without one. The console wizard makes one silently while you are choosing an engine. The CLI expects you to have done it.

It also has a requirement that surprises people: the subnet group needs subnets in at least two availability zones, and AWS documents that as applying to Single-AZ deployments too, so the instance can be converted to Multi-AZ later. A single-AZ database still needs a multi-AZ subnet group.

VPC=$(aws ec2 describe-vpcs --region us-east-1 \
  --filters "Name=isDefault,Values=true" \
  --query 'Vpcs[0].VpcId' --output text)

SUBNETS=$(aws ec2 describe-subnets --region us-east-1 \
  --filters "Name=vpc-id,Values=$VPC" \
  --query 'Subnets[].SubnetId' --output text)

aws rds create-db-subnet-group --region us-east-1 \
  --db-subnet-group-name xfusion-rds-subnet-group \
  --db-subnet-group-description "Subnet group for xfusion-rds" \
  --subnet-ids $SUBNETS

$SUBNETS is deliberately unquoted there. --subnet-ids wants a space-separated list, and --output text produces exactly that.

One more habit worth stealing. The task said "8.4.x" without pinning a patch level, so rather than guess, I asked AWS which 8.4 versions are actually orderable on that instance class in that region:

VER=$(aws rds describe-orderable-db-instance-options --region us-east-1 \
  --engine mysql --db-instance-class db.t3.micro \
  --query "OrderableDBInstanceOptions[?starts_with(EngineVersion, '8.4')].EngineVersion" \
  --output text | tr '\t' '\n' | sort -uV | tail -1)

describe-orderable-db-instance-options beats describe-db-engine-versions here, because a version can exist and still not be available on db.t3.micro. And sort -V is version sort, which puts 8.4.11 after 8.4.9. Plain sort would not.

That query cost me a detour first. Written with backticks around the version prefix, it fails:

In function starts_with(), invalid type for value: 8.4,
expected one of: ['string'], received: "number"

In JMESPath, backticks delimit JSON literals, so `8.4` is the number 8.4. String literals use single quotes. Backticks appear to work for most words only because invalid JSON degrades to a string, and they break the moment the content parses as a number, a boolean, or null. Use single quotes for strings every time.

Everything after that was one call and a wait of five to ten minutes with no output, which is normal. Do your discovery during the wait, not before it.

State you have to go looking for

The stash and the missing subnet group are the same shape of problem. Something exists, or does not exist, and the tool in front of you has no opinion about telling you. git status is silent about parked work. The console is silent about the resource it built for you, right up until you try the same thing without it.

So here is the Day 31 question. What is true about your environment right now that nothing on your screen is telling you?

Day 31 down. Sixty-nine to go.

Day 30: Hard Reset Forgets on Purpose, and NAT Needs a Guardrail Switched Off

Nnamdi Felix Ibe — Sat, 22 Aug 2026 05:52:22 +0000

Thirty days in, and today both tasks required deliberately switching off something that exists to protect you. Git refuses to let you overwrite shared history. EC2 refuses to forward packets that aren't addressed to it. Both refusals are correct. Both had to go.

One Git task, one AWS task. Hard reset a branch and force-push it, then build a NAT instance so a private EC2 instance can reach S3. The tasks come from the KodeKloud Engineer platform.

Hard reset: the undo with no record

Three days ago on Day 27, I wrote, in bold, that you should never rewrite history you have already pushed. Today's task is doing precisely that. So let me deal with that head-on rather than pretend the two posts don't touch.

cd /usr/src/kodekloudrepos/official

git branch --show-current
git log --oneline

git reset --hard <commit-id>

git log --oneline
git status

reset --hard moves the branch pointer backwards and takes the working tree and index with it. Everything after that commit is gone from the branch. Compare that to revert, which writes a new commit undoing an old one and leaves both on the record. Revert adds. Reset removes.

There are three modes, and only one is dangerous. --soft moves the pointer and leaves your changes staged. --mixed, the default, moves it and unstages them. --hard moves it and throws away uncommitted work entirely. That last part has no undo — the reflog can recover commits, but nothing recovers changes you never committed.

Then the push is rejected because the branch is no longer a fast-forward. Git is not being difficult; it is telling you that accepting this push would destroy commits sitting on the remote.

# What the task asked for
git push origin master --force

# What you should almost always type instead
git push origin master --force-with-lease

--force-with-lease is the one worth building into your fingers. It checks that the remote is still where you last saw it and refuses if anyone has pushed since. Plain --force overwrites their work silently. When you're working alone, the two are identical. When you're not, one of them costs a colleague their afternoon.

So does Day 27's rule still hold? Yes. Force-pushing a branch only you use, or a lab repo, is fine. Force-pushing a shared branch rewrites history other people already pulled — their next pull produces divergent branches and a merge that reintroduces exactly what you removed. The rule was never "never force-push." It was "know whose history you're rewriting."

The NAT instance: an invisible default that drops everything

A private EC2 instance had a cron job uploading a file to S3 every minute, and every upload was failing. No internet access. The fix is a NAT instance in a public subnet — cheaper than a NAT Gateway, and considerably more educational.

The first lesson arrived before I built anything. The task said the VPC, private subnet and private EC2 already existed, implying I only needed the public subnet and the NAT instance. So I ran discovery anyway:

aws ec2 describe-internet-gateways --region us-east-1 \
  --filters "Name=attachment.vpc-id,Values=$VPC" \
  --query 'InternetGateways[].InternetGatewayId' --output text

Empty. There was no internet gateway attached to the VPC at all. A NAT instance in a public subnet with no route to an IGW is useless — the NAT instance needs internet access before it can hand any out. The task description was wrong about its own environment, and discovery is what caught it.

With the IGW, public subnet and route table in place, the NAT configuration itself is three things in user-data:

dnf install -y iptables-services
sysctl -w net.ipv4.ip_forward=1
IFACE=$(ip -o -4 route show to default | awk '{print $5}' | head -1)
iptables -t nat -A POSTROUTING -o "$IFACE" -j MASQUERADE
service iptables save

ip_forward is off by default and lets the kernel route packets between interfaces. MASQUERADE rewrites the source IP of outbound packets to the NAT instance's own address and reverses it on the way back — that is literally the network address translation. And IFACE is detected rather than hardcoded, because AL2023 names interfaces ens5, not eth0.

Then the line that decides whether any of it works:

aws ec2 modify-instance-attribute --instance-id $NAT_ID --no-source-dest-check

By default, EC2 drops any packet whose source or destination IP is not the instance's own address. That is a sensible anti-spoofing guardrail, and it is exactly what a NAT instance does all day long. Leave it on, and the instance discards every forwarded packet. No error. No log line. The cron job just keeps failing and everything you built looks correct.

That is what makes it the classic NAT instance necessity: it is not a mistake you can see. It is a default you have to know about in advance.

One more detail worth stealing. When routing the private subnet through the instance, target the ENI, not the instance ID:

ENI=$(aws ec2 describe-instances --instance-ids $NAT_ID \
  --query 'Reservations[0].Instances[0].NetworkInterfaces[0].NetworkInterfaceId' --output text)

aws ec2 create-route --route-table-id $PRIV_RTB \
  --destination-cidr-block 0.0.0.0/0 --network-interface-id $ENI

And if that route shows blackhole rather than active, the instance wasn't running when you created it. Run aws ec2 wait instance-running first.

For the record: NAT instances are a cost play and an exam topic. In production, you use a NAT Gateway, which is managed, highly available per AZ, and scales to 100 Gbps. You pay per gigabyte for the privilege of never thinking about ip_forward again.

Guardrails you turn off knowingly

Git's push rejection and EC2's source/destination check are the same kind of thing: a default that says no to something that is usually a mistake. Both tasks needed that not to be overridden. The difference between a professional and an incident is whether you knew what the guardrail was for before you switched it off.

So here is the Day 30 question, a third of the way in. When you disable a safety check, is it because you understand why it exists, or because it was in the way?

Day 30 down. Seventy to go.

Day 29: A Pull Request Needs Two People, and Peering Needs Two Routes

Nnamdi Felix Ibe — Wed, 19 Aug 2026 19:51:41 +0000

Some things cannot be finished alone, by design. A change that only you have seen is not reviewed. A network connection that only works in one direction is not a connection. Day 29 was two versions of the same idea: both sides have to agree, and half of it is worse than none because half of it looks like it should work.

One Git task, one AWS task. Open a pull request, get it reviewed and merged, then peer a public VPC with a private one so instances on each side can talk. The tasks come from the KodeKloud Engineer platform.

Pull requests: the gate, not the merge

The first thing worth saying is that a pull request is not a Git feature. Git has no idea what one is. It is a workflow layer that hosting platforms bolt on top — Gitea here, GitHub and GitLab elsewhere — and underneath, a merged PR is git merge and nothing more.

Which raises the obvious question: if the merge is trivial, what is the PR for?

The gate. A named reviewer, a recorded approval, a discussion attached to the change, and a permanent record of who agreed to what. On this task, that meant opening the request against master from a feature branch, adding a second user as reviewer, then logging in as that user to read the diff, approve it, and merge with a merge commit.

# Everything before and after the PR is ordinary git
git log --oneline fox-grapes        # what you are proposing
git log --oneline master            # what it is going into

# ...PR happens in the web UI...

git checkout master
git pull origin master
git log --oneline --graph --all --decorate

Two things catch people out. Base and compare get reversed constantly — base is where the code is going, compare is where it comes from. Flip them, and you propose merging master into your feature branch, which usually shows nothing and reads as a broken UI rather than a mistake.

And the reviewer has to be a genuinely different user. Approving your own pull request is the one move that empties the mechanism of all meaning, which is why most platforms refuse it.

Worth knowing the third option too: "create a merge commit" preserves the branch shape, exactly the --no-ff behaviour from Day 25. Squash flattens the branch to one commit. Rebase replays them with no merge commit at all. Same code, three different stories in the history.

One small thing that trips everyone once: merging on the server does nothing to your local clone. git pull afterwards or your master sits silently behind.

VPC peering: one connection, two route tables

The AWS task was peering a public VPC with a private one so an instance in each could reach the other. And it opened with a problem that had nothing to do with peering at all.

SSH to the public instance simply hung. No refusal, no error — just silence. The cause was in the security group:

"IpPermissions": [{
  "IpProtocol": "tcp", "FromPort": 22, "ToPort": 22,
  "IpRanges": [],
  "UserIdGroupPairs": [{ "GroupId": "sg-xxxxxxxx" }]
}]

That rule says "allow SSH from anything already inside this security group" — and the group it references is the one attached to the instance itself. Traffic from another instance in that group would match. My client host, sitting outside AWS entirely, never could.

That is the tell worth memorising: UserIdGroupPairs populated and IpRanges empty means the rule is SG-referenced, not CIDR-based. Nothing external gets in, and because security groups drop rather than reject, you get a hang instead of a message.

Then a second problem, which I liked. The task wanted my public key in the instance's authorized_keys. But writing that file needs SSH, and SSH needs that file.

EC2 Instance Connect breaks the loop. It pushes a public key to the instance out-of-band through the AWS API, valid for sixty seconds — long enough to get in and write the key permanently:

aws ec2-instance-connect send-ssh-public-key --region us-east-1 \
  --instance-id <instance-id> --instance-os-user ec2-user \
  --availability-zone us-east-1b \
  --ssh-public-key file:///root/.ssh/id_rsa.pub \
&& ssh -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa ec2-user@<public-ip> \
  "mkdir -p ~/.ssh && chmod 700 ~/.ssh && echo '$(cat /root/.ssh/id_rsa.pub)' >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

The && is not stylistic. Sixty seconds is not enough to run two commands you are typing by hand, and the second one failing on an expired key looks exactly like a permissions problem.

With access sorted, the peering itself is four checks, and this is the part that matters:

# 1. The connection must be ACTIVE, not pending-acceptance
aws ec2 accept-vpc-peering-connection --vpc-peering-connection-id <pcx-id>

# 2 and 3. A route on EACH side, pointing to the other's CIDR
aws ec2 create-route --route-table-id $DEF_RTB \
  --destination-cidr-block 10.1.0.0/16 --vpc-peering-connection-id <pcx-id>

aws ec2 create-route --route-table-id $PRIV_RTB \
  --destination-cidr-block 172.31.0.0/16 --vpc-peering-connection-id <pcx-id>

# 4. The target SG must allow the traffic type from the source CIDR
aws ec2 authorize-security-group-ingress --group-id <private-sg> \
  --protocol icmp --port -1 --cidr 172.31.0.0/16

Peering is bidirectional by nature but not by configuration. One connection object, two route tables, and you have to edit both. A route on only one side produces traffic that arrives and cannot reply, which presents as a total failure rather than a partial one.

Three details worth carrying forward. --port -1 with ICMP means all ICMP types — ping is ICMP, not TCP, so opening TCP ports does precisely nothing for it. If all four checks pass and it still fails, look at network ACLs, which, unlike security groups, are stateless and need explicit rules in both directions. And peering is not transitive: A peered to B and B peered to C does not let A reach C.

One honest note. I opened port 22 to 0.0.0.0/0 to get moving in a throwaway lab, which directly contradicts what I wrote on Day 22 about scoping SSH to /32. In a lab, that is a shortcut. Anywhere real, it is the thing you get audited for, and I would rather flag it than leave it in a code block.

Two small mistakes that cost me time, both embarrassing and both instructive. I pasted a placeholder straight into the shell — <sg-id> makes bash attempt an input redirect and throw No such file or directory. The angle brackets are notation, not syntax. And I ran ping "$PRIVATE_IP" on the remote host, where that variable had never been set. Local shell variables do not travel over SSH.

Half a handshake is not a handshake

An approval you gave yourself is not a review. A route on one side is not a path. Both tasks fail in the same quiet way: everything you configured is correct, and the thing still does not work, because the other half was never done.

So here is the Day 29 question. Where in your setup have you built one direction of something and assumed the other side agreed?

Day 29 down. Seventy-one to go.