DEV Community: Nnamdi Felix Ibe

100 Days of DevOps and Cloud (AWS), Day 20: Wiring Nginx to PHP Over a Socket, and Giving EC2 a Role Without Keys

Nnamdi Felix Ibe — Thu, 30 Jul 2026 06:53:10 +0000

The best way to give one component access to another is usually the one that leaves no secret lying around. A local socket instead of an open network port. A temporary role instead of a hardcoded key. Day 20 was two versions of that idea, Nginx talking to PHP over a Unix socket, and EC2 getting AWS permissions through a role.

One Linux task, one AWS task. Wire Nginx to PHP-FPM through a Unix socket, then create an IAM role that EC2 instances can assume. The tasks come from the KodeKloud Engineer platform.

Nginx and PHP-FPM: connected by a socket, not a port

Nginx serves static files itself, but it cannot run PHP. It hands PHP requests off to PHP-FPM, a separate process that does the actual execution. The question is how the two talk, and here the answer is a Unix socket:

location ~ \.php$ {
    include fastcgi_params;
    fastcgi_pass unix:/var/run/php-fpm/default.sock;   # PHP-FPM over a Unix socket
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
}

The location ~ \.php$ regex sends only .php requests to PHP-FPM; everything else Nginx serves statically. fastcgi_pass points at a socket file, not an IP and port. You could connect them over TCP with 127.0.0.1:9000 instead, but the Unix socket is the better local choice. It is a file on disk, so there is no network stack in the path and nothing listening on a port that could be reached from off the box. Less overhead, smaller surface.

The trap is ownership. Nginx and PHP-FPM have to agree on who owns that socket file, or Nginx gets permission-denied when it tries to connect. So PHP-FPM is configured to run as the nginx user and create the socket with matching ownership:

# In /etc/php-fpm.d/www.conf
# user = nginx
# group = nginx
# listen = /var/run/php-fpm/default.sock
# listen.owner = nginx
# listen.group = nginx

If PHP downloads as a file instead of running, or you get a 502, socket ownership is the first place to look.

IAM roles: permissions without a stored key

On AWS, the task was an IAM role for EC2. A role is how you grant an instance permission to call AWS services without putting an access key on the box. The instance assumes the role and receives temporary, rotating credentials instead.

# Trust policy: who is allowed to assume this role (the EC2 service)
aws iam create-role \
  --role-name EC2-Custom-Role \
  --assume-role-policy-document file://ec2-role-trust-policy.json

# Permissions policy: what the role can do once assumed
aws iam attach-role-policy \
  --role-name EC2-Custom-Role \
  --policy-arn <policy-arn>

A role has two policies doing two different jobs. The trust policy says who can assume it, here ec2.amazonaws.com. The permissions policy says what it can do once assumed. Two documents, two questions, do not mix them up.

Here is the step the task leaves out, and the one that trips people moving from the console to the CLI. A role by itself cannot be attached to an EC2 instance. EC2 uses an instance profile, a container that wraps the role, and in the console AWS creates that wrapper for you invisibly. From the CLI you do it by hand:

aws iam create-instance-profile --instance-profile-name EC2-Custom-Profile
aws iam add-role-to-instance-profile \
  --instance-profile-name EC2-Custom-Profile \
  --role-name EC2-Custom-Role

Then you attach the instance profile, not the role, to the instance. Miss this, and you have a perfectly good role that EC2 simply cannot use (per the AWS IAM docs).

Connect without leaking

Both tasks were about connecting two things the clean way. The Unix socket links Nginx and PHP without opening a port or leaving a network path exposed. The role links EC2 to AWS without a stored key that could leak. The lazy versions, a TCP port and a hardcoded credential, both work and both widen your attack surface for nothing in return.

So here is the Day 20 question. Would you rather connect your components the convenient way and carry the exposure, or the clean way that leaves nothing lying around to steal?

Day 20 down. Eighty to go.

100 Days of DevOps and Cloud (AWS), Day 19: Apache Deploys, and a Policy Is Just a Document Until It's Attached

Nnamdi Felix Ibe — Tue, 28 Jul 2026 19:32:11 +0000

A policy sitting in your AWS account grants nothing to anyone. It is a document until you attach it to a person, and that attach step is the moment the permissions become real. Day 19 had that on the AWS side, and its Linux twin, application files that do nothing until they land in the right directory and the server reloads.

One Linux task, one AWS task. Deploy a web application with Apache httpd, moving the files across with SCP, then attach an IAM policy to a user. The tasks come from the KodeKloud Engineer platform.

Apache: set the port, ship the files, reload

yum install -y httpd
systemctl enable --now httpd

# Point Apache at the required port
grep -n -i listen /etc/httpd/conf/httpd.conf
vi /etc/httpd/conf/httpd.conf     # change Listen 80 to the required port
systemctl reload httpd

Apache's listen port lives in httpd.conf on the Listen line. Running grep with -n gives you the line number so you change that one directive and leave the rest alone. Then reload rather than restart, so the new port applies without dropping live connections. Next, the files:

# From the jump server, copy the app files over
sudo scp -r /home/thor/media user@stapp01:/tmp
sudo scp -r /home/thor/apps  user@stapp01:/tmp

# On the app server, move them into the web root
mv /tmp/media /var/www/html/
mv /tmp/apps  /var/www/html/

The files arrive from a jump server over scp, a two-hop pattern you see any time the app server has no direct internet access. They land in /tmp first, then move into the web root at /var/www/html. Nothing serves until the files are actually under the web root. Then confirm it:

curl -4 http://stapp01:<port>/media

curl -4 forces IPv4, which saves you from odd IPv6 resolution behavior in a dual-stack setup.

IAM: attaching is what makes a policy active

On AWS, the task was to attach an existing policy to a user. The policy already existed, the user already existed, and on their own they did nothing for each other. The attach is the wiring:

# Find the user and the policy
aws iam list-users
aws iam list-policies --no-only-attached --max-items 3

# Attach the policy to the user
aws iam attach-user-policy \
  --user-name iam_userBob \
  --policy-arn arn:aws:iam::123456789012:policy/my-policy

# Verify it stuck
aws iam list-attached-user-policies --user-name iam_userBob

The policy ARN is the exact identifier, and attaching the wrong one is how you hand out access you never meant to, so you check the ARN before attaching and list the attached policies after. And a callback to Day 17: this is exactly the case groups solve. Attaching directly to Bob works, but if five people need the same access you are back to five attach commands and five things to unpick later. Attach the policy to a group and add the users to the group instead.

Created is not connected

Both tasks share one quiet trap. Creating a thing and activating it are separate steps. The app files existed on the jump server and served nothing until they reached the web root. The policy existed in the account and granted nothing until it was attached. Half-finished work looks a lot like finished work, right up until you test it and nothing happens.

So here is the Day 19 question. When you call something done, do you mean the pieces exist, or do you mean you watched it work end to end?

Day 19 down. Eighty-one to go.

100 Days of DevOps and Cloud (AWS), Day 18: MariaDB's Secure Install, and Why 'Read-Only' Needs More Than ec2:Describe

Nnamdi Felix Ibe — Mon, 27 Jul 2026 07:34:20 +0000

The most secure version of a fresh install is the one with its convenient defaults stripped back out. New database software ships open enough to get you started fast, which is exactly the state an attacker is hoping to find. Day 18 was about closing that gap on MariaDB, then granting its mirror image on AWS access so narrow it can only look.

One Linux task, one AWS task. Install MariaDB and secure it, then write a read-only IAM policy for EC2. The tasks come from the KodeKloud Engineer platform.

MariaDB: install, then take the defaults away

sudo yum install -y mariadb-server
sudo systemctl enable --now mariadb

Then the step that matters most:

# Strip the insecure defaults
sudo mysql_secure_installation

This single command is the whole security story of a fresh MariaDB. Left alone, a new install has an empty root password, an anonymous user anyone can connect as, a test database open to everyone, and remote root login switched on. mysql_secure_installation walks you through removing every one of those. Skipping it is how databases end up reachable on the internet with no password, which happens far more than it should. With that done, create the database, user, and grants:

CREATE DATABASE kodekloud_db2;
CREATE USER 'kodekloud_cap'@'localhost' IDENTIFIED BY 'securepassword';
GRANT ALL ON kodekloud_db2.* TO 'kodekloud_cap'@'localhost';
FLUSH PRIVILEGES;

MariaDB users have two parts: the name and the host they connect from. 'kodekloud_cap'@'localhost' is a different user from 'kodekloud_cap'@'%', even with the same name and password. localhost means local connections only, % means from any host. If the app connects locally, you grant to localhost, if it connects over the network, you grant to the remote host or %. Get this wrong, and you get access-denied errors that have nothing to do with the password. And FLUSH PRIVILEGES reloads the grant tables so the changes take effect now rather than after a restart.

IAM read-only: narrow enough to only look

On the AWS side, a read-only policy for EC2. This is the least-privilege idea from Day 16 pointed at a specific job, someone who needs to see the EC2 console but must not change anything.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "ec2:Describe*",
      "elasticloadbalancing:Describe*",
      "autoscaling:Describe*",
      "cloudwatch:Describe*",
      "cloudwatch:GetMetricStatistics",
      "cloudwatch:ListMetrics"
    ],
    "Resource": "*"
  }]
}

aws iam create-policy \
  --policy-name ec2-readonly-console-policy \
  --policy-document file://ec2-readonly-console.json

Here is the part I did not expect. ec2:Describe* on its own gives read-only API access, but it does not make the EC2 console usable. The console page pulls in load balancers, auto scaling groups, and CloudWatch metric graphs, and each of those is a separate service. Without elasticloadbalancing:Describe*, autoscaling:Describe*, and the CloudWatch read actions, the console throws permission errors across the page even though the core EC2 data loads fine. "Read-only for EC2" in practice means read access across the handful of services the console stitches together. AWS even ships a managed policy, AmazonEC2ReadOnlyAccess, that bundles exactly these, worth reaching for unless you need something trimmed or custom.

Two sides of the same discipline

MariaDB's secure install and a read-only policy are the same instinct facing opposite directions. One takes away access that was open by default. The other grants only the access a job genuinely needs. Both start from closed and open up on purpose, which is the reverse of how most work gets done: start open and lock it down later if anyone remembers.

So here is the Day 18 question. Would you rather begin locked down and grant access deliberately, or begin open and hope you close the doors before someone else finds them?

Day 18 down. Eighty-two to go.

100 Days of DevOps and Cloud (AWS), Day 17: Postgres Privileges and IAM Groups Are the Same Lesson

Nnamdi Felix Ibe — Fri, 24 Jul 2026 06:10:38 +0000

Access control is the same problem everywhere. Decide who gets in, and decide exactly what they can do once they are. Day 17 came at that idea from two directions, a Postgres database on Linux and an IAM group on AWS, and the shape underneath both is identical.

One Linux task, one AWS task. Install PostgreSQL, then create a database, a user, and grant that user access. On AWS, create an IAM group to manage permissions for a set of users at once. The tasks come from the KodeKloud Engineer platform.

PostgreSQL: a database, a user, and a grant

PostgreSQL runs its own user system, separate from Linux logins, and it operates as the postgres OS user. So you drop into its shell as that user:

# Log in as the postgres superuser
sudo -u postgres psql

From there, it is three statements: a database, a user with a password, and a grant that links them.

CREATE DATABASE my_database;
CREATE USER my_user WITH ENCRYPTED PASSWORD 'securepassword';
GRANT ALL PRIVILEGES ON DATABASE my_database TO my_user;

WITH ENCRYPTED PASSWORD stores a hash rather than the plain text, which is the difference between a password and a liability. The GRANT is what actually connects the user to the database.

Here is the part that catches people, and it caught me. GRANT ALL PRIVILEGES ON DATABASE does less than it sounds like. It grants database-level rights, connecting, creating schemas, and making temp tables. It does not grant access to the tables inside. For that, you need a second grant:

GRANT ALL ON ALL TABLES IN SCHEMA public TO my_user;

So "I granted the user everything" and "the user can actually read the tables" are two separate statements. Miss the second and your user connects fine, then cannot touch a single row, which is a confusing way to lose an afternoon. If you would rather stay out of the psql shell, createuser and createdb do the same job from the command line, and you set the password with ALTER USER afterwards.

IAM groups: manage permissions for people, not per person

On AWS, the task was an IAM group. Groups exist for the same reason grants do, managing access without repeating yourself. Attach a policy to five users one by one, and you now have five things to update every time the rules change. Attach it to a group, and you have one.

# Create the group
aws iam create-group --group-name Nautilus_Admin

# Confirm it exists
aws iam get-group --group-name Nautilus_Admin
aws iam list-groups

Creating the group is only the opening move. A group with no policy and no members does nothing at all. The next two commands are what make it useful, attaching a policy and adding people:

aws iam attach-group-policy --group-name Nautilus_Admin --policy-arn <policy-arn>
aws iam add-user-to-group --group-name Nautilus_Admin --user-name Bob

Two things worth knowing. A group is not an identity, you cannot sign in as a group, only as a user or a role. And groups do not nest, you cannot put a group inside another group. It is a flat container for users, and that is deliberate.

Same idea, two systems

Both tasks are access control wearing different clothes. Postgres asks who connects and what they can touch. IAM asks who you are and what you are allowed to do. The habit that carries across both is setting access at the level that scales, a grant on the right scope, a policy on the group instead of the individual. Do it, person by person, and it works right up until it becomes a mess nobody can audit.

So here is the Day 17 question. Would you rather set permissions once at the level that scales, or keep patching them one user at a time until you have lost track of who can do what?

Day 17 down. Eighty-three to go.

100 Days of DevOps and Cloud (AWS), Day 16: A Load Balancer in Ten Lines of Nginx, and Least-Privilege IAM

Nnamdi Felix Ibe — Wed, 22 Jul 2026 04:44:35 +0000

You do not need a cloud load balancer to load balance. Ten lines of Nginx will spread traffic across a pool of servers, and understanding those ten lines teaches you what the managed services are quietly doing for you. Day 16 was that, plus setting up IAM the way it is meant to be done, with least privilege instead of a blank check.

One Linux task, one AWS task. Turn Nginx into a load balancer in front of three app servers, then create an IAM user, group, and read-only policy. The tasks come from the KodeKloud Engineer platform.

Nginx load balancer: an upstream block and a proxy_pass

On the load balancer host, install Nginx and get it running:

yum install -y nginx
systemctl enable nginx
systemctl start nginx

On each app server: confirm the port Apache is running on

sudo ss -tulp | grep httpd
# Alternative: systemctl status httpd

Then the config that turns it into a load balancer:

# On the load balancer: edit the Nginx config to set up upstream load balancing
vi /etc/nginx/nginx.conf

# Define the pool of backend servers
upstream app_servers {
    server stapp01:5000;
    server stapp02:5000;
    server stapp03:5000;
}

server {
    listen 80;
    location / {
        proxy_pass http://app_servers;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

The upstream block names a group of backend servers. The proxy_pass points incoming requests at that group. That is the load balancer. By default Nginx spreads requests across the pool in round-robin order, one server after another in turn, and if you want a different strategy like least-connections or IP hash, it is a single extra directive. The proxy_set_header lines matter more than they look. Without them, every backend sees the request as coming from the load balancer itself, so these pass along the client's real IP and protocol, which is what keeps your backend logs and any IP-based logic honest.

nginx -t                 # test the config
systemctl reload nginx   # apply it without dropping connections
curl http://stlb01:80

Same discipline as Day 15, test, then reload. Reload rather than restart here, so the new config applies without cutting live connections.

IAM: grant exactly what is needed, nothing more

The AWS task was IAM done the right way, a user, a group, and a policy that grants only what is needed:

aws iam create-user --user-name Bob
aws iam create-group --group-name Admins

Then the policy document:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "ec2:DescribeInstances",
      "ec2:DescribeImages",
      "ec2:DescribeTags",
      "ec2:DescribeSnapshots"
    ],
    "Resource": "*"
  }]
}

aws iam create-policy \
  --policy-name ec2-readonly-policy \
  --policy-document file://policy.json

aws iam list-policies --scope Local

This is least privilege in practice. The policy allows four specific read-only actions, the Describe calls, and nothing else. The lazy version would be "Action": "ec2:*", which hands over full control of EC2 for what is supposed to be read-only access. Do not do that. Grant the exact actions the role needs and widen only when a real need shows up. The "Resource": "*" looks broad, but most EC2 Describe actions do not support resource-level restrictions, so it is expected here, the actions are already the tight part.

One small syntax trap. The --policy-document value is file://policy.json, with the double slash. Miss it, and the CLI treats your filename as a literal policy string and fails.

The task stopped at creating these pieces. To actually put them to work, you take two more steps, attach the policy to the group with attach-group-policy, and add the user to the group with add-user-to-group, so Bob inherits the read-only access through the group. Managing permissions at the group level instead of per user is the habit that keeps IAM sane as a team grows.

Where this leaves us

Both tasks were about doing the fundamental version properly. A load balancer you build yourself teaches you what the managed one abstracts away. An IAM policy scoped to four actions teaches you the security posture that ec2:* quietly throws away. This whole first stretch of the challenge has been exactly that, the boring, correct version of each skill, because that is the version that holds up when it counts.

So here is the Day 16 question. Would you rather reach for the broad, convenient option and hope it never bites you, or build the tight version now while the stakes are still just a lab?

Day 16 down. Eighty-four to go.

100 Days of DevOps and Cloud (AWS), Day 15: HTTPS on Nginx, and the Incremental Trick That Makes Snapshots Cheap

Nnamdi Felix Ibe — Mon, 20 Jul 2026 21:10:23 +0000

Test before you reload. That one habit is the whole difference between a config change and an outage, and it is baked into how you work with Nginx. Day 15 was TLS on Nginx, where a single broken line can take the site down if you skip the check, and then EBS snapshots, which are quietly one of the best deals in AWS storage.

One Linux task, one AWS task. Configure Nginx to serve HTTPS with provided certificates, then take a point-in-time snapshot of an EBS volume. The tasks come from the KodeKloud Engineer platform.

Nginx TLS: place the certs, wire them up, test before reload

Nginx on RHEL and CentOS comes from the EPEL repository:

sudo yum install epel-release -y
yum install -y nginx
systemctl enable nginx
systemctl start nginx

Then put the certificate and key where the system keeps them:

# Public certificate goes with the certs
mv cert.pem /etc/pki/tls/certs/
# Private key goes in the private directory, root-only
mv key.pem  /etc/pki/tls/private/
sudo chmod 600 /etc/pki/tls/private/key.pem

A word on placement, because it matters. The public certificate belongs in /etc/pki/tls/certs/. The private key should go in /etc/pki/tls/private/, readable only by root. My lab notes dropped both into the certs directory to keep it simple, but in practice the private key gets stricter treatment than the certificate, because it is the secret half of the pair. Keep them separate. Then point Nginx at both:

vi /etc/nginx/nginx.conf
# In the TLS server block:
# ssl_certificate     /etc/pki/tls/certs/cert.pem;
# ssl_certificate_key /etc/pki/tls/private/key.pem;

Now the habit worth burning in:

# ALWAYS validate the config before applying it
nginx -t

# Only if the test passes
systemctl restart nginx

nginx -t parses the config and tells you if anything is wrong without touching the running server. Restart on a broken config, and Nginx can fail to come back up, taking the site with it. Test first, every single time, then restart. Finally, confirm HTTPS is answering:

# -k skips cert verification, fine for a self-signed test cert
curl -Ik https://hostname

The -k flag skips certificate verification, which is fine for a self-signed cert in a lab. In production you want the real chain to validate, so treat skipping the check as a test-only shortcut.

EBS snapshots: full backups at incremental cost

The AWS task was a point-in-time snapshot of an EBS volume, the standard way to back up a cloud disk:

# Snapshot a volume, with a description and a tag
aws ec2 create-snapshot \
  --volume-id vol-0e9bbb3943d871f56 \
  --description "Nautilus Snapshot" \
  --tag-specifications 'ResourceType=snapshot,Tags=[{Key=purpose,Value=nautilus-vol-ss}]'

# Watch it move from pending to completed
aws ec2 describe-snapshots --filters "Name=tag:purpose,Values=nautilus-vol-ss"

Here is what makes snapshots such a good deal. They are incremental. The first snapshot copies the whole volume, but every snapshot after that stores only the blocks that changed since the last one. Take a daily snapshot of a 100 GB volume where a few hundred megabytes change each day, and you pay for those few hundred megabytes, not 100 GB a day. AWS still lets you restore any single snapshot as a full volume, it tracks the block references behind the scenes, so you get full backups at incremental cost.

Two practical notes. Snapshot creation is asynchronous, the call returns immediately, and the status moves from pending to completed on its own schedule, so a script should wait on completed rather than assume. And a snapshot is more than a backup, you can build a new volume from it or copy it to another region for disaster recovery.

The through-line

Both tasks were about protecting something valuable with a cheap, repeatable action. nginx -t protects your uptime for the cost of two seconds before every reload. Snapshots protect your data for the cost of a few changed blocks. The good habits in this work are rarely expensive, they are just easy to skip until the day skipping them hurts.

So here is the Day 15 question. Would you rather build the cheap safety checks into your routine now, or explain later why the one time you skipped them was the time it mattered?

Day 15 down. Eighty-five to go.

100 Days of DevOps and Cloud (AWS), Day 14: Restoring a Broken httpd, and the One EC2 Command With No Undo

Nnamdi Felix Ibe — Sun, 19 Jul 2026 21:08:36 +0000

Some commands you can walk back. Terminating an EC2 instance is not one of them. Day 14 paired a recoverable problem, a web server knocked over by a rogue process, with an unrecoverable one, deleting a server on purpose, and the contrast is the whole lesson.

One Linux task, one AWS task. Track down the process blocking Apache and restore the service, then terminate an EC2 instance and confirm it is gone. The tasks come from the KodeKloud Engineer platform.

httpd: diagnose in order, then restore

When httpd will not start, resist the urge to guess. Work in order. Start with what the service itself reports:

# What does httpd think is wrong
systemctl status httpd
systemctl start httpd

The status output usually names the problem, and a failed bind on the port is the classic one. Before assuming a rogue process, check that httpd's own config is sane, because a wrong port or hostname produces the same "won't start" symptom:

# Check the configured listen port and server name
grep -i listen /etc/httpd/conf/httpd.conf
vi /etc/httpd/conf/httpd.conf
# Fix the ServerName directive if it is wrong: ServerName hostname:<port>

If the config is fine and the port is genuinely taken, then you go hunting for the process holding it:

# Find the PID on the conflicting port
sudo su -
yum install -y net-tools
netstat -tulpn

# Clear it, then bring httpd back
kill -9 <PID>
systemctl enable httpd
systemctl start httpd
systemctl status httpd

kill -9 is SIGKILL, the instant, no-cleanup kill. It is the right tool when a process is wedged and ignoring a polite request, but as a default habit, it is worth trying a plain kill first. The order that matters here is diagnostic: config before process, gentle signal before forceful one. Rushing to kill -9 at the first sign of trouble is how you mask the real cause instead of fixing it.

Terminating EC2: the command with no undo

Day 9 was about protecting an instance from termination. Day 14 is the other side of that lever, actually terminating one, on purpose:

# Terminate the instance
aws ec2 terminate-instances --instance-ids i-1234567890abcdef0

# Confirm the state moved to 'terminated'
aws ec2 describe-instances \
  --instance-ids i-1234567890abcdef0 \
  --query "Reservations[*].Instances[*].State.Name"

There is no recycle bin here. Once an instance is terminated, it and its instance-store data are gone for good. The thing worth checking before you run this is what happens to the storage. By default, the root EBS volume is set to delete on termination, so it goes with the instance. Extra volumes you attached usually survive, they detach instead of being deleted, but do not assume it. The DeleteOnTermination flag is set per volume, and you should know its value before you pull the trigger.

One quirk that confuses people. A terminated instance still appears in describe-instances for a little while, in the terminated state, before it disappears completely. That is normal, not a failed termination. And if this is a box you cannot afford to lose, that is precisely what Day 9's termination protection is for. Turn it on, and this command bounces instead of executing.

The contrast

The two tasks sit at opposite ends of reversibility. A downed web server is a puzzle you can take your time on, because nothing is lost while you diagnose. A termination is a decision you make once and cannot take back. Knowing which kind of action you are about to perform, recoverable or final, is what should change how carefully you move.

So here is the Day 14 question. Before you run a command, do you actually know whether you can undo it, or do you find out only when you try and can't?

Day 14 down. Eighty-six to go.

100 Days of DevOps and Cloud (AWS), Day 13: Firewall Rules Are About Order, and Creating an AMI Reboots the Instance

Nnamdi Felix Ibe — Fri, 17 Jul 2026 19:12:36 +0000

A firewall is not a list of rules. It is an ordered list of rules, and the order is the whole thing. Put the same two rules in the wrong sequence, and you either block traffic you meant to allow or allow traffic you meant to block. Day 13 drove that home, then taught me that snapshotting a server has a side effect the name does not warn you about.

One Linux task, one AWS task. Lock a port down so only the load balancer can reach it, then capture a running EC2 instance as an AMI. The tasks come from the KodeKloud Engineer platform.

iptables: allow the one source, drop the rest, in that order

The goal was to let one machine, the load balancer, reach a port and block everyone else. That is two rules, and their order decides whether it works. First, install and enable the iptables service:

sudo yum install -y iptables-services
systemctl enable iptables
systemctl start iptables

Then the two rules that do the work:

# Allow the port FROM the load balancer's IP only
iptables -R INPUT 5 -p tcp --destination-port 5004 -s 172.16.238.14 -j ACCEPT

# Drop all other traffic to that port
iptables -A INPUT -p tcp --destination-port 5004 -j DROP

iptables reads the INPUT chain top to bottom and stops at the first rule that matches. So the ACCEPT for the load balancer has to come before the DROP for everyone else. If the DROP sat first, it would match the load balancer's packets too and drop them before the ACCEPT was ever reached. The -R INPUT 5 replaces the rule at position 5 to slot the ACCEPT into the right place, and -A appends the DROP to the end, which is exactly where a catch-all belongs.

# Persist the rules so a reboot doesn't wipe them
service iptables save

# Confirm what is saved
sudo iptables-save
iptables -L

Same persistence trap as Day 12. service iptables save writes the rules to /etc/sysconfig/iptables. Until you run it, everything you configured lives in memory only and vanishes on reboot. And a naming gotcha: iptables-save does not save, it prints the current rules to your screen. The command that actually persists is the service iptables save one.

AMI: a full-instance image, with a reboot in the middle

The AWS task was to capture a running instance as an AMI, a reusable image you can launch fresh instances from:

# Create the image from the instance
aws ec2 create-image \
  --instance-id i-1234567890abcdef0 \
  --name "My server" \
  --description "An AMI for my server"

# It takes a few minutes to become available
aws ec2 describe-images --image-ids ami-xxxxxxxxxx

Here is the detail my notes glossed over, and it matters. By default, create-image reboots the instance. AWS shuts it down and restarts it to snapshot the volumes in a consistent state, with all buffered and in-memory data flushed to disk first (per the AWS create-image docs). That is the safe default, but an unexpected reboot on a production box in the middle of the day is not something you want to trigger by accident. If you cannot take the reboot, you add --no-reboot, and then AWS warns it cannot guarantee the filesystem integrity of the image. So it is a real tradeoff, consistency with a reboot or no reboot with a small risk, not a free action either way.

Two more things. The AMI includes snapshots of every attached EBS volume by default, so it is a full-instance image, not just the root disk. And AMIs are region-specific. If you need one elsewhere, you copy it with aws ec2 copy-image.

What connects them

Both tasks looked like a single action and were really about a hidden mechanic. The firewall worked or failed on rule order, something invisible until you know to look for it. The AMI looked like a passive snapshot but carried an active reboot. The theme of the week keeps coming back: the command is easy, and the behaviour underneath it is where the real knowledge lives.

So here is the Day 13 question. Would you rather run commands and get surprised by what they do underneath, or learn the mechanics so nothing about the outcome catches you off guard?

Day 13 down. Eighty-seven to go.

100 Days of DevOps and Cloud (AWS), Day 12: A Port Conflict Killed httpd, and an Attached Volume Isn't a Mounted One

Nnamdi Felix Ibe — Thu, 16 Jul 2026 06:48:07 +0000

Two services cannot listen on the same port. So when Apache refuses to start, the most common reason is boring and specific, something else grabbed the port first. Day 12 was about finding that something and clearing it, then handling the AWS storage version of a job that looks finished but isn't.

One Linux task, one AWS task. Work out why httpd will not come up on its port and fix the conflict, then attach an EBS volume to an EC2 instance. The tasks come from the KodeKloud Engineer platform.

httpd: find the process on the port, then clear it

When httpd fails to bind its port, you need to know what is already sitting there. netstat shows every listening socket and the process behind it:

# Install net-tools if netstat isn't present
yum install -y net-tools

# List listening TCP/UDP ports with the owning PID
netstat -tulpn

# Narrow to the port you care about
netstat -tulpn | grep <port>

The -tulpn flags are worth learning as one unit: TCP and UDP, listening sockets, numeric ports, and the PID. That last piece, the PID, is what lets you act. Once you know which process is squatting on the port, clear it and start httpd:

# Stop the conflicting process
sudo kill -9 <PID>

# Now httpd can bind the port
systemctl enable httpd
systemctl start httpd
systemctl status httpd

A quick word on kill -9. It sends SIGKILL, which ends a process instantly with no chance to clean up. It works, but it is a blunt instrument. Try a plain kill first, which sends SIGTERM and lets the process shut down gracefully, and reach for -9 only when that gets ignored. With the port free, the firewall still has to allow the traffic in:

# Allow inbound traffic on the port, inserted at the top of INPUT
iptables -I INPUT -p tcp --destination-port <port> -j ACCEPT

# Persist the rules so they survive a reboot
service iptables save
systemctl enable iptables

iptables -I inserts the ACCEPT rule at the top of the INPUT chain, so it is checked before any DROP rule further down. And the step people forget, service iptables save, writes the rules to disk. Skip it, and every rule you just added disappears on the next reboot.

EBS: attaching is not mounting

The AWS task was to attach an existing EBS volume to a running instance:

# Attach the volume (same AZ as the instance, always)
aws ec2 attach-volume \
  --volume-id vol-xxxxxxxxxx \
  --instance-id i-xxxxxxxxxx \
  --device /dev/sdf

# Check the attachment state
aws ec2 describe-volumes --volume-ids vol-xxxxxxxxxx

Two things to know. First, the volume and the instance have to be in the same availability zone, the same rule that governs snapshots and network interfaces. Second, and this is the one that catches people, attaching is not mounting. The API call succeeds, and the volume is now wired to the instance, but the operating system still sees a raw disk. It is not usable until you go inside the instance and finish the job:

# Inside the instance
lsblk                        # see the new disk
sudo mkfs -t xfs /dev/xvdf   # format it, only if it is blank
sudo mount /dev/xvdf /data   # mount it somewhere

And a modern wrinkle worth knowing. You asked AWS for /dev/sdf, but on current Nitro-based instances, the OS often shows the volume as an NVMe device like /dev/nvme1n1 instead. The name you request and the name the kernel assigns can differ, so trust lsblk over the device string you passed to the API.

The shared lesson

Both tasks have the same shape. The command returning success is not the same as the job being done. httpd needed the port cleared and the firewall opened. The EBS volume needed formatting and mounting after the attach. Cloud APIs and system services both hand you a green result at the halfway mark, and knowing that is what separates "it says it worked" from "it actually works."

So here is the Day 12 question. Would you rather trust the success message and move on, or know exactly which half of the job that success message actually covers?

Day 12 down. Eighty-eight to go.

100 Days of DevOps and Cloud (AWS), Day 11: A WAR File Deploys Itself, and an ENI Is a Network Card You Can Move

Nnamdi Felix Ibe — Wed, 15 Jul 2026 18:51:09 +0000

Deploying a Java web app to Tomcat sounds like it should need a deploy tool. It does not. You copy one file into one folder, and Tomcat does the rest. Day 11 was a good reminder that a lot of what people call deployment is just putting the right file in the right place.

One Linux task, one AWS task. Install and configure Tomcat, then deploy a WAR file to it, and attach a second network interface to an EC2 instance. The tasks come from the KodeKloud Engineer platform.

Tomcat: install, set the port, drop in the WAR

Tomcat is a servlet container, the thing that actually runs Java web applications. Install it from the package manager, and you get the service plus a standard directory layout under /usr/share/tomcat:

sudo su -
yum update -y
yum install -y tomcat zip

# Enable, start, verify
systemctl enable tomcat
systemctl start tomcat
systemctl status tomcat

The HTTP port lives in server.xml:

vi /usr/share/tomcat/conf/server.xml
# Find the Connector with protocol="HTTP/1.1" and change its port
systemctl restart tomcat

Change the Connector with protocol="HTTP/1.1", and only that one. There is a second Connector for the AJP protocol further down, leave it alone, it is not your HTTP port. Restart Tomcat to apply the change. Then deploy the application:

# Copy the WAR into Tomcat's webapps directory
cp /tmp/ROOT.war /usr/share/tomcat/webapps/

# Optionally expand it in place to inspect or edit files
cd /usr/share/tomcat/webapps
unzip ROOT.war

# Test
curl -i http://localhost:<port>

Here is the part that surprises people. Deploying is copying the WAR into the webapps directory. Tomcat watches that folder and automatically expands and deploys anything you drop in. A WAR named ROOT.war becomes the application at the root path. Unzipping it yourself is optional, handy when you want to look inside or change a file, but Tomcat would have expanded it on its own.

If the app server has no direct internet access, you move the WAR onto it from a jump server with scp, which is how the file got there to begin with. That jump-server-then-scp pattern shows up constantly once you work inside locked-down networks.

ENI: a network card you can move between instances

An Elastic Network Interface is a virtual network card. Every instance has one by default at device index 0, the primary. This task attached a second, pre-existing ENI as a secondary interface:

# List available interfaces to get the ENI ID
aws ec2 describe-network-interfaces

# Attach it as a secondary interface (index 1; index 0 is the primary)
aws ec2 attach-network-interface \
  --network-interface-id eni-0dc56a8d4640ad10a \
  --instance-id i-1234567890abcdef0 \
  --device-index 1

# Verify
aws ec2 describe-network-interfaces \
  --network-interface-ids eni-0dc56a8d4640ad10a

The interesting part is what an ENI carries. It holds its own private IP, any Elastic IP, its MAC address, and its security groups, and it keeps all of that when you detach it from one instance and attach it to another. That is what makes it a failover tool. If an instance dies, you can move its ENI, and its entire network identity to a standby instance, and traffic follows without anything upstream needing to change.

The one hard rule is the same one that governs EBS volumes. The ENI and the instance have to be in the same availability zone. Network interfaces do not cross zones.

The pattern underneath

Both tasks were about how little some things actually take once you understand them. A Java deployment is a file copied into a watched folder. A failover network setup is one interface moved between two machines. The complexity people imagine is often just unfamiliarity, and it shrinks fast once you have done the thing a single time.

So here is the Day 11 question. Would you rather keep treating deployment and networking as intimidating black boxes, or open them once and find out how simple the moving parts really are?

Day 11 down. Eighty-nine to go.

100 Days of DevOps and Cloud (AWS), Day 10: Automating Backups Over SSH, and What an Elastic IP Really Costs Now

Nnamdi Felix Ibe — Mon, 13 Jul 2026 16:32:46 +0000

A backup you have to babysit is not a backup. The whole point is that it runs without you, at 2 am, while you sleep, which means it cannot stop to ask anyone for a password. Day 10 was about making that automation genuinely hands-off and then giving a server an address that does not move.

One Linux task, one AWS task. Write a Bash script that zips a directory and ships it to a backup server over SSH, then associate an Elastic IP with an EC2 instance so its public address survives a reboot. The tasks come from the KodeKloud Engineer platform.

The backup script: zip, ship, and never type a password

The script itself is short. Zip a directory, then copy the archive to a remote backup server with scp:

#!/bin/bash
# Zip the target directory
zip -r /backup/xfusioncorp_official.zip /var/www/html/official

# Copy the archive to the backup server
scp /backup/xfusioncorp_official.zip clint@stbkp01:/backup

The catch is the scp line. By default, it asks for the remote user's password, and a script cannot type one. So before this can run unattended, you set up key-based SSH between the two machines, exactly like Day 7:

# On the source server, once
ssh-keygen                       # generate a key pair if you don't have one
ssh-copy-id clint@stbkp01        # push the public key to the backup server

Now the source server logs into the backup server with its key, no prompt, and scp runs silently. Make the script executable and run it:

chmod +x /scripts/official_backup.sh
/scripts/official_backup.sh

Two practical notes. The /backup directory has to exist before zip writes to it, or the script fails on the first line. And this is the natural partner to Day 6's cron. Drop this script into a crontab line, and the backup runs itself every night. That is the moment a script stops being a command you run and becomes a system that runs on its own.

Elastic IP: a fixed address, and a new line on the bill

On the AWS side, the task was to give an instance a stable public IP. A normal EC2 public IP is temporary, it changes every time the instance stops and starts, which breaks anything pointing at it. An Elastic IP is a public IPv4 address you allocate and hold onto, and it stays put across reboots. Associating one is straightforward:

# Find the allocation ID of your Elastic IP
aws ec2 describe-addresses

# Associate it with the instance
aws ec2 associate-address \
  --instance-id i-0b263919b6498b123 \
  --allocation-id eipalloc-64d5890a

# Confirm the association
aws ec2 describe-addresses --allocation-ids eipalloc-64d5890a

Here is where I have to correct some old advice, including what my own notes said. It used to be true that an Elastic IP was free as long as it was attached to a running instance, and you only paid for idle ones. That is no longer the case. Since February 1, 2024, AWS charges for every public IPv4 address, about $0.005 an hour, whether it is attached or not, which works out to roughly $3.60 a month per address (per the AWS pricing announcement). There is a free tier of 750 hours a month for the first year, but the old attached-equals-free model is gone. The practical advice is what it always was: do not hoard public IPv4, except now it costs you even while in use.

One more detail. The default quota is 5 Elastic IPs per region. It is adjustable, but it exists to discourage exactly that hoarding.

What Day 10 was really about

Both tasks were about doing permanence properly. The backup script only works if the trust between the two servers is set up once and then stays out of the way. The Elastic IP only helps if you treat it as a resource you own and pay for, not a free convenience. Automation and cloud both pay you back for setting things up deliberately and knowing what they cost.

So here is the Day 10 question. Would you rather wire up automation that runs itself and know exactly what each piece costs, or keep doing the work by hand because the setup felt like too much effort up front?

Day 10 down. Ninety to go.

100 Days of DevOps and Cloud (AWS), Day 9: A Database That Won't Start Is Usually a Permissions Problem

Nnamdi Felix Ibe — Sun, 12 Jul 2026 20:35:47 +0000

When a service refuses to start, the instinct is to blame the service. Usually, the service itself is fine, and something around it is broken. Day 9's dead MariaDB was a clean example of exactly that.

One Linux task, one AWS task. Bring a failed MariaDB service back to life by reading the logs rather than guessing, then enable termination protection for an EC2 instance so it cannot be deleted on a bad day. The tasks come from the KodeKloud Engineer platform, the same as the rest of this series.

MariaDB: read the logs, don't guess

A database that will not start feels opaque, and that is what makes it stressful. It is not actually opaque. MariaDB, like most services, tells you exactly why it failed if you look in the right place, so the discipline is looking before you start changing things.

# Where does it stand right now
sudo systemctl status mariadb

# Read the service's own log, newest lines first
sudo tail -30 /var/log/mariadb/mariadb.log

# Narrow the systemd journal to the failure window
sudo journalctl -u mariadb --since "2025-11-09 12:43:00" --until "2025-11-09 12:45:00"

systemctl status gives you the headline, the last few log lines and whether systemd even tried to start it. The service log and journalctl give you the details. Reading the tail first is the shortcut, the newest lines are almost always the ones that matter.

In this case, the log pointed at the runtime directory. MariaDB runs as the mysql user, and it needs to write its socket and PID file into /run/mariadb. That directory was owned by root, so the mysql user could not write to it, and the service died on startup. The fix is one line:

# Who owns the runtime directory
ls -ld /run/mariadb

# Hand it to the user the service actually runs as
sudo chown mysql:mysql /run/mariadb

# Restart and confirm
sudo systemctl restart mariadb
sudo systemctl status mariadb

That is the whole lesson in miniature. The database engine was never broken. A directory it depended on had the wrong owner, and everything downstream fell over. While you are in there, confirm the service is set to come back after a reboot too:

# Make sure the fix survives a reboot
sudo systemctl is-enabled mariadb

EC2 termination protection: a guard against the irreversible

The AWS task is the safety net for the worst kind of mistake, deleting a server you still needed. Terminating an EC2 instance is permanent, so termination protection exists to make it hard to do by accident. Turn it on with one flag:

# Enable termination protection
aws ec2 modify-instance-attribute \
  --instance-id i-1234567890abcdef0 \
  --disable-api-termination

Same naming quirk I hit with stop protection on Day 8. The flag is --disable-api-termination, which sounds destructive but is the switch that protects the instance. It disables the terminate API for that box. To lift it later, you use --no-disable-api-termination.

The note verified this by actually calling terminate-instances and watching it get refused. That works, but if the protection had not applied, you would have deleted the instance to find out. Ask AWS directly instead:

# Confirm protection is on without risking the instance
aws ec2 describe-instance-attribute \
  --instance-id i-1234567890abcdef0 \
  --attribute disableApiTermination

If it returns true, you are protected, and terminate calls will bounce with an error. Two things to keep in mind. Termination protection and stop protection are separate switches, so a box you want fully guarded needs both. And this only blocks the API path, it does nothing about someone deleting the attached volumes, so treat it as one layer, not the whole wall.

The thread

Both halves of Day 9 were about the same instinct, slowing down before you act. The database rewarded a careful read of the logs over a panicked restart. The EC2 flag made the destructive action deliberately awkward to perform. Speed feels productive, but the operators who last are the ones who look first.

So here is the Day 9 question. Would you rather move fast and hope the thing you are about to break is not the thing you needed, or spend two minutes confirming before you commit?

Day 9 down. Ninety-one to go.