My Server Had Zero Inbound Ports. I Still Got a Shell.

A shell prompt on a server with no SSH key, no port 22, and no inbound firewall rules at all.
From it, a query against a database my own laptop cannot reach, not because of a password,
but because no route to it exists.

That is the end state. Here is the build.

The problem

A SaaS company, 500 paying customers, everything on one server: web app, MySQL, and file
uploads on one box with one public IP. It went down for eleven minutes when the server
restarted. Worse, the logs showed thousands of automated login attempts nightly, and the
database sat at the same address as the app.

The fix is not a better password. It is making the database unreachable.

What actually makes a subnet public

A VPC with four subnets across two availability zones. Public for things that face customers,
private for things that must not.

The precise version, and it is a common interview question: a subnet is public because its
route table sends 0.0.0.0/0 to an internet gateway. Private subnets have no such route.
Not a checkbox, just the presence or absence of a line.

Public points at the IGW. Private points at a NAT gateway, outbound only.

Security groups referencing security groups

Three groups, chained: the ALB accepts HTTP from anywhere, app servers accept 3000 only from
the ALB's security group, the database accepts 3306 only from the app servers' group.

The rules name groups, not IPs. Replace a server tomorrow with a different address and
the rule still works, because it grants access to anything wearing that badge.

Hitting an app server directly on 3000 from a browser timed out. Not refused. Timeout
means packets silently dropped, usually a firewall or missing route. Refused means you
reached the machine but nothing was listening. Useful distinction when debugging.

Redundancy plus health checks

Two app servers in two availability zones, physically separate data centres. An ALB in front
spreads traffic and, more importantly, requests /health every thirty seconds. Three
consecutive failures and that server stops receiving traffic. The failure still happens; it
stops being an outage.

Same URL, different servers, neither reachable directly.

IAM roles, not access keys

Access keys in a config file end up on disk, in the repo, on every laptop that cloned it, and
in backups. They do not expire. Bots scan public repos for them constantly and find them
within minutes.

A role attached to the instance gets temporary credentials, rotated automatically. Nothing to
leak, nothing to rotate, and it cannot be lifted off the machine the way a stolen key can.

The least-privilege policy needs two statements, which is where most first attempts fail:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-bucket" },
    { "Effect": "Allow", "Action": ["s3:GetObject","s3:PutObject","s3:DeleteObject"],
      "Resource": "arn:aws:s3:::my-bucket/*" }
  ]
}

Bucket actions take the bucket ARN. Object actions take /*. Grant one without the other
and you get confusing half-failures.

After swapping the broad managed policy for that one, uploads still worked and aws s3 ls
returned AccessDenied. The denial is the win. A compromised server now reaches one bucket
and nothing else.

RDS: no shell, and that is the point

AWS handles backups, patching, storage, hardware. There is no SSH into an RDS instance,
because the shell existed to do maintenance that is no longer yours. You get a stable
endpoint that survives hardware replacement and failover.

Private subnets, public access off, app servers only.

Session Manager instead of SSH

No key pair, no SSH rule. An agent on the instance holds an outbound connection to AWS, and
sessions come back through it.

No port 22 to scan. No key to store, copy, lose, or rotate. Authentication through your IAM
identity, so removing someone's AWS access locks them out of every server at once. Every
session logged in CloudTrail under a real name. And it reaches private subnets, so no bastion
host.

Trap worth knowing: Amazon Linux ships the MariaDB client, which uses
--ssl-verify-server-cert, not MySQL's --ssl-mode=VERIFY_IDENTITY.

Cost, and the runbook that failed

Cloud bills for existing, not for using. An idle NAT gateway is the same price as a busy one,
roughly a dollar a day.

Budget alarm before any resource existed. Tags on creation. Teardown runbook written up front,
then tested, and testing it found a real gap: deleting the VPC failed because the NAT gateway
and its network interface had to go first, individually. That is exactly how orphaned
resources end up billing for months.

On method

I zoned out on the videos for this module, repeatedly. So I changed approach: every concept
as a specific problem first, named plainly, then built. Ten concepts before a single
resource. By the time I was in the console I was confirming something I understood rather
than following steps I did not.

If videos are not landing for you, that is worth knowing about yourself early.

Full build, screenshots, diagram, retro, and the tested teardown runbook:
https://github.com/vivianokose/nexaops-operations-lab/tree/main/09-cloud-fundamentals