I did something this week that sounds like a mistake and was actually the most useful thing I have learned in months.
I locked myself out of my own server. On purpose. A server that was running fine, serving a live website, with no way back in. No SSH, no password, and no keyboard to plug in, because it lives in an Amazon data centre I will never enter.
Then I recovered it. And now I understand something most engineers only read about.

Logged back into the server that had locked me out.
Let me walk through the whole thing, because the recovery only makes sense once you see how the server was built.
Renting a computer by the hour
The cloud is someone else's computer that you rent by the hour. You get a slice of a machine in a warehouse, use it, and hand it back when you are done. Amazon calls this service EC2, and one rented machine is an "instance."
The mental shift that matters: you pay for the machine existing, not for using it. A server you forgot about costs the same as one serving thousands of people. That single fact shapes every decision in this module, and it is why cost discipline shows up at the end.
Locking the front door before building anything
The very first step had nothing to do with servers. It was locking down the account itself.
When you sign up for AWS, you get a "root" account that can do anything, including run up unlimited charges if the credentials leak. There is a whole economy of bots scanning the internet for leaked AWS keys, and they spin up expensive servers within minutes of finding one.
So the first move is to lock root away behind multi-factor authentication and create a weaker, everyday identity to actually work with. That everyday identity is an IAM user. Root is the master key you lock in a drawer; the IAM user is the normal key you carry.

Root account secured with multi-factor authentication.

An everyday IAM user, in an Administrators group, so I never work as root.
The server that would not let me in
With the account secured, I launched my first instance. And immediately hit a wall.
SSH timed out. Not "permission denied," which would mean I reached the server but was not allowed in. A timeout, which means the knock got no answer at all.
My security group, AWS's firewall, was correct: SSH open to my IP, web traffic open to everyone. So why the silence?
I worked through it one layer at a time. Was my key wrong? A wrong key gets rejected, not ignored. Had my home IP changed? It matched. Was the subnet missing its route to the internet? The gateway was there.
Then I found it. A network ACL, a second firewall sitting above the security group, set to deny all traffic. Someone had configured this account's network in a non-standard way long before I arrived. My rules were perfect; the layer above them blocked everything.
I made a call that felt like the professional one: stop patching someone else's confusing setup, and build my own clean network from scratch. A fresh VPC, one public subnet, a proper gateway, all wired correctly. I launched into that, and SSH worked instantly.
The instance running in my own clean network, finally reachable.
The lesson stuck harder than any happy-path tutorial could have taught it. A timeout is not one problem, it is a checklist: security group, your IP, the route, the network ACL. And when you inherit a mess, rebuilding clean often beats untangling.
Hardening, against real threats this time
I had hardened Linux servers before, but always on my own laptop where the threats were hypothetical. This one had a public address, which means bots start probing it within minutes of it going live.
Same baseline, real stakes: patch everything first, put up a firewall that denies everything by default and opens only SSH and the web ports, and run fail2ban to jail anything that hammers the login repeatedly.
The firewall: everything denied by default, only SSH and the web ports open.

fail2ban standing guard, ready to jail repeat login offenders.
Giving it a name and a padlock
A raw IP address is no way to reach a website. So I pointed a subdomain of my own domain, app.viviancloud.site, at the server, while carefully leaving my main domain on its existing site. One domain, different subdomains for different things, which is exactly how real companies organise this.
Then HTTPS. For years, certificates cost money and were fiddly. A nonprofit called Let's Encrypt made them free and automatic. One tool, one command, and my site loaded with a padlock and a certificate that renews itself. The plain-HTTP postcard became a sealed envelope.

My own page, served from the EC2 instance.

Reachable by name at app.viviancloud.site.

The padlock: HTTPS live, certificate valid and auto-renewing.
Data on its own disk
I attached a second disk, separate from the one the operating system lives on. The reason is practical: keep your data apart from your OS, and you can rebuild the OS whenever you like without touching the data. Format it, mount it, and add it to the system's startup config so the mount survives a reboot. I proved it by actually rebooting and watching the disk come back on its own.

The separate data disk, still mounted after a full reboot.
The lockout, and the way back
Now the part I opened with.
To learn recovery for real, I created the disaster. I emptied the file that lists which keys are allowed to log in, disconnected, and confirmed I was locked out. Permission denied, on my own running server.
But before breaking anything, I took a snapshot: a full backup of the disk, my undo button.

A snapshot first. Never break something risky without a backup.
Here is the recovery, and the single idea that makes it possible: the file that decides who can log in lives on the disk, and a disk is a movable object.
So I stopped the server, detached its disk, and attached that disk to a second "rescue" instance as a spare drive. From the rescue instance, I could reach into the locked server's files, and I wrote my key back onto its login list, with the exact ownership and permissions SSH insists on. Then I detached the disk, reattached it to the original server as its boot drive, and started it up.
I logged in. The same key that got "permission denied" ten minutes earlier now worked.
It is exactly like being locked out of your house, unbolting the front door, carrying it to a locksmith who fits a new lock you have a key for, and rehanging it. The house never changed. You fixed the lock by taking the door somewhere you could work on it.
The deeper realisation: a server accepts any key whose public half is written in its login file. So a lost key is never a dead end. You do not recover the old key, you generate a new one and install it. The lock is always replaceable.
There are faster recovery routes when you set them up in advance, but the disk-swap always works, with no agent and no preparation. It is the universal fallback, which is why it is the one worth learning first.
Building carefully, tearing down completely
The module ended where cloud work always should: accounting for the cost and knowing how to remove everything.
Total spend for all of this: zero dollars, by staying on free-tier resources and stopping the instance between sessions. I set a billing alarm as a tripwire, and wrote a teardown runbook listing every resource and how to remove it in order, because a resource you forget is a resource you keep paying for.

A billing alarm: emails me if estimated charges ever cross five dollars.
What I actually learned
Not just the steps. The principles underneath them.
That cloud bills on existence, not use. That you default everything to closed and open only what each service needs. That a lost key is a lock to replace, not a catastrophe. And that knowing how to cleanly destroy infrastructure matters as much as knowing how to build it.
I locked myself out of a server and got back in by hand. I will not forget how that works.


Top comments (0)