A broke man’s guide to running enterprise grade identity software on AWS
A little while ago I set myself a challenge. I wanted to take a serious piece of identity software, the kind big companies use to handle everyone’s logins and run it properly on Amazon’s cloud. Not a cut down demo on my laptop but the real thing. Two servers sitting in two different data centers, a proper database behind them, firewalls, monitoring and alarms. And the whole thing had to cost me nothing.
This is the story of how I got there. I have tried to keep it plain and honest including the bits where things went wrong, because that is usually where the learning actually happens.
What I was actually building
The software at the center of all this is called WSO2 Identity Server. Think of it as a receptionist whose only job is checking who you are. Instead of every app you build handling its own usernames and passwords, they all ask this one server whether you really are who you claim to be and whether you are allowed in. It is the same idea as the “Sign in with Google” button, except here I am running the identity server myself.
The goal was to wrap that receptionist in a proper, safe building. One public front door, two receptionists in case one falls ill, a locked records room for the data, cameras watching everything and a guard who turns away anyone poking at the staff entrance. Most tutorials skip all of that and just put the receptionist on the pavement. I wanted the whole building.
And I wrote every piece of it as code (IaC), using a tool called Terraform, so I can build the entire thing with one command and tear it all down again with another.
The money rule that shaped everything
Here is the constraint that made this interesting. Newer AWS accounts come with a free plan that hands you some credits, somewhere around a hundred to two hundred dollars worth and the lovely part is that the account simply cannot be charged. When the credits run out Amazon pauses things rather than sending you a bill. So “no cost to me” was a real guarantee and not wishful thinking.
The twist is that those credits still drain by the hour while things are switched on. So the trick was never to leave it all running. I would build the whole setup, test it, take my screenshots and then destroy it again, every single session. Because it is all written as code, bringing it back is one command and about ten minutes. The entire project cost me a few dollars of credits in the end.
Phase 0
Before any servers existed I set up the environment that saves you later. I created a budget alarm that emails me if spending ever climbs, which on the free plan is more of a comfort blanket than a real risk. I set up a safe, encrypted place for Terraform to keep track of what it has built. And I copied the two big installers, the WSO2 software and the Java it runs on into Amazon storage once, so that every rebuild pulls them quickly from inside AWS instead of dragging them across the internet again. All of it makes every later step faster and calmer.
Phase 1
This is the part people skip and later regret. I carved out my own private slice of Amazon’s network and split it into three layers. A public layer where the front door lives. A private layer for the servers. And a sealed off layer for the database that has no route to the internet at all.
Then I put firewalls between them. The front door only accepts secure web traffic from the outside world. The servers only accept traffic from the front door. The database only accepts traffic from the servers. Each layer can only talk to the next one in line.
The mental model that finally clicked for me goes like this. There are two kinds of firewalls here. One is fine grained and controls exactly who may talk to whom. The other is a blunt instrument that seals off an entire zone. You really want both and now I understand why.
Phase 2
Next came the database, a managed MySQL database from Amazon. It sits in that sealed off layer and I made sure of two things. It has no public address so I genuinely cannot reach it from my laptop, which is exactly the point. And its passwords are generated automatically and kept in a digital vault called Secrets Manager , never written into any file or into the code. The servers fetch them only when they start up.
There was a neat side effect. Because the database is not reachable from outside, I could not set up its tables from my laptop either. That job had to be done from inside the network, which became a task for the next phase.
Figure 1. The database running privately inside AWS
Figure 2. The database instance as it appears in the AWS console
Phase 3
Now the fun part. I wanted a server that could build itself from nothing. Here is the flow I automated. A plain Amazon Linux machine starts up empty. A tiny startup note tells it to fetch a larger setup script from storage and run it. That script installs Java, unpacks WSO2, grabs the database password from the vault and writes out the server’s configuration with the right database address and credentials already filled in. Then it connects to the database, sets up the tables and starts the server.
One thing I am quietly proud of is that there is no SSH involved anywhere. To get a shell on a server I use Amazon’s Session Manager , so there is no open remote login port and no key for me to lose.
Then came the satisfying moment of proof. The server logged that it had started in 60 seconds. Its health check came back healthy. The database now held well over 200 tables which proved the server was really using the cloud database and not some throwaway local one. And there were zero connection errors.
Figure 3. The server up and running in AWS
The plot twist nobody warns you about
My very first attempt to launch a server failed. The error simply said the server type I had chosen was not eligible for the free plan.
It turns out the free plan only lets you launch certain blessed server types. The obvious small one I picked was not on the list. And the truly tiny ones that are on the list have only 1 GB of memory, which is nowhere near enough for WSO2 which wants at least 4 GB.
The way out took some digging. Amazon has newer flex servers that are on the eligible list and come with real memory. One of them has 4 GB and another has 8 GB. I went with the 8 GB one and suddenly everything fit, still at no cost. This is the kind of thing you only ever learn by walking straight into it.
Phase 4
A single server is not high availability. So I added a second one in the other data center and put a load balancer in front of both. The load balancer is the single public web address that everyone visits. It quietly checks both servers every 30 seconds and only sends visitors to a healthy one. It also keeps each visitor stuck to the same server during their session so that logins flow smoothly.
I also gave it a certificate so all the traffic is encrypted. I do not own a domain name for this yet, so the certificate is a self-made one which means browsers show a warning even though the encryption itself is real. Swapping in a proper certificate later is a tiny change.
Figure 4. The load balancer — the single front door
I made the servers fully self-sufficient too. Each one when it boots, installs itself, sets up the database if it happens to be the first to arrive and starts serving. No manual steps. That is the thing that makes the idea of killing a server and having a fresh one quietly take its place actually work.
The clustering headache
The two servers need to find each other and behave as one system, so that a change made on one is known to the other. WSO2 has a mode that is supposed to discover the other servers on AWS automatically. It quietly did nothing. The library that powers that feature is not included in this version, so each server thought it was completely alone. Two servers, each convinced it was the only one.
To fix this I had each server ask Amazon directly at startup, which servers carry the same cluster label and then configure itself from that answer. After that change the logs showed exactly what I wanted. One server was elected as the leader and the other joined it. One cluster and two members, working as a pair.
Proof it all works
Both servers showed up as healthy behind the load balancer, one in each data center. Visiting the public address took me to the login page over a secure connection, the admin console loaded and the standard identity document was served correctly. A real request through the load balancer using the admin account came back successful and a request with a wrong password was correctly refused. So the whole chain works from the public internet, through the load balancer, into a server and down to the database with real authentication at the end.
Figure 5. The admin console, served through the load balance
Figure 6. The identity server publishing its OpenID configuration
Figure 7. The identity server publishing its OpenID configuration
Phase 5
A system you cannot see into is a liability, so I wired up monitoring. A small agent on each server ships the important logs up to Amazon’s cloud logging. The main server log, the audit log that records who logged in and who failed, and the web access log.
Then I set up alarms that watch those logs and the load balancer for trouble. A sudden spike of errors. A burst of failed logins, which is the classic sign of someone guessing passwords. Server errors, an unhealthy server or a database working too hard. Each one sends me an email.
Finally, I built a dashboard that pulls it all together in one place, so I can glance at traffic, the number of healthy servers, memory use, database load and a live count of failed logins.
Figure 8. The monitoring dashboard
Figure 9. The alarms standing guard
Figure 10. The CloudWatch overview, every alarm green and watching quietly
I tested the whole alert chain from start to finish, and a real test email landed in my inbox. So I knew that when I later attacked my own system, the failed login alarm would actually fire. That alarm is the tripwire the attack demo would spring.
Phase 6
Up to this point the admin console was reachable by anyone on the internet which is not acceptable for the control panel of an identity system. So I closed it off.
The load balancer now checks where each request is coming from. The admin pages only answer if the request comes from my own internet address. Everyone else gets a flat refusal, a plain forbidden response. The normal login pages stay public, because they have to be. I tested it both ways. It loads for me and it slams shut for everyone else.
Figure 11. Anyone else trying the admin console gets turned away
I also built a web firewall that I can switch on, which blocks common attacks and more importantly for what came next, limits how fast a single attacker can hammer the login. I tightened what the servers themselves are allowed to reach out to, down to only the things they genuinely need. And I turned on account lockout so that after a handful of wrong passwords an account locks itself for a while, even if the next guess happens to be correct.
There is one real world wrinkle worth admitting. My home internet address changes every so often and the console lock is pinned to it. So now and then I have to update one line and apply it again.
Phases 7 and 8
This is the part I built the whole thing for. Two demonstrations. First, a real person logging in. Then, proof that the system shrugs off both a server dying and an actual attack.
A real login
I built a tiny demo app that signs a user in through the identity server using the standard OpenID Connect flow and then shows the identity it got back. I registered the app, created a test user and logged in. The app received a signed token and when I looked inside it, the token had genuinely been issued by my own server, for my own test user. Logins work.
Figure 12. The little demo app, signed in through my identity server
Figure 13. Registering the demo app inside WSO2
Figure 14. The sign in options switched on for the app
Killing a server in the middle of everything
Then the honest stress test. While logins were happening, I abruptly killed one of the two servers. The result was exactly what real high availability looks like, resilient. A couple of requests stumbled for a few seconds while the load balancer noticed the server was gone. Then every login succeeded again on the survivor. And the system healed itself. A fresh server was launched to replace the dead one all on its own, back to two servers across two data centers without me lifting a finger. Because the servers keep no unique data of their own and everything lives in the shared database, losing one lost nothing.
While that server was down, the monitoring did its job too and the unhealthy server alarm dropped into my inbox without me asking.
Figure 15. The unhealthy server alarm arriving by email
Attacking my own login
Finally, the fun one. I pointed a script at my own login page and hammered it with around 150 wrong passwords, a small taste of what a real password guessing attack looks like. Then I sat back and watched every layer of defense catch it.
The account locked itself. After a handful of wrong tries the target account was locked and from then on even the correct password was refused. So the attacker is stopped even if they eventually guess right.
Figure 16. The account locking itself after too many wrong tries.
It was all recorded. Every failed attempt flowed up into cloud logging and a counter I had set up tallied 55 failures.
Figure 17. Failed logins piling up in the logs
An alarm fired and emailed me. The too many failed logins alarm tripped and sent a real warning email, the detection layer working live in front of my eyes.
Figure 18. The alarm going off
Figure 19. And the warning landing in my inbox
The firewall blocked the flood. Once the attacker crossed the speed limit, the web firewall started turning requests away at the very edge before they ever reached a server. The firewall’s own console tells the story plainly. Out of 535 requests in that window it let 385 through and blocked 150 of them.
Figure 20. The firewall’s running tally of allowed and blocked requests
You can even see where the blocking happens. The traffic flows into the rate limit rule and a thick slice of it is stopped right there while the rest carries on to the servers.
Figure 21. The traffic hitting the rate limit rule and a slice being blocked
The spikes in blocked traffic line up exactly with my two attack bursts.
Figure 22. Allowed and blocked totals spiking during the attack
Figure 23. The rate limit rule doing the blocking during the bursts
The firewall could even tell that a good slice of this traffic was not a real browser at all. It flagged around 160 of the requests as unverified bot activity rather than genuine visitors.
Figure 24. The firewall flagging the attack as unverified bots
Here is the wider view of everything the firewall was seeing across that traffic, and a closer look at the individual requests it turned away.
Figure 25. the firewall’s traffic overview
Figure 26. A closer look at the requests it turned away
Three separate layers, the app locking accounts, the cloud watching and alerting and the firewall blocking at the door, each caught the same attack in its own way. That is what people mean by defense in depth, shown working rather than just claimed.
I want to be honest about one thing here, because it is the most useful lesson of the lot. My failed login alarm did not work on the first try. My first guess at what a failed login looks like in the logs was simply wrong, so the alarm sat there doing nothing. I only found the real signal by running the attack and reading the actual log lines it produced and then fixing the alarm to match. You cannot trust a detector you have never tested.
Figure 27. The real audit log lines from the attack, the ones I had to learn to read
Figure 28.Reading the real log lines and fixing the detector to match
What it cost and what I walked away with
The whole thing ran on free credits. While it is switched on it costs a few cents an hour and switched off, which is how it sits between sessions, it costs essentially nothing. The entire project came to a few dollars of credits and the free plan could not have charged me even if I had forgotten to turn it off.
A few lessons really stuck with me. The two kinds of firewall finally make sense, one for fine control and one for sealing off whole zones and you want both. The free tier has sharp edges and it will not let you launch ordinary servers, which you find out by walking into the wall. The phrase about clustering happening automatically often means it does nothing of the sort and you end up wiring it together yourself. High availability is not instant and a dying server causes a few seconds of wobble before things settle. And you genuinely cannot trust a security alarm you have not tested against a real attack.
If I were to take this further, I would add a proper domain name and certificate, build a ready-made server image so new servers start in seconds instead of minutes and double up the networking so even that has no single point of failure. Those are the natural steps from a great learning lab towards something you would actually run in production.
Everything I built is written down as code in the repository, every firewall rule, every config file, every demo script. If you fancy it, you can stand the whole thing up, attack it yourself and tear it back down. That honestly, was the best way I have ever found to actually learn this stuff.
The entire stack is open source on GitHub at:
It is all Terraform, split into modules for the network, the database, the compute fleet, the load balancer, the monitoring and the firewall, alongside the cloud init scripts and the attack demo. Run terraform apply to stand the whole thing up in about ten minutes and terraform destroy to wipe it back to nothing.
Top comments (0)