I have been writing code professionally for ten years. React, Node, PostgreSQL, PHP before that. I have shipped a 17-module ERP that 100 people use every day to run a business. I manage the servers it runs on, the DNS, the hosting, the deploys.
Until a few weeks ago I thought that meant I understood infrastructure.
It does not. What it means is that I have been very good at keeping one machine alive.
The system I actually run
Our ERP has 17 modules. Inventory, procurement, order management, reporting, the rest. Real users, real transactions, real money moving through it.
Here is the architecture:
┌─────────────────────────┐
│ One server │
│ │
Users ────────► │ Node.js API │
│ PostgreSQL │
│ File storage │
│ Background jobs │
│ │
└─────────────────────────┘
That is the whole diagram. There is nothing I left out.
It works. It has worked for 1 year. Uptime has been fine. Nobody has complained loudly enough for it to become a priority.
But I drew it out properly for the first time last month, and two things became obvious that I had been carefully not looking at.
Problem one: everything is one thing
If that server goes down, all 17 modules go down. Not degraded. Down. The API, the database, the files, the scheduled jobs, all of it lives in the same place and dies at the same time.
I have no answer to "what happens if this machine fails" other than "I restore a backup and we lose a few hours."
For a system that runs a business, that is not a strategy. That is a thing I have been getting away with.
Problem two: the reporting module
This one is more specific and more embarrassing.
When a user clicks "generate report," the request hits the API and the API generates the report synchronously. That takes about 50 seconds for a typical report, longer for the month-end ones.
During those 50 seconds, that request is holding a connection and burning CPU. Ten users click it at the same time at month end, which is exactly when they all do, and the entire ERP slows to a crawl. Not just reporting. Everything. Order entry gets slow because someone is exporting a spreadsheet.
My solution so far has been to buy a bigger server. It worked. It works every time. It also costs money every month for capacity that sits idle for 27 days out of 30, and it does not fix anything, it just moves the failure threshold further out.
This is not a database problem. I spent a while treating it as one, adding indexes and tuning queries. It is an architecture problem. The work should never have been on the request path in the first place.
What I actually don't know
Being specific about this because I think vagueness here is how people avoid starting.
I have never designed a system across multiple availability zones
I have never used a message queue in production
I do not know what RTO and RPO mean in any way I could defend in a meeting
I have never written infrastructure as code. Every server I run was configured by hand, by me, in a terminal, with steps I did not document
I have configured nginx as a load balancer, but I could not tell you what a managed load balancer does differently
I understand DNS well and networking poorly, which I have recently learned are not the same skill
That last one surprised me. I have managed DNS for 40+ properties for years. I assumed that meant I knew networking. Then I read about VPC subnets and route tables and realised my hosting provider had been doing that layer for me the entire time and I had never once thought about it.
The plan
I am studying for AWS Solutions Architect Associate, and then migrating pieces of this ERP onto AWS for real.
The certificate is a side effect. What I want is the thing the syllabus forces me to learn, which is the set of questions I currently do not know to ask.
First project: get report generation off the request path.
Requests go onto a queue instead of blocking
Workers pick them up, generate, write the output to object storage
User gets notified when it is ready
If a worker dies mid-job, the message goes back on the queue instead of vanishing
I know roughly what this looks like conceptually. I have never built it. I expect the gap between those two states to be where all the interesting content comes from.
After that: managed database with a standby in a second availability zone, CDN in front of the static assets, and eventually the whole thing described in Terraform so it can be rebuilt without me remembering what I did.
The commitment
Every two weeks, here, for the next twelve months. What I built, what I got wrong, what it cost.
I am publishing this partly so I cannot quietly stop. But mostly because the posts I have found most useful while figuring out where to start were the ones where someone admitted what did not work. There is a lot of AWS content that walks you through a clean path to a working result. There is much less that says "I did this and it fell over and here is why."
I would like to write the second kind.
One ask
If you have made this same transition, from building applications to designing the systems they run on, I would rather hear what took you longest to understand than what you got right.
Specifically: what is the thing you got wrong early that you only understood much later?
Top comments (1)
I really appreciate your candor. Specifically in this statement.
“_ What I want is the thing the syllabus forces me to learn, which is the set of questions I currently do not know to ask._”
I don’t know a lot of people (in person) who wrestle with that question but it is one I think about whenever I’m learning something or trying out something new.
Good luck on your training and I hope that you find it rewarding and extremely practical.