Boundaries Before Networks
The slide was titled "Target Architecture."
Five boxes. Five services. Clean arrows between them: Users, Billing, Notifications, Orders, Reporting. Each one with its own database, its own repository, its own deployment pipeline.
I made that slide. I presented it with confidence. The team was excited. It looked like the kind of system real companies build.
We had six backend developers.
It took me almost a year to admit that the slide was the most expensive mistake of my career so far.
Why It Felt Right
I want to be fair to the version of me who made that decision, because his reasons weren't stupid.
- The monolith was getting messy. Our Django project had grown to dozens of apps. Models imported from everywhere. A change in billing could break notifications, and nobody could explain why.
- Scaling was coming. We expected traffic to grow fast. Microservices would let us scale each part on its own.
- Teams would grow. Separate services meant separate ownership. People could work without stepping on each other.
- It was what good engineers did. Every conference talk, every architecture blog, every system design interview pointed the same way.
Every one of those reasons was real.
But each one solved a problem we expected to have. Not a problem we actually had.
What It Actually Cost
The costs didn't arrive all at once. They arrived the way architecture costs usually do: quietly, one sprint at a time.
Every feature became a distributed feature
Before the split, adding a discount to an order meant changing a few files and running the tests.
After the split, it meant touching Orders, Billing and Notifications. Three repositories. Three pull requests. Three deployments, in the right order. One contract between them that somebody had to keep in sync by hand.
A two-day task became a one-week task.
We lost transactions
In the monolith, creating an order and charging the customer happened inside one database transaction. If one step failed, everything rolled back.
Across services, there was no single transaction. We needed retries, idempotency keys, compensation logic, and a background job to clean up orders that were created but never paid.
We spent months building infrastructure to get back something Django's transaction.atomic() used to give us for free.
Debugging became archaeology
A user reported a missing email. Before, I would open the logs and follow one request.
After, the request passed through an API gateway, two services and a message queue. Each one had its own logs, its own timestamps, and its own idea of what happened. Finding one bug could take an afternoon.
Local development broke
New developers needed five services, three databases, Redis and RabbitMQ running before they could test a single screen. Onboarding went from one day to almost a week.
The team was too small for the boxes
Five services. Six developers. That's barely one person per service, which meant no real ownership, just everyone context-switching across everything.
We didn't build a distributed system because we needed one. We built one because it looked like what serious systems look like.
The Moment I Knew
There was no single disaster. Just one ordinary planning meeting.
We were estimating a small feature: letting users download an invoice as a PDF. A product manager asked why it would take two sprints.
I started explaining. The invoice data lived in Billing. The user details lived in Users. The order lines lived in Orders. Reporting had its own copy of half of it, slightly out of date.
Halfway through, I heard myself and stopped.
The feature wasn't hard. Our architecture made it hard.
That night I went back to the original slide and asked the question I should have asked a year earlier:
"Which of these problems did we actually have on the day we made this decision?"
The honest answer was: messy code boundaries. That's it. Not scale. Not team size. Just a monolith without structure.
And you don't fix messy boundaries by adding network calls between them.
What I'd Build Instead
If I could go back, I'd build a modular monolith.
One codebase. One deployment. One database. But with strict internal boundaries, the same kind of boundaries microservices force on you, without the network in between.
In Django, that looks something like this:
project/
├── orders/
│ ├── models.py
│ ├── services.py # the only public entry point
│ └── selectors.py # read-only queries
├── billing/
│ ├── models.py
│ ├── services.py
│ └── selectors.py
└── notifications/
├── services.py
└── tasks.py
The rules are simple, and they matter more than the folder structure:
- Apps talk to each other only through
services.pyandselectors.py. No importing another app's models directly. - Business logic lives in the service layer, not in views, serializers or model
save()methods. - Cross-app side effects go through clear functions or signals you can trace, not hidden coupling.
- Each app could become a service later, because its boundary already exists.
That last point is the key. A modular monolith doesn't reject microservices. It delays them until you have a real reason, and makes the split cheap when that day comes.
Get the boundaries right first. Decide where the network goes later.
When Splitting Actually Makes Sense
I'm not saying microservices are wrong. I'm saying they're a solution with a price, and you should only pay it when you have the problem.
Before splitting a service out, I now ask for evidence of at least one of these:
- A real scaling difference. One part of the system needs 10× the resources of everything else, and you've measured it.
- A real team boundary. A separate team owns this area, deploys on its own schedule, and is slowed down by everyone else's releases.
- A real reliability need. A failure in this part must not take down the rest, and you can't isolate it any other way.
- A real technology need. It genuinely needs a different language, runtime or data store.
"It feels cleaner" isn't on the list. "We might need it someday" isn't either.
What We Did About It
We didn't rewrite everything back into one project overnight. That would have been the same mistake in reverse.
We merged slowly. Users and Billing went back into the main codebase first, because almost every feature touched both. We kept Notifications as a separate worker, because it really did have different scaling and failure needs.
Five services became two. Features got faster. Onboarding got shorter. And debugging went back to following one request through one log.
The Real Lesson
The mistake wasn't choosing microservices. The mistake was designing for the company we imagined, instead of the team we had.
Architecture decisions feel like technical choices. They're really bets about the future. And the more confidently you draw the boxes, the easier it is to forget that you're betting.
These days, before I present any "target architecture," I add one more slide. It has one question on it:
"What problem do we have today that this solves?"
If I can't answer it with something real, the boxes stay on the whiteboard.
Your turn 👇
What's an architecture decision you'd reverse if you could? I'd genuinely like to hear the ones other engineers learned the hard way. Share yours in the comments.
If this was useful, follow me here for more stories from the backend trenches, and subscribe to The Backend Tech Lead on Substack for weekly blueprints.
Top comments (0)