The API worked.
Requests returned 200 OK.
The database had the right data.
Tests were passing.
The frontend was happy.
And yet, the system was becoming harder to operate every week.
This is one of the easiest traps in backend development:
A working API does not necessarily mean a working architecture.
An endpoint can be perfectly implemented while the system around it is slowly becoming fragile.
The Endpoint Looks Fine
Imagine a simple endpoint:
POST /orders/
It creates an order and returns:
{
"id": 123,
"status": "created"
}
The implementation might look completely reasonable:
def create_order(data):
order = Order.objects.create(**data)
send_confirmation_email(order)
update_inventory(order)
notify_warehouse(order)
return order
Nothing looks obviously wrong.
The API works.
The tests pass.
The response is correct.
But now imagine that:
- sending the email takes 800ms
- inventory is handled by another service
- warehouse notification sometimes takes 2 seconds
- one of those services occasionally times out
Suddenly, creating an order is no longer just creating an order.
The problem is not the endpoint.
The problem is the architecture behind it.
Correctness Is Not the Same as Architecture
There are several different questions we should ask about a backend system.
Does the API work?
Does it return the expected response?
Is the business logic correct?
Does it enforce the right rules?
Is the system reliable?
What happens when dependencies fail?
Is it scalable?
What happens when traffic increases?
Is it maintainable?
Can another engineer safely change it six months from now?
These are different properties.
A system can be correct and still be difficult to scale.
It can be fast and still be unreliable.
It can pass every test and still have a bad architecture.
One Request Can Do Too Much
A common pattern is gradually adding responsibilities to an endpoint.
It starts simple:
Request
↓
Validate
↓
Save
↓
Response
Then requirements arrive.
"Also send an SMS."
"Also update analytics."
"Also notify the admin."
"Also generate the invoice."
"Also sync with the external service."
Eventually:
┌── Email
├── SMS
├── Analytics
Request → API → DB ─┼── Invoice
├── External API
└── Notifications
Every individual operation might be valid.
The architecture is the problem.
The request lifecycle has become coupled to everything that happens after the database write.
Failure Makes It More Complicated
Suppose the database transaction succeeds.
Then the external API fails.
Create Order
↓
Database ✓
↓
Send Notification
↓
External API ✗
Should the order be rolled back?
Usually not.
But now the API needs to deal with partial failure.
Maybe we retry.
What if the retry succeeds but the API request times out?
The client might retry the entire request.
Now we could create the order twice.
So we need idempotency.
Then an idempotency key.
Then somewhere to store that key.
A seemingly simple endpoint has turned into a distributed systems problem.
Not because someone wrote terrible code.
Because multiple responsibilities were connected to one request.
The Synchronous Everything Problem
One of the easiest architectural mistakes is doing everything synchronously.
def create_order(data):
order = create_order(data)
send_email(order)
send_sms(order)
sync_with_partner(order)
generate_report(order)
return order
The API response now depends on all of these operations.
If the slowest operation takes three seconds:
API latency ≈ database + email + SMS + partner API + report
Even if creating the order itself takes only 20ms.
A better architecture might be:
Request
↓
Create Order
↓
Commit
↓
Response
│
└── Background Work
├── Email
├── SMS
├── Partner Sync
└── Report
The important part is not simply "use a queue."
The important part is:
Separate operations that must happen before the response from operations that can happen after it.
Sometimes that means a background worker.
Sometimes an event.
Sometimes an outbox.
Sometimes simply moving non-critical work outside the request path.
The architecture should follow the actual business requirements.
The Database Can Hide Problems Too
Consider a perfectly valid ORM query:
orders = Order.objects.all()
for order in orders:
print(order.customer.name)
It works.
For 20 orders, it might be fine.
For 20,000 orders, you may have an N+1 query problem.
The API contract didn't change.
The endpoint still works.
But the system doesn't behave well under realistic load.
The same thing happens with:
- missing indexes
- unbounded queries
- large payloads
- expensive serialization
- unnecessary joins
- repeated external calls
A working endpoint is not proof that the data access pattern is healthy.
"We'll Add Caching Later"
Caching is another common architectural band-aid.
A query becomes slow.
Instead of asking why, we add Redis.
Sometimes that's exactly the right solution.
But sometimes the real problem is:
Missing index
or:
N+1 queries
or:
Fetching data we don't need
or:
Doing an expensive aggregation on every request
Adding a cache can make the endpoint faster while hiding the actual bottleneck.
Now we have:
API
↓
Redis
↓
Database
plus:
- cache invalidation
- TTL decisions
- stale data
- cache misses
- cache failures
The system is faster.
But also more complicated.
Performance improvements should start with measurement and understanding the bottleneck.
Microservices Don't Automatically Fix Architecture
Sometimes the response to a growing monolith is:
"Let's split it into microservices."
But moving bad boundaries into separate services doesn't create good architecture.
Imagine:
Order Service
↓
Payment Service
↓
Inventory Service
↓
Notification Service
If every request needs all four services synchronously, you've created a distributed monolith.
Instead of:
Function call
you now have:
Network call
And network calls introduce:
- latency
- timeouts
- retries
- partial failures
- service discovery
- observability requirements
- deployment coordination
Microservices can solve real organizational and scaling problems.
But service boundaries should exist because there is a meaningful boundary in the system.
Not simply because the application has become large.
Architecture Is About Boundaries
One of the most useful questions in backend development is:
What should be allowed to depend on what?
For example:
HTTP Layer
↓
Application Logic
↓
Domain Logic
↓
Persistence
The exact architecture doesn't have to look like this.
The important idea is boundaries.
Your HTTP handler shouldn't need to know how an email provider works.
Your domain logic shouldn't need to know about HTTP status codes.
Your database model shouldn't become the entire business layer.
Your notification system shouldn't determine whether an order is valid.
Good boundaries make change cheaper.
Ask What Happens When Things Fail
A happy-path architecture is easy to design.
The real architecture appears when something fails.
Ask questions like:
What if the database is slow?
Does the entire application become slow?
What if Redis is unavailable?
Can the application continue?
What if an external API times out?
Do we retry? How many times?
What if the client retries the request?
Can we safely process it twice?
What if a worker crashes halfway through a job?
Can the job be retried safely?
What if the same event is processed twice?
Is the operation idempotent?
These questions often reveal more about an architecture than another diagram of the happy path.
The API Is Only the Surface
When we review an API, we often focus on:
- REST conventions
- response structure
- status codes
- validation
- serializers
Those things matter.
But they don't tell the whole story.
A backend system has many layers:
API Contract
│
↓
Application Logic
│
↓
Data Access
│
↓
Infrastructure
│
↓
External Systems
The API is the surface.
The architecture is everything underneath it.
A Better Definition of "It Works"
Instead of asking only:
Does the API work?
Ask:
Does the system behave correctly when the happy path disappears?
A production backend should have reasonable answers for:
- dependency failures
- retries
- timeouts
- duplicate requests
- concurrent operations
- large datasets
- increasing traffic
- partial failures
- background job failures
- database contention
Not every application needs an elaborate solution for all of these.
That's important.
Architecture is not about preparing a small application for billions of users.
It's about preparing the system for the problems it is actually expected to face.
Keep the Architecture Boring
Good backend architecture is often surprisingly boring.
You might end up with:
API
↓
PostgreSQL
And that's fine.
Maybe you later need:
API
↓
PostgreSQL
↓
Background Worker
That's fine too.
Then perhaps traffic grows:
┌── API
Load Balancer ──┼── API
└── API
│
↓
PostgreSQL
│
↓
Workers
Architecture should evolve because requirements evolve.
Not because technology is available.
You don't get architectural maturity by adding more boxes to a diagram.
You get it by understanding why each box exists.
The Real Test
A good architecture should make the next change easier.
If adding a new notification provider requires changing the order creation flow, you have coupling.
If changing the database requires rewriting business logic, you have coupling.
If one external service being unavailable prevents unrelated functionality from working, you have coupling.
If scaling one feature requires scaling the entire application, you may have a boundary problem.
The goal isn't to eliminate coupling.
That's impossible.
The goal is to make coupling intentional and understandable.
Final Thought
The API can work perfectly.
The tests can be green.
The deployment can be successful.
And the architecture can still be wrong for the system you're building.
That's why backend engineering is more than writing endpoints.
It's about understanding:
- dependencies
- boundaries
- failure modes
- data ownership
- consistency
- concurrency
- scaling
- operational cost
The API is what the client sees.
The architecture is what determines whether the system remains healthy as the system grows.
A working API is a feature.
A sustainable architecture is a system.
Read the full version
This article is part of my backend engineering notes.
I publish the full version, along with other practical backend architecture articles, on my personal site:
If you're interested in backend engineering, system design, APIs, databases, and the trade-offs behind real-world systems, you may find the other notes useful too.
Top comments (0)