7 backend lessons I learned the hard way

I'm building my first mobile app with microservices. Here are 7 backend lessons I learned the hard way.

1. Own your data, or nobody does.
   Each service owns its own data. No shared database that every service reads and writes directly. The moment two services share a table, a change in one breaks the other, and nobody can say who owns the truth anymore.

2. Make retries safe.
   Networks fail and clients retry. If the same request twice creates two charges, two rows, or two emails, the retry becomes the bug. The rule I keep: the same request, safely repeated, must produce the same result, with no duplicate side effect.

3. Put the rule in the database, not just the app.
   Validation in the app is good. A constraint in the database is the one every future app, script, and service has to respect. Apps get rewritten. The database is the last writer standing.

4. Auth lives in one place.
   One auth service, verified at the boundary. Not a login check copied into every handler, drifting a little each time it is copied. When auth changes, it should change in one place.

5. Version your contracts.
   Services talk through APIs, and APIs are promises. Change a field, rename a response, or drop an endpoint without warning, and you break callers who did nothing wrong. Version it, keep the old one working, and give callers a path forward.

6. Observe before you scale.
   Shipping is not the same as knowing. CI/CD that deploys small changes, metrics that show what is actually happening, and logs you can search will tell you more than a bigger server will. If you cannot see it, you cannot fix it.

7. Ship the smallest backend that works, then harden it.
   A small backend you can change beats a clever one you are afraid to touch. Get the core working, watch how it behaves, then add the complexity it has actually earned.

None of these is clever. Together they are the difference between a backend you can keep building on and one that fights you every time you touch it.

I write about backend systems that look small until they aren't. Follow along if that's your world; the longer pieces live on my Substack.