Most software doesn't fail because the developer forgot how to write code.
It fails because the code was built around assumptions that eventually stopped being true.
We assume the network will respond.
We assume data will be fresh.
We assume a process will keep running.
We assume a database will always be available.
We assume an external API will behave exactly as documented.
We assume the state in our application represents the real state of the system.
Most of these assumptions are reasonable.
The problem is that software systems eventually encounter the situations where they aren't true.
And that's where reliability starts.
The Happy Path Creates False Confidence
Consider a simple API call:
Application
↓
API Request
↓
Response
↓
Process Data
It works.
You test it locally.
You deploy it.
Everything looks fine.
Then one day:
Application
↓
API Request
↓
Timeout
Now the application has a decision to make.
Should it retry?
Should it stop?
Was the request processed?
Is the local state still valid?
A single line of code suddenly becomes a system-design problem.
That's why the happy path is not enough.
A reliable system needs to define what happens when the expected sequence is interrupted.
A Timeout Doesn't Always Mean Failure
One of the most dangerous assumptions in distributed software is:
No response = operation failed
That isn't necessarily true.
Imagine an application sends a request:
Client
|
| request
↓
Server
The server processes it successfully.
But before the response reaches the client:
Server
|
| response
X
Network failure
The client sees:
TIMEOUT
But the operation may already have happened.
If the client blindly retries, it may perform the same operation twice.
This is why reliable systems often need concepts such as:
- idempotency;
- request identifiers;
- state reconciliation;
- retries with limits;
- explicit unknown states.
The important lesson is that communication failure and operation failure are not always the same thing.
Stale Data Is Still Data
Another dangerous assumption is that data is valid simply because it exists.
Imagine a service receives information from another system.
The last update arrived one minute ago.
The application still has a perfectly valid object in memory.
Nothing is technically broken.
But the information may now be useless.
Last Update
↓
Time Passes
↓
Local Data Remains
↓
Application Uses It
This is why systems that depend on real-time information often need freshness checks.
Instead of asking only:
Do I have data?
the application should sometimes ask:
How old is this data?
That small difference can completely change system behavior.
State Is Harder Than Functions
Developers naturally think in terms of functions:
fetch()
calculate()
save()
send()
But long-running systems are often easier to understand in terms of state.
For example:
STARTING
↓
CONNECTING
↓
READY
↓
RUNNING
↓
RECOVERING
↓
READY
Now failures become transitions rather than unexpected events.
For example:
RUNNING
↓
CONNECTION_LOST
↓
RECOVERING
↓
SYNCING
↓
RUNNING
This way of thinking is useful for many kinds of software:
- background workers;
- payment systems;
- monitoring services;
- distributed applications;
- desktop applications;
- data pipelines;
- automated trading systems.
The exact states change.
The principle doesn't.
Recovery Is Different From Restarting
A common approach to failure is:
Something went wrong
↓
Restart application
Sometimes that works.
But restarting a process doesn't automatically restore the state that existed before the failure.
Imagine:
Application
↓
Operation started
↓
Process crashes
When the application starts again, it needs to know:
Did the operation finish?
Did it partially finish?
Is it still running?
Should it be retried?
Is the previous state trustworthy?
This is why recovery often requires synchronization with the external system.
A useful pattern is:
Restart
↓
Load Local State
↓
Query External State
↓
Compare
↓
Reconcile
↓
Resume
The application doesn't simply continue from where it thinks it stopped.
It verifies reality first.
External Systems Will Eventually Behave Differently
Even when an API is well designed, the surrounding environment can change.
A response can be delayed.
A field can be missing.
A rate limit can be reached.
Credentials can expire.
A service can temporarily become unavailable.
A dependency can return an unexpected error.
This doesn't mean that every possible failure needs hundreds of lines of defensive code.
It means important assumptions should be explicit.
For example:
if data is stale:
stop processing
if dependency is unavailable:
retry with backoff
if state is inconsistent:
reconcile
if recovery fails repeatedly:
stop and alert
The exact implementation depends on the application.
The important part is that the behavior is intentional.
Not Everything Should Be Retried
Retries are useful.
They are also easy to misuse.
A simple pattern looks like:
error
↓
retry
↓
error
↓
retry
↓
error
↓
retry
This can turn a small problem into a much bigger one.
For temporary network failures, a limited retry with backoff may be reasonable.
For an inconsistent financial transaction, blindly retrying may be dangerous.
For corrupted application state, retrying the same operation may accomplish nothing.
A reliable system should first ask:
What kind of failure is this?
Then choose the recovery strategy.
Sometimes the correct answer is retry.
Sometimes it is reconciliation.
Sometimes it is simply stopping.
Observability Changes Everything
When something fails in production, the question isn't only:
What exception happened?
A more useful question is:
What was the system doing when it happened?
Good logs can describe a sequence:
Request started
Dependency connected
Data received
Validation passed
Operation started
Timeout detected
State changed: RUNNING → RECOVERING
Reconciliation started
State restored
System resumed
This gives you a story.
Without that story, debugging becomes guesswork.
For long-running software, logging and monitoring aren't decorations added after development.
They are part of the architecture.
Reliability Is About Controlled Failure
Perfect software doesn't exist.
Networks fail.
Servers fail.
Processes crash.
Dependencies become unavailable.
Data becomes stale.
The goal isn't to prevent every failure.
The goal is to make failure predictable.
A reliable system should be able to answer:
What failed?
What state are we in?
Can we safely retry?
Do we need to reconcile?
Should we stop?
How do we know that recovery succeeded?
That's a much stronger design than simply adding more error handling.
What I Learned From Building Real Systems
Working on larger projects changed how I think about software.
The interesting part is often not the main algorithm.
It's everything surrounding it.
State.
Networking.
Retries.
Validation.
Recovery.
Logging.
Concurrency.
External dependencies.
A small function can be perfectly correct while the overall system is still unreliable.
The difficult engineering work begins when you ask:
What happens when my assumptions are wrong?
That's the question that turns a prototype into a real system.
I've been exploring these ideas while building different parts of CryptoBot, where external APIs, real-time data, execution state, and recovery all have to work together.
The project is available here:
https://github.com/pavloaser23/crypto-trading-bot
Final Thought
The software usually isn't broken when the unexpected happens.
The unexpected situation is often exactly what the software should have been designed to handle.
The real problem is the assumption that everything will continue working exactly as it did during the first successful test.
Good engineering starts when we stop designing only for:
everything works
and start designing for:
something will eventually go wrong
Because it will.
The question is whether the system knows what to do next.
Top comments (0)