Farhad Rahimi Klie

Posted on Jan 23

How to Write Unbreakable Production Code

#production #performance #programming #code

A Practical Guide for Engineers Who Care About Reliability

Introduction

Most code does not fail because of syntax errors or lack of intelligence.
It fails because of assumptions, edge cases, scale, and time.

“Unbreakable code” does not mean bug-free code. That is a myth.
It means code that survives real users, real data, real load, and real failures without collapsing.

The goal is to help you write software that:

Fails predictably
Is observable and debuggable
Is resistant to misuse and change
Can be safely maintained for years

1. Design for Failure, Not for Success

The fundamental mistake

Most developers write code assuming:

Inputs are valid
Dependencies are available
Networks are reliable
Disk and memory are infinite

In production, every one of these assumptions is wrong.

Production mindset

You must assume:

APIs will timeout
Data will be malformed
Files will disappear
Memory will fragment
Clocks will drift
Users will do the worst possible thing

Practical rules

Every external call must have a timeout
Every operation must handle failure
Every function must define what happens when things go wrong

If you don’t define failure behavior, the runtime will define it for you.

2. Be Explicit, Not Clever

Clever code breaks silently

Short, clever, “smart” code often:

Hides intent
Encourages misuse
Makes debugging painful

Explicit code survives teams and time

Production code is read far more often than it is written.

Prefer this mindset:

Clear > Short
Obvious > Elegant
Boring > Smart

Example principle

Name variables after meaning, not mechanics
Split complex expressions into steps
Avoid magic values
Avoid “just trust me” logic

Unbreakable code is self-documenting through structure, not comments.

3. Validate Everything at the Boundary

Trust nothing that enters your system

Every boundary is hostile:

HTTP requests
Files
Environment variables
Database records
User input
Config files

Boundary validation rules

Validate type, range, format, and presence
Reject invalid data early
Fail fast with clear errors

Key insight

Once data is validated at the boundary, internal code can be simpler and safer.

Do not spread validation everywhere.
Centralize it at entry points.

4. Make Invariants Impossible to Break

What is an invariant?

An invariant is a condition that must always be true.

Examples:

A user ID is never null
Money is never negative
Passwords are never stored in plain text
State transitions follow strict rules

How unbreakable systems enforce invariants

Through types
Through constructors
Through encapsulation
Through restricted APIs

Rule

If breaking an invariant is possible, someone will do it.

Do not rely on “developer discipline”.
Rely on compiler checks, runtime checks, and architecture.

5. Defensive Programming Without Paranoia

Defensive ≠ Messy

Defensive programming does not mean:

Endless if statements
Catching every exception everywhere
Swallowing errors

It means:

Checking assumptions
Failing loudly
Protecting critical paths

Practical defensive techniques

Assertions for impossible states
Guard clauses for invalid inputs
Explicit error types instead of generic ones
No silent failures

If something should never happen, crash or alert.

Silent corruption is worse than downtime.

6. Logging Is Part of the Code, Not an Afterthought

Production debugging reality

You will debug:

Without a debugger
Without reproducing locally
Under time pressure
With partial data

Logs are your only truth.

Logging rules

Log intent, not noise
Include identifiers (request ID, user ID)
Log failures with context
Never log secrets

Anti-patterns

Logging everything
Logging nothing
Logging without structure

Good logs turn chaos into evidence.

7. Errors Are Data, Not Strings

The string error problem

Returning error strings leads to:

Fragile comparisons
Localization issues
Impossible automation

Production-grade error handling

Use structured errors
Attach metadata
Categorize errors (validation, IO, auth, logic)

Result

Easier retries
Better metrics
Clearer incident response

Errors should be actionable, not just readable.

8. Write Code That Can Be Deleted

Longevity is not about permanence

The best production code:

Is easy to change
Is easy to remove
Has minimal coupling

How to achieve this

Small modules
Clear interfaces
No hidden dependencies
Feature isolation

If deleting a feature feels dangerous, the code is already broken.

9. Concurrency and State: Be Ruthless

Shared state is the enemy

Most catastrophic production bugs come from:

Race conditions
Improper locking
Implicit shared state

Survival rules

Minimize shared mutable state
Prefer immutability
Make state transitions explicit
Serialize when correctness matters more than speed

Performance bugs are annoying.
Concurrency bugs destroy trust.

10. Test Behavior, Not Implementation

Unit tests are not enough

Unbreakable systems use:

Unit tests
Integration tests
Property-based tests
Failure tests

What to test

Contracts
Edge cases
Error paths
Boundary conditions

Key idea

Test what the system promises, not how it currently works.

Refactors should not break tests.
Behavior changes should.

11. Configuration Is Code

Configuration causes production outages

Missing values
Wrong formats
Environment drift

Treat configuration seriously

Validate on startup
Fail fast if invalid
Version it
Document it

If the system starts successfully, it should be safe to run.

12. Simplicity Is a Feature

Complexity compounds over time

Every abstraction has a cost:

Cognitive
Operational
Debugging

Production wisdom

Start simple
Add complexity only when forced
Remove complexity aggressively

Unbreakable systems are not complex.
They are disciplined.

Conclusion

Writing unbreakable production code is not about brilliance.
It is about humility.

Humility toward failure
Humility toward users
Humility toward future maintainers
Humility toward time

Production code must assume:

“Someone else will maintain this under pressure, at 3 AM, during an outage.”

If your code survives that scenario, it is on the path to being unbreakable.