A few years ago, I worked on a system that looked completely fine in development.
The architecture was reasonable. The API was fast. Tests were passing. Deployments were automated, and the team had enough confidence to start talking about the next set of features.
Then production traffic arrived.
Within the first week, jobs started disappearing.
Not all of them. That would have been easier.
Roughly one out of every few hundred background jobs would enter the queue, appear to start processing, and then never reach the final state. No useful error. No obvious exception. The customer would simply keep seeing:
processing
At first, we blamed the worker.
We increased timeouts, added retries, restarted processes, and inspected the queue manually. Everything looked normal.
The bug stayed.
That was the first important lesson: when a system fails intermittently, the part that looks broken is often not the part that is actually broken.
The real failure was between systems
The application had three important pieces:
- API server
- job queue
- PostgreSQL database
The API created a database record and then pushed a job to the queue.
The worker processed that job and updated the record afterward.
Simple enough.
The problem was that we treated those two operations as if they were one operation.
They weren't.
Sometimes the database transaction had not fully committed before the worker picked up the queued job.
The worker would start immediately, query the record, fail to find it, throw an error, and retry.
Usually the retry worked.
Usually.
Under heavier load, multiple retries could happen before the database became visible to that worker connection. Eventually the retry limit was exhausted.
The queue considered the job failed, but our application still considered it active.
That was how we created jobs that stayed in processing forever.
Nothing was actually processing them.
The UI was telling the truth according to the database.
The queue was also telling the truth according to the queue.
The system itself was lying because we had no reliable agreement between those two truths.
Fixing one bug exposed another
We changed the flow so the job was published only after the database transaction committed.
That solved the disappearing jobs.
For about two days.
Then users started reporting duplicate results.
Now some jobs were being processed twice.
The first instinct was to prevent duplicate queue messages.
That turned out to be the wrong abstraction.
Queues can deliver the same message more than once. Workers can crash after completing work but before acknowledging the message. Networks can time out while the server successfully processes the request.
Exactly-once delivery sounds nice in diagrams.
In production, I prefer assuming that something will eventually happen twice.
We made the worker idempotent.
Instead of saying:
receive job -> perform action
we moved closer to:
receive job
-> acquire processing state
-> check whether result already exists
-> perform operation
-> persist result atomically
-> mark completed
Processing the same job twice could no longer create two final results.
That was a much stronger guarantee than trying to prove that duplicate delivery would never happen.
Then the deployment broke
A few weeks later, we introduced a database migration.
The change looked harmless: a new non-null column with a default value.
It passed staging.
Production was much larger.
The migration locked a heavily used table long enough to cause API requests to pile up. Connection usage climbed. Workers started waiting on queries. Latency exploded.
Health checks began failing.
Then Kubernetes did exactly what we had configured it to do.
It restarted the pods.
Which made the situation worse.
More pods came online, created more database connections, retried more requests, and increased pressure on the database we were already trying to recover.
Our automation was functioning correctly.
Our assumptions were wrong.
We rolled back the deployment, changed the migration strategy, introduced the column without the expensive operation, backfilled it gradually, and applied the constraint separately.
After that incident, I stopped treating database migrations as deployment details.
They are production code.
A migration has resource usage, locking behavior, failure modes, rollback concerns, and compatibility requirements just like any service.
The biggest improvement wasn't another framework
After several incidents like this, the team became much better at debugging.
Not because we suddenly became smarter.
We finally improved observability.
Every job received a correlation ID.
Logs became structured.
Queue retries became visible.
We tracked job duration, failure count, retry count, queue depth, database latency, and the number of jobs stuck in intermediate states.
We added alerts for abnormal state age instead of only monitoring HTTP errors.
That changed everything.
Before, debugging started with:
"Something is wrong."
Afterward, it started with:
"Jobs from worker group B are spending six times longer waiting for a database connection, beginning five minutes after deployment."
That difference is enormous.
Good observability doesn't prevent failure.
It reduces the amount of guessing you do after failure.
What stayed with me
I've seen developers become frustrated because they spent three days fixing something that they expected to finish in three hours.
I've done it too.
But software rarely fails according to our schedule.
A difficult production issue doesn't mean the architecture is hopeless.
A bad deployment doesn't mean the project is finished.
A broken implementation doesn't mean the idea was wrong.
Sometimes you retry.
Sometimes you roll back.
Sometimes you remove an entire abstraction and rebuild that part from scratch.
And sometimes the uncomfortable answer is that your original design was simply wrong.
That's still progress.
The real danger starts when failure becomes personal.
"This feature failed" slowly turns into "I can't build this."
Those are completely different statements.
Engineering is mostly the process of replacing wrong assumptions with better ones.
We assume the queue won't duplicate work.
Then it does.
We assume staging represents production.
Then production disagrees.
We assume retries improve reliability.
Then retries create a storm.
We assume monitoring means watching CPU and HTTP 500s.
Then customers get stuck for hours while every dashboard stays green.
Each failure removes one bad assumption.
That's why I don't think failure is endless.
Failures have causes.
Causes can be investigated.
Systems can be measured.
Architectures can be changed.
Code can be deleted and rewritten.
Giving up is different.
Once you stop investigating, stop testing, stop rebuilding, and stop asking what assumption was wrong, there is nothing left for engineering to solve.
Failure isn't endless.
Giving up is.
Top comments (1)
Thanks, this was a nice read!