Handling Production Incidents

Everyone working in an organization that has systems running in production has probably experienced one or more production incidents, either in the past or at the moment. If it's happening right now, God be with you, lol.

And if you've never experienced a production incident, you definitely will sooner or later.

By production incidents, I mean situations where bugs make their way into production, a newly deployed fix crashes the server, one or more services become unavailable, or a feature that was previously working suddenly stops working as expected.

When these things happen, how do you respond?

I generally consider two approaches when handling production incidents:

1. Fix-forward.
2. Rollback.

1. Fix-forward

Fix-forward is when you deploy a new change to address an issue in production rather than reverting to a previous version.

For example, let's say you deploy a new version of your application and discover that a particular API endpoint is returning a 500 Internal Server Error because of a bug in the newly introduced logic.

If the issue is well understood and you can confidently implement, test, and deploy a safe fix quickly, you can fix forward.

The idea is to correct the problem with another deployment rather than undoing the previous one.

However, you need to be careful. During a critical incident, deploying another change without properly understanding the problem could introduce even more issues. Speed is important, but so is ensuring that the fix actually addresses the problem.

2. Rollback

Rollback is reverting an application or service to a previously stable version.

I consider rollback when a production incident is serious and a safe fix cannot be deployed immediately, or when reverting to the previous version is the quickest and safest way to restore service.

However, there's a caveat.

Before rolling back, you need to ensure that the previous application version is still compatible with the current database schema and other services.

For example, you might have introduced a new database column or changed an existing database structure during the deployment. If the previous application version doesn't support that change, rolling back the application could introduce another problem.

The same applies to frontend and backend compatibility. You might roll back the frontend to a previous version that expects an API response different from what the current backend provides.

Also, rolling back application code doesn't automatically reverse database changes or undo transactions that have already happened.

This is why database migrations should ideally be backward-compatible, especially when working with systems that require minimal downtime.

3. A hypothetical scenario: A payment service goes down

Let's put these approaches into a practical scenario.

Imagine you're working on a fintech application that allows users to transfer money between bank accounts.

Your team deploys a new version of the backend, which includes some improvements to the payment processing service.

Shortly after deployment, monitoring alerts start coming in. You notice that payment requests are returning 500 errors, and users can no longer complete transfers.

What do you do?

First, you investigate.

You check your application logs, monitoring dashboards, and error reports to understand what went wrong. You discover that a change introduced in the latest deployment is causing the payment processing endpoint to fail.

Now you have a decision to make.

Option A: Fix-forward

If you've identified the problem and the fix is straightforward, you can implement the correction, run the necessary tests, and deploy a new version.

For example, you discover that a condition in the payment validation logic is incorrectly rejecting valid transactions. You correct the condition, test the affected scenarios, and deploy the fix.

You then monitor the service to confirm that payment requests are being processed successfully.

Option B: Rollback

Now imagine the issue is more complicated. The new payment processing logic has introduced several unexpected failures, and you cannot confidently deliver a tested fix immediately.

If the previous application version is compatible with the current database schema and other services, you can roll back to that version to restore the payment service.

But remember, you must also consider transactions that were already initiated before the rollback. Some might have failed, while others might have been processed successfully even though the application returned an error.

You need to ensure that these transactions are reconciled before attempting to process them again. Otherwise, you risk duplicate payments.

The objective is not just to get the service running again but to ensure that users' funds and transaction records remain consistent.

4. How I approach production incidents

Regardless of whether I choose to fix forward or roll back, there are a few things I consider when handling production incidents.

1. Identify the issue

The first thing is understanding what is broken. I check application logs, error reports, monitoring dashboards, and recent deployments to identify the possible cause.

2. Assess the impact

I determine how much of the system is affected. Is the entire application down, or is it just one service? Are all users affected, or is it a particular group? Are we dealing with failed payments, authentication issues, or something less critical?

This helps establish the severity and urgency of the incident.

3. Mitigate the issue

Once I have a reasonable understanding of the problem, I determine the safest way to restore the affected functionality.

This could mean fixing forward, rolling back, disabling an affected feature, or temporarily routing traffic away from an unhealthy service.

4. Verify the fix

After deploying a fix or completing a rollback, I verify that the affected functionality is working as expected.

For a payment service, for example, I wouldn't just check that the endpoint returns a successful response. I'd also verify that transactions are being processed correctly and that the relevant records are being persisted.

5. Monitor the system

I continue monitoring the application to ensure the issue hasn't returned.

I pay attention to error rates, response times, service health, and relevant business metrics.

Getting the application back up doesn't necessarily mean the incident is completely resolved.

5. What happens after the incident?

One thing I believe is important is understanding why the incident happened in the first place.

Once the system is stable, the team should conduct a post-incident review to understand the root cause, what went wrong, how the incident was handled, and what can be done to prevent a recurrence.

This might involve improving automated tests, introducing additional monitoring and alerts, reviewing deployment processes, or making database migrations safer.

The goal isn't to find someone to blame. It's to make the system more resilient.

Conclusion

Production incidents are part of working with software systems, especially when you're building and maintaining applications that people depend on.

What matters is how quickly and safely you can identify the problem, restore service, and prevent the same issue from happening again.

Whether you choose to fix forward or roll back, always consider the wider system. Your application doesn't operate in isolation. It depends on databases, APIs, background workers, and other services.

A fix that restores one part of the system but breaks another isn't really a fix.