Five Costs Engineers Don’t Know About Breaking A Critical Path In The App

Introduction

When you break an app in production, the first thing leaders are interested to know is the estimated impact.

The impact is what largely decides what course of action you take thereafter. If you perform retrospectives or postmortems you’d try to be more accurate with your estimates, however, based on my personal experience, most developers fail to see impact beyond direct revenue or crashes.

In this post, I will discuss five different types of costs you incur when you break an app in production.

The five types of costs associated with breaking a flow in your app are:

1. Explicit Loss of Revenue
2. Collateral Damage to the Business
3. Cost of Investigation
4. Cost of Fixing
5. Cost of Remediation

Let’s look at each one of them in a little more detail.

Cost 1: Explicit Loss of Revenue

When you break a critical path in your app, this leads to a loss in revenue.

Depending on the business model you operate under, the revenue loss could be immediate or lagging. For example, if you are selling goods or items that a user will consume, you could notice a drop in revenue soon enough, provided you have sufficient monitoring in place. On the other hand, if you are selling subscriptions, you could lose out on your subscribers in the subsequent cycle which could be a lagging indicator or you would fail to convert users into paying customers.

This is what most people are able to figure out since this is what the bottom lines are made up of.

Cost 2: Collateral Damage to The Business

Apart from losing out on sales and conversion for certain items or subscriptions. You also end up hurting various other aspects of your business when you break a critical path in your app.

Brand Perception

A lot of apps today depend on some form of organic growth. A dent in the brand perception can hurt the business significantly. Since brand perception is not something that you can measure easily. Its effects show up on the charts over months and sometimes even quarters.

App Ratings

Another thing that can easily get affected here is the overall ratings in various app stores. Users that are savvy and care about app ratings can easily pick your competitor over you just because of that number.

User Retention and Engagement

An app or an experience that doesn’t work for a user will eventually lead to them leaving your app sooner than you’d expect them to. Driving users out of your app will bring the overall retention and engagement numbers down. This hurts a lot of businesses that rely on a network of how many active users exist on their app at any given point in time.

Customer Acquisition Cost

The more users you acquire over time, the harder it becomes to target the relevant people with the ads for your product. This leads to a rise in customer acquisition cost. The users that have installed your app either organically or through a paid ad, contribute to this if they leave the ecosystem due to a bad experience.

Cost 3: Cost of Investigation

Depending on the nature of the fire and its impact, you’d start investigating the issues.

Needless to say this is one of the things engineers enjoy the most. However, pulling people away from their regular tasks to deal with fires has a huge impact on productivity since it requires context switching. You also need people who have domain knowledge and technical understanding of the areas that are currently dealing with fire. That kind of expertise may not always be available.

In addition to engineering resources, you would also end up involving Quality Assurance to ensure the fix works as expected and product managers to plan a proper course of action whenever needed.

Cost 4: Cost of Fixing

Once the investigation is complete and you have isolated the issue. The process to fix the issue starts.

Here someone would either commit a new change or revert an old commit, run the tests, and ensure everything works fine. Wait for your continuous integration to pass. Cherry-pick the commit into a new branch. Generate the artifacts, run your regression tests, and eventually submit the builds to the relevant stores.

All of this again, costs time and effort. None of which is free.

Cost 5: Cost of Remediation

Often times when you break the app, product and engineering are not the only ones that are dealing with the fire.

Remember the times when you had to take to social media to get the attention of a business you were frustrated with? Yeah, that happens too and that’s where your marketing and communications teams come in. They have to deal with any negative PR that gets generated due to this issue. In addition to this, your customers could also start flooding your customer service teams.

An increase in customer service inbound leads to either an increased backlog of tickets your team has to deal with or some additional expense your company has to bear by adding more representatives to cater to this issue.

Conclusion

I speak from years of experience working on apps operating in very different industries and teams of all sizes big and small. It is not common to come across engineers who think about these things before shipping their next feature, fix, or a refactor.

Before you make a change be it a feature or a fix, ask yourself. What can go wrong and if there is a way I can either prevent it from happening today or I can immediately mitigate the crash/problem by guarding my change behind a feature flag? It will always be appreciated by your teammates and your managers.