DEV Community

Cover image for Crisis Response: How Rapid Risk Aversion Becomes Technical Debt
Paimon Sorornejad
Paimon Sorornejad

Posted on

Crisis Response: How Rapid Risk Aversion Becomes Technical Debt

Intro

The world is currently in a crisis. As the spread of virus makes its way through countries and continents, businesses are ramping up their crisis response teams. This response is not isolated to any one form of business, and ranges from corporate, industrial, agricultural and educational. Businesses with critical IT footprints are rapidly reacting to possibilities of large scale remote asset management, not just with bare-metal, but with people.

This rapid response is a necessity for companies to continue to serve their customers, protect their people, and preserve their revenue. Unfortunately, there is an unavoidable side effect of rapid response in IT, and that is technical debt.

What Is Technical Debt

Technical debt is a natural byproduct of business driven IT. As funded projects make their way through delivery, developers are often left with risk based decision making that help stay on course, both financially and temporally.

These decisions can be as simple as leaving out comments in code, to more complex situations where features are built rapidly to serve an immediate business need without a long term support strategy.

Many businesses are finding themselves in this exact dilemma as they are responding to the current COVID-19 crisis. As engineers rearrange production priorities, many are left to quickly gap fill for potential business impacts and increased observability. As financial impacts are being felt around the world, there is little time to waste to ensure that the business continues delivery while strengthening their response and recovery plan.

Do Now, Fix Later

Rapid response has a direct negative impact to well architected solutions, assuming otherwise is a denial that technical debt exists. This denial however is present in many organizations, and it is unfortunately a part of historical business culture that didn't embrace failures as learning opportunities. Doing now, and fixing later is more often than not required and essential to the sustained operation of a business. With that being said, we need to make sure that we stay honest that there is a "fix later" phase, which means we need to accept that our solution isn't perfect, and there is always room for increased conformity and improvement.

Embracing Technical Debt

For a business to succeed as a force in IT, the fact that technical debt is an influencer to innovation must be embraced. Engineers need to understand that there are different solutions to common problems, and problems with no immediate solutions need to be documented and shared.

Technical debt can range from physical challenges ("we do not have the data transfer bandwidth to meet our SLO"), technical challenges ("our automation is taking too long to complete"), as well as financial challenges ("my current infrastructure is expensive to scale out") but it is important to note that initially one doesn't outweigh the other.

As we encounter these debts, we need to encourage ourselves and our staff to document them in a common place for everyone to see. Teams that have the luxury of being located in the same physical location should use visual cues like a whiteboard, post its, or even pin ups. The goal is to ensure that these debts are always in sight and not forgotten.

Virtual teams are not exempt from this process, and should use any content management tool at their disposal. For example, the mechanism for tracking can range from a shared cloud to-do list, to tools like bug trackers or agile project management software. The purpose is to ensure that all members of the team have the ability to contribute, and encouraged to do so.

These debts should be simple and conform to a set of loosely defined rules. For example, treat it like a "tweet" from your favorite influencer. The most impactful messages are the ones that are short, and direct. We are not trying to propose a solution, we are simply stating the problem we encounter while doing our jobs, or a time when we had to break away from a defined pattern. Keeping it as simple as possible ensures that the practice is not a distraction to our daily jobs.

Paying Back Debts

As time goes on, there will be certain debts that are encountered multiple times, and possibly by multiple people. The most important part of debt tracking is recording them as they reoccur, for example with a check mark for physical lists, or some sort of virtual marker. This will help build a natural priority to approaching our debts, and which ones may have the largest return on investment. Remember, the reason some of these debts were created was due to financial, technical or timing constraints therefore not all of them can be solved, or will need to be solved. Some will be the most expensive financially to solve, but provide the least amount of ROI because they are not encountered often.

Now that debts are being tracked, they must be reviewed and reminded of. Depending on the size of the organization, its agility, and other possible factors, reviews can be done after several weeks, or even months. This should become part of the normal business practice so that the debts are embraced as a team, and members become less adverse to avoiding them.

Respond Rapidly, Resolve Rapidly

Technical debt is unavoidable, and its introduction to the business landscape is required to respond effectively not only to crisis, but shifting production priorities, employee turnaround, as well as accelerated delivery timelines. We must encourage our teams to embrace these debts, not shy away from them, and expose them as soon as possible.

As teams are responding to pandemic, rapid deployment of infrastructure, architecture, and solutions will naturally need to take the quickest path to production. There will be processes that will break for the sake of agility, and checks and balances that will be temporarily bypassed. Stay vigilant to your priorities, but don't lose sight of any shortcuts that were needed to be successful.

Top comments (0)