Intro
We can hear quite often that cybersecurity in a companies with low maturity is expensive and it make a questions about what benefits it brings to the business. Why does a development need AppSec\DevSecOps specialists? Why is the cost of fixing a code defect at the design level cheaper than at the development stage, and even more so at the release and production stage. So, and how Shift Left can help save a company money in the long term.
In the material below, we will consider the relevance of the issue, evaluate the effectiveness of AppSec\DevSecOps in investing in the final high-quality product, analyze some metrics, look at the criteria for assessing the effectiveness of AppSec\DevSecOps processes and, finally draw a conclusion - who needs AppSec\DevSecOps and when.
[General statistics]
Let's take a look at some stats:
- Recent studies show 210% new vulnerabilities per year in the National Vulnerability Database (NVD)
- 92% of developers feel pressure to release code to market faster
- Top 50 US university coding programs currently don't require their students to take secure coding courses
Let's consider that almost 95% of data breaches last year were on web apps, and 56% of the biggest incidents in the last 5 years tie back to web app security issues. It often takes more than eight months to find a web app exploit, which means that your business and your customers can be exposed to attackers for an extended period of time.
Attacks on web apps have cost over $7.6 billion, representing 42% of all financial losses from attacks.
What are the difficulties?
Cost to Remediate Vulnerabilities
It often takes up to 7 hours for a vulnerability to be detected, prioritized, and remediated – making your application a sitting duck. When your team does learn of a vulnerability, you need to act quickly to remediate the problem.
Studies show that the average time to detect, prioritize, and remediate one vulnerability is 7 hours.
Let's look at the calculations:
- A team is faced with 5,000 vulnerabilities
- They fix at least 30% of the vulnerabilities = 1,500 vulnerabilities to fix
- 1,500 vulnerabilities @ 7 hours each = 10,500 hours of developer work
- 10,500 hours of developer work @ $72/ hour* = $757,215
The total average cost to remediate vulnerabilities is $757,215 annually.
Let's look at the obvious bills first: the downtime and PR agony of a big, public software failure. Here are a few recent cautionary tales that illustrate the impact of a catastrophic failure or breach.
- The big Facebook outage in 2021 was reported to cost $65 million in advertising revenue, and (temporarily) tanked Mark Zuckerberg's personal wealth to the tune of $6 billion.
- Twitter has famously suffered an ongoing series of outages since Elon Musk's extensive layoffs, during which period the stock price also plummeted to a 3rd of its former value.
- The infamous SolarWinds attack cost 18,000 clients an average of $12 million each. Impacted companies in the U.S. reported an average of a 14% impact on their annual revenue. SolarWinds itself had $40 million in upfront recovery costs, plus their stock plummeted from $25/share in 2020 to about $9.
- In September 2022, hackers stole $160 million from crypto platform Wintermute. In March 2022, hackers stole $620 million in Ethereum cryptocurrency fromm play-to-earn game Axie Infinity. In June 2023, hackers stole $35 million in crypto from Atomic Wallet.
While crypto is clearly a major target, it's far from the only one. According to Cybersecurity Ventures, cybercrime costs alone will reach $10.5 trillion annually by 2025, and the US will shoulder at least one-third of that cost.
Additional Risk. Software errors and loss of investor trust
Boeing's Starliner has suffered from problems related to software errors since 2018, with two failed attempts to launch. Boeing has reported losses totaling $595 million related to the project, and their stock price has suffered significantly.
Investors made their displeasure known when Slack outages showed up in their quarterly earnings, as they'd failed to meet the standards set out in their SLAs. The market responded with a chilly 14% drop in the stock valuation.
British Airways wiped about $200M off their stock price when they stranded hundreds of passengers in airports during a major systems outage.
And to cite just one of many Tesla reliability headlines, Tesla stock fell 5.7% after announcing a major patch to 362,000 of their self-driving vehicles. The company's self-driving software has been plagued with widespread issues for years which have regularly chipped away at the stock price.
[Software Bugs Examples]
Government sector
You might think that in spacecraft engineering, there's a lot that could go wrong. Yes, that's right. Moreover, NASA had several failures because of the bug. On July 22, 1962, their spacecraft Mariner 1 probe heading toward Venus was destroyed just in 293 seconds after launch. Why did it happen? Engineers missed a hyphen in the code. Because of this, the spacecraft was "wrecked by the most expensive hyphen in history." The cost of program failure was $18.5 million in 1962. Today the cost of such a mistake could be approximately $554 million.
Did they learn the lesson and start to make better testing? Not so fast. In 1999, NASA's Mars Climate Orbiter got lost in space after a 286-day journey from Earth. Spacecraft orbited too close to Mars' surface and disintegrated. The reason was that one engineering team used English units of measurement (inches, feet, and pounds). In contrast, the other team used the more conventional metric system (millimeters, meters, and kilograms) for key operations. At the end of the project, two teams forgot to convert different systems into one. The cost of a bug in code is $125 million.
One more punch for NASA was the Genesis crash in 2004. It was meant to bring back space material from beyond Earth's moon. Genesis returned to Earth three years after takeoff with samples of the solar wind for analysis. But it didn't land smoothly. It crashed in Utah. As a result, many of the probe's precious samples were destroyed and polluted, though some were recovered. A NASA report released in 2009 said that Lockheed Martin workers had inverted the position of the probe's accelerometers. Hence, the spacecraft never knew it was decelerating into the Earth's atmosphere and, therefore, never deployed its parachute. This gap in testing cost NASA over $260 million.
Commercial sector
PayPal
One of the largest online payment platforms in the world, PayPal, has also faced a lot of troubles because of software defects. One beautiful morning Chris Reynolds from Pennsylvania became reacher on a $92 quadrillion thanks to a smallPayPal error. They accidentally credit this amount of money to his account. What a surprise to Chris it was when he checked his monthly statement. But such an error was quickly recognized and fixed. By the time Chris Reynolds had logged in, his account had returned to zero.
One more severe security bug in PayPal that they have from 2021 isn't fixed until today. This new unpatched bug lets hackers steal money from PayPal users. With the help of this program defect, attackers can trick victims into unknowingly completing attacker-directed transactions with a single click.
Knight
What if some computer bug made you buy high and sell low? What if such a bug would cost you $440 million? Unreal? But that's exactly what happened to Knight, which nearly bankrupted them.
In 2012 Knight was the largest trader in the U.S., with average daily trading of over $21 billion. In August morning of that year, Knight activated a new trading software… with a bug. When New York Stock Exchange opened that day, the faulty software sent Knight on an acquisition spree. Soon it was buying shares in about 150 companies worth about $7 billion in the first hour.
According to the stock exchange rules, Knight must pay for those shares three days later, but they couldn't as they had no source of funds behind them. Of course, Knight tried to cancel the deal, but the chairman of the Securities and Exchange Commission, Mary Schapiro, refused. Only six stock transactions were reversed.
When Knight understood that the trades would stand, they had to save themselves by selling off the stocks for nothing. Goldman Sachs stepped in and bought all of Knight's unwanted positions for $440 million. By the next summer, the company was acquired by its competitor, Getco LLC. 17 years of dedicated work disappeared in less than one hour.
What went wrong? Several factors caused the failure. Yet, one of the most important was a flag that had previously been used to enable Power Peg was repurposed for use in new functionality. In other words, the program believed it was in a test environment and executed trades as quickly as possible without caring about losing the spread value.
The Security Exchange Commission's report highlighted many other factors. Yet, the critical factor was missing formal code review and unit QA process to check that the software had deployed correctly.
The Exponential Cost of Failure
As seen above, best practices such as DevSecOps and automating SAST throughout the SDLC can produce significant savings by finding and fixing defects and vulnerabilities. The results are higher quality and more secure code that forms the foundation of the software applications or software powering devices. What you will see in this section is that there are other cost factors to consider when measuring the ROI of using SAST solution that are not as concrete to calculate.
The average enterprise individual data breach costs a company $4.24 million, the highest average total cost in the 17-year history of IBM's annual "Cost of a Data Breach Report" for 2021. While this seems astronomically high, you have to consider all the factors involved in solving such a breach. Not to mention the lasting damage to an organization's brand and reputation. In 2020, it was estimated that software defects of all kinds, including software vulnerabilities, cost the economy $2 trillion. Unsurprisingly, this is due to software defects making their way through the entire software development phase to manifest in products delivered to customers.
Here are some things to consider when evaluating the real cost of security vulnerabilities and other software failures:
Risk and liability are high with safety critical devices such as critical infrastructure controls, medical and automotive systems, and aircraft electronics. Failure here could cause human injury or even death. The Prius brake issue turned out to be a software failure that cost Toyota $5 billion to remedy which included the recall of four million vehicles. The Boeing 737MAX accidents and grounding of the airplane is likely to cost Boeing $19 billion.
Brand and reputation might be difficult to monetize but it certainly is a large problem for corporations that have fallen afoul of a large security incident. The Equifax breach and the more recent Solar Winds supply chain attack are two prominent examples. Data breaches increased by 17% in 2021 with several high profile cases like zero day vulnerabilities in Microsoft Exchange Server and the Log4J/Log4shell vulnerability.
Customer experience is a leading differentiator in many of today's applications. Poor implementation (design and coding defects), poor security and poor quality all result in poor customer experience. For example, performance can be a significant customer experience issue: Amazon found that for every 100ms of latency in their online applications costs them 1% in sales. Google found that 500ms delay in search page results dropped traffic by 20%. Customers are flush with choices in today's market, customer experience is key in keeping them.
Patching and recalls are inevitable when serious security vulnerabilities or defects are found. In the Toyota case, they had to recall four million vehicles to patch their software. It's expensive to recall and patch your own software but you are also downloading huge costs on to your customers. Organizations are spending thousands of hours and millions of dollars on patch management for software deployed in their environments. There is both an internal and external cost for security vulnerabilities and defects in software. Every unpatched piece of your code in customer's hands is a liability for them and you as it opens up new threat vectors.
Compliance , especially for public companies is a critical part of their business. Failure to manage the risk to your business due to security incidents can lead to heavy consequences from the Securities Exchange Commission (SEC) or Federal Trade Commission (FTC). For example, the Equifax data breach resulted in $575 million fines payable to the FTC and CFPB. The Home Depot breach was $200 million in fines and the Capital One breach resulted in $190 million. Note these fines are over and above every other cost and liability resulting from the breach.
Cybersecurity insurance coverage and premiums are beginning to be impacted by software quality, safety and security issues. Insurers will begin raising rates or possibly even denying coverage to organization not following DevSecOps best practices. The SolarWinds attack cost cyber insurance vendors more than $90 million.
There is tremendous opportunity in reducing these downstream costs with improved software development, shifting security left and automating testing practices.
[Economy AppSec\DevSecOps]
Effect of software errors on developer labor costs
The time your developers spend finding and fixing bugs is time that costs the business money, plus the opportunity costs of not building new features. As we just covered above, finding and fixing bugs is a big-ticket item in your development lifecycle. So exactly how much does it cost you?
- According to VentureBeat, developers spend 20% of their time fixing bugs
- The average salary for a Software Engineer in the USA is hovering around $100,000
- That's about $20,000/year spent on fixing software errors, per developer
On top of developer salary expenses, unaddressed software errors affect end users, and when your product doesn't work as intended for your customers, support costs start to mount.
You should be aiming for this kind of reactive work (finding and fixing errors, support costs) to take no more than 20% of developer hours, allowing 80% on proactive work (building features and improving your products) rather than vice versa. If we assume that your team adheres to this 80/20 standard, based on a 40-hour work week, the average software developer spends 32 hours fixing errors and replicating issues each month. If you have 50 developers, the combined 1,600 hours of reactive work could potentially ramp up the cost of software errors to US$83,000 in lost time per month.
Then there's the opportunity costs of time spent away from building regular features for customers. Reducing 1,600 hours by 50% to 800 hours would save your company US$41,500 every month in salary expenses, while increasing overall product and software quality.
Impact of errors on developer turnover
If you're asking your development team to dedicate huge portions of their working life to the monotonous task of bug-fixing, you're not helping their job satisfaction. We've established that errors shouldn't occupy more than 20% of developer hours, but there are some nasty stats that show a different reality:
- 26% of developers in a global survey said that they spend half their time fixing bugs
- 10% in the Western U.S. said this occupies up to 75% of their time
- 44% called bug fixing their biggest pain point
- 55% said it kept them from building new features and functionality
- 12% felt "resentful" about manual bug-fixing
- 7% said it makes them want to quit.
This is especially confronting given that the tech sector has the highest staff attrition rates of any industry, and within this, software engineers have an even higher turnover rate of 21.7% (for context, the average across all industries is 10.9%). Plus, the business cost to hire a developer averages $50,000.
The upshot? It's a very good investment to give your development team the right tools so that they can spend less time digging through log files and more time doing rewarding work. Plus, this means they're building new features that strengthen your market offering, so it's a win-win.
Cost of Fixing Bugs
Bugs in code should be found and fixed during the testing phase of the web development life cycle. Otherwise, the real impact of software bugs might cost more than we can imagine. For example, the research of the Systems Sciences Institute at IBM shows that the cost to fix an error found after a product release costs from 4x to 5x times more than the one uncovered during the design stage. Moreover, it costs 100x times more when software bugs are identified in the maintenance phase.
Clearly, it's harder to fix issues when the product is launched and released. The later bugs are tracked, the more negative consequences they have and the more complicated they can be to resolve. Furthermore, late and slow bug fixing can affect product functionality and brand image. Moreover, late bug fixing causes further code changes that might conflict with the initial one – adding to the cost, time, and effort. So it's essential to track and fix bugs during the early stages of development.
How Much Would Bug Fixing Cost You?
In one of our previous articles, "How to hire a dedicated development team: 9 steps to simplify the hiring process" – read here, we showed statistics on the average hourly rate of software development worldwide. Due to this data, we can calculate how much bug fixes cost globally.
The average time to fix the functional (major level) bug before the launch stage is around 12 hours. So, to count the cost of fixing a bug, we take the developer's hourly rate multiplied by the time to fix it. Here is what we received:
| North America | $80*12h = $960 |
|---|---|
| Western Europe | $75*12h = $900 |
| Eastern Europe | $55*12h = $660 |
| Africa | $31*12h = $372 |
| Asia Pacific | $28*12h = $336 |
Now it would be much easier to calculate and understand the cost of the bug fixes in the maintenance phase. Simply multiply all those numbers by x100.
The high cost of late bug fixing is not the only problem. Business owners should consider that fixing bugs in the already released product causes a domino effect. When developers begin to change and fix one part of the code, it ripples on other parts of software code, sometimes even on the website design. Thus, delinquent bug fixing might provoke the second round of SDLC , adding an extra cost to that code change.
Due to late bug fixes, your customers receive a slow and buggy application. You lost your revenue. Moreover, instead of releasing new product features, improving user experience, and moving forward in product development, the engineering team stucks in the fixing process.
According to the CPSQ report, in 2020, the total Cost of Poor Software Quality in the US was $2.08 trillion , and here you can add minus customer interest, plus failed IT projects and time lost. To avoid failure of a new project, we should strive for about 20% reactive work (finding and fixing defects, support) and 80% proactive work (building new features and improving our product). If you delay bug fixing, you start a snowball effect and get 80% reactive work compared to only 20% proactive.
As a matter of fact, delayed bugfixes can affect everything in your business project. It starts from budget overruns, low revenue and results in indirect costs like customer loyalty, brand reputation, wasted time, and the slow death of a project.
The total cost of software bugs can be hard to specify, but a detailed understanding of what software bugs is, and their impact on your business is the first step to reducing wasteful spending. Prioritizing, timely fixing, and focusing on critical errors profits with a successful project and reduces costs.
Experts' opinion
The cleanup cost for fixing a bug in a homegrown Web application ranges anywhere from $400 to $4,000 to repair, depending on the vulnerability and the way it's fixed.
Security experts traditionally have been hesitant to calculate the actual cost associated with bug fixes because there are so many variables, including the severity of the vulnerability, differences in man-hour rates, and the makeup of the actual fix.
John Steven, senior director for advanced technology consulting at Cigital, says Grossman's numbers are "dead on." "Cross-site scripting costs very little to fix, for instance, but the regression rate and 'new findings' rates are very high," says Steven, who has done some number-crunching of his own.
Stevens says security remediation typically occurs outside of the normal development and quality-assurance cycle. It costs an organization about $250 to understand a vulnerability finding, $300 to communicate a vulnerability internally and to get "action," and around $240 to verify the fix itself, he says. A simple bug can take about an hour and a half to fix, he says, or $160, for example, at about $105 per man-hour.
"Endemic problems, like authorization, that require integration with tools take more like 80 to 100 hours," Stevens says, so Grossman's estimate for those cases is right on target, he says.
With XSS, enterprises aren't typically fixing just one XSS bug at a time, either. "Developers tend to fix in batches. So no one fixes [just] one cross-site scripting [bug]," Stevens says. Instead, it's more like eight to 20 at a time, he adds, and while some bugs only cost about $400 to fix, others can cost **$9,000 to $11,000 **to fix.
A cross-site request forgery (CSRF) vulnerability that requires encryption can require 80 to 100 man-hours of resources to repair, he says. But a low-budget $400 XSS fix is likely to cause more problems later. "Retests will uncover related problems or the same problem elsewhere as a result of that kind of 'fix,'" Stevens says.
Still, large sites are facing a major reality check in the costs associated with cleaning up their bugs: WhiteHat's Grossman says it's safe to say that most Websites today are full of vulnerabilities, and finding them is a major challenge. The cost of finding those bugs depends on the route an enterprise takes, whether it's a one-time consultant's vulnerability assessment of $10,000 per site, or a much less expensive vulnerability scan, which is somewhere around $1,000. And that's just finding the bugs, not fixing them, Grossman says.
Next reading AppSec and DevSecOps: part 2 - cost of a bug, cases, effectiveness assessment, ROI will be soon..
Top comments (1)
Hi Ivan, welcome to Dev.to! Great post!
Follow me on High-performing engineering teams, Twitter and Linkedin for more Career, Leadership and Growth advice.