DEV Community: Nick Harley

11 of the most costly software errors in history

Nick Harley — Mon, 10 Dec 2018 22:24:11 +0000

In this post, we take a look at some of the biggest disasters over the years to see what happens when software errors cause chaos!

NASA's Mars Climate Orbiter

On its mission to Mars in 1998 the Climate Orbiter spacecraft was ultimately lost in space. Although the failure bemused engineers for some time it was revealed that a sub contractor on the engineering team failed to make a simple conversion from English units to metric. An embarrassing lapse that sent the $125 million craft fatally close to Mars' surface after attempting to stabilize its orbit too low. Flight controllers believe the spacecraft ploughed into Mars' atmosphere where the associated stresses crippled its communications, leaving it hurtling on through space in an orbit around the sun.

Ariane 5 Flight 501

Europe’s newest un-manned satellite-launching rocket reused working software from its predecessor, the Ariane 4. Unfortunately, the Ariane 5’s faster engines exploited a bug that was not found in previous models. Thirty-six seconds into its maiden launch the rocket's engineers hit the self destruct button following multiple computer failures. In essence, the software had tried to cram a 64-bit number into a 16-bit space. The resulting overflow conditions crashed both the primary and backup computers (which were both running the exact same software).

The Ariane 5 had cost nearly $8 billion to develop, and was carrying a $500 million satellite payload when it exploded. Read more here.

EDS Child Support System

In 2004, EDS introduced a highly complex IT system to the U.K.’s Child Support Agency (CSA). At the exact same time, the Department for Work and Pensions (DWP) decided to restructure the entire agency. The two pieces of software were completely incompatible, and irreversible errors were introduced as a result. The system somehow managed to overpay 1.9 million people, underpay another 700,000, had US$7 billion in uncollected child support payments, a backlog of 239,000 cases, 36,000 new cases “stuck” in the system, and has cost the UK taxpayers over US$1 billion to date.

Soviet Gas Pipeline Explosion

The Soviet pipeline had a level of complexity that would require advanced automated control software. The CIA was tipped off to the Soviet intentions to steal the control system's plans. Working with the Canadian firm that designed the pipeline control software, the CIA had the designers deliberately create flaws in the programming so that the Soviets would receive a compromised program. It is claimed that in June 1982, flaws in the stolen software led to a massive explosion along part of the pipeline, causing the largest non-nuclear explosion in the planet’s history.

Bitcoin Hack, Mt. Gox

Launched in 2010, Japanese bitcoin exchange, Mt. Gox, was the largest in the world. After being hacked in June, 2011, Mt. Gox stated that they’d lost over 850,000 bitcoins (worth around half a billion US dollars at the time of writing).

Although around 200,000 of the bitcoins were recovered, Mark Karpeles admits "We had weaknesses in our system, and our bitcoins vanished.”

Heathrow Terminal 5 Opening

Just before the opening of Heathrow's Terminal 5 in the UK, staff tested the brand new baggage handling system built to carry the vast amounts of luggage checked in each day. Engineers tested the system thoroughly before opening the Terminal to the public with over 12,000 test pieces of luggage. It worked flawlessly on all test runs only to find on the Terminal's opening day the system simply could not cope. It is thought that "real life" scenarios such as removing a bag from the system manually when a passenger had left an important item in their luggage, had caused the entire system to become confused and shut down.

Over the following 10 days some 42,000 bags failed to travel with their owners, and over 500 flights were cancelled.

The Mariner 1 Spacecraft

On a mission to fly-by Venus in 1962, this spacecraft barely made it out of Cape Canaveral when a software-coding error caused the rocket to veer dangerously off-course, threatening to crash back to earth. Alarmed, NASA engineers on the ground issued a self-destruct command. A review board later determined that the omission of a hyphen in coded computer instructions allowed the transmission of incorrect guidance signals to the spacecraft. The cost for the rocket was reportedly more than $18 million at the time.

The Morris Worm

A program developed by a Cornell University student for what he said was supposed to be a harmless experiment wound up spreading wildly and crashing thousands of computers in 1988 because of a coding error. It was the first widespread worm attack on the fledgling Internet. The graduate student, Robert Tappan Morris, was convicted of a criminal hacking offense and fined $10,000. Morris's lawyer claimed at the trial that his client's program helped improve computer security.

Costs for cleaning up the mess may have gone as high as $100 Million. Morris, who interestingly co-founded the startup incubator Y Combinator, is now a professor at the Massachusetts Institute of Technology. A disk with the worm's source code is now housed at the University of Boston.

Patriot Missile Error

Sometimes, the cost of a software glitch can't be measured in dollars. In February of 1991, a U.S. Patriot missile defence system in Saudi Arabia, failed to detect an attack on an Army barracks. A government report found that a software problem led to an inaccurate tracking calculation that became worse the longer the system operated. On the day of the incident, the system had been operating for more than 100 hours, and the inaccuracy was serious enough to cause the system to look in the wrong place for the incoming missile. The attack killed 28 American soldiers. Prior to the incident, Army officials had fixed the software to improve the Patriot systems accuracy. That modified software reached the base the day after the attack.

Pentium FDIV bug

When a math professor discovered and publicized a flaw in Intel's popular Pentium processor in 1994, the company's response was to replace chips upon request to users who could prove they were affected. Intel calculated that the error caused by the flaw would happen so rarely that the vast majority of users wouldn't notice. Angry customers demanded a replacement for anyone who asked, and Intel agreed. The episode cost Intel $475 million.

Knight's $440 Million Error

One of the biggest American market makers for stocks struggled to stay afloat after a software bug triggered a $440 million loss in just 30 minutes. The firm's shares lost 75 percent in two days after the faulty software flooded the market with unintended trades. Knight’s trading algorithms reportedly started pushing erratic trades through on nearly 150 different stocks, sending them into spasms.

Honourable mention: NOAA-19 Satellite

Although not a software error, on September 6, 2003, this satellite was badly damaged while being worked on at the Lockheed Martin Space Systems factory. The satellite fell to the floor as a team was turning it to a horizontal position. An inquiry into the mishap determined that it was caused by a lack of procedural discipline throughout the facility. Turns out that while the turn-over cart used during the procedure was in storage, a technician removed twenty-four bolts securing an adapter plate to it without documenting the action. The team subsequently using the cart to turn the satellite failed to check the bolts, as specified in the procedure, before attempting to move the satellite.

Repairs to the satellite cost $135 million.

Don't want to get caught out by your software bugs? Get automatically notified of your software errors with instant notifications. Book a demo with an experienced team member or sign up for a free trial.

5 Application Performance Monitoring tools for improving user experiences

Nick Harley — Wed, 22 Nov 2017 20:57:16 +0000

Not all user experiences are equal. People can have good and bad ones, even when using exactly the same website or mobile application.

Meanwhile, developers code away and release new features thinking users aren't complaining, and everything is fine (when it's not).

When was the last time you submitted a report for a software error? Many customers won't take the time to create a support ticket when they experience a slow loading landing page, they'll just close your browser and leave. Fortunately, these wasted interactions are preventable using Application Performance Monitoring tools. In this post we've compared a selection of tools currently on the market for software teams (including our own for comparison).

Why you need application performance monitoring tools

Part of the problem is that different operating systems, browsers, connection speeds, devices and locations mean that any number of combinations could be interacting with your application.

Local testing is useful but fails to identify the vast amount of variables that users can encounter when trying to use your app.

Synthetic testing also comes up short. How do you fake millions of users interacting with your application and get an accurate view of what may happen when you deploy into production?

Nothing will beat real user interactions with your application in production for the most accurate view. Getting a window into this activity is also a lot easier than you think if you have the right tools.

Here we take a look at five Application Performance Monitoring tools that will monitor your application in production and beyond.

1. New Relic

New Relic is widely used in software engineering teams, particularly for their APM (Application Performance Monitoring) tool, as they've been around for many years now.

Handy features like the User Interaction Traces allow you to track individual user interactions in your application by providing code-level data traces. You get visual timelines for any activity in your app giving you insight into how your app uses memory, database, and CPU. Interaction traces also provide insight into how your data is optimized across the different threads in your app so you can dive into this code and make sure there are no inefficiencies.

With a ton of insights into how the elements of our application interact, New Relic helps tune the experience for users and identify potentially disastrous issues before they become problems.

Though New Relic is not as strong on the error and crash diagnosis as other dedicated crash reporting and error tracking solutions, you can pair it up with a tool like [Raygun][1] to give you the best of both worlds. You'll get APM data plus dedicated error diagnosis and user session tracking.

Key features

App availability monitoring, alerting, and notification
Production Thread Profiler feature
Automatic application topology mapping
Platform plugins
Histograms and percentiles
Performance data API access
Real-User response time, throughput, and breakdown by layer
JVM performance analyzer
Error detection, alerting, and analysis
Database call response time and throughput
Code-level diagnostics, transaction tracing, and stack trace
App response time, throughput, and breakdown by component
Slow SQL and SQL performance details
Apdex score
Real-User breakdown by web page, browser, geography
Custom dashboards
Track individual business transactions
X-Ray sessions for business transactions
Cross-application tracing for distributed apps
Availability, scalability, deployment reports

2. Pingdom

Usually, a particular part of the world is vital to your business. So, if your app is painfully slow for customers located there, you'll need to know about it. For this reason, Real User Monitoring (RUM) tools are essential for identifying performance problems and their environments.

With Pingdom, you can filter data from specific users for greater insight on the regional performance of your website. By setting your performance levels for what is considered satisfactory, you can ensure your most valuable customers get the site experience they deserve.

Pingdom also monitors your website for downtime, alerting you right away when a disaster happens, and it goes offline. You can't deliver exceptional software experiences for your users if your website isn't loading at all and Pingdom's uptime monitoring has you covered here.

Key features

Monitor websites and servers
Detailed reports
Important interactions
Analyze and come up with a solution to fix it
Real data from real visitors equals real performance data
Share your data with others
Available for free on your mobile devices

3. Datadog

Datadog wants you to view all your application performance metrics in one place — ; and does so well. There are a large number of integrations that allow you to connect various other tools as well as what Datadog provides out of the box.

Though other performance monitoring tools have collaborative functionality, Datadog puts this at the heart of the product and makes sure teams consider monitoring various parts of their applications together.

DataDog also charts a vast variety of data on simple-to-use dashboards and is not limited to monitoring data only. Where New Relic builds different agents to monitor various aspects of your application code, DataDog serves this information in one dashboard.

Key features

Free trial for an unlimited number of hosts
120+ turn-key integrations for data aggregation
Clean graphs of StatsD and other integrations
Slice and dice graphs and alerts by tags, roles, and more
Outlier detection for hosts behaving abnormally
Alert notifications via e-mail and PagerDuty
Alert on any metric, for a single host or an entire cluster
Full API access in more than 15 languages
Overlay metrics and events across disparate sources
Out-of-the-box and customizable monitoring dashboards
Easy way to compute rates, ratios, averages, or integrals
Sampling intervals of 10 seconds
Mute all alerts with one click during upgrades and maintenance
Tools for team collaboration
Easy-to-use search for hosts, metrics, and tags

4. Raygun

The Raygun Platform gives your development team a unique view of how users are experiencing your software applications. Mainly, Raygun focuses on error monitoring alongside identifying performance issues that users encounter with real user monitoring and application performance monitoring.

One of the features that saves the most time is smart error grouping, which collects errors underneath a single root cause so you don't get flooded with notifications.

To help isolate and diagnose performance issues, page performance breakdowns are viewable for every user session to see which specific assets were the cause of a slow loading page or mobile app view.

Another time-saving feature of Raygun is the ability to identify authenticated users, so you get a view of which specific users encountered problems, and to what extent. This is great for surfacing VIPs or Enterprise customers who have bad user experiences. It works by enabling the developer to search a specific email address to produce a user profile. The unique profile will have a list of every error or crash that user has encountered, which browser they use, and all the devices they use to access your app.

The deployment tracking feature is also useful for surfacing problems with releases. When you release a new update or version, Raygun can identify issues that were introduced, resolved or fixed in each version.

Pretty neat for immediately spotting a poor release before they get into the hand of your customers!

Though functionality is different to what tools like Google Analytics provide, Raygun does also incorporate some high-level analytics including which locations are experiencing the fastest and slowest loading times.

Key features

Full-text search and filtering
Track multiple applications using any language and framework
Error, crash and performance tracking in one platform
Unlimited end users
Discover critical bugs
Add comments and mention team members on issues
Version/deployment tracking
Attach tags, custom data objects, and user data
JavaScript source maps
Automatic error reporting
Seamless integration that takes minutes
Intelligent error grouping
Error notifications via email, Slack, HipChat etc
Full stack traces

5. AppDynamics

Firmly placing it's offering towards the Enterprise end of the market, AppDynamics is one of the performance monitoring tools providing APM and End User Monitoring in one solution.

AppDynamics has instrumentation to handle application data collection if the application is in a supported language. (Java, .NET, PHP, Python, C++ or Node.js.) They also collect data from mobile devices, mobile apps, and browsers.

Tracing transactions from the user through all the components within the application means AppDynamics can relate all of the infrastructure and log messages back to that single transaction. AppDynamics automatically discovers and names these exchanges enabling you to correlate data and visualize topologies of applications.

Key features:

End User Monitoring
Application health dashboard
Real-time business transaction monitoring
Reports and visibility into your application
Visualize and manage your entire app
Operational dashboards
Detect business impact and performance spikes
Analyze impact of agile releases
Isolate bottlenecks in your application
Deep code-level diagnostics in production
Identify root cause with complete code diagnostics
Dynamic scaling in the cloud
Automatic business transaction discovery
Diagnose the root cause of the problems 90% faster
Discover and visualize your application topology and business
Troubleshoot performance and availability issues
Set up proactive alerting to find problems
Monitor 24/7 what matters most-your key business transaction
Troubleshoot bottlenecks 90% faster
Monitor hybrid environments with Java, .NET, PHP, and Node.js

Overview

In conclusion, each of the performance monitoring tools above offers something slightly different, but are all built around the same goal — seeing what your users are doing when they encounter problems. Usually, developers have to hunt around for clues as to why their software is not performing as expected. Application Performance Monitoring tools present the symptoms of the problem in a clear and visual way to aid in diagnosis and ultimate resolution.

Ultimately, performance problems are a huge contributor to dissatisfactory software experiences. Therefore, it's down to the software engineering team to be proactive in making sure their applications are performing for users in the way they are intended.

Teams slow to adopt such visibility stand to lose out to their more innovative competitors that took more care of how users experience their application.