DEV Community

Cover image for The Cost of an Error. What are the consequences of the most notorious software errors?
Serverspace.us
Serverspace.us

Posted on

The Cost of an Error. What are the consequences of the most notorious software errors?

September 6, 1989 the Parisians remembered for a long time — on that day more than 41 thousand residents of the French capital received official letters from the police that they had committed cruel murders and robberies, although in fact the recipients had only violated traffic rules. The cause of the mishap turned out to be a failure in the computer system of the Parisian gendarmerie, which added gray hair to ordinary citizens. This incident was not the first in the history of high technology, and certainly not the last. Let's remember the most loud and widely known computer errors that led to amusing and sometimes rather sad consequences.

There is no program in the world, except perhaps "Hello World!", that is completely error-proof. What about large applications consisting of millions of lines of code and developed by teams of programmers. Of course, such software products are thoroughly documented and pass through the careful hands of testers before they go into production, but the harsh reality brings its corrections to the logic of their operation anyway.

Sometimes the software stability is affected by external circumstances and operating conditions, much more often the cause of the problems lies somewhere between the office chair and the keyboard. And it is very good if the failure will only lead to the loss of a couple of important reports. Sometimes the consequences can be much more serious.

The Biggest Bank Mistake in American History

On a warm May day in 1996, Sylvester Dorsey, a collector for Peoples Gas Light and Coke, a private Chicago gas service company, went out to lunch. On the way, he decided to stop at a nearby ATM to check his bank card balance. When Dorsey received a receipt, he couldn't believe his own eyes. He turned out to be the owner of a modest $924.8 million fortune. "I showed the check to a friend who was nearby, and we just screamed with excitement," Dorsey recalled later.

Another lucky man was Jeff Ferrera, a computer engineer at Zenith Electronics, who called the bank Friday morning in an attempt to check his balance. After listening to the automated voice mail message, Jeff called back again, recorded the robot's voice on a tape recorder and set the recording as a greeting on his phone. Now every caller to Ferrera heard the following message on the receiver: "the available balance in your primary account is currently $924,844,208 and 32 cents..." One can only imagine how Jeff himself felt when the answering machine first said these words.

Image description
The same fate befell 825 other clients of First National Bank of Chicago — all of them suddenly became multimillionaires. However, the happiness did not last long: only one day. By the evening of the day, bank employees discovered that an unfortunate computer glitch had been the cause of the account holders' enormous wealth. The debit card software had miscalculated the parameters of recent transactions and transferred a staggering $763.8 billion to clients, which was more than six times the total value of all First National Bank of Chicago's assets.

By the evening the accounts of the new millionaires had already been frozen, and the sums mistakenly credited had been safely withdrawn. According to the bank representatives, none of the clients managed to escape with an unexpected windfall to the islands of the Caribbean archipelago, so the real financial losses of the company were minimal. But to this day this incident is considered to be the largest banking mistake in U.S. history, caused by a glitch in the computer program.

By the way, two years before this story, something similar happened at Chemical Bank in New York, but with the opposite mathematical sign. A glitchy computer halved all the deposits stored in the head office database, and instead of a few hundred happy millionaires the bank received an army of angry customers. The consequences of this failure were dealt with by technicians for several days.

But this is PayPal!

However, Sylvester Dorsey, Jeff Ferrera, and the 825 other Chicago bank customers all look like paupers compared to a man named Chris Reynolds from Pennsylvania. The evolution of electronic payment systems and digital currencies has made monetary transactions easier, more convenient, and faster, but it has also increased the risk of problems caused by errors in the software used to process those payments.

Like many Americans, Chris Reynolds used PayPal, and naively thought it was the best financial service in the world. His high opinion of the benefits and features of PayPal was reinforced when he found $92,233,720,368,547,800 in his account one morning on June 30, 2013.

Once again: $92 quadrillion. For a used car-parts dealer on eBay, that was a pretty handsome sum: the fortune of the richest man on the planet that year, telecommunications magnate Carlos Slim, was a little short of Reynolds' wealth, with a mere measly $67 billion.

Image description
Chris even printed out his account statement as a memento. However, after he took the sage advice from the TV show "Computerman" namely, "try to get out and get back in," the miraculous obsession dissipated. He had 145 bucks and 25 cents on his balance sheet again, and Carlos Slim was back in his place of honor as the richest man on Earth. The fairy tale was over, and the untold treasure had turned into a pumpkin.

PayPal admitted the failure of its server software, and as compensation offered the upset Reynolds to transfer any reasonable amount of money to any charitable purpose. When the BBC reporters asked the hapless quadri-billionaire what he would spend the money on if he really got it, he said "I would pay off the U.S. foreign debt”.

Your bill, sir!

However, not only bank computers make financial mistakes, as the Brotherton family from Lancashire county, located in England on the shores of the Irish Sea, was able to see for themselves. Linda and Nigel Brotherton decided to change the electricity supplier: previously they used the services of Scottish Power, but one day they thought it would be more profitable to buy electricity from the company Npower. But they were wrong.

An Npower employee inspected the electric meter at the Brazertons' house and reset the meter to zero. But the company's computer thought that a "0" on the indicator meant that the meter had spun a full cycle since the last time it reported electricity use, and the available numbers simply ran out. The next month, the venerable family received a receipt that said their electric bill had increased slightly, from 87 pounds to 53,480,062 pounds, or about $90 million.

Image description
The investigation revealed that the software used by Npower simply did not include such an operation as resetting the meter manually, and the company representative did not know this. The programmers' error cost Nigel Brotherton and his wife a fair amount of nerve cells, but they still did not have to pay £53 million.

Minus 460 million in 45 minutes

On Wednesday, August 1, 2012, the office of the investment company Knight Capital began work at 8am, as usual. When they turned on their computers, the first thing they did was to check their emails, and among the spam they found automatic messages saying that the Power Peg program running on the server was not properly configured. Nobody paid attention to these warnings, because Power Peg had not been used for almost 10 years, since 2003. It was completely in vain.

At 9am, the New York Stock Exchange opened, and Knight Capital's automated trading systems began creating orders to buy and sell assets. Within 45 minutes, the company had already lost more than $4.5 million, and soon the firm's total loss from the soulless trades reached $460 million, pushing Knight Capital to the edge of bankruptcy. Automated algorithms of other players took advantage of this situation, which caused the shares of some companies on the New York Stock Exchange to jump by 300 percent.

Image description
An investigation later showed that on the eve of that unfortunate day, a software update was installed on Knight Capital's servers, which negligently turned on the outdated Power Peg application, which had been disabled long ago due to its uselessness. In the test mode, this application sells shares at the current price and immediately buys them back at the market rate (which is usually higher than the selling price), paying absolutely no attention to the value of securities - its task is to make as many deals as possible in a unit of time. After decommissioning this program, the developers removed the check from its code as to whether the application is running on a test server in a local network, or whether it is operating in a real working environment.

As it turned out, the installed update launched Power Peg on a server connected to the New York Stock Exchange, after which the program started working in test mode and began to register a huge number of crazy deals, rapidly draining the company's capital. Knight Capital was later fined $12 million by the Securities and Exchange Commission for violating financial risk management rules.

Apple Maps

Some software errors do not seem to cause direct financial losses, but sometimes they cause indirect losses. The first iPhone models used maps and navigation from Google, but in the struggle with its main competitor, Apple decided to get rid of the Google Maps application. In 2012, Cupertino developed its own version of maps for iOS, but unlike Google, which spent many years and millions of dollars to create its service, Apple decided that the task can be solved faster and much more economically. Information about roads, bridges, architectural objects and places of interest is gathered in Google from thousands of different sources, stored in several distributed databases, and a powerful software system collects all this information together. Apple didn't have all these resources at the beginning.

As a result, on iPhones and iPads many lakes, bridges and train stations were absent from their usual places, the Washington Monument moved to the next street, the Publix supermarket in Jacksonville, Florida, became a hospital, and the main train station of the capital of New Zealand, the city of Auckland, was in the middle of the ocean. In 3D, some parts of the map looked even more fantastic: highways were folded in a comb, rushed vertically into the sky and twisted into a Mobius strip, bridges went under water, and buildings were piled up in the middle of the water surface.

Image description
Of course, none of the Apple card users dived their car off the Oakland waterfront in an attempt to catch the departing train, but the credibility of the company's software was still slightly undermined. And reputations are very expensive these days.

Conclusion

In most cases, it is not only companies that suffer from errors in programs, but also ordinary users. Sometimes developers try to make up for this and compensate people for the inconvenience they have caused. But sometimes they do it in a very strange way. For example, a customer service call to First National Bank of Chicago cost as much as three dollars. After the incident with the erroneous crediting of 924.8 million to debit cards, the bank's management made heart-to-heart conversations with its employees a free service. But only for those customers who themselves informed the institution about the unexpected wealth that had fallen on them and voluntarily returned the money. Incredible kindness, isn't it?

We think each of us has faced various bugs in software. We will be glad if you share the most interesting bugs from your practice in the comments!

Top comments (0)