DEV Community

Cover image for When S3 Went Down, AWS Couldn't Change Its Own Status Icons
CatNight A
CatNight A

Posted on Edited on

When S3 Went Down, AWS Couldn't Change Its Own Status Icons

On February 28, 2017, you could open the AWS status page and get two different answers depending on where you looked. At the top, a banner carried updates about an outage in S3, the service many websites use to store photos and other files. The trouble was in Northern Virginia. A little farther down, the icons for individual services were stuck. For two hours, even the people running the page couldn't change them.

The AWS status page on the day of the outage, with a notice above the list of services

This screenshot was taken after service was restored. You can see “RESOLVED” in the banner. Publickey's coverage from the time

The trouble started with a slow billing system. An engineer on Amazon's S3 team followed a procedure to take a small set of servers offline. One value in the command was wrong. It took down more servers than intended, including some that helped S3 keep track of where it had put everyone's files.

Say your photo app stores its pictures in S3. You tap yesterday's photo of your cat to send it to a friend. The app asks S3 for the picture by name. That name tells S3 which picture you want, but it still has to look up where the picture lives. The part that keeps track of those locations is called an index.

Enough servers had gone offline that the index had to restart and check its records. You can see the order in which things came back if you follow that same cat for one more photo.

AWS S3 outage status page screenshot

At 1:18 p.m. Pacific, S3 could retrieve files normally again. You could open yesterday's picture. Now suppose the cat finally looks straight at the camera. You take a new photo and tap Upload. That takes one more step. S3 has to find somewhere to put it.

A separate part of S3 did that job. It had to wait for the index before it could recover. Uploads returned to normal at 1:54 p.m. That's 36 minutes when the same photo app could show you an old picture and still have trouble saving a new one.

That leaves those stubborn little icons. To change one, the person on duty had to use an internal tool. And that tool relied on S3 in Northern Virginia. The team could still post a banner and send updates on Twitter. It was the ordinary job of changing a status icon that the outage had taken away from them.

Afterward, AWS changed the tool to run in more than one region. If Northern Virginia went down again, the team could still change the icons from somewhere else.

Read the AWS team's account of the outage.

Top comments (6)

Collapse
 
acatnigh profile image
CatNight A •

@xulingfeng Bro, show some support!

Collapse
 
xulingfeng profile image
xulingfeng •

🙌great

Collapse
 
acatnigh profile image
CatNight A •

😁😁

Collapse
 
acatnigh profile image
CatNight A •

给一点提议=w=😁

Thread Thread
 
xulingfeng profile image
xulingfeng •

🤣太晚了,该睡觉了,明天还要上班

Thread Thread
 
acatnigh profile image
CatNight A •

🤪