DEV Community

Cover image for The 2017 S3 Outage Was One Wrong Input. AWS Fixed the Tool, Not the Engineer
Gabriel Anhaia
Gabriel Anhaia

Posted on Edited on AI-assisted

The 2017 S3 Outage Was One Wrong Input. AWS Fixed the Tool, Not the Engineer

#aws #devops #ai #security

At 9:37 AM Pacific on February 28, 2017, an authorized member of the Amazon S3 team ran a command from an established playbook. It was meant to take a small number of servers out of one subsystem in us-east-1. AWS's own summary of the event says what went wrong next: "one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended."

S3 in that region was not back to normal until 1:54 PM, per the same summary. That is more than four hours, from one input.

AWS's fix went into the tool. If you let a coding agent run shell commands, that fix applies to you, because an agent enters inputs faster than any engineer does.

What AWS says happened

The S3 team was debugging the S3 billing system, which was running slower than expected. The command was supposed to remove a few servers from a subsystem the billing process used. The bad input removed far more, and some of those servers supported two other subsystems, AWS wrote:

  • The index subsystem, which manages the metadata and location of every S3 object in the region. GET, LIST, PUT and DELETE requests all need it.
  • The placement subsystem, which allocates storage for new objects and needs the index to work.

Both needed a full restart. AWS was candid that it had "not completely restarted the index subsystem or the placement subsystem in our larger regions for many years," and after years of growth, the restart and the safety checks on the metadata took longer than expected. The index could serve GET, LIST and DELETE again by 12:26 PM and was fully recovered at 1:18 PM. Placement finished at 1:54 PM, which brought PUT back (AWS).

The damage reached past S3. The summary lists the S3 console, new EC2 instance launches, EBS volumes that needed data from an S3 snapshot, and AWS Lambda among the affected services. Even the AWS Service Health Dashboard could not show individual service status until 11:37 AM, because its admin console depended on S3.

A black silhouette of an engineer at a large control dial, the dial's needle stopped by a burnt-orange block at the minimum mark, with a row of server racks behind
The 2017 fix in one picture: the tool itself refuses to go below the floor.

The fix went into the tool

"While removal of capacity is a key operational practice, in this instance, the tool used allowed too much capacity to be removed too quickly," AWS wrote. Then the change: the tool now removes capacity more slowly, and it refuses when a removal would take any subsystem below its minimum required capacity. AWS's reason, in its own words: "This will prevent an incorrect input from triggering a similar event in the future." It also said it was "auditing our other operational tools to ensure we have similar safety checks."

None of the changes AWS listed is about the person who entered the input. The tool was changed so the same mistake fails safely the next time anyone makes it.

The rest of the list was about recovering faster. AWS moved up its work to split the index subsystem into smaller cells, and it moved the Health Dashboard's admin console to run across multiple regions.

Read as a design rule, it fits in one line: a destructive tool should cap how much it does per run and refuse to cross a floor, whatever input it gets.

Your agent types faster than you do

A coding agent runs commands you didn't type, built from inputs it worked out on its own. Usually that is fine. But the S3 failure mode is one an agent can reproduce well. An agent that passes 20 where the runbook meant 2, or acts on whatever cluster your kubeconfig currently points at, is the S3 scenario again: an authorized operator, a known command, one bad argument, and no pause before it runs.

The approval prompt helps only while you read it. By the fortieth prompt of an afternoon, approving on reflex is easy. So the first guard should not depend on your attention. Put it in the script.

Put the floor inside the script

Here is the AWS fix as a small wrapper around kubectl scale. It refuses to remove more than MAX_STEP replicas per run, refuses to go below MIN_REPLICAS, won't start without an explicit cluster context, and only prints the command unless you pass --apply:

#!/usr/bin/env bash
# scripts/scale-down.sh DEPLOYMENT COUNT [--apply]
set -euo pipefail

deploy="${1:?deployment name}"
remove="${2:?replicas to remove}"
mode="${3:-}"

ctx="${KUBE_CONTEXT:?set KUBE_CONTEXT}"
MIN_REPLICAS="${MIN_REPLICAS:-3}"
MAX_STEP="${MAX_STEP:-2}"

case "$remove" in
  ''|*[!0-9]*) echo "COUNT must be a number" >&2; exit 2 ;;
esac

current=$(kubectl --context "$ctx" get deployment "$deploy" \
  -o jsonpath='{.spec.replicas}')
target=$(( current - remove ))

if (( remove > MAX_STEP )); then
  echo "refusing: $remove is more than MAX_STEP=$MAX_STEP" >&2
  exit 1
fi
if (( target < MIN_REPLICAS )); then
  echo "refusing: $current -> $target is below" \
       "MIN_REPLICAS=$MIN_REPLICAS" >&2
  exit 1
fi

cmd=(kubectl --context "$ctx" scale deployment "$deploy"
  --current-replicas="$current" --replicas="$target")

if [[ "$mode" != "--apply" ]]; then
  echo "dry run: ${cmd[*]}"
  exit 0
fi
"${cmd[@]}"
Enter fullscreen mode Exit fullscreen mode

Both AWS changes are here. MAX_STEP is "remove capacity more slowly": a 20 typed where 2 was meant gets refused instead of applied. MIN_REPLICAS is the floor. KUBE_CONTEXT has no default, and both kubectl calls pass it as --context, so the script never acts on whatever cluster your kubeconfig happens to point at. Export it in the shell you start the agent from, so the agent's command stays a plain ./scripts/scale-down.sh api 2. --current-replicas is a precondition from kubectl itself. The scale only goes through if the deployment still has the replica count the script just read, so a change made in between makes it fail rather than land on a stale number. The dry run is the default, so an agent's first attempt shows you the exact kubectl line it would run. I haven't run this script against a cluster myself, so set the floor for your own deployment and try it on a staging one first.

The same shape fits most destructive commands an agent might reach for: a database reset that refuses when the connection string points anywhere but localhost, a cleanup job that caps deletions per run, a migration runner that won't drop a table without --apply. The input can be wrong, so the tool checks it.

Deny rules are the second layer

Next, steer the agent to the wrapper and away from the raw command. In Claude Code, permission rules go in .claude/settings.json, and the docs say "Rules are evaluated in order: deny, then ask, then allow":

{
  "permissions": {
    "allow": ["Bash(./scripts/scale-down.sh *)"],
    "ask": ["Bash(./scripts/scale-down.sh * --apply)"],
    "deny": [
      "Bash(kubectl scale *)",
      "Bash(kubectl delete *)"
    ]
  }
}
Enter fullscreen mode Exit fullscreen mode

Dry runs go through without a prompt. Anything ending in --apply matches the ask rule and prompts, even though it also matches the allow rule, because ask is checked first. Raw kubectl scale and kubectl delete are refused.

Codex has the same idea in rules files, for example ~/.codex/rules/default.rules. When several rules match, the most restrictive decision wins: forbidden, then prompt, then allow. OpenAI marks rules as experimental.

prefix_rule(
    pattern = ["kubectl", "scale"],
    decision = "forbidden",
    justification = "Use scripts/scale-down.sh",
)
Enter fullscreen mode Exit fullscreen mode

To see how Codex classifies a command before you trust the rule, the docs give a checker. Here it is pointed at the rule above:

codex execpolicy check --pretty \
  --rules ~/.codex/rules/default.rules \
  -- kubectl scale deployment api --replicas=0
Enter fullscreen mode Exit fullscreen mode

The caveat comes from Anthropic's own page: a deny rule "covers the invocation Claude usually produces and isn't a security boundary around the program." Its example is that Bash(rm *) stops rm -rf build/ but not /bin/rm -rf build/ or bash -c 'rm -rf build/'. For enforcement that doesn't depend on the command text, the same docs point to sandboxing.

So neither layer is complete alone. The rule steers the agent to the script. The script refuses a bad number when it is used. If an agent should never be able to scale production at all, give it credentials that can't. That puts the floor outside the agent entirely.

Read what the agent ran

I build Hermes IDE, so weigh this section with that in mind. Hermes runs each agent (Claude Code, Codex and others) in its own real terminal, so the agent's own permission prompts and rules still apply; Hermes doesn't replace them. It never answers a permission prompt for you and never types into a terminal on its own.

In the ⌘N sheet (Ctrl+Shift+N on Windows and Linux) you pick the agent, model, effort and account per task, and the task starts on its own hermes/ branch and worktree, with the task as the agent's first prompt.

The Hermes New task sheet with a refund-endpoint task typed, chips for Claude Code, the orders-api project, a new hermes/ worktree, Accept edits, model and effort, and the preview of the claude command Hermes will run
The ⌘N New task sheet in Hermes 2.3.4: agent, project, a new hermes/ worktree, the Accept edits approval mode, model and effort, and a preview of the claude command Hermes will run.

When an agent stops on an approval prompt, it counts in the attention inbox, and ⌘I (Ctrl+Shift+I) jumps to the agent that has waited longest. You answer in that agent's terminal, where its prompt shows the command it wants to run. For a prompted command, that is your last look at the input before it runs.

After the fact, the Review Desk (⌘G, or Ctrl+Shift+G) shows what changed by turn or by file, with risk flags. Every agent turn is saved, so you can diff or restore a single turn, with undo.

The Hermes Review Desk in By turn view for a Claude Code task in orders-api, listing two turns, turn 1's diff of src/db.ts adding a refunded status, the changed files and a drafted commit message
The Review Desk (⌘G) by turn in Hermes 2.3.4: two turns of a Claude Code task, turn 1's diff of src/db.ts, the changed files and a commit message drafted from the turns.

Be clear about the limits. Hermes observes what agents report and what git shows. It doesn't block commands, so the deny rule and the floor in your script are still the guards. Restoring a turn puts files back in your worktree; it won't bring back replicas an agent scaled away. Hermes is free, source-available under BSL 1.1, and runs on macOS, Windows and Linux. It is not affiliated with Nous Research, Anthropic, OpenAI or any other agent vendor. Download it or read the code on GitHub.

Which destructive script in your repo would an agent find first, and does it have a floor? Tell me in the comments.

If this was useful

The S3 summary is really a capacity story: two subsystems had a minimum below which a whole region stopped serving, and the tool that removed servers didn't know where that line was. My System Design Pocket Guide: Fundamentals covers those core building blocks for scalable systems, so you can see where the floors in your own system sit before a script, or an agent, finds them. If you run several agents at once, Hermes IDE is where I keep an eye on which of them is waiting on me.

System Design Pocket Guide: Fundamentals — Core Building Blocks for Scalable Systems

Top comments (0)