Originally published at trustboundarystudio.com. The video version, with diagrams, is on YouTube.
Key facts
- When: 22 to 23 March 2019. Discovered 17 July 2019, after an outside party emailed Capital One's responsible disclosure address.
- Scale: Personal data of 106 million people.
- Entry: Server-side request forgery through a misconfigured web application firewall, returning IMDSv1 credentials.
- What made it catastrophic: The IAM role attached to the firewall instance could read S3 buckets across the account.
- Penalty: $80 million OCC civil money penalty, August 2020. $190 million class action settlement.
- Primary sources: OCC Consent Order 2020-036. FBI criminal complaint. MIT Sloan case study.
In March 2019 someone obtained the personal data of 106 million people from
Capital One. No malware. No zero-day. No stolen employee password. They asked a
web application firewall to make a request on their behalf, and it did.
That much is well known. The part that gets lost is what happened next, and
what the regulator actually penalised eighteen months later. Because the
Office of the Comptroller of the Currency's consent order does not mention
server-side request forgery at all.
The system
Capital One had moved a large part of its IT operations into AWS starting
around 2015, further and faster than most banks its size. That is not the
failure. That is ordinary modernisation, and it mostly went well.
Credit card applications going back to 2005 sat in S3. In front of the
application layer sat a web application firewall running on an EC2 instance.
Its job was to inspect incoming requests and block malicious ones. It was a
security control, and it is the thing that got used to break in.
To do that job, the firewall instance had an IAM role attached.
An IAM role is a set of permissions. Attach one to an EC2 instance and anything
running on that instance can use those permissions. No password, no key on
disk. It is how almost every workload in AWS talks to other AWS services, and
it is a genuinely good design. It is also one of the most consequential
configuration decisions most teams make once and never look at again.
Why the metadata service answered
The firewall was misconfigured in a way that allowed server-side request
forgery. Normally you send a request to a server and it answers. In an SSRF you
send a request that convinces the server to fetch something for you and hand
you the response. That matters because of where the server is standing. You are
outside. It is inside. Anything it can reach, you can now reach through it.
On an EC2 instance there is one address that is always reachable and always
interesting: 169.254.169.254. That is the instance metadata service. It is
link-local, so it exists only from the perspective of the instance itself, and
you cannot route to it from the internet. What it returns includes temporary
credentials for whatever role is attached.
This is not a flaw. It is the mechanism that means you do not hardcode access
keys into your application, which is one of the better patterns AWS ever
shipped. But in 2019 that service, now called IMDSv1, answered any plain HTTP
GET originating from the instance. No token. No authentication. If you could
make the instance issue a request, you got credentials back.
Three commands
The FBI complaint describes three.
The first obtained security credentials. The role appears in the indictment
only as *****-WAF-Role; the rest is redacted, and anyone quoting a full role
name is guessing. The firewall handed over its own credentials, working as
designed at every individual step.
The second listed the names of folders and buckets in Capital One's storage.
The third copied data out of them.
This is where an interesting incident becomes a catastrophic one. Ask what a
web application firewall actually needs. It inspects traffic, matches patterns,
blocks or forwards requests. There is a plausible reason for it to reach S3:
rule sets, configuration, logging. There is no reason for it to enumerate
storage across the account and read the contents.
So the third command was not an exploit. It was a copy. Ordinary S3 operations,
correctly authenticated, fully permitted, and at the API level
indistinguishable from legitimate traffic.
The encryption did not help
Capital One's own statement says they encrypt as standard, and then says this:
Due to the particular circumstances of this incident, the unauthorized access
also enabled the decrypting of data.
The credentials that could read the data could also decrypt it. That is what
encryption at rest is for: a stolen disk, not a valid caller. It is worth being
precise about this, because "the data was encrypted" is repeated constantly as
though it were mitigation, and here it was not.
The intrusion was not a chain of escalating exploits. It was one boundary
crossing followed by entirely legitimate use of over-granted permissions.
117 days, and it was an email
The intrusion happened on 22 and 23 March 2019. Capital One found out on 17
July.
It was not detection tooling that ended it. Someone noticed the data described
on a public GitHub page and wrote to Capital One's responsible disclosure
address. An outside party, reading a public post, told the bank it had been
breached.
What the regulator actually found
The financial consequences were an $80 million civil money penalty from the OCC
and a $190 million class action settlement. The consent order is the part worth
reading, and it is not about the SSRF.
The OCC found that Capital One failed to establish effective risk assessment
processes before migrating its IT operations to the cloud. That internal audit
failed to identify the control gaps. And that the board failed to hold
management accountable.
Not "you got hacked". You did not know what your own environment allowed.
That distinction is the reason this incident is still worth studying. A
vulnerability is a thing you fix. Not knowing what your permissions grant is a
condition you live in, and it is invisible right up until the moment it is not.
Which control would have held
Three candidates, and the order matters.
Fix the firewall misconfiguration. True, and the weakest of the three,
because it assumes you never ship a vulnerability. You will.
The metadata service. On 19 November 2019, four months after this became
public, AWS shipped IMDSv2. It requires a session token obtained by an HTTP PUT
before it will answer, and the choice of PUT is deliberate: most misconfigured
firewalls and reverse proxies do not forward PUT at all. AWS was explicit that
this is defence in depth against exactly this class of problem. If you are
running EC2 today with v1 still enabled, that is the actionable item here.
The IAM role. This is the real one. The first two stop this attack.
Scoping the role limits every attack.
Had that role carried read access to the buckets a firewall actually needs, the
same SSRF, the same stolen credentials and the same three commands would have
returned firewall configuration. Still an incident. Still an investigation. Not
106 million people.
That is the difference between a vulnerability and a catastrophe: not whether
someone gets in, but how far the credentials they find will carry them. Least
privilege is not a compliance checkbox. It decides the size of your worst day.
The pattern underneath
It is not the SSRF that recurs. It is the assumption beneath it, that a service
inside your perimeter is trustworthy because it is inside your perimeter.
The firewall was trusted because it was internal. The metadata service answered
because the request came from the instance. The role was broad because scoping
it properly is tedious and nothing had gone wrong yet.
Every one of those decisions was locally reasonable. That is what a trust
boundary failure looks like in practice. Not a dramatic break-in, but a series
of sensible choices, one of which granted far more than anybody checked.
Go and look at your instance roles this week. Not the ones you wrote recently.
The ones attached to something that has been running since before you arrived.
Sources
Every claim above comes from primary documents rather than coverage of them.
- OCC Consent Order #2020-036 (AA-EC-20-51), 6 August 2020. https://www.occ.gov/static/enforcement-actions/ea2020-036.pdf
- FBI criminal complaint, US District Court at Seattle, 2019.
- Neto & Madnick, "A Case Study of the Capital One Data Breach", MIT Sloan,
- Capital One incident disclosure and FAQ, July 2019.
- AWS, "Defense in depth: open firewalls, reverse proxies and SSRF vulnerabilities with EC2 IMDS", November 2019.
Corrections are welcome, and any made are listed, dated, at the end of this article.
Questions this answers
Was the Capital One breach caused by SSRF?
The entry point was server-side request forgery through a misconfigured web application firewall. What made it a 106-million-record breach was the IAM role attached to that instance, which could read S3 buckets across the account. The regulator's consent order does not mention SSRF at all.
Why did encryption not protect the data?
The data was encrypted at rest, but the stolen credentials were valid, and credentials that can read the data can also decrypt it. Encryption at rest protects against a stolen disk, not a legitimate caller. Capital One's own statement says the unauthorised access also enabled the decrypting of data.
What did the OCC actually penalise Capital One for?
Failing to establish effective risk assessment processes before migrating IT operations to the cloud, an internal audit that did not identify the control gaps, and a board that did not hold management accountable. The penalty was $80 million, in August 2020.
What is IMDSv2 and would it have stopped this?
IMDSv2 is the second version of the EC2 instance metadata service, released 19 November 2019. It requires a session token obtained by an HTTP PUT before answering, and most misconfigured firewalls do not forward PUT. It would have blocked this particular path. Scoping the IAM role would have limited every path.
The video version
Top comments (0)