Every discussion of hiring bias reaches for Amazon’s scrapped resume screener within two minutes. It rests on one news story, from one publication, sourced to people who were not named, and Amazon has never published a technical account. That does not make it false. It changes how you are entitled to cite it.
What the report said
On 10 October 2018 Reuters published a piece by Jeff Dastin reporting that Amazon had built and then abandoned an experimental machine learning tool for reviewing job applicants’ resumes. The reporting attributes its account to five people familiar with the effort, who spoke on condition of anonymity. The core claims are that a team in Edinburgh began the work in 2014, that the goal was a system which could rate candidates on a one-to-five scale, and that by 2015 the team realised the model was not rating candidates for software developer and other technical roles in a gender-neutral way.
Reuters reports that the group was disbanded by the beginning of 2017, and that Amazon said the tool was never used by recruiters to evaluate candidates. The original is at Reuters, 10 October 2018. Every specific number and date in this section comes from that piece and should be attributed to it rather than to Amazon.
Nothing here is legal advice, and nothing here is a finding of fact about Amazon. No regulator adjudicated this tool and no court examined it. If you are assessing your own screening model, get the assessment done on your own data under privilege where that is available to you.
The described mechanism
The mechanism Reuters describes is the reason the story survived. The models were reportedly trained on resumes submitted to the company over a ten-year period, a set which skewed heavily male, reflecting the industry’s composition rather than any judgement about the candidates. The system, in effect, learned what a successful applicant historically looked like and reproduced it.
The two concrete symptoms reported are the useful part. The system was said to penalise resumes containing the word “women’s” — as in “women’s chess club captain” — and to downgrade graduates of two all-women’s colleges. Reuters reports that Amazon edited the models to make them neutral to those particular terms, but that there was no guarantee the system would not find other ways to sort candidates that proved discriminatory. That last sentence is the whole problem of proxy features stated in one line by the engineers who hit it.
What is not established
Being disciplined here is what makes the page worth citing.
- There is no published model, dataset or evaluation. No architecture, no training corpus description beyond “resumes over ten years”, no measured disparity, no adverse-impact ratio. Anyone quoting a percentage for how much the model disadvantaged women is inventing it.
- Amazon’s position is that it was never used to evaluate candidates. The reporting includes that statement. It has not been tested by any adjudicator, and it is neither confirmed nor contradicted by anything public.
- No legal claim was ever brought over it. There is no EEOC charge, no consent decree, no settlement. Presenting it as an enforcement case is simply wrong.
- The story is second-hand within the industry. Much of what circulates is a paraphrase of a paraphrase, and the drift is visible: the tool is variously described as having been deployed, as having rejected real applicants, and as having been discovered by an external audit. None of that is in the original.
Why the failure mode is the durable part
Strip the attribution problems away and what remains is a mechanism that does not depend on the story being true in every detail, because it follows from how supervised learning works. If the label is “this person was hired” or “this resume advanced”, the model is learning the historical decision function, including whatever bias that function carried. It is not learning who is good at the job, because nobody labelled that.
The proxy problem follows immediately. Removing gender as a feature does nothing, because the text of a resume is saturated with correlates: institution names, sport and society memberships, career gaps, phrasing conventions, even the distribution of verbs. A high-capacity model over free text will find them. This is the same structural point that drives the SafeRent tenant-screening claim in housing: the protected characteristic does not need to be an input for the output to track it.
The generative era did not retire this. A large model asked to rank candidates from resume text has the same exposure through a different route — not a learned coefficient on an all-women’s college, but the pretraining distribution’s associations, invoked implicitly and with no coefficient anyone can inspect or edit. Amazon could at least find the offending term and neutralise it.
What would happen to the same tool now
In 2015 there was almost no instrument aimed squarely at this. That has changed, and the change is what makes a 2018 story still worth reading.
In New York City, an automated employment decision tool used to substantially assist a hiring decision has required an annual independent bias audit with published results since enforcement of Local Law 144 began in July 2023 — see the bias audit requirement. In Illinois, HB 3773 amended the Human Rights Act to address AI in employment decisions, with obligations commencing 1 January 2026; the Illinois position sets out what it covers. In the EU, a system intended for recruitment or candidate filtering falls in Annex III of the AI Act and carries the full high-risk regime, including the data-governance duties in Article 10 that bear directly on an unrepresentative historical training set.
A tool built the way the Reuters piece describes and then quietly killed would today generate documentation obligations, an audit trail and, in several jurisdictions, a disclosure to candidates. Whether that would have caught the problem earlier is unknowable. What it would certainly have produced is a written record of the training data, which is the artefact whose absence is the reason this case is still argued from a news story eight years later.
Top comments (0)