DEV Community

Multigrid
Multigrid

Posted on • Originally published at multigrid.ai

NYC Local Law 144: What the Bias Audit Must Calculate

Local Law 144 does not ask whether your tool is fair. It asks for a set of ratios, computed on named categories, by someone independent, within the last year, published on your website.

The duty, and the one-year clock

New York City Local Law 144 of 2021 sits in the Administrative Code at sections 20-870 to 20-874. It took effect on 1 January 2023 and the Department of Consumer and Worker Protection began enforcing it on 5 July 2023, after adopting final rules — Title 6, Chapter 5, Subchapter T of the Rules of the City of New York — on 6 April 2023. The core prohibition is simple: an employer or employment agency may not use an automated employment decision tool to screen a candidate or employee for an employment decision within the city unless the tool has been the subject of a bias audit conducted no more than one year before its use, and a summary of the results has been made publicly available.

The one-year window is a rolling requirement, not a one-off. An audit dated 3 March 2026 stops supporting lawful use on 3 March 2027, and because each day of use is treated as a separate violation, letting the audit lapse is not a single infraction. The DCWP’s guidance page for the law is at nyc.gov.

Not legal advice. Whether your tool is an AEDT at all is the prior question and it is not obvious — see the definition page — and getting it wrong in either direction has costs.

What makes an auditor independent

The rules define an independent auditor as a person or group capable of exercising objective and impartial judgement on all issues within the scope of the audit. The rules then disqualify three situations: the auditor was involved in using, developing or distributing the tool; the auditor has an employment relationship with an employer or employment agency that seeks to use the tool, or with a vendor that developed or distributed it; and the auditor has a direct financial interest, or a material indirect financial interest, in such an employer, agency or vendor.

Two consequences follow that surprise people. Your own data science team cannot audit your own tool, however competent. And a vendor may commission an audit of its product from an independent third party, which many do — but the employer remains the party that must make a summary publicly available, so relying on a vendor’s audit without republishing anything leaves the employer’s own duty unsatisfied.

The arithmetic: selection rate and impact ratio

For a tool that selects candidates, the audit computes a selection rate for each category: the number of individuals in that category who were selected, divided by the number of individuals in that category who applied. For a tool that scores candidates, it computes a scoring rate: the rate at which individuals in a category receive a score above the median score for the whole sample.

The impact ratio is then the comparison the rules actually require to be published:

selection rate (category) = selected in category / applicants in category

impact ratio (selection) = selection rate of category
                           / selection rate of the most selected category

scoring rate (category)  = scored above the sample median in category
                           / applicants in category

impact ratio (scoring)   = scoring rate of category
                           / scoring rate of the highest scoring category

worked example
  category A: 400 applicants, 100 selected -> 0.250
  category B: 200 applicants,  30 selected -> 0.150
  most selected category is A
  impact ratio for B = 0.150 / 0.250 = 0.60
Enter fullscreen mode Exit fullscreen mode

A published summary must give, for each category, the number of applicants or candidates, the selection or scoring rate, and the impact ratio. It must also state the source and explanation of the data used, the number of individuals the tool assessed who fall into an unknown category, and the distribution date of the tool.

The rules do not set a pass mark. There is no threshold in Local Law 144 below which an impact ratio is unlawful — the 0.80 figure people quote comes from the federal Uniform Guidelines on Employee Selection Procedures, which is a different instrument with a different purpose. Publishing a ratio of 0.60 is fully compliant with Local Law 144 and may be extremely useful evidence to a plaintiff under Title VII or under Illinois HB 3773. That gap between the disclosure duty and the discrimination duty is the most important thing to understand about this law.

Categories, including intersectional ones

The categories are the race and ethnicity and sex categories used in the EEOC’s EEO-1 Component 1 report — see the EEOC’s data collection page for the category list itself. The April 2023 rules require three sets of calculations, and the third is the one that catches people out:

  • Sex categories on their own.
  • Race and ethnicity categories on their own.
  • Intersectional categories — each combination of sex with race and ethnicity, which is where a disparity affecting one group can be invisible in both single-axis views.

A category may be excluded from the impact ratio calculation where it represents less than 2% of the data being used, provided the summary says which categories were excluded and gives the number of individuals in them. That exclusion exists because a ratio computed on four applicants is noise, not a finding — but it must be disclosed rather than quietly applied.

Historical data, test data, and the gaps

The audit is to use historical data — the tool’s own real use data. An employer may rely on an audit conducted using the historical data of other employers or employment agencies that used the same tool, which is the mechanism that makes vendor-commissioned audits workable. Test data may be used only where the entity has insufficient historical data, and where it is used the summary must say so and explain why.

The practical obstacle is that many employers do not collect race, ethnicity or sex at the point in the funnel where the tool operates, because collecting it feels like the opposite of fair hiring. The rules do not require you to collect it, and they accommodate the gap through the unknown-category disclosure — but an audit in which most applicants are in the unknown category tells nobody anything, and it is published. Where demographic data is collected voluntarily and kept separate from the selection process, that separation is itself part of the compliance story and should be documented.

Penalties run from the DCWP: $500 for a first violation and for each additional violation occurring on the same day, and $500 to $1,500 for each subsequent violation, with each day of unlawful use counting separately. Failure to provide the required notices is a separate violation again, which is the subject of the notice page.

Related

Top comments (0)