NYC Local Law 144 became effective July 5, 2023 — requiring employers and employment agencies using automated employment decision tools (AEDTs) in hiring or promotion decisions affecting NYC candidates to conduct annual independent bias audits and publish the results.
Two years of published audit results is enough to draw conclusions. Not about whether the law is working in a policy sense — that's a different debate — but about what the audits are actually finding, what the compliance landscape looks like, and what the published results tell practitioners about AI hiring tool governance more broadly.
What Two Years of Audits Shows
Adverse impact ratios are frequently above 0.8 — but not uniformly.
The most common finding across published bias audits: selection rate ratios for most demographic categories fall above the 0.8 four-fifths threshold that indicates adverse impact under EEOC Uniform Guidelines. This is the pattern vendors point to when marketing their tools as "passing" the bias audit.
What the published audits also show: variability is significant across tools, across demographic categories, and across the specific job roles being screened. Resume screening tools that show no adverse impact for white vs. Black candidates may show meaningful disparities for Hispanic candidates or for specific gender categories. The headline "passed the bias audit" often conceals within-audit variation that deserves scrutiny.
Testing methodology varies enormously across auditors.
NYC Local Law 144 specifies that audits be conducted by independent auditors but doesn't specify methodology in detail. The result: audit reports vary dramatically in what they test, how they present results, and how much information is disclosed.
Some published audits report selection rates and adverse impact ratios for five or six demographic categories with detailed confidence intervals and statistical significance assessments. Others report the same metrics for two categories without statistical analysis. Both technically satisfy the disclosure requirement — which tells you more about what the law requires than about what the tools are actually doing.
The Archuz breakdown of NYC Local Law 144 covers what a credible audit methodology should include and how to evaluate published audit results against those standards.
Intersectional analysis is almost universally absent.
Virtually no published NYC Local Law 144 audit includes intersectional analysis — testing for disparities in outcomes for Black women as a combined category, for example, rather than Black candidates and women as separate analyses. This is a significant methodological gap, because single-axis analysis can miss patterns of intersectional discrimination that don't appear when each characteristic is examined independently.
Compliance is uneven and enforcement has been selective.
A 2025 audit of employer compliance with NYC Local Law 144 found significant non-compliance: many employers using AEDTs in NYC hiring hadn't published bias audit results, and some weren't aware the requirement applied to them. Enforcement actions have been issued but don't yet reflect the full scope of non-compliance in the employer population.
What This Means for Employers
"The vendor says they passed the bias audit" is not sufficient diligence. Vendor-commissioned audits have an inherent interest alignment problem — the vendor is the client paying the auditor, and the auditor is competing for repeat business. Some vendors commission audits from multiple auditors and publish the most favorable results. Employers should review published audit results themselves rather than relying on vendor summaries.
The audit is about the tool in the specific deployment context. NYC Local Law 144 audits test the AEDT's outputs — the selection rates the tool produces in actual use. But different deployment contexts can produce different results: an AI hiring tool that shows acceptable adverse impact ratios when screening for one role category may show problematic ratios when screening for a different role. Employers should understand which roles and contexts the published audit covers.
The publication requirement creates a paper trail that plaintiffs can use. Published bias audit results that show disparities in protected categories — even above the four-fifths threshold — can be used as evidence in discrimination litigation. An employer who published an audit showing an adverse impact ratio of 0.81 for one category may find that result becomes relevant if a candidate in that category files a discrimination claim. This isn't a reason to not publish; it's a reason to understand what you're publishing.
The annual cadence matters. A bias audit from 2023 covering a tool that has been updated twice since then isn't current evidence of that tool's current behavior. Employers should confirm that the audit they're relying on covers the current version of the tool in its current deployment configuration.
The Broader Governance Lesson
NYC Local Law 144 is the most specific US employment AI disclosure requirement in force. Its experience over two years reveals something important about bias audit requirements generally: disclosure requirements without methodology standards produce wildly variable information quality.
When an employer publishes an NYC Local Law 144 audit that tests two demographic categories with no statistical significance analysis, and another publishes an audit testing eight categories with full statistical rigor, both are "compliant" with the disclosure requirement — but they're not providing equivalent governance assurance.
For governance practitioners designing AI bias audit programs beyond what NYC Local Law 144 requires: the methodology matters more than the disclosure. A rigorous internal bias assessment is more governance value than a technically compliant public disclosure that doesn't test what matters.
Further reading:
Top comments (0)