Bias audit

What NYC Local Law 144 requires

New York City's Local Law 144 of 2021 is the best-known bias audit rule. It is enforced by the city's Department of Consumer and Worker Protection (DCWP), and enforcement began on 5 July 2023. The points below follow DCWP's AEDT FAQ. This is a summary, not legal advice.

• Which tools. An automated employment decision tool (AEDT) uses machine learning, statistical modelling, data analytics or AI, helps with employment decisions, and substantially assists or replaces discretionary decision-making. Screening counts, not only the final hire. Ranking or match-score features inside an applicant tracking system can fall in scope, depending on how the employer uses them. Using a tool to search a resume bank or run outreach to people who have not applied is not covered.
• When. The tool must have had a bias audit within the past year before each use.
• Who audits. An independent auditor: not employed by the employer or the vendor, not involved in building or using the tool, and without a financial interest in either.
• Who is responsible. The employer or employment agency, not the vendor. A vendor may commission an audit, but an employer can rely on a multi-employer audit only if it contributed its own data or is using the tool for the first time.
• What is measured. Selection or scoring rates and impact ratios across sex categories, race and ethnicity categories, and intersectional categories. Categories under 2 percent of the data may be excluded.
• What data. Historical data from real use. Test data is allowed when historical data is insufficient. Imputed or inferred demographics are not allowed.
• What is published. A summary with the audit date, the data source, the number of people in an unknown category, and the counts, rates and impact ratios for every category, plus the date the tool was first used.
• Notice. Candidates and employees who live in the city must be told, 10 business days before use, that an AEDT will be used and which qualifications it assesses, with instructions for requesting an accommodation.

The law requires the audit, not any particular response to its results. Anti-discrimination law still applies to what the results show.

Worked example: auditing a scoring tool

For a tool that outputs a score rather than a yes or no, the city's rules use a scoring rate: the share of a category that scores above the median score of everyone assessed. Ledgerly, a fictional fintech hiring in New York, has an independent auditor review last year's 1,000 scored applicants. Half the pool, by definition, scored above the median.

The impact ratio for female applicants is 45 / 55 = 0.82. The auditor repeats the table for race and ethnicity, then for each sex and race combination, where small cells often show bigger gaps than the totals. Any intersectional category under 2 percent of the 1,000 (fewer than 20 people) may be left out of the calculations. Ledgerly publishes the summary, keeps its notice live on its careers page, and, because one intersectional ratio comes in at 0.71, asks which inputs drive it. That last step is bias mitigation, which the law does not require but common sense does.

Bias audit vs internal adverse impact analysis

Both rest on the same arithmetic, explained under adverse impact.

Bias audits beyond New York City

• Colorado. SB26-189, signed on 14 May 2026, repealed and reenacted the state's 2024 AI law as a law on automated decision-making technology. For decisions including employment, the legislature's summary describes notice, a plain-language description of the tool's role within 30 days after an adverse decision, a right to request meaningful human review, and record keeping for at least three years, with main duties starting 1 January 2027. It does not describe a bias audit requirement.
• European Union. The AI Act treats recruitment and selection tools as high-risk. It uses risk management, data governance (including examining training data for possible bias), human oversight and logging rather than a published audit of selection rates.

Laws in this area change often. Check the current text before relying on any summary.

Common mistakes

• Relying on the vendor's audit by default. It covers you only if you supplied data to it or this is your first use of the tool.
• Inferring demographics. Imputed race or sex cannot be used. Collect self-reported data where you lawfully can, or use test data and explain why.
• Forgetting the notice. The audit alone does not satisfy the law.
• Stopping at the totals. Intersectional categories are required and often where the gaps are.
• Losing the data. A yearly audit needs a year of scores and outcomes. Plan for auditability from the first day of use.

How ScoringFactory approaches it

ScoringFactory ranks candidates for a team and cites each score to the line in the record; it never rejects anyone, and the hiring manager decides. Whether a given deployment is an AEDT depends on how the employer uses it, so employers hiring in New York City should assess that with counsel, and the scores and records the product keeps can support an audit. Practices are on the trust page. See also AI candidate scoring and AI governance.