How algorithmic bias in AI hiring tools can replicate workplace discrimination

Companies increasingly use software built on machine learning to screen résumés, rank candidates and even conduct early-stage interviews. Researchers and regulators have found that several of these systems produce different outcomes for applicants of different genders, races or ages, even when the underlying qualifications are identical.

What is algorithmic bias in hiring?

Algorithmic bias occurs when a machine-learning system's outputs systematically favor or disadvantage certain groups, independent of job-relevant qualifications. Public comments on a draft standard from the U.S. National Institute of Standards and Technology (NIST), SP 1270, 'Towards a Standard for Identifying and Managing Bias in Artificial Intelligence,' published on NIST's website, describe this as a socio-technical problem: bias can enter not only through the statistical design of an algorithm but through the historical data it is trained on and the human and institutional context in which it is deployed.

In hiring specifically, a model is usually trained on past résumés, past hiring decisions, or text generated by a large language model (LLM). If the workforce that produced that historical data was itself skewed by gender, race or age — a common feature of many industries — the model can learn to reproduce that skew as if it were a measure of merit.

A documented case: Amazon's scrapped recruiting engine

One of the earliest and most widely cited examples involves an experimental recruiting tool Amazon built starting in 2014 to score job applicants from one to five stars, according to the American Civil Liberties Union (ACLU). The system was trained by observing patterns in résumés the company had received over roughly a decade, a pool that was overwhelmingly male, reflecting male dominance in the technical roles involved.

The ACLU reports that the tool penalized résumés containing the word 'women's' — for example, 'women's rugby team' — and downgraded graduates of two all-women's colleges, while favoring resumes that used verbs more common in men's technical résumés, such as 'executed' and 'captured.' Amazon's engineers attempted to fix the pattern but, according to the ACLU's account of reporting on the project, concluded the underlying problem could not be guaranteed away and shut the tool down in 2015, about a year after it began being tested.

What recent research on AI resume screening found

Newer research has tested the large language models now embedded in many commercial hiring products. A University of Washington study presented in October 2024 at the AAAI/ACM Conference on Artificial Intelligence, Ethics and Society tested three open-source LLMs — from Mistral AI, Salesforce and Contextual AI — by assigning 120 names associated with white and Black men and women to more than 550 real résumés and ranking them against over 500 real job postings across nine occupations, producing more than three million résumé-to-job comparisons.

Lead author Kyra Wilson and co-author Aylin Caliskan, both at the University of Washington's Information School, found that résumés with white-associated names were favored 85% of the time compared with 9% for Black-associated names, and male-associated names were preferred 52% of the time compared with 11% for female-associated names. The models never favored Black male-associated names over white male names, a pattern the researchers said became visible only when race and gender were analyzed together rather than separately.

Key findings at a glance

How the bias mechanism works

In each documented case, the pattern traces back to training data or associations the model absorbed from text it was trained on, rather than to an explicit instruction to discriminate. The Amazon tool learned from a historically male applicant pool. The LLMs tested by University of Washington researchers carried name-based racial and gender associations from the broader text corpora on which they were trained.

Comments on NIST's draft bias standard note that because these systems operate within existing social and institutional structures, removing bias from the dataset alone does not guarantee a fair outcome — a conclusion that matches Amazon's own experience, as described by the ACLU, when engineers' repeated fixes failed to resolve the underlying pattern.

How regulators are responding

New York City's Local Law 144, enforced by the Department of Consumer and Worker Protection since July 2023, requires employers using an 'automated employment decision tool' for hiring or promotion to commission an independent bias audit annually, publicly post a summary of the audit results, and give candidates at least ten business days' notice before the tool is used, according to the law firm Pivot Point Security's summary of the measure. Violations can draw civil penalties ranging from $500 to $1,500 per violation, according to the same summary.

California's Civil Rights Council adopted new regulations under the state's Fair Employment and Housing Act, effective October 1, 2025, clarifying that an employer's use of an 'automated-decision system' can violate state law if it harms applicants or employees based on protected characteristics such as race, gender or disability, according to the Council's own June 2025 announcement. The regulations require employers to retain automated-decision data for at least four years and note that certain automated assessments eliciting disability-related information may count as unlawful medical inquiries.

What this means for job applicants and employers

• Bias found in hiring AI to date has tracked existing social patterns in training data — historical résumés, online images, or general-purpose text — rather than appearing at random.
• Documented effects include favoring résumés from historically dominant demographic groups and producing systematically different outcomes for identical underlying qualifications.
• New York City and California now impose specific legal obligations — audits, record-keeping, or both — on employers using these tools, though requirements differ by jurisdiction.
• University of Washington researchers say bias identified in one demographic category (e.g., race alone) can mask different effects visible only when categories are combined, such as race and gender together.