Case Study 1: The Hiring Algorithm

The Scenario

Nexus Consulting Group is a large professional services firm with 8,500 employees across 23 offices. In 2021, Nexus implemented an AI-powered resume screening system — call it ScreenPro — to address what its HR leadership described as an unsustainable volume of applications: 45,000 applications per year for approximately 2,000 positions, processed by a team of 12 recruiters.

ScreenPro was trained on Nexus's historical hiring data: approximately 60,000 applications from the previous decade, the 18,500 candidates who received interviews, and the 12,300 who were ultimately hired. The system learned to identify patterns in the applications of people who were hired and to rank new applicants by their similarity to successful hires.

Eighteen months into deployment, an internal audit revealed disturbing patterns. Compared to manual review:

• Female applicants were 34% less likely to be ranked in the top tier, even when controlling for qualifications
• Applicants with names suggesting South Asian or Black backgrounds were screened out at significantly higher rates
• Graduates of certain women's colleges were consistently penalized
• Gaps in employment history — common among applicants who had taken time for caregiving responsibilities — were treated as negative signals

Nexus's Chief People Officer commissioned an investigation. The findings were instructive. The historical data reflected Nexus's past hiring practices — and Nexus, like many professional services firms, had historically hired disproportionately from elite universities, had a male-dominated senior workforce, and had promoted and retained male employees at higher rates than female employees. ScreenPro had not introduced bias; it had learned and amplified the bias already present in the historical data. The system had successfully automated Nexus's historical pattern of discrimination.

The investigation also found that ScreenPro's scoring was not interpretable: the system could rank applicants but could not explain why any given applicant received a particular score. The factors contributing to scores were distributed across hundreds of variables in ways that no human reviewer could examine or audit without specialized machine learning expertise.

When this became public — through a whistleblower report to a journalist — Nexus faced regulatory scrutiny, significant reputational damage, and a class action lawsuit from rejected applicants.

The Philosophical Questions

1. Winner's Politics of Artifacts Applied to ScreenPro

Langdon Winner argues that artifacts have politics: technologies embed values and social relations in their design that shape distributions of benefit and harm.

What values and social relations did ScreenPro embed? The answer requires looking at multiple levels:

The decision to use historical data as training: This seems methodologically natural — use past success to predict future success. But it encodes a crucial assumption: that the patterns of past hiring reflect genuine merit rather than historical bias. At Nexus, this assumption was false. The decision to use historical data was not a purely technical choice; it was a value choice that treated historical patterns as legitimate baselines.

The optimization target: ScreenPro was optimized to identify candidates similar to historical hires. But "similar to" is doing enormous philosophical work. Similar in qualifications? In communication style? In socioeconomic background? In the signals (prestigious university, linear career path, no employment gaps) that correlated with hiring success in a historically biased selection process? The choice of optimization target encoded a particular vision of what a good candidate looks like — a vision derived from Nexus's history of discriminatory preference.

The interpretability gap: The fact that ScreenPro could not explain its rankings was not a technical oversight — it was a design choice prioritizing accuracy over transparency. This choice had political consequences: it made the system difficult to audit, difficult to challenge, and therefore difficult to hold accountable. The opacity was not neutral; it protected the system from scrutiny.

Winner's key insight applies here: the politics of ScreenPro are not in its "use" but in its design. The discrimination was built in — not because anyone intended to discriminate, but because the design choices reflected and reproduced existing patterns of power.

2. Is Algorithmic Screening More "Objective" Than Human Review?

A common argument for algorithmic hiring tools is that they are more objective than human reviewers, who bring unconscious biases, inconsistency, and idiosyncratic preferences. The algorithm, on this view, simply identifies patterns in data — it doesn't have in-group favoritism, it doesn't favor candidates who remind it of itself, it doesn't have good days and bad days.

This argument is philosophically confused in ways that matter.

"Objective" relative to what standard? Objectivity is not an absolute — it is always relative to some criterion. ScreenPro was objective in the sense of being consistent: it applied the same algorithm to every applicant. But consistency in applying a biased algorithm is not fairness; it is consistent unfairness. The algorithm treated all female applicants with similar backgrounds the same way — it consistently ranked them lower than comparable male applicants. This is precisely the wrong kind of objectivity.

The deeper issue: there is no view from nowhere in hiring. Every hiring decision encodes judgments about what matters — what qualifications are relevant, what experiences are valued, what signals of future performance to look for. These are not facts about the world that an algorithm can neutrally read off; they are value judgments that must be made by human beings who are accountable for them.

When a human recruiter makes a biased decision, there is at least in principle a person who can be held responsible, challenged, trained, and corrected. When an algorithm makes a biased decision, the responsibility is diffused — across the designers, the trainers, the data, the deployment decision, the auditors who failed to catch the problem. The appearance of objectivity may actually reduce accountability rather than increase it.

3. Feminist Critique: Whose "Success" Is Being Modeled?

The feminist critique of ScreenPro cuts deeper than bias correction. It asks: what vision of professional success was embedded in the training data?

A career path that correlates with success at Nexus — linear advancement, no employment gaps, attendance at elite universities, long hours, geographical mobility — is not neutral. It is a career path that is historically easier to follow if you are male, if you have a partner who handles domestic responsibilities, if you have resources to attend elite institutions, and if you have no caregiving responsibilities.

The ScreenPro training data did not just record who was hired and who succeeded at Nexus. It recorded who was hired and who succeeded at Nexus under conditions that systematically advantaged certain kinds of workers over others. Training a system to replicate these patterns is not identifying merit; it is automating a particular structural advantage.

This points to a deeper question: what would a genuinely fair hiring algorithm look like? One possibility: identify candidates who have the cognitive and interpersonal capacities needed for the job, regardless of the specific path by which they developed those capacities. But this requires being explicit about what the job actually requires — a hard question that many organizations avoid by using proxies (prestigious degree, linear career path) that are correlated with performance in ways that may reflect structural advantage more than genuine capacity.

4. Applying the Governance Question

The Nexus case raises the question of how algorithmic decision-making should be governed. Several philosophical frameworks suggest different answers.

A consequentialist might focus on outcomes: the question is whether the system produces more good outcomes (better hires, fairer process) than alternatives. The evidence suggests ScreenPro failed on both dimensions — it was both unfair and arguably produced worse hiring decisions than it should have.

A rights-based framework would focus on the rights of job applicants to be judged on their merits, without discrimination. On this view, the critical issue is not just bad outcomes but the violation of a right: applicants have a right to equal consideration, and ScreenPro violated that right systematically.

The Heideggerian analysis points to a different concern: what is lost when the hiring judgment is delegated to a system? The judgment that a particular person would be a valuable colleague is not a computation — it involves attending to a particular person, understanding their specific situation, exercising practical wisdom about how their particular combination of strengths and limitations would function in the specific context. Reducing this to algorithmic pattern-matching may impoverish the decision in ways that neither the consequentialist nor the rights-based framework fully captures.

Discussion Questions

1. Nexus's CEO, in response to the scandal, said: "We removed human bias from the process — we didn't intend for the algorithm to discriminate." Is this a philosophically coherent defense? What does Winner's framework say about the relevance of intention to the politics of artifacts?

2. Imagine redesigning ScreenPro from scratch. What philosophical principles should guide the design? What data should and should not be used for training? What should the optimization target be? Who should be involved in the design process, and why?

3. Some critics of algorithmic hiring argue that all algorithmic screening should be banned from high-stakes decisions. Others argue that with proper design and auditing, algorithmic tools can make hiring fairer than human review. What is the most philosophically defensible position?

4. The ScreenPro case involves discrimination against protected classes (gender, race/ethnicity). But ScreenPro also penalized employment gaps associated with caregiving. Should a hiring algorithm be permitted to penalize employment gaps? What philosophical framework is most useful for answering this?

5. Return to the Meridian Health AI from earlier chapters. What are the structural similarities and differences between the Meridian case and the Nexus case? What does the comparison reveal about the general problem of algorithmic bias in high-stakes decisions?