Case Study 2: James's Algorithm Audit — The Logistic Regression Behind the Risk Scores

Contributors

Case Study 2: James's Algorithm Audit — The Logistic Regression Behind the Risk Scores

The Setup

Professor James Washington has spent the entire semester studying a predictive policing algorithm. In Chapter 1, he asked whether the algorithm showed racial bias. In Chapter 9, he calculated conditional probabilities and found that false positive rates differed by race. In Chapter 16, he tested whether those differences were statistically significant. In Chapter 23, he built a multiple regression model controlling for criminal history and found that race still had a measurable effect.

Now, in Chapter 24, the final piece falls into place: James discovers that the algorithm itself is a logistic regression model. The "risk score" that judges use to make bail decisions is the output of a sigmoid function applied to a linear combination of defendant characteristics. The mysterious black box was never a black box at all.

"All this time," James tells his research assistant, "I've been studying the outputs of the algorithm — the risk scores, the predictions, the error rates. Now I can study the model itself. I can see the weights. I can compute the odds ratios. I can trace exactly how race enters the prediction."

The Algorithm

The recidivism prediction algorithm works like this:

Inputs: Defendant age at first arrest, number of prior offenses, severity of current charge (scale 1-5), employment status, and race.
Model: A logistic regression that outputs P(reoffend within 2 years).
Threshold: If P > 0.50, the defendant is classified as "high risk."
Output: The risk classification (high/low) is shared with the judge at the bail hearing.

import numpy as np
import pandas as pd
import statsmodels.api as sm
from sklearn.metrics import (confusion_matrix, classification_report,
                             roc_curve, auc, accuracy_score)
import matplotlib.pyplot as plt

# ============================================================
# JAMES'S ALGORITHM AUDIT — THE FULL INVESTIGATION
# ============================================================

np.random.seed(2024)
n = 1000

# Defendant characteristics
age_first_arrest = np.random.normal(22, 5, n).clip(14, 50).astype(int)
prior_offenses = np.random.poisson(2.5, n)
charge_severity = np.random.choice([1, 2, 3, 4, 5], n,
                                    p=[0.15, 0.25, 0.30, 0.20, 0.10])
employed = np.random.binomial(1, 0.50, n)
race = np.random.choice([0, 1], n, p=[0.42, 0.58])  # 0=White, 1=Black

# Systemic factors: prior offenses are correlated with race
# due to differential policing (not differential behavior)
prior_offenses = prior_offenses + (race * np.random.poisson(1, n))

# True recidivism model
log_odds = (-1.0
            - 0.03 * age_first_arrest
            + 0.35 * prior_offenses
            + 0.15 * charge_severity
            - 0.40 * employed
            + 0.20 * race)  # direct race effect in the data
prob_reoffend = 1 / (1 + np.exp(-log_odds))
reoffended = np.random.binomial(1, prob_reoffend)

defendants = pd.DataFrame({
    'age_first_arrest': age_first_arrest,
    'prior_offenses': prior_offenses,
    'charge_severity': charge_severity,
    'employed': employed,
    'race': race,
    'race_label': np.where(race == 0, 'White', 'Black'),
    'reoffended': reoffended
})

print("=" * 60)
print("DEFENDANT DATA OVERVIEW")
print("=" * 60)
print(f"Total defendants: {n}")
print(f"White: {(defendants['race']==0).sum()} ({(defendants['race']==0).mean():.1%})")
print(f"Black: {(defendants['race']==1).sum()} ({(defendants['race']==1).mean():.1%})")
print(f"\nOverall recidivism rate: {defendants['reoffended'].mean():.1%}")
print(f"White recidivism rate:  {defendants[defendants['race']==0]['reoffended'].mean():.1%}")
print(f"Black recidivism rate:  {defendants[defendants['race']==1]['reoffended'].mean():.1%}")

Part 1: The Algorithm's Coefficients

James first examines the model that the algorithm uses — the logistic regression with all five predictors, including race.

# ============================================================
# THE ALGORITHM: Logistic regression WITH race as a predictor
# ============================================================

predictors_with_race = ['age_first_arrest', 'prior_offenses',
                         'charge_severity', 'employed', 'race']

X_with = sm.add_constant(defendants[predictors_with_race])
model_with_race = sm.Logit(defendants['reoffended'], X_with).fit()

print("\n" + "=" * 60)
print("MODEL 1: THE ALGORITHM (includes race)")
print("=" * 60)
print(model_with_race.summary())

# Odds ratios
print("\n--- Odds Ratios ---")
or_with = pd.DataFrame({
    'Coefficient': model_with_race.params.round(4),
    'Odds Ratio': np.exp(model_with_race.params).round(3),
    'p-value': model_with_race.pvalues.round(4)
})
print(or_with)

James examines the output carefully:

"The coefficient for race is approximately 0.20, which translates to an odds ratio of about 1.22. This means that, even after controlling for age at first arrest, prior offenses, charge severity, and employment status, being Black is associated with approximately 22% higher odds of being classified as likely to reoffend."

But is this the whole story? James suspects that prior offenses — the strongest predictor — may itself be contaminated by differential policing.

Part 2: The Race-Neutral Alternative

James builds a second model that excludes race as a direct predictor.

# ============================================================
# THE ALTERNATIVE: Logistic regression WITHOUT race
# ============================================================

predictors_without_race = ['age_first_arrest', 'prior_offenses',
                            'charge_severity', 'employed']

X_without = sm.add_constant(defendants[predictors_without_race])
model_without_race = sm.Logit(defendants['reoffended'], X_without).fit()

print("\n" + "=" * 60)
print("MODEL 2: RACE-NEUTRAL ALTERNATIVE")
print("=" * 60)
print(model_without_race.summary())

# Compare odds ratios
print("\n--- Comparison: Does removing race change other coefficients? ---")
comparison = pd.DataFrame({
    'OR with race': np.exp(model_with_race.params[1:]).round(3),
    'OR without race': np.exp(model_without_race.params[1:]).round(3)
})
# Only compare the common predictors
common_preds = ['age_first_arrest', 'prior_offenses',
                'charge_severity', 'employed']
for pred in common_preds:
    or_w = np.exp(model_with_race.params[pred])
    or_wo = np.exp(model_without_race.params[pred])
    print(f"  {pred}: OR with race = {or_w:.3f}, "
          f"OR without race = {or_wo:.3f}")

James notices something crucial: when race is removed from the model, the coefficient for prior offenses increases. This is because prior offenses is correlated with race (due to differential policing), so when race is removed, prior offenses absorbs some of the race effect.

"Removing race from the model doesn't remove racial bias from the predictions," James writes in his report. "It just hides it inside a proxy variable."

Part 3: Differential Error Rates

This is the analysis that matters most. James evaluates both models' predictions, broken down by race.

# ============================================================
# ERROR RATE ANALYSIS BY RACE
# ============================================================

# Predictions from both models
defendants['prob_with_race'] = model_with_race.predict(X_with)
defendants['pred_with_race'] = (defendants['prob_with_race'] >= 0.5).astype(int)

defendants['prob_without_race'] = model_without_race.predict(X_without)
defendants['pred_without_race'] = (defendants['prob_without_race'] >= 0.5).astype(int)

print("\n" + "=" * 60)
print("ERROR RATE ANALYSIS BY RACE")
print("=" * 60)

for model_name, pred_col in [("WITH race", 'pred_with_race'),
                               ("WITHOUT race", 'pred_without_race')]:
    print(f"\n{'=' * 50}")
    print(f"Model {model_name}")
    print(f"{'=' * 50}")

    for race_val, race_name in [(0, 'White'), (1, 'Black')]:
        subset = defendants[defendants['race'] == race_val]
        cm = confusion_matrix(subset['reoffended'], subset[pred_col])

        if cm.shape == (2, 2):
            tn, fp, fn, tp = cm.ravel()
        else:
            continue

        total = len(subset)
        accuracy = (tp + tn) / total
        fpr = fp / (fp + tn) if (fp + tn) > 0 else 0
        fnr = fn / (fn + tp) if (fn + tp) > 0 else 0
        sensitivity = tp / (tp + fn) if (tp + fn) > 0 else 0
        specificity = tn / (tn + fp) if (tn + fp) > 0 else 0
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0

        print(f"\n  {race_name} defendants (n={total}):")
        print(f"    Accuracy:    {accuracy:.3f}")
        print(f"    Sensitivity: {sensitivity:.3f}  (catches {sensitivity:.1%} of actual reoffenders)")
        print(f"    Specificity: {specificity:.3f}  (correctly clears {specificity:.1%} of non-reoffenders)")
        print(f"    FPR:         {fpr:.3f}  ({fpr:.1%} of non-reoffenders wrongly flagged)")
        print(f"    FNR:         {fnr:.3f}  ({fnr:.1%} of reoffenders wrongly cleared)")
        print(f"    Precision:   {precision:.3f}  (of those flagged, {precision:.1%} actually reoffend)")

Part 4: ROC Curves by Race

# ============================================================
# ROC CURVES BY RACE
# ============================================================

fig, axes = plt.subplots(1, 2, figsize=(16, 7))

for idx, (model_name, prob_col) in enumerate([
    ("WITH race", 'prob_with_race'),
    ("WITHOUT race", 'prob_without_race')
]):
    ax = axes[idx]

    for race_val, race_name, color in [(0, 'White', 'steelblue'),
                                         (1, 'Black', 'coral')]:
        subset = defendants[defendants['race'] == race_val]
        fpr_r, tpr_r, _ = roc_curve(subset['reoffended'],
                                     subset[prob_col])
        auc_r = auc(fpr_r, tpr_r)

        ax.plot(fpr_r, tpr_r, color=color, linewidth=2,
                label=f'{race_name} (AUC = {auc_r:.3f})')

    ax.plot([0, 1], [0, 1], color='gray', linestyle='--',
            label='Random')
    ax.set_xlabel('False Positive Rate', fontsize=11)
    ax.set_ylabel('True Positive Rate', fontsize=11)
    ax.set_title(f'ROC Curve — Model {model_name}', fontsize=13)
    ax.legend(loc='lower right')
    ax.grid(True, alpha=0.3)

plt.suptitle("James's Fairness Audit: ROC Curves by Race",
             fontsize=14, fontweight='bold')
plt.tight_layout()
plt.show()

Part 5: The Proxy Variable Problem

James digs deeper into why removing race doesn't fix the bias.

# ============================================================
# THE PROXY VARIABLE PROBLEM
# ============================================================

print("\n" + "=" * 60)
print("THE PROXY VARIABLE PROBLEM")
print("=" * 60)

# Prior offenses by race
print("\nPrior offenses by race:")
print(defendants.groupby('race_label')['prior_offenses']
      .agg(['mean', 'median', 'std']).round(2))

# Employment by race
print("\nEmployment rate by race:")
print(defendants.groupby('race_label')['employed'].mean().round(3))

# Correlation between race and predictors
print("\nCorrelations with race (0=White, 1=Black):")
for pred in ['prior_offenses', 'charge_severity',
             'employed', 'age_first_arrest']:
    corr = defendants['race'].corr(defendants[pred])
    print(f"  {pred}: r = {corr:.3f}")

James explains: "Prior offenses is correlated with race — not because Black individuals commit more crimes, but because Black communities experience more intensive policing. More police presence means more arrests for the same behavior. The algorithm treats prior arrests as a neutral predictor, but prior arrests are themselves a product of biased policing.

"This is the fundamental problem with 'race-neutral' algorithms. You can remove race from the model, but if the remaining predictors are correlated with race — and they are, because they are products of a society with racial inequality — then the predictions will still differ by race. The bias is laundered through the proxy variables."

Part 6: The Impossibility Theorem

James encounters a mathematical result that deepens the analysis.

# ============================================================
# THE FAIRNESS IMPOSSIBILITY DEMONSTRATION
# ============================================================

print("\n" + "=" * 60)
print("THE FAIRNESS IMPOSSIBILITY DEMONSTRATION")
print("=" * 60)

# Different base rates by race
for race_val, race_name in [(0, 'White'), (1, 'Black')]:
    subset = defendants[defendants['race'] == race_val]
    base_rate = subset['reoffended'].mean()
    print(f"\n{race_name} base rate (recidivism): {base_rate:.3f}")

print("\nChouldechova (2017) proved: when base rates differ between")
print("groups, it is MATHEMATICALLY IMPOSSIBLE for an algorithm to")
print("simultaneously satisfy:")
print("  1. Equal false positive rates across groups")
print("  2. Equal false negative rates across groups")
print("  3. Equal positive predictive values across groups")
print("\n(except in trivial cases)")
print("\nThis means SOME form of unfairness is inevitable.")
print("The question is: which kind of unfairness is least harmful?")

James's Report to the Judicial Review Board

Based on his semester-long investigation, James drafts a report:

Executive Summary

The predictive policing algorithm used in bail decisions is a logistic regression model — a standard statistical tool. Our audit reveals three findings:

Finding 1: Race directly enters the prediction. The algorithm includes race as a predictor variable. Even after controlling for criminal history, charge severity, and employment, being Black is associated with approximately 22% higher odds of being classified as high-risk (OR = 1.22, p < 0.05).

Finding 2: Removing race doesn't remove bias. When we rebuilt the model without race as a direct predictor, the false positive rate for Black defendants remained higher than for White defendants. This is because other predictors — particularly prior offenses — are themselves correlated with race due to differential policing practices. These variables serve as proxies for race.

Finding 3: Perfect fairness is mathematically impossible. Because recidivism base rates differ between racial groups (due to systemic factors), no algorithm can simultaneously equalize false positive rates, false negative rates, and predictive values across groups. This is a mathematical fact, not a limitation of this particular algorithm.

Recommendations:

Remove race as a direct predictor. While this doesn't eliminate racial disparities in predictions, it removes the most explicit source of differential treatment.

Set different thresholds by group to equalize false positive rates. If the cost of a false positive (wrongly denying bail to someone who would not have reoffended) is considered the most important fairness criterion, the threshold for Black defendants should be higher than for White defendants.

Report error rates by race alongside overall accuracy. Judges who use the algorithm should know not just its overall accuracy but also its differential error rates.

Consider whether the algorithm should be used at all. Dressel and Farid (2018) showed that untrained humans predict recidivism about as accurately as the algorithm. If a simple tool with fewer variables performs comparably, the added complexity (and potential for hidden bias) may not be justified.

Address the upstream causes. The algorithm reflects the data it was trained on. If the data reflects biased policing, the algorithm will reproduce that bias. Fixing the algorithm is necessary but not sufficient — the policing practices that generate biased data must also be reformed.

Discussion Questions

James writes that "the bias is laundered through the proxy variables." What does he mean? Can you think of other situations where a facially neutral variable serves as a proxy for a protected characteristic?
The impossibility theorem says that when base rates differ, perfect fairness across multiple criteria is impossible. Does this mean we should give up on fairness? Or does it mean we need to choose which kind of fairness to prioritize — and if so, who should make that choice?
Professor Washington asks his students: "If you could only equalize ONE metric across racial groups — false positive rate, false negative rate, or positive predictive value — which would you choose? Why?"
The algorithm assigns a probability to each defendant. A judge then uses a threshold (0.50) to make a binary decision. Should the judge see the probability (e.g., "62% risk") or only the binary classification (e.g., "high risk")? What are the advantages and dangers of each approach?
Connect this case study to the Bayes' theorem material from Chapter 9. If the base rate of recidivism is 30% for White defendants and 45% for Black defendants, and the algorithm has the same sensitivity (80%) and specificity (70%) for both groups, calculate the positive predictive value for each group. What does this tell you about the meaning of a "high risk" classification for each group?
James's recommendation #4 suggests that the algorithm might not be necessary. Under what circumstances would you argue that an algorithm is better than human judgment, even with its known biases? Under what circumstances would you argue for human judgment?