AI/ML Explained: Fairness, Bias & Debiasing

Model bias isn't "one line of code got it wrong"—it's like an ETL data pipeline where collection, sampling, labeling, and aggregation each inject systematic skew, which then compounds and amplifies inside the model. Just like distributed systems: contamination often isn't at the endpoint but at some upstream sampling node that quietly dropped data, and everything downstream inherits it unknowingly. "Garbage in, garbage out" is true, but bias is subtler—data that is compliant, complete, and bug-free can still be biased.

To govern bias, first know where it enters. The literature decomposes pipeline bias into several independent injection points—this matters because different sources need entirely different fixes; conflating them wastes effort:

• Historical bias—data faithfully reflects an unjust world. A role has historically skewed male; the model learns a "fact," but that fact itself needs correcting;
• Representation bias—sampling leaves some groups underrepresented. Face datasets are sparse in dark-skinned samples → the model has systematically higher error on those groups;
• Measurement bias—what you want to predict can't be measured directly, so you use a proxy label. You want to predict "crime" but only have "arrest" data, and enforcement intensity varies across groups—the proxy-vs-truth gap becomes model bias;
• Aggregation bias—forcing one model onto a heterogeneous population, optimal for no one;
• Learning/evaluation bias—the loss minimizes aggregate error, naturally sacrificing minority groups; an equally skewed test set then hides the problem.

import pandas as pd

# A "clean-looking" hiring history: no missing values, no errors, but biased
df = pd.read_csv("hiring.csv")  # cols: gender, hired(0/1)

# ① Representation check: are group sample sizes wildly unequal?
print(df["gender"].value_counts(normalize=True))

# ② Historical-bias check: how much does the positive-label base rate differ?
#    The model will treat this historical gap as "signal" and amplify it
base_rates = df.groupby("gender")["hired"].mean()
print(base_rates)
print("base-rate disparity:", base_rates.max() - base_rates.min())
# Large gap → training on it directly = baking historical injustice into the model

"Fairness" is not one metric—it's like an SLA definition. For the same system you can promise "equal throughput per tenant" or "equal error rate per tenant"; these are two different contracts and you can't satisfy everyone at once. Fairness metrics translate "fair" into computable mathematical conditions, and which definition you pick is itself a value judgment, not a technical detail.

"Is this model fair?" is unanswerable until you specify fair by which definition. The two mainstream group-fairness definitions differ at the root: look at outputs, or look at errors?

• Demographic Parity (statistical parity)—requires the positive prediction rate to be equal across groups: P(Ŷ=1 | A=x) = P(Ŷ=1 | A=y). Intuition: the acceptance rate is the same for every group, regardless of each group's true qualification rate. It only looks at the model's output distribution;
• Equalized Odds—requires true positive rate (TPR) and false positive rate (FPR) to be equal across groups. Intuition: among those who are truly qualified, every group is accepted with the same probability (equal TPR); among the unqualified, every group is falsely accepted with the same probability (equal FPR). It checks whether the model's "way of making mistakes" is consistent across groups;
• Equal Opportunity—a relaxed version of equalized odds requiring only equal TPR (qualified people treated fairly), ignoring FPR. Use it when "missing truly qualified people" is the main harm.

Key distinction: demographic parity ignores the truth Y, looking only at output rates; equalized odds is conditioned on the truth, requiring consistent error distributions. So if two groups genuinely have different qualification rates, these two definitions clash—force acceptance rates equal and you necessarily create a higher error rate in some group. This isn't a bug; it's the mathematical inevitability of the next card.

from fairlearn.metrics import (MetricFrame,
    demographic_parity_difference, equalized_odds_difference)
from sklearn.metrics import selection_rate, true_positive_rate

# y_true = truth, y_pred = prediction, A = sensitive attribute (e.g. gender)
mf = MetricFrame(
    metrics={"selection_rate": selection_rate, "TPR": true_positive_rate},
    y_true=y_true, y_pred=y_pred, sensitive_features=A)

print(mf.by_group)          # each metric, per group

# One number summarizing the gap: 0 = perfectly fair, larger = less fair
print("demographic parity diff:", demographic_parity_difference(
    y_true, y_pred, sensitive_features=A))     # range of selection rates
print("equalized odds diff:", equalized_odds_difference(
    y_true, y_pred, sensitive_features=A))     # larger of TPR/FPR ranges
# Often one is small and the other large—proof they measure different "fairness"

Fairness has its own CAP theorem. Just as distributed systems can't simultaneously have consistency (C), availability (A), and partition tolerance (P), a risk-scoring model's three fairness notions—calibration, FPR balance, TPR balance—are mathematically impossible to satisfy at once when group base rates differ. This isn't bad engineering; it's a proven ceiling: you must choose which to sacrifice.

In 2016 two groups of researchers (Kleinberg et al.; and Chouldechova) almost simultaneously proved this result, prompted by the famous COMPAS recidivism controversy—ProPublica charged the algorithm with a higher false-positive rate for Black defendants, while vendor Northpointe countered that the algorithm was calibrated (a given score means the same thing across groups). The impossibility theorem reveals: both are right, because the two fairness notions they want simply cannot coexist.

The three conditions in plain terms:

• Calibration: the model says "70% will reoffend"—then among people scored this in each group, the actual reoffense rate should be 70%. Scores mean the same across groups;
• FPR balance: among people who won't reoffend, each group is falsely flagged high-risk at the same rate;
• TPR balance: among people who will reoffend, each group is correctly identified at the same rate.

The theorem: as long as two groups have different base rates (different true reoffense rates) and the model isn't 100% perfect, the three above can hold at most two at a time. Why? Calibration forces "same score = same true risk," but when one group has a higher overall base rate, to maintain calibration the model must, in that group, either produce more false positives or miss more true positives—errors cannot be aligned across groups. The base-rate difference gets "squeezed" into the error rates; there's no escape.

import numpy as np
# A minimal example of how "calibration" and "FPR balance" strangle each other
# Two groups, different base rates: A true-positive rate 0.6, B 0.3

def rates(y_true, score, thr=0.5):
    pred = (score >= thr).astype(int)
    tpr = pred[y_true == 1].mean()          # true positive rate
    fpr = pred[y_true == 0].mean()          # false positive rate
    return tpr, fpr

# Both groups' models are perfectly calibrated (score=true prob), base rates differ
rng = np.random.default_rng(0)
yA = (rng.random(10000) < 0.6).astype(int)  # A: high base rate
yB = (rng.random(10000) < 0.3).astype(int)  # B: low base rate
sA = np.where(yA==1, rng.random(10000)*.5+.5, rng.random(10000)*.5)
sB = np.where(yB==1, rng.random(10000)*.5+.5, rng.random(10000)*.5)

print("A TPR,FPR:", rates(yA, sA))
print("B TPR,FPR:", rates(yB, sB))
# Same threshold for both (calibrated) → but FPRs differ; to equalize FPR you'd
# need different thresholds → then a given score means different things → breaks
# calibration. No solution.

Debiasing is like data quality governance—you can clean at ingest (pre-processing), enforce in write constraints (in-processing), or patch at read (post-processing), each position with its own cost. Causal fairness goes further: instead of settling for "looks equal on correlations," it asks a counterfactual—"if this person's sensitive attribute were changed, would the decision change?"—upgrading fairness from statistical correlation to causal mechanism.

Knowing the sources and the metrics, how do you fix it? Three interventions, classified by where in the pipeline they act:

• Pre-processing—change the data: reweight, resample, or transform feature representations to strip sensitive info. The classic is Bolukbasi 2016 word-embedding debiasing—find the "gender direction" in embeddings and project it out of occupation words. Pro: decoupled from downstream; con: may lose useful signal;
• In-processing—change the objective: add a fairness constraint to the loss, or use adversarial debiasing—add a discriminator whose job is to guess the sensitive attribute from the representation, and train the main model until it can't. Pro: best trade-off; con: requires touching model internals;
• Post-processing—change the output: leave the model alone, only adjust decision thresholds per group. Hardt et al. 2016 proved group-specific thresholds can exactly achieve equalized odds. Pro: no retraining, works on a black-box API; con: explicitly treats groups differently, which may raise legal/ethical concerns.

Causal fairness is a different route. All the metrics above are correlational; they can't tell whether "zip code affects the decision" because zip code is relevant on its own or because it's a proxy for race. Counterfactual fairness (Kusner et al. 2017) defines it thus: a decision is fair if and only if, in the counterfactual world—flip the individual's sensitive attribute A to another value and update all downstream variables caused by A along the causal graph—the prediction distribution is unchanged. It requires first drawing a causal DAG, distinguishing "legitimate descendants of A" from "discriminatory paths that should be blocked."

from fairlearn.postprocessing import ThresholdOptimizer
from sklearn.ensemble import GradientBoostingClassifier

# Post-processing debiasing: no retraining, tune per-group thresholds for equalized odds
base = GradientBoostingClassifier().fit(X_tr, y_tr)

fair = ThresholdOptimizer(
    estimator=base,
    constraint="equalized_odds",   # target fairness definition (see previous card)
    predict_method="predict_proba",
    prefit=True)

# Fitting requires the sensitive attribute—it learns thresholds per group
fair.fit(X_tr, y_tr, sensitive_features=A_tr)
y_fair = fair.predict(X_te, sensitive_features=A_te)
# Cost: overall accuracy usually drops slightly—the "fairness tax" you pay
# under the impossibility theorem. Note: inference also needs the sensitive
# attribute, which may raise compliance constraints in deployment.

• Fairness and Machine Learning (Barocas, Hardt, Narayanan) — the authoritative free textbook; the clearest treatment of bias sources and fairness definitions, start here
• Kleinberg, Mullainathan, Raghavan 2016 — Inherent Trade-Offs in the Fair Determination of Risk Scores — the foundational impossibility-theorem paper
• Chouldechova 2017 — Fair Prediction with Disparate Impact — the COMPAS controversy and an independent proof of the impossibility result
• Hardt, Price, Srebro 2016 — Equality of Opportunity in Supervised Learning — the equal-opportunity definition + post-processing debiasing
• Kusner et al. 2017 — Counterfactual Fairness — the landmark work defining fairness via causal inference
• Bolukbasi et al. 2016 — Debiasing Word Embeddings — a classic on gender bias in word vectors and pre-processing debiasing