Measuring disparate impact in human and machine decisions.
Clinical Snapshot
PICO Framework
| P — Population | Pedestrians subject to police stop-and-frisk decisions in New York City (n = 2.2 million stops); broader conceptual application to any decision-making context involving protected characteristics (loan approvals, hiring, clinical triage) |
| I — Intervention | Risk-adjusted regression — a three-step analytic method that (1) estimates decision value/risk via machine learning on all available covariates, (2) measures outcome disparities after adjusting for estimated risk alone, and (3) applies sensitivity analysis for risk mismeasurement |
| C — Comparator | Traditional covariate-adjusted regression (standard disparate-impact statistical testing using observed covariates in a regression model) |
| O — Outcomes | Primary: magnitude of racial disparity in police stop decisions after risk adjustment; Secondary: methodological comparison of risk-adjusted versus traditional regression in detecting unjustified disparate impact; sensitivity of disparity estimates to potential mismeasurement of risk |
Bottom Line
This methodological paper introduces risk-adjusted regression, a three-step framework for detecting unjustified disparities in human and machine decisions. Applied to 2.2 million NYPD stop-and-frisk records, it demonstrates that conventional covariate-adjusted regression substantially underestimates racial disparities by inadvertently adjusting away legitimate evidence of discrimination. The core innovation is adjusting only for a machine-learning-derived risk estimate — capturing all legitimate justifications for a decision — rather than for individual covariates that may serve as proxies for protected characteristics. The sensitivity analysis for risk mismeasurement is a notable methodological strength. For clinicians and health system leaders, this framework offers a principled approach to auditing AI-assisted clinical tools and administrative decisions for demographic disparities. Limitations include reliance on a single US policing dataset, untestable assumptions about the completeness of the risk model, and potential endogeneity in officer-recorded covariates. The paper is newly published with no independent replications. Before adoption in Australian healthcare equity auditing, the method requires validation in clinical datasets and alignment with Australian anti-discrimination law. Nonetheless, it represents a significant conceptual advance for equity-focused health services research.
Key Findings
P Value: Not reported in abstract
Effect Size: Traditional statistical tests of discrimination substantially understate the magnitude of risk-adjusted racial disparities in police stops; specific numerical effect sizes are reported in the full manuscript but not quantified in the abstract
Primary Outcome: Racial disparities in NYPD pedestrian stop decisions, measured using risk-adjusted regression versus traditional covariate-adjusted regression
Nnt Or Sensitivity: Not applicable in traditional clinical sense; the analytic output is a disparity ratio or difference in stop rates between racial groups after risk adjustment, with sensitivity bounds from the mismeasurement analysis
Confidence Interval: Not reported in abstract; sensitivity analysis for risk mismeasurement is provided in Step 3 of the method
Clinical Application
Implementation requires: (1) access to individual-level decision data with covariates, (2) capacity to fit machine learning risk models, and (3) statistical expertise to conduct sensitivity analyses. This is feasible for health system researchers and regulators but not routine clinical practice. The method is computationally accessible with standard statistical software. In Australia, the method is directly relevant to: (1) TGA oversight of AI-based medical devices and clinical decision support tools, where the Therapeutic Goods Act 1989 and emerging AI regulatory frameworks require demonstration of equitable performance across demographic groups; (2) RACGP guidance on algorithmic clinical decision support, where equity auditing is increasingly expected; (3) Medicare and PBS data linkage studies examining differential access to subsidised medicines or procedures by Indigenous Australians, culturally and linguistically diverse populations, and socioeconomically disadvantaged groups; (4) the Australian Human Rights Commission's work on automated decision-making and discrimination. The method could be applied to audit AI tools used in Australian hospital systems, aged care allocation, or NDIS eligibility decisions. However, Australian legal doctrine on disparate impact differs from US civil rights law, and direct legal translation requires caution. The method is proposed for any context where algorithmic or human decisions may produce disparate outcomes across protected groups, including clinical triage, diagnostic tool deployment, resource allocation, and insurance underwriting. In healthcare, relevant applications include AI-assisted diagnostic tools, clinical risk scoring, organ allocation, and treatment recommendation systems.
Abstract
Empirical analyses have grown increasingly important in discrimination litigation with the greater availability of detailed data on individuals and decisions. A popular analytic strategy is to estimate disparities after adjusting for observed covariates, typically with a regression model, in hopes of ferreting out discriminatory intent. This approach, however, is ill-suited to auditing algorithms that are now commonly used to aid decisions, which typically do not include race or other legally protected factors as inputs. Motivated by legal understandings of disparate impact, we introduce an approach that aims to measure "unjustified" disparities in both human and machine decisions. Our method, which we call risk-adjusted regression, proceeds in three steps. In the first step, we combine all available information in a machine learning model to estimate the value, or inversely, the risk, of taking a certain action, such as approving a loan application or hiring a job candidate. Second, we measure disparities in decisions after adjusting for these risk estimates alone. Finally, in the third step, we assess the sensitivity of results to potential mismeasurement of risk. We demonstrate this approach on a detailed dataset of 2.2 million police stops of pedestrians in New York City, and show that traditional statistical tests of discrimination can substantially understate the magnitude of (risk-adjusted) racial disparities.
References
- 1.Jung, J., Corbett-Davies, S., Gaebler, J. D., Shroff, R., & Goel, S. (2026). Measuring disparate impact in human and machine decisions. Proceedings of the National Academy of Sciences of the United States of America. https://doi.org/10.1073/pnas.2509765122
This content is for educational purposes for healthcare professionals only and does not constitute clinical advice. Clinical decisions should be based on individual patient assessment, current guidelines, and appropriate specialist consultation. Editorial Standards · Privacy Policy · Terms of Service