DaSSWeb | Forks over knives: Predictive inconsistency in criminal justice algorithmic risk assessment tools

Forks over knives: Predictive inconsistency in criminal justice algorithmic risk assessment tools
(joint work with Travis Greene, Ching-Fu Lin, Han-Wei Liu, and Jan Fell)

Speaker
Galit Shmueli
National Tsing Hua University, Taiwan

Abstract
Big data and algorithmic risk prediction tools promise to improve criminal justice systems by reducing human biases and inconsistencies in decision-making. Yet different, equally justifiable choices when developing, testing and deploying these socio-technical tools can lead to disparate predicted risk scores for the same individual. Synthesizing diverse perspectives from machine learning, statistics, sociology, criminology, law, philosophy and economics, we conceptualize this phenomenon as predictive inconsistency. We describe sources of predictive inconsistency at different stages of algorithmic risk assessment tool development and deployment and consider how future technological developments may amplify predictive inconsistency. We argue, however, that in a diverse and pluralistic society we should not expect to completely eliminate predictive inconsistency. Instead, to bolster the legal, political and scientific legitimacy of algorithmic risk prediction tools, we propose identifying and documenting relevant and reasonable ‘forking paths’ to enable quantifiable, reproducible multiverse and specification curve analyses of predictive inconsistency at the individual level.

Short Bio
Galit Shmueli is the Tsing Hua Distinguished Professor at the Institute of Service Science, College of Technology Management, National Tsing Hua University, Taiwan. Earlier she was Associate Professor at University of Maryland’s Smith School of Business, and then the SRITNE Chaired Professor of Data Analytics and Associate Professor of Statistics & Information Systems at the Indian School of Business. Prof. Shmueli’s research focuses on statistical and machine learning methodology with applications in information systems and healthcare, and an emphasis on human behavior. Since her 2010 Statistical Science paper “To Explain or To Predict?” (2000+ citations), she’s been investigating how predictive methodology can enhance causal explanatory goals, and how causal explanatory methodology can enhance predictive goals. Prof. Shmueli authors multiple books, including the popular textbook Data Mining for Business Analytics and has over 100 publications in peer-reviewed journals and books. Prof. Shmueli teaches courses on machine learning, forecasting analytics, interactive visualization, research methods, and other business analytics topics. Her online teaching videos are highly subscribed, and she has won multiple teaching awards. Prof. Shmueli is the inaugural Editor-in-Chief of the INFORMS Journal on Data Science, and has served on editorial boards of top journals in statistics and information systems. She is an IMS Fellow and ISI elected member.

Link Zoom