“Trust Junk” Leads to Unjustified Support for Highly Discriminatory Predictive Models

Authors

Michael Correll (Northeastern University), Lucy Havens (Northeastern University), Mahsan Nourani (Northeastern University)

Presentation

Session: Can We Trust This Chart? (Asking for a Friend)
Time: Wednesday, Nov 11, 08:27 – 08:36 (US/Eastern) · session 08:00 – 09:30
Location: Hall America south

Keywords

XAI, Data Rhetoric, Information Visualization

Abstract

The persuasive power of data visualizations can go awry: for instance, in an explainable AI (XAI) context, visualizations can produce over-trust of predictive models. In this paper, we use a crowdsourced study to show that providing accurate (but superfluous or irrelevant) data in a model explanation can, in fact, result in unjustified trust and other positive beliefs about a model, even when the model is patently discriminatory and unfair. Our results suggest that XAI designers and developers need to consider the implicit or explicit rhetorics of their work, and beware of the potential of visualizations to imbue models with unearned trust. Supplemental material is available at https://osf.io/wufqz/}{https://osf.io/wufqz/.

For Practitioners

Any practitioner working in deploying and explaining ML models as part of their work. They would learn that more is not always better in explainable AI, and that even the best explanatory techniques don't get them off the hook from thinking about the rhetorical power of data visualization. It's also bad news for people hoping that just sticking a dashboard of SHAP values by their model is sufficient to let people act as auditors for ML models in their organizations.