Less Is More Reflective: Intervention Intensity and Metacognitive Disruption in Visual Analytics
Authors
Mengyu Chen (Emory University), Yixin Bai (University of Maryland), Zheyuan Lin (Emory University), Emily Wall (Emory University)
Presentation
- Session
- Did you see that? Are you sure?
- Time
- Wednesday, Nov 11, 14:12 – 14:24 (US/Eastern) · session 13:00 – 14:30
- Location
- Hall Essex center
Keywords
Metacognition, Metacognitive Awareness, Visual Analytics, Socratic Questioning, Interaction Trace, Reflective Thinking
Abstract
Exploratory data visualization tools often prioritize efficiency over reflective thinking, leaving users vulnerable to cognitive biases and superficial analyses. This capacity for reflective thinking, specifically, the ability to monitor one’s own analytical process, is known as metacognitive awareness, yet it remains largely unsupported in visualization design. In this paper, we investigate two potential interventions for improving metacognitive awareness: Socratic prompting, which triggers reflection through contextual questioning, and visualizing interaction traces, which surface behavioral patterns to support self-monitoring. We conducted a between-subjects study (N=36) comparing contextual Socratic prompts delivered via Wizard-of-Oz and a bias-aware visualization interface that surfaces interaction traces against a control condition. We measured metacognitive awareness before and after completing a visualization exploration task using an adapted Metacognitive Awareness Inventory (MAI). Contrary to our expectations, we observed a significant decrease in MAI scores for participants in the interaction trace group, while the Socratic group and control group showed no significant decline. Our findings reveal how certain design features intended to support awareness may instead lead to cognitive overload, distraction, or superficial engagement, ultimately diminishing it. This work contributes empirical support for the growing body of work on promoting reflection with visualizations, highlighting the importance of evaluating interventions not only for their intended effects but also for their unintended consequences. All Supplemental Materials are available at https://osf.io/bnamf.
For Practitioners
Visual analytics/dashboard designers, UX designers building bias-awareness or self-monitoring features, learning analytics/EdTech designers, and visualization tool developers -- anyone designing feedback mechanisms meant to support user reflection or decision-making. More broadly, this work is a reminder that transparency and feedback features might not automatically help users and they can backfire depending on how persistent or actionable they are. Practitioners designing any kind of user-facing feedback or awareness tool should evaluate not just whether people find a feature useful or usable, but whether it actually changes their thinking and behavior in the intended way, and watch for unintended side effects like distraction or over-reliance on the system's cues.