Why Question Answering Modality Matters\in Visualization Studies and Tests
Authors
Anne-Flore Cabouat (Université Paris-Saclay, Inria, Institut Polytechnique de Paris, CNRS, LISN, i3), Lily W. Ge (Northwestern University), Tobias Isenberg (Université Paris-Saclay, CNRS, Inria, LISN), Samuel Huron (Télécom Paris, Institut Polytechnique de Paris), Matthew Kay (Northwestern University), Petra Isenberg (Université Paris-Saclay, CNRS, Inria, LISN)
Links
Sign in to access the preprint PDF.
Sign in- Download Supplemental Material
Keywords
Visualization evaluation, Answer modality
Abstract
In this position paper, we argue that answer modality choices can affect the validity of visualization evaluations. Visualization evaluation studies frequently rely on specific answer modalities such as multiple-choice questions, open-ended numerical estimates, verbal explanations, or spatial interactions. Yet, answer modality is often treated as a neutral reporting mechanism rather than as a factor that may influence reasoning strategies, representational transformations, and ultimately measured performance and the conclusions researchers can draw from their observations. We discuss two broad classes of modality effects on participant performance: changes in participants' task-solving strategies and changes induced by representational transformations between the visual stimulus and the expected response format. We further argue that answer modality determines what evidence about participants' reasoning is preserved for analysis. We ground our reflections in observations from prior empirical work and propose practical guidance for reasoning about answer modality as part of visualization evaluation design.