Inkling: Exploring How Students Design Visualizations with Integrated Sketching, Natural Language, and Code Editing

Authors

Sadie M Fisher (California Polytechnic State University, San Luis Obispo), Austin P Wright (California Polytechnic State University), Ayaan M Kazerouni (California Polytechnic State University)

Abstract

Visualization authors can use a number of tools to design and author data graphics, including those with alternative input modalities. For example, sketching is useful for exploring ideas, natural language for expressing intents that are not easily visually expressible, and direct code editing for precise refinement. Prior work exploring these modalities has attended to one modality at a time. However, a data visualization designer may benefit from the ability to fluidly move between modalities for different stages of the design process. We present Inkling, a prototype multimodal editor for data visualization in which users can combine sketching, natural language instructions, and direct code editing to design visualizations that are bound to their data. We conducted pilot user studies with six visualization novices to gain an initial insight into how they used the three modalities to complete assigned visualization tasks. This work is meant to inform further larger-scale empirical studies into the effectiveness of multimodal visualization editing. Our qualitative analysis of screen recordings and audio transcriptions revealed consistent patterns of usage of the three interaction modes.