VizPilot: Automated Onboarding for SVG-based Composite Visualizations using Multimodal LLMs
Authors
Nishaanthini Gnanavel (Nanyang Technological University), Yong WANG (Nanyang Technological University)
Presentation
- Session
- Kids these days don't learn anything anymore, let's fix that!
- Time
- Thursday, Nov 12, 15:48 – 16:00 (US/Eastern) · session 15:00 – 16:30
- Location
- Hall America north
Links
Sign in to access the preprint PDF.
Sign in
Keywords
Composite Visualization, Visualization Onboarding, Multimodal Large Language Models, SVG
Abstract
Composite visualizations integrate multiple visualizations in a meaningful manner and show strong expressiveness in representing complex dataset. They have been widely proposed by visualization developers in different applications. However, they often impose a high initial cognitive load on novice users, due to their intrinsic composite designs. Existing visualization onboarding approaches have attempted to mitigate this issue, but they are typically platform-dependent, require substantial manual authoring effort, and struggle with the structural complexity of composite visualizations, making them unable to be used for general composite visualizations. In this paper, we present VizPilot, a novel automated visualization onboarding approach that reverse-engineers composite visualization structure to enable an interactive onboarding experience directly from raw visualization artifacts. The proposed approach consists of two primary modules: the Composite Visualization Analyzer and the Onboarding Interface. Leveraging Multimodal Large Language Models (MLLMs), the Analyzer employs a two stage pipeline: The Semantic Inference stage decomposes the visualization into individual visual components and extracts structured knowledge using a five category taxonomy to generate atomic explanation units, and the Semantic Mapping stage anchors these explanations to precise SVG elements via hierarchical DOM traversal, enabling reliable highlighting and interactions. We implement VizPilot as a browser extension. To initiate the workflow, a visualization developer provides a brief description and optional interaction source code and VizPilot then automatically extracts the SVG and generates bitmap image to drive the multimodal reasoning. Then, the interactive Onboarding Interface will show the structured outputs from the Composite Visualization Analyzer, which supports both Narrative Scrollytelling, providing guided, step-by-step explanations with synchronized highlighting, and Free Exploration for on-demand inquiry by users. We extensively evaluate VizPilot through a comparative analysis of different input modalities to assess the pipeline robustness, a usage scenario to illustrate reduced authoring effort, and a user study to confirm its effect in reducing users’ cognitive load. The overall results demonstrate the effectiveness and usability of VizPilot in facilitating an easy onboarding of composite visualizations.
For Practitioners
This paper is intended for visualization developers who design and deploy visualization systems. They can leverage VizPilot to automatically generate interactive onboarding experiences for SVG-based composite visualizations, reducing the manual effort required to create onboarding content. By integrating VizPilot into the visualization development workflow, developers can improve the learnability and accessibility of composite visualizations, enabling users to understand and interact with them more effectively.