ResonaVis: Visualizing Interactive Music Data to Support Reflective Music Composition for Therapeutic Contexts
Authors
Abhishek Karwankar (University of Delaware), Elise Ruggiero (University of Delaware), Daniel Stevens (University of Delaware), Matthew Louis Mauriello (University of Delaware)
Presentation
- Session
- Data really is everywhere
- Time
- Tuesday, Nov 10, 15:00 – 15:12 (US/Eastern) · session 15:00 – 16:30
- Location
- Hall Essex north
Links
Sign in to access the preprint PDF.
Sign in- Download Supplemental Material
Keywords
Music; Autism; Software; Prototyping; Interactive interfaces
Abstract
Designing music for therapeutic contexts requires navigating complex relationships between musical structure and listeners' sensory responses, yet composers often lack structured representations of these interactions to inform compositional decisions, relying instead on intuition. We present ResonaVis, an interactive visualization system to help composers analyze interaction and audio data from prior sessions with children with Autism Spectrum Disorder (ASD), informing future compositions. ResonaVis integrates audio features and interaction logs to capture how children engage with layered musical compositions, representing this engagement through coordinated visualizations of temporal transitions, layer co-occurrence, rhythmic activity, and spectral characteristics. Rather than prescribing strategies or directly supporting therapy sessions, the system surfaces patterns in past session data, supporting data-informed reflection during iterative composition. We evaluated ResonaVis through a mixed-methods study with eight music students and a follow-up case study with two experienced composers. Results demonstrate good usability (SUS = 72.23), exceeding benchmarks for early prototypes, and indicate that participants were able to identify interaction patterns, reason about layer relationships, and make informed compositional decisions. Despite moderate cognitive demands, participants reported high perceived performance and low frustration, suggesting productive engagement. Participants' confidence in interpreting interaction and acoustic data when composing for individuals with ASD also increased significantly across multiple dimensions—rhythm and dynamics, pitch and timbre, and spectral characteristics (all p < 0.05). Finally, qualitative findings suggest the potential to shift composers from intuition-driven toward more adaptive, data-informed composition. This work contributes a visualization design space for therapeutic music interaction data, an integrated system for compositional reflection, and empirical evidence that visualization tools can support analytical reasoning and confidence in data-informed creative practices.
For Practitioners
ResonaVis is intended for practitioners who design and reflect on interactive music experiences for therapeutic contexts, particularly composers creating functional music for children's therapy sessions. By visualizing relational, temporal, and acoustic dimensions of past interaction data together, ResonaVis enables composers to identify which musical and structural choices were associated with meaningful engagement patterns in prior sessions, and to carry these insights forward into future compositions. Beyond composers, the system's design goals and encoding strategies (e.g., pairing Sankey diagrams with covariance matrices to relate discrete interaction events to continuous acoustic features) offer a reusable analytical pattern for HCI and visualization practitioners building reflective tools for other time-series-plus-relational-log domains, such as educational technology or accessible instrument design. Music therapists may also draw on findings surfaced through ResonaVis-style analysis to inform session planning.