Beyond One Output: Visualizing and Comparing Distributions of Language Model Generations
Authors
Emily Reif (University of Washington), Claire Y Yang (University of Washington), Jared Hwang (University of Washington), Deniz Nazar (University of Washington), Noah Smith (University of Washington), Jeffrey Heer (University of Washington)
Presentation
- Session
- Modern talking-visually
- Time
- Tuesday, Nov 10, 13:12 – 13:24 (US/Eastern) · session 13:00 – 14:30
- Location
- Hall America center
Links
Sign in to access the preprint PDF.
Sign in- Download Supplemental Material
Keywords
Large language models, Human–AI interaction, Visualization, Distributional visualization.
Abstract
Users typically interact with and evaluate language models via single outputs, where each output is just one sample from a broad distribution of possible completions. This interaction hides distributional structure such as modes, uncommon edge cases, and sensitivity to small prompt changes, leading users to over-generalize from anecdotes when iterating on prompts for open-ended tasks. We conduct a formative study with researchers who use LMs (n = 13) examining when stochasticity matters in practice, how they reason about distributions over language, and where current workflows break down. Informed by this study, we introduce GROVE, an interactive visualization that represents multiple LM generations as overlapping paths through a text graph, revealing shared structure, branching points, and clusters while preserving access to raw outputs. We evaluate across three crowdsourced user studies (N = 47, 44, and 40 participants) targeting complementary distributional tasks. Our results support a hybrid workflow: graph summaries improve structural judgments (e.g., assessing LM output diversity), while direct output inspection remains more effective for detail-oriented tasks.
For Practitioners
This paper would primarily interest AI/ML researchers, prompt engineers, and human-AI interaction designers. Practitioners can apply these insights and the GROVE visualization tool to improve LLM transparency, calibrate trust, and evaluate whether a model is robust by examining output distributions for hidden modes, failure points, and sensitivity rather than relying on misleading single-output samples.