SFB-TRR 161 funding period 3 approved

We are excited to announce that the SFB-TRR 161 has been approved by the DFG for a third funding period! Our project A07 | Visual Attention Modeling for Optimization of Information Visualizations launched in September 2020 has so far published 6 conference and journal articles. Stay tuned for the next four years!

Despite the importance of human vision for perceiving and understanding information visualizations, dominant approaches to quantify users’ visual attention require special-purpose eye tracking equipment. However, such equipment is not always available, has to be calibrated to each user, and is limited to post-hoc optimization of information visualizations.

This project aims to integrate automatic quantification of spatio-temporal visual attention directly into the visualization design process without the need for any eye tracking equipment. To achieve these goals, the project takes inspiration from computational models of visual attention (saliency models) that mimic basic perceptual processing to reproduce attentive behavior. Originally introduced in computational neuroscience, saliency models have been tremendously successful in several research fields, particularly in computer vision. In contrast, few works have investigated the use of saliency models in information visualization. We will develop new methods for data-driven attention prediction for information visualizations as well as joint modeling of bottom-up and top-down visual attention, and use them to investigate attention-driven optimization and real-time adaptation of static and dynamic information visualizations.

1. 

   Scanpath Prediction on Information Visualisations

   Yao Wang, Mihai Bâce, Andreas Bulling

   IEEE Transactions on Visualization and Computer Graphics (TVCG), 30 (7), pp. 3902–3914, 2023.

   Abstract Links BibTeX Project

   We propose Unified Model of Saliency and Scanpaths (UMSS) – a model that learns to predict multi-duration saliency and scanpaths (i.e. sequences of eye fixations) on information visualisations. Although scanpaths provide rich information about the importance of different visualisation elements during the visual exploration process, prior work has been limited to predicting aggregated attention statistics, such as visual saliency. We present in-depth analyses of gaze behaviour for different information visualisation elements (e.g. Title, Label, Data) on the popular MASSVIS dataset. We show that while, overall, gaze patterns are surprisingly consistent across visualisations and viewers, there are also structural differences in gaze dynamics for different elements. Informed by our analyses, UMSS first predicts multi-duration element-level saliency maps, then probabilistically samples scanpaths from them. Extensive experiments on MASSVIS show that our method consistently outperforms state-of-the-art methods with respect tto several, widely used scanpath and saliency evaluation metrics. Our method achieves a relative improvement in sequence score of 11.5 % for scanpath prediction, and a relative improvement in Pearson correlation coefficient of up to 23.6 % for saliency prediction. These results are auspicious and point towards richer user models and simulations of visual attention on visualisations without the need for any eye tracking equipment.

   doi: 10.1109/TVCG.2023.3242293

   Paper: wang23_tvcg.pdf

   Supplementary Material: wang23_tvcg_sup.pdf

   Dataset: https://darus.uni-stuttgart.de/dataset.xhtml?persistentId=doi:10.18419/darus-3361

   @article{wang23_tvcg, title = {Scanpath Prediction on Information Visualisations}, author = {Wang, Yao and Bâce, Mihai and Bulling, Andreas}, year = {2023}, journal = {IEEE Transactions on Visualization and Computer Graphics (TVCG)}, volume = {30}, number = {7}, pages = {3902--3914}, doi = {10.1109/TVCG.2023.3242293} }

1. 

   Improving Neural Saliency Prediction with a Cognitive Model of Human Visual Attention

   Ekta Sood, Lei Shi, Matteo Bortoletto, Yao Wang, Philipp Müller, Andreas Bulling

   Proc. the 45th Annual Meeting of the Cognitive Science Society (CogSci), pp. 3639–3646, 2023.

   Abstract Links BibTeX Project

   We present a novel method for saliency prediction that leverages a cognitive model of visual attention as an inductive bias. This approach is in stark contrast to recent purely data-driven saliency models that achieve performance improvements mainly by increased capacity, resulting in high computational costs and the need for large-scale training datasets. We demonstrate that by using a cognitive model, our method achieves competitive performance to the state of the art across several natural image datasets while only requiring a fraction of the parameters. Furthermore, we set the new state of the art for saliency prediction on information visualizations, demonstrating the effectiveness of our approach for cross-domain generalization. We further provide augmented versions of the full MSCOCO dataset with synthetic gaze data using the cognitive model, which we used to pre-train our method. Our results are highly promising and underline the significant potential of bridging between cognitive and data-driven models, potentially also beyond attention.

   Paper: sood23_cogsci.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/neural-saliency-prediction-with-a-cognitive-model/

   Supplementary Material: sood23_cogsci_sup.pdf

   Dataset: https://collaborative-ai.org/research/datasets/MSCOCOEMMAFigureQAEMMA/

   @inproceedings{sood23_cogsci, author = {Sood, Ekta and Shi, Lei and Bortoletto, Matteo and Wang, Yao and Müller, Philipp and Bulling, Andreas}, title = {Improving Neural Saliency Prediction with a Cognitive Model of Human Visual Attention}, booktitle = {Proc. the 45th Annual Meeting of the Cognitive Science Society (CogSci)}, year = {2023}, pages = {3639--3646} }

1. 

   Adapting visualizations and interfaces to the user

   Francesco Chiossi, Johannes Zagermann, Jakob Karolus, Nils Rodrigues, Priscilla Balestrucci, Daniel Weiskopf, Benedikt Ehinger, Tiare Feuchtner, Harald Reiterer, Lewis L. Chuang, Marc Ernst, Andreas Bulling, Sven Mayer, Albrecht Schmidt

   it - Information Technology, 64 (4-5), pp. 133–143, 2022.

   Abstract Links BibTeX Project

   Adaptive visualization and interfaces pervade our everyday tasks to improve interaction from the point of view of user performance and experience. This approach allows using several user inputs, whether physiological, behavioral, qualitative, or multimodal combinations, to enhance the interaction. Due to the multitude of approaches, we outline the current research trends of inputs used to adapt visualizations and user interfaces. Moreover, we discuss methodological approaches used in mixed reality, physiological computing, visual analytics, and proficiency-aware systems. With this work, we provide an overview of the current research in adaptive systems.

   doi: 10.1515/itit-2022-0035

   Paper: chiossi22_it.pdf

   @article{chiossi22_it, title = {Adapting visualizations and interfaces to the user}, author = {Chiossi, Francesco and Zagermann, Johannes and Karolus, Jakob and Rodrigues, Nils and Balestrucci, Priscilla and Weiskopf, Daniel and Ehinger, Benedikt and Feuchtner, Tiare and Reiterer, Harald and Chuang, Lewis L. and Ernst, Marc and Bulling, Andreas and Mayer, Sven and Schmidt, Albrecht}, pages = {133--143}, volume = {64}, number = {4-5}, journal = {it - Information Technology}, doi = {10.1515/itit-2022-0035}, year = {2022} }
2. 

   VisRecall: Quantifying Information Visualisation Recallability via Question Answering

   Yao Wang, Chuhan Jiao, Mihai Bâce, Andreas Bulling

   IEEE Transactions on Visualization and Computer Graphics (TVCG), 28 (12), pp. 4995-5005, 2022.

   Abstract Links BibTeX Project

   Despite its importance for assessing the effectiveness of communicating information visually, fine-grained recallability of information visualisations has not been studied quantitatively so far. In this work, we propose a question-answering paradigm to study visualisation recallability and present VisRecall - a novel dataset consisting of 200 visualisations that are annotated with crowd-sourced human (N = 305) recallability scores obtained from 1,000 questions of five question types. Furthermore, we present the first computational method to predict recallability of different visualisation elements, such as the title or specific data values. We report detailed analyses of our method on VisRecall and demonstrate that it outperforms several baselines in overall recallability and FE-, F-, RV-, and U-question recallability. Our work makes fundamental contributions towards a new generation of methods to assist designers in optimising visualisations.

   doi: 10.1109/TVCG.2022.3198163

   Paper: wang22_tvcg.pdf

   Supplementary Material: wang22_tvcg_sup.pdf

   Dataset: https://darus.uni-stuttgart.de/dataset.xhtml?persistentId=doi:10.18419/darus-2826

   @article{wang22_tvcg, title = {VisRecall: Quantifying Information Visualisation Recallability via Question Answering}, author = {Wang, Yao and Jiao, Chuhan and Bâce, Mihai and Bulling, Andreas}, year = {2022}, journal = {IEEE Transactions on Visualization and Computer Graphics (TVCG)}, volume = {28}, number = {12}, pages = {4995-5005}, doi = {10.1109/TVCG.2022.3198163} }

1. 

   Impact of Gaze Uncertainty on AOIs in Information Visualisations

   Yao Wang, Maurice Koch, Mihai Bâce, Daniel Weiskopf, Andreas Bulling

   ETRA Workshop on Eye Tracking and Visualization (ETVIS), pp. 1–6, 2022.

   Abstract Links BibTeX Project

   Gaze-based analysis of areas of interest (AOIs) is widely used in information visualisation research to understand how people explore visualisations or assess the quality of visualisations concerning key characteristics such as memorability. However, nearby AOIs in visualisations amplify the uncertainty caused by the gaze estimation error, which strongly influences the mapping between gaze samples or fixations and different AOIs. We contribute a novel investigation into gaze uncertainty and quantify its impact on AOI-based analysis on visualisations using two novel metrics: the Flipping Candidate Rate (FCR) and Hit Any AOI Rate (HAAR). Our analysis of 40 real-world visualisations, including human gaze and AOI annotations, shows that gaze uncertainty frequently and significantly impacts the analysis conducted in AOI-based studies. Moreover, we analysed four visualisation types and found that bar and scatter plots are usually designed in a way that causes more uncertainty than line and pie plots in gaze-based analysis.

   doi: 10.1145/3517031.3531166

   Paper: wang22_etvis.pdf

   Supplementary Material: wang22_etvis_sup.pdf

   @inproceedings{wang22_etvis, title = {Impact of Gaze Uncertainty on AOIs in Information Visualisations}, author = {Wang, Yao and Koch, Maurice and B{\^a}ce, Mihai and Weiskopf, Daniel and Bulling, Andreas}, year = {2022}, pages = {1--6}, booktitle = {ETRA Workshop on Eye Tracking and Visualization (ETVIS)}, doi = {10.1145/3517031.3531166} }

1. 

   Visual Analytics and Annotation of Pervasive Eye Tracking Video

   Kuno Kurzhals, Nils Rodrigues, Maurice Koch, Michael Stoll, Andrés Bruhn, Andreas Bulling, Daniel Weiskopf

   Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA), pp. 1-9, 2020.

   Abstract Links BibTeX Project

   We propose a new technique for visual analytics and annotation of long-term pervasive eye tracking data for which a combined analysis of gaze and egocentric video is necessary. Our approach enables two important tasks for such data for hour-long videos from individual participants: (1) efficient annotation and (2) direct interpretation of the results. Exemplary time spans can be selected by the user and are then used as a query that initiates a fuzzy search of similar time spans based on gaze and video features. In an iterative refinement loop, the query interface then provides suggestions for the importance of individual features to improve the search results. A multi-layered timeline visualization shows an overview of annotated time spans. We demonstrate the efficiency of our approach for analyzing activities in about seven hours of video in a case study and discuss feedback on our approach from novices and experts performing the annotation task.

   doi: 10.1145/3379155.3391326

   Paper: kurzhals20_etra.pdf

   @inproceedings{kurzhals20_etra, title = {Visual Analytics and Annotation of Pervasive Eye Tracking Video}, author = {Kurzhals, Kuno and Rodrigues, Nils and Koch, Maurice and Stoll, Michael and Bruhn, Andrés and Bulling, Andreas and Weiskopf, Daniel}, year = {2020}, booktitle = {Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA)}, doi = {10.1145/3379155.3391326}, pages = {1-9} }

Here are some related news you might like to read next:

• DFG proposal accepted
• Andreas Bulling appointed as Humboldt Henriette Herz Scout
• Andreas Bulling among the top 2% of most cited researchers in their field
• Constantin Ruhdorfer receives the Rul Gunzenhäuser Prize
• Andreas Bulling joins the IMPRS-IS executive board