Yao Wang

Institute for Visualisation and Interactive Systems, University of Stuttgart

+49 711 685 60055 Pfaffenwaldring 5a, 70569 Stuttgart, Germany University of Stuttgart, SimTech Building, Room 01.031 Google Scholar

Biography

Yao Wang is a PhD student in the Perceptual User Interfaces group since September 2020. He holds a Bachelor's degree in Intelligence Software and Technology and a Master’s degree in Computer Software and Technology both from Peking University. He works now on visual attention modelling for optimization of information visualizations, which integrated to Project A07 in SFB-TRR 161. He did a research stay at Aalto University (Finland) in User Interface Group with Prof. Antti Oulasvirta from January to April 2023.

The first research goal is acquiring large-scale humanlike attention data without eye-tracking equipment on information visualizations. He turns to crowdsourcing approaches (webcam or mouse-clicking) or gaze synthetic models. The second research goal is to computationally model human visual behaviour (saliency map, scanpath) under different tasks (top-down) in information visualizations. The third research goal is to develop a task-driven computational model to optimize the information visualisation by maximising metrics (e.g. recallability, gaze uncertainty).

Teaching

2022

Mensch-Computer-Interaktion (Teaching Assistant)Bachelor

 

Machine Perception and Learning (Tutor)Master

2021

Mensch-Computer-Interaktion (Teaching Assistant)Bachelor

 

Machine Learning and Computer Vision for HCI (Fachpraktikum)Master

2020

Machine Learning and Computer Vision for HCI (Fachpraktikum)Master

---

Supervision

2023

Gaze Uncertainty Prediction of Information Visualisations (B.Sc.) Sichun Zeng

 

Analyzing Neural Attention in Chart Question Answering (M.Sc.) Yingpeng Ma§§

 

GPT-4-based Visualization Reasoning Dataset (B.Sc.) Jo Hollenbeck

 

Large-scale Information Visualization Saliency Dataset Collection (B.Sc.) Abdullah Abdelhafez (German University of Cairo)**

 

Task-driven Attention Prediction on Information Visualisations (HiWi Project) Weitian Wang

2022

Gaze Uncertainty Prediction and Correction on InfoVis (HiWi Project) Qian-Rong Yi

 

Visual Question Answering through Attention Modelling with Curiosity-driven Reinforcement Learning (M.Sc.) Stoyan Dimitrov*

 

Predicting Recallability from Gaze Behaviour on InfoVis (B.Sc.) Sarra Bakhchali

 

Understanding Gaze Behaviour during Question-Answering on Information Visualisations (HiWi Project) Nuo Chen

2021

VisRecall: Quantifying Information Visualisation Recallability via Question Answering (Erasmus Internship) Chuhan Jiao (Aalto University)

 

Scanpath Modeling on Information Visualisations (HiWi Project) Saiteja Malyala

 

Implementation of Analysis Toolbox for Information Visualisations (HiWi Project) Sruthi Radhakrishnan

 

Multi-view 3D Saliency (M.Sc.) Qi Dai

 

Joint Learning Model for Saliency and Scanpath Prediction (M.Sc.) Jiaqi Qin

 

Multi Duration Saliency Dataset Annotation Toolox (HiWi Project) Luwei Liu

2020

Deep Learning Engineering in Saliency Modeling (HiWi Project) Wei Zhang

*Co-supervised with Ekta Sood
**Co-supervised with Mayar Elfares
§Co-supervised with Susanne Hindennach
§§Co-supervised with Susanne Hindennach and Ekta Sood

---

Open Thesis Projects

 

Saliency Inference AttackBachelor/Master

---

Publications

(Jiao et al., 2024; Wang et al., 2024; Wang et al., 2024; Wang et al., 2024; Sood et al., 2023; Wang et al., 2023; Wang et al., 2022; Wang et al., 2022; Wei et al., 2020; Chen et al., 2018; Hu et al., 2018)

1. 

   VisRecall++: Analysing and Predicting Visualisation Recallability from Gaze Behaviour

   Yao Wang, Yue Jiang, Zhiming Hu, Constantin Ruhdorfer, Mihai Bâce, Andreas Bulling

   Proc. ACM on Human-Computer Interaction (PACM HCI), 8(ETRA), pp. 1–18, 2024.

   Abstract Links BibTeX Project

   Question answering has recently been proposed as a promising means to assess the recallability of information visualisations. However, prior works are yet to study the link between visually encoding a visualisation in memory and recall performance. To fill this gap, we propose VisRecall++ – a novel 40-participant recallability dataset that contains gaze data on 200 visualisations and five question types, such as identifying the title, and finding extreme values.We measured recallability by asking participants questions after they observed the visualisation for 10 seconds.Our analyses reveal several insights, such as saccade amplitude, number of fixations, and fixation duration significantly differ between high and low recallability groups.Finally, we propose GazeRecallNet – a novel computational method to predict recallability from gaze behaviour that outperforms several baselines on this task.Taken together, our results shed light on assessing recallability from gaze behaviour and inform future work on recallability-based visualisation optimisation.

   doi: 10.1145/3655613

   Paper: wang24_etra.pdf

   Supplementary Material: wang24_etra_sup.pdf

   @article{wang24_etra, title = {VisRecall++: Analysing and Predicting Visualisation Recallability from Gaze Behaviour}, author = {Wang, Yao and Jiang, Yue and Hu, Zhiming and Ruhdorfer, Constantin and Bâce, Mihai and Bulling, Andreas}, year = {2024}, journal = {Proc. ACM on Human-Computer Interaction (PACM HCI)}, pages = {1--18}, volume = {8}, number = {ETRA}, doi = {10.1145/3655613} }

1. 

   Saliency3D: a 3D Saliency Dataset Collected on Screen

   Yao Wang, Qi Dai, Mihai Bâce, Karsten Klein, Andreas Bulling

   Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA), pp. 1–9, 2024.

   Abstract Links BibTeX Project

   While visual saliency has recently been studied in 3D, the experimental setup for collecting 3D saliency data can be expensive and cumbersome. To address this challenge, we propose a novel experimental design that utilizes an eye tracker on a screen to collect 3D saliency data. Our experimental design reduces the cost and complexity of 3D saliency dataset collection. We first collect gaze data on a screen, then we map them to 3D saliency data through perspective transformation. Using this method, we collect a 3D saliency dataset (49,276 fixations) comprising 10 participants looking at sixteen objects. Moreover, we examine the viewing preferences for objects and discuss our findings in this study. Our results indicate potential preferred viewing directions and a correlation between salient features and the variation in viewing directions.

   doi: 10.1145/3649902.3653350

   Paper: wang24_etras.pdf

   @inproceedings{wang24_etras, title = {Saliency3D: a 3D Saliency Dataset Collected on Screen}, author = {Wang, Yao and Dai, Qi and B{\^a}ce, Mihai and Klein, Karsten and Bulling, Andreas}, year = {2024}, pages = {1--9}, booktitle = {Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA)}, doi = {10.1145/3649902.3653350} }
2. 

   SalChartQA: Question-driven Saliency on Information Visualisations

   Yao Wang, Weitian Wang, Abdullah Abdelhafez, Mayar Elfares, Zhiming Hu, Mihai Bâce, Andreas Bulling

   Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 1–14, 2024.

   Abstract Links BibTeX Project

   Understanding the link between visual attention and user’s needs when visually exploring information visualisations is under-explored due to a lack of large and diverse datasets to facilitate these analyses. To fill this gap, we introduce SalChartQA – a novel crowd-sourced dataset that uses the BubbleView interface as a proxy for human gaze and a question-answering (QA) paradigm to induce different information needs in users. SalChartQA contains 74,340 answers to 6,000 questions on 3,000 visualisations. Informed by our analyses demonstrating the tight correlation between the question and visual saliency, we propose the first computational method to predict question-driven saliency on information visualisations. Our method outperforms state-of-the-art saliency models, improving several metrics, such as the correlation coefficient and the Kullback-Leibler divergence. These results show the importance of information needs for shaping attention behaviour and paving the way for new applications, such as task-driven optimisation of visualisations or explainable AI in chart question-answering.

   doi: 10.1145/3613904.3642942

   Paper: wang24_chi.pdf

   Supplementary Material: wang24_chi_sup.pdf

   Dataset: https://darus.uni-stuttgart.de/dataset.xhtml?persistentId=doi:10.18419/darus-3884

   @inproceedings{wang24_chi, title = {SalChartQA: Question-driven Saliency on Information Visualisations}, author = {Wang, Yao and Wang, Weitian and Abdelhafez, Abdullah and Elfares, Mayar and Hu, Zhiming and B{\^a}ce, Mihai and Bulling, Andreas}, year = {2024}, pages = {1--14}, booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)}, doi = {10.1145/3613904.3642942} }

1. 

   DiffGaze: A Diffusion Model for Continuous Gaze Sequence Generation on 360° Images

   Chuhan Jiao, Yao Wang, Guanhua Zhang, Mihai Bâce, Zhiming Hu, Andreas Bulling

   arXiv:2403.17477, pp. 1–13, 2024.

   Abstract Links BibTeX Project

   We present DiffGaze, a novel method for generating realistic and diverse continuous human gaze sequences on 360° images based on a conditional score-based denoising diffusion model. Generating human gaze on 360° images is important for various human-computer interaction and computer graphics applications, e.g. for creating large-scale eye tracking datasets or for realistic animation of virtual humans. However, existing methods are limited to predicting discrete fixation sequences or aggregated saliency maps, thereby neglecting crucial parts of natural gaze behaviour. Our method uses features extracted from 360° images as condition and uses two transformers to model the temporal and spatial dependencies of continuous human gaze. We evaluate DiffGaze on two 360° image benchmarks for gaze sequence generation as well as scanpath prediction and saliency prediction. Our evaluations show that DiffGaze outperforms state-of-the-art methods on all tasks on both benchmarks. We also report a 21-participant user study showing that our method generates gaze sequences that are indistinguishable from real human sequences. Taken together, our evaluations not only demonstrate the effectiveness of DiffGaze but also point towards a new generation of methods that faithfully model the rich spatial and temporal nature of natural human gaze behaviour.

   Paper Access: https://arxiv.org/abs/2403.17477

   @techreport{jiao24_arxiv, title = {DiffGaze: A Diffusion Model for Continuous Gaze Sequence Generation on 360° Images}, author = {Jiao, Chuhan and Wang, Yao and Zhang, Guanhua and B{\^a}ce, Mihai and Hu, Zhiming and Bulling, Andreas}, year = {2024}, pages = {1--13}, url = {https://arxiv.org/abs/2403.17477} }

1. 

   Scanpath Prediction on Information Visualisations

   Yao Wang, Mihai Bâce, Andreas Bulling

   IEEE Transactions on Visualization and Computer Graphics (TVCG), (), pp. 1–15, 2023.

   Abstract Links BibTeX Project

   We propose Unified Model of Saliency and Scanpaths (UMSS) – a model that learns to predict multi-duration saliency and scanpaths (i.e. sequences of eye fixations) on information visualisations. Although scanpaths provide rich information about the importance of different visualisation elements during the visual exploration process, prior work has been limited to predicting aggregated attention statistics, such as visual saliency. We present in-depth analyses of gaze behaviour for different information visualisation elements (e.g. Title, Label, Data) on the popular MASSVIS dataset. We show that while, overall, gaze patterns are surprisingly consistent across visualisations and viewers, there are also structural differences in gaze dynamics for different elements. Informed by our analyses, UMSS first predicts multi-duration element-level saliency maps, then probabilistically samples scanpaths from them. Extensive experiments on MASSVIS show that our method consistently outperforms state-of-the-art methods with respect tto several, widely used scanpath and saliency evaluation metrics. Our method achieves a relative improvement in sequence score of 11.5 % for scanpath prediction, and a relative improvement in Pearson correlation coefficient of up to 23.6 % for saliency prediction. These results are auspicious and point towards richer user models and simulations of visual attention on visualisations without the need for any eye tracking equipment.

   doi: 10.1109/TVCG.2023.3242293

   Paper: wang23_tvcg.pdf

   Supplementary Material: wang23_tvcg_sup.pdf

   Dataset: https://darus.uni-stuttgart.de/dataset.xhtml?persistentId=doi:10.18419/darus-3361

   @article{wang23_tvcg, title = {Scanpath Prediction on Information Visualisations}, author = {Wang, Yao and Bâce, Mihai and Bulling, Andreas}, year = {2023}, journal = {IEEE Transactions on Visualization and Computer Graphics (TVCG)}, volume = {}, number = {}, pages = {1--15}, doi = {10.1109/TVCG.2023.3242293} }

1. 

   Improving Neural Saliency Prediction with a Cognitive Model of Human Visual Attention

   Ekta Sood, Lei Shi, Matteo Bortoletto, Yao Wang, Philipp Müller, Andreas Bulling

   Proc. the 45th Annual Meeting of the Cognitive Science Society (CogSci), pp. 3639–3646, 2023.

   Abstract Links BibTeX Project

   We present a novel method for saliency prediction that leverages a cognitive model of visual attention as an inductive bias. This approach is in stark contrast to recent purely data-driven saliency models that achieve performance improvements mainly by increased capacity, resulting in high computational costs and the need for large-scale training datasets. We demonstrate that by using a cognitive model, our method achieves competitive performance to the state of the art across several natural image datasets while only requiring a fraction of the parameters. Furthermore, we set the new state of the art for saliency prediction on information visualizations, demonstrating the effectiveness of our approach for cross-domain generalization. We further provide augmented versions of the full MSCOCO dataset with synthetic gaze data using the cognitive model, which we used to pre-train our method. Our results are highly promising and underline the significant potential of bridging between cognitive and data-driven models, potentially also beyond attention.

   Paper: sood23_cogsci.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/neural-saliency-prediction-with-a-cognitive-model/

   Supplementary Material: sood23_cogsci_sup.pdf

   Dataset: https://perceptualui.org/research/datasets/MSCOCOEMMAFigureQAEMMA/

   @inproceedings{sood23_cogsci, author = {Sood, Ekta and Shi, Lei and Bortoletto, Matteo and Wang, Yao and Müller, Philipp and Bulling, Andreas}, title = {Improving Neural Saliency Prediction with a Cognitive Model of Human Visual Attention}, booktitle = {Proc. the 45th Annual Meeting of the Cognitive Science Society (CogSci)}, year = {2023}, pages = {3639--3646} }

1. 

   VisRecall: Quantifying Information Visualisation Recallability via Question Answering

   Yao Wang, Chuhan Jiao, Mihai Bâce, Andreas Bulling

   IEEE Transactions on Visualization and Computer Graphics (TVCG), 28(12), pp. 4995-5005, 2022.

   Abstract Links BibTeX Project Oral presentation

   Despite its importance for assessing the effectiveness of communicating information visually, fine-grained recallability of information visualisations has not been studied quantitatively so far. In this work, we propose a question-answering paradigm to study visualisation recallability and present VisRecall - a novel dataset consisting of 200 visualisations that are annotated with crowd-sourced human (N = 305) recallability scores obtained from 1,000 questions of five question types. Furthermore, we present the first computational method to predict recallability of different visualisation elements, such as the title or specific data values. We report detailed analyses of our method on VisRecall and demonstrate that it outperforms several baselines in overall recallability and FE-, F-, RV-, and U-question recallability. Our work makes fundamental contributions towards a new generation of methods to assist designers in optimising visualisations.

   doi: 10.1109/TVCG.2022.3198163

   Paper: wang22_tvcg.pdf

   Supplementary Material: wang22_tvcg_sup.pdf

   Dataset: https://darus.uni-stuttgart.de/dataset.xhtml?persistentId=doi:10.18419/darus-2826

   @article{wang22_tvcg, title = {VisRecall: Quantifying Information Visualisation Recallability via Question Answering}, author = {Wang, Yao and Jiao, Chuhan and Bâce, Mihai and Bulling, Andreas}, year = {2022}, journal = {IEEE Transactions on Visualization and Computer Graphics (TVCG)}, volume = {28}, number = {12}, pages = {4995-5005}, doi = {10.1109/TVCG.2022.3198163} }

1. 

   Impact of Gaze Uncertainty on AOIs in Information Visualisations

   Yao Wang, Maurice Koch, Mihai Bâce, Daniel Weiskopf, Andreas Bulling

   ETRA Workshop on Eye Tracking and Visualization (ETVIS), pp. 1–6, 2022.

   Abstract Links BibTeX Project

   Gaze-based analysis of areas of interest (AOIs) is widely used in information visualisation research to understand how people explore visualisations or assess the quality of visualisations concerning key characteristics such as memorability. However, nearby AOIs in visualisations amplify the uncertainty caused by the gaze estimation error, which strongly influences the mapping between gaze samples or fixations and different AOIs. We contribute a novel investigation into gaze uncertainty and quantify its impact on AOI-based analysis on visualisations using two novel metrics: the Flipping Candidate Rate (FCR) and Hit Any AOI Rate (HAAR). Our analysis of 40 real-world visualisations, including human gaze and AOI annotations, shows that gaze uncertainty frequently and significantly impacts the analysis conducted in AOI-based studies. Moreover, we analysed four visualisation types and found that bar and scatter plots are usually designed in a way that causes more uncertainty than line and pie plots in gaze-based analysis.

   doi: 10.1145/3517031.3531166

   Paper: wang22_etvis.pdf

   Supplementary Material: wang22_etvis_sup.pdf

   @inproceedings{wang22_etvis, title = {Impact of Gaze Uncertainty on AOIs in Information Visualisations}, author = {Wang, Yao and Koch, Maurice and B{\^a}ce, Mihai and Weiskopf, Daniel and Bulling, Andreas}, year = {2022}, pages = {1--6}, booktitle = {ETRA Workshop on Eye Tracking and Visualization (ETVIS)}, doi = {10.1145/3517031.3531166} }

1. 

   Semantic 3D Reconstruction with Learning MVS and 2D Segmentation of Aerial Images

   Zizhuang Wei, Yao Wang, Hongwei Yi, Yisong Chen, Guoping Wang

   Applied Sciences, 10(4), pp. 1275, 2020.

   Abstract Links BibTeX Project

   Semantic modeling is a challenging task that has received widespread attention in recent years. With the help of mini Unmanned Aerial Vehicles (UAVs), multi-view high-resolution aerial images of large-scale scenes can be conveniently collected. In this paper, we propose a semantic Multi-View Stereo (MVS) method to reconstruct 3D semantic models from 2D images. Firstly, 2D semantic probability distribution is obtained by Convolutional Neural Network (CNN). Secondly, the calibrated cameras poses are determined by Structure from Motion (SfM), while the depth maps are estimated by learning MVS. Combining 2D segmentation and 3D geometry information, dense point clouds with semantic labels are generated by a probability-based semantic fusion method. In the final stage, the coarse 3D semantic point cloud is optimized by both local and global refinements. By making full use of the multi-view consistency, the proposed method efficiently produces a fine-level 3D semantic point cloud. The experimental result evaluated by re-projection maps achieves 88.4% Pixel Accuracy on the Urban Drone Dataset (UDD). In conclusion, our graph-based semantic fusion procedure and refinement based on local and global information can suppress and reduce the re-projection error.

   doi: 10.3390/app10041275

   Paper: wei20_as.pdf

   @article{wei20_as, title = {Semantic 3D Reconstruction with Learning MVS and 2D Segmentation of Aerial Images}, author = {Wei, Zizhuang and Wang, Yao and Yi, Hongwei and Chen, Yisong and Wang, Guoping}, year = {2020}, journal = {Applied Sciences}, doi = {10.3390/app10041275}, pages = {1275}, volume = {10}, number = {4} }

1. 

   Large-Scale Structure from Motion with Semantic Constraints of Aerial Images

   Yu Chen, Yao Wang, Peng Lu, Yisong Chen, Guoping Wang

   Proc. Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pp. 347-359, 2018.

   Abstract Links BibTeX Project

   Structure from Motion (SfM) and semantic segmentation are two branches of computer vision. However, few previous methods integrate the two branches together. SfM is limited by the precision of traditional feature detecting method, especially in complicated scenes. As the research field of semantic segmentation thrives, we could gain semantic information of high confidence in each specific task with little effort. By utilizing semantic segmentation information, our paper presents a new way to boost the accuracy of feature point matching. Besides, with the semantic constraints taken from the result of semantic segmentation, a new bundle adjustment method with equality constraint is proposed. By exploring the sparsity of equality constraint, it indicates that constrained bundle adjustment can be solved by Sequential Quadratic Programming (SQP) efficiently. The proposed approach achieves state of the art accuracy, and, by grouping the descriptors together by their semantic labels, the speed of putative matches is slightly boosted. Moreover, our approach demonstrates a potential of automatic labeling of semantic segmentation. In a nutshell, our work strongly verifies that SfM and semantic segmentation benefit from each other.

   doi: 10.1007/978-3-030-03398-9_30

   Paper: chen18_prcv.pdf

   @inproceedings{chen18_prcv, title = {Large-Scale Structure from Motion with Semantic Constraints of Aerial Images}, author = {Chen, Yu and Wang, Yao and Lu, Peng and Chen, Yisong and Wang, Guoping}, year = {2018}, booktitle = {Proc. Chinese Conference on Pattern Recognition and Computer Vision (PRCV)}, doi = {10.1007/978-3-030-03398-9_30}, pages = {347-359} }
2. 

   Sobel Heuristic Kernel for Aerial Semantic Segmentation

   Tao Hu, Yao Wang, Yisong Chen, Peng Lu, Heng Wang, Guoping Wang

   IEEE International Conference on Image Processing (ICIP), pp. 3074-3078, 2018.

   Abstract Links BibTeX Project

   Misclassification in semantic segmentation mostly occurs in the pixels around the semantic contour. In this work, we address the task of aerial image segmentation by borrowing the kernel prior from classical edge detecting operator. We propose a module called Sobel Heuristic Kernel(SHK). Our work makes several main contributions and experimentally shows good performance. To the best of our knowledge, we are the first to combine traditional edge detection method and deep learning method in semantic segmentation. Our SHK module reaches state of the art in the Inria Aerial Image Labeling dataset.

   doi: 10.1109/ICIP.2018.8451170

   Paper: hu18_icip.pdf

   @inproceedings{hu18_icip, title = {Sobel Heuristic Kernel for Aerial Semantic Segmentation}, author = {Hu, Tao and Wang, Yao and Chen, Yisong and Lu, Peng and Wang, Heng and Wang, Guoping}, year = {2018}, booktitle = {IEEE International Conference on Image Processing (ICIP)}, doi = {10.1109/ICIP.2018.8451170}, pages = {3074-3078} }