Yao Wang

Institute for Visualisation and Interactive Systems, University of Stuttgart

+49 711 685 60055 Pfaffenwaldring 5a, 70569 Stuttgart, Germany University of Stuttgart, SimTech Building, Room 01.031 Google Scholar

Biography

Yao Wang is a PhD student in the Perceptual User Interfaces group since September 2020. He holds a Bachelor's degree in Intelligence Software and Technology and a Master’s degree in Computer Software and Technology both from Peking University. He works now on visual attention modelling for optimization of information visualizations, which integrated to Project A07 in SFB-TRR 161. He did a research stay at Aalto University (Finland) in User Interface Group with Prof. Antti Oulasvirta from January to April 2023.

The first research goal is acquiring large-scale humanlike attention data without eye-tracking equipment on information visualizations. He turns to crowdsourcing approaches (webcam or mouse-clicking) or gaze synthetic models. The second research goal is to computationally model human visual behaviour (saliency map, scanpath) under different tasks (top-down) in information visualizations. The third research goal is to develop a task-driven computational model to optimize the information visualisation by maximising metrics (e.g. recallability, gaze uncertainty).

Teaching

2022

Mensch-Computer-Interaktion (Teaching Assistant)Bachelor

 

Machine Perception and Learning (Tutor)Master

2021

Mensch-Computer-Interaktion (Teaching Assistant)Bachelor

 

Machine Learning and Computer Vision for HCI (Fachpraktikum)Master

2020

Machine Learning and Computer Vision for HCI (Fachpraktikum)Master

---

Supervision

2023

Gaze Uncertainty Prediction of Information Visualisations (B.Sc.) Sichun Zeng

 

Analyzing Neural Attention in Chart Question Answering (M.Sc.) Yingpeng Ma§

 

GPT-4-based Visualization Reasoning Dataset (B.Sc.) Jo Hollenbeck

 

Large-scale Information Visualization Saliency Dataset Collection (B.Sc.) Abdullah Abdelhafez (German University of Cairo)**

 

Task-driven Attention Prediction on Information Visualisations (HiWi Project) Weitian Wang

2022

Gaze Uncertainty Prediction and Correction on InfoVis (HiWi Project) Qian-Rong Yi

 

Visual Question Answering through Attention Modelling with Curiosity-driven Reinforcement Learning (M.Sc.) Stoyan Dimitrov*

 

Predicting Recallability from Gaze Behaviour on InfoVis (B.Sc.) Sarra Bakhchali

 

Understanding Gaze Behaviour during Question-Answering on Information Visualisations (HiWi Project) Nuo Chen

2021

VisRecall: Quantifying Information Visualisation Recallability via Question Answering (Erasmus Internship) Chuhan Jiao (Aalto University)

 

Scanpath Modeling on Information Visualisations (HiWi Project) Saiteja Malyala

 

Implementation of Analysis Toolbox for Information Visualisations (HiWi Project) Sruthi Radhakrishnan

 

Multi-view 3D Saliency (M.Sc.) Qi Dai

 

Joint Learning Model for Saliency and Scanpath Prediction (M.Sc.) Jiaqi Qin

 

Multi Duration Saliency Dataset Annotation Toolox (HiWi Project) Luwei Liu

2020

Deep Learning Engineering in Saliency Modeling (HiWi Project) Wei Zhang

*Co-supervised with Ekta Sood
**Co-supervised with Mayar Elfares
§Co-supervised with Susanne Hindennach

---

Open Thesis Projects

 

Saliency Inference AttackBachelor/Master

---

Publications

(Sood et al., 2023; Wang et al., 2023; Wang et al., 2022; Wang et al., 2022; Wei et al., 2020; Chen et al., 2018; Hu et al., 2018)

1. 

   Scanpath Prediction on Information Visualisations

   Yao Wang, Mihai Bâce, Andreas Bulling

   IEEE Transactions on Visualization and Computer Graphics (TVCG), (), pp. 1–15, 2023.

   Abstract Links BibTeX Project

   We propose Unified Model of Saliency and Scanpaths (UMSS) – a model that learns to predict multi-duration saliency and scanpaths (i.e. sequences of eye fixations) on information visualisations. Although scanpaths provide rich information about the importance of different visualisation elements during the visual exploration process, prior work has been limited to predicting aggregated attention statistics, such as visual saliency. We present in-depth analyses of gaze behaviour for different information visualisation elements (e.g. Title, Label, Data) on the popular MASSVIS dataset. We show that while, overall, gaze patterns are surprisingly consistent across visualisations and viewers, there are also structural differences in gaze dynamics for different elements. Informed by our analyses, UMSS first predicts multi-duration element-level saliency maps, then probabilistically samples scanpaths from them. Extensive experiments on MASSVIS show that our method consistently outperforms state-of-the-art methods with respect tto several, widely used scanpath and saliency evaluation metrics. Our method achieves a relative improvement in sequence score of 11.5 % for scanpath prediction, and a relative improvement in Pearson correlation coefficient of up to 23.6 % for saliency prediction. These results are auspicious and point towards richer user models and simulations of visual attention on visualisations without the need for any eye tracking equipment.

   doi: 10.1109/TVCG.2023.3242293

   Paper: wang23_tvcg.pdf

   Supplementary Material: wang23_tvcg_sup.pdf

   Dataset: https://darus.uni-stuttgart.de/dataset.xhtml?persistentId=doi:10.18419/darus-3361

   @article{wang23_tvcg, title = {Scanpath Prediction on Information Visualisations}, author = {Wang, Yao and Bâce, Mihai and Bulling, Andreas}, year = {2023}, journal = {IEEE Transactions on Visualization and Computer Graphics (TVCG)}, volume = {}, number = {}, pages = {1--15}, doi = {10.1109/TVCG.2023.3242293} }

1. 

   Improving Neural Saliency Prediction with a Cognitive Model of Human Visual Attention

   Ekta Sood, Lei Shi, Matteo Bortoletto, Yao Wang, Philipp Müller, Andreas Bulling

   Proc. the 45th Annual Meeting of the Cognitive Science Society (CogSci), pp. 3639–3646, 2023.

   Abstract Links BibTeX Project

   We present a novel method for saliency prediction that leverages a cognitive model of visual attention as an inductive bias. This approach is in stark contrast to recent purely data-driven saliency models that achieve performance improvements mainly by increased capacity, resulting in high computational costs and the need for large-scale training datasets. We demonstrate that by using a cognitive model, our method achieves competitive performance to the state of the art across several natural image datasets while only requiring a fraction of the parameters. Furthermore, we set the new state of the art for saliency prediction on information visualizations, demonstrating the effectiveness of our approach for cross-domain generalization. We further provide augmented versions of the full MSCOCO dataset with synthetic gaze data using the cognitive model, which we used to pre-train our method. Our results are highly promising and underline the significant potential of bridging between cognitive and data-driven models, potentially also beyond attention.

   Paper: sood23_cogsci.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/neural-saliency-prediction-with-a-cognitive-model/

   Supplementary Material: sood23_cogsci_sup.pdf

   Dataset: https://perceptualui.org/research/datasets/MSCOCOEMMAFigureQAEMMA/

   @inproceedings{sood23_cogsci, author = {Sood, Ekta and Shi, Lei and Bortoletto, Matteo and Wang, Yao and Müller, Philipp and Bulling, Andreas}, title = {Improving Neural Saliency Prediction with a Cognitive Model of Human Visual Attention}, booktitle = {Proc. the 45th Annual Meeting of the Cognitive Science Society (CogSci)}, year = {2023}, pages = {3639--3646} }

1. 

   VisRecall: Quantifying Information Visualisation Recallability via Question Answering

   Yao Wang, Chuhan Jiao, Mihai Bâce, Andreas Bulling

   IEEE Transactions on Visualization and Computer Graphics (TVCG), 28(12), pp. 4995-5005, 2022.

   Abstract Links BibTeX Project Oral presentation

   Despite its importance for assessing the effectiveness of communicating information visually, fine-grained recallability of information visualisations has not been studied quantitatively so far. In this work, we propose a question-answering paradigm to study visualisation recallability and present VisRecall - a novel dataset consisting of 200 visualisations that are annotated with crowd-sourced human (N = 305) recallability scores obtained from 1,000 questions of five question types. Furthermore, we present the first computational method to predict recallability of different visualisation elements, such as the title or specific data values. We report detailed analyses of our method on VisRecall and demonstrate that it outperforms several baselines in overall recallability and FE-, F-, RV-, and U-question recallability. Our work makes fundamental contributions towards a new generation of methods to assist designers in optimising visualisations.

   doi: 10.1109/TVCG.2022.3198163

   Paper: wang22_tvcg.pdf

   Supplementary Material: wang22_tvcg_sup.pdf

   Dataset: https://darus.uni-stuttgart.de/dataset.xhtml?persistentId=doi:10.18419/darus-2826

   @article{wang22_tvcg, title = {VisRecall: Quantifying Information Visualisation Recallability via Question Answering}, author = {Wang, Yao and Jiao, Chuhan and Bâce, Mihai and Bulling, Andreas}, year = {2022}, journal = {IEEE Transactions on Visualization and Computer Graphics (TVCG)}, volume = {28}, number = {12}, pages = {4995-5005}, doi = {10.1109/TVCG.2022.3198163} }

1. 

   Impact of Gaze Uncertainty on AOIs in Information Visualisations

   Yao Wang, Maurice Koch, Mihai Bâce, Daniel Weiskopf, Andreas Bulling

   ETRA Workshop on Eye Tracking and Visualization (ETVIS), pp. 1–6, 2022.

   Abstract Links BibTeX Project

   Gaze-based analysis of areas of interest (AOIs) is widely used in information visualisation research to understand how people explore visualisations or assess the quality of visualisations concerning key characteristics such as memorability. However, nearby AOIs in visualisations amplify the uncertainty caused by the gaze estimation error, which strongly influences the mapping between gaze samples or fixations and different AOIs. We contribute a novel investigation into gaze uncertainty and quantify its impact on AOI-based analysis on visualisations using two novel metrics: the Flipping Candidate Rate (FCR) and Hit Any AOI Rate (HAAR). Our analysis of 40 real-world visualisations, including human gaze and AOI annotations, shows that gaze uncertainty frequently and significantly impacts the analysis conducted in AOI-based studies. Moreover, we analysed four visualisation types and found that bar and scatter plots are usually designed in a way that causes more uncertainty than line and pie plots in gaze-based analysis.

   doi: 10.1145/3517031.3531166

   Paper: wang22_etvis.pdf

   Supplementary Material: wang22_etvis_sup.pdf

   @inproceedings{wang22_etvis, title = {Impact of Gaze Uncertainty on AOIs in Information Visualisations}, author = {Wang, Yao and Koch, Maurice and B{\^a}ce, Mihai and Weiskopf, Daniel and Bulling, Andreas}, year = {2022}, pages = {1--6}, booktitle = {ETRA Workshop on Eye Tracking and Visualization (ETVIS)}, doi = {10.1145/3517031.3531166} }

1. 

   Semantic 3D Reconstruction with Learning MVS and 2D Segmentation of Aerial Images

   Zizhuang Wei, Yao Wang, Hongwei Li, Yisong Chen, Guoping Wang

   Applied Sciences, 10(4), pp. 1275, 2020.

   Abstract Links BibTeX Project

   Semantic modeling is a challenging task that has received widespread attention in recent years. With the help of mini Unmanned Aerial Vehicles (UAVs), multi-view high-resolution aerial images of large-scale scenes can be conveniently collected. In this paper, we propose a semantic Multi-View Stereo (MVS) method to reconstruct 3D semantic models from 2D images. Firstly, 2D semantic probability distribution is obtained by Convolutional Neural Network (CNN). Secondly, the calibrated cameras poses are determined by Structure from Motion (SfM), while the depth maps are estimated by learning MVS. Combining 2D segmentation and 3D geometry information, dense point clouds with semantic labels are generated by a probability-based semantic fusion method. In the final stage, the coarse 3D semantic point cloud is optimized by both local and global refinements. By making full use of the multi-view consistency, the proposed method efficiently produces a fine-level 3D semantic point cloud. The experimental result evaluated by re-projection maps achieves 88.4% Pixel Accuracy on the Urban Drone Dataset (UDD). In conclusion, our graph-based semantic fusion procedure and refinement based on local and global information can suppress and reduce the re-projection error.

   doi: 10.3390/app10041275

   Paper: wei20_as.pdf

   @article{wei20_as, title = {Semantic 3D Reconstruction with Learning MVS and 2D Segmentation of Aerial Images}, author = {Wei, Zizhuang and Wang, Yao and Li, Hongwei and Chen, Yisong and Wang, Guoping}, year = {2020}, journal = {Applied Sciences}, doi = {10.3390/app10041275}, pages = {1275}, volume = {10}, number = {4} }

1. 

   Large-Scale Structure from Motion with Semantic Constraints of Aerial Images

   Yu Chen, Yao Wang, Peng Lu, Yisong Chen, Guoping Wang

   Proc. Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pp. 347-359, 2018.

   Abstract Links BibTeX Project

   Structure from Motion (SfM) and semantic segmentation are two branches of computer vision. However, few previous methods integrate the two branches together. SfM is limited by the precision of traditional feature detecting method, especially in complicated scenes. As the research field of semantic segmentation thrives, we could gain semantic information of high confidence in each specific task with little effort. By utilizing semantic segmentation information, our paper presents a new way to boost the accuracy of feature point matching. Besides, with the semantic constraints taken from the result of semantic segmentation, a new bundle adjustment method with equality constraint is proposed. By exploring the sparsity of equality constraint, it indicates that constrained bundle adjustment can be solved by Sequential Quadratic Programming (SQP) efficiently. The proposed approach achieves state of the art accuracy, and, by grouping the descriptors together by their semantic labels, the speed of putative matches is slightly boosted. Moreover, our approach demonstrates a potential of automatic labeling of semantic segmentation. In a nutshell, our work strongly verifies that SfM and semantic segmentation benefit from each other.

   doi: 10.1007/978-3-030-03398-9_30

   Paper: chen18_prcv.pdf

   @inproceedings{chen18_prcv, title = {Large-Scale Structure from Motion with Semantic Constraints of Aerial Images}, author = {Chen, Yu and Wang, Yao and Lu, Peng and Chen, Yisong and Wang, Guoping}, year = {2018}, booktitle = {Proc. Chinese Conference on Pattern Recognition and Computer Vision (PRCV)}, doi = {10.1007/978-3-030-03398-9_30}, pages = {347-359} }
2. 

   Sobel Heuristic Kernel for Aerial Semantic Segmentation

   Tao Hu, Yao Wang, Yisong Chen, Peng Lu, Heng Wang, Guoping Wang

   IEEE International Conference on Image Processing (ICIP), pp. 3074-3078, 2018.

   Abstract Links BibTeX Project

   Misclassification in semantic segmentation mostly occurs in the pixels around the semantic contour. In this work, we address the task of aerial image segmentation by borrowing the kernel prior from classical edge detecting operator. We propose a module called Sobel Heuristic Kernel(SHK). Our work makes several main contributions and experimentally shows good performance. To the best of our knowledge, we are the first to combine traditional edge detection method and deep learning method in semantic segmentation. Our SHK module reaches state of the art in the Inria Aerial Image Labeling dataset.

   doi: 10.1109/ICIP.2018.8451170

   Paper: hu18_icip.pdf

   @inproceedings{hu18_icip, title = {Sobel Heuristic Kernel for Aerial Semantic Segmentation}, author = {Hu, Tao and Wang, Yao and Chen, Yisong and Lu, Peng and Wang, Heng and Wang, Guoping}, year = {2018}, booktitle = {IEEE International Conference on Image Processing (ICIP)}, doi = {10.1109/ICIP.2018.8451170}, pages = {3074-3078} }