Lei Shi

Institute for Visualisation and Interactive Systems, University of Stuttgart

  Pfaffenwaldring 5a, 70569 Stuttgart, Germany University of Stuttgart, SimTech Building Google Scholar

Biography

Lei Shi is a post-doctoral researcher in the Collaborative Artificial Intelligence group since March 2022. He received his PhD degree from the University of Antwerp, Belgium, in the InViLab research group. He received his master's degree (cum laude) in Mechatronics from the Tallinn University of Technology, Estonia and his bachelor's degree in Electrical Engineering and Automation from Shanghai Maritime University, China. His research interests include eye tracking, intention prediction and graph neural networks.

Teaching

2024

Machine Perception and Learning (Teaching Assistant)Master

2023

Machine Perception and Learning (Teaching Assistant)Master

2022

Machine Perception and Learning (Teaching Assistant)Master

---

Supervision

2024

Long-term action anticipation using latent video diffusion model (Research Project) Yao Tong

 

Multi-modal Graphomer for Action Recognition in Egocentric Videos (M.Sc.) Deniz Bickici

 

Human Hands-Objects-Aware Transformer-based Egocentric Gaze Estimation (Research Project) Obaidah Theeb

 

Gaze-based Transformer for Improving Action Recognition in Egocentric Videos (M.Sc.) Mengze Lu

2023

Inferring Other Agents' Goal in Collaborative Environments Using Graphs (B.Sc.) Ruben Werbke

 

Stationary Vehicle Classification Based on Scene Understanding (M.Sc.) Weitian Wang

---

Open Thesis Projects

 

Multi-modal Graphomer for Action Recognition in Egocentric VideosMaster

---

Publications

(Shi et al., 2025; Bortoletto et al., 2024; Abdessaied et al., 2024; Bortoletto et al., 2024; Bortoletto et al., 2024; Penzkofer et al., 2024; Hindennach et al., 2024; Shi et al., 2024; Shi et al., 2024; Bortoletto et al., 2024; Hindennach et al., 2024; Abdessaied et al., 2024; Sood et al., 2023; Zhang et al., 2023; Shi et al., 2022; Shi et al., 2021; Shi et al., 2021; Shi et al., 2021; Shi et al., 2020; Shi et al., 2020; Shi et al., 2019; Shi et al., 2019; Copot et al., 2019)

1. ActionDiffusion: An Action-aware Diffusion Model for Procedure Planning in Instructional Videos

   Lei Shi, Paul-Christian Bürkner, Andreas Bulling

   Proc. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. , 2025.

   Abstract Links BibTeX Project

   We present ActionDiffusion - a novel diffusion model for procedure planning in instructional videos that is the first to take temporal inter-dependencies between actions into account. Our approach is in stark contrast to existing methods that fail to exploit the rich information content available in the particular order in which actions are performed. Our method unifies the learning of temporal dependencies between actions and denoising of the action plan in the diffusion process by projecting the action information into the noise space. This is achieved 1) by adding action embeddings in the noise masks in the noise-adding phase and 2) by introducing an attention mechanism in the noise prediction network to learn the correlations between different action steps. We report extensive experiments on three instructional video benchmark datasets (CrossTask, Coin, and NIV) and show that our method outperforms previous state-of-the-art methods on all metrics on CrossTask and NIV and all metrics except accuracy on Coin dataset. We show that by adding action embeddings into the noise mask the diffusion model can better learn action temporal dependencies and increase the performances on procedure planning.

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/ActionDiffusion_WACV2025

   @inproceedings{shi25_wacv, author = {Shi, Lei and Bürkner, Paul-Christian and Bulling, Andreas}, title = {ActionDiffusion: An Action-aware Diffusion Model for Procedure Planning in Instructional Videos}, booktitle = {Proc. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)}, year = {2025}, pages = {} }

1. 

   Mindful Explanations: Prevalence and Impact of Mind Attribution in XAI Research

   Susanne Hindennach, Lei Shi, Filip Miletic, Andreas Bulling

   Proc. ACM on Human-Computer Interaction (PACM HCI), 8 (CSCW), pp. 1–42, 2024.

   Abstract Links BibTeX Project Best Paper Honourable Mention Award

   When users perceive AI systems as mindful, independent agents, they hold them responsible instead of the AI experts who created and designed these systems. So far, it has not been studied whether explanations support this shift in responsibility through the use of mind-attributing verbs like "to think". To better understand the prevalence of mind-attributing explanations we analyse AI explanations in 3,533 explainable AI (XAI) research articles from the Semantic Scholar Open Research Corpus (S2ORC). Using methods from semantic shift detection, we identify three dominant types of mind attribution: (1) metaphorical (e.g. "to learn" or "to predict"), (2) awareness (e.g. "to consider"), and (3) agency (e.g. "to make decisions"). We then analyse the impact of mind-attributing explanations on awareness and responsibility in a vignette-based experiment with 199 participants. We find that participants who were given a mind-attributing explanation were more likely to rate the AI system as aware of the harm it caused. Moreover, the mind-attributing explanation had a responsibility-concealing effect: Considering the AI experts’ involvement lead to reduced ratings of AI responsibility for participants who were given a non-mind-attributing or no explanation. In contrast, participants who read the mind-attributing explanation still held the AI system responsible despite considering the AI experts’ involvement. Taken together, our work underlines the need to carefully phrase explanations about AI systems in scientific writing to reduce mind attribution and clearly communicate human responsibility.

   doi: 10.1145/3641009

   Paper: hindennach24_pacm.pdf

   Paper Access: https://medium.com/acm-cscw/be-mindful-when-using-mindful-descriptions-in-explanations-about-ai-bfc7666885c6

   @article{hindennach24_pacm, title = {Mindful Explanations: Prevalence and Impact of Mind Attribution in XAI Research}, author = {Hindennach, Susanne and Shi, Lei and Miletic, Filip and Bulling, Andreas}, year = {2024}, pages = {1--42}, volume = {8}, number = {CSCW}, doi = {10.1145/3641009}, journal = {Proc. ACM on Human-Computer Interaction (PACM HCI)}, url = {https://medium.com/acm-cscw/be-mindful-when-using-mindful-descriptions-in-explanations-about-ai-bfc7666885c6} }

1. 

   Explicit Modelling of Theory of Mind for Belief Prediction in Nonverbal Social Interactions

   Matteo Bortoletto, Constantin Ruhdorfer, Lei Shi, Andreas Bulling

   Proc. 27th European Conference on Artificial Intelligence (ECAI), pp. 866–873, 2024.

   Abstract Links BibTeX Project

   We propose MToMnet - a Theory of Mind (ToM) neural network for predicting beliefs and their dynamics during human social interactions from multimodal input. ToM is key for effective nonverbal human communication and collaboration, yet, existing methods for belief modelling have not included explicit ToM modelling or have typically been limited to one or two modalities. MToMnet encodes contextual cues (scene videos and object locations) and integrates them with person-specific cues (human gaze and body language) in a separate MindNet for each person. Inspired by prior research on social cognition and computational ToM, we propose three different MToMnet variants: two involving fusion of latent representations and one involving re-ranking of classification scores. We evaluate our approach on two challenging real-world datasets, one focusing on belief prediction, while the other examining belief dynamics prediction. Our results demonstrate that MToMnet surpasses existing methods by a large margin while at the same time requiring a significantly smaller number of parameters. Taken together, our method opens up a highly promising direction for future work on artificial intelligent systems that can robustly predict human beliefs from their non-verbal behaviour and, as such, more effectively collaborate with humans.

   doi: 10.3233/FAIA240573

   Paper: bortoletto24_ecai.pdf

   @inproceedings{bortoletto24_ecai, author = {Bortoletto, Matteo and Ruhdorfer, Constantin and Shi, Lei and Bulling, Andreas}, title = {Explicit Modelling of Theory of Mind for Belief Prediction in Nonverbal Social Interactions}, booktitle = {Proc. 27th European Conference on Artificial Intelligence (ECAI)}, year = {2024}, pages = {866--873}, doi = {10.3233/FAIA240573} }
2. 

   Multi-Modal Video Dialog State Tracking in the Wild

   Adnen Abdessaied, Lei Shi, Andreas Bulling

   Proc. 18th European Conference on Computer Vision (ECCV), pp. 1–25, 2024.

   Abstract Links BibTeX Project

   We present MST-MIXER – a novel video dialog model operating over a generic multi-modal state tracking scheme. Current models that claim to perform multi-modal state tracking fall short of two major aspects: (1) They either track only one modality (mostly the visual input) or (2) they target synthetic datasets that do not reflect the complexity of real-world in the wild scenarios. Our model addresses these two limitations in an attempt to close this crucial research gap. Specifically, MST-MIXER first tracks the most important constituents of each input modality. Then, it predicts the missing underlying structure of the selected constituents of each modality by learning local latent graphs using a novel multi-modal graph structure learning method. Subsequently, the learned local graphs and features are parsed together to form a global graph operating on the mix of all modalities which further refines its structure and node embeddings. Finally, the fine-grained graph node features are used to enhance the hidden states of the backbone Vision-Language Model (VLM). MST-MIXER achieves new state-of-the-art results on five challenging benchmarks.

   Paper: abdessaied24_eccv.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/MST-MIXER

   @inproceedings{abdessaied24_eccv, author = {Abdessaied, Adnen and Shi, Lei and Bulling, Andreas}, title = {Multi-Modal Video Dialog State Tracking in the Wild}, booktitle = {Proc. 18th European Conference on Computer Vision (ECCV)}, year = {2024}, pages = {1--25} }
3. 

   Benchmarking Mental State Representations in Language Models

   Matteo Bortoletto, Constantin Ruhdorfer, Lei Shi, Andreas Bulling

   Proc. ICML 2024 Workshop on Mechanistic Interpretability, pp. 1–21, 2024.

   Abstract Links BibTeX Project

   While numerous works have assessed the generative performance of language models (LMs) on tasks requiring Theory of Mind reasoning, research into the models’ internal representation of mental states remains limited. Recent work has used probing to demonstrate that LMs can represent beliefs of themselves and others. However, these claims are accompanied by limited evaluation, making it difficult to assess how mental state representations are affected by model design and training choices. We report an extensive benchmark with various LM types with different model sizes, fine-tuning approaches, and prompt designs to study the robustness of mental state representations and memorisation issues within the probes. Our results show that the quality of models’ internal representations of the beliefs of others increases with model size and, more crucially, with fine-tuning. We are the first to study how prompt variations impact probing performance on theory of mind tasks. We demonstrate that models’ representations are sensitive to prompt variations, even when such variations should be beneficial. Finally, we complement previous activation editing experiments on Theory of Mind tasks and show that it is possible to improve models’ reasoning performance by steering their activations without the need to train any probe.

   doi:

   Paper: bortoletto24_icmlw.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/mental-states-in-LMs

   Paper Access: https://openreview.net/forum?id=yEwEVoH9Be

   @inproceedings{bortoletto24_icmlw, author = {Bortoletto, Matteo and Ruhdorfer, Constantin and Shi, Lei and Bulling, Andreas}, title = {Benchmarking Mental State Representations in Language Models}, booktitle = {Proc. ICML 2024 Workshop on Mechanistic Interpretability}, year = {2024}, pages = {1--21}, doi = {}, url = {https://openreview.net/forum?id=yEwEVoH9Be} }
4. 

   Limits of Theory of Mind Modelling in Dialogue-Based Collaborative Plan Acquisition

   Matteo Bortoletto, Constantin Ruhdorfer, Adnen Abdessaied, Lei Shi, Andreas Bulling

   Proc. 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1–16, 2024.

   Abstract Links BibTeX Project

   Recent work on dialogue-based collaborative plan acquisition (CPA) has suggested that Theory of Mind (ToM) modelling can improve missing knowledge prediction in settings with asymmetric skill-sets and knowledge. Although ToM was claimed to be important for effective collaboration, its real impact on this novel task remains under-explored. By representing plans as graphs and by exploiting task-specific constraints we show that, as performance on CPA nearly doubles when predicting one’s own missing knowledge, the improvements due to ToM modelling diminish. This phenomenon persists even when evaluating existing baseline methods. To better understand the relevance of ToM for CPA, we report a principled performance comparison of models with and without ToM features. Results across different models and ablations consistently suggest that learned ToM features are indeed more likely to reflect latent patterns in the data with no perceivable link to ToM. This finding calls for a deeper understanding of the role of ToM in CPA and beyond, as well as new methods for modelling and evaluating mental states in computational collaborative agents.

   doi:

   Paper: bortoletto24_acl.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/limits-of-tom

   @inproceedings{bortoletto24_acl, author = {Bortoletto, Matteo and Ruhdorfer, Constantin and Abdessaied, Adnen and Shi, Lei and Bulling, Andreas}, title = {Limits of Theory of Mind Modelling in Dialogue-Based Collaborative Plan Acquisition}, booktitle = {Proc. 62nd Annual Meeting of the Association for Computational Linguistics (ACL)}, year = {2024}, pages = {1--16}, doi = {} }
5. 

   VSA4VQA: Scaling A Vector Symbolic Architecture To Visual Question Answering on Natural Images

   Anna Penzkofer, Lei Shi, Andreas Bulling

   Proc. 46th Annual Meeting of the Cognitive Science Society (CogSci), 2024.

   Abstract Links BibTeX Project Oral Presentation

   While Vector Symbolic Architectures (VSAs) are promising for modelling spatial cognition, their application is currently limited to artificially generated images and simple spatial queries. We propose VSA4VQA – a novel 4D implementation of VSAs that implements a mental representation of natural images for the challenging task of Visual Question Answering (VQA). VSA4VQA is the first model to scale a VSA to complex spatial queries. Our method is based on the Semantic Pointer Architecture (SPA) to encode objects in a hyper-dimensional vector space. To encode natural images, we extend the SPA to include dimensions for object’s width and height in addition to their spatial location. To perform spatial queries we further introduce learned spatial query masks and integrate a pre-trained vision-language model for answering attribute-related questions. We evaluate our method on the GQA benchmark dataset and show that it can effectively encode natural images, achieving competitive performance to state-of-the-art deep learning methods for zero-shot VQA.

   Paper: penzkofer24_cogsci.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/VSA4VQA

   Paper Access: https://escholarship.org/uc/item/26j7v1nf.

   @inproceedings{penzkofer24_cogsci, author = {Penzkofer, Anna and Shi, Lei and Bulling, Andreas}, title = {{VSA4VQA}: {Scaling} {A} {Vector} {Symbolic} {Architecture} {To} {Visual} {Question} {Answering} on {Natural} {Images}}, booktitle = {Proc. 46th Annual Meeting of the Cognitive Science Society (CogSci)}, year = {2024}, volume = {46}, url = {https://escholarship.org/uc/item/26j7v1nf.} }
6. 

   Explaining Disagreement in Visual Question Answering Using Eye Tracking

   Susanne Hindennach, Lei Shi, Andreas Bulling

   Proc. International Workshop on Pervasive Eye Tracking and Mobile Gaze-Based Interaction (PETMEI), pp. 1–7, 2024.

   Abstract Links BibTeX Project

   When presented with the same question about an image, human annotators often give valid but disagreeing answers indicating that their reasoning was different. Such differences are lost in a single ground truth label used to train and evaluate visual question answering (VQA) methods. In this work, we explore whether visual attention maps, created using stationary eye tracking, provide insight into the reasoning underlying disagreement in VQA. We first manually inspect attention maps in the recent VQA-MHUG dataset and find cases in which attention differs consistently for disagreeing answers. We further evaluate the suitability of four different similarity metrics to detect attention differences matching the disagreement. We show that attention maps plausibly surface differences in reasoning underlying one type of disagreement, and that the metrics complementarily detect them. Taken together, our results represent an important first step to leverage eye-tracking to explain disagreement in VQA.

   doi: 10.1145/3649902.3656356

   Paper: hindennach24_petmei.pdf

   @inproceedings{hindennach24_petmei, title = {Explaining Disagreement in Visual Question Answering Using Eye Tracking}, author = {Hindennach, Susanne and Shi, Lei and Bulling, Andreas}, year = {2024}, pages = {1--7}, doi = {10.1145/3649902.3656356}, booktitle = {Proc. International Workshop on Pervasive Eye Tracking and Mobile Gaze-Based Interaction (PETMEI)} }
7. 

   Inferring Human Intentions from Predicted Action Probabilities

   Lei Shi, Paul-Christian Bürkner, Andreas Bulling

   Proc. Workshop on Theory of Mind in Human-AI Interaction at CHI 2024, pp. 1–7, 2024.

   Abstract Links BibTeX Project

   Inferring human intentions is a core challenge in human-AI collab-oration but while Bayesian methods struggle with complex visual input, deep neural network (DNN) based methods do not provide uncertainty quantifications. In this work we combine both approaches for the first time and show that the predicted next action probabilities contain information that can be used to infer the underlying user intention. We propose a two-step approach to human intention prediction: While a DNN predicts the probabilities of the next action, MCMC-based Bayesian inference is used to infer the underlying intention from these predictions. This approach not only allows for the independent design of the DNN architecture but also the subsequently fast, design-independent inference of human intentions. We evaluate our method using a series of experiments on the Watch-And-Help (WAH) and a keyboard and mouse interaction dataset. Our results show that our approach can accurately predict human intentions from observed actions and the implicit information contained in next action probabilities. Furthermore, we show that our approach can predict the correct intention even if only a few actions have been observed.

   Paper: shi24_chiw.pdf

   @inproceedings{shi24_chiw, author = {Shi, Lei and Bürkner, Paul-Christian and Bulling, Andreas}, title = {Inferring Human Intentions from Predicted Action Probabilities}, booktitle = {Proc. Workshop on Theory of Mind in Human-AI Interaction at CHI 2024}, year = {2024}, pages = {1--7} }
8. 

   Neural Reasoning About Agents’ Goals, Preferences, and Actions

   Matteo Bortoletto, Lei Shi, Andreas Bulling

   Proc. 38th AAAI Conference on Artificial Intelligence (AAAI), pp. 456–464, 2024.

   Abstract Links BibTeX Project

   We propose the Intuitive Reasoning Network (IRENE) – a novel neural model for intuitive psychological reasoning about agents’ goals, preferences, and actions that can generalise previous experiences to new situations. IRENE combines a graph neural network for learning agent and world state representations with a transformer to encode the task context. When evaluated on the challenging Baby Intuitions Benchmark, IRENE achieves new state-of-the-art performance on three out of its five tasks – with up to 48.9 % improvement. In contrast to existing methods, IRENE is able to bind preferences to specific agents, to better distinguish between rational and irrational agents, and to better understand the role of blocking obstacles. We also investigate, for the first time, the influence of the training tasks on test performance. Our analyses demonstrate the effectiveness of IRENE in combining prior knowledge gained during training for unseen evaluation tasks.

   doi: 10.1609/aaai.v38i1.27800

   Paper: bortoletto24_aaai.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/IRENE

   @inproceedings{bortoletto24_aaai, author = {Bortoletto, Matteo and Shi, Lei and Bulling, Andreas}, title = {Neural Reasoning About Agents’ Goals, Preferences, and Actions}, booktitle = {Proc. 38th AAAI Conference on Artificial Intelligence (AAAI)}, year = {2024}, volume = {38}, number = {1}, pages = {456--464}, doi = {10.1609/aaai.v38i1.27800} }
9. 

   VD-GR: Boosting Visual Dialog with Cascaded Spatial-Temporal Multi-Modal GRaphs

   Adnen Abdessaied, Lei Shi, Andreas Bulling

   Proc. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 5805–5814, 2024.

   Abstract Links BibTeX Project

   We propose VD-GR – a novel visual dialog model that combines pre-trained language models (LMs) with graph neural networks (GNNs). Prior works mainly focused on one class of models at the expense of the other, thus missing out on the opportunity of combining their respective benefits. At the core of VD-GR is a novel integration mechanism that alternates between spatial-temporal multi-modal GNNs and BERT layers, and that covers three distinct contributions: First, we use multi-modal GNNs to process the features of each modality (image, question, and dialog history) and exploit their local structures before performing BERT global attention. Second, we propose hub-nodes that link to all other nodes within one modality graph, allowing the model to propagate information from one GNN (modality) to the other in a cascaded manner. Third, we augment the BERT hidden states with fine-grained multi-modal GNN features before passing them to the next VD-GR layer. Evaluations on VisDial v1.0, VisDial v0.9, VisDialConv, and VisPro show that VD-GR achieves new state-of-the-art results across all four datasets

   Paper: abdessaied24_wacv.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/VDGR

   @inproceedings{abdessaied24_wacv, author = {Abdessaied, Adnen and Shi, Lei and Bulling, Andreas}, title = {VD-GR: Boosting Visual Dialog with Cascaded Spatial-Temporal Multi-Modal GRaphs}, booktitle = {Proc. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)}, year = {2024}, pages = {5805--5814} }

1. 

   ActionDiffusion: An Action-aware Diffusion Model for Procedure Planning in Instructional Videos

   Lei Shi, Paul Burkner, Andreas Bulling

   arXiv:2403.08591, pp. 1–6, 2024.

   Abstract Links BibTeX Project

   We present ActionDiffusion – a novel diffusion model for procedure planning in instructional videos that is the first to take temporal inter-dependencies between actions into account in a diffusion model for procedure planning. This approach is in stark contrast to existing methods that fail to exploit the rich information content available in the particular order in which actions are performed. Our method unifies the learning of temporal dependencies between actions and denoising of the action plan in the diffusion process by projecting the action information into the noise space. This is achieved 1) by adding action embeddings in the noise masks in the noiseadding phase and 2) by introducing an attention mechanism in the noise prediction network to learn the correlations between different action steps. We report extensive experiments on three instructional video benchmark datasets (CrossTask, Coin, and NIV) and show that our method outperforms previous state-of-the-art methods on all metrics on CrossTask and NIV and all metrics except accuracy on Coin dataset. We show that by adding action embeddings into the noise mask the diffusion model can better learn action temporal dependencies and increase the performances on procedure planning

   Paper Access: https://arxiv.org/abs/2403.08591

   @techreport{shi24_arxiv, title = {ActionDiffusion: An Action-aware Diffusion Model for Procedure Planning in Instructional Videos}, author = {Shi, Lei and Burkner, Paul and Bulling, Andreas}, year = {2024}, pages = {1--6}, url = {https://arxiv.org/abs/2403.08591} }

1. 

   Improving Neural Saliency Prediction with a Cognitive Model of Human Visual Attention

   Ekta Sood, Lei Shi, Matteo Bortoletto, Yao Wang, Philipp Müller, Andreas Bulling

   Proc. the 45th Annual Meeting of the Cognitive Science Society (CogSci), pp. 3639–3646, 2023.

   Abstract Links BibTeX Project

   We present a novel method for saliency prediction that leverages a cognitive model of visual attention as an inductive bias. This approach is in stark contrast to recent purely data-driven saliency models that achieve performance improvements mainly by increased capacity, resulting in high computational costs and the need for large-scale training datasets. We demonstrate that by using a cognitive model, our method achieves competitive performance to the state of the art across several natural image datasets while only requiring a fraction of the parameters. Furthermore, we set the new state of the art for saliency prediction on information visualizations, demonstrating the effectiveness of our approach for cross-domain generalization. We further provide augmented versions of the full MSCOCO dataset with synthetic gaze data using the cognitive model, which we used to pre-train our method. Our results are highly promising and underline the significant potential of bridging between cognitive and data-driven models, potentially also beyond attention.

   Paper: sood23_cogsci.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/neural-saliency-prediction-with-a-cognitive-model/

   Supplementary Material: sood23_cogsci_sup.pdf

   Dataset: https://collaborative-ai.org/research/datasets/MSCOCOEMMAFigureQAEMMA/

   @inproceedings{sood23_cogsci, author = {Sood, Ekta and Shi, Lei and Bortoletto, Matteo and Wang, Yao and Müller, Philipp and Bulling, Andreas}, title = {Improving Neural Saliency Prediction with a Cognitive Model of Human Visual Attention}, booktitle = {Proc. the 45th Annual Meeting of the Cognitive Science Society (CogSci)}, year = {2023}, pages = {3639--3646} }
2. 

   Exploring Natural Language Processing Methods for Interactive Behaviour Modelling

   Guanhua Zhang, Matteo Bortoletto, Zhiming Hu, Lei Shi, Mihai Bâce, Andreas Bulling

   Proc. IFIP TC13 Conference on Human-Computer Interaction (INTERACT), pp. 1–22, 2023.

   Abstract Links BibTeX Project Best Student Paper Nomination

   Analysing and modelling interactive behaviour is an important topic in human-computer interaction (HCI) and a key requirement for the development of intelligent interactive systems. Interactive behaviour has a sequential (actions happen one after another) and hierarchical (a sequence of actions forms an activity driven by interaction goals) structure, which may be similar to the structure of natural language. Designed based on such a structure, natural language processing (NLP) methods have achieved groundbreaking success in various downstream tasks. However, few works linked interactive behaviour with natural language. In this paper, we explore the similarity between interactive behaviour and natural language by applying an NLP method, byte pair encoding (BPE), to encode mouse and keyboard behaviour. We then analyse the vocabulary, i.e., the set of action sequences, learnt by BPE, as well as use the vocabulary to encode the input behaviour for interactive task recognition. An existing dataset collected in constrained lab settings and our novel out-of-the-lab dataset were used for evaluation. Results show that this natural language-inspired approach not only learns action sequences that reflect specific interaction goals, but also achieves higher F1 scores on task recognition than other methods. Our work reveals the similarity between interactive behaviour and natural language, and presents the potential of applying the new pack of methods that leverage insights from NLP to model interactive behaviour in HCI.

   Paper: zhang23_interact.pdf

   @inproceedings{zhang23_interact, title = {Exploring Natural Language Processing Methods for Interactive Behaviour Modelling}, author = {Zhang, Guanhua and Bortoletto, Matteo and Hu, Zhiming and Shi, Lei and B{\^a}ce, Mihai and Bulling, Andreas}, booktitle = {Proc. IFIP TC13 Conference on Human-Computer Interaction (INTERACT)}, pages = {1--22}, year = {2023}, publisher = {Springer} }

1. Evaluating Dropout Placements in Bayesian Regression Resnet

   Lei Shi, Cosmin Copot, Steve Vanlanduit

   Journal of Artificial Intelligence and Soft Computing Research, 12 (1), pp. 61–73, 2022.

   Abstract Links BibTeX Project

   Deep Neural Networks (DNNs) have shown great success in many fields. Various network architectures have been developed for different applications. Regardless of the complexities of the networks, DNNs do not provide model uncertainty. Bayesian Neural Networks (BNNs), on the other hand, is able to make probabilistic inference. Among various types of BNNs, Dropout as a Bayesian Approximation converts a Neural Network (NN) to a BNN by adding a dropout layer after each weight layer in the NN. This technique provides a simple transformation from a NN to a BNN. However, for DNNs, adding a dropout layer to each weight layer would lead to a strong regularization due to the deep architecture. Previous researches [1, 2, 3] have shown that adding a dropout layer after each weight layer in a DNN is unnecessary. However, how to place dropout layers in a ResNet for regression tasks are less explored. In this work, we perform an empirical study on how different dropout placements would affect the performance of a Bayesian DNN. We use a regression model modified from ResNet as the DNN and place the dropout layers at different places in the regression ResNet. Our experimental results show that it is not necessary to add a dropout layer after every weight layer in the Regression ResNet to let it be able to make Bayesian Inference. Placing Dropout layers between the stacked blocks i.e. Dense+Identity+Identity blocks has the best performance in Predictive Interval Coverage Probability (PICP). Placing a dropout layer after each stacked block has the best performance in Root Mean Square Error (RMSE).

   doi: 10.2478/JAISCR-2022-0005

   @article{shi22_jaiscr, author = {Shi, Lei and Copot, Cosmin and Vanlanduit, Steve}, title = {Evaluating Dropout Placements in Bayesian Regression Resnet}, journal = {Journal of Artificial Intelligence and Soft Computing Research}, year = {2022}, volume = {12}, number = {1}, pages = {61--73}, doi = {10.2478/JAISCR-2022-0005} }

1. Gaze Gesture Recognition by Graph Convolutional Networks

   Lei Shi, Cosmin Copot, Steve Vanlanduit

   Frontiers in Robotics and AI, 8, 2021.

   Abstract Links BibTeX Project

   Gaze gestures are extensively used in the interactions with agents/computers/robots. Either remote eye tracking devices or head-mounted devices (HMDs) have the advantage of hands-free during the interaction. Previous studies have demonstrated the success of applying machine learning techniques for gaze gesture recognition. More recently, graph neural networks (GNNs) have shown great potential applications in several research areas such as image classification, action recognition, and text classification. However, GNNs are less applied in eye tracking researches. In this work, we propose a graph convolutional network (GCN)–based model for gaze gesture recognition. We train and evaluate the GCN model on the HideMyGaze! dataset. The results show that the accuracy, precision, and recall of the GCN model are 97.62%, 97.18%, and 98.46%, respectively, which are higher than the other compared conventional machine learning algorithms, the artificial neural network (ANN) and the convolutional neural network (CNN).

   doi: 10.3389/frobt.2021.709952

   @article{shi21_frai, author = {Shi, Lei and Copot, Cosmin and Vanlanduit, Steve}, title = {Gaze Gesture Recognition by Graph Convolutional Networks}, journal = {Frontiers in Robotics and AI}, year = {2021}, volume = {8}, paper = {222}, doi = {10.3389/frobt.2021.709952} }

2. GazeEMD: Detecting Visual Intention in Gaze-Based Human-Robot Interaction

   Lei Shi, Cosmin Copot, Steve Vanlanduit

   Robotics, 10 (2), pp. 1–18, 2021.

   Abstract Links BibTeX Project

   In gaze-based Human-Robot Interaction (HRI), it is important to determine human visual intention for interacting with robots. One typical HRI interaction scenario is that a human selects an object by gaze and a robotic manipulator will pick up the object. In this work, we propose an approach, GazeEMD, that can be used to detect whether a human is looking at an object for HRI application. We use Earth Mover’s Distance (EMD) to measure the similarity between the hypothetical gazes at objects and the actual gazes. Then, the similarity score is used to determine if the human visual intention is on the object. We compare our approach with a fixation-based method and HitScan with a run length in the scenario of selecting daily objects by gaze. Our experimental results indicate that the GazeEMD approach has higher accuracy and is more robust to noises than the other approaches. Hence, the users can lessen cognitive load by using our approach in the real-world HRI scenario.

   doi: 10.3390/robotics10020068

   @article{shi21_robotics, author = {Shi, Lei and Copot, Cosmin and Vanlanduit, Steve}, title = {{GazeEMD}: Detecting Visual Intention in Gaze-Based Human-Robot Interaction}, journal = {Robotics}, year = {2021}, volume = {10}, number = {2}, paper = {68}, pages = {1--18}, doi = {10.3390/robotics10020068} }

3. A Bayesian Deep Neural Network for Safe Visual Servoing in Human–Robot Interaction

   Lei Shi, Cosmin Copot, Steve Vanlanduit

   Frontiers in Robotics and AI, 8, pp. 1–13, 2021.

   Abstract Links BibTeX Project

   Safety is an important issue in human–robot interaction (HRI) applications. Various research works have focused on different levels of safety in HRI. If a human/obstacle is detected, a repulsive action can be taken to avoid the collision. Common repulsive actions include distance methods, potential field methods, and safety field methods. Approaches based on machine learning are less explored regarding the selection of the repulsive action. Few research works focus on the uncertainty of the data-based approaches and consider the efficiency of the executing task during collision avoidance. In this study, we describe a system that can avoid collision with human hands while the robot is executing an image-based visual servoing (IBVS) task. We use Monte Carlo dropout (MC dropout) to transform a deep neural network (DNN) to a Bayesian DNN, and learn the repulsive position for hand avoidance. The Bayesian DNN allows IBVS to converge faster than the opposite repulsive pose. Furthermore, it allows the robot to avoid undesired poses that the DNN cannot avoid. The experimental results show that Bayesian DNN has adequate accuracy and can generalize well on unseen data. The predictive interval coverage probability (PICP) of the predictions along x, y, and z directions are 0.84, 0.94, and 0.95, respectively. In the space which is unseen in the training data, the Bayesian DNN is also more robust than a DNN. We further implement the system on a UR10 robot, and test the robustness of the Bayesian DNN and the IBVS convergence speed. Results show that the Bayesian DNN can avoid the poses out of the reach range of the robot and it lets the IBVS task converge faster than the opposite repulsive pose.

   doi: 10.3389/frobt.2021.687031

   @article{shi21_frai_2, author = {Shi, Lei and Copot, Cosmin and Vanlanduit, Steve}, title = {A Bayesian Deep Neural Network for Safe Visual Servoing in Human–Robot Interaction}, journal = {Frontiers in Robotics and AI}, year = {2021}, volume = {8}, paper = {687031}, pages = {1--13}, doi = {10.3389/frobt.2021.687031} }

1. Visual Intention Classification by Deep Learning for Gaze-based Human-Robot Interaction

   Lei Shi, Cosmin Copot, Steve Vanlanduit

   IFAC-PapersOnLine, 53 (5), pp. 750-755, 2020.

   Abstract Links BibTeX Project

   In this work, we propose a deep learning model to classify a human’s visual intention in gaze-based Human-Robot Interaction(HRI). We consider a scenario in which a human wears a pair of eye tracking glasses and can select an object by gaze and a robotic manipulator picks up the object. A neural network is trained as a binary classifier to classify if a human is looking at an object. The network architecture is based on Fully Convolutional Net(FCN), Convolutional Block Attention Modules(CBAM) and Residual Blocks. We evaluate our model with two experiments. In one experiment we test the performance in the scenario where only a single object exists and the other one multiple objects exist. The results show that our proposed network is accurate and it can generalize well. The F1 score on the single object is 0.971 and 0.962 on multiple objects.

   doi: 10.1016/j.ifacol.2021.04.168

   @article{shi20_ifac, author = {Shi, Lei and Copot, Cosmin and Vanlanduit, Steve}, title = {Visual Intention Classification by Deep Learning for Gaze-based Human-Robot Interaction}, journal = {IFAC-PapersOnLine}, year = {2020}, volume = {53}, number = {5}, pages = {750-755}, doi = {10.1016/j.ifacol.2021.04.168} }

1. A Deep Regression Model for Safety Control in Visual Servoing Applications

   Lei Shi, Cosmin Copot, Steve Vanlanduit

   Proc. IEEE International Conference on Robotic Computing (IRC), pp. 360–366, 2020.

   Abstract Links BibTeX Project

   In Human-Robot Interaction scenarios, a human often needs to interact or closely working with objects and/or the robot. Hence the safety aspect needs to be taken care of in the Human-Robot Interaction scenarios. In this paper, we apply a deep learning approach to learning an optimal repulsive pose. The end effector of the robot will move the optimal repulsive pose if the human hand is too close to the end effector. We use a ResNet based deep regression model to learn the weights between the input i.e. the hand position + Tool Center Point position and output i.e. the repulsive pose. We evaluate the model with different readouts and loss functions. With the Fully Connected readout, the Mean absolute Error in the x, y and z directions are between 7.4 mm and 7.7 mm. The model inference time is also smaller than the computation time of calculating the optimal repulsive pose online.

   doi: 10.1109/IRC.2020.00063

   @inproceedings{shi20_irc, author = {Shi, Lei and Copot, Cosmin and Vanlanduit, Steve}, title = {A Deep Regression Model for Safety Control in Visual Servoing Applications}, booktitle = {Proc. IEEE International Conference on Robotic Computing (IRC)}, year = {2020}, pages = {360--366}, doi = {10.1109/IRC.2020.00063} }

1. A Performance Analysis of Invariant Feature Descriptors in Eye Tracking based Human Robot Collaboration

   Lei Shi, Cosmin Copot, Stijn Derammelaere, Steve Vanlanduit

   Proc. International Conference on Control, Automation and Robotics (ICCAR), pp. 256–260, 2019.

   Abstract Links BibTeX Project

   For eye tracking applications in Human Robot Collaboration (HRC), it is essential for the robot to be aware of where the human gaze is located in the scene. Using feature detectors and feature descriptors, the human gaze can be projected to the image from which robot could know where a human is looking at. The motion that occurs during the collaboration may affect the performance of the descriptor. In this paper, we analyse the performance of SIFT, SURF, AKAZE, BRISK and ORB feature descriptor in a real scene for eye tracking in HRC where different variances co-exist. We use a robotic arm and two cameras to test the descriptors instead of directly testing on eye tracking glasses in order that different accelerations can be tested quantitatively. Results show that BRISK, AKAZE and SURF are more favourable considering accuracy, stability and computation time.

   doi: 10.1109/ICCAR.2019.8813478

   @inproceedings{shi19_iccar, author = {Shi, Lei and Copot, Cosmin and Derammelaere, Stijn and Vanlanduit, Steve}, title = {A Performance Analysis of Invariant Feature Descriptors in Eye Tracking based Human Robot Collaboration}, booktitle = {Proc. International Conference on Control, Automation and Robotics (ICCAR)}, year = {2019}, pages = {256--260}, doi = {10.1109/ICCAR.2019.8813478} }

2. Application of Visual Servoing and Eye Tracking Glass in Human Robot Interaction: A case study

   Lei Shi, Cosmin Copot, Steve Vanlanduit

   Proc. International Conference on System Theory, Control and Computing (ICSTCC), pp. 515–520, 2019.

   Abstract Links BibTeX Project

   Human gazes reveal a lot of information about their intention, which could be used for intuitive Human Robot Interaction(HRI). Using gaze as a modality to interact with robots has huge value in increasing freedom for human. It has a strong application potential in different domains. In this paper, we propose a system for eye tracking based HRI with which a human can select an object by his/her gaze and a robot manipulator receives the intended object and waits for the confirmation from the human before it starts the grasping task. We use Image-based Visual Servoing (IBVS) to guide the robot manipulator to perform the grasping automatically which can also bring more robustness to the system. The first hand results indicate that the use of both visual servoing features and eye tracking technique can provide successful results in term of human robot interaction.

   doi: 10.1109/ICSTCC.2019.8886064

   @inproceedings{shi19_icstcc, author = {Shi, Lei and Copot, Cosmin and Vanlanduit, Steve}, title = {Application of Visual Servoing and Eye Tracking Glass in Human Robot Interaction: A case study}, booktitle = {Proc. International Conference on System Theory, Control and Computing (ICSTCC)}, year = {2019}, pages = {515--520}, doi = {10.1109/ICSTCC.2019.8886064} }

3. Automatic Tuning Methodology of Visual Servoing System Using Predictive Approach

   Cosmin Copot, Lei Shi, Steve Vanlanduit

   Proc. IEEE International Conference on Control and Automation (ICCA), pp. 776–781, 2019.

   Abstract Links BibTeX Project

   In this paper, a tuning methodology based on predictive model approach for visual servoing system is investigated. The proposed approach uses features prediction based method to estimate the camera velocity and thus to calculate the optimal control tuning parameter. In order to have a faster convergence and in the same time a desired behaviour of the servoing system, a new control parameter is computed every sampling time. To evaluate the designed tuning strategy, a visual servoing architecture with an eye-in-hand configuration has been considered. The experimental results showed that the proposed tuning algorithm based on prediction model has a stable and convergent behavior when dealing with visual servoing applications. To the knowledge of the authors, the proposed methodology is the first approach which enables an automatic selection of control parameter for the proportional visual control law.

   doi: 10.1109/ICCA.2019.8899522

   @inproceedings{copot19_icca, author = {Copot, Cosmin and Shi, Lei and Vanlanduit, Steve}, title = {Automatic Tuning Methodology of Visual Servoing System Using Predictive Approach}, booktitle = {Proc. IEEE International Conference on Control and Automation (ICCA)}, year = {2019}, pages = {776--781}, doi = {10.1109/ICCA.2019.8899522} }