Lei Shi

Institute for Visualisation and Interactive Systems, University of Stuttgart

  Pfaffenwaldring 5a, 70569 Stuttgart, Germany University of Stuttgart, SimTech Building Google Scholar

Biography

Lei Shi is a post-doctoral researcher in the Perceptual User Interfaces group since March 2022. He received his PhD degree from the University of Antwerp, Belgium, in the InViLab research group. He received his master's degree (cum laude) in Mechatronics from the Tallinn University of Technology, Estonia and his bachelor's degree in Electrical Engineering and Automation from Shanghai Maritime University, China. His research interests include eye tracking, intention prediction and graph neural networks.

Teaching

2022

Machine Perception and Learning (Tutor) (summer)Master

 

Machine Perception and Learning (Teaching Assistant) (winter)Master

Open Thesis Projects

 

Evaluating Embedding Methods of Action in Goal Prediction TaskBachelor/Master

 

Next Question Prediction in Goal-oriented Visual Question Answering by a Theory of Mind modelBachelor/Master

 

Inferring Other Agents' Goal in Collaborative Environments Using GraphsBachelor

Publications

1. 

   Improving Neural Saliency Prediction with a Cognitive Model of Human Visual Attention

   Ekta Sood, Lei Shi, Matteo Bortoletto, Yao Wang, Philipp Müller, Andreas Bulling

   Proc. the 45th Annual Meeting of the Cognitive Science Society (CogSci), pp. 3639–3646, 2023.

   Abstract Links BibTeX Project

   We present a novel method for saliency prediction that leverages a cognitive model of visual attention as an inductive bias. This approach is in stark contrast to recent purely data-driven saliency models that achieve performance improvements mainly by increased capacity, resulting in high computational costs and the need for large-scale training datasets. We demonstrate that by using a cognitive model, our method achieves competitive performance to the state of the art across several natural image datasets while only requiring a fraction of the parameters. Furthermore, we set the new state of the art for saliency prediction on information visualizations, demonstrating the effectiveness of our approach for cross-domain generalization. We further provide augmented versions of the full MSCOCO dataset with synthetic gaze data using the cognitive model, which we used to pre-train our method. Our results are highly promising and underline the significant potential of bridging between cognitive and data-driven models, potentially also beyond attention.

   Paper: sood23_cogsci.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/neural-saliency-prediction-with-a-cognitive-model/

   Supplementary Material: sood23_cogsci_sup.pdf

   Dataset: https://perceptualui.org/research/datasets/MSCOCOEMMAFigureQAEMMA/

   @inproceedings{sood23_cogsci, author = {Sood, Ekta and Shi, Lei and Bortoletto, Matteo and Wang, Yao and Müller, Philipp and Bulling, Andreas}, title = {Improving Neural Saliency Prediction with a Cognitive Model of Human Visual Attention}, booktitle = {Proc. the 45th Annual Meeting of the Cognitive Science Society (CogSci)}, year = {2023}, pages = {3639--3646} }
2. 

   Exploring Natural Language Processing Methods for Interactive Behaviour Modelling

   Guanhua Zhang, Matteo Bortoletto, Zhiming Hu, Lei Shi, Mihai Bâce, Andreas Bulling

   Proc. IFIP TC13 Conference on Human-Computer Interaction (INTERACT), pp. 1–22, 2023.

   Abstract Links BibTeX Project

   Analysing and modelling interactive behaviour is an important topic in human-computer interaction (HCI) and a key requirement for the development of intelligent interactive systems. Interactive behaviour has a sequential (actions happen one after another) and hierarchical (a sequence of actions forms an activity driven by interaction goals) structure, which may be similar to the structure of natural language. Designed based on such a structure, natural language processing (NLP) methods have achieved groundbreaking success in various downstream tasks. However, few works linked interactive behaviour with natural language. In this paper, we explore the similarity between interactive behaviour and natural language by applying an NLP method, byte pair encoding (BPE), to encode mouse and keyboard behaviour. We then analyse the vocabulary, i.e., the set of action sequences, learnt by BPE, as well as use the vocabulary to encode the input behaviour for interactive task recognition. An existing dataset collected in constrained lab settings and our novel out-of-the-lab dataset were used for evaluation. Results show that this natural language-inspired approach not only learns action sequences that reflect specific interaction goals, but also achieves higher F1 scores on task recognition than other methods. Our work reveals the similarity between interactive behaviour and natural language, and presents the potential of applying the new pack of methods that leverage insights from NLP to model interactive behaviour in HCI.

   Paper: zhang23_interact.pdf

   @inproceedings{zhang23_interact, title = {Exploring Natural Language Processing Methods for Interactive Behaviour Modelling}, author = {Zhang, Guanhua and Bortoletto, Matteo and Hu, Zhiming and Shi, Lei and B{\^a}ce, Mihai and Bulling, Andreas}, booktitle = {Proc. IFIP TC13 Conference on Human-Computer Interaction (INTERACT)}, pages = {1--22}, year = {2023}, publisher = {Springer} }

1. Evaluating Dropout Placements in Bayesian Regression Resnet

   Lei Shi, Cosmin Copot, Steve Vanlanduit

   Journal of Artificial Intelligence and Soft Computing Research, 12(1), pp. 61–73, 2022.

   Abstract Links BibTeX Project

   Deep Neural Networks (DNNs) have shown great success in many fields. Various network architectures have been developed for different applications. Regardless of the complexities of the networks, DNNs do not provide model uncertainty. Bayesian Neural Networks (BNNs), on the other hand, is able to make probabilistic inference. Among various types of BNNs, Dropout as a Bayesian Approximation converts a Neural Network (NN) to a BNN by adding a dropout layer after each weight layer in the NN. This technique provides a simple transformation from a NN to a BNN. However, for DNNs, adding a dropout layer to each weight layer would lead to a strong regularization due to the deep architecture. Previous researches [1, 2, 3] have shown that adding a dropout layer after each weight layer in a DNN is unnecessary. However, how to place dropout layers in a ResNet for regression tasks are less explored. In this work, we perform an empirical study on how different dropout placements would affect the performance of a Bayesian DNN. We use a regression model modified from ResNet as the DNN and place the dropout layers at different places in the regression ResNet. Our experimental results show that it is not necessary to add a dropout layer after every weight layer in the Regression ResNet to let it be able to make Bayesian Inference. Placing Dropout layers between the stacked blocks i.e. Dense+Identity+Identity blocks has the best performance in Predictive Interval Coverage Probability (PICP). Placing a dropout layer after each stacked block has the best performance in Root Mean Square Error (RMSE).

   doi: 10.2478/JAISCR-2022-0005

   @article{shi22_jaiscr, author = {Shi, Lei and Copot, Cosmin and Vanlanduit, Steve}, title = {Evaluating Dropout Placements in Bayesian Regression Resnet}, journal = {Journal of Artificial Intelligence and Soft Computing Research}, year = {2022}, volume = {12}, number = {1}, pages = {61--73}, doi = {10.2478/JAISCR-2022-0005} }

1. Gaze Gesture Recognition by Graph Convolutional Networks

   Lei Shi, Cosmin Copot, Steve Vanlanduit

   Frontiers in Robotics and AI, 8, 2021.

   Abstract Links BibTeX Project

   Gaze gestures are extensively used in the interactions with agents/computers/robots. Either remote eye tracking devices or head-mounted devices (HMDs) have the advantage of hands-free during the interaction. Previous studies have demonstrated the success of applying machine learning techniques for gaze gesture recognition. More recently, graph neural networks (GNNs) have shown great potential applications in several research areas such as image classification, action recognition, and text classification. However, GNNs are less applied in eye tracking researches. In this work, we propose a graph convolutional network (GCN)–based model for gaze gesture recognition. We train and evaluate the GCN model on the HideMyGaze! dataset. The results show that the accuracy, precision, and recall of the GCN model are 97.62%, 97.18%, and 98.46%, respectively, which are higher than the other compared conventional machine learning algorithms, the artificial neural network (ANN) and the convolutional neural network (CNN).

   doi: 10.3389/frobt.2021.709952

   @article{shi21_frai, author = {Shi, Lei and Copot, Cosmin and Vanlanduit, Steve}, title = {Gaze Gesture Recognition by Graph Convolutional Networks}, journal = {Frontiers in Robotics and AI}, year = {2021}, volume = {8}, paper = {222}, doi = {10.3389/frobt.2021.709952} }

2. GazeEMD: Detecting Visual Intention in Gaze-Based Human-Robot Interaction

   Lei Shi, Cosmin Copot, Steve Vanlanduit

   Robotics, 10(2), pp. 1–18, 2021.

   Abstract Links BibTeX Project

   In gaze-based Human-Robot Interaction (HRI), it is important to determine human visual intention for interacting with robots. One typical HRI interaction scenario is that a human selects an object by gaze and a robotic manipulator will pick up the object. In this work, we propose an approach, GazeEMD, that can be used to detect whether a human is looking at an object for HRI application. We use Earth Mover’s Distance (EMD) to measure the similarity between the hypothetical gazes at objects and the actual gazes. Then, the similarity score is used to determine if the human visual intention is on the object. We compare our approach with a fixation-based method and HitScan with a run length in the scenario of selecting daily objects by gaze. Our experimental results indicate that the GazeEMD approach has higher accuracy and is more robust to noises than the other approaches. Hence, the users can lessen cognitive load by using our approach in the real-world HRI scenario.

   doi: 10.3390/robotics10020068

   @article{shi21_robotics, author = {Shi, Lei and Copot, Cosmin and Vanlanduit, Steve}, title = {{GazeEMD}: Detecting Visual Intention in Gaze-Based Human-Robot Interaction}, journal = {Robotics}, year = {2021}, volume = {10}, number = {2}, paper = {68}, pages = {1--18}, doi = {10.3390/robotics10020068} }

3. A Bayesian Deep Neural Network for Safe Visual Servoing in Human–Robot Interaction

   Lei Shi, Cosmin Copot, Steve Vanlanduit

   Frontiers in Robotics and AI, 8, pp. 1–13, 2021.

   Abstract Links BibTeX Project

   Safety is an important issue in human–robot interaction (HRI) applications. Various research works have focused on different levels of safety in HRI. If a human/obstacle is detected, a repulsive action can be taken to avoid the collision. Common repulsive actions include distance methods, potential field methods, and safety field methods. Approaches based on machine learning are less explored regarding the selection of the repulsive action. Few research works focus on the uncertainty of the data-based approaches and consider the efficiency of the executing task during collision avoidance. In this study, we describe a system that can avoid collision with human hands while the robot is executing an image-based visual servoing (IBVS) task. We use Monte Carlo dropout (MC dropout) to transform a deep neural network (DNN) to a Bayesian DNN, and learn the repulsive position for hand avoidance. The Bayesian DNN allows IBVS to converge faster than the opposite repulsive pose. Furthermore, it allows the robot to avoid undesired poses that the DNN cannot avoid. The experimental results show that Bayesian DNN has adequate accuracy and can generalize well on unseen data. The predictive interval coverage probability (PICP) of the predictions along x, y, and z directions are 0.84, 0.94, and 0.95, respectively. In the space which is unseen in the training data, the Bayesian DNN is also more robust than a DNN. We further implement the system on a UR10 robot, and test the robustness of the Bayesian DNN and the IBVS convergence speed. Results show that the Bayesian DNN can avoid the poses out of the reach range of the robot and it lets the IBVS task converge faster than the opposite repulsive pose.

   doi: 10.3389/frobt.2021.687031

   @article{shi21_frai_2, author = {Shi, Lei and Copot, Cosmin and Vanlanduit, Steve}, title = {A Bayesian Deep Neural Network for Safe Visual Servoing in Human–Robot Interaction}, journal = {Frontiers in Robotics and AI}, year = {2021}, volume = {8}, paper = {687031}, pages = {1--13}, doi = {10.3389/frobt.2021.687031} }

1. Visual Intention Classification by Deep Learning for Gaze-based Human-Robot Interaction

   Lei Shi, Cosmin Copot, Steve Vanlanduit

   IFAC-PapersOnLine, 53(5), pp. 750-755, 2020.

   Abstract Links BibTeX Project

   In this work, we propose a deep learning model to classify a human’s visual intention in gaze-based Human-Robot Interaction(HRI). We consider a scenario in which a human wears a pair of eye tracking glasses and can select an object by gaze and a robotic manipulator picks up the object. A neural network is trained as a binary classifier to classify if a human is looking at an object. The network architecture is based on Fully Convolutional Net(FCN), Convolutional Block Attention Modules(CBAM) and Residual Blocks. We evaluate our model with two experiments. In one experiment we test the performance in the scenario where only a single object exists and the other one multiple objects exist. The results show that our proposed network is accurate and it can generalize well. The F1 score on the single object is 0.971 and 0.962 on multiple objects.

   doi: 10.1016/j.ifacol.2021.04.168

   @article{shi20_ifac, author = {Shi, Lei and Copot, Cosmin and Vanlanduit, Steve}, title = {Visual Intention Classification by Deep Learning for Gaze-based Human-Robot Interaction}, journal = {IFAC-PapersOnLine}, year = {2020}, volume = {53}, number = {5}, pages = {750-755}, doi = {10.1016/j.ifacol.2021.04.168} }

1. A Deep Regression Model for Safety Control in Visual Servoing Applications

   Lei Shi, Cosmin Copot, Steve Vanlanduit

   Proc. IEEE International Conference on Robotic Computing (IRC), pp. 360–366, 2020.

   Abstract Links BibTeX Project

   In Human-Robot Interaction scenarios, a human often needs to interact or closely working with objects and/or the robot. Hence the safety aspect needs to be taken care of in the Human-Robot Interaction scenarios. In this paper, we apply a deep learning approach to learning an optimal repulsive pose. The end effector of the robot will move the optimal repulsive pose if the human hand is too close to the end effector. We use a ResNet based deep regression model to learn the weights between the input i.e. the hand position + Tool Center Point position and output i.e. the repulsive pose. We evaluate the model with different readouts and loss functions. With the Fully Connected readout, the Mean absolute Error in the x, y and z directions are between 7.4 mm and 7.7 mm. The model inference time is also smaller than the computation time of calculating the optimal repulsive pose online.

   doi: 10.1109/IRC.2020.00063

   @inproceedings{shi20_irc, author = {Shi, Lei and Copot, Cosmin and Vanlanduit, Steve}, title = {A Deep Regression Model for Safety Control in Visual Servoing Applications}, booktitle = {Proc. IEEE International Conference on Robotic Computing (IRC)}, year = {2020}, pages = {360--366}, doi = {10.1109/IRC.2020.00063} }

1. A Performance Analysis of Invariant Feature Descriptors in Eye Tracking based Human Robot Collaboration

   Lei Shi, Cosmin Copot, Stijn Derammelaere, Steve Vanlanduit

   Proc. International Conference on Control, Automation and Robotics (ICCAR), pp. 256–260, 2019.

   Abstract Links BibTeX Project

   For eye tracking applications in Human Robot Collaboration (HRC), it is essential for the robot to be aware of where the human gaze is located in the scene. Using feature detectors and feature descriptors, the human gaze can be projected to the image from which robot could know where a human is looking at. The motion that occurs during the collaboration may affect the performance of the descriptor. In this paper, we analyse the performance of SIFT, SURF, AKAZE, BRISK and ORB feature descriptor in a real scene for eye tracking in HRC where different variances co-exist. We use a robotic arm and two cameras to test the descriptors instead of directly testing on eye tracking glasses in order that different accelerations can be tested quantitatively. Results show that BRISK, AKAZE and SURF are more favourable considering accuracy, stability and computation time.

   doi: 10.1109/ICCAR.2019.8813478

   @inproceedings{shi19_iccar, author = {Shi, Lei and Copot, Cosmin and Derammelaere, Stijn and Vanlanduit, Steve}, title = {A Performance Analysis of Invariant Feature Descriptors in Eye Tracking based Human Robot Collaboration}, booktitle = {Proc. International Conference on Control, Automation and Robotics (ICCAR)}, year = {2019}, pages = {256--260}, doi = {10.1109/ICCAR.2019.8813478} }

2. Application of Visual Servoing and Eye Tracking Glass in Human Robot Interaction: A case study

   Lei Shi, Cosmin Copot, Steve Vanlanduit

   Proc. International Conference on System Theory, Control and Computing (ICSTCC), pp. 515–520, 2019.

   Abstract Links BibTeX Project

   Human gazes reveal a lot of information about their intention, which could be used for intuitive Human Robot Interaction(HRI). Using gaze as a modality to interact with robots has huge value in increasing freedom for human. It has a strong application potential in different domains. In this paper, we propose a system for eye tracking based HRI with which a human can select an object by his/her gaze and a robot manipulator receives the intended object and waits for the confirmation from the human before it starts the grasping task. We use Image-based Visual Servoing (IBVS) to guide the robot manipulator to perform the grasping automatically which can also bring more robustness to the system. The first hand results indicate that the use of both visual servoing features and eye tracking technique can provide successful results in term of human robot interaction.

   doi: 10.1109/ICSTCC.2019.8886064

   @inproceedings{shi19_icstcc, author = {Shi, Lei and Copot, Cosmin and Vanlanduit, Steve}, title = {Application of Visual Servoing and Eye Tracking Glass in Human Robot Interaction: A case study}, booktitle = {Proc. International Conference on System Theory, Control and Computing (ICSTCC)}, year = {2019}, pages = {515--520}, doi = {10.1109/ICSTCC.2019.8886064} }

3. Automatic Tuning Methodology of Visual Servoing System Using Predictive Approach

   Cosmin Copot, Lei Shi, Steve Vanlanduit

   Proc. IEEE International Conference on Control and Automation (ICCA), pp. 776–781, 2019.

   Abstract Links BibTeX Project

   In this paper, a tuning methodology based on predictive model approach for visual servoing system is investigated. The proposed approach uses features prediction based method to estimate the camera velocity and thus to calculate the optimal control tuning parameter. In order to have a faster convergence and in the same time a desired behaviour of the servoing system, a new control parameter is computed every sampling time. To evaluate the designed tuning strategy, a visual servoing architecture with an eye-in-hand configuration has been considered. The experimental results showed that the proposed tuning algorithm based on prediction model has a stable and convergent behavior when dealing with visual servoing applications. To the knowledge of the authors, the proposed methodology is the first approach which enables an automatic selection of control parameter for the proportional visual control law.

   doi: 10.1109/ICCA.2019.8899522

   @inproceedings{copot19_icca, author = {Copot, Cosmin and Shi, Lei and Vanlanduit, Steve}, title = {Automatic Tuning Methodology of Visual Servoing System Using Predictive Approach}, booktitle = {Proc. IEEE International Conference on Control and Automation (ICCA)}, year = {2019}, pages = {776--781}, doi = {10.1109/ICCA.2019.8899522} }