Matteo Bortoletto

Institute for Visualisation and Interactive Systems, University of Stuttgart

+49 711 685 60147 Pfaffenwaldring 5a, 70569 Stuttgart, Germany University of Stuttgart, SimTech Building, Room 01.027 Google Scholar

Biography

Matteo Bortoletto is a PhD student in the Perceptual User Interfaces group since February 2022. He holds a Bachelor’s degree in Physics and a Master’s degree in Physics of Data from the University of Padua. His research interests include reinforcement learning, multi-agent systems and machine theory of mind.

Open Thesis Projects

 

Learning About Agents Using Graph TransformersBachelor/Master

 

Theory of Mind in Object-context ScenariosBachelor/Master

 

Inferring Other Agents' Goal in Collaborative Environments Using GraphsBachelor

 

Improving Common-Sense Reasoning Tasks with a Cognitive Model of Human Visual AttentionMaster

Teaching

2022

Machine Perception and Learning (Tutor) (summer)Master

 

Machine Perception and Learning (Tutor) (winter)Master

---

Publications

(Zhang et al., 2023; Sood et al., 2023)

1. 

   Exploring Natural Language Processing Methods for Interactive Behaviour Modelling

   Guanhua Zhang, Matteo Bortoletto, Zhiming Hu, Lei Shi, Mihai Bâce, Andreas Bulling

   Proc. IFIP TC13 Conference on Human-Computer Interaction (INTERACT), pp. 1–22, 2023.

   Abstract Links BibTeX Project

   Analysing and modelling interactive behaviour is an important topic in human-computer interaction (HCI) and a key requirement for the development of intelligent interactive systems. Interactive behaviour has a sequential (actions happen one after another) and hierarchical (a sequence of actions forms an activity driven by interaction goals) structure, which may be similar to the structure of natural language. Designed based on such a structure, natural language processing (NLP) methods have achieved groundbreaking success in various downstream tasks. However, few works linked interactive behaviour with natural language. In this paper, we explore the similarity between interactive behaviour and natural language by applying an NLP method, byte pair encoding (BPE), to encode mouse and keyboard behaviour. We then analyse the vocabulary, i.e., the set of action sequences, learnt by BPE, as well as use the vocabulary to encode the input behaviour for interactive task recognition. An existing dataset collected in constrained lab settings and our novel out-of-the-lab dataset were used for evaluation. Results show that this natural language-inspired approach not only learns action sequences that reflect specific interaction goals, but also achieves higher F1 scores on task recognition than other methods. Our work reveals the similarity between interactive behaviour and natural language, and presents the potential of applying the new pack of methods that leverage insights from NLP to model interactive behaviour in HCI.

   Paper: zhang23_interact.pdf

   @inproceedings{zhang23_interact, title = {Exploring Natural Language Processing Methods for Interactive Behaviour Modelling}, author = {Zhang, Guanhua and Bortoletto, Matteo and Hu, Zhiming and Shi, Lei and B{\^a}ce, Mihai and Bulling, Andreas}, booktitle = {Proc. IFIP TC13 Conference on Human-Computer Interaction (INTERACT)}, pages = {1--22}, year = {2023}, publisher = {Springer} }
2. 

   Improving Neural Saliency Prediction with a Cognitive Model of Human Visual Attention

   Ekta Sood, Lei Shi, Matteo Bortoletto, Yao Wang, Philipp Müller, Andreas Bulling

   Proc. the 45th Annual Meeting of the Cognitive Science Society (CogSci), pp. 3639–3646, 2023.

   Abstract Links BibTeX Project

   We present a novel method for saliency prediction that leverages a cognitive model of visual attention as an inductive bias. This approach is in stark contrast to recent purely data-driven saliency models that achieve performance improvements mainly by increased capacity, resulting in high computational costs and the need for large-scale training datasets. We demonstrate that by using a cognitive model, our method achieves competitive performance to the state of the art across several natural image datasets while only requiring a fraction of the parameters. Furthermore, we set the new state of the art for saliency prediction on information visualizations, demonstrating the effectiveness of our approach for cross-domain generalization. We further provide augmented versions of the full MSCOCO dataset with synthetic gaze data using the cognitive model, which we used to pre-train our method. Our results are highly promising and underline the significant potential of bridging between cognitive and data-driven models, potentially also beyond attention.

   Paper: sood23_cogsci.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/neural-saliency-prediction-with-a-cognitive-model/

   Supplementary Material: sood23_cogsci_sup.pdf

   Dataset: https://perceptualui.org/research/datasets/MSCOCOEMMAFigureQAEMMA/

   @inproceedings{sood23_cogsci, author = {Sood, Ekta and Shi, Lei and Bortoletto, Matteo and Wang, Yao and Müller, Philipp and Bulling, Andreas}, title = {Improving Neural Saliency Prediction with a Cognitive Model of Human Visual Attention}, booktitle = {Proc. the 45th Annual Meeting of the Cognitive Science Society (CogSci)}, year = {2023}, pages = {3639--3646} }