Florian Strohm

Institute for Visualisation and Interactive Systems, University of Stuttgart

+49 711 685 61723 Pfaffenwaldring 5a, 70569 Stuttgart, Germany University of Stuttgart, SimTech Building, Room 01.031 Google Scholar

Biography

Florian Strohm is a PhD student in the Perceptual User Interfaces group since January 2020. He holds a Bachelor's and Master’s degree in Computer Science from the University of Stuttgart. During his Master’s studies he focused on the fields of Computer Vision and Machine Learning, and he won the Rul Gunzenhäuser Prize for his master thesis. His research interests are analyzing cognitive processes such as how humans encode visual stimuli in memory and the synthetic reconstruction of such stimuli.

Teaching

2022

Machine Perception and Learning (Tutor) (summer)Master

 

Machine Perception and Learning (Tutor) (winter)Master

2021

Mensch-Computer-Interaktion (Teaching Assistant)Bachelor

 

Machine Learning and Computer Vision for HCI (Fachpraktikum)Master

2020

Machine Learning and Computer Vision for HCI (Fachpraktikum)Master

 

Mensch-Computer-Interaktion (Teaching Assistant)Bachelor

2018

Information Retrieval and Text Mining (Teaching Assistant)Bachelor/Master

2017

Rechnerorganisation (Tutor)Bachelor

---

Supervision

2023

Web Interface for Interactive Face Reconstruction (B.Sc.) Lewin Fritzenschaft

 

Disentangled Face Embeddings (B.Sc.) Michael Linder

2022

Video-based Representation Learning for Emotion Recognition (M.Sc.) Shangrui Nie**

 

Interpreting Neural Attention in NLP with Human Visual Attention (M.Sc. Research Project) Shangrui Nie**

 

Conditional GAN-based Mask Guided Portrait Editing (M.Sc. Research Project) Prabhakar Panday

 

Augmenting fMRI-based Image Reconstruction with Image Labelling (M.Sc.) Jamie Ullerich

 

Intention Recognition via Eye Gaze for Hierarchical Imitation Learning (M.Sc.) Anna Penzkofer

 

Fine-Grained Facial Composite Generation using Facial Landmarks (B.Sc.) Markus Kaltenecker

2021

Image Reconstruction from Human Brain Activity by Variational Autoencoder and Adversarial Learning (M.Sc.) Mariia Podguzova**

 

Analysing the Influence of Facial Feature Alteration on Person Identification (B.Sc.) Britta Schulz

*Co-supervised with Ekta Sood

---

Open Thesis Projects

 

Mental Face Reconstruction by Interactive MorphingMaster

 

Fixation Importance PredictionMaster

---

Publications

(Strohm & Klinger, 2018; Strohm et al., 2021; Sood et al., 2021; Strohm et al., 2022; Penzkofer et al., 2023)

1. 

   Int-HRL: Towards Intention-based Hierarchical Reinforcement Learning

   Anna Penzkofer, Simon Schaefer, Florian Strohm, Mihai Bâce, Stefan Leutenegger, Andreas Bulling

   Proc. Adaptive and Learning Agents Workshop (ALA), pp. 1-7, 2023.

   Abstract Links BibTeX Project

   While deep reinforcement learning (RL) agents outperform humans on an increasing number of tasks, training them requires data equivalent to decades of human gameplay. Recent hierarchical RL methods have increased sample efficiency by incorporating information inherent to the structure of the decision problem but at the cost of having to discover or use human-annotated sub-goals that guide the learning process. We show that intentions of human players, i.e. the precursor of goal-oriented decisions, can be robustly predicted from eye gaze even for the long-horizon sparse rewards task of Montezuma’s Revenge – one of the most challenging RL tasks in the Atari2600 game suite. We propose Int-HRL: Hierarchical RL with intention-based sub-goals that are inferred from human eye gaze. Our novel sub-goal extraction pipeline is fully automatic and replaces the need for manual sub-goal annotation by human experts. Our evaluations show that replacing hand-crafted sub-goals with automatically extracted intentions leads to a HRL agent that is significantly more sample efficient than previous methods.

   doi:

   Paper: penzkofer23_ala.pdf

   @inproceedings{penzkofer23_ala, author = {Penzkofer, Anna and Schaefer, Simon and Strohm, Florian and Bâce, Mihai and Leutenegger, Stefan and Bulling, Andreas}, title = {Int-HRL: Towards Intention-based Hierarchical Reinforcement Learning}, booktitle = {Proc. Adaptive and Learning Agents Workshop (ALA)}, year = {2023}, doi = {}, pages = {1-7} }

1. 

   Facial Composite Generation with Iterative Human Feedback

   Florian Strohm, Ekta Sood, Dominike Thomas, Mihai Bâce, Andreas Bulling

   Proceedings of the NeurIPS Workshop Gaze Meets ML (GMML), pp. 1–19, 2022.

   Abstract Links BibTeX Project Oral presentation

   We propose the first method in which human and AI collaborate to iteratively reconstruct the human’s mental image of another person’s face only from their eye gaze. Current tools for generating digital human faces involve a tedious and time-consuming manual design process. While gaze-based mental image reconstruction represents a promising alternative, previous methods still assumed prior knowledge about the target face, thereby severely limiting their practical usefulness. The key novelty of our method is a collaborative, iterative query engine: Based on the user’s gaze behaviour in each iteration, our method predicts which images to show to the user in the next iteration. Results from two human studies (N=12 and N=22) show that our method can visually reconstruct digital faces that are more similar to the mental image, and is more usable compared to other methods. As such, our findings point at the significant potential of human-AI collaboration for reconstructing mental images, potentially also beyond faces, and of human gaze as a rich source of information and a powerful mediator in said collaboration.

   doi:

   Paper: strohm22_gmml.pdf

   @inproceedings{strohm22_gmml, title = {Facial Composite Generation with Iterative Human Feedback}, author = {Strohm, Florian and Sood, Ekta and Thomas, Dominike and Bâce, Mihai and Bulling, Andreas}, year = {2022}, booktitle = {Proceedings of the NeurIPS Workshop Gaze Meets ML (GMML)}, doi = {}, pages = {1--19} }

1. 

   Neural Photofit: Gaze-based Mental Image Reconstruction

   Florian Strohm, Ekta Sood, Sven Mayer, Philipp Müller, Mihai Bâce, Andreas Bulling

   Proc. IEEE International Conference on Computer Vision (ICCV), pp. 245-254, 2021.

   Abstract Links BibTeX Project

   We propose a novel method that leverages human fixations to visually decode the image a person has in mind into a photofit (facial composite). Our method combines three neural networks: An encoder, a scoring network, and a decoder. The encoder extracts image features and predicts a neural activation map for each face looked at by a human observer. A neural scoring network compares the human and neural attention and predicts a relevance score for each extracted image feature. Finally, image features are aggregated into a single feature vector as a linear combination of all features weighted by relevance which a decoder decodes into the final photofit. We train the neural scoring network on a novel dataset containing gaze data of 19 participants looking at collages of synthetic faces. We show that our method significantly outperforms a mean baseline predictor and report on a human study that shows that we can decode photofits that are visually plausible and close to the observer’s mental image. Code and dataset available upon request.

   doi: 10.1109/ICCV48922.2021.00031

   Paper: strohm21_iccv.pdf

   Code: Available upon request.

   Dataset: Available upon request.

   @inproceedings{strohm21_iccv, title = {Neural Photofit: Gaze-based Mental Image Reconstruction}, author = {Strohm, Florian and Sood, Ekta and Mayer, Sven and Müller, Philipp and Bâce, Mihai and Bulling, Andreas}, year = {2021}, booktitle = {Proc. IEEE International Conference on Computer Vision (ICCV)}, doi = {10.1109/ICCV48922.2021.00031}, pages = {245-254} }
2. 

   VQA-MHUG: A gaze dataset to study multimodal neural attention in VQA

   Ekta Sood, Fabian Kögel, Florian Strohm, Prajit Dhar, Andreas Bulling

   Proc. ACL SIGNLL Conference on Computational Natural Language Learning (CoNLL), pp. 27–43, 2021.

   Abstract Links BibTeX Project Oral presentation

   We present VQA-MHUG - a novel 49-participant dataset of multimodal human gaze on both images and questions during visual question answering (VQA) collected using a high-speed eye tracker. We use our dataset to analyze the similarity between human and neural attentive strategies learned by five state-of-the-art VQA models: Modulated Co-Attention Network (MCAN) with either grid or region features, Pythia, Bilinear Attention Network (BAN), and the Multimodal Factorized Bilinear Pooling Network (MFB). While prior work has focused on studying the image modality, our analyses show - for the first time - that for all models, higher correlation with human attention on text is a significant predictor of VQA performance. This finding points at a potential for improving VQA performance and, at the same time, calls for further research on neural text attention mechanisms and their integration into architectures for vision and language tasks, including but potentially also beyond VQA.

   doi: 10.18653/v1/2021.conll-1.3

   Paper: sood21_conll.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/vqa-mhug-interpretability

   Dataset: https://perceptualui.org/research/datasets/VQA-MHUG/

   @inproceedings{sood21_conll, title = {VQA-MHUG: A gaze dataset to study multimodal neural attention in VQA}, author = {Sood, Ekta and Kögel, Fabian and Strohm, Florian and Dhar, Prajit and Bulling, Andreas}, booktitle = {Proc. ACL SIGNLL Conference on Computational Natural Language Learning (CoNLL)}, year = {2021}, pages = {27--43}, doi = {10.18653/v1/2021.conll-1.3}, publisher = {Association for Computational Linguistics} }

1. 

   An Empirical Analysis of the Role of Amplifiers, Downtoners, and Negations in Emotion Classification in Microblogs

   Florian Strohm, Roman Klinger

   IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pp. 673-681, 2018.

   Abstract Links BibTeX Project

   The effect of amplifiers, downtoners, and negations has been studied in general and particularly in the context of sentiment analysis. However, there is only limited work which aims at transferring the results and methods to discrete classes of emotions, e.g., joy, anger, fear, sadness, surprise, and disgust. For instance, it is not straight-forward to interpret which emotion the phrase "not happy" expresses. With this paper, we aim at obtaining a better understanding of such modifiers in the context of emotion-bearing words and their impact on document-level emotion classification, namely, microposts on Twitter. We select an appropriate scope detection method for modifiers of emotion words, incorporate it in a document-level emotion classification model as additional bag of words and show that this approach improves the performance of emotion classification. In addition, we build a term weighting approach based on the different modifiers into a lexical model for the analysis of the semantics of modifiers and their impact on emotion meaning. We show that amplifiers separate emotions expressed with an emotion-bearing word more clearly from other secondary connotations. Downtoners have the opposite effect. In addition, we discuss the meaning of negations of emotion-bearing words. For instance we show empirically that "not happy" is closer to sadness than to anger and that fear words in the scope of downtoners often express surprise.

   doi: 10.1109/DSAA.2018.00087

   Paper: strohm18_dsaa.pdf

   @inproceedings{strohm18_dsaa, author = {Strohm, Florian and Klinger, Roman}, booktitle = {IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA)}, title = {An Empirical Analysis of the Role of Amplifiers, Downtoners, and Negations in Emotion Classification in Microblogs}, year = {2018}, pages = {673-681}, doi = {10.1109/DSAA.2018.00087} }