Julian Steil

Max Planck Institute for Informatics

+49 681 9325 2132 Campus E1 4, 66123 Saarbrücken, Germany Google Scholar

Biography

Julian Steil was a PhD student in the Perceptual User Interfaces group from 2014 to 2019. He holds a Bachelor's degree in Computer Science and a Master's degree in Visual Computing from Saarland University. His research interests include Mobile Eye Tracking, Image Processing and Computer Vision, Machine Learning and Pattern Recognition, as well as Human-Computer Interaction.

Publications

1. 

   InvisibleEye: Fully Embedded Mobile Eye Tracking Using Appearance-Based Gaze Estimation

   Julian Steil, Marc Tonsen, Yusuke Sugano, Andreas Bulling

   ACM SIGMOBILE Mobile Computing and Communications Review, 23 (2), pp. 30-34, 2019.

   Abstract Links BibTeX Project

   Despite their potential for a range of exciting new applications, mobile eye trackers suffer from several fundamental usability problems. InvisibleEye is an innovative approach for mobile eye tracking that uses millimetre-size RGB cameras that can be fully embedded into normal glasses frames, as well as appearance-based gaze estimation to directly estimate gaze from the eye images. Through evaluation on three large-scale, increasingly realistic datasets, we show that InvisibleEyes can achieve a person-specific gaze estimation accuracy of up to 2.04° using three camera pairs with a resolution of only 3x3 pixels.

   Paper: steil19_sigmobile.pdf

   @article{steil19_sigmobile, author = {Steil, Julian and Tonsen, Marc and Sugano, Yusuke and Bulling, Andreas}, title = {InvisibleEye: Fully Embedded Mobile Eye Tracking Using Appearance-Based Gaze Estimation}, journal = {ACM SIGMOBILE Mobile Computing and Communications Review}, year = {2019}, volume = {23}, number = {2}, pages = {30-34} }

1. 

   PrivacEye: Privacy-Preserving Head-Mounted Eye Tracking Using Egocentric Scene Image and Eye Movement Features

   Julian Steil, Marion Koelle, Wilko Heuten, Susanne Boll, Andreas Bulling

   Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA), pp. 1–10, 2019.

   Abstract Links BibTeX Project Best Video Award

   Eyewear devices, such as augmented reality displays, increasingly integrate eye tracking but the first-person camera required to map a user’s gaze to the visual scene can pose a significant threat to user and bystander privacy. We present PrivacEye, a method to detect privacy-sensitive everyday situations and automatically enable and disable the eye tracker’s first-person camera using a mechanical shutter. To close the shutter in privacy-sensitive situations, the method uses a deep representation of the first-person video combined with rich features that encode users’ eye movements. To open the shutter without visual input, PrivacEye detects changes in users’ eye movements alone to gauge changes in the "privacy level" of the current situation. We evaluate our method on a first-person video dataset recorded in daily life situations of 17 participants, annotated by themselves for privacy sensitivity, and show that our method is effective in preserving privacy in this challenging setting.

   doi: 10.1145/3314111.3319913

   Paper: steil19_etra.pdf

   Supplementary Material: steil19_etra_sup.pdf

   Video: https://www.youtube.com/watch?v=Gy61255F8T8

   @inproceedings{steil19_etra, title = {PrivacEye: Privacy-Preserving Head-Mounted Eye Tracking Using Egocentric Scene Image and Eye Movement Features}, author = {Steil, Julian and Koelle, Marion and Heuten, Wilko and Boll, Susanne and Bulling, Andreas}, year = {2019}, booktitle = {Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA)}, pages = {1--10}, doi = {10.1145/3314111.3319913}, video = {https://www.youtube.com/watch?v=Gy61255F8T8} }
2. 

   Privacy-Aware Eye Tracking Using Differential Privacy

   Julian Steil, Inken Hagestedt, Michael Xuelin Huang, Andreas Bulling

   Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA), pp. 1–9, 2019.

   Abstract Links BibTeX Project Best Paper Award

   With eye tracking being increasingly integrated into virtual and augmented reality (VR/AR) head-mounted displays, preserving users’ privacy is an ever more important, yet under-explored, topic in the eye tracking community. We report a large-scale online survey (N=124) on privacy aspects of eye tracking that provides the first comprehensive account of with whom, for which services, and to which extent users are willing to share their gaze data. Using these insights, we design a privacy-aware VR interface that uses differential privacy, which we evaluate on a new 20-participant dataset for two privacy sensitive tasks: We show that our method can prevent user re-identification and protect gender information while maintaining high performance for gaze-based document type classification. Our results highlight the privacy challenges particular to gaze data and demonstrate that differential privacy is a potential means to address them. Thus, this paper lays important foundations for future research on privacy-aware gaze interfaces.

   doi: 10.1145/3314111.3319915

   Paper: steil19_etra_2.pdf

   @inproceedings{steil19_etra_2, title = {Privacy-Aware Eye Tracking Using Differential Privacy}, author = {Steil, Julian and Hagestedt, Inken and Huang, Michael Xuelin and Bulling, Andreas}, year = {2019}, booktitle = {Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA)}, doi = {10.1145/3314111.3319915}, pages = {1--9} }

1. 

   Fixation Detection for Head-Mounted Eye Tracking Based on Visual Similarity of Gaze Targets

   Julian Steil, Michael Xuelin Huang, Andreas Bulling

   Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA), pp. 1–9, 2018.

   Abstract Links BibTeX Project

   Fixations are widely analysed in human vision, gaze-based interaction, and experimental psychology research. However, robust fixation detection in mobile settings is profoundly challenging given the prevalence of user and gaze target motion. These movements feign a shift in gaze estimates in the frame of reference defined by the eye tracker’s scene camera. To address this challenge, we present a novel fixation detection method for head-mounted eye trackers. Our method exploits that, independent of user or gaze target motion, target appearance remains about the same during a fixation. It extracts image information from small regions around the current gaze position and analyses the appearance similarity of these gaze patches across video frames to detect fixations. We evaluate our method using fine-grained fixation annotations on a five-participant indoor dataset (MPIIEgoFixation) with more than 2,300 fixations in total. Our method outperforms commonly used velocity- and dispersion-based algorithms, which highlights its significant potential to analyse scene image information for eye movement detection.

   doi: 10.1145/3204493.3204538

   Paper: steil18_etra.pdf

   @inproceedings{steil18_etra, author = {Steil, Julian and Huang, Michael Xuelin and Bulling, Andreas}, title = {Fixation Detection for Head-Mounted Eye Tracking Based on Visual Similarity of Gaze Targets}, booktitle = {Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA)}, year = {2018}, pages = {1--9}, doi = {10.1145/3204493.3204538} }
2. 

   Forecasting User Attention During Everyday Mobile Interactions Using Device-Integrated and Wearable Sensors

   Julian Steil, Philipp Müller, Yusuke Sugano, Andreas Bulling

   Proc. ACM International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI), pp. 1–13, 2018.

   Abstract Links BibTeX Project Best Paper Award

   Visual attention is highly fragmented during mobile interactions but the erratic nature of attention shifts currently limits attentive user interfaces to adapt after the fact, i.e. after shifts have already happened. We instead study attention forecasting – the challenging task of predicting users’ gaze behavior (overt visual attention) in the near future. We present a novel long-term dataset of everyday mobile phone interactions, continuously recorded from 20 participants engaged in common activities on a university campus over 4.5 hours each (more than 90 hours in total). We propose a proof-of-concept method that uses device-integrated sensors and body-worn cameras to encode rich information on device usage and users’ visual scene. We demonstrate that our method can forecast bidirectional attention shifts and whether the primary attentional focus is on the handheld mobile device. We study the impact of different feature sets on performance and discuss the significant potential but also remaining challenges of forecasting user attention during mobile interactions.

   doi: 10.1145/3229434.3229439

   Paper: steil18_mobilehci.pdf

   @inproceedings{steil18_mobilehci, author = {Steil, Julian and M{\"{u}}ller, Philipp and Sugano, Yusuke and Bulling, Andreas}, title = {Forecasting User Attention During Everyday Mobile Interactions Using Device-Integrated and Wearable Sensors}, booktitle = {Proc. ACM International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI)}, year = {2018}, doi = {10.1145/3229434.3229439}, pages = {1--13} }

1. 

   InvisibleEye: Mobile Eye Tracking Using Multiple Low-Resolution Cameras and Learning-Based Gaze Estimation

   Marc Tonsen, Julian Steil, Yusuke Sugano, Andreas Bulling

   Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), 1 (3), pp. 1–21, 2017.

   Abstract Links BibTeX Project Distinguished Paper Award

   Analysis of everyday human gaze behaviour has significant potential for ubiquitous computing, as evidenced by a large body of work in gaze-based human-computer interaction, attentive user interfaces, and eye-based user modelling. However, current mobile eye trackers are still obtrusive, which not only makes them uncomfortable to wear and socially unacceptable in daily life, but also prevents them from being widely adopted in the social and behavioural sciences. To address these challenges we present InvisibleEye, a novel approach for mobile eye tracking that uses millimetre-size RGB cameras that can be fully embedded into normal glasses frames. To compensate for the cameras’ low image resolution of only a few pixels, our approach uses multiple cameras to capture different views of the eye, as well as learning-based gaze estimation to directly regress from eye images to gaze directions. We prototypically implement our system and characterise its performance on three large-scale, increasingly realistic, and thus challenging datasets: 1) eye images synthesised using a recent computer graphics eye region model, 2) real eye images recorded of 17 participants under controlled lighting, and 3) eye images recorded of four participants over the course of four recording sessions in a mobile setting. We show that InvisibleEye achieves a top person-specific gaze estimation accuracy of 1.79° using four cameras with a resolution of only 5×5 pixels. Our evaluations not only demonstrate the feasibility of this novel approach but, more importantly, underline its significant potential for finally realising the vision of invisible mobile eye tracking and pervasive attentive user interfaces.

   doi: 10.1145/3130971

   Paper: tonsen17_imwut.pdf

   @article{tonsen17_imwut, author = {Tonsen, Marc and Steil, Julian and Sugano, Yusuke and Bulling, Andreas}, title = {InvisibleEye: Mobile Eye Tracking Using Multiple Low-Resolution Cameras and Learning-Based Gaze Estimation}, journal = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT)}, year = {2017}, doi = {10.1145/3130971}, volume = {1}, number = {3}, pages = {1--21} }

1. 

   3D Gaze Estimation from 2D Pupil Positions on Monocular Head-Mounted Eye Trackers

   Mohsen Mansouryar, Julian Steil, Yusuke Sugano, Andreas Bulling

   Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA), pp. 197-200, 2016.

   Abstract Links BibTeX Project

   3D gaze information is important for scene-centric attention analysis, but accurate estimation and analysis of 3D gaze in real-world environments remains challenging. We present a novel 3D gaze estimation method for monocular head-mounted eye trackers. In contrast to previous work, our method does not aim to infer 3D eye- ball poses, but directly maps 2D pupil positions to 3D gaze directions in scene camera coordinate space. We first provide a detailed discussion of the 3D gaze estimation task and summarize different methods, including our own. We then evaluate the performance of different 3D gaze estimation approaches using both simulated and real data. Through experimental validation, we demonstrate the effectiveness of our method in reducing parallax error, and we identify research challenges for the design of 3D calibration procedures.

   doi: 10.1145/2857491.2857530

   Paper: mansouryar16_etra.pdf

   @inproceedings{mansouryar16_etra, author = {Mansouryar, Mohsen and Steil, Julian and Sugano, Yusuke and Bulling, Andreas}, title = {3D Gaze Estimation from 2D Pupil Positions on Monocular Head-Mounted Eye Trackers}, booktitle = {Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA)}, year = {2016}, pages = {197-200}, doi = {10.1145/2857491.2857530} }

1. 

   Discovery of Everyday Human Activities From Long-term Visual Behaviour Using Topic Models

   Julian Steil, Andreas Bulling

   Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 75-85, 2015.

   Abstract Links BibTeX Project

   Human visual behaviour has significant potential for activity recognition and computational behaviour analysis, but previous works focused on supervised methods and recognition of predefined activity classes based on short-term eye movement recordings. We propose a fully unsupervised method to discover users’ everyday activities from their long-term visual behaviour. Our method combines a bag-of-words representation of visual behaviour that encodes saccades, fixations, and blinks with a latent Dirichlet allocation (LDA) topic model. We further propose different methods to encode saccades for their use in the topic model. We evaluate our method on a novel long-term gaze dataset that contains full-day recordings of natural visual behaviour of 10 participants (more than 80 hours in total). We also provide annotations for eight sample activity classes (outdoor, social interaction, focused work, travel, reading, computer work, watching media, eating) and periods with no specific activity. We show the ability of our method to discover these activities with performance competitive with that of previously published supervised methods.

   doi: 10.1145/2750858.2807520

   Paper: steil15_ubicomp.pdf

   @inproceedings{steil15_ubicomp, author = {Steil, Julian and Bulling, Andreas}, title = {Discovery of Everyday Human Activities From Long-term Visual Behaviour Using Topic Models}, booktitle = {Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)}, year = {2015}, doi = {10.1145/2750858.2807520}, pages = {75-85} }