GSoC 2021 Project Idea 26.1: Eye-tracker based on a convolutional neural network in Python + TensorFlow/Pytorch

Current eye-trackers generally rely on previous-generation computer-vision algorithms and the best ones are also expensive and closed-source. A recent publication from Google Research has shown that it is possible to obtain very good performance using a simple convolutional neural network (CNN) running off a simple mobile phone camera. The details of the CNN have been made available, but the actual implementation is not available. The goal of the project is to implement this algorithm in an open-source package and then explore various extensions including a) incorporating head-position estimation for eye-in-head measurements; and b) extending the algorithm to higher sampling-rates and incorporating filtered estimates of eye-position that take the time-series of previously estimated eye-positions into account.

Mentor: Suresh Krishna @suresh.krishna

1 Like

This project seems really interesting!
I’ve implemented a few projects in TensorFlow before as well
Can you tell me, How can I know more about this?
Like maybe a Github Repo or something?

1 Like

@suresh.krishna , please could you help with details that @shubh is looking for?
Thanks :+1:

Hi !

Thanks for the interest.

There is no Github repo yet. The source paper is this one : Accelerating eye movement research via accurate and affordable smartphone eye tracking – Google Research

I am in touch with Vidhya Navalpakkam (the senior author) and she is happy to assist with project translation.

As you can see from the paper, there is a model description but the code is not open source, and will not be open-sourced. So the first part of the project is to simply replicate the algorithm from that paper. The second part would be to implement various extensions as I mention above.

Creating a functional, open-source eye-tracker is likely to be of great interest in many areas.


Very interesting project! It is so occurred that my research is also about convolutional neural network using Pytroch. Thank you for the source paper. As I understand the simple mobile phone camera supposed be anodroid? Is there something using NNAPI?

Hi @suresh.krishna, I am Neelay Shah (website), a pre-final year undergraduate student at BITS Pilani in India. I enjoyed reading the paper you’ve shared and am excited about this project.
I was trying to download the GazeCapture dataset used in the paper for getting started on a minimal re-implementation of the paper in PyTorch. However, something seems to be off with the registration process on the website and hence I’m not able to download the dataset. Could you please help me in gaining access to the dataset?
Thank you.

Hi, Thanks for the interest !

In the paper, they used an Android camera whose output was sent to a central server for individualized tuning of the network etc. But on-device training is something Google is interested in, and that is why there is an interest in keeping the model small as well.

However, for the purposes of the project, one does not have to be chained to Android or any other particular system… I am not familiar with NNAPI, but any project that essentially collects a sequence of video-frames and tries to predict gaze and/or eye-position from them would fit.

Hi Neelay… thanks for the interest !

Ah yes, seems to be a problem there. I will check with the authors of the dataset. In any case, alternative datasets can be generated - it is not an issue for the project itself if this dataset becomes unavailable.

ps. The website admin has been roped in and is looking into the issue.

1 Like

Hi @suresh.krishna, what do you think would be some good initial steps towards a project proposal?

@Neelay The website registration works now for the gaze capture data. Also associated with it is a neural network ([1606.05814] Eye Tracking for Everyone) with apparently poorer performance compared to the work from Google.

A good initial step would be to understand the Google paper and come up with an outline of how to implement their network - doesn’t have to be on Android for now. From there, one could outline how to create pipelines to evaluate the model with other datasets or newly collected datasets, extend it (and in what directions), etc. Then one could actually start the implementation phase and see what problems one runs into – and in the normal course, there will be many, like temporarily not being to able to download the data :slight_smile:

1 Like