Accurate and efficient measurement of visual function is difficult in infants and young children because of limited cooperation, inability to provide cognitive verbal responses and lack of efficient behavioural methods. This is important in the clinical and research context where detection and treatment of eye conditions in infancy is dependent on measurement of visual function. Visual deprivation in infants disrupts normal visual development and affects multiple visual functions that are important in visually guided behaviors in everyday life such as contrast sensitivity, motion perception, contour integration, and face recognition. At present there are no reliable automated objective methods for measuring visual functions in infants and young children below the age of 3 years.
In this AI for health project, that continues work from GSoC 2022 and GSoC 2023, we will address these limitations by developing a deep-learning based eye-tracker to test infant visual function by automatic monitoring of their eye-movements. Previous projects made progress towards developing an eye-tracker module that can work both with software-based and hardware-based eye-trackers and with developing a visual stimulus presentation module. Both these modules will be developed further in this year’s GSoC so that a full proof of concept can be developed. The project involves a) the development of an application with a suite of visual stimuli and analytical procedures to probe multiple visual functions; b) testing and further developing a deep-learning based infant eye-tracker and c) developing a GUI and controller that holds the display, eye-tracking and analysis components together.
Skill level: Intermediate/advanced
Required skills: Comfortable with Python. Experience with image/video processing and using deep-learning based image-processing models. Ideally, also comfortable with Android/iOS app development and especially ARKit/equivalent, but not necessary.
Time commitment: Full-time (350 h, large project)
Lead mentor: Arvind Chandna (@arvindchandna, arvind@vision4children.org.uk)
Project website: GitHub - Shazam213/automated-preferential-looking: This project aims to develop a ready-to-deploy application suite that will address these limitations by integrating hardware devices or deep learning-based infant eye trackers, and visual stimuli analysis into a user-friendly graphical user interface (GUI).
Backup mentors: Suresh Krishna (@suresh.krishna, suresh.krishna@mcgill.ca), Soham Mulye (mulyesoham@gmail.com)
Tech keywords: Health AI, Infant vision, Image processing, App development, Python, health AI, IOS/Android, ML based eye-tracking, PyTorch, Deep Learning
1 Like
Hello Sir,
My name is Mohamed Atef. I am an Egyptian senior AI student. I am interested in your project and I took a quick look at the GitHub repository, however, I have some questions:
1 - Did you use a specific dataset for training the tracking model?
2 - What advancements do you want in the project in detail?
3 - I did not understand how a hardware device will be integrated, could you elaborate?
I also want to know more about the project to conduct more research, so any guidelines will be helpful.
Thanks in advance.
Hi @suresh.krishna @arvindchandna,
I am Dhruvanshu Joshi, a pre final year student at VJTI. I am a computer vision and deep learning enthusiast. I have tried to use these skills to solve real world complex problems through my projects like Pothole detection which uses computer vision and depth estimation to detect road potholes. I also am the global runner up of OpenCV Spatial AI competition, 2022. I have spent quite some time as an open source contributor and worked as a open source summer intern at PyMC during the summer of 2023. I am interested in working on the project titled “ Efficient app-based measurement of visual functions in infants and young children (350 h)” as a part of GSOC, 2024.
I have been through the code base and built the project locally including the necessary psychopy installation. I see that the original implementation is based on icatcher+ which is trained on the Lookit dataset. I believe that integration of api developed by Ioannis Valasakis with the current implementation would be a valid next step to replace the need for keyboard inputs in the experiments.
It would be really helpful if you can guide me to some task/issue that I can solve in order to contribute to this project. I am attaching my resume here for your reference.
Yours Sincerely,
Dhruvanshu Joshi
Hello !!
My name is Tvisha Vedant and I am currently a second-year B.Tech Computer Science student at Veermata Jijabai Technological Institute,Mumbai
I have a strong background in various areas of computer science, including natural language processing (NLP), image processing , web development,Pytorch , using libraries like numpy/pandas and version control with Git/GitHub. I had contributed in building a web-app which controlled the music volume and playback using gestures using Media-pipe.I have worked extensively with LLMS like langchain, Transformer architecture, and NLP, which I applied in developing a healthcare chatbot(using the Llama model from hugging face after testing a lot many models).
Having gone through the eye tracking projects of GSOC’22 and GSOC’23 I would like to contribute to its extention under GSOC’24. I have started building the repo locally.
I am new to the world of open source with just a few month’s experience but the entire concept and setting of open source contributions appeals me !!
Looking forward to collaborating with you.
Email:
tnvedant_b22@ce.vjti.ac.in
My github repo:
https://github.com/tvilight4
My resume:
Resume
Do share any updates regarding the tasks to be performed for GSOC’24
@Dhruvanshu_Joshi @Mohamed1 @Tvisha_Vedant - thanks for the interest. The first step is to go through the code-base, learn how to use Psychopy and understand how the eye-tracker component is working. The possible extensions include improving the eye-tracker quality, increasing the library of visual stimuli that can be generated, developing a functional and well-designed GUI that interfaces with the eye-tracker and stimulus components, etc. Additional extensions will be posted here in a week.
Feel free to post here if you have any issues. Please first do some homework and read up, and only ask after you have put in some work and thought - and please describe in detail what you did. It is important that you do this before you ask for help - we have busy day-jobs and we are helping many of you at the same time.
1 Like
We have compiled a list of feature enhancements below. In the interim, get familiar with the code-base and test it out yourself to better understand the feature upgrades. Feel free to reach out to @soham_mulye , if you have any issues. Please think before you reach out though, do some homework and read up, and only ask after you have put in some work and thought - and please describe in detail what you did. It is important that you do this before you ask for help - we have busy day-jobs and we are helping many of you at the same time
Some example features to be added to the project:
- Enhance the GUI to offer a more interactive and user-friendly experience.
- Change the stimuli to an attractive cartoon character and use an additional auditory cue with each change whenever possible to keep infants engaged in the experiments.
- Also allow smooth flow of stimuli across the screen from one point to another.
- Add more stimulus classes (needs Psychopy knowledge).
- Test the software - create an installation routine that works more reliably in case bugs are found.
- The deep learning model’s current limitation is that it only detects gaze direction in either left or right directions, so it cannot be utilised in studies with stimuli in all four hemispheres. Fix the issue.
- Provide functionality for easy integration of eye tracking hardware devices such as Tobii eye trackers, allowing anyone having the hardware to run the programme. (Link to past api created as part of GSoC 2022: GitHub - wizofe/ao-baby-tracker: Google Summer of Code 2022 - Eye tracking project for neonates)
1 Like
Hello everyone! my name is Atharv Sabde, and I am an undergraduate student at SPPU, India. I am reaching out to express my interest in participating in GSOC 2024 and potentially collaborating with your organization.
With a huge passion for machine learning and generative AI, along with a keen interest in computer vision, I have dedicated myself to continuous learning and skill development over the past few years. As evidence of my commitment, I am proud to hold the title of Kaggle 2X Expert in both notebooks and discussions.
Throughout my academic journey, I have embarked on various projects that have enabled me to apply my knowledge effectively. Notable among them are projects such as the LLM chatbot developed using the Gemini API, a face detection model in the domain of computer vision, and a heart failure prediction model utilizing machine learning techniques.
After thoroughly reviewing the list of project ideas, I am particularly drawn to the this project. Its potential impact and alignment with my interests make it an ideal choice for my involvement in GSOC 2024.
In terms of technical proficiency, I have acquired a solid foundation in Python, TensorFlow, PyTorch, deep learning, neural networks, and web development through both coursework and practical projects. I am enthusiastic about embracing new challenges and continuously enhancing my skills while contributing meaningfully to the project.
To provide you with a comprehensive view of my capabilities, I invite you to explore my GitHub profile at AtharvSabde (Atharv Sabde) · GitHub , and Kaggle profile at Atharv Sabde | Expert | Kaggle , where you can find detailed insights into my projects and contributions.
Can you please tell me what next steps should I take to get selected for this project?
@Atharv_Sabde thank you for your interest. I would suggest going through the feature list that has to be implemented in the project along with the codebase and understanding different components and their implementation. Also run the program on your system once to understand the project better. your previous work seems interesting and you should use that in your proposal and mention how will that be useful for this project.
Please feel free to ask questions here as needed, but please do your own thinking and homework first, and spend time on your posts/questions.
Hello @arvindchandna @suresh.krishna @soham_mulye ,
I am Om Khare, and I am pursuing a B. Tech. in Computer Engineering from the College Of Engineering Pune (COEP). I have worked extensively on Machine Learning, Data Analysis, and Computer Vision projects. I interned at Deutsche Bank, wherein I was one of the very few interns selected to work on live ML and data analysis projects. I have also done honors courses in deep learning and big data. I have three publications, two in computer vision-related projects and another in applied machine learning. Currently, my final year btech project is related to Computer Vision and I have a good experience developing android applications; hence, I am a perfect match for this project and am excited to work on it!
Hope to get in touch with you soon!
Email: omkhare02@gmail.com
Resume: Resume
Github: OmKhare (Om Khare) · GitHub
@Om_Khare @Atharv_Sabde - thank you for the interest. The main step to get selected is to prepare a proposal, of course, that takes the features mentioned above into account, proposes a plan to extend the project, gives technical details and also shows (off) your previous coding experience in order to show that you can do this. @soham_mulye , last year’s GSoC intern is a mentor this year and so will help as needed and possible.
Hi @suresh.krishna @arvindchandna ,
I am Sarthak Puri, a B.Tech Computer Science student at Chandigarh University. I have expertise in machine learning, deep learning and app development and find this project align with my skills and experience till now so well that it feels as though I was destined to work on it. Can you provide me with your email, I would like to connect with you directly and discuss one key aspect of this project before drafting the proposal.
Thanks & Regards
@Sarthakpurii you can direct message me here
@suresh.krishna
At the current point of time, GUI has been implemented with TkInter with which I am proficient as well, but to create a comprehensive and user-friendly application, Flutter will be the best choice given its cross platform characteristic as well. Also integrating deep learning models to flutter apps is also not a puzzling task, with the help of tensorflow lite.
Before proceeding with drafting my proposal, I wanted to discuss with you and understand your views on incorporating Flutter for the app development part of this project.
sure. no problem with proposing to use flutter
That’s really good to hear. I will continue to work on my proposal and having it reviewed by you before submitting it would be immensely beneficial.
And I couldn’t properly introduce myself earlier which I thought I will be doing on mail
So I am Sarthak Puri, a B.Tech Computer Science student at Chandigarh University. I have expertise in machine learning, deep learning and app development with flutter. I did an internship at SHMT as ML engineer which also focuses on the intersection of technology and healthcare. I have done many similar projects like emotion detection etc in the field of computer vision which stole my attention to contribute to this project.
I look forward to the possibility of contributing to this meaningful project under your guidance. Thank you for considering my inquiry.
Resume
1 Like
Hello @suresh.krishna ,
I’m geared up with a strategy to make this project a success, inspired by Soham Mulye’s remarkable GSoC’23 work. I’ve also had detailed discussion of this project with Soham which provided me valuable insights. I’d love to share the planned approach with you but given the public nature of this space, I prefer discussing it in a more secure setting. Could you share a preferred contact method, like an email or LinkedIn, for a brief 5-minute discussion? Your insights would be greatly appreciated!
Regards
@Sarthakpurii direct messages here are private.
Hey @suresh.krishna @arvindchandna,
I hope you’re doing well! I wanted to express my keen interest in contributing to your Project for GSoC 2024. I’ve put together a draft proposal and would greatly appreciate your expert advice and feedback on it. Could you please provide me with the best way to contact you? Whether it’s through email, a messaging platform, or another preferred method.
you can send me a direct message here with a link to the proposal. @darshil929