Develop a machine learning-based registration framework that makes use of the Diffeomorphic Registration implemented in DIPY. The goal is to train a CNN model that can compute the deformation field in an unsupervised setting. This project will also involve leveraging the reconstruction module in DIPY to perform image fusion via inter-modality registration.
- Understand Diffeomorphic Registration. Look at dipy.align module and DIPY tutorial.
- Understand advanced Image Reconstruction Models such as Free-Water-DTI, DKI, etc. in DIPY
- Implement a Deep Neural Net (e.g. a 2D/ 3D CNN) or a Multivariate Model for Co-learning
Skills required: Python, Deep Learning, Tensorflow, Registration, strong math skills.
Mentors: Shreyas Fadnavis (firstname.lastname@example.org), @bramsh Bramsh Chandio (email@example.com)
Hi @malin, @bramsh, and Shreyas. I am interested in this project. I have experience with Deep Learning and Tensorflow and have been working with deep neural net architectures. Can you help me where would be the best starting point for this project?
Hi @rahullabs, @ShreyasFadnavis can help you with this.
Hi @rahullabs! Thank you for your interest I would recommend understanding the Registration API of DIPY. We also have extensive examples of different registration methods which can be found here: https://www.dipy.org/documentation/1.1.1./examples_built/syn_registration_2d/#example-syn-registration-2d (and similar if you move back and forth between the examples). The deformation field is what we intend to learn via a machine learning architecture and CNN is just a commonly used method for the same. You may also want to look at: https://profs.etsmtl.ca/hlombaert/thinplates/ for a good understanding of the spline the CNN should learn!
@ShreyasFadnavis thanks for the guide… I will look into all the material you suggested.
Hi @bramsh and @ShreyasFadnavis . I am Saksham, a sophomore at IIT Delhi. I am interested in this project. I have worked with tensorflow and deep neural networks particularly in computer vision tasks. I have a strong mathematical background and a good understanding on deep learning algorithms. How can I get started with this project building upto my proposal?
You can start by making a small enhancement/bugfix/documentation fix/etc to DIPY. It can help you get some idea about how things would work during the GSoC. The fix does not need to be related to your proposal.
Getting started with DIPY tutorial https://dipy.org/documentation/1.1.1./examples_built/quick_start/#example-quick-start
DIPY GSoC 2020 GitHub page https://github.com/dipy/dipy/wiki/Google-Summer-of-Code-2020
And for more materials to read about ML topics refer to @ShreyasFadnavis’s reply above.
Thank you for your interest! As per @bramsh 's suggestion, make sure to make a PR on DIPY’s GitHub… This way, you will also get familiarized with DIPY’s API. I would also suggest to read up some literature on thin-plate splines, diffeomorphic registration (see: https://dipy.org/documentation/1.1.1./examples_built/syn_registration_3d/ and https://www.youtube.com/watch?v=knuyM6bEQvs) to get an idea of how it works. To get some basics in place, see: http://user.engineering.uiowa.edu/~aip/papers/bookstein-89.pdf. As a part of this project, we will first start with a 2D case and then move to 3D. Feel free to get creative with the deep learning architecture! Deep learning is not compulsory to learn this spline, you can also implement any other method that you feel can do the job It might be simple to think of this problem in a supervised setting first, where you simulate the ground truth by adding a known deformation field… Does this help?
Hi,I am Shubham Shaswat,currently in final year pursuing Computer Science at HIT Kolkata. I would like to contribute to the project.Any suggestion to get started would be much appreciated.
also are there any chat room for discussing the project?
Please read comments above on this post by me and @ShreyasFadnavis for details about the project.
You can discuss project in this current thread.
You can also checkout more details on DIPY’s GSoC page here https://github.com/dipy/dipy/wiki/Google-Summer-of-Code-2020.
@bramsh Thanks,I will look into it.
Hi,Going through the links and tutorials as listed,I got some rough idea about the objective of the topics,still I need clarifications.
This is the brief idea about what I have understood:
Image Registration is matching points from one image (source) to another image(target) correspond to the same physical regions in both.
In our case we have a static image and moving image,our goal is to find deformation field.
So,If we going to build a Deep Learning model,does this mean our input will be these two images and the target would be finding a deformation field?
Also,I find this http://www.cs.ucf.edu/~bagci/teaching/mic16/lec15.pdf,does it useful in our case?
Hi @ShubhamShaswat! Yes, the tutorial is useful and you do have the correct idea. The goal is to do a non-rigid registration using thin-plate splines representing the deformation field.
The Deep Learning phase would involve something (like you guessed correctly) similar to: Cao. et.al. “Deformable Image Registration Based on Similarity-Steered CNN Regression”, MICCAI, 2017. Does this help?
Thanks,I will look into this paper.Also,I think it would be better if I share an architecture of our model including input pipeline so that we know we are on a right track.
Not sure what you mean, but, its a good idea to keep sharing anyway
Hi @ShreyasFadnavis and @bramsh
I am Kunal Mehta currently in my Final Year of Computer Engineering at K.J Somaiya.I have previously worked on various Machine Learning, Deep Learning and would like to contribute as much as I can towards this project.I have gone through previous discussions as well as resources and documentation which @ShreyasFadnavis mentioned.I have also worked on basic Dipy tutorial to improve my understanding towards this topic.
Here is the jupyter notebook on colab which I was working on following the tutorial to improve my knowledge on Dipy.
Could you please help identify issues that are related to our topic
Hi,After looking these papers:
2.Non-rigid image registration using fully convolutional networks with
deep self-supervision (Another paper that I found usefull)
Our main goal is to find and optimal 𝐷𝑣 (deformation field) for objective function is:
min(𝐷𝑣) −𝑆 (𝐼𝑓(𝑣),𝐼𝑚(𝐷𝑣 ∘ 𝑣)) + 𝜆𝑅(𝐷𝑣)
In Paper 1,The deep learning network is used is this:
The author used Similarity Guidance to bridge the gap between two input images. In order to obtain the very accurate deformation field of each image pair, SyN is first applied on the intensity images with careful parameter tuning, and then Diffeomorphic Demons.The Deformation field after these 2 stages, is used as Ground Truth for model Training.
In paper 2:
They paper mentioned that they don’t need deformation field in training.The model is optimized by maximizing image similarity with smoothness regularization of deformation fields.
I am looking at these two papers as reference in our project.So,if there anything more you want to add or point out any mistakes will be much appreciated.
I have also question regarding the use of current Dipy modules in these models.
Hi @KUNAL_MEHTA! Thank you for your interest. I see that you have created a replica of some DIPY tutorials, which is good. I am not sure if you mean GitHub issues here, but, generally speaking, they are not very beginner friendly. However, you can take a look at different issues related to registration such as: https://github.com/dipy/dipy/issues/1838
It will give you a good insight into the DIPY API
Hi @ShubhamShaswat, thank you for sharing that information:
I am also adding some links and references here:
- Miao, et. al. " A CNN Regression Approach for Real-Time 2D/3D Registration", IEEE TMI 2016
- Krebs, et. al. “Robust Non-rigid Registration Through Agent-Based Action Learning”, MICCAI, 2017
All of the above are plausible and good approaches
Thanks @ShreyasFadnavis for sharing the resources,however I can’t access the second paper published on IEEE.