Using nipype via Docker

,
1

Hey Folks, I installed Docker Desktop on my machine (I am restricted to use Windows 10) and pulled the nipype image via docker pull nipype/nipype. I also set up three folders in my home directory called /data, /output and /workdir. /data already contains some NIFTI-files and /workdir is the directory where I would like to create and save my python scripts. So … what now? How am I going to be able to start writing code? What do I have to do to ‘tell’ my nipype-container where the code should be saved, where the data is, etc.? How do I even start writing code? I guess I have to do something like @oesteban described in Using Nipype under Docker? Sorry, these questions might be rather unspecific, but I have the feeling I didn’t really get how this all works even after spending 2-3 hours with the nipype-tutorial today.

Greetings,

Johannes

1 Like
2

Ahoi hoi @JohannesWiesner,

yeah, the whole thing, that is the big picture of how these tools integrate with one another, is not super straightforward.

In principle, you have the critical components ready to go. Now it’s about defining the workflow that’s best for you. Let’s start from the setup you currently have: /data where your input data is, /output where the outputs should go and /workdir where the python scripts should be stored. The crucial part is that you make these paths and things within them available in your nipype container (the running instance of the nipype image). In other words, you need to map paths on your host machine (your windows10) to paths inside the container, kinda a like stargate hehe (or containergate). With that you can do the following: store your data and scripts/workflows on your machine, but run the scripts/workflows inside the container. So basically all processing is happening inside the container while storage is happening outside the container. You can think of the container as an application that executes your scripts/workflow in a virtual, isolated environment.

Here’s a toy example:

docker run -it --rm 
-v /path/to/data/on/host:/path/to/data/in/container
-v /path/to/output/on/host:/path/to/output/in/container 
-v /path/to/workdir/on/host:/path/to/workdir/in/container 
nipype/nipype

When running this command in your shell (whatever that is in windows10, NB: depending on your precise version you need docker toolbox or docker desktop, WSL, etc., please check this section from the MTL Brainhack School) you should end up “within” the container shell and should be able to access your data at /path/to/data/in/container, should be able to write output to /path/to/output/in/container and find/run scripts/workflows placed in /path/to/workdir/in/container as these point to correspondingly mapped path on your host machine. Therefore, you can write/change scripts on your host machine and run them inside the container:

python /path/to/workdir/in/container/my_cool_workflow.py

NB: make sure to adapt the input and output paths of your script/workflow to the paths inside the container.

HTH, cheers, Peer

P.S.: sorry for the repetitive writing. Here are some other links/resources that might be helpful: short docker intro, long docker intro, one day docker workshop.

2 Likes
3

Hi Peer, thanks for your detailed explanation on nipype & docker, that really sorted a few things out for me! After trying out what you proposed I think I got on the right track but of course, as always further questions came up and I hope that this reply is not completely overloaded with questions and text (probably it is). I tried to break it down in a ‘What I did so far part’ and a ‘Questions’ part in order to give it at least some structure.

What I did so far:

I followed your instructions and those from the the nipype tutorial (just to be able to use some already written scripts) and did the following:

docker run -it --rm -v C:/Users/Johannes.Wiesner/Documents/testing/test_nipype/workdir/:/home/neuro/nipype_tutorial -v C:/Users/Johannes.Wiesner/Documents/testing/test_nipype/data/:/data -v C:/Users/Johannes.Wiesner/Documents/testing/test_nipype/output/:/output -p 8888:8888 miykael/nipype_tutorial jupyter notebook

I then extracted all the content from the nipype_tutorial repository into my local working directory C:/Users/Johannes.Wiesner/Documents/testing/test_nipype/workdir/ and also downloaded and extracted the ds000114 dataset into my data directory.

I then ran two example scripts, namely notebooks/introduction_quickstart_non-neuroimaging and notebooks/introduction_quickstart_non-neuroimaging and both scripts ran succesfully without any errors.

Then I also tried to to use the nipype docker image and ran:

docker run -it --rm -v C:/Users/Johannes.Wiesner/Documents/testing/test_nipype/workdir/:/home/neuro/nipype_tutorial -v C:/Users/Johannes.Wiesner/Documents/testing/test_nipype/data/:/data -v C:/Users/Johannes.Wiesner/Documents/testing/test_nipype/output/:/output -p 8888:8888 nipype/nipype

For that created a dummy script that simply smoothes one nifti-file in the data set:

import nipype.interfaces.spm as spm
smooth = spm.Smooth()
# Note that you have to decompress the original file first
smooth.inputs.in_files = '/data/ds000114/sub-01/ses-test/anat/sub-01_ses-test_T1w.nii'
smooth.inputs.fwhm = [4, 4, 4]
smooth.run()

Again, everything seemed to work fine.

Further Questions that came up:

However, I was not able to start jupyter-notebook from within the nipype container, which I found confusing in the beginning. Shouldn’t the ‘real’ nipype be able to to everything what the tutorial image can do? It took me while to find out that the nipype_tutorial uses Neurodocker which can build containers that include nipype, all the standard neuroimaging tools, conda, jupyter, etc.

1.) So am I right that I have to use the Neurodocker if I want to be able to have all the functionality from nipype with an additional IDE like jupyter notebook? If so, is there a newbie-friendly way to create a “minimal” docker image using Neurodocker that just has the standard software packages and all what the “typical user” needs (maybe there isn’t because there is no such thing as a standard solution)?

I also have a few “optional” questions that are not super important (i.e. they are not important in order to use nipype) but are still gnawing on my nerves and I would be very happy if would understand a little bit more to be more confident in using docker & nipype.

2.) I have the feeling that WSL/WSL2 is often mentioned in the context of docker, nipype and Windows. When and why is it important to install WSL on my Windows 10 machine? For example your wrote:

When running this command in your shell (whatever that is in windows10, NB: depending on your precise version you need docker toolbox or docker desktop , WSL , etc., please check this section from the MTL Brainhack School )

From what I’ve tried it seems that I could do everything what I wanted without having to install WSL on my machine (which is not straightforward by the way on my machine due to extremely strict IT-restrictions, e.g. I am not even able to open the Microsoft Store). So when and why do I need WSL? Also I thought the whole magic behind Docker was to be not be dependent on the OS? Isn’t it paradoxical then that you still have to install WSL on your Windows computer?

3.) I haven’t really understood why one needs to type in the following lines when running a docker image: -p 8888:8888.

Even after trying to get a better intuition on docker and ports (by doing the do the most straightforward thing: ‘what are ports docker?’) I haven’t really got satisfactory answers to get around this…I have the feeling that I get circular answers like “docker uses ports and that’s why you have to do type this in” instead of really understanding why I have to do this (where are these numbers like 8888:8888 coming from?). Of course I don’t want to blame anyone, maybe (probably) the whole thing is simply not straightfoward to explain in a few sentences to a IT-newbie like me.

4.) When running a docker image, one always seems to be provided with three links to choose from. The display looks something like this:

To access the notebook, open this file in a browser:
        file://foo
    Or copy and paste one of these URLs:
        http://bar
     or http://foobar

Whenever I run a container, always only the last link works while the first two just lead to dead ends. Again, this is not a big problem, because I could do what I wanted, but why do I get provided with three links in the first place, and am I the only one that can always only use the last link?

Outro:

Again, I hope that this reply is not a complete overkill, while writing this I also thought about breaking all those questions up into separate neurostars questions but then I thought, since this question already has such a generic title (“Using nipype via docker”) it might be also a good idea to post them all in here.

Greetings,

Johannes