Jupyter notebook in singularity image

jupyter
singularity

#1

Hey,

I have started looking into the possibility of running a jupyter notebook from inside a singularity container. I know that jupyter has docker recipes and containers ready to go but I plan to run this on a machine that only allows singularity. Someone suggested neurodocker to me which looks very nice but doesn’t seem to have any singularity recipes that either exposes ports (like the 8888 port used by the default notebook) or runs notebooks.

I don’t know my way around singularity very well but it seems to me that exposing ports is something that is not intended by design. Has anyone found a way around that or is that just a bad plan in general?

Thanks for any advice,
Seb


#2

Hi, Seb – I do exactly what you’re suggesting.

On the remote computing site:

  1. singularity exec -B </drive-to-bind:/drive-to-bind> <my_singularity_image> /neurodocker/startup.sh jupyter-notebook --ip=* &

Then, from a local machine, set up the ssh tunnel:

  1. ssh -N -f -L localhost:1234:<remote_computer>:8888

Then open chrome on your localmachine and point it to: localhost:1234

and you should see your notebooks.


#3

Hey Todd,

thanks for that. I may be a bit dense but I can’t seem to get this to run even locally (I am fine with the ssh forward). Would you mind sharing a singularity recipe that you found to work with your command (whether locally or remotely). I’ll check that against mine and figure out what’s wrong.

Best,
Seb


#4

For me, the easiest thing to do was to use miykael’s nipype tutorial environment. He’s used neurodocker to create an image that has everything I needed (and much, much more). That dockerfile is here: https://hub.docker.com/r/miykael/nipype_tutorial/~/dockerfile/

You can (and perhaps should, as I’m sure it’s a valuable learning exercise) use neurodocker to create your own recipe and then specify that you’d like it to include jupyter, but if you just want to confirm the concept works, you should be able to do this:

  1. singularity pull docker://miykael/nipype_tutorial
  2. <Go get lunch while the singularity image builds>
  3. singularity exec -B /om:/om nipype_tutorial.simg /neurodocker/startup.sh jupyter-notebook --ip=* &

(/om is our file system. Substitute your own appropriately.)

When you run that, you should see messages about jupyter-notebook starting up and being grouchy about your choices about security. From there, it’s just ssh tunneling.


#5

Yeah, that’s the stuff, thank you!
Looks like my problem was that I didn’t mount any filesystem location to where jupyter wants to write (in my case /run/user). With that in place I got it to run locally. Many thanks for your help!

For posterity, here is what I did:

  1. Get basic notebook container (nipype tutorial failed for me with tar errors) edit: this seems to be a version issue. Updating from singularity 2.4.2 to 2.5.2 fixes this.
    singularity pull docker://jupyter/minimal-notebook
  2. new folder in my homedir for jupyter to have write permission in:
    mkdir -p ~/place_to_mount
  3. Mount new dir to container /run/user. That’s where my setup wants to write. It appears to create a folder with the UID of the user in the container that runs the notebook (1000 in my case).
    singularity exec -B ~/place_to_mount:/run/user minimal-notebook.simg jupyter-notebook
  4. Follow the onscreen instruction and go for
    http://localhost:8888/?token=yourspecialsecret
    in browser

#6

Great! Glad you got it to work!

Also for posterity, I also saw tar errors on the nipype_tutorial pull:
“Exploding layer: sha256:bc52bd2b04a70441db735ce4aabac9523b45a00074d67c42e55ff560d3460872.tar.gz
WARNING: Warning reading tar header: Ignoring malformed pax extended attribute”

but that didn’t stop the eventual singularity build, using singularity 2.6.0.

Your container is definitely smaller, if you don’t need any neuroimaging tools to be installed alongside jupyter.


#7

Interesting. I use singularity 2.4.2 (I think that’s from the ubuntu 18.04 repo) and it gives me this:

gzip: /home/surchs/.singularity/docker/sha256:a70e0e77f5f3656c35f85a6c6741d8e01872121dd85833e8ae843859d86d4273.tar.gz: not in gzip format
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
Cleaning up...
ERROR: pulling container failed!

So I guess I should update my singularity. Thanks!

edit: updating singularity to 2.5.2 does get rid of the error!