Jupyter notebook in singularity image

,
1

Hey,

I have started looking into the possibility of running a jupyter notebook from inside a singularity container. I know that jupyter has docker recipes and containers ready to go but I plan to run this on a machine that only allows singularity. Someone suggested neurodocker to me which looks very nice but doesn’t seem to have any singularity recipes that either exposes ports (like the 8888 port used by the default notebook) or runs notebooks.

I don’t know my way around singularity very well but it seems to me that exposing ports is something that is not intended by design. Has anyone found a way around that or is that just a bad plan in general?

Thanks for any advice,
Seb

2

Hi, Seb – I do exactly what you’re suggesting.

On the remote computing site:

  1. singularity exec -B </drive-to-bind:/drive-to-bind> <my_singularity_image> /neurodocker/startup.sh jupyter-notebook --ip=* &

Then, from a local machine, set up the ssh tunnel:

  1. ssh -N -f -L localhost:1234:<remote_computer>:8888

Then open chrome on your localmachine and point it to: localhost:1234

and you should see your notebooks.

1 Like
3

Hey Todd,

thanks for that. I may be a bit dense but I can’t seem to get this to run even locally (I am fine with the ssh forward). Would you mind sharing a singularity recipe that you found to work with your command (whether locally or remotely). I’ll check that against mine and figure out what’s wrong.

Best,
Seb

4

For me, the easiest thing to do was to use miykael’s nipype tutorial environment. He’s used neurodocker to create an image that has everything I needed (and much, much more). That dockerfile is here: https://hub.docker.com/r/miykael/nipype_tutorial/~/dockerfile/

You can (and perhaps should, as I’m sure it’s a valuable learning exercise) use neurodocker to create your own recipe and then specify that you’d like it to include jupyter, but if you just want to confirm the concept works, you should be able to do this:

  1. singularity pull docker://miykael/nipype_tutorial
  2. <Go get lunch while the singularity image builds>
  3. singularity exec -B /om:/om nipype_tutorial.simg /neurodocker/startup.sh jupyter-notebook --ip=* &

(/om is our file system. Substitute your own appropriately.)

When you run that, you should see messages about jupyter-notebook starting up and being grouchy about your choices about security. From there, it’s just ssh tunneling.

5

Yeah, that’s the stuff, thank you!
Looks like my problem was that I didn’t mount any filesystem location to where jupyter wants to write (in my case /run/user). With that in place I got it to run locally. Many thanks for your help!

For posterity, here is what I did:

  1. Get basic notebook container (nipype tutorial failed for me with tar errors) edit: this seems to be a version issue. Updating from singularity 2.4.2 to 2.5.2 fixes this.
    singularity pull docker://jupyter/minimal-notebook
  2. new folder in my homedir for jupyter to have write permission in:
    mkdir -p ~/place_to_mount
  3. Mount new dir to container /run/user. That’s where my setup wants to write. It appears to create a folder with the UID of the user in the container that runs the notebook (1000 in my case).
    singularity exec -B ~/place_to_mount:/run/user minimal-notebook.simg jupyter-notebook
  4. Follow the onscreen instruction and go for
    http://localhost:8888/?token=yourspecialsecret
    in browser
1 Like
6

Great! Glad you got it to work!

Also for posterity, I also saw tar errors on the nipype_tutorial pull:
“Exploding layer: sha256:bc52bd2b04a70441db735ce4aabac9523b45a00074d67c42e55ff560d3460872.tar.gz
WARNING: Warning reading tar header: Ignoring malformed pax extended attribute”

but that didn’t stop the eventual singularity build, using singularity 2.6.0.

Your container is definitely smaller, if you don’t need any neuroimaging tools to be installed alongside jupyter.

7

Interesting. I use singularity 2.4.2 (I think that’s from the ubuntu 18.04 repo) and it gives me this:

gzip: /home/surchs/.singularity/docker/sha256:a70e0e77f5f3656c35f85a6c6741d8e01872121dd85833e8ae843859d86d4273.tar.gz: not in gzip format
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
Cleaning up...
ERROR: pulling container failed!

So I guess I should update my singularity. Thanks!

edit: updating singularity to 2.5.2 does get rid of the error!

9

Hi guys,

I’m new here and hoping to revive this topic and get a little help as I’m relatively new to all this. I’d appreciate any help. My lab’s server is a centos distribution of linux and we use tmux along with singularity containers. After I shell into my singularity image in tmux, how do i get a jupyter notebook to open on my browser? jupyter is installed in the container btw. So the succession of commands is:

  1. ssh …
  2. tmux
  3. srun -p gpu -N1 --cpus-per-task=6 --gres=gpu:1 --mem=20000 --time=4-0:00:00 --pty bash
  4. singularity shell -nv --writable …sif

what do i do here now? The basic: jupyter notebook --no-browser --port=7800 gives back this error:

Traceback (most recent call last): File “/opt/conda/bin/jupyter-notebook”, line 11, in sys.exit(main()) File “/opt/conda/lib/python3.6/site-packages/jupyter_core/application.py”, line 266, in launch_instance return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs) File “/opt/conda/lib/python3.6/site-packages/traitlets/config/application.py”, line 657, in launch_instance app.initialize(argv) File “”, line 2, in initialize File “/opt/conda/lib/python3.6/site-packages/traitlets/config/application.py”, line 87, in catch_config_error return method(app, *args, **kwargs) File “/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py”, line 1628, in initialize self.init_configurables() File “/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py”, line 1319, in init_configurables connection_dir=self.runtime_dir, File “/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py”, line 556, in get return self.get(obj, cls) File “/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py”, line 535, in get value = self._validate(obj, dynamic_default()) File “/opt/conda/lib/python3.6/site-packages/jupyter_core/application.py”, line 99, in _runtime_dir_default ensure_dir_exists(rd, mode=0o700) File “/opt/conda/lib/python3.6/site-packages/jupyter_core/utils/ init .py”, line 13, in ensure_dir_exists os.makedirs(path, mode=mode) File “/opt/conda/lib/python3.6/os.py”, line 210, in makedirs makedirs(head, mode, exist_ok) File “/opt/conda/lib/python3.6/os.py”, line 220, in makedirs mkdir(name, mode) OSError: [Errno 30] Read-only file system: ‘/run/user/19241’

Thank you and sorry if this sounds easy.