FMRIPREP from Singularity: UnboundLocalError

[Gilles_de_Hollander]

Nope

https://gist.github.com/Gilles86/75c3b848f1973d947f0838a1020f7f76#file-fmriprep-1-3-0-post1

[rwblair]

The permission error is coming from git-annex not being able to write to /opt/templateflow/ in the container. If we bind a directory from the host machine into /opt/templateflow/ we should be able to write to it.

First we need a directory to mount to:
mkdir $HOME/templateflow

Then singularity command would look something like:
singularity run -B $HOME/templateflow:/opt/templateflow fmriprep-1.3.0.post1.simg /home/gholland/data/bias_task/fmriprep_sourcedata/ /home/gholland/data/bias_task/derivatives/ participant --participant-label 01 --fs-license-file ~/license.txt

This will put a directory in your home directory called templateworkflow where the downloaded templates will be stored. Any other calls to fmriprep via singularity would need to bind this directory to use the downloaded templates.

@Gilles_de_Hollander can you try running with this command again?

Couple of things I noticed while looking at singularity documentation. First is that in 3.0 they recommend setting an environment variable for default bindings:
export SINGULARITY_BIND="/opt,/data:/mnt"

I didn’t notice that recommendation in previous editions, and I’d be curious to see if that environment variable is set on your HPC. Then there was the release note for 2.6.1:

disables instance features for mount commands, disables instance join for start command, and disables daemon start for action commands

If we ever see issues with only 2.6.1 related to permissions this might need to look into the specific changes singularity made.

[Benjamin_Zimmerman]

I was having the same issue as @Gilles_de_Hollander, except that I was having trouble with Singularity 2.6.1-dist. Your suggestion of binding a directory into /opt/templateflow/ worked for me, with fmriprep-1.3.0.post1.

[oesteban]

Any progress here? @PhilK, @Gilles_de_Hollander

[effigies]

Just to take a survey, how is everybody creating and running their singularity images? Please respond with:

1. Your command to build the container, e.g. singularity build ... or docker run ... docker2singularity ...
2. Your HPC environment (TACC, Sherlock, etc.)
3. Your execution command (e.g. singularity run ...)

@philk @Gilles_de_Hollander @Benjamin_Zimmerman @TribikramT @dlevitas

Also anybody else running into this problem. I’m going to try to resolve this issue for good, and I need to be able to reproduce the problem.

[dlevitas]

1).

docker run --privileged -t --rm \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /host/path/where/to/output/singularity/image:/output \
    singularityware/docker2singularity \
    poldracklab/fmriprep:1.3.0

scp -r *fmriprep*1.3.0 server/account/home/dir

2). HPC environment is Carbonate

3).

unset PYTHONPATH; singularity run -B $HOME/templateflow:/opt/templateflow $HOME/*fmriprep*img ...

I’m currently running it, and haven’t noticed any errors, but I’ll wait until it’s finished to check again.
Using singularity version 2.6.1

[Benjamin_Zimmerman]

1. singularity build my_images/fmriprep-latest.simg docker://poldracklab/fmriprep:latest
   (this step worked fine)

2. I’m not sure. They’re in the process of setting up the cluster here.

3. singularity run --cleanenv -B -B $HOME/templateflow:/opt/templateflow my_images/fmriprep-latest.simg Documents/fmriprep_test/Nifti/ Documents/fmriprep_test/output_singularity/ participant --ignore fieldmaps --fs-license-file $FS_LICENSE

(this worked, but gives a permissions error without -B $HOME/templateflow:/opt/templateflow )

[effigies]

Hi all, can you try with poldracklab/fmriprep:tfpatch? From my own experiment, I was able to reproduce the error you had with a new singularity image built from poldracklab/fmriprep:1.3.0.post1, and I did not see it with the test image.

There is a possibility that this is time-dependent, however. It may be that git annex only attempts to write to the lockfile if enough time has elapsed that it thinks it should check whether it’s up-to-date, and thus this is only “fixed” in the sense that I generated the image and tested it quickly enough. I will check again tomorrow, to see if it continues to work for me.

If this works for everyone, I will release 1.3.1 tomorrow.

[TribikramT]

Hi @effigies

Please see the answers below:

1) Command to build the container: /mnt/singularity/tmp SINGULARITY_LOCALCACHEDIR=/mnt/singularity/tmp SINGULARITY_CACHEDIR=/mnt/singularity/tmp /usr/local/singularity/3.0.2/bin/singularity build fmriprep-1.3.0.simg docker://poldracklab/fmriprep:1.3.0

2) HPC environment: https://docs.massive.org.au/M3/m3users.html

3) Execution command: --cleanenv -B /projects:/projects -B /scratch:/scratch /usr/local/fmriprep/1.3.0/bin/fmriprep-1.3.0.simg fmriprep <args>

[dlevitas]

I tried the tfpatch using my step #1, which failed

Status: Downloaded newer image for poldracklab/fmriprep:tfpatch
Size: 19192 MB for the singularity container
(1/9) Creating an empty image...
Creating a sparse image with a maximum size of 19192MiB...
Using given image size of 19192
Formatting image (/sbin/mkfs.ext3)
Done. Image can be found at: /tmp/poldracklab_fmriprep_tfpatch-2019-02-13-e53e60559a90.img
(2/9) Importing filesystem...
tar: opt/freesurfer/bin/mri_make_register: Cannot open: No error information
tar: opt/freesurfer/bin/mri_mark_temporal_lobe: Cannot open: No error information
tar: opt/freesurfer/bin/mri_mergelabels: Cannot open: No error information
tar: opt/freesurfer/bin/mri_motion_correct2: Cannot open: No error information
tar: opt/freesurfer/bin/mri_ms_EM: Cannot open: No error information
tar: opt/freesurfer/bin/mri_ms_LDA: Cannot open: No error information
tar: opt/freesurfer/bin/mri_multiscale_segment: Cannot open: No error information
tar: opt/freesurfer/bin/mri_parse_sdcmdir: Cannot open: No error information
tar: opt/freesurfer/bin/mri_path2label: Cannot open: No error information
tar: opt/freesurfer/bin/mri_probedicom: Cannot open: No error information
tar: opt/freesurfer/bin/mri_reorient_LR.csh: Cannot open: No error information
tar: opt/freesurfer/bin/mri_rf_label: Cannot open: No error information

...

tar: usr: Cannot mkdir: Read-only file system
tar: usr/local/miniconda/lib/python3.7/site-packages/nilearn/regions/signal_extraction.py: Cannot open: No such file or directory
tar: usr: Cannot mkdir: Read-only file system
tar: usr/local/miniconda/lib/python3.7/site-packages/nilearn/regions/tests: Cannot mkdir: No such file or directory
tar: usr: Cannot mkdir: Read-only file system
tar: usr/local/miniconda/lib/python3.7/site-packages/nilearn/regions/tests/__init__.py: Cannot open: No such file or directory
tar: usr: Cannot mkdir: Read-only file system
tar: usr/local/miniconda/lib/python3.7/site-packages/nilearn/regions/tests/__pycache__: Cannot mkdir: No such file or directory
tar: usr: Cannot mkdir: Read-only file system
tar: usr/local/miniconda/lib/python3.7/site-packages/nilearn/regions/tests/__pycache__/__init__.cpython-37.pyc: Cannot open: No such file or directory
tar: usr: Cannot mkdir: Read-only file system

I tried using Singularity (singularity build $HOME/fmriprep-tfpatch.simg docker://poldracklab/fmriprep:tfpatch) but got this error after ~26min:

Write failed because Disk quota exceeded

FATAL ERROR:Failed to write to output filesystem
ERROR: Failed squashing image, left template directory at: /tmp/.singularity-build.IHhBwX

I find this odd, because I submitted this as a job with 80gb vmem. This is my first attempt at generating an image with Singularity though, so I’m unfamiliar with how it normally works.

[Gilles_de_Hollander]

1. Your command to build the container, e.g. singularity build ... or docker run ... docker2singularity ...

singularity build fmriprep-1.3.0.post1.simg docker://poldracklab/fmriprep:1.3.0.post1

2. Your HPC environment (TACC, Sherlock, etc.)

SLURM ?
https://userinfo.surfsara.nl/systems/cartesius/description

3. Your execution command (e.g. singularity run ... )
   singularity run fmriprep-1.3.0.simg /home/gholland/data/bias_task/fmriprep_sourcedata/

/home/gholland/data/bias_task/derivatives/ participant --participant-label 01 --fs-license-file ~/license.txt

[Gilles_de_Hollander]

Hi all, can you try with poldracklab/fmriprep:tfpatch ? From my own experiment, I was able to reproduce the error you had with a new singularity image built from poldracklab/fmriprep:1.3.0.post1 , and I did not see it with the test image.

This solves the issue for me!

[effigies]

Is the UnboundLocalError resolved as well?

[Gilles_de_Hollander]

Yes. Fmriprep now starts to run properly on my data…

[oesteban]

This has been a puzzling one. Thank you all for your patience and collaborative spirit!

@PhilK, @Benjamin_Zimmerman can you confirm the good news on your end?

[Benjamin_Zimmerman]

Yes! Good news confirmed. Fmriprep:tfpatch is working for me.Thanks for your work!

[PhilK]

@effigies @oesteban It’s now running successfully for me as well with tfpatch! Thanks a lot for your fantastic work!