fMRIPrep 1.3.x on Singularity - git-annex permission errors

[Gilles_de_Hollander]

A new error

https://gist.github.com/Gilles86/75c3b848f1973d947f0838a1020f7f76

[effigies]

Ah, thanks. We really need to start directly testing Singularity images; this is a permissions issue related to the fact that files generated by Docker are not generally writeable. In a few hours we should have an :unstable image you can try.

[effigies]

Can you try with poldracklab/fmriprep:unstable?

[oesteban]

I’ve just triggered the deployment of a new post release (version 1.3.0.post1) which should not have this problem (I’ve tested it myself and could not replicate). It should be ready in a few hours (when all these steps should are green - https://circleci.com/workflow-run/ed055daa-988f-4ac1-8bea-7c610009f54a).

[Gilles_de_Hollander]

Nope

https://gist.github.com/Gilles86/75c3b848f1973d947f0838a1020f7f76#file-fmriprep-1-3-0-post1

[rwblair]

The permission error is coming from git-annex not being able to write to /opt/templateflow/ in the container. If we bind a directory from the host machine into /opt/templateflow/ we should be able to write to it.

First we need a directory to mount to:
mkdir $HOME/templateflow

Then singularity command would look something like:

singularity run -B $HOME/templateflow:/opt/templateflow \
                fmriprep-1.3.0.post1.simg /home/gholland/data/bias_task/fmriprep_sourcedata/ \
                                                          /home/gholland/data/bias_task/derivatives/ \
                participant --participant-label 01 --fs-license-file ~/license.txt

This will put a directory in your home directory called templateworkflow where the downloaded templates will be stored. Any other calls to fmriprep via singularity would need to bind this directory to use the downloaded templates.

@Gilles_de_Hollander can you try running with this command again?

Couple of things I noticed while looking at singularity documentation. First is that in 3.0 they recommend setting an environment variable for default bindings:
export SINGULARITY_BIND="/opt,/data:/mnt"

I didn’t notice that recommendation in previous editions, and I’d be curious to see if that environment variable is set on your HPC. Then there was the release note for 2.6.1:

disables instance features for mount commands, disables instance join for start command, and disables daemon start for action commands

If we ever see issues with only 2.6.1 related to permissions this might need to look into the specific changes singularity made.

[Benjamin_Zimmerman]

I was having the same issue as @Gilles_de_Hollander, except that I was having trouble with Singularity 2.6.1-dist. Your suggestion of binding a directory into /opt/templateflow/ worked for me, with fmriprep-1.3.0.post1.

[oesteban]

Any progress here? @PhilK, @Gilles_de_Hollander

[effigies]

Just to take a survey, how is everybody creating and running their singularity images? Please respond with:

1. Your command to build the container, e.g. singularity build ... or docker run ... docker2singularity ...
2. Your HPC environment (TACC, Sherlock, etc.)
3. Your execution command (e.g. singularity run ...)

@philk @Gilles_de_Hollander @Benjamin_Zimmerman @TribikramT @dlevitas

Also anybody else running into this problem. I’m going to try to resolve this issue for good, and I need to be able to reproduce the problem.

[dlevitas]

1).

docker run --privileged -t --rm \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /host/path/where/to/output/singularity/image:/output \
    singularityware/docker2singularity \
    poldracklab/fmriprep:1.3.0

scp -r *fmriprep*1.3.0 server/account/home/dir

2). HPC environment is Carbonate

3).

unset PYTHONPATH; singularity run -B $HOME/templateflow:/opt/templateflow $HOME/*fmriprep*img ...

I’m currently running it, and haven’t noticed any errors, but I’ll wait until it’s finished to check again.
Using singularity version 2.6.1

[Benjamin_Zimmerman]

1. singularity build my_images/fmriprep-latest.simg docker://poldracklab/fmriprep:latest
   (this step worked fine)

2. I’m not sure. They’re in the process of setting up the cluster here.

3. singularity run --cleanenv -B -B $HOME/templateflow:/opt/templateflow my_images/fmriprep-latest.simg Documents/fmriprep_test/Nifti/ Documents/fmriprep_test/output_singularity/ participant --ignore fieldmaps --fs-license-file $FS_LICENSE

(this worked, but gives a permissions error without -B $HOME/templateflow:/opt/templateflow )

[effigies]

Hi all, can you try with poldracklab/fmriprep:tfpatch? From my own experiment, I was able to reproduce the error you had with a new singularity image built from poldracklab/fmriprep:1.3.0.post1, and I did not see it with the test image.

There is a possibility that this is time-dependent, however. It may be that git annex only attempts to write to the lockfile if enough time has elapsed that it thinks it should check whether it’s up-to-date, and thus this is only “fixed” in the sense that I generated the image and tested it quickly enough. I will check again tomorrow, to see if it continues to work for me.

If this works for everyone, I will release 1.3.1 tomorrow.

[TribikramT]

Hi @effigies

Please see the answers below:

1) Command to build the container: /mnt/singularity/tmp SINGULARITY_LOCALCACHEDIR=/mnt/singularity/tmp SINGULARITY_CACHEDIR=/mnt/singularity/tmp /usr/local/singularity/3.0.2/bin/singularity build fmriprep-1.3.0.simg docker://poldracklab/fmriprep:1.3.0

2) HPC environment: https://docs.massive.org.au/M3/m3users.html

3) Execution command: --cleanenv -B /projects:/projects -B /scratch:/scratch /usr/local/fmriprep/1.3.0/bin/fmriprep-1.3.0.simg fmriprep <args>

[dlevitas]

I tried the tfpatch using my step #1, which failed

Status: Downloaded newer image for poldracklab/fmriprep:tfpatch
Size: 19192 MB for the singularity container
(1/9) Creating an empty image...
Creating a sparse image with a maximum size of 19192MiB...
Using given image size of 19192
Formatting image (/sbin/mkfs.ext3)
Done. Image can be found at: /tmp/poldracklab_fmriprep_tfpatch-2019-02-13-e53e60559a90.img
(2/9) Importing filesystem...
tar: opt/freesurfer/bin/mri_make_register: Cannot open: No error information
tar: opt/freesurfer/bin/mri_mark_temporal_lobe: Cannot open: No error information
tar: opt/freesurfer/bin/mri_mergelabels: Cannot open: No error information
tar: opt/freesurfer/bin/mri_motion_correct2: Cannot open: No error information
tar: opt/freesurfer/bin/mri_ms_EM: Cannot open: No error information
tar: opt/freesurfer/bin/mri_ms_LDA: Cannot open: No error information
tar: opt/freesurfer/bin/mri_multiscale_segment: Cannot open: No error information
tar: opt/freesurfer/bin/mri_parse_sdcmdir: Cannot open: No error information
tar: opt/freesurfer/bin/mri_path2label: Cannot open: No error information
tar: opt/freesurfer/bin/mri_probedicom: Cannot open: No error information
tar: opt/freesurfer/bin/mri_reorient_LR.csh: Cannot open: No error information
tar: opt/freesurfer/bin/mri_rf_label: Cannot open: No error information

...

tar: usr: Cannot mkdir: Read-only file system
tar: usr/local/miniconda/lib/python3.7/site-packages/nilearn/regions/signal_extraction.py: Cannot open: No such file or directory
tar: usr: Cannot mkdir: Read-only file system
tar: usr/local/miniconda/lib/python3.7/site-packages/nilearn/regions/tests: Cannot mkdir: No such file or directory
tar: usr: Cannot mkdir: Read-only file system
tar: usr/local/miniconda/lib/python3.7/site-packages/nilearn/regions/tests/__init__.py: Cannot open: No such file or directory
tar: usr: Cannot mkdir: Read-only file system
tar: usr/local/miniconda/lib/python3.7/site-packages/nilearn/regions/tests/__pycache__: Cannot mkdir: No such file or directory
tar: usr: Cannot mkdir: Read-only file system
tar: usr/local/miniconda/lib/python3.7/site-packages/nilearn/regions/tests/__pycache__/__init__.cpython-37.pyc: Cannot open: No such file or directory
tar: usr: Cannot mkdir: Read-only file system

I tried using Singularity (singularity build $HOME/fmriprep-tfpatch.simg docker://poldracklab/fmriprep:tfpatch) but got this error after ~26min:

Write failed because Disk quota exceeded

FATAL ERROR:Failed to write to output filesystem
ERROR: Failed squashing image, left template directory at: /tmp/.singularity-build.IHhBwX

I find this odd, because I submitted this as a job with 80gb vmem. This is my first attempt at generating an image with Singularity though, so I’m unfamiliar with how it normally works.

[Gilles_de_Hollander]

1. Your command to build the container, e.g. singularity build ... or docker run ... docker2singularity ...

singularity build fmriprep-1.3.0.post1.simg docker://poldracklab/fmriprep:1.3.0.post1

2. Your HPC environment (TACC, Sherlock, etc.)

SLURM ?
https://userinfo.surfsara.nl/systems/cartesius/description

3. Your execution command (e.g. singularity run ... )
   singularity run fmriprep-1.3.0.simg /home/gholland/data/bias_task/fmriprep_sourcedata/

/home/gholland/data/bias_task/derivatives/ participant --participant-label 01 --fs-license-file ~/license.txt

[Gilles_de_Hollander]

Hi all, can you try with poldracklab/fmriprep:tfpatch ? From my own experiment, I was able to reproduce the error you had with a new singularity image built from poldracklab/fmriprep:1.3.0.post1 , and I did not see it with the test image.

This solves the issue for me!

[orduek]

I encounter the same issue. Is there any fix for that or should I revert to 1.2.6?

[oesteban]

In this thread, two issues have been discussed. For us to help, please report further details:

• Execution framework (container, OS, versions of fmriprep, containers, etc.)
• Exact traceback of the error

[orduek]

Sure. I’m running singularity on slurm cluster.
I receive the following error:

tch60/oad4/output participant --fs-license-file /home/oad4/freesurferLicense/license.txt --participant-label 1445 --skip_bids_validation -w /gpfs/ysm/scratch60/oad4/work/
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/site-packages/nilearn/datasets/neurovault.py:16: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Container
/usr/local/miniconda/lib/python3.7/site-packages/datalad/utils.py:71: DeprecationWarning: dist() and linux_distribution() functions are deprecated in Python 3.5
  = platform.linux_distribution()[:2]
/usr/local/miniconda/lib/python3.7/site-packages/datalad/utils.py:847: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  is_decorating = not kwargs and len(args) == 1 and isinstance(args[0], collections.Callable)
/usr/local/miniconda/lib/python3.7/site-packages/datalad/support/param.py:30: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  _KNOWN_ARGS = getargspec(argparse.Action.__init__)[0] + ['action']
/usr/local/miniconda/lib/python3.7/site-packages/datalad/interface/base.py:216: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  args, varargs, varkw, defaults = getargspec(func)
/usr/local/miniconda/lib/python3.7/site-packages/skimage/__init__.py:80: ResourceWarning: unclosed file <_io.TextIOWrapper name='/usr/local/miniconda/lib/python3.7/site-packages/pytest.py' mode='r' encoding='utf-8'>
  imp.find_module('pytest')
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
190305-19:40:22,197 nipype.workflow IMPORTANT:
	 
    Running fMRIPREP version 1.3.0.post1:
      * BIDS dataset path: /gpfs/ysm/scratch60/oad4/RVLdata.
      * Participant list: ['1445'].
      * Run identifier: 20190305-194022_54002913-506e-4ed3-a099-8f5a5e7febea.
    
/usr/local/miniconda/lib/python3.7/site-packages/networkx/classes/reportviews.py:95: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Mapping, Set, Iterable
/usr/local/miniconda/lib/python3.7/site-packages/datalad/interface/base.py:480: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  argspec = getargspec(call)
/usr/local/miniconda/lib/python3.7/site-packages/datalad/interface/base.py:480: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  argspec = getargspec(call)
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 751, in build_workflow
    ignore_aroma_err=opts.ignore_aroma_denoising_errors,
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/base.py", line 218, in init_fmriprep_wf
    ignore_aroma_err=ignore_aroma_err,
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/base.py", line 461, in init_single_subject_wf
    name="anat_preproc_wf",
  File "/usr/local/miniconda/lib/python3.7/site-packages/smriprep/workflows/anatomical.py", line 237, in init_anat_preproc_wf
    omp_nthreads=omp_nthreads)
  File "/usr/local/miniconda/lib/python3.7/site-packages/smriprep/workflows/anatomical.py", line 621, in init_skullstrip_ants_wf
    t1_skull_strip.inputs.brain_template = get_template(skull_strip_template, 'res-01_T1w.nii.gz')
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 24, in get
    out_file = api.get(filepath)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/interface/utils.py", line 478, in eval_func
    return return_func(generator_func)(*args, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/interface/utils.py", line 466, in return_func
    results = list(results)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/interface/utils.py", line 421, in generator_func
    result_renderer, result_xfm, _result_filter, **_kwargs):
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/interface/utils.py", line 490, in _process_results
    for res in results:
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/distribution/get.py", line 622, in __call__
    jobs=jobs):
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/support/gitrepo.py", line 301, in newfunc
    result = func(self, files_new, *args, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/support/annexrepo.py", line 1300, in get
    files, ['--not', '--in', 'here'])
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/support/annexrepo.py", line 1361, in _get_expected_files
    'find', opts=expr, files=files
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/support/annexrepo.py", line 2381, in _run_annex_command_json
    raise e
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/support/annexrepo.py", line 2297, in _run_annex_command_json
    **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/support/annexrepo.py", line 1053, in _run_annex_command
    raise e
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/support/annexrepo.py", line 1045, in _run_annex_command
    return self.cmd_call_wrapper.run(cmd_list, env=env, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/cmd.py", line 670, in run
    cmd, env=self.get_git_environ_adjusted(env), *args, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/cmd.py", line 530, in run
    raise CommandError(str(cmd), msg, status, out[0], out[1])
datalad.support.exceptions.CommandError: CommandError: command '['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'find', '--json', '--not', '--in', 'here', '--', 'tpl-OASIS30ANTs_res-01_T1w.nii.gz']' failed with exitcode 1
Failed to run ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'find', '--json', '--not', '--in', 'here', '--', 'tpl-OASIS30ANTs_res-01_T1w.nii.gz'] under '/opt/templateflow/tpl-OASIS30ANTs'. Exit code=1. out= err=git-annex: .git/annex/journal.lck: openFd: permission denied (Permission denied)

[oesteban]

There are two possibilities to overcome this one:

1. Bind a templateflow folder:

singularity run -B $HOME/templateflow:/opt/templateflow \
            fmriprep-1.3.0.post1.simg ...

2. Wait for fmriprep-1.3.1, which should be released soon.