Error creating singularity image for fmriprep

Hey neurostars,
I am stuck while trying to create a singularity image as described in the fmriprep.readthedocs.io
Specifically, I try

docker run --privileged -t --rm     -v /var/run/docker.sock:/var/run/docker.sock     -v /Users/Ralf/Desktop/image:/output     singularityware/docker2singularity     poldracklab/fmriprep:latest

and this starts nicely, but then results in thousands of mkdir/read/write errors:

Done. Image can be found at: /tmp/poldracklab_fmriprep_latest-2017-12-02-cce608d015a5.img

(2/9) Importing filesystemā€¦
tar: usr/lib/apt: Cannot mkdir: No error information
tar: usr/lib/apt: Cannot mkdir: Read-only file system
ā€¦
tar: bin: Cannot change mode to rwxr-xr-x: Read-only file system
tar: Exiting with failure status due to previous errors
cas3401schmaelzle:singularity-vm Ralf$

I tried to sudo and various other things I was capable off, but could not get it to work
Thanks Ralf.

I was not able to replicate this issue (using Docker for Windows). Did you try restarting Docker and trying again?

Hi Chris,
thanks for the quick reply. I tried restarting, but to no avail. I also updated Docker to the most recent, but it persisted. I am working on a Mac Sierra, which might have to do with it.

Will retry periodically, and perhaps on other machines - couldnā€™t find anything related in issues/stackoverflow though. If sb runs into solution, Iā€™d be grateful.

Otherwise Iā€™ll try to work around ā€¦As far as I understand, I should be able to copy a singularity image prepared by sb else to the cluster where I intend to use it, right?.. But getting that to run (it complains about bindings of folder from the clusterā€™s filesystem) is a different challengeā€¦

thanks & best ralf.

Have you tried converting other images? For example ubuntu or poldracklab/mriqc. Are you getting the same errors?

Yes - images are portable. You can use one created by someone else.

thank you. That helped. After minor hickups, trying to build mriqc led me to a solution path: At first it complained that the docker image was full/out of free space. I searched for help on that and found mentions of this issue for MAC, plus a hack suggesting to delete the Docker.qcow2. This iwill require re-downloading things in the future and thus slow down pace, but at least it works. The mriqc container built successfully, and so did the fmriprep afterwards.
now on to getting it to run on the cluster ā€¦
thanks and best r

1 Like

I am continuing this thread (not sure I should open a new one).
After overcoming a few issues regarding docker2singularity, I finally managed to get the singularity-version of fmriprep going. It produces some output in the anatomy folders , but them seems to crash at the end of the anatomical processing and before the functionals. In the func folder, I have realignment paramenters, but no normalized/preprocessed functional data.
The error is

File ā€œ/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/interfaces/mni.pyā€, line 147, in _run_interface
term_out = _write_outputs(interface_result.runtime,
AttributeError: ā€˜NoneTypeā€™ object has no attribute ā€˜runtimeā€™

Does anyone have an idea what to do? I searched stackoverflow, but could really get at what the runtime means.
Thanks a ton and Best, Ralf.

Hi Ralf,

Would it be possible for you to share some of these data you are using? Iā€™ll try to replicate on my own and see if there is something we should fix.

Hi Oesteban.
I will send you a link to the email address I found on github (code@oscaresteā€¦). Of note, I can successfully run the same dataset locally with fmriprep-docker. After I noticed I did that with fmriprep0.7, I updated, faced some new freesurfer license issues, but finally got it to work and it also ran successfully. It thus seems that the error is somehow hidden in my singularity-docker-cluster setup (which I get via ā€œsingularity pull https://github.com/rschmaelzle/fmriprepā€). But as said, it starts running ok and then hits an error somewhere during the anatomy.
Thanks a ton
Best Ralf

Hi Ralf, Iā€™ve received the invitation to your data. Iā€™ll have a look ASAP. Thanks a lot!

Hi all,
I went to our clusterā€™s user support to look into the singularity issue ā€¦ Given that I was able to run successfully via Docker (but not via singularity), I would not be surprised if it ran for you if using Docker or ā€œyourā€ signularity on your cluster (not sure what happens if you pull my singularity: singularity pull shub://rschmaelzle/fmriprep - that might be worth a try). Based on their advice, we also did ā€œmodule purgeā€ to get rid of any paths that may linger in the system. However, this did not solve the issue and I am getting the same error message in niworkflows/interfaces/mni.py, indicating that ā€œAttributeError: ā€˜NoneTypeā€™ object has no attribute ā€˜runtimeā€™ā€
From the surface, it looked like this thread could be related: https://github.com/poldracklab/mriqc/issues/433, but this is just a guess.
Best, R

Hi @rschmaelzle, sorry for the lack of response.

Since your last post we have released a lot of new FMRIPREP versions. In particular, the error you submitted looks like a problem related to the underlying Nipype of FMRIPREP. That is one of the most changing elements across these releases.

Although I could not look into your data for the problem, Iā€™d bet this problem should be fixed now. Could you please check the latest release (1.0.5) and check real quick?.

Otherwise I can try to replicate with your data, but weā€™ve been really busy since you posted and for now that does not seem to change. Iā€™ll raise the priority if you check yourself and this is still happening.

Again, sorry for the silence and sorry I canā€™t be more helpful.

Cheers,
Oscar

Hi @oesteban & all
no worries, I can always work on my local machine ā€¦ but in the long run Iā€™d hope to get all the fancy new tools to work on our cluster ā€¦
I just rebuilt the singularity image for the updated fmriprep and reran the code, but again hit the error:
Standard error:

Return code: 137.
180129-02:49:46,557 workflow WARNING:
 [Node] Error on "fmriprep_wf.single_subject_101_wf.anat_preproc_wf.t1_2_mni" (/mnt/home/schmaelz/work/fmriprep_wf/single_subject_101_wf/anat_preproc_wf/t1_2_mni)
Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 11, in <module>
load_entry_point('fmriprep==1.0.5', 'console_scripts', 'fmriprep')()
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py", line 270, in main
fmriprep_wf.run(**plugin_settings)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/workflows.py", line 602, in run
runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/base.py", line 168, in run
self._clean_queue(jobid, graph, result=result))
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/base.py", line 227, in _clean_queue
raise RuntimeError("".join(result['traceback']))
RuntimeError: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 68, in run_node
result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 487, in run
result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 571, in _run_interface
return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 650, in _run_command
result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base/core.py", line 510, in run
runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/interfaces/report_base.py", line 54, in _run_interface
ReportCapableInterface, self)._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/interfaces/mni.py", line 147, in _run_interface
term_out = _write_outputs(interface_result.runtime,
AttributeError: 'NoneType' object has no attribute 'runtime'

[schmaelz@dev-intel14 ~]$

I am fairly confident that the dataset would run on your singularity, but have little clue why it doesnā€™t on mine. As said, it is not too time-critical, but I want to report here also in case others hit similar issues.
best, r

I wonder if this could be caused by environment variables on your host machine. Could you try passing -e to singularity when trying to run the container? https://github.com/singularityware/singularity/issues/445

It also doesnā€™t hurt to make sure your dataset is BIDS compliant using the latest version of the BIDS Validator.

@rschmaelzle - we found the source of the problem. It should be fixed in version 1.0.7

@ChrisGorgolewski thank you. I rebuild with version 1.0.7 and was able to move beyond this error! Thanks!

Unfortuntately, however, fmriprep still doesnā€™t run through with my data. I still fails in mni.py with the ā€œRuntimeError: Robust spatial normalization failed after 3 retries.ā€ I will try a dataset from openneuro or so to make sure this is not dataset specific (though, as said, it ran in local (noncluster) docker-fmriprep).

If anyone has a great idea ā€¦
Best R

If you could upload an example subject to http://OpenNeuro.org we can try looking into finding a solution.

We run into something similar in MRIQC recently. The problem was incorrect qform/sform of the input data. Make sure you are running it on raw data and using a decent dicom to nifti converter (such as dcm2niix).

@ChrisGorgolewski thanks. I reran both, the subject in question, as well as one from the bart task from openneuro using fmriprep-docker - successfully. However, attempting to run them via singularity (fmriprep1.07 container) gives the same error (failed after 3 retries). Of note, the output for my dataset (processed via docker) mentions that sform has been overwritten by qform, no such mention is made in the bart-output. Best Ralf

Do you have PYTHONPATH variable set in the environment you run singularity on? Are you running singularity and Docker on the same system?

Thanks Oscar and Chris. It now works on our cluster. It had worked on docker, and after the recent update, the remaining error while using singularity was a glitch on our side, having to do with too large computing requests. Now it runs and all is well. Thank you.
best r

2 Likes