Fmriprep error when set --nthreads

1

Hi guys,

I ran fmriprep 0.5.2 on one of our cluster node locally. When I set --nthreads > 1 (4 cores cpu on that node), fmriprep stopped and thrown errors like below.

/usr/local/miniconda/lib/python3.6/site-packages/sklearn/externals/joblib/_multiprocessing_helpers.py:28: UserWarning: [Errno 2] No such file or directory. joblib will operate in serial mode
warnings.warn(’%s. joblib will operate in serial mode’ % (e,))
170703-21:03:24,511 workflow INFO:
Creating bold processing workflow for “/seastor/zhifang/ds_test/sub-07/func/sub-01_task-rest_bold.nii.gz”.
170703-21:03:24,610 workflow INFO:
Slice-timing correction will be included.
170703-21:03:24,961 workflow INFO:
Fieldmap estimation: type “phasediff” found
/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/fsl/preprocess.py:1597: UserWarning: This has not been fully tested. Please report any failures.
warn(‘This has not been fully tested. Please report any failures.’)
170703-21:03:26,763 workflow INFO:
Creating FreeSurfer processing flow.
Traceback (most recent call last):
File “/usr/local/miniconda/bin/fmriprep”, line 11, in
load_entry_point(‘fmriprep==0.5.2’, ‘console_scripts’, ‘fmriprep’)()
File “/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py”, line 157, in main
create_workflow(opts)
File “/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py”, line 280, in create_workflow
fmriprep_wf.run(**plugin_settings)
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/workflows.py”, line 569, in run
runner = plugin_mod(plugin_args=plugin_args)
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py”, line 161, in init
self.pool = NonDaemonPool(processes=self.processors)
File “/usr/local/miniconda/lib/python3.6/multiprocessing/pool.py”, line 150, in init
self._setup_queues()
File “/usr/local/miniconda/lib/python3.6/multiprocessing/pool.py”, line 243, in _setup_queues
self._inqueue = self._ctx.SimpleQueue()
File “/usr/local/miniconda/lib/python3.6/multiprocessing/context.py”, line 112, in SimpleQueue
return SimpleQueue(ctx=self.get_context())
File “/usr/local/miniconda/lib/python3.6/multiprocessing/queues.py”, line 324, in init
self._rlock = ctx.Lock()
File “/usr/local/miniconda/lib/python3.6/multiprocessing/context.py”, line 67, in Lock
return Lock(ctx=self.get_context())
File “/usr/local/miniconda/lib/python3.6/multiprocessing/synchronize.py”, line 163, in init
SemLock.init(self, SEMAPHORE, 1, 1, ctx=ctx)
File “/usr/local/miniconda/lib/python3.6/multiprocessing/synchronize.py”, line 60, in init
unlink_now)
FileNotFoundError: [Errno 2] No such file or directory

Here’s my command:

singularity run -e /seastor/zhifang/toolbox/bin/fmriprep.img /seastor/zhifang/ds_test /seastor/zhifang/ds_test/derivatives participant --participant_label 07 --nthreads 2 --omp-nthreads 1 --mem_mb 12000 --output-space T1w template fsnative fsaverage --template MNI152NLin2009cAsym --use-aroma --no-submm-recon -w /seastor/zhifang/ds_test/work --write-graph

I created this fmriprep image using docker2singularity.sh from https://github.com/singularityware/docker2singularity. When I ran fmriprep on my laptop with docker, everything worked fine without error.

So I don’t know this error is from docker to singularity conversion or something goes wrong on our cluster node.

Any suggestion? Thansk!

2

Interesting. Is /seastore automatically mounted by Singularity on your cluster? If so you need to pass it as a mount point when creating a container using docker2singularity (see https://github.com/singularityware/docker2singularity/blob/master/README.md#my-clusterhpc-requires-singularity-images-to-include-specific-mount-points).

3

I didn’t include path mounting during docker2singularity conversion. Instead I modified Singularity config file on our cluster, so /seastor will always auto-mount to any singularity image.

I do find a workaround to this problem after try and error. I disable /tmp auto binding in singularity config file and bind it to a user defined directory instead of host’s /tmp. Multiprocessing seems works fine now. But I still don’t know what happens…
Also, I encountered another error during recon-all processing. tkregister2 thrown an error that shared libraries libGL.so.1 was not found. I’m trying to export /usr/lib/x86_64-linux-gnu/mesa to $LD_LIBRARY_PATH and hopefully could fix the error. Any suggestion?

Thanks!

4

Sorry. I found in the end I didn’t just disable /tmp auto binding. Actually I use -e -c option that should disable all path binding and clean host env. And I only use -B /seastor:/seastor and -B /user-defined/path:/tmp.

1 Like