FMRIPREP returning OSError

1

Hello,

I’m having some issues with running fmriprep on a supercomputer using singularity. The images get created, but then fmriprep just stalls (for days) and doesn’t produce any sort of .html or figures. The error I’m posting below happens just after the bids validation step (I’ve run it without the bids validation step and still get the error). Any help on why it’s doing this would be greatly appreciated. I’ve tried googling it and I even updated fmriprep to the latest version (1.4.1) last night, but I’m still getting this error.

exception calling callback for <Future at 0x2aaad8837e48 state=finished raised OSError>
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 487, in run
    self, report_type='postexec', is_mapnode=isinstance(self, MapNode))
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 149, in write_report
    result = node.result  # Locally cache result
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 197, in result
    return _load_resultfile(self.output_dir(), self.name)[0]
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 359, in load_resultfile
    basedir=path).items()):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 478, in modify_paths
    val, relative=relative, basedir=basedir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 484, in modify_paths
    modify_paths(val, relative=relative, basedir=basedir))
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 498, in modify_paths
    raise IOError('File %s not found' % out)
OSError: File /projects/adapt_lab/shared/TPOT/bids_data/derivatives/working_test/fmriprep_wf/single_subject_TPOT004_wf/func_preproc_ses_1_task_affect_acq_1_wf/func_derivatives_wf/raw_sources/sub-TPOT004/ses-1/func/sub-TPOT004_ses-1_task-affect_acq-1_bold.nii.gz not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/concurrent/futures/process.py", line 232, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 72, in run_node
    result['result'] = node.result
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 197, in result
    return _load_resultfile(self.output_dir(), self.name)[0]
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 359, in load_resultfile
    basedir=path).items()):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 478, in modify_paths
    val, relative=relative, basedir=basedir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 484, in modify_paths
    modify_paths(val, relative=relative, basedir=basedir))
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 498, in modify_paths
    raise IOError('File %s not found' % out)
OSError: File /projects/adapt_lab/shared/TPOT/bids_data/derivatives/working_test/fmriprep_wf/single_subject_TPOT004_wf/func_preproc_ses_1_task_affect_acq_1_wf/func_derivatives_wf/raw_sources/sub-TPOT004/ses-1/func/sub-TPOT004_ses-1_task-affect_acq-1_bold.nii.gz not found
"""

Additionally, here is the code I’m using to run fmriprep.


singularity run --bind "${group_dir}":"${group_dir}" $image $bids_dir $derivatives participant \
--participant_label $subid \
 -w $working_dir \
 -t $task \
--output-spaces T1w MNI152NLin2009cAsym fsaverage5 fsnative \
--omp-nthreads 8 --nthreads 8 --mem_mb 30000 \
--write-graph \
--fs-license-file $FS_LICENSE
2

Are you parallelizing the script across subjects?

3

Yes! I have a separate batch script if you’d like to see it. Is that causing issues?

4

Seems like two processes racing. Please launch just one instance and see whether you see this error with this subject.

If you are trying to select tasks with -t it is potentially the root of the problem - it may not work. Please let us know whether -t had any effect on the processing.

5

I tried running it with only specifying one task in the argument (-t "rest") and that worked! My question is, if I have to run fmriprep for each task, how do I do that while still keeping the output .html from the previous run of fmriprep? In other words, how does sub-TPOT011.hmtl for “rest” not get overwritten when I run fmriprep for “affect” task? Thanks for your help!

6

I think you are right, unless you share the reportlets/ folder under the work directory across runs, they will be overwritten.