Fmri prep 1.5.1 crashes when running with singularity

Hi fmriprep experts,
We recently updated to the latest version of fmriprep 1.5.1. on HPC. We are running using singularity container. We are having a lot of problems with the runs crashing now. We preprocessed the same subjects with the previous version using singularity and it worked fine. The container with the same version 1.5.1 also runs fine on the local machine with docker container.

this is our singularity command:
singularity run --cleanenv --bind ${WORK}/rest_lang:/data /extra/dkp/singularity-images/fmriprep.sif /data /data/derivatives participant --participant_label ${participant} --fs-license-file ${WORK}/license.txt -w /data/work --skip_bids_validation --use-aroma

error file:
Node: fmriprep_wf.single_subject_111_wf.bids_info
Working directory: /data/work/fmriprep_wf/single_subject_111_wf/bids_inf

bids_dir = /data
bids_validate = False
in_file =

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 69, in run_node
result[‘result’] = node.run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 479, in run
result = self._run_interface(execute=True)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 569, in _run_interface
return self._run_command(execute)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 662, in _run_command
result = self._interface.run(cwd=outdir)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 382, in run
runtime = self._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/bids.py”, line 164, in _run_interface
self.inputs.bids_validate)
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/utils/bids.py”, line 213, in _init_layout
layout = BIDSLayout(str(bids_dir), validate=validate)
File “/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/layout.py”, line 218, in init
indexer.index_metadata()
File “/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/index.py”, line 207, in index_metadata
with open(bf.path, ‘r’) as handle:
FileNotFoundError: [Errno 2] No such file or directory: ‘/data/work/fmriprep_wf/single_subject_111_wf/func_preproc_task_rest_run_001_wf/bold_stc_wf/slice_timing_correction/_0xfccdd5ca72c699c768e61a3f06670a49_unfinished.json’

Hey @akielar,

As I mentioned, I’m not familiar with the HPC but does this seem like the issue you’re running into?


Based on those posts, I’m wondering if putting your work directory inside your derivatives directory would help? (so instead of /data/work/ in parallel with /data/derivatives, use /data/derivatives/work so that derivatives/work/ is in parallel with derivatives/fmriprep/

-Saren

Thanks for suggestions. It is running now.

Cool, let me know if that worked for you!