Fmriprep OSErrors - File not found and Read-only file system

Hi preppers!

I’ve been trying to run a command similar to below on fmriprep1.5.0.rc1 on our HPC cluster and seem to be having some problems:

  1. An OSError in my main logfile indicating that it can’t find elementary data that I’m assuming should just be copied into the derivates folder. The script continues well after this though but eventually hangs.

  2. A permissions error that results in several crashes. I’ve seen a few errors like this reported on the site, but they seem to have only been noted when running under singularity (and now apparently fixed).

I can’t attach the logs as a new user but have pasted in relevant excerpts below. Any guidance would be super appreciated :slight_smile:

Cheers,

Arjun

A) Command:

fmriprep /data/project/abc/03_Sorted/02C_Faces_BIDS /data/project/abc/03_Sorted/02D_fmriprep_Faces participant -w derivatives --participant_label $i -t faces --longitudinal --output-spaces MNIPediatricAsym:cohort-3 --use-aroma --dummy-scans 0 --ignore fieldmaps --fs-license-file /data/project/abc/X_Scripts_AS/license.txt

B) Crash example (there are two others but all seem to reference the same issue):

Node: fmriprep_wf.single_subject_ABC004_wf.anat_preproc_wf.anat_norm_wf.msk_select
Working directory: /data/project/abc/03_Sorted/02C_Faces_BIDS/derivatives/fmriprep_wf/single_subject_ABC004_wf/anat_preproc_wf/anat_norm_wf/_template_MNIPediatricAsym/msk_select

Node inputs:

desc = brain
function_str = def _get_template(template, template_spec, suffix='T1w', desc=None):
    from niworkflows.utils.misc import get_template_specs
    template_spec['suffix'] = suffix
    template_spec['desc'] = desc
    return get_template_specs(template, template_spec=template_spec)[0]

suffix = mask
template = MNIPediatricAsym
template_spec = {'cohort': '3', 'suffix': 'mask', 'desc': 'brain', 'atlas': None, 'resolution': 1}

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 316, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 472, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 563, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 643, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 375, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/utility/wrappers.py", line 144, in _run_interface
    out = function_handle(**args)
  File "<string>", line 5, in _get_template
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/utils/misc.py", line 50, in get_template_specs
    tpl_target_path = get_template(in_template, **template_spec)
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 39, in get
    _s3_get(filepath)
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 139, in _s3_get
    with filepath.open('wb') as f:
  File "/usr/local/miniconda/lib/python3.7/pathlib.py", line 1176, in open
    opener=self._opener)
  File "/usr/local/miniconda/lib/python3.7/pathlib.py", line 1030, in _opener
    return self._accessor.open(self, flags, mode)
OSError: [Errno 30] Read-only file system: '/home/fmriprep/.cache/templateflow/tpl-MNIPediatricAsym/cohort-3/tpl-MNIPediatricAsym_cohort-3_res-1_desc-brain_mask.nii.gz'

C) File not found example. There is alot more but thought this gives the idea:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 487, in run
    self, report_type='postexec', is_mapnode=isinstance(self, MapNode))
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 149, in write_report
    result = node.result  # Locally cache result
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 197, in result
    return _load_resultfile(self.output_dir(), self.name)[0]
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 359, in load_resultfile
    basedir=path).items()):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 478, in modify_paths
    val, relative=relative, basedir=basedir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 484, in modify_paths
    modify_paths(val, relative=relative, basedir=basedir))
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 498, in modify_paths
    raise IOError('File %s not found' % out)
OSError: File /data/project/abc/03_Sorted/02C_Faces_BIDS/derivatives/fmriprep_wf/single_subject_ABC004_wf/func_preproc_ses_B_task_faces_wf/func_derivatives_wf/raw_sources/sub-ABC004/ses-B/func/sub-ABC004_ses-B_task-faces_bold.nii.gz not found

For the error message B) above (Read-only file system) it turns out the fmriprep command on our cluster was actually an alias for a singularity command, so it was the same issue as reported before.

I don’t want to get excited prematurely but it looks like it’s running now after setting the singularity environment to a writeable directory.

setenv SINGULARITYENV_TEMPLATEFLOW_HOME /SomeWritableDir/templateflow/

We have recently updated our documentation to better help Singularity users. Please check it out and do not hesitate to suggest improvements based on your experience!

https://fmriprep.readthedocs.io/en/latest/singularity.html

1 Like

Thanks, will check it out!

MANAGED BY INCF