Fmriprep NodeExecutionError fsl2itk

Software Support
1

Summary of what happened:

Hello,

I’m running the NKI RS dataset through fmriprep using the precomputed fastsurfer (deep learning implementation of freesurfer) outputs. While processing a total of 2353 sessions through fmriprep in parallel, most of the jobs (> 2000) fail with the exact same crash report (see below). I’m unsure what the issue might be and would appreciate some guidance in understanding and fixing this issue.

Command used (and if a helper script was used, a link to the helper script or the command generated):

fmriprep /lscratch/$SLURM_JOB_ID/sub-A00051691_ses-BAS1/input \
/lscratch/$SLURM_JOB_ID/sub-A00051691_ses-BAS1/fmriprep_output/ \
participant \
-w /lscratch/$SLURM_JOB_ID/sub-A00051691_ses-BAS1/wrk/sub-A00051691 \
--fs-subjects-dir /lscratch/$SLURM_JOB_ID/sub-A00051691_ses-BAS1/fs_tmpdir/ses-BAS1/ \
--nthreads 10 \
--participant-label sub-A00051691 \
--output-spaces fsaverage6 fsLR MNI152NLin2009cAsym \
--fs-no-reconall \
--skip_bids_validation --clean-workdir

Version:

fMRIPrep version 23.1.4

Environment (Docker, Singularity / Apptainer, custom installation):

fmriprep installed by HPC admins. I access it by module load fmriprep and run the command above. Under the hood, I believe it’s a singularity container.

Following are the environment variables set during processing:

export SINGULARITY_CACHEDIR=/data/arshithab/.cache
export SINGULARITY_TMPDIR=/data/arshithab/singularity_tmp
export SINGULARITY_BINDPATH=/gpfs,/spin1,/fdb,/data,/lscratch,/vf

Data formatted according to a validatable standard? Please provide the output of the validator:

BIDS validator is currently running on the dataset and since it’s about 3TB in size, it’s taking a while. I’ll add the validator results as soon as it completes.

Relevant log outputs (up to 20 lines):

Node: fmriprep_23_1_wf.single_subject_A00051691_wf.func_preproc_ses_BAS1_task_rest_acq_645_wf.bold_hmc_wf.fsl2itk
Working directory: /lscratch/22720357/sub-A00051691_ses-BAS1/wrk/sub-A00051691/fmriprep_23_1_wf/single_subject_A00051691_wf/func_preproc_ses_BAS1_task_rest_acq_645_wf/bold_hmc_wf/fsl2itk

Node inputs:

in_files = <undefined>
in_reference = <undefined>
in_source = <undefined>
num_threads = 8

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node fsl2itk.

Traceback:
        Traceback (most recent call last):
          File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
            setattr(outputs, key, val)
          File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
            value = super(File, self).validate(objekt, name, value, return_pathlike=True)
          File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
            self.error(objekt, name, str(value))
          File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/traits/base_trait_handler.py", line 74, in error
            raise TraitError(
        traits.trait_errors.TraitError: The 'itk_transform' trait of a C3dAffineToolOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/lscratch/22720357/sub-A00051691_ses-BAS1/wrk/sub-A00051691/fmriprep_23_1_wf/single_subject_A00051691_wf/func_preproc_ses_BAS1_task_rest_acq_645_wf/bold_hmc_wf/fsl2itk/tmp-1jc7jg9s/MAT_0113_itk-00113.txt' <class 'str'> was specified.

        During handling of the above exception, another exception occurred:

        Traceback (most recent call last):
          File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 397, in run
            runtime = self._run_interface(runtime)
          File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/niworkflows/interfaces/itk.py", line 78, in _run_interface
            itk_outs = list(
          File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
            yield _result_or_cancel(fs.pop())
          File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
            return fut.result(timeout)
          File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/_base.py", line 458, in result
            return self.__get_result()
          File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
            raise self._exception
          File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/thread.py", line 58, in run
            result = self.fn(*self.args, **self.kwargs)
          File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/niworkflows/interfaces/itk.py", line 225, in _mat2itk
            ).run()
          File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 400, in run
            outputs = self.aggregate_outputs(runtime)
          File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
            raise FileNotFoundError(msg)
        FileNotFoundError: No such file or directory '/lscratch/22720357/sub-A00051691_ses-BAS1/wrk/sub-A00051691/fmriprep_23_1_wf/single_subject_A00051691_wf/func_preproc_ses_BAS1_task_rest_acq_645_wf/bold_hmc_wf/fsl2itk/tmp-1jc7jg9s/MAT_0113_itk-00113.txt' for output 'itk_transform' of a C3dAffineTool interface

Screenshots / relevant information:

  1. The dataset is bids valid.
  2. I’m independently processing every session as if it’s single subject. To implement this, I’m using the workflow discussed here
  3. I delete the working directory after each session has been processed.
  4. Each slurm job is allocated with 32g of memory and 16 cpus for processing one session through fmriprep

Thanks,
Arsh

2

Hi @Arsh,

Does the error persist with the most recent version (23.2.1 at this time)? Also it looks like you removed the part of the software support template that asks what kind of environment you are running fmriprep in. Please provide that information and we encourage not to leave out that information in the future.

Best,
Steven

3

I haven’t tried it with the latest version, but I’ll try it out now.

And apologies for not following the template, I used “Software Support” tag after drafting the message so the template didn’t show up. I’ll give it another shot.

4

Thanks for editing the original post, I was doing the same but you beat me to it. I’ve updated it with environment information though.

5

I don’t see the environment information there at the moment.

6

Sorry, my evening brain is not being very sharp. I’ve updated the post now. thank you for your patience!

7

Hi Steven - Using the latest version of fmriprep (v23.2.1) via singularity container worked without errors. For future reference, here’s a sample command I used to process single session of the nki dataset

singularity run --cleanenv \
-B /lscratch/$SLURM_JOB_ID/sub-A00054895:/tmpdir \
-B /home/arshithab:/home /data/MLDSST/containers/fmriprep-23.2.1.simg \
/tmpdir/input /tmpdir/fmriprep_outputs participant \
-w /tmpdir/work \
--fs-license-file /home/fs_license.txt \
--fs-subjects-dir /tmpdir/fs_tmpdir/ses-BAS1/ \
--nthreads 10 \
--participant-label sub-A00054895 \
--output-spaces fsaverage6 fsLR MNI152NLin2009cAsym \
--mem_mb 24000 \
--fs-no-reconall --skip_bids_validation \
--clean-workdir --stop-on-first-crash --notrack