Error occurring in the SyN node of FMRIprep

CARLOS_LEANDRO_SILVA · January 16, 2024, 6:36pm

Summary of what happened:

I am encountering an error during the execution of the ‘syn’ node in FMRIPrep pipeline . The error message indicates an issue related to the file ‘fmap_syn0Warp.nii.gz’ in the specified path. I would like to understand better why this error is occurring during the execution of the ‘syn’ node. Is there anything specific about the settings or parameters that could be causing this problem? How can I diagnose and rectify this error in the context of SyN registration?

Command used (and if a helper script was used, a link to the helper script or the command generated):

#!/bin/bash
echo Start
for sub in 003 017 024 031 041 049 054 066 076 082 092 097 106 115 123 128; do 
fmriprep-docker /media/pisa/0ABA/Elisa/bids /media/pisa/0ABA/Elisa/derivatives participant --participant-label ${sub} --fs-license-file /home/pisa/Desktop/license.txt --output-spaces MNI152NLin2009cAsym --stop-on-first-crash --write-graph --use-syn-sdc -w /media/pisa/0ABA/Elisa/workdir
done

Version:

fMRIPrep-23.1.4

Environment (Docker, Singularity, custom installation):

Docker

Data formatted according to a validatable standard? Please provide the output of the validator:

Yes

Relevant log outputs (up to 20 lines):

Node: fmriprep_23_1_wf.single_subject_002_wf.fmap_preproc_wf.wf_auto_00000.syn
Working directory: /scratch/fmriprep_23_1_wf/single_subject_002_wf/fmap_preproc_wf/wf_auto_00000/syn

Node inputs:

args = <undefined>
collapse_output_transforms = True
convergence_threshold = [1e-06, 1e-08]
convergence_window_size = [5, 2]
dimension = 3
environ = {'NSLOTS': '8'}
fixed_image = <undefined>
fixed_image_mask = <undefined>
fixed_image_masks = <undefined>
float = <undefined>
initial_moving_transform = <undefined>
initial_moving_transform_com = <undefined>
initialize_transforms_per_stage = False
interpolation = Linear
interpolation_parameters = <undefined>
invert_initial_moving_transform = <undefined>
metric = [['Mattes', 'Mattes'], ['Mattes', 'Mattes']]
metric_item_trait = <undefined>
metric_stage_trait = <undefined>
metric_weight = [[0.5, 0.5], [0.5, 0.5]]
metric_weight_item_trait = 1.0
metric_weight_stage_trait = <undefined>
moving_image = <undefined>
moving_image_mask = <undefined>
moving_image_masks = <undefined>
num_threads = 8
number_of_iterations = [[200, 100], [10]]
output_inverse_warped_image = False
output_transform_prefix = fmap_syn
output_warped_image = False
radius_bins_item_trait = 5
radius_bins_stage_trait = <undefined>
radius_or_number_of_bins = [[48, 48], [48, 48]]
random_seed = <undefined>
restore_state = <undefined>
restrict_deformation = <undefined>
sampling_percentage = [[0.8, 0.8], [1.0, 1.0]]
sampling_percentage_item_trait = <undefined>
sampling_percentage_stage_trait = <undefined>
sampling_strategy = [['Random', 'Random'], [None, None]]
sampling_strategy_item_trait = <undefined>
sampling_strategy_stage_trait = <undefined>
save_state = <undefined>
shrink_factors = [[1, 1], [1]]
sigma_units = ['vox', 'vox']
smoothing_sigmas = [[2.0, 0.0], [0.0]]
transform_parameters = [(0.8, 6.0, 3.0), (0.8, 2.0, 1.0)]
transforms = ['SyN', 'SyN']
use_estimate_learning_rate_once = <undefined>
use_histogram_matching = [True, True]
verbose = False
winsorize_lower_quantile = 0.001
winsorize_upper_quantile = 0.998
write_composite_transform = False

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node syn.

Cmdline:
	antsRegistration --collapse-output-transforms 1 --dimensionality 3 --initialize-transforms-per-stage 0 --interpolation Linear --output fmap_syn --transform SyN[ 0.8, 6.0, 3.0 ] --metric Mattes[ /scratch/fmriprep_23_1_wf/single_subject_002_wf/syn_preprocessing_auto_00000/anat2epi/clipped_noise_corrected_trans.nii.gz, /scratch/fmriprep_23_1_wf/single_subject_002_wf/fmap_preproc_wf/wf_auto_00000/clip_epi/clipped.nii.gz, 0.5, 48, Random, 0.8 ] --metric Mattes[ /scratch/fmriprep_23_1_wf/single_subject_002_wf/fmap_preproc_wf/wf_auto_00000/lap_anat_norm/clipped_noise_corrected_trans_maths_norm.nii.gz, /scratch/fmriprep_23_1_wf/single_subject_002_wf/fmap_preproc_wf/wf_auto_00000/lap_epi_norm/clipped_maths_norm.nii.gz, 0.5, 48, Random, 0.8 ] --convergence [ 200x100, 1e-06, 5 ] --smoothing-sigmas 2.0x0.0vox --shrink-factors 1x1 --use-histogram-matching 1 --restrict-deformation 0.1x1.0x0.1 --masks [ /scratch/fmriprep_23_1_wf/single_subject_002_wf/syn_preprocessing_auto_00000/mask2epi/sub-002_ses-01_T1w_noise_corrected_template_corrected_xform_rbrainmask_trans_uint8.nii, /scratch/fmriprep_23_1_wf/single_subject_002_wf/fmap_preproc_wf/wf_auto_00000/epi_umask/clipped_mask_union.nii.gz ] --transform SyN[ 0.8, 2.0, 1.0 ] --metric Mattes[ /scratch/fmriprep_23_1_wf/single_subject_002_wf/syn_preprocessing_auto_00000/anat2epi/clipped_noise_corrected_trans.nii.gz, /scratch/fmriprep_23_1_wf/single_subject_002_wf/fmap_preproc_wf/wf_auto_00000/clip_epi/clipped.nii.gz, 0.5, 48 ] --metric Mattes[ /scratch/fmriprep_23_1_wf/single_subject_002_wf/fmap_preproc_wf/wf_auto_00000/lap_anat_norm/clipped_noise_corrected_trans_maths_norm.nii.gz, /scratch/fmriprep_23_1_wf/single_subject_002_wf/fmap_preproc_wf/wf_auto_00000/lap_epi_norm/clipped_maths_norm.nii.gz, 0.5, 48 ] --convergence [ 10, 1e-08, 2 ] --smoothing-sigmas 0.0vox --shrink-factors 1 --use-histogram-matching 1 --restrict-deformation 0.1x1.0x0.1 --masks [ /scratch/fmriprep_23_1_wf/single_subject_002_wf/syn_preprocessing_auto_00000/mask2epi/sub-002_ses-01_T1w_noise_corrected_template_corrected_xform_rbrainmask_trans_uint8.nii, /scratch/fmriprep_23_1_wf/single_subject_002_wf/fmap_preproc_wf/wf_auto_00000/epi_umask/clipped_mask_union.nii.gz ] --winsorize-image-intensities [ 0.001, 0.998 ]  --write-composite-transform 0
Stdout:

Stderr:
	Killed
Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
	    setattr(outputs, key, val)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/traits/trait_types.py", line 2699, in validate
	    return TraitListObject(self, object, name, value)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/traits/trait_list_object.py", line 582, in __init__
	    super().__init__(
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/traits/trait_list_object.py", line 213, in __init__
	    super().__init__(self.item_validator(item) for item in iterable)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/traits/trait_list_object.py", line 213, in <genexpr>
	    super().__init__(self.item_validator(item) for item in iterable)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/traits/trait_list_object.py", line 865, in _item_validator
	    return trait_validator(object, self.name, value)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
	    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
	    self.error(objekt, name, str(value))
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/traits/base_trait_handler.py", line 74, in error
	    raise TraitError(
	traits.trait_errors.TraitError: Each element of the 'reverse_forward_transforms' trait of a RegistrationOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/scratch/fmriprep_23_1_wf/single_subject_002_wf/fmap_preproc_wf/wf_auto_00000/syn/fmap_syn0Warp.nii.gz' <class 'str'> was specified.

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 400, in run
	    outputs = self.aggregate_outputs(runtime)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
	    raise FileNotFoundError(msg)
	FileNotFoundError: No such file or directory '['/scratch/fmriprep_23_1_wf/single_subject_002_wf/fmap_preproc_wf/wf_auto_00000/syn/fmap_syn0Warp.nii.gz']' for output 'reverse_forward_transforms' of a FixHeaderRegistration interface

Screenshots / relevant information:

├── dataset_description.json
├── sub-002
│ ├── ses-01
│ │ ├── anat
│ │ │ ├── sub-002_ses-01_T1w.json
│ │ │ └── sub-002_ses-01_T1w.nii
│ │ ├── dwi
│ │ │ ├── sub-002_ses-01_dwi.bval
│ │ │ ├── sub-002_ses-01_dwi.bvec
│ │ │ ├── sub-002_ses-01_dwi.json
│ │ │ └── sub-002_ses-01_dwi.nii
│ │ └── func
│ │ ├── sub-002_ses-01_task-rest_bold.json
│ │ └── sub-002_ses-01_task-rest_bold.nii
│ └── ses-02
│ ├── anat
│ │ ├── sub-002_ses-02_T1w.json
│ │ └── sub-002_ses-02_T1w.nii
│ ├── dwi
│ │ ├── sub-002_ses-02_dwi.bval
│ │ ├── sub-002_ses-02_dwi.bvec
│ │ ├── sub-002_ses-02_dwi.json
│ │ └── sub-002_ses-02_dwi.nii
│ └── func
│ ├── sub-002_ses-02_task-rest_bold.json
│ └── sub-002_ses-02_task-rest_bold.nii

effigies · January 16, 2024, 6:42pm

The node was killed by the operating system, indicating that it ran out of memory. What resources are available on the system?

CARLOS_LEANDRO_SILVA · January 16, 2024, 6:48pm

13th Gen Intel(R) Core™ i7-13700KF
RAM: 64 Gb

effigies · January 16, 2024, 8:02pm

That should be plenty. Are other large processes running on the system at the same time? And what are the dimensions/voxel sizes of your T1w and BOLD images?

Steven · January 16, 2024, 8:12pm

It could also be that your default docker settings are limiting you to some lower memory. E.g. (from my Docker desktop settings),

CARLOS_LEANDRO_SILVA · January 17, 2024, 1:24pm

Thank you, effigies, for your response!

No, there are no processes running at the same time.

The dimensions are:

T1w: 512x512
BOLD: 64x64

CARLOS_LEANDRO_SILVA · January 17, 2024, 1:27pm

Thank you, Steven, for your response.
I’m using linux, in this case, Docker can use all available host memory, right?

effigies · January 17, 2024, 2:16pm

Is this 512x512x512? If so, that’s likely the problem. The SyN-SDC process used to downsample anatomicals to BOLD resolution, but it now uses original resolutions. High-res T1w images will cause it to use a lot of memory and take excessive time. There’s an open issue (Excessive SyN runtime · Issue #422 · nipreps/sdcflows · GitHub) to reduce this.

CARLOS_LEANDRO_SILVA · January 17, 2024, 2:24pm

No, sorry.

T1w
Captura de ecrã de 2024-01-17 11-20-43

BOLD
Captura de ecrã de 2024-01-17 11-19-50

Is there any way to pre-process this still using SyN-SDC?

effigies · January 17, 2024, 2:44pm

You could downsample your T1w image and use the sdcflows 2.8 CLI to create a fieldmap. Then pass the output directory as an input to fMRIPrep 23.2.0. Something like (I haven’t tested this, so you’ll probably need to fiddle):

$ pipx run sdcflows $DOWNSAMPLED_BIDS $OUT/sdcflows participant --participant-label $LABEL
$ pipx run fmriprep-docker $BIDS $OUT/fmriprep participant \
    --participant-label $LABEL --derivatives fmaps=$OUT/sdcflows

At the very least it would be a good test of whether handling arbitrary precomputed derivatives is working properly.

CARLOS_LEANDRO_SILVA · January 17, 2024, 4:53pm

I wanted to update you on the progress. I’ve successfully completed the downsampling of the image as per the requirements. However, I’m currently encountering difficulties when attempting to run the sdcflows package.

Please, take a look:

sdcflows /media/pisa/0ABA196EBA1958111/Elisa/downsampled_bids /media/pisa/0ABA196EBA1958111/Elisa/sdcflows participant --participant-label 002
240117-13:45:26,437 cli IMPORTANT:
	 Running SDCFlows 2.8.0:
             * BIDS dataset path: /media/pisa/0ABA196EBA1958111/Elisa/downsampled_bids.
             * Output folder: /media/pisa/0ABA196EBA1958111/Elisa/sdcflows.
             * Analysis levels: ['participant'].

Traceback (most recent call last):
  File "/home/pisa/fsl/bin/sdcflows", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/pisa/fsl/lib/python3.11/site-packages/sdcflows/cli/main.py", line 176, in main
    sdcflows_wf.run(**_plugin)
  File "/home/pisa/fsl/lib/python3.11/site-packages/nipype/pipeline/engine/workflows.py", line 638, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/home/pisa/fsl/lib/python3.11/site-packages/niworkflows/engine/plugin.py", line 162, in run
    self._generate_dependency_list(graph)
  File "/home/pisa/fsl/lib/python3.11/site-packages/niworkflows/engine/plugin.py", line 348, in _generate_dependency_list
    self.depidx = to_scipy_sparse_array(graph, nodelist=self.procs, format="lil")
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pisa/fsl/lib/python3.11/site-packages/networkx/utils/backends.py", line 412, in __call__
    return self.orig_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pisa/fsl/lib/python3.11/site-packages/networkx/convert_matrix.py", line 570, in to_scipy_sparse_array
    raise nx.NetworkXError("Graph has no nodes or edges")
networkx.exception.NetworkXError: Graph has no nodes or edges

effigies · January 17, 2024, 4:55pm

Looks like a bug. Could you open an issue?

CARLOS_LEANDRO_SILVA · January 17, 2024, 5:10pm

Sure! I will open
Do you know another way to pre-process using FMRIPrep and still using SyN-SDC?
I’m not sure if it would be a good idea to use the input from fmriprep with the images already downsampled.

CARLOS_LEANDRO_SILVA · January 17, 2024, 6:24pm

Thanks a lot, effigies! You’re a magician with FMRIPrep – confirmed, the resolution is the problem.

effigies · January 17, 2024, 6:51pm

The idea is to use downsampled images for calculating the fieldmap, and then rerun with the full resolution, using the pre-calculated fieldmap as an additional input. So the “other” approach (working around the bug in sdcflows) would be to run minimal fmriprep on the downsampled dataset, throw out everything but the fieldmap outputs, and then run fmriprep on the full-res dataset, passing in the fieldmaps:

$ pipx run fmriprep-docker $DOWNSAMPLED_BIDS $OUT/fmriprep-ds participant \
    --participant-label $LABEL --level minimal --fs-no-reconall
$ rm -r $OUT/fmriprep-ds/sub-$LABEL/{anat,func}
$ pipx run fmriprep-docker $BIDS $OUT/fmriprep participant \
    --participant-label $LABEL --derivatives fmaps=$OUT/fmriprep-ds

Note that the precomputed derivatives stuff is very new, and it’s been mostly tested on the anatomical derivatives. It’s possible you’ll find more bugs.

CARLOS_LEANDRO_SILVA · January 18, 2024, 2:55pm

Thank you for sharing your approach for handling the fieldmap calculation. Using downsampled images for initial fieldmap computation and then incorporating the pre-calculated fieldmap into the full-resolution dataset sounds great. The provided script gives a clear outline of the steps involved.
I reduced the resolution of the anatomical image slightly using FSL’s FLIRT, and then ran FMRIPrep. It’s working well.