Summary of what happened:
I am running fmriprep-v23.2.1 using direct B0 field mapping for the susceptibility distortion correction. When fmriprep reaches a point with something related to the template space and field map, an exception about a “BrokenProcessPool” is encountered. Fmriprep is stuck at the point and continuously prints out the exception.
Command used (and if a helper script was used, a link to the helper script or the command generated):
singularity run \
-B /scratch/g/jjane/jheff/experiments/epilepsy_padre/derivatives/fmriprep-v23.2.1/sdc-none/fmriprep/sourcedata/freesurfer:/fs_subjects_dir \
-B /scratch/g/jjane/jheff/experiments/epilepsy_padre/bids/fmap-gredirect1:/data \
-B /scratch/g/jjane/jheff/experiments/epilepsy_padre/derivatives/fmriprep-v23.2.1/sdc-gredirect1/fmriprep:/out \
-B /scratch/g/jjane/jheff/experiments/epilepsy_padre/tmp/fmriprep-v23.2.1/sub-10027s_rand-ZiEEZP:/work \
-B /hpc/apps/freesurfer/6.0.0/license.txt:/opt/freesurfer/license.txt \
/scratch/g/jjane/jheff/experiments/epilepsy_padre/singularity_images/fmriprep-v23.2.1.sif \
/data /out participant \
--participant_label 10027s \
--nprocs 16 \
--omp-nthreads 4 \
--output-spaces MNI152NLin2009cAsym:res-2 T1w \
--cifti-output 91k \
--no-submm-recon \
--fs-license-file /opt/freesurfer/license.txt \
--work-dir /work \
--fs-subjects-dir /fs_subjects_dir
Version:
fmriprep 23.2.1
Environment (Docker, Singularity / Apptainer, custom installation):
Singularity
Data formatted according to a validatable standard? Please provide the output of the validator:
-------------------------------------------------------------------------------bids-validator@1.14.1
(node:1036458) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
e[33m1: [WARN] The recommended file /README is missing. See Section 03 (Modality agnostic files) of the BIDS specification. (code: 101 - README_FILE_MISSING)e[39m
e[36m Please visit https://neurostars.org/search?q=README_FILE_MISSING for existing conversations about this issue.e[39m
e[34me[4mSummary:e[24me[39m e[34me[4mAvailable Tasks:e[24me[39m e[34me[4mAvailable Modalities:e[39me[24m
2227 Files, 71.04GB SDTD MRI
84 - Subjects rest
1 - Session storymath
Relevant log outputs (up to 20 lines):
[Node] Setting-up "fmriprep_23_2_wf.sub_10027s20141203_wf.bold_task_storymath_run_2_wf.bold_std_wf.fmap_recon" in "/work/fmriprep_23_2_wf/sub_10027s20141203_wf/bold_task_storymath_run_2_wf/bold_std_wf/_in_tuple_MNI152NLin2009cAsym.res2/fmap_recon".
240425-17:41:51,987 nipype.workflow INFO:
[Node] Setting-up "fmriprep_23_2_wf.sub_10027s20141203_wf.bold_task_storymath_run_3_wf.bold_std_wf.fmap_recon" in "/work/fmriprep_23_2_wf/sub_10027s20141203_wf/bold_task_storymath_run_3_wf/bold_std_wf/_in_tuple_MNI152NLin2009cAsym.res2/fmap_recon".
240425-17:41:52,20 nipype.workflow INFO:
[Node] Executing "fmap_recon" <fmriprep.interfaces.resampling.ReconstructFieldmap>
240425-17:41:52,20 nipype.workflow INFO:
[Node] Executing "fmap_recon" <fmriprep.interfaces.resampling.ReconstructFieldmap>
240425-17:41:52,20 nipype.workflow INFO:
[Node] Executing "fmap_recon" <fmriprep.interfaces.resampling.ReconstructFieldmap>
240425-17:41:52,21 nipype.workflow INFO:
[Node] Executing "fmap_recon" <fmriprep.interfaces.resampling.ReconstructFieldmap>
240425-17:41:52,21 nipype.workflow INFO:
[Node] Executing "fmap_recon" <fmriprep.interfaces.resampling.ReconstructFieldmap>
240425-17:41:52,22 nipype.workflow INFO:
[Node] Executing "fmap_recon" <fmriprep.interfaces.resampling.ReconstructFieldmap>
240425-17:41:52,23 nipype.workflow INFO:
[Node] Executing "fmap_recon" <fmriprep.interfaces.resampling.ReconstructFieldmap>
exception calling callback for <Future at 0x7fba87691d20 state=finished raised BrokenProcessPool>
Traceback (most recent call last):
File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/_base.py", line 342, in _invoke_callbacks
callback(self)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
exception calling callback for <Future at 0x7fba87692c50 state=finished raised BrokenProcessPool>
Traceback (most recent call last):
File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/_base.py", line 342, in _invoke_callbacks
callback(self)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/_base.py", line 342, in _invoke_callbacks
callback(self)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
exception calling callback for <Future at 0x7fba97b21b40 state=finished raised BrokenProcessPool>
Traceback (most recent call last):
File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/_base.py", line 342, in _invoke_callbacks
Screenshots / relevant information:
I have only ran the command for one participant. Here are the fieldmap json contents:
{
"Units": "Hz",
"IntendedFor": [
"func/sub-10027s_task-SDTD_run-1_bold.nii.gz",
"func/sub-10027s_task-SDTD_run-2_bold.nii.gz",
"func/sub-10027s_task-storymath_run-1_bold.nii.gz",
"func/sub-10027s_task-storymath_run-2_bold.nii.gz",
"func/sub-10027s_task-storymath_run-3_bold.nii.gz",
"func/sub-10027s_task-rest_run-1_bold.nii.gz",
"func/sub-10027s_task-rest_run-2_bold.nii.gz"
],
"group_file": "/group/jbinder/work/Epilepsy/padre/derivatives/bfields/sub-10027s_bfield.nii.gz"
}
Here are the magnitude json contents:
{
"IntendedFor": [
"func/sub-10027s_task-SDTD_run-1_bold.nii.gz",
"func/sub-10027s_task-SDTD_run-2_bold.nii.gz",
"func/sub-10027s_task-storymath_run-1_bold.nii.gz",
"func/sub-10027s_task-storymath_run-2_bold.nii.gz",
"func/sub-10027s_task-storymath_run-3_bold.nii.gz",
"func/sub-10027s_task-rest_run-1_bold.nii.gz",
"func/sub-10027s_task-rest_run-2_bold.nii.gz"
],
"group_file": "/group/jbinder/work/Epilepsy/padre/derivatives/bfields/sub-10027s_magnitude.nii.gz"
}
The command successfully ran when the level was set to minimal. I am going to run fmriprep again with the precomputed minimal outputs as inputs to fmriprep and see if that will complete without error.