Hello all,
I am trying to get fMRIPREP v 23.1.2 to run on my university’s HPC via singularity (v. 3.10.3). I got it to run ok (though long and resource intensive) on my local machine via docker. I believe it is failing due to a freesurfer or templateflow issue but I can’t tell.
Here is my setup
export SINGULARITYENV_TEMPLATEFLOW_HOME=/templates
export TEMPLATEFLOW_HOME=/home/gshearre/.templateflow
export FS_LICENSE=/project/brainmri/software/freesurfer/license.txt
Here is my command:
singularity run --cleanenv -B /project/brainmri/data/bbc/BIDS:/data -B /home/gshearre/.templateflow:/templates -B /gscratch/gshearre:/work -B /project/brainmri/software/freesurfer/license.txt:/license.txt:ro /project/brainmri/software/fmriprep-latest.simg /data /data/derivatives/fMRIPrep-23.1.2 participant --participant-label 20001 -w /work/ -vv --omp-nthreads 8 --nthreads 12 --mem_mb 30000 --fs-license-file /license.txt
Here is my initial error
Error setting node input:
Node: gen_ref
input: fixed_image
results_file: /work/fmriprep_23_1_wf/single_subject_20001_wf/func_preproc_ses_2_task_resting_wf/bold_std_trans_wf/_std_target_MNI152NLin2009cAsym.resnative/select_tpl/result_select_tpl.pklz
value: /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_T1w.nii.gz
exception calling callback for <Future at 0x150c4863d000 state=finished raised BrokenProcessPool>
Further error in the crash log
Node: fmriprep_23_1_wf.single_subject_20001_wf.func_preproc_ses_3_task_clamp_wf.bold_std_trans_wf.gen_ref
Working directory: /work/fmriprep_23_1_wf/single_subject_20001_wf/func_preproc_ses_3_task_clamp_wf/bold_std_trans_wf/_std_target_MNI152NLin2009cAsym.resnative/gen_ref
Node inputs:
fixed_image = <undefined>
fov_mask = None
keep_native = True
moving_image = <undefined>
xform_code = None
Traceback (most recent call last):
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 454, in run
cached, updated = self.is_cached()
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 342, in is_cached
hashed_inputs, hashvalue = self._get_hashval()
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 548, in _get_hashval
self._get_inputs()
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 619, in _get_inputs
self.set_input(key, deepcopy(output_value))
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 312, in set_input
setattr(self.inputs, parameter, deepcopy(val))
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
value = super(File, self).validate(objekt, name, value, return_pathlike=True)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
self.error(objekt, name, str(value))
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/traits/base_trait_handler.py", line 74, in error
raise TraitError(
traits.trait_errors.TraitError: The 'fixed_image' trait of a _GenerateSamplingReferenceInputSpec instance must be a pathlike object or string representing an existing file, but a value of '/templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_T1w.nii.gz' <class 'str'> was specified.
Error setting node input:
Node: gen_ref
input: fixed_image
results_file: /work/fmriprep_23_1_wf/single_subject_20001_wf/func_preproc_ses_3_task_clamp_wf/bold_std_trans_wf/_std_target_MNI152NLin2009cAsym.resnative/select_tpl/result_select_tpl.pklz
value: /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_T1w.nii.gz
Here is what I have tried
Going into the container using shell and checking the paths
They seem to bind correctly
singularity shell --cleanenv -B /project/brainmri/data/bbc/BIDS:/data -B /home/gshearre/.templateflow:/templates:ro -B /gscratch/gshearre:/work -B /project/brainmri/software/freesurfer/license.txt:/license.txt:ro /project/brainmri/software/fmriprep-latest.simg /data /data/derivatives/fMRIPrep-23.1.2 participant --participant-label 20001 -w /work/ -vv --omp-nthreads 8 --nthreads 12 --mem_mb 30000 --verbose --fs-license-file /license.txt
Following this github issue
I tried checking:
cat /proc/sys/crypto/fips_enabled
it returned 0
TLDR
Can’t get fMRIPREP container to run via singularity, suspect binding or something with freesurfer or template flow but not sure since it looks ok in the container.
Potential next steps?
- Blood sacrifice to the MRI gods?
- Crying?
Thanks,
Grace
I guess it was the freesurfer file.
