Summary of what happened:
I have been getting an error in fmriprep that leads to a KILL(9) error. I had previously been advised this was a memory issue and have since moved to use a high performance cluster (Hoffman2), but am still getting the error. In looking at the log files, the issue appears to revolve around a file not being found that fMRIprep should be producing in the working directory:
FileNotFoundError: [Errno 2] No such file or directory: ‘/u/home/j/juschnei/schneider_uploads/WL_fMRI/data/work/fmriprep_wf/single_subject_WLfMRI037_wf/func_preproc_ses_Session1_task_wl_run_03_wf/initial_boldref_wf/validate_1st/_0xfbdb57434ca7925163e36d897a1e9d4f_unfinished.json’
Command used (and if a helper script was used, a link to the helper script or the command generated):
#!/bin/bash
#$ -cwd
# error = Merged with joblog
#$ -o fmriprep.$JOB_ID.$TASK_ID
#$ -j y
#$ -pe shared 1
#$ -l h_rt=8:00:00,h_data=40G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m a
# Job array indexes (parallel processing; change 2 to the number of participants)
#$ -t 1-7:1
# Load the job environment:
. /u/local/Modules/default/init/modules.sh
module use /u/project/CCN/apps/modulefiles
# Load the necessary modules
module load singularity
# Set up directories and parameters
BIDS_DIR="/u/home/j/juschnei/schneider_uploads/WL_fMRI/data/BIDS"
OUTPUT_DIR="/u/home/j/juschnei/schneider_uploads/WL_fMRI/data/fmriprep"
WORK_DIR="/u/home/j/juschnei/schneider_uploads/WL_fMRI/data/fmriprep/work"
SINGULARITY_IMG="/u/project/CCN/apps/fmriprep/rh7/20.2.1/fmriprep-20.2.1.sif"
FS_LICENSE="/u/home/j/juschnei/CCN-apps/freesurfer/rh7/7.1.1/license.txt"
#FS_LICENSE="/u/home/j/juschnei/schneider_uploads/WL_fMRI/license.txt"
#BIDS_FILTER_FILE="/u/home/j/juschnei/schneider_uploads/WL_fMRI/bids_filter.json"
#readarray -t bidsfmri_in < /u/home/j/juschnei/schneider_uploads/WL_fMRI/bids_fmri_list.txt
#(( i=$SGE_TASK_ID -1))
export SINGULARITYENV_FS_LICENSE="/u/home/j/juschnei/CCN-apps/freesurfer/rh7/7.1.1/license.txt"
singularity run --cleanenv \
$SINGULARITY_IMG \
$BIDS_DIR $OUTPUT_DIR participant \
# --participant-label ${bidsfmri_in[$i]} \
# --bids-filter-file $BIDS_FILTER_FILE \
-w $WORK_DIR \
--skip_bids_validation \
--validate=FALSE \
--fs-license-file $FS_LICENSE \
Version:
fmiprep 20.2.1
Environment (Docker, Singularity / Apptainer, custom installation):
Singularity
Data formatted according to a validatable standard? Please provide the output of the validator:
The data is formatted in BIDS format and has passed the online validator.
PASTE VALIDATOR OUTPUT HERE
Relevant log outputs (up to 20 lines):
Node: _n4_correct1
Working directory: /u/home/j/juschnei/schneider_uploads/WL_fMRI/data/work/fmriprep_wf/single_subject_WLfMRI035_wf/anat_preproc_wf/anat_template_wf/n4_correct/mapflow/_n4_correct1
Node inputs:
args = <undefined>
bias_image = <undefined>
bspline_fitting_distance = <undefined>
bspline_order = <undefined>
convergence_threshold = <undefined>
copy_header = True
dimension = 3
environ = {'NSLOTS': '1'}
histogram_sharpening = <undefined>
input_image = /u/home/j/juschnei/schneider_uploads/WL_fMRI/data/BIDS/sub-WLfMRI035/ses-Session2/anat/sub-WLfMRI035_ses-Session2_T1w.nii.gz
mask_image = <undefined>
n_iterations = <undefined>
num_threads = 1
output_image = <undefined>
rescale_intensities = False
save_bias = False
shrink_factor = <undefined>
weight_image = <undefined>
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 522, in aggregate_outputs
setattr(outputs, key, val)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
value = super(File, self).validate(objekt, name, value, return_pathlike=True)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
self.error(objekt, name, str(value))
File "/usr/local/miniconda/lib/python3.7/site-packages/traits/trait_handlers.py", line 172, in error
value )
traits.trait_errors.TraitError: The 'output_image' trait of a N4BiasFieldCorrectionOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/u/home/j/juschnei/schneider_uploads/WL_fMRI/data/work/fmriprep_wf/single_subject_WLfMRI035_wf/anat_preproc_wf/anat_template_wf/n4_correct/mapflow/_n4_correct1/sub-WLfMRI035_ses-Session2_T1w_corrected.nii.gz' <class 'str'> was specified.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
result = self._run_interface(execute=True)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
return self._run_command(execute)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
result = self._interface.run(cwd=outdir)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 420, in run
runtime = self._post_run_hook(runtime)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/mixins/fixheader.py", line 127, in _post_run_hook
outputs = self.aggregate_outputs(runtime=runtime).get_traitsfree()
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 529, in aggregate_outputs
raise FileNotFoundError(msg)
FileNotFoundError: No such file or directory '/u/home/j/juschnei/schneider_uploads/WL_fMRI/data/work/fmriprep_wf/single_subject_WLfMRI035_wf/anat_preproc_wf/anat_template_wf/n4_correct/mapflow/_n4_correct1/sub-WLfMRI035_ses-Session2_T1w_corrected.nii.gz' for output 'output_image' of a N4BiasFieldCorrection interface
Screenshots / relevant information:
My html file suggests this is across a number of nodes:
Errors
Node Name: fmriprep_wf.single_subject_WLfMRI036_wf.func_preproc_ses_Session1_task_wl_run_01_wf.initial_boldref_wf.gen_ref Node Name: fmriprep_wf.single_subject_WLfMRI036_wf.func_preproc_ses_Session1_task_wl_run_02_wf.initial_boldref_wf.gen_ref Node Name: fmriprep_wf.single_subject_WLfMRI036_wf.anat_preproc_wf.brain_extraction_wf.lap_tmpl Node Name: fmriprep_wf.single_subject_WLfMRI036_wf.func_preproc_ses_Session2_task_rest_wf.initial_boldref_wf.gen_ref Node Name: fmriprep_wf.single_subject_WLfMRI036_wf.func_preproc_ses_Session2_task_langloc_run_01_wf.initial_boldref_wf.gen_ref Node Name: fmriprep_wf.single_subject_WLfMRI036_wf.func_preproc_ses_Session1_task_wm_run_02_wf.initial_boldref_wf.gen_ref Node Name: fmriprep_wf.single_subject_WLfMRI036_wf.func_preproc_ses_Session1_task_wl_run_04_wf.initial_boldref_wf.gen_ref Node Name: fmriprep_wf.single_subject_WLfMRI036_wf.func_preproc_ses_Session2_task_cogflex_run_01_wf.initial_boldref_wf.gen_ref Node Name: fmriprep_wf.single_subject_WLfMRI036_wf.func_preproc_ses_Session2_task_cogflex_run_02_wf.initial_boldref_wf.gen_ref Node Name: fmriprep_wf.single_subject_WLfMRI036_wf.func_preproc_ses_Session1_task_wl_run_03_wf.initial_boldref_wf.gen_ref Node Name: fmriprep_wf.single_subject_WLfMRI036_wf.func_preproc_ses_Session1_task_wm_run_01_wf.initial_boldref_wf.gen_ref Node Name: fmriprep_wf.single_subject_WLfMRI036_wf.func_preproc_ses_Session2_task_langloc_run_02_wf.initial_boldref_wf.gen_ref Node Name: _t1w_conform1