FmriPrep Inconsistently Failing During Preprocessing Due To "File Not Found Error"

Software Support
1

Summary of what happened:

I encountered a nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node unifize, followed by a FileNotFoundError: [Errno 2] No such file or directory error when running fMRIPrep despite the file being present.This error is inconsistent and sometimes resolves when rerunning the same dataset.

Command used (and if a helper script was used, a link to the helper script or the command generated):

3dUnifize -clfrac 0.2 -rbt 18.3 65.0 90.0 -prefix uni.nii.gz -T2 -input /mfs/io/groups/dmello/projects/cerebellum_reliability_work_temp/fmriprep_23_1_wf/single_subject_MSC01_wf/func_preproc_ses_func06_task_glasslexical_run_01_wf/bold_t1_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/unifize/vol0000_xform-00000_clipped_merged_average_corrected_brain_masked.nii.gz

Version:

23.1.4

Environment (Docker, Singularity / Apptainer, custom installation):

Apptainer

Data formatted according to a validatable standard? Please provide the output of the validator:

PASTE VALIDATOR OUTPUT HERE

Relevant log outputs (up to 20 lines):

Inputs:
args: -clfrac 0.2 -rbt 18.3 65.0 90.0
cl_frac:
environ: {}
epi:
gm:
in_file:
no_duplo:
num_threads: 1
out_file: uni.nii.gz
outputtype: NIFTI_GZ
quiet:
rbt:
scale_file:
t2: True
t2_up:
urad:
Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node unifize.

Cmdline:
	3dUnifize -clfrac 0.2 -rbt 18.3 65.0 90.0 -prefix uni.nii.gz -T2 -input /mfs/io/groups/dmello/projects/cerebellum_reliability_work_temp/fmriprep_23_1_wf/single_subject_MSC01_wf/func_preproc_ses_func06_task_glasslexical_run_01_wf/bold_t1_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/unifize/vol0000_xform-00000_clipped_merged_average_corrected_brain_masked.nii.gz
Stdout:

Stderr:
	++ 3dUnifize: AFNI version=AFNI_23.2.02 (Jul 28 2023) [64-bit]
	 + Pre-processing: IAV..UW[s4034]Im
	++ Output dataset ./uni.nii.gz
	++ ===== Elapsed = 46.5 sec
Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 400, in run
	    outputs = self.aggregate_outputs(runtime)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 429, in aggregate_outputs
	    predicted_outputs = self._list_outputs()  # Predictions from _list_outputs
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/afni/base.py", line 249, in _list_outputs
	    outputs = super(AFNICommand, self)._list_outputs()
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 902, in _list_outputs
	    outputs[out_name] = os.path.abspath(fname)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/posixpath.py", line 384, in abspath
	    cwd = os.getcwd()
	FileNotFoundError: [Errno 2] No such file or directory```

### Screenshots / relevant information:
<!-- Below this line, provide any other information that might be meaningful. This could be screenshots, troubleshooting steps you have tried, or information about your operating system, etc. -->
Please let me know if I can provide any additional information. Thanks in advance!
_____
2

Hi @adiDe, and welcome to neurostars!

Please state the fmriprep command.

Additionally, what resources (cpus, memory) did you devote to the job?

Is the error subject-specific or happening to everyone?

Does the error persist when using a fresh working directory?

You might also consider upgrading fmriprep to the most recent version (24.1.1 at this time).

Best,
Steven

3

Hi Steven,

Thank you for the response!
fmriprep command:

unset PYTHONPATH; apptainer run -B $root_dir -B $fmriprep_dir $fmriprep_dir/fmriprep-23.1.4.sif \
    $bids_root_dir/$dataset $output_folder -w $workfolder \
    participant \
    --participant-label $subj \
    --skip-bids-validation \
    --md-only-boilerplate \
    --fs-license-file $FS_LICENSE \
    --fs-no-reconall \
    --output-spaces MNI152NLin2009cAsym:res-2 \
    --mem_mb $mem_mb \

Variables:

#!/bin/bash
export dataset=$1
export subj=$2
#Directories
export root_dir=/mfs/io/groups/dmello/projects
export bids_root_dir=$root_dir/cerebellum_reliability
export code_dir=$bids_root_dir/code
export datadir=$bids_root_dir/derivatives
export preproc_dir=$datadir/fmriprep
export workfolder=$root_dir/cerebellum_reliability_work_temp
export software_dir=/mfs/io/groups/dmello/software
export fmriprep_dir=$software_dir/fmriprep
export afni_dir=$software_dir/afni
export heudiconv_dir=$software_dir/heudiconv

Resources: Memory (100 gb) within our fmriprep file. Our launcher utilizes:

#!/bin/bash
#SBATCH -N 1               # Total # of nodes (must be 1 for serial)
#SBATCH -n 1               # Total # of mpi tasks (should be 1 for serial)
#SBATCH -t 36:00:00        # Run time (hh:mm:ss)

The error is happening to random subjects and may or may not disappear after rerunning preprocessing on the same subject (i.e. 2 weeks ago preprocessing of another subject resulted in this error but after rerunning this week it preprocessed correctly).

Yes the error does persistent in fresh working directories.

Sounds good, I will try upgrading the fmriprep to the most recent version.
Please let me know if i can provide any additional information.
Thanks in advance,
Adi

4

Hi @adiDe,

It doesn’t look like you are specifying memory or cpu in the sbatch header, which will control the resources given to you by the job scheduler. The arguments in fmriprep will only be important if the job scheduler is giving you enough resources.

Also --fs-no-reconall is not recommended. Additionally, you should include -e as a flag in your apptainer run preamble.

Can you also return the bids validator output (do not skip BIDS validation) to ensure there are no other errors?

Best,
Steven

5

Hi @Steven,

Thank you for all your help!

I have attached the bids validator output for one of the datasets that is returning this error below:

(Use `node --trace-warnings ...` to show where the warning was created)
	1: [WARN] Not all subjects contain the same files. Each subject should contain the same number of files with the same naming unless some files are known to be missing. (code: 38 - INCONSISTENT_SUBJECTS)
		./sub-MSC01/ses-func03/func/sub-MSC01_ses-func03_task-glasslexical_bold.nii.gz
			Evidence: Subject: sub-MSC01; Missing file: sub-MSC01_ses-func03_task-glasslexical_bold.nii.gz
		./sub-MSC01/ses-func03/func/sub-MSC01_ses-func03_task-glasslexical_events.tsv
			Evidence: Subject: sub-MSC01; Missing file: sub-MSC01_ses-func03_task-glasslexical_events.tsv
		./sub-MSC01/ses-func10/func/sub-MSC01_ses-func10_task-glasslexical_run-03_bold.nii.gz
			Evidence: Subject: sub-MSC01; Missing file: sub-MSC01_ses-func10_task-glasslexical_run-03_bold.nii.gz
		./sub-MSC01/ses-func10/func/sub-MSC01_ses-func10_task-glasslexical_run-03_events.tsv
			Evidence: Subject: sub-MSC01; Missing file: sub-MSC01_ses-func10_task-glasslexical_run-03_events.tsv
		./sub-MSC01/ses-struct01/anat/sub-MSC01_ses-struct01_run-03_T1w.nii.gz
			Evidence: Subject: sub-MSC01; Missing file: sub-MSC01_ses-struct01_run-03_T1w.nii.gz
		./sub-MSC01/ses-struct01/anat/sub-MSC01_ses-struct01_run-03_T2w.nii.gz
			Evidence: Subject: sub-MSC01; Missing file: sub-MSC01_ses-struct01_run-03_T2w.nii.gz
		./sub-MSC01/ses-struct01/anat/sub-MSC01_ses-struct01_run-03_mod-T1w_defacemask.nii.gz
			Evidence: Subject: sub-MSC01; Missing file: sub-MSC01_ses-struct01_run-03_mod-T1w_defacemask.nii.gz
		./sub-MSC01/ses-struct01/anat/sub-MSC01_ses-struct01_run-03_mod-T2w_defacemask.nii.gz
			Evidence: Subject: sub-MSC01; Missing file: sub-MSC01_ses-struct01_run-03_mod-T2w_defacemask.nii.gz
		./sub-MSC01/ses-struct02/anat/sub-MSC01_ses-struct02_run-03_T2w.nii.gz
			Evidence: Subject: sub-MSC01; Missing file: sub-MSC01_ses-struct02_run-03_T2w.nii.gz
		./sub-MSC01/ses-struct02/anat/sub-MSC01_ses-struct02_run-03_mod-T2w_defacemask.nii.gz
			Evidence: Subject: sub-MSC01; Missing file: sub-MSC01_ses-struct02_run-03_mod-T2w_defacemask.nii.gz
		... and 118 more files having this issue (Use --verbose to see them all).

	Please visit https://neurostars.org/search?q=INCONSISTENT_SUBJECTS for existing conversations about this issue.

	2: [WARN] Not all subjects/sessions/runs have the same scanning parameters. (code: 39 - INCONSISTENT_PARAMETERS)
		./sub-MSC06/ses-func08/func/sub-MSC06_ses-func08_task-rest_bold.nii.gz

	Please visit https://neurostars.org/search?q=INCONSISTENT_PARAMETERS for existing conversations about this issue.

	3: [WARN] Tabular file contains custom columns not described in a data dictionary (code: 82 - CUSTOM_COLUMN_WITHOUT_DESCRIPTION)
		./sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-glasslexical_run-01_events.tsv
			Evidence: Columns: correct not defined, please define in: /events.json, /task-glasslexical_events.json,/run-01_events.json,/task-glasslexical_run-01_events.json,/sub-MSC01/sub-MSC01_events.json,/sub-MSC01/sub-MSC01_task-glasslexical_events.json,/sub-MSC01/sub-MSC01_run-01_events.json,/sub-MSC01/sub-MSC01_task-glasslexical_run-01_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_task-glasslexical_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_run-01_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_task-glasslexical_run-01_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-glasslexical_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_run-01_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-glasslexical_run-01_events.json
		./sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-glasslexical_run-02_events.tsv
			Evidence: Columns: correct not defined, please define in: /events.json, /task-glasslexical_events.json,/run-02_events.json,/task-glasslexical_run-02_events.json,/sub-MSC01/sub-MSC01_events.json,/sub-MSC01/sub-MSC01_task-glasslexical_events.json,/sub-MSC01/sub-MSC01_run-02_events.json,/sub-MSC01/sub-MSC01_task-glasslexical_run-02_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_task-glasslexical_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_run-02_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_task-glasslexical_run-02_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-glasslexical_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_run-02_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-glasslexical_run-02_events.json
		./sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-memoryfaces_events.tsv
			Evidence: Columns: correct not defined, please define in: /events.json, /task-memoryfaces_events.json,/sub-MSC01/sub-MSC01_events.json,/sub-MSC01/sub-MSC01_task-memoryfaces_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_task-memoryfaces_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-memoryfaces_events.json
		./sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-memoryscenes_events.tsv
			Evidence: Columns: correct not defined, please define in: /events.json, /task-memoryscenes_events.json,/sub-MSC01/sub-MSC01_events.json,/sub-MSC01/sub-MSC01_task-memoryscenes_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_task-memoryscenes_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-memoryscenes_events.json
		./sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-memorywords_events.tsv
			Evidence: Columns: correct not defined, please define in: /events.json, /task-memorywords_events.json,/sub-MSC01/sub-MSC01_events.json,/sub-MSC01/sub-MSC01_task-memorywords_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_task-memorywords_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-memorywords_events.json
		./sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-glasslexical_run-01_events.tsv
			Evidence: Columns: correct not defined, please define in: /events.json, /task-glasslexical_events.json,/run-01_events.json,/task-glasslexical_run-01_events.json,/sub-MSC01/sub-MSC01_events.json,/sub-MSC01/sub-MSC01_task-glasslexical_events.json,/sub-MSC01/sub-MSC01_run-01_events.json,/sub-MSC01/sub-MSC01_task-glasslexical_run-01_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_task-glasslexical_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_run-01_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_task-glasslexical_run-01_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-glasslexical_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_run-01_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-glasslexical_run-01_events.json
		./sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-glasslexical_run-02_events.tsv
			Evidence: Columns: correct not defined, please define in: /events.json, /task-glasslexical_events.json,/run-02_events.json,/task-glasslexical_run-02_events.json,/sub-MSC01/sub-MSC01_events.json,/sub-MSC01/sub-MSC01_task-glasslexical_events.json,/sub-MSC01/sub-MSC01_run-02_events.json,/sub-MSC01/sub-MSC01_task-glasslexical_run-02_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_task-glasslexical_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_run-02_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_task-glasslexical_run-02_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-glasslexical_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_run-02_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-glasslexical_run-02_events.json
		./sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-memoryfaces_events.tsv
			Evidence: Columns: correct not defined, please define in: /events.json, /task-memoryfaces_events.json,/sub-MSC01/sub-MSC01_events.json,/sub-MSC01/sub-MSC01_task-memoryfaces_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_task-memoryfaces_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-memoryfaces_events.json
		./sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-memoryscenes_events.tsv
			Evidence: Columns: correct not defined, please define in: /events.json, /task-memoryscenes_events.json,/sub-MSC01/sub-MSC01_events.json,/sub-MSC01/sub-MSC01_task-memoryscenes_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_task-memoryscenes_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-memoryscenes_events.json
		./sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-memorywords_events.tsv
			Evidence: Columns: correct not defined, please define in: /events.json, /task-memorywords_events.json,/sub-MSC01/sub-MSC01_events.json,/sub-MSC01/sub-MSC01_task-memorywords_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_task-memorywords_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-memorywords_events.json
		... and 490 more files having this issue (Use --verbose to see them all).

	Please visit https://neurostars.org/search?q=CUSTOM_COLUMN_WITHOUT_DESCRIPTION for existing conversations about this issue.

	4: [WARN] The onset of the last event is less than half the total duration of the corresponding scan. This design is suspiciously short.  (code: 86 - SUSPICIOUSLY_SHORT_EVENT_DESIGN)
		./sub-MSC02/ses-func06/func/sub-MSC02_ses-func06_task-memoryfaces_events.tsv

	Please visit https://neurostars.org/search?q=SUSPICIOUSLY_SHORT_EVENT_DESIGN for existing conversations about this issue.

	5: [WARN] Not all subjects contain the same sessions. (code: 97 - MISSING_SESSION)
		./sub-MSC01/ses-func11
			Evidence: Subject: sub-MSC01; Missing session: ses-func11
		./sub-MSC03/ses-func11
			Evidence: Subject: sub-MSC03; Missing session: ses-func11
		./sub-MSC04/ses-func11
			Evidence: Subject: sub-MSC04; Missing session: ses-func11
		./sub-MSC05/ses-func11
			Evidence: Subject: sub-MSC05; Missing session: ses-func11
		./sub-MSC06/ses-func11
			Evidence: Subject: sub-MSC06; Missing session: ses-func11
		./sub-MSC07/ses-func11
			Evidence: Subject: sub-MSC07; Missing session: ses-func11
		./sub-MSC08/ses-func11
			Evidence: Subject: sub-MSC08; Missing session: ses-func11
		./sub-MSC09/ses-func11
			Evidence: Subject: sub-MSC09; Missing session: ses-func11

	Please visit https://neurostars.org/search?q=MISSING_SESSION for existing conversations about this issue.

        Summary:                   Available Tasks:        Available Modalities: 
        2189 Files, 25.19GB        glasslexical            MRI                   
        10 - Subjects              memorywords                                   
        13 - Sessions              memoryscenes                                  
                                   memoryfaces                                   
                                   motor                                         
                                   rest

Please let me know if I can provide any more information. I will try specifying the CPU and Memory in the sbatch header and updating the apptainer run until the.

Thank you!

6

Hi Steven,
After updating the CPU (10 nodes) and mem (64GB) I am still running into the same error (cleared the working directories as well). I added the --node – trace-warnings to the validator call with this result:

Singularity> node --trace-warnings $(which bids-validator) /mfs/io/groups/...
/projects/cerebellum_reliability/ds000224
bids-validator@1.14.6
(node:3945737) Warning: Closing directory handle on garbage collection
	1: [WARN] Not all subjects contain the same files. Each subject should contain the same number of files with the same naming unless some files are known to be missing. (code: 38 - INCONSISTENT_SUBJECTS)
		./sub-MSC01/ses-func03/func/sub-MSC01_ses-func03_task-glasslexical_bold.nii.gz
			Evidence: Subject: sub-MSC01; Missing file: sub-MSC01_ses-func03_task-glasslexical_bold.nii.gz
		./sub-MSC01/ses-func03/func/sub-MSC01_ses-func03_task-glasslexical_events.tsv
			Evidence: Subject: sub-MSC01; Missing file: sub-MSC01_ses-func03_task-glasslexical_events.tsv
		./sub-MSC01/ses-func10/func/sub-MSC01_ses-func10_task-glasslexical_run-03_bold.nii.gz
			Evidence: Subject: sub-MSC01; Missing file: sub-MSC01_ses-func10_task-glasslexical_run-03_bold.nii.gz
		./sub-MSC01/ses-func10/func/sub-MSC01_ses-func10_task-glasslexical_run-03_events.tsv
			Evidence: Subject: sub-MSC01; Missing file: sub-MSC01_ses-func10_task-glasslexical_run-03_events.tsv
		./sub-MSC01/ses-struct01/anat/sub-MSC01_ses-struct01_run-03_T1w.nii.gz
			Evidence: Subject: sub-MSC01; Missing file: sub-MSC01_ses-struct01_run-03_T1w.nii.gz
		./sub-MSC01/ses-struct01/anat/sub-MSC01_ses-struct01_run-03_T2w.nii.gz
			Evidence: Subject: sub-MSC01; Missing file: sub-MSC01_ses-struct01_run-03_T2w.nii.gz
		./sub-MSC01/ses-struct01/anat/sub-MSC01_ses-struct01_run-03_mod-T1w_defacemask.nii.gz
			Evidence: Subject: sub-MSC01; Missing file: sub-MSC01_ses-struct01_run-03_mod-T1w_defacemask.nii.gz
		./sub-MSC01/ses-struct01/anat/sub-MSC01_ses-struct01_run-03_mod-T2w_defacemask.nii.gz
			Evidence: Subject: sub-MSC01; Missing file: sub-MSC01_ses-struct01_run-03_mod-T2w_defacemask.nii.gz
		./sub-MSC01/ses-struct02/anat/sub-MSC01_ses-struct02_run-03_T2w.nii.gz
			Evidence: Subject: sub-MSC01; Missing file: sub-MSC01_ses-struct02_run-03_T2w.nii.gz
		./sub-MSC01/ses-struct02/anat/sub-MSC01_ses-struct02_run-03_mod-T2w_defacemask.nii.gz
			Evidence: Subject: sub-MSC01; Missing file: sub-MSC01_ses-struct02_run-03_mod-T2w_defacemask.nii.gz
		... and 118 more files having this issue (Use --verbose to see them all).

	Please visit https://neurostars.org/search?q=INCONSISTENT_SUBJECTS for existing conversations about this issue.

	2: [WARN] Not all subjects/sessions/runs have the same scanning parameters. (code: 39 - INCONSISTENT_PARAMETERS)
		./sub-MSC06/ses-func08/func/sub-MSC06_ses-func08_task-rest_bold.nii.gz

	Please visit https://neurostars.org/search?q=INCONSISTENT_PARAMETERS for existing conversations about this issue.

	3: [WARN] Tabular file contains custom columns not described in a data dictionary (code: 82 - CUSTOM_COLUMN_WITHOUT_DESCRIPTION)
		./sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-glasslexical_run-01_events.tsv
			Evidence: Columns: correct not defined, please define in: /events.json, /task-glasslexical_events.json,/run-01_events.json,/task-glasslexical_run-01_events.json,/sub-MSC01/sub-MSC01_events.json,/sub-MSC01/sub-MSC01_task-glasslexical_events.json,/sub-MSC01/sub-MSC01_run-01_events.json,/sub-MSC01/sub-MSC01_task-glasslexical_run-01_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_task-glasslexical_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_run-01_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_task-glasslexical_run-01_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-glasslexical_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_run-01_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-glasslexical_run-01_events.json
		./sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-glasslexical_run-02_events.tsv
			Evidence: Columns: correct not defined, please define in: /events.json, /task-glasslexical_events.json,/run-02_events.json,/task-glasslexical_run-02_events.json,/sub-MSC01/sub-MSC01_events.json,/sub-MSC01/sub-MSC01_task-glasslexical_events.json,/sub-MSC01/sub-MSC01_run-02_events.json,/sub-MSC01/sub-MSC01_task-glasslexical_run-02_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_task-glasslexical_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_run-02_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_task-glasslexical_run-02_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-glasslexical_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_run-02_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-glasslexical_run-02_events.json
		./sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-memoryfaces_events.tsv
			Evidence: Columns: correct not defined, please define in: /events.json, /task-memoryfaces_events.json,/sub-MSC01/sub-MSC01_events.json,/sub-MSC01/sub-MSC01_task-memoryfaces_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_task-memoryfaces_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-memoryfaces_events.json
		./sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-memoryscenes_events.tsv
			Evidence: Columns: correct not defined, please define in: /events.json, /task-memoryscenes_events.json,/sub-MSC01/sub-MSC01_events.json,/sub-MSC01/sub-MSC01_task-memoryscenes_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_task-memoryscenes_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-memoryscenes_events.json
		./sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-memorywords_events.tsv
			Evidence: Columns: correct not defined, please define in: /events.json, /task-memorywords_events.json,/sub-MSC01/sub-MSC01_events.json,/sub-MSC01/sub-MSC01_task-memorywords_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_events.json,/sub-MSC01/ses-func01/sub-MSC01_ses-func01_task-memorywords_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_events.json,/sub-MSC01/ses-func01/func/sub-MSC01_ses-func01_task-memorywords_events.json
		./sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-glasslexical_run-01_events.tsv
			Evidence: Columns: correct not defined, please define in: /events.json, /task-glasslexical_events.json,/run-01_events.json,/task-glasslexical_run-01_events.json,/sub-MSC01/sub-MSC01_events.json,/sub-MSC01/sub-MSC01_task-glasslexical_events.json,/sub-MSC01/sub-MSC01_run-01_events.json,/sub-MSC01/sub-MSC01_task-glasslexical_run-01_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_task-glasslexical_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_run-01_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_task-glasslexical_run-01_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-glasslexical_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_run-01_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-glasslexical_run-01_events.json
		./sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-glasslexical_run-02_events.tsv
			Evidence: Columns: correct not defined, please define in: /events.json, /task-glasslexical_events.json,/run-02_events.json,/task-glasslexical_run-02_events.json,/sub-MSC01/sub-MSC01_events.json,/sub-MSC01/sub-MSC01_task-glasslexical_events.json,/sub-MSC01/sub-MSC01_run-02_events.json,/sub-MSC01/sub-MSC01_task-glasslexical_run-02_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_task-glasslexical_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_run-02_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_task-glasslexical_run-02_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-glasslexical_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_run-02_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-glasslexical_run-02_events.json
		./sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-memoryfaces_events.tsv
			Evidence: Columns: correct not defined, please define in: /events.json, /task-memoryfaces_events.json,/sub-MSC01/sub-MSC01_events.json,/sub-MSC01/sub-MSC01_task-memoryfaces_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_task-memoryfaces_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-memoryfaces_events.json
		./sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-memoryscenes_events.tsv
			Evidence: Columns: correct not defined, please define in: /events.json, /task-memoryscenes_events.json,/sub-MSC01/sub-MSC01_events.json,/sub-MSC01/sub-MSC01_task-memoryscenes_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_task-memoryscenes_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-memoryscenes_events.json
		./sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-memorywords_events.tsv
			Evidence: Columns: correct not defined, please define in: /events.json, /task-memorywords_events.json,/sub-MSC01/sub-MSC01_events.json,/sub-MSC01/sub-MSC01_task-memorywords_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_events.json,/sub-MSC01/ses-func02/sub-MSC01_ses-func02_task-memorywords_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_events.json,/sub-MSC01/ses-func02/func/sub-MSC01_ses-func02_task-memorywords_events.json
		... and 490 more files having this issue (Use --verbose to see them all).

	Please visit https://neurostars.org/search?q=CUSTOM_COLUMN_WITHOUT_DESCRIPTION for existing conversations about this issue.

	4: [WARN] The onset of the last event is less than half the total duration of the corresponding scan. This design is suspiciously short.  (code: 86 - SUSPICIOUSLY_SHORT_EVENT_DESIGN)
		./sub-MSC02/ses-func06/func/sub-MSC02_ses-func06_task-memoryfaces_events.tsv

	Please visit https://neurostars.org/search?q=SUSPICIOUSLY_SHORT_EVENT_DESIGN for existing conversations about this issue.

	5: [WARN] Not all subjects contain the same sessions. (code: 97 - MISSING_SESSION)
		./sub-MSC01/ses-func11
			Evidence: Subject: sub-MSC01; Missing session: ses-func11
		./sub-MSC03/ses-func11
			Evidence: Subject: sub-MSC03; Missing session: ses-func11
		./sub-MSC04/ses-func11
			Evidence: Subject: sub-MSC04; Missing session: ses-func11
		./sub-MSC05/ses-func11
			Evidence: Subject: sub-MSC05; Missing session: ses-func11
		./sub-MSC06/ses-func11
			Evidence: Subject: sub-MSC06; Missing session: ses-func11
		./sub-MSC07/ses-func11
			Evidence: Subject: sub-MSC07; Missing session: ses-func11
		./sub-MSC08/ses-func11
			Evidence: Subject: sub-MSC08; Missing session: ses-func11
		./sub-MSC09/ses-func11
			Evidence: Subject: sub-MSC09; Missing session: ses-func11

	Please visit https://neurostars.org/search?q=MISSING_SESSION for existing conversations about this issue.

        Summary:                   Available Tasks:        Available Modalities: 
        2189 Files, 25.19GB        memoryscenes            MRI                   
        10 - Subjects              motor                                         
        13 - Sessions              glasslexical                                  
                                   rest                                          
                                   memoryfaces                                   
                                   memorywords                                   


	If you have any questions, please post on https://neurostars.org/tags/bids.```

Please let me know if I can provide any other details. Thank you for all your help!
7

Hi @adiDe,

Can you run seff $jobid but use the actual job number from slurm, and return the outputs?

Best,
Steven

8

Below is the output of sacct with the same information:

JobID           Cluster      User     Group      State ExitCode   NNodes  AllocCPUS   TotalCPU    Elapsed     MaxRSS     ReqMem 
------------ ---------- --------- --------- ---------- -------- -------- ---------- ---------- ---------- ---------- ---------- 
139464               g2 .........      ....  COMPLETED      0:0        1         10  00:26.764   13:36:46                   60G 
139464.batch         g2                      COMPLETED      0:0        1         10  00:26.764   13:36:46     15656K

In this batch I ran data for 3 subjects and only 1 successfully completed without errors.
Please let me know if I can provide any additional information.

Thanks in advance

9

Hi @adiDe,

Can you try more memory for those failing subjects? Also can you show the full script with the new SBATCH header to make sure it was formatted correctlty?

Best,
Steven

11

I will try upping the memory for each of the failing subjects. Below is the updated sbatch header and full script:

#!/bin/bash
#SBATCH -J cerebellum_reliability   # Job name
#SBATCH -p ----          # Queue (partition) name
#SBATCH -N 1               # Total # of nodes (must be 1 for serial)
#SBATCH -n 1               # Total # of mpi tasks (should be 1 for serial)
#SBATCH -t 36:00:00        # Run time (hh:mm:ss)
#SBATCH --mem=60GB
#SBATCH --cpus-per-task=10

#Convert virtual memory from gb to mb
mem=`echo "${mem//[!0-9]/}"` #remove gb at end
mem_mb=`echo $(((mem*1000)-5000))` #reduce some memory for buffer space during pre-processing

export FS_LICENSE=$fmriprep_dir/license.txt
cd $bids_root_dir

#Run fmriprep
unset PYTHONPATH; apptainer run -e -B $root_dir -B $fmriprep_dir $fmriprep_dir/fmriprep-23.1.4.sif \
    $bids_root_dir/$dataset $output_folder -w $workfolder \
    participant \
    --participant-label $subj \
    --skip-bids-validation \
    --md-only-boilerplate \
    --fs-license-file $FS_LICENSE \
    --fs-no-reconall \
    --output-spaces MNI152NLin2009cAsym:res-2 \
    --mem_mb $mem_mb \