Summary of what happened:
I have been preprocessing a dataset with fMRIprep, and did this successfully for the first 11 subjects. However, last week, I launched fMRIprep for the next 5 subjects, and the first three ran and have now finished successfully, but the last two failed, with errors in the logs that the Repetition Time is not defined, as well as warnings that the slice timing is not defined. I now cannot get fMRIprep to run for any participants. But when I check their .json files, the āRepetitionTimeā and āSliceTimingā fields are defined. I have compared them carefully to the .json files for subjects that ran successfully before this problem occurred, and they are exactly the same (aside from the acquisition time). Iām stumped about what could be going wrong, especially since nothing seems to have changed between the subjects that already ran successfully and all the rest of them. I would greatly appreciate any ideas! More details about what Iāve tried and noticed in the Screenshots/relevant information section.
Command used (and if a helper script was used, a link to the helper script or the command generated):
I believe what I used is a helper script, so here is an example of the command generated by it:
sbatch --no-requeue -n 1 --mem=100000 --time=40:0:0 --cpus-per-task=12 --job-name="sub-035_fmriprep" --output=/proj/lindquistgiovanello/projects/EPIC/logs/FMRIprep_logs/Output-sub-035_fmriprep-jobid-%j.out --wrap="unset PYTHONPATH; export SINGULARITYENV_TEMPLATEFLOW_HOME=/proj/hng/singularity_imgs/template_flow; singularity run -B ${TEMPLATEFLOW_HOME:-$HOME/.cache/templateflow}:/proj/hng/singularity_imgs/template_flow,/proj,/pine,/work,/nas02,/nas -e --no-home /proj/hng/singularity_imgs/fmriprep_1.5.3.sif /proj/lindquistgiovanello/projects/EPIC/data_BIDS /proj/lindquistgiovanello/projects/EPIC/data_fmriprep/fmriprep participant --participant-label 035 -w /work/users/n/g/ngfrye --fs-license-file /proj/hng/singularity_imgs/license.txt --nthreads 12 "
Version:
fMRIprep version 21.0.2, but have also tried with version 22.1.1 and version 1.5.3 with the same results.
Environment (Docker, Singularity, custom installation):
Iām a bit new to this and Iām not totally sure what this means, but I believe Singularity. Iām running fMRIprep through an fMRI processing pipeline that we use at my institution called clpipe. So it runs on an HPC, and it implements fMRIprep for preprocessing.
Data formatted according to a validatable standard? Please provide the output of the validator:
The data are in BIDS format. Some context on that in the relevant information section below - in short, I am working from niftiās rather than DICOMS and manually created .jsonās and put the data into BIDS format. Once I did this for one subject, I ran BIDS validator to check that it worked, and got the following output:
e[33m1: [WARN] Task scans should have a corresponding events.tsv file. If this is a resting state scan you can ignore this warning or rename the task to include the word "rest". (code: 25 - EVENTS_TSV_MISSING)e[39m
./sub-001/func/sub-001_task-AR1_bold.nii.gz
Task scans should have a corresponding events.tsv file. It can be included one of the following locations: /task-AR1_events.tsv, /events.tsv, /sub-001/sub-001_task-AR1_events.tsv, /sub-001/sub-001_events.tsv, /sub-001/func/sub-001_task-AR1_events.tsv, /sub-001/func/sub-001_events.tsv
./sub-001/func/sub-001_task-AR2_bold.nii.gz
Task scans should have a corresponding events.tsv file. It can be included one of the following locations: /task-AR2_events.tsv, /events.tsv, /sub-001/sub-001_task-AR2_events.tsv, /sub-001/sub-001_events.tsv, /sub-001/func/sub-001_task-AR2_events.tsv, /sub-001/func/sub-001_events.tsv
./sub-001/func/sub-001_task-AR3_bold.nii.gz
Task scans should have a corresponding events.tsv file. It can be included one of the following locations: /task-AR3_events.tsv, /events.tsv, /sub-001/sub-001_task-AR3_events.tsv, /sub-001/sub-001_events.tsv, /sub-001/func/sub-001_task-AR3_events.tsv, /sub-001/func/sub-001_events.tsv
./sub-001/func/sub-001_task-AR4_bold.nii.gz
Task scans should have a corresponding events.tsv file. It can be included one of the following locations: /task-AR4_events.tsv, /events.tsv, /sub-001/sub-001_task-AR4_events.tsv, /sub-001/sub-001_events.tsv, /sub-001/func/sub-001_task-AR4_events.tsv, /sub-001/func/sub-001_events.tsv
./sub-001/func/sub-001_task-ER_bold.nii.gz
Task scans should have a corresponding events.tsv file. It can be included one of the following locations: /task-ER_events.tsv, /events.tsv, /sub-001/sub-001_task-ER_events.tsv, /sub-001/sub-001_events.tsv, /sub-001/func/sub-001_task-ER_events.tsv, /sub-001/func/sub-001_events.tsv
e[33m2: [WARN] Nifti file's header field for unit information for x, y, z, and t dimensions empty or too short (code: 41 - NIFTI_UNIT)e[39m
./sub-001/anat/sub-001_T1w.nii.gz
Nifti file's header field for unit information for x, y, z, and t dimensions empty or too short
./sub-001/anat/sub-001_T2w.nii.gz
Nifti file's header field for unit information for x, y, z, and t dimensions empty or too short
e[34me[4mSummary:e[24me[39m e[34me[4mAvailable Tasks:e[24me[39m e[34me[4mAvailable Modalities:e[24me[39m
21 Files, 16.27GB AR1 /participants
1 - Subject AR2 T1w
1 - Session AR4 T2w
AR3 bold
rest
ER
Then, about a month later, when this issue arose, I ran BIDS validator again (this time on all of our participants as I had at this point gotten them all into BIDS format). This is the output:
[31m1: [ERR] You have to define 'RepetitionTime' for this file. (code: 10 - REPETITION_TIME_MUST_DEFINE)e[39m
./sub-008/func/sub-008_task-AR1_bold.nii.gz
./sub-035/func/sub-035_task-AR3_bold.nii.gz
./sub-035/func/sub-035_task-AR4_bold.nii.gz
./sub-035/func/sub-035_task-AR5_bold.nii.gz
./sub-035/func/sub-035_task-ER_bold.nii.gz
./sub-036/func/sub-036_task-AR1_bold.nii.gz
./sub-036/func/sub-036_task-AR2_bold.nii.gz
./sub-036/func/sub-036_task-AR3_bold.nii.gz
./sub-036/func/sub-036_task-ER_bold.nii.gz
./sub-037/func/sub-037_task-AR1_bold.nii.gz
e[31m... and 125 more files having this issue (Use --verbose to see them all).e[39m
e[31m2: [ERR] Not a valid JSON file. (code: 27 - JSON_INVALID)e[39m
./sub-016/anat/sub-016_T2w.json
@ line: 2 character: 5
Evidence: "Modality": "MR",
./sub-016/anat/sub-016_T2w.json
@ line: 2 character: 15
Evidence: "Modality": "MR",
./sub-016/anat/sub-016_T2w.json
@ line: 3 character: 28
Evidence: "MagneticFieldStrength": 3,
./sub-016/anat/sub-016_T2w.json
@ line: 3 character: 5
./sub-016/anat/sub-016_T2w.json
@ line: 2 character: 15
Evidence: "Modality": "MR",
./sub-016/anat/sub-016_T2w.json
@ line: 3 character: 5
Evidence: "MagneticFieldStrength": 3,
e[31m3: [ERR] You have to define 'TaskName' for this file. (code: 50 - TASK_NAME_MUST_DEFINE)e[39m
./sub-008/func/sub-008_task-AR1_bold.nii.gz
./sub-035/func/sub-035_task-AR3_bold.nii.gz
./sub-035/func/sub-035_task-AR4_bold.nii.gz
./sub-035/func/sub-035_task-AR5_bold.nii.gz
./sub-035/func/sub-035_task-ER_bold.nii.gz
./sub-036/func/sub-036_task-AR1_bold.nii.gz
./sub-036/func/sub-036_task-AR2_bold.nii.gz
./sub-036/func/sub-036_task-AR3_bold.nii.gz
./sub-036/func/sub-036_task-ER_bold.nii.gz
./sub-037/func/sub-037_task-AR1_bold.nii.gz
e[31m... and 125 more files having this issue (Use --verbose to see them all).e[39m
e[31m4: [ERR] A json sidecar file was found without a corresponding data file (code: 90 - SIDECAR_WITHOUT_DATAFILE)e[39m
./sub-008/anat/sub-008_T1w.json
./sub-008/anat/sub-008_T2w.json
./sub-015/anat/sub-015_T1w.json
./sub-015/anat/sub-015_T2w.json
./sub-016/anat/sub-016_T1w.json
e[33m1: [WARN] You should define 'SliceTiming' for this file. If you don't provide this information slice time correction will not be possible. (code: 13 - SLICE_TIMING_NOT_DEFINED)e[39m
./sub-008/func/sub-008_task-AR1_bold.nii.gz
./sub-035/func/sub-035_task-AR3_bold.nii.gz
./sub-035/func/sub-035_task-AR4_bold.nii.gz
./sub-035/func/sub-035_task-AR5_bold.nii.gz
./sub-035/func/sub-035_task-ER_bold.nii.gz
./sub-036/func/sub-036_task-AR1_bold.nii.gz
./sub-036/func/sub-036_task-AR2_bold.nii.gz
./sub-036/func/sub-036_task-AR3_bold.nii.gz
./sub-036/func/sub-036_task-ER_bold.nii.gz
./sub-037/func/sub-037_task-AR1_bold.nii.gz
e[33m... and 125 more files having this issue (Use --verbose to see them all).e[39m
e[33m2: [WARN] Task scans should have a corresponding events.tsv file. If this is a resting state scan you can ignore this warning or rename the task to include the word "rest". (code: 25 - EVENTS_TSV_MISSING)e[39m
./sub-001/func/sub-001_task-AR1_bold.nii.gz
./sub-001/func/sub-001_task-AR2_bold.nii.gz
./sub-001/func/sub-001_task-AR3_bold.nii.gz
./sub-001/func/sub-001_task-AR4_bold.nii.gz
./sub-001/func/sub-001_task-ER_bold.nii.gz
./sub-002/func/sub-002_task-AR1_bold.nii.gz
./sub-002/func/sub-002_task-AR2_bold.nii.gz
./sub-002/func/sub-002_task-AR3_bold.nii.gz
./sub-002/func/sub-002_task-AR4_bold.nii.gz
./sub-003/func/sub-003_task-AR2_bold.nii.gz
e[33m... and 246 more files having this issue (Use --verbose to see them all).e[39m
e[33m3: [WARN] Not all subjects contain the same files. Each subject should contain the same number of files with the same naming unless some files are known to be missing. (code: 38 - INCONSISTENT_SUBJECTS)e[39m
./sub-001/func/sub-001_task-AR5_bold.json
Evidence: Subject: sub-001; Missing file: sub-001_task-AR5_bold.json
./sub-001/func/sub-001_task-AR5_bold.nii.gz
Evidence: Subject: sub-001; Missing file: sub-001_task-AR5_bold.nii.gz
./sub-002/func/sub-002_task-AR5_bold.json
Evidence: Subject: sub-002; Missing file: sub-002_task-AR5_bold.json
./sub-002/func/sub-002_task-AR5_bold.nii.gz
Evidence: Subject: sub-002; Missing file: sub-002_task-AR5_bold.nii.gz
./sub-002/func/sub-002_task-ER_bold.json
Evidence: Subject: sub-002; Missing file: sub-002_task-ER_bold.json
./sub-002/func/sub-002_task-ER_bold.nii.gz
Evidence: Subject: sub-002; Missing file: sub-002_task-ER_bold.nii.gz
./sub-002/func/sub-002_task-resting_bold.json
Evidence: Subject: sub-002; Missing file: sub-002_task-resting_bold.json
./sub-002/func/sub-002_task-resting_bold.nii.gz
Evidence: Subject: sub-002; Missing file: sub-002_task-resting_bold.nii.gz
./sub-003/func/sub-003_task-AR1_bold.json
Evidence: Subject: sub-003; Missing file: sub-003_task-AR1_bold.json
./sub-003/func/sub-003_task-AR1_bold.nii.gz
Evidence: Subject: sub-003; Missing file: sub-003_task-AR1_bold.nii.gz
e[33m... and 92 more files having this issue (Use --verbose to see them all).e[39m
e[33m4: [WARN] Not all subjects/sessions/runs have the same scanning parameters. (code: 39 - INCONSISTENT_PARAMETERS)e[39m
./sub-003/func/sub-003_task-AR2_bold.nii.gz
./sub-003/func/sub-003_task-AR3_bold.nii.gz
./sub-003/func/sub-003_task-AR4_bold.nii.gz
./sub-012/func/sub-012_task-ER_bold.nii.gz
./sub-057/func/sub-057_task-ER_bold.nii.gz
./sub-057/func/sub-057_task-resting_bold.nii.gz
e[33m5: [WARN] Nifti file's header field for unit information for x, y, z, and t dimensions empty or too short (code: 41 - NIFTI_UNIT)e[39m
./sub-001/anat/sub-001_T1w.nii.gz
./sub-001/anat/sub-001_T2w.nii.gz
./sub-004/anat/sub-004_T1w.nii.gz
./sub-004/anat/sub-004_T2w.nii.gz
./sub-005/anat/sub-005_T1w.nii.gz
./sub-005/anat/sub-005_T2w.nii.gz
./sub-006/anat/sub-006_T1w.nii.gz
./sub-006/anat/sub-006_T2w.nii.gz
./sub-007/anat/sub-007_T1w.nii.gz
./sub-007/anat/sub-007_T2w.nii.gz
e[33m... and 56 more files having this issue (Use --verbose to see them all).e[39m
e[34me[4mSummary:e[24me[39m e[34me[4mAvailable Tasks:e[24me[39m e[34me[4mAvailable Modalities:e[24me[39m
731 Files, 731.46GB AR1 /participants
46 - Subjects AR2 T1w
1 - Session AR4 T2w
ER bold
AR3
rest
AR5
Relevant log outputs (up to 20 lines):
Here are the first 20 lines of the log output from fMRIprep for the last subject that was processed successfully (which was submitted at the same time as the first subjects that failed):
WARNING: DEPRECATED USAGE: Forwarding SINGULARITYENV_TEMPLATEFLOW_HOME as environment variable will not be supported in the future, use APPTAINERENV_TEMPLATEFLOW_HOME instead
INFO: underlay of /etc/localtime required more than 50 (107) bind mounts
bids-validator@1.8.0
e[33m1: [WARN] NIfTI file's header field for unit information for x, y, z, and t dimensions empty or too short (code: 41 - NIFTI_UNIT)e[39m
./sub-031/anat/sub-031_T1w.nii.gz
./sub-031/anat/sub-031_T2w.nii.gz
e[36m Please visit https://neurostars.org/search?q=NIFTI_UNIT for existing conversations about this issue.e[39m
e[33m2: [WARN] The Authors field of dataset_description.json should contain an array of fields - with one author per field. This was triggered based on the presence of only one author field. Please ignore if all contributors are already properly listed. (code: 102 - TOO_FEW_AUTHORS)e[39m
e[36m Please visit https://neurostars.org/search?q=TOO_FEW_AUTHORS for existing conversations about this issue.e[39m
e[33m3: [WARN] The recommended file /README is very small. Please consider expanding it with additional information about the dataset. (code: 213 - README_FILE_SMALL)e[39m
e[36m Please visit https://neurostars.org/search?q=README_FILE_SMALL for existing conversations about this issue.e[39m
Now here are the first 20 lines of the log output from fMRIprep for the first subject that failed:
WARNING: DEPRECATED USAGE: Forwarding SINGULARITYENV_TEMPLATEFLOW_HOME as environment variable will not be supported in the future, use APPTAINERENV_TEMPLATEFLOW_HOME instead
INFO: underlay of /etc/localtime required more than 50 (107) bind mounts
bids-validator@1.8.0
e[31m1: [ERR] You have to define 'RepetitionTime' for this file. (code: 10 - REPETITION_TIME_MUST_DEFINE)e[39m
./sub-035/func/sub-035_task-AR3_bold.nii.gz
./sub-035/func/sub-035_task-AR4_bold.nii.gz
./sub-035/func/sub-035_task-AR5_bold.nii.gz
./sub-035/func/sub-035_task-ER_bold.nii.gz
e[36m Please visit https://neurostars.org/search?q=REPETITION_TIME_MUST_DEFINE for existing conversations about this issue.e[39m
e[33m1: [WARN] You should define 'SliceTiming' for this file. If you don't provide this information slice time correction will not be possible. 'Slice Timing' is the time at which each slice was acquired within each volume (frame) of the acquisition. Slice timing is not slice order -- rather, it is a list of times containing the time (in seconds) of each slice acquisition in relation to the beginning of volume acquisition. (code: 13 - SLICE_TIMING_NOT_DEFINED)e[39m
./sub-035/func/sub-035_task-AR3_bold.nii.gz
./sub-035/func/sub-035_task-AR4_bold.nii.gz
./sub-035/func/sub-035_task-AR5_bold.nii.gz
./sub-035/func/sub-035_task-ER_bold.nii.gz
e[36m Please visit https://neurostars.org/search?q=SLICE_TIMING_NOT_DEFINED for existing conversations about this issue.e[39m
Screenshots / relevant information:
Background info on the data:
There are only niftiās for these data, no DICOMS, so I have had to find a way to get the data into BIDS format to be compatable with fMRIprep and with my institutionās processing pipeline. Here are the steps I have done to do that: I created .json files for each task, using .jsonās from another study as a template and filling in all the needed metadata, which I do have access to. I created a BIDS file structure and moved the niftiās into the proper folders, renamed them according to BIDS conventions, and uploaded the .jsons to accompany them, aslo naming them according to BIDS conventions. Then, for the .jsons accompanying functional runs, I opened each and edited the āAcquisitionTimeā field according to study records, and sometimes changed the āSeriesNumberā if the study records indicated the protocol was adjusted in a particular scan and tasks were re-ordered. The .jsons for the structural runs I was able to get straight from Flywheel where they were automatically generated (unlike our functional runs), so I didnāt change anything about them and just moved them into the appropriate folders. After doing that and running BIDS validator, there was an error indicating that the nifti headers on our functional runs did not contain the units for the TR or voxel dimensions, so I opened and edited each functional nifti header using flsedithd and adding those fields. After doing this, BIDS validator ran without errors, and I was able to process the first 11 participants with fmriprep.
Other notes:
On the first subject that failed, these errors/warnings pointed to only the last 4 tasks, and on the second subject that failed and all other subjects I have tried since, it points to all tasks.
I have also checked the nifti headers, and they are exactly the same as those for subjects that already ran successfully. When I ran BIDS validator again, it now shows a number of additional errors and warnings that similarly donāt make sense to me, such as needing to define āTaskNameā for the file, when it is also defined in all of the files and is the same as in the .jsonās for subjects that ran successfully. I also see in the BIDS validator output that it is not giving errors/warnings for the subjects that already were ran successfully, even though I canāt find anything thatās different about them.
I have tried using different versions of fMRIprep, as well as different versions of the pipeline I am using to implement it (clpipe)