I am preprocessing a task fmri dataset on an hpc cluster using slurm + singularity.
The log of the slurm job finishes with an exit code = 1 but I can’t find any errors. I previously ran the exact same singularity command on the exact same dataset (before fixing my json sidecars to get fmap correction) and the exit code was 0. Does anyone have an idea as to what might be happening and how I could debug this?
Command used (and if a helper script was used, a link to the helper script or the command generated):
Dear @Steven,
Thank you for your reply and sorry for missing the post template!
Here is some more information:
This problem appears across the whole dataset, all subjects have an exit code 1 (I catch the exit code with echo "Completed with return code: $?" just after $SING_CMD)
The data fails BIDS validation as I still haven’t generated the events.tsv files but this has not been a problem when running fmriprep previously.
The final HTML outputs seem normal and indicate that there are no errors to report.
I first thought I could just trust the HTML outputs (reporting no errors) but then when I checked the logs from previous run of fmriprep on the same data (the only difference was that SDC was not performed due to error in B0 json sidecar), the exit code was 0 for all subjects. Thus I am hesitant to proceed with further analysis until I understand what caused the difference in exit code.
bids-validator@1.14.8
(node:1) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
1: [ERR] Files with such naming scheme are not part of BIDS specification. This error is most commonly caused by typos in file names that make them not BIDS compatible. Please consult the specification and make sure your files are named correctly. If this is not a file naming issue (for example when including files not yet covered by the BIDS specification) you should include a ".bidsignore" file in your dataset (see https://github.com/bids-standard/bids-validator#bidsignore for details). Please note that derived (processed) data should be placed in /derivatives folder and source data (such as DICOMS or behavioural logs in proprietary formats) should be placed in the /sourcedata folder. (code: 1 - NOT_INCLUDED)
./bids_conversion_log.txt
Evidence: bids_conversion_log.txt
./bids_validation_1.txt
Evidence: bids_validation_1.txt
./sub-01/ses-1/anat/sub-01_ses-1_run-1_T1w_defaced.nii
Evidence: sub-01_ses-1_run-1_T1w_defaced.nii
./sub-01/ses-1/file_names_lookup.json
Evidence: file_names_lookup.json
./sub-01/ses-1/fmap/sub-01_ses-1_task-learning_run-1_physio.log
Evidence: sub-01_ses-1_task-learning_run-1_physio.log
./sub-01/ses-1/fmap/sub-01_ses-1_task-learning_run-2_physio.log
Evidence: sub-01_ses-1_task-learning_run-2_physio.log
./sub-01/ses-1/fmap/sub-01_ses-1_task-test_run-3_physio.log
Evidence: sub-01_ses-1_task-test_run-3_physio.log
./sub-01/ses-1/func/sub-01_ses-1_task-learning_run-1_physio.log
Evidence: sub-01_ses-1_task-learning_run-1_physio.log
./sub-01/ses-1/func/sub-01_ses-1_task-learning_run-2_physio.log
Evidence: sub-01_ses-1_task-learning_run-2_physio.log
./sub-01/ses-1/func/sub-01_ses-1_task-test_run-3_physio.log
Evidence: sub-01_ses-1_task-test_run-3_physio.log
... and 565 more files having this issue (Use --verbose to see them all).
Please visit https://neurostars.org/search?q=NOT_INCLUDED for existing conversations about this issue.
1: [WARN] Task scans should have a corresponding events.tsv file. If this is a resting state scan you can ignore this warning or rename the task to include the word "rest". (code: 25 - EVENTS_TSV_MISSING)
./sub-01/ses-1/func/sub-01_ses-1_task-learning_run-1_bold.nii
./sub-01/ses-1/func/sub-01_ses-1_task-learning_run-2_bold.nii
./sub-01/ses-1/func/sub-01_ses-1_task-test_run-3_bold.nii
./sub-02/ses-1/func/sub-02_ses-1_task-learning_run-1_bold.nii
./sub-02/ses-1/func/sub-02_ses-1_task-learning_run-2_bold.nii
./sub-02/ses-1/func/sub-02_ses-1_task-test_run-3_bold.nii
./sub-03/ses-1/func/sub-03_ses-1_task-learning_run-1_bold.nii
./sub-03/ses-1/func/sub-03_ses-1_task-learning_run-2_bold.nii
./sub-03/ses-1/func/sub-03_ses-1_task-test_run-3_bold.nii
./sub-04/ses-1/func/sub-04_ses-1_task-learning_run-1_bold.nii
... and 203 more files having this issue (Use --verbose to see them all).
Please visit https://neurostars.org/search?q=EVENTS_TSV_MISSING for existing conversations about this issue.
2: [WARN] Not all subjects contain the same files. Each subject should contain the same number of files with the same naming unless some files are known to be missing. (code: 38 - INCONSISTENT_SUBJECTS)
./sub-01/ses-1/anat/sub-01_ses-1_run-2_T1w.nii
Evidence: Subject: sub-01; Missing file: sub-01_ses-1_run-2_T1w.nii
./sub-02/ses-1/anat/sub-02_ses-1_run-2_T1w.nii
Evidence: Subject: sub-02; Missing file: sub-02_ses-1_run-2_T1w.nii
./sub-03/ses-1/anat/sub-03_ses-1_run-2_T1w.nii
Evidence: Subject: sub-03; Missing file: sub-03_ses-1_run-2_T1w.nii
./sub-04/ses-1/anat/sub-04_ses-1_run-2_T1w.nii
Evidence: Subject: sub-04; Missing file: sub-04_ses-1_run-2_T1w.nii
./sub-05/ses-1/anat/sub-05_ses-1_run-2_T1w.nii
Evidence: Subject: sub-05; Missing file: sub-05_ses-1_run-2_T1w.nii
./sub-06/ses-1/anat/sub-06_ses-1_run-2_T1w.nii
Evidence: Subject: sub-06; Missing file: sub-06_ses-1_run-2_T1w.nii
./sub-07/ses-1/anat/sub-07_ses-1_run-2_T1w.nii
Evidence: Subject: sub-07; Missing file: sub-07_ses-1_run-2_T1w.nii
./sub-08/ses-1/anat/sub-08_ses-1_run-2_T1w.nii
Evidence: Subject: sub-08; Missing file: sub-08_ses-1_run-2_T1w.nii
./sub-09/ses-1/anat/sub-09_ses-1_run-2_T1w.nii
Evidence: Subject: sub-09; Missing file: sub-09_ses-1_run-2_T1w.nii
./sub-12/ses-1/anat/sub-12_ses-1_run-2_T1w.nii
Evidence: Subject: sub-12; Missing file: sub-12_ses-1_run-2_T1w.nii
... and 56 more files having this issue (Use --verbose to see them all).
Please visit https://neurostars.org/search?q=INCONSISTENT_SUBJECTS for existing conversations about this issue.
3: [WARN] The recommended file /README is very small. Please consider expanding it with additional information about the dataset. (code: 213 - README_FILE_SMALL)
./README
Please visit https://neurostars.org/search?q=README_FILE_SMALL for existing conversations about this issue.
Summary: Available Tasks: Available Modalities:
2783 Files, 58.68GB Reward Pairs MRI
71 - Subjects
1 - Session
If you have any questions, please post on https://neurostars.org/tags/bids.
I would try to fix those errors, there are so many invalid names that I can’t get a full scope of what is wrong (e.g., you mentioned SDC - but it is unclear if those are named correctly).
Yes the validator output is a bit overwhelming, however most of those errors come from our physiological logs. Also this did not cause any problem when running fmriprep previously.
For the physio logs you can patch those in a .bidsignore. The _defaced.nii is a problem because “defaced” is not a BIDS valid suffix. Probably *rec-defaced_T1w.nii.gz would be better.
bids-validator@1.14.8
(node:1023331) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
1: [WARN] Task scans should have a corresponding events.tsv file. If this is a resting state scan you can ignore this warning or rename the task to include the word "rest". (code: 25 - EVENTS_TSV_MISSING)
./sub-01/ses-1/func/sub-01_ses-1_task-learning_run-1_bold.nii
./sub-01/ses-1/func/sub-01_ses-1_task-learning_run-2_bold.nii
./sub-01/ses-1/func/sub-01_ses-1_task-test_run-3_bold.nii
./sub-02/ses-1/func/sub-02_ses-1_task-learning_run-1_bold.nii
./sub-02/ses-1/func/sub-02_ses-1_task-learning_run-2_bold.nii
./sub-02/ses-1/func/sub-02_ses-1_task-test_run-3_bold.nii
./sub-03/ses-1/func/sub-03_ses-1_task-learning_run-1_bold.nii
./sub-03/ses-1/func/sub-03_ses-1_task-learning_run-2_bold.nii
./sub-03/ses-1/func/sub-03_ses-1_task-test_run-3_bold.nii
./sub-04/ses-1/func/sub-04_ses-1_task-learning_run-1_bold.nii
... and 203 more files having this issue (Use --verbose to see them all).
Please visit https://neurostars.org/search?q=EVENTS_TSV_MISSING for existing conversations about this issue.
2: [WARN] The recommended file /README is very small. Please consider expanding it with additional information about the dataset. (code: 213 - README_FILE_SMALL)
./README
Please visit https://neurostars.org/search?q=README_FILE_SMALL for existing conversations about this issue.
Summary: Available Tasks: Available Modalities:
2203 Files, 50.9GB Reward Pairs MRI
71 - Subjects
1 - Session
If you have any questions, please post on https://neurostars.org/tags/bids.
Is it possible there is something running before or after your fmriprep command that is failing? Do you have set +e or -e in your script anywhere? I ask because I looked at one of my own runs of fmriprep 23.2.1 and it doesn’t return any error code at the end of the process, so it might be something else.
I don’t have set +e or -e anywhere in my script. What I did is that I added --debug to my singularity command and am now seeing this in the log:
DEBUG [U=646287714,P=3446887]PostStartProcess() Post start process
You are using fMRIPrep-23.2.1, and a newer version of fMRIPrep is available: 24.0.1.
Please check out our documentation about how and when to upgrade:
https://fmriprep.readthedocs.io/en/latest/faq.html#upgrading
DEBUG [U=646287714,P=3446887]Master() Child exited with exit status 1
I am not too familiar with Singularity but from the U and P codes, I think that the child process in question is in deed the fmriprep command.
FYI I have asked my university’s IT team for assistance in debugging so that they can help me figure out if the issue could be cause by Singularity or SLURM. I will update this thread when I know more.
By update you mean use a later version of fmriprep?
I just ran the same version inside a singularity shell and the behaviour was the same: fmriprep reporting no error but the exit status is 1.
I will give it one last try by running a different version but if I have no success I will just use the current outputs as they seem to be good.
In either case, thanks a lot for the time you put into helping me!