Exit code 1 despite no error in logs

Summary of what happened:

Hi,

I am preprocessing a task fmri dataset on an hpc cluster using slurm + singularity.
The log of the slurm job finishes with an exit code = 1 but I can’t find any errors. I previously ran the exact same singularity command on the exact same dataset (before fixing my json sidecars to get fmap correction) and the exit code was 0. Does anyone have an idea as to what might be happening and how I could debug this?

Command used (and if a helper script was used, a link to the helper script or the command generated):

singularity run \
--cleanenv \
-B /home/hfluhr/shares-hare/ds-learning-habits:/data \
-B /scratch/hfluhr/workflows/ds-learninghabits:/work \
-B /home/hfluhr/shares-hare/ds-learning-habits/derivatives:/out \
/home/hfluhr/data/containers/fmriprep23 
/data \
/out/fmriprep-23.2.1 \
participant \
--participant-label sub-01 \
-w /work \
--skip-bids-validation \
-vvv

Version:

23.2.1

Environment (Docker, Singularity / Apptainer, custom installation):

Singularity

Data formatted according to a validatable standard? Please provide the output of the validator:

PASTE VALIDATOR OUTPUT HERE

Relevant log outputs (up to 20 lines):

PASTE LOG OUTPUT HERE

Screenshots / relevant information:


Hi @hugofluhr, and welcome to neurostars!

In the future, please keep the Software Support post template for these kinds of post, you can see I added it back in for you this time.

It is hard to help without seeing the fMRIPrep log. Other important information include:

  1. Are the final HTMLs output?
  2. Are the data BIDS valid? (Return the BIDS validator output)
  3. Is this error subject specific or across the whole dataset?

Best,
Steven

Dear @Steven,
Thank you for your reply and sorry for missing the post template!
Here is some more information:

  • This problem appears across the whole dataset, all subjects have an exit code 1 (I catch the exit code with echo "Completed with return code: $?" just after $SING_CMD)
  • The data fails BIDS validation as I still haven’t generated the events.tsv files but this has not been a problem when running fmriprep previously.
  • The final HTML outputs seem normal and indicate that there are no errors to report.

I first thought I could just trust the HTML outputs (reporting no errors) but then when I checked the logs from previous run of fmriprep on the same data (the only difference was that SDC was not performed due to error in B0 json sidecar), the exit code was 0 for all subjects. Thus I am hesitant to proceed with further analysis until I understand what caused the difference in exit code.

Best regards,
Hugo

Hi @hugofluhr,

Can you share the BIDS validation report?

Best,
Steven

Sure, here it is:

bids-validator@1.14.8
(node:1) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
        1: [ERR] Files with such naming scheme are not part of BIDS specification. This error is most commonly caused by typos in file names that make them not BIDS compatible. Please consult the specification and make sure your files are named correctly. If this is not a file naming issue (for example when including files not yet covered by the BIDS specification) you should include a ".bidsignore" file in your dataset (see https://github.com/bids-standard/bids-validator#bidsignore for details). Please note that derived (processed) data should be placed in /derivatives folder and source data (such as DICOMS or behavioural logs in proprietary formats) should be placed in the /sourcedata folder. (code: 1 - NOT_INCLUDED)
                ./bids_conversion_log.txt
                        Evidence: bids_conversion_log.txt
                ./bids_validation_1.txt
                        Evidence: bids_validation_1.txt
                ./sub-01/ses-1/anat/sub-01_ses-1_run-1_T1w_defaced.nii
                        Evidence: sub-01_ses-1_run-1_T1w_defaced.nii
                ./sub-01/ses-1/file_names_lookup.json
                        Evidence: file_names_lookup.json
                ./sub-01/ses-1/fmap/sub-01_ses-1_task-learning_run-1_physio.log
                        Evidence: sub-01_ses-1_task-learning_run-1_physio.log
                ./sub-01/ses-1/fmap/sub-01_ses-1_task-learning_run-2_physio.log
                        Evidence: sub-01_ses-1_task-learning_run-2_physio.log
                ./sub-01/ses-1/fmap/sub-01_ses-1_task-test_run-3_physio.log
                        Evidence: sub-01_ses-1_task-test_run-3_physio.log
                ./sub-01/ses-1/func/sub-01_ses-1_task-learning_run-1_physio.log
                        Evidence: sub-01_ses-1_task-learning_run-1_physio.log
                ./sub-01/ses-1/func/sub-01_ses-1_task-learning_run-2_physio.log
                        Evidence: sub-01_ses-1_task-learning_run-2_physio.log
                ./sub-01/ses-1/func/sub-01_ses-1_task-test_run-3_physio.log
                        Evidence: sub-01_ses-1_task-test_run-3_physio.log
                ... and 565 more files having this issue (Use --verbose to see them all).

        Please visit https://neurostars.org/search?q=NOT_INCLUDED for existing conversations about this issue.

        1: [WARN] Task scans should have a corresponding events.tsv file. If this is a resting state scan you can ignore this warning or rename the task to include the word "rest". (code: 25 - EVENTS_TSV_MISSING)
                ./sub-01/ses-1/func/sub-01_ses-1_task-learning_run-1_bold.nii
                ./sub-01/ses-1/func/sub-01_ses-1_task-learning_run-2_bold.nii
                ./sub-01/ses-1/func/sub-01_ses-1_task-test_run-3_bold.nii
                ./sub-02/ses-1/func/sub-02_ses-1_task-learning_run-1_bold.nii
                ./sub-02/ses-1/func/sub-02_ses-1_task-learning_run-2_bold.nii
                ./sub-02/ses-1/func/sub-02_ses-1_task-test_run-3_bold.nii
                ./sub-03/ses-1/func/sub-03_ses-1_task-learning_run-1_bold.nii
                ./sub-03/ses-1/func/sub-03_ses-1_task-learning_run-2_bold.nii
                ./sub-03/ses-1/func/sub-03_ses-1_task-test_run-3_bold.nii
                ./sub-04/ses-1/func/sub-04_ses-1_task-learning_run-1_bold.nii
                ... and 203 more files having this issue (Use --verbose to see them all).

        Please visit https://neurostars.org/search?q=EVENTS_TSV_MISSING for existing conversations about this issue.

        2: [WARN] Not all subjects contain the same files. Each subject should contain the same number of files with the same naming unless some files are known to be missing. (code: 38 - INCONSISTENT_SUBJECTS)
                ./sub-01/ses-1/anat/sub-01_ses-1_run-2_T1w.nii
                        Evidence: Subject: sub-01; Missing file: sub-01_ses-1_run-2_T1w.nii
                ./sub-02/ses-1/anat/sub-02_ses-1_run-2_T1w.nii
                        Evidence: Subject: sub-02; Missing file: sub-02_ses-1_run-2_T1w.nii
                ./sub-03/ses-1/anat/sub-03_ses-1_run-2_T1w.nii
                        Evidence: Subject: sub-03; Missing file: sub-03_ses-1_run-2_T1w.nii
                ./sub-04/ses-1/anat/sub-04_ses-1_run-2_T1w.nii
                        Evidence: Subject: sub-04; Missing file: sub-04_ses-1_run-2_T1w.nii
                ./sub-05/ses-1/anat/sub-05_ses-1_run-2_T1w.nii
                        Evidence: Subject: sub-05; Missing file: sub-05_ses-1_run-2_T1w.nii
                ./sub-06/ses-1/anat/sub-06_ses-1_run-2_T1w.nii
                        Evidence: Subject: sub-06; Missing file: sub-06_ses-1_run-2_T1w.nii
                ./sub-07/ses-1/anat/sub-07_ses-1_run-2_T1w.nii
                        Evidence: Subject: sub-07; Missing file: sub-07_ses-1_run-2_T1w.nii
                ./sub-08/ses-1/anat/sub-08_ses-1_run-2_T1w.nii
                        Evidence: Subject: sub-08; Missing file: sub-08_ses-1_run-2_T1w.nii
                ./sub-09/ses-1/anat/sub-09_ses-1_run-2_T1w.nii
                        Evidence: Subject: sub-09; Missing file: sub-09_ses-1_run-2_T1w.nii
                ./sub-12/ses-1/anat/sub-12_ses-1_run-2_T1w.nii
                        Evidence: Subject: sub-12; Missing file: sub-12_ses-1_run-2_T1w.nii
                ... and 56 more files having this issue (Use --verbose to see them all).

        Please visit https://neurostars.org/search?q=INCONSISTENT_SUBJECTS for existing conversations about this issue.

        3: [WARN] The recommended file /README is very small. Please consider expanding it with additional information about the dataset. (code: 213 - README_FILE_SMALL)
                ./README

        Please visit https://neurostars.org/search?q=README_FILE_SMALL for existing conversations about this issue.

        Summary:                   Available Tasks:        Available Modalities: 
        2783 Files, 58.68GB        Reward Pairs            MRI                   
        71 - Subjects                                                            
        1 - Session                                                              

        If you have any questions, please post on https://neurostars.org/tags/bids.

I would try to fix those errors, there are so many invalid names that I can’t get a full scope of what is wrong (e.g., you mentioned SDC - but it is unclear if those are named correctly).

Yes the validator output is a bit overwhelming, however most of those errors come from our physiological logs. Also this did not cause any problem when running fmriprep previously.

For the physio logs you can patch those in a .bidsignore. The _defaced.nii is a problem because “defaced” is not a BIDS valid suffix. Probably *rec-defaced_T1w.nii.gz would be better.

thanks a lot for your help! I will take care of the bids-validation errors and post here with an update

I fixed the bids-validation problems by

  • using a .bidsignore for files that we want to keep but are not bids compatible.
  • renaming the refaced T1 to *T1w.nii and archiving the T1 containing the face elsewhere.

My problem persists (exit code 1 after fMRIPrep finished successfully!)
Any insight on how to properly investigate this problem would be appreciated.

What is the updated BIDS validation report?

new BIDS validation report:

bids-validator@1.14.8
(node:1023331) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
        1: [WARN] Task scans should have a corresponding events.tsv file. If this is a resting state scan you can ignore this warning or rename the task to include the word "rest". (code: 25 - EVENTS_TSV_MISSING)
                ./sub-01/ses-1/func/sub-01_ses-1_task-learning_run-1_bold.nii
                ./sub-01/ses-1/func/sub-01_ses-1_task-learning_run-2_bold.nii
                ./sub-01/ses-1/func/sub-01_ses-1_task-test_run-3_bold.nii
                ./sub-02/ses-1/func/sub-02_ses-1_task-learning_run-1_bold.nii
                ./sub-02/ses-1/func/sub-02_ses-1_task-learning_run-2_bold.nii
                ./sub-02/ses-1/func/sub-02_ses-1_task-test_run-3_bold.nii
                ./sub-03/ses-1/func/sub-03_ses-1_task-learning_run-1_bold.nii
                ./sub-03/ses-1/func/sub-03_ses-1_task-learning_run-2_bold.nii
                ./sub-03/ses-1/func/sub-03_ses-1_task-test_run-3_bold.nii
                ./sub-04/ses-1/func/sub-04_ses-1_task-learning_run-1_bold.nii
                ... and 203 more files having this issue (Use --verbose to see them all).

        Please visit https://neurostars.org/search?q=EVENTS_TSV_MISSING for existing conversations about this issue.

        2: [WARN] The recommended file /README is very small. Please consider expanding it with additional information about the dataset. (code: 213 - README_FILE_SMALL)
                ./README

        Please visit https://neurostars.org/search?q=README_FILE_SMALL for existing conversations about this issue.

        Summary:                  Available Tasks:        Available Modalities: 
        2203 Files, 50.9GB        Reward Pairs            MRI                   
        71 - Subjects                                                           
        1 - Session                                                             


        If you have any questions, please post on https://neurostars.org/tags/bids.

Can you share a full job log? E.g. the slurm log?

Sure, here is one!
fmriprep-15601299-1.txt (189.2 KB)

Is it possible there is something running before or after your fmriprep command that is failing? Do you have set +e or -e in your script anywhere? I ask because I looked at one of my own runs of fmriprep 23.2.1 and it doesn’t return any error code at the end of the process, so it might be something else.

I don’t have set +e or -e anywhere in my script. What I did is that I added --debug to my singularity command and am now seeing this in the log:

DEBUG   [U=646287714,P=3446887]PostStartProcess()            Post start process
You are using fMRIPrep-23.2.1, and a newer version of fMRIPrep is available: 24.0.1.
Please check out our documentation about how and when to upgrade:
https://fmriprep.readthedocs.io/en/latest/faq.html#upgrading
DEBUG   [U=646287714,P=3446887]Master()                      Child exited with exit status 1

I am not too familiar with Singularity but from the U and P codes, I think that the child process in question is in deed the fmriprep command.

FYI I have asked my university’s IT team for assistance in debugging so that they can help me figure out if the issue could be cause by Singularity or SLURM. I will update this thread when I know more.

Perhaps the presence of the warning message changes the exit code? What happens if you update? In either case I think the outputs you have are usable.

By update you mean use a later version of fmriprep?
I just ran the same version inside a singularity shell and the behaviour was the same: fmriprep reporting no error but the exit status is 1.

I will give it one last try by running a different version but if I have no success I will just use the current outputs as they seem to be good.

In either case, thanks a lot for the time you put into helping me!

Yes, I mean most up to date version which will not produce the warning to upgrade.