fMRIPrep not running on HPC using Singularity

I am trying to get fMRIPREP v 23.2.1 to run on my school’s HPC via singularity and I’m not sure why my job is getting killed. I am trying to run 1 subject only (sub-002), and after I am able to do that I would like to be able to loop through the rest (total of 113).

I’m using a shell script to run singularity on a SLURM system.

I am able to submit a job successfully and checked that it was running (approximately ~12 hours or so) before it quits with nothing added to my derivatives/fmriprep/ folder. I’m not sure why I’m having the issue, as a postdoc in my lab was able to successfully run a very similar script on different data with no error.

This is the line I use to run the code:

sbatch code/mypreps.sh 002

Here is my script

#! /bin/bash
#SBATCH --partition=standard
#SBATCH --job-name=fmriPrep
#SBATCH -c 4
#SBATCH -o /mnt/chrastil/users/erica/projects/mlindivPreproc/code/gridlog/mlindiv-prep_%j.txt
#SBATCH -e /mnt/chrastil/users/erica/projects/mlindivPreproc/code/gridlog/slurmoutput/%j_slurm.err

# Variables
subs=("$@")
bidsDir=/mnt/chrastil/data/MLINDIV
containers=/mnt/chrastil/users/erica/projects/mlindivPreproc/code/singularity
fsLicense=/mnt/chrastil/users/erica/projects/mlindivPreproc/code/freesurfer

current_date_time="`date +%m-%d-%Y_%H:%M:%S`";


for id in "${subs[@]}"
do

    # Run fMRIprep
    echo "Starting fMRIprep $current_date_time" 
    singularity run -B $bidsDir:/data -B /mnt/chrastil/users/erica/projects/mlindivPreproc:/out -B $fsLicense:/fs \
    --cleanenv $containers/fmriprep_v23-2-1.simg \
    /data \
    /out/derivatives/fmriprep \
    -w /out/work/ \
    participant \
    --participant-label $id \
    --fs-license /fs/license.txt \
    --n_cpus $SLURM_CPUS_PER_TASK
done

Here is the output of my fMRIprep

Starting fMRIprep 05-03-2024_14:46:29
bids-validator@1.14.1
	e[33m1: [WARN] The recommended file /README is very small. Please consider expanding it with additional information about the dataset. (code: 213 - README_FILE_SMALL)e[39m
		./README

e[36m	Please visit https://neurostars.org/search?q=README_FILE_SMALL for existing conversations about this issue.e[39m

        e[34me[4mSummary:e[24me[39m                    e[34me[4mAvailable Tasks:e[24me[39m        e[34me[4mAvailable Modalities:e[39me[24m 
        4590 Files, 615.43GB        explore                 MRI                   
        104 - Subjects              findTarget                                    
        1 - Session                 rest                                          


e[36m	If you have any questions, please post on https://neurostars.org/tags/bids.e[39m
240504-09:56:34,514 nipype.workflow IMPORTANT:
	 Running fMRIPrep version 23.2.1

         License NOTICE ##################################################
         fMRIPrep 23.2.1
         Copyright 2023 The NiPreps Developers.
         
         This product includes software developed by
         the NiPreps Community (https://nipreps.org/).
         
         Portions of this software were developed at the Department of
         Psychology at Stanford University, Stanford, CA, US.
         
         This software is also distributed as a Docker container image.
         The bootstrapping file for the image ("Dockerfile") is licensed
         under the MIT License.
         
         This software may be distributed through an add-on package called
         "Docker Wrapper" that is under the BSD 3-clause License.
         #################################################################


Error in crash log

(node:1530365) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
/var/spool/slurm/d/job42278/slurm_script: line 33: 1505171 Killed                  singularity run -B $bidsDir:/data -B /mnt/chrastil/users/erica/projects/mlindivPreproc:/out -B $fsLicense:/fs --cleanenv $containers/fmriprep_v23-2-1.simg /data /out/derivatives/fmriprep -w /out/work/ participant --participant-label $id --fs-license /fs/license.txt

Here is my the structure of my data

sub-002
├── anat
│   ├── sub-002_T1w.json
│   └── sub-002_T1w.nii.gz
├── dwi
│   ├── sub-002_run-01_dwi.bval
│   ├── sub-002_run-01_dwi.bvec
│   ├── sub-002_run-01_dwi.json
│   ├── sub-002_run-01_dwi.nii.gz
│   ├── sub-002_run-02_dwi.bval
│   ├── sub-002_run-02_dwi.bvec
│   ├── sub-002_run-02_dwi.json
│   └── sub-002_run-02_dwi.nii.gz
├── fmap
│   ├── sub-002_magnitude1.json
│   ├── sub-002_magnitude1.nii.gz
│   ├── sub-002_magnitude2.json
│   ├── sub-002_magnitude2.nii.gz
│   ├── sub-002_phasediff.json
│   └── sub-002_phasediff.nii.gz
├── func
│   ├── sub-002_task-explore_run-01_bold.json
│   ├── sub-002_task-explore_run-01_bold.nii.gz
│   ├── sub-002_task-explore_run-01_events.tsv
│   ├── sub-002_task-explore_run-02_bold.json
│   ├── sub-002_task-explore_run-02_bold.nii.gz
│   ├── sub-002_task-explore_run-02_events.tsv
│   ├── sub-002_task-findTarget_run-01_bold.json
│   ├── sub-002_task-findTarget_run-01_bold.nii.gz
│   ├── sub-002_task-findTarget_run-01_events.tsv
│   ├── sub-002_task-findTarget_run-02_bold.json
│   ├── sub-002_task-findTarget_run-02_bold.nii.gz
│   ├── sub-002_task-findTarget_run-02_events.tsv
│   ├── sub-002_task-findTarget_run-03_bold.json
│   ├── sub-002_task-findTarget_run-03_bold.nii.gz
│   ├── sub-002_task-findTarget_run-03_events.tsv
│   ├── sub-002_task-findTarget_run-04_bold.json
│   ├── sub-002_task-findTarget_run-04_bold.nii.gz
│   ├── sub-002_task-findTarget_run-04_events.tsv
│   ├── sub-002_task-findTarget_run-05_bold.json
│   ├── sub-002_task-findTarget_run-05_bold.nii.gz
│   ├── sub-002_task-findTarget_run-05_events.tsv
│   ├── sub-002_task-findTarget_run-06_bold.json
│   ├── sub-002_task-findTarget_run-06_bold.nii.gz
│   ├── sub-002_task-findTarget_run-06_events.tsv
│   ├── sub-002_task-rest_run-01_bold.json
│   ├── sub-002_task-rest_run-01_bold.nii.gz
│   └── sub-002_task-rest_run-01_events.tsv
└── sub-002_scans.tsv

Here is what I have tried

I have followed the Troubleshooting section in (https://fmriprep.org/en/1.5.1/singularity.html) and was able to successfully check my access to the input data folder and BIDS validity, check my access to the output data folder and write permissions (also verified by the error files being written), checked my access and permissions to my $HOME folder.

Hi @eward and welcome to neurostars!

Killed messages usually indicate out of memory. How much memory/cpu are you devoting to the process? I see that you did not specify memory in your sbatch header, so you are probably using a very low default value not suitable for fmriprep.

Best,
Steven

Thanks for such a quick response Steven! I’ll find out how much memory per CPU I can use, but I did allocate 4 CPUs, which I’ve been told should be enough to run fMRIprep.

Hi @eward,

The number of cpus is mainly going to affect the speed at which fmriprep runs, and to a minor extent, the amount of memory it will consume. The total memory devoted to the job is the most important thing to determine here. Especially with as many images you have, you’ll need a good bit (32GB would be good to try, and then adjust as needed from there).

Best,
Steven

I see. Thanks for the clarification and the suggestion Steven, I’ll try that and report back. (Hopefully with a successful run haha.)

Best,
Erica

An update:

I ran the same subject through my lab’s compute cluster and I am now getting this error:

ValueError: 'desc' is not a recognized entity.

I found here that “This error usually happens when derivatives that are not inside a derivatives/ subdirectory are found in the input dataset,” but I could not find much more on the matter.

I am running my script (using the line mentioned above) from a directory that contains a subdirectories named “code” and “derivatives/fmriprep/logs” (see below):

(current_working_directory) $ ls
code  derivatives  hopper_output_002.txt  work

Is this issue stemming from the existing derivatives folder in my BIDS data path? Please correct me if I’m wrong, but I don’t see any fmri derivatives in the in the BIDS directory (other that what is contained in the derivatives/ subdirectory.

(directory_where_lab_data_is_stored) $ ls
CHANGES                   sourcedata  sub-011  sub-022  sub-032  sub-043  sub-053  sub-064  sub-074  sub-085  sub-098  sub-109
code                      sub-002     sub-012  sub-023  sub-033  sub-044  sub-054  sub-065  sub-075  sub-086  sub-099  sub-110
dataset_description.json  sub-003     sub-013  sub-024  sub-034  sub-045  sub-055  sub-066  sub-076  sub-087  sub-100  sub-111
derivatives               sub-004     sub-014  sub-025  sub-035  sub-046  sub-056  sub-067  sub-078  sub-088  sub-101  sub-112
events.json               sub-005     sub-015  sub-026  sub-036  sub-047  sub-057  sub-068  sub-079  sub-089  sub-102  sub-113
old_MLINDIV_dontUse       sub-006     sub-017  sub-027  sub-037  sub-048  sub-058  sub-069  sub-080  sub-091  sub-103  task-explore_bold.json
participants.json         sub-007     sub-018  sub-028  sub-038  sub-049  sub-059  sub-070  sub-081  sub-093  sub-104  task-findTarget_bold.json
participants.tsv          sub-008     sub-019  sub-029  sub-039  sub-050  sub-060  sub-071  sub-082  sub-094  sub-106  task-rest_bold.json
README                    sub-009     sub-020  sub-030  sub-040  sub-051  sub-061  sub-072  sub-083  sub-095  sub-107  work
scans.json                sub-010     sub-021  sub-031  sub-041  sub-052  sub-062  sub-073  sub-084  sub-096  sub-108

Hi @eward,

There are some files/folders there that do not look BIDS valid. Please return the output of the validator.

Best,
Steven

Hi Steven,

Here is the output of the bids-validation using my lab’s cluster:

bids-validator /data
bids-validator@1.14.1
(node:2749394) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
        1: [WARN] Not all subjects contain the same files. Each subject should contain the same number of files with the same naming unless some files are known to be missing. (code: 38 - INCONSISTENT_SUBJECTS)
                ./sub-002/anat/sub-002_T2w.json
                        Evidence: Subject: sub-002; Missing file: sub-002_T2w.json
                ./sub-002/anat/sub-002_T2w.nii.gz
                        Evidence: Subject: sub-002; Missing file: sub-002_T2w.nii.gz
                ./sub-004/anat/sub-004_T2w.json
                        Evidence: Subject: sub-004; Missing file: sub-004_T2w.json
                ./sub-004/anat/sub-004_T2w.nii.gz
                        Evidence: Subject: sub-004; Missing file: sub-004_T2w.nii.gz
                ./sub-004/dwi/sub-004_run-02_dwi.bval
                        Evidence: Subject: sub-004; Missing file: sub-004_run-02_dwi.bval
                ./sub-004/dwi/sub-004_run-02_dwi.bvec
                        Evidence: Subject: sub-004; Missing file: sub-004_run-02_dwi.bvec
                ./sub-004/dwi/sub-004_run-02_dwi.json
                        Evidence: Subject: sub-004; Missing file: sub-004_run-02_dwi.json
                ./sub-004/dwi/sub-004_run-02_dwi.nii.gz
                        Evidence: Subject: sub-004; Missing file: sub-004_run-02_dwi.nii.gz
                ./sub-004/func/sub-004_task-findTarget_run-02_bold.json
                        Evidence: Subject: sub-004; Missing file: sub-004_task-findTarget_run-02_bold.json
                ./sub-004/func/sub-004_task-findTarget_run-02_bold.nii.gz
                        Evidence: Subject: sub-004; Missing file: sub-004_task-findTarget_run-02_bold.nii.gz
                ... and 199 more files having this issue (Use --verbose to see them all).

        Please visit https://neurostars.org/search?q=INCONSISTENT_SUBJECTS for existing conversations about this issue.

        2: [WARN] Not all subjects/sessions/runs have the same scanning parameters. (code: 39 - INCONSISTENT_PARAMETERS)
                ./sub-002/func/sub-002_task-findTarget_run-01_bold.nii.gz
                ./sub-002/func/sub-002_task-findTarget_run-02_bold.nii.gz
                ./sub-002/func/sub-002_task-findTarget_run-03_bold.nii.gz
                ./sub-002/func/sub-002_task-findTarget_run-04_bold.nii.gz
                ./sub-002/func/sub-002_task-findTarget_run-05_bold.nii.gz
                ./sub-002/func/sub-002_task-findTarget_run-06_bold.nii.gz
                ./sub-003/func/sub-003_task-explore_run-01_bold.nii.gz
                ./sub-003/func/sub-003_task-findTarget_run-01_bold.nii.gz
                ./sub-003/func/sub-003_task-findTarget_run-02_bold.nii.gz
                ./sub-003/func/sub-003_task-findTarget_run-03_bold.nii.gz
                ... and 646 more files having this issue (Use --verbose to see them all).

        Please visit https://neurostars.org/search?q=INCONSISTENT_PARAMETERS for existing conversations about this issue.

        3: [WARN] The recommended file /README is very small. Please consider expanding it with additional information about the dataset. (code: 213 - README_FILE_SMALL)
                ./README

        Please visit https://neurostars.org/search?q=README_FILE_SMALL for existing conversations about this issue.

        Summary:                    Available Tasks:        Available Modalities: 
        4585 Files, 615.41GB        explore                 MRI                   
        104 - Subjects              findTarget                                    
        1 - Session                 rest                                          


        If you have any questions, please post on https://neurostars.org/tags/bids.

That surprises me because files like that should not pass validation. Are you using a bidsignore file?

No, there is no bidsignore file.

I did make some changes and I was able to get outputs to my derivatives/fmirprep/ folder:

  • I copied one subject to a new directory named data in the parent directory preproc where I’m executing the script code/mypreps.sh to run fMRIprep, and I am now setting this as my BIDS directory.
  • I ran this using my lab’s cluster- not using the SLURM system which has more resources and would be better for me to use for the rest of my ~100 subjects.

So my directory has now has the structure:

preproc
├── code
│   ├── freesurfer
│       └── license.txt
│   ├── gridlog
│       ├── ...
│       └── slurmoutput
│       ├── ...
│   ├── mypreps.sh
│   └── singularity
│       ├── fmriprep_v23-2-1.simg
│       └── mriqc_v23-1-0.simg


├── derivatives
│   ├── fmriprep
│   │   ├── logs
│   │   ├── sourcedata
│   │   └──  ...

├── data
│   ├── sub-002
│   │   └──  (with same anat, dwi, fmap, and func structure as above)

Current issues

  1. Although I was able to get an output to the derivatives folder, it may not feasible for me to copy the rest of the data to my own directory (since it is very large).
    Potential solutions are to:
  • Symlink the lab’s data to my new BIDS directory? (Although I’m not sure how this will interact with the Singularity containers- but was a suggestion from a postdoc in my lab)
  • Ask for permission to modify our lab’s data folder organization to be BIDS validated and include my fMRIprep execution script there
  1. I am running into a TemplateFlow error when running on the SLURM system. I presume this is because I don’t have internet access when running things on that cluster? Is there a way around this?
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='templateflow.s3.amazonaws.com', port=443): Max retries exceeded with url: /tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_T1w.nii.gz (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7f5cd1f870a0>: Failed to resolve 'templateflow.s3.amazonaws.com' ([Errno -3] Temporary failure in name resolution)"))

Thanks,
Erica

Hi @eward,

Ideally derivatives and code would live inside data (where data is your BIDS root directory with the raw subject data folders).

Symlinking is possible, just be careful and intentional with your mounting strings in the singularity command, keeping in mind both the target and destination of the symlink are mounted. Ideally you just run it from a BIDS valid dataset without symlinks.

Regarding templatflow error, you can download this directory I made here (templateflow.zip - Google Drive) with the files downloaded, upload it to your cluster, run “export SINGULARITYENV_TEMPLATEFLOW_HOME=/path/to/templateflow”, and make sure it is mounted in singularity.

Best,
Steven