I am trying to get fMRIPREP v 23.2.1 to run on my school’s HPC via singularity and I’m not sure why my job is getting killed. I am trying to run 1 subject only (sub-002), and after I am able to do that I would like to be able to loop through the rest (total of 113).
I’m using a shell script to run singularity on a SLURM system.
I am able to submit a job successfully and checked that it was running (approximately ~12 hours or so) before it quits with nothing added to my derivatives/fmriprep/ folder. I’m not sure why I’m having the issue, as a postdoc in my lab was able to successfully run a very similar script on different data with no error.
Starting fMRIprep 05-03-2024_14:46:29
bids-validator@1.14.1
e[33m1: [WARN] The recommended file /README is very small. Please consider expanding it with additional information about the dataset. (code: 213 - README_FILE_SMALL)e[39m
./README
e[36m Please visit https://neurostars.org/search?q=README_FILE_SMALL for existing conversations about this issue.e[39m
e[34me[4mSummary:e[24me[39m e[34me[4mAvailable Tasks:e[24me[39m e[34me[4mAvailable Modalities:e[39me[24m
4590 Files, 615.43GB explore MRI
104 - Subjects findTarget
1 - Session rest
e[36m If you have any questions, please post on https://neurostars.org/tags/bids.e[39m
240504-09:56:34,514 nipype.workflow IMPORTANT:
Running fMRIPrep version 23.2.1
License NOTICE ##################################################
fMRIPrep 23.2.1
Copyright 2023 The NiPreps Developers.
This product includes software developed by
the NiPreps Community (https://nipreps.org/).
Portions of this software were developed at the Department of
Psychology at Stanford University, Stanford, CA, US.
This software is also distributed as a Docker container image.
The bootstrapping file for the image ("Dockerfile") is licensed
under the MIT License.
This software may be distributed through an add-on package called
"Docker Wrapper" that is under the BSD 3-clause License.
#################################################################
Error in crash log
(node:1530365) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
/var/spool/slurm/d/job42278/slurm_script: line 33: 1505171 Killed singularity run -B $bidsDir:/data -B /mnt/chrastil/users/erica/projects/mlindivPreproc:/out -B $fsLicense:/fs --cleanenv $containers/fmriprep_v23-2-1.simg /data /out/derivatives/fmriprep -w /out/work/ participant --participant-label $id --fs-license /fs/license.txt
I have followed the Troubleshooting section in (https://fmriprep.org/en/1.5.1/singularity.html) and was able to successfully check my access to the input data folder and BIDS validity, check my access to the output data folder and write permissions (also verified by the error files being written), checked my access and permissions to my $HOME folder.
Killed messages usually indicate out of memory. How much memory/cpu are you devoting to the process? I see that you did not specify memory in your sbatch header, so you are probably using a very low default value not suitable for fmriprep.
Thanks for such a quick response Steven! I’ll find out how much memory per CPU I can use, but I did allocate 4 CPUs, which I’ve been told should be enough to run fMRIprep.
The number of cpus is mainly going to affect the speed at which fmriprep runs, and to a minor extent, the amount of memory it will consume. The total memory devoted to the job is the most important thing to determine here. Especially with as many images you have, you’ll need a good bit (32GB would be good to try, and then adjust as needed from there).
I ran the same subject through my lab’s compute cluster and I am now getting this error:
ValueError: 'desc' is not a recognized entity.
I found here that “This error usually happens when derivatives that are not inside a derivatives/ subdirectory are found in the input dataset,” but I could not find much more on the matter.
I am running my script (using the line mentioned above) from a directory that contains a subdirectories named “code” and “derivatives/fmriprep/logs” (see below):
(current_working_directory) $ ls
code derivatives hopper_output_002.txt work
Is this issue stemming from the existing derivatives folder in my BIDS data path? Please correct me if I’m wrong, but I don’t see any fmri derivatives in the in the BIDS directory (other that what is contained in the derivatives/ subdirectory.
Here is the output of the bids-validation using my lab’s cluster:
bids-validator /data
bids-validator@1.14.1
(node:2749394) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
1: [WARN] Not all subjects contain the same files. Each subject should contain the same number of files with the same naming unless some files are known to be missing. (code: 38 - INCONSISTENT_SUBJECTS)
./sub-002/anat/sub-002_T2w.json
Evidence: Subject: sub-002; Missing file: sub-002_T2w.json
./sub-002/anat/sub-002_T2w.nii.gz
Evidence: Subject: sub-002; Missing file: sub-002_T2w.nii.gz
./sub-004/anat/sub-004_T2w.json
Evidence: Subject: sub-004; Missing file: sub-004_T2w.json
./sub-004/anat/sub-004_T2w.nii.gz
Evidence: Subject: sub-004; Missing file: sub-004_T2w.nii.gz
./sub-004/dwi/sub-004_run-02_dwi.bval
Evidence: Subject: sub-004; Missing file: sub-004_run-02_dwi.bval
./sub-004/dwi/sub-004_run-02_dwi.bvec
Evidence: Subject: sub-004; Missing file: sub-004_run-02_dwi.bvec
./sub-004/dwi/sub-004_run-02_dwi.json
Evidence: Subject: sub-004; Missing file: sub-004_run-02_dwi.json
./sub-004/dwi/sub-004_run-02_dwi.nii.gz
Evidence: Subject: sub-004; Missing file: sub-004_run-02_dwi.nii.gz
./sub-004/func/sub-004_task-findTarget_run-02_bold.json
Evidence: Subject: sub-004; Missing file: sub-004_task-findTarget_run-02_bold.json
./sub-004/func/sub-004_task-findTarget_run-02_bold.nii.gz
Evidence: Subject: sub-004; Missing file: sub-004_task-findTarget_run-02_bold.nii.gz
... and 199 more files having this issue (Use --verbose to see them all).
Please visit https://neurostars.org/search?q=INCONSISTENT_SUBJECTS for existing conversations about this issue.
2: [WARN] Not all subjects/sessions/runs have the same scanning parameters. (code: 39 - INCONSISTENT_PARAMETERS)
./sub-002/func/sub-002_task-findTarget_run-01_bold.nii.gz
./sub-002/func/sub-002_task-findTarget_run-02_bold.nii.gz
./sub-002/func/sub-002_task-findTarget_run-03_bold.nii.gz
./sub-002/func/sub-002_task-findTarget_run-04_bold.nii.gz
./sub-002/func/sub-002_task-findTarget_run-05_bold.nii.gz
./sub-002/func/sub-002_task-findTarget_run-06_bold.nii.gz
./sub-003/func/sub-003_task-explore_run-01_bold.nii.gz
./sub-003/func/sub-003_task-findTarget_run-01_bold.nii.gz
./sub-003/func/sub-003_task-findTarget_run-02_bold.nii.gz
./sub-003/func/sub-003_task-findTarget_run-03_bold.nii.gz
... and 646 more files having this issue (Use --verbose to see them all).
Please visit https://neurostars.org/search?q=INCONSISTENT_PARAMETERS for existing conversations about this issue.
3: [WARN] The recommended file /README is very small. Please consider expanding it with additional information about the dataset. (code: 213 - README_FILE_SMALL)
./README
Please visit https://neurostars.org/search?q=README_FILE_SMALL for existing conversations about this issue.
Summary: Available Tasks: Available Modalities:
4585 Files, 615.41GB explore MRI
104 - Subjects findTarget
1 - Session rest
If you have any questions, please post on https://neurostars.org/tags/bids.
I did make some changes and I was able to get outputs to my derivatives/fmirprep/ folder:
I copied one subject to a new directory named data in the parent directory preproc where I’m executing the script code/mypreps.sh to run fMRIprep, and I am now setting this as my BIDS directory.
I ran this using my lab’s cluster- not using the SLURM system which has more resources and would be better for me to use for the rest of my ~100 subjects.
Although I was able to get an output to the derivatives folder, it may not feasible for me to copy the rest of the data to my own directory (since it is very large).
Potential solutions are to:
Symlink the lab’s data to my new BIDS directory? (Although I’m not sure how this will interact with the Singularity containers- but was a suggestion from a postdoc in my lab)
Ask for permission to modify our lab’s data folder organization to be BIDS validated and include my fMRIprep execution script there
I am running into a TemplateFlow error when running on the SLURM system. I presume this is because I don’t have internet access when running things on that cluster? Is there a way around this?
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='templateflow.s3.amazonaws.com', port=443): Max retries exceeded with url: /tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_T1w.nii.gz (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7f5cd1f870a0>: Failed to resolve 'templateflow.s3.amazonaws.com' ([Errno -3] Temporary failure in name resolution)"))
Ideally derivatives and code would live inside data (where data is your BIDS root directory with the raw subject data folders).
Symlinking is possible, just be careful and intentional with your mounting strings in the singularity command, keeping in mind both the target and destination of the symlink are mounted. Ideally you just run it from a BIDS valid dataset without symlinks.
Regarding templatflow error, you can download this directory I made here (templateflow.zip - Google Drive) with the files downloaded, upload it to your cluster, run “export SINGULARITYENV_TEMPLATEFLOW_HOME=/path/to/templateflow”, and make sure it is mounted in singularity.