I am trying to get fMRIPREP v 23.2.1 to run on my school’s HPC via singularity and I’m not sure why my job is getting killed. I am trying to run 1 subject only (sub-002), and after I am able to do that I would like to be able to loop through the rest (total of 113).
I’m using a shell script to run singularity on a SLURM system.
I am able to submit a job successfully and checked that it was running (approximately ~12 hours or so) before it quits with nothing added to my derivatives/fmriprep/ folder. I’m not sure why I’m having the issue, as a postdoc in my lab was able to successfully run a very similar script on different data with no error.
This is the line I use to run the code:
sbatch code/mypreps.sh 002
Here is my script
#! /bin/bash
#SBATCH --partition=standard
#SBATCH --job-name=fmriPrep
#SBATCH -c 4
#SBATCH -o /mnt/chrastil/users/erica/projects/mlindivPreproc/code/gridlog/mlindiv-prep_%j.txt
#SBATCH -e /mnt/chrastil/users/erica/projects/mlindivPreproc/code/gridlog/slurmoutput/%j_slurm.err
# Variables
subs=("$@")
bidsDir=/mnt/chrastil/data/MLINDIV
containers=/mnt/chrastil/users/erica/projects/mlindivPreproc/code/singularity
fsLicense=/mnt/chrastil/users/erica/projects/mlindivPreproc/code/freesurfer
current_date_time="`date +%m-%d-%Y_%H:%M:%S`";
for id in "${subs[@]}"
do
# Run fMRIprep
echo "Starting fMRIprep $current_date_time"
singularity run -B $bidsDir:/data -B /mnt/chrastil/users/erica/projects/mlindivPreproc:/out -B $fsLicense:/fs \
--cleanenv $containers/fmriprep_v23-2-1.simg \
/data \
/out/derivatives/fmriprep \
-w /out/work/ \
participant \
--participant-label $id \
--fs-license /fs/license.txt \
--n_cpus $SLURM_CPUS_PER_TASK
done
Here is the output of my fMRIprep
Starting fMRIprep 05-03-2024_14:46:29
bids-validator@1.14.1
e[33m1: [WARN] The recommended file /README is very small. Please consider expanding it with additional information about the dataset. (code: 213 - README_FILE_SMALL)e[39m
./README
e[36m Please visit https://neurostars.org/search?q=README_FILE_SMALL for existing conversations about this issue.e[39m
e[34me[4mSummary:e[24me[39m e[34me[4mAvailable Tasks:e[24me[39m e[34me[4mAvailable Modalities:e[39me[24m
4590 Files, 615.43GB explore MRI
104 - Subjects findTarget
1 - Session rest
e[36m If you have any questions, please post on https://neurostars.org/tags/bids.e[39m
240504-09:56:34,514 nipype.workflow IMPORTANT:
Running fMRIPrep version 23.2.1
License NOTICE ##################################################
fMRIPrep 23.2.1
Copyright 2023 The NiPreps Developers.
This product includes software developed by
the NiPreps Community (https://nipreps.org/).
Portions of this software were developed at the Department of
Psychology at Stanford University, Stanford, CA, US.
This software is also distributed as a Docker container image.
The bootstrapping file for the image ("Dockerfile") is licensed
under the MIT License.
This software may be distributed through an add-on package called
"Docker Wrapper" that is under the BSD 3-clause License.
#################################################################
Error in crash log
(node:1530365) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
/var/spool/slurm/d/job42278/slurm_script: line 33: 1505171 Killed singularity run -B $bidsDir:/data -B /mnt/chrastil/users/erica/projects/mlindivPreproc:/out -B $fsLicense:/fs --cleanenv $containers/fmriprep_v23-2-1.simg /data /out/derivatives/fmriprep -w /out/work/ participant --participant-label $id --fs-license /fs/license.txt
Here is my the structure of my data
sub-002
├── anat
│ ├── sub-002_T1w.json
│ └── sub-002_T1w.nii.gz
├── dwi
│ ├── sub-002_run-01_dwi.bval
│ ├── sub-002_run-01_dwi.bvec
│ ├── sub-002_run-01_dwi.json
│ ├── sub-002_run-01_dwi.nii.gz
│ ├── sub-002_run-02_dwi.bval
│ ├── sub-002_run-02_dwi.bvec
│ ├── sub-002_run-02_dwi.json
│ └── sub-002_run-02_dwi.nii.gz
├── fmap
│ ├── sub-002_magnitude1.json
│ ├── sub-002_magnitude1.nii.gz
│ ├── sub-002_magnitude2.json
│ ├── sub-002_magnitude2.nii.gz
│ ├── sub-002_phasediff.json
│ └── sub-002_phasediff.nii.gz
├── func
│ ├── sub-002_task-explore_run-01_bold.json
│ ├── sub-002_task-explore_run-01_bold.nii.gz
│ ├── sub-002_task-explore_run-01_events.tsv
│ ├── sub-002_task-explore_run-02_bold.json
│ ├── sub-002_task-explore_run-02_bold.nii.gz
│ ├── sub-002_task-explore_run-02_events.tsv
│ ├── sub-002_task-findTarget_run-01_bold.json
│ ├── sub-002_task-findTarget_run-01_bold.nii.gz
│ ├── sub-002_task-findTarget_run-01_events.tsv
│ ├── sub-002_task-findTarget_run-02_bold.json
│ ├── sub-002_task-findTarget_run-02_bold.nii.gz
│ ├── sub-002_task-findTarget_run-02_events.tsv
│ ├── sub-002_task-findTarget_run-03_bold.json
│ ├── sub-002_task-findTarget_run-03_bold.nii.gz
│ ├── sub-002_task-findTarget_run-03_events.tsv
│ ├── sub-002_task-findTarget_run-04_bold.json
│ ├── sub-002_task-findTarget_run-04_bold.nii.gz
│ ├── sub-002_task-findTarget_run-04_events.tsv
│ ├── sub-002_task-findTarget_run-05_bold.json
│ ├── sub-002_task-findTarget_run-05_bold.nii.gz
│ ├── sub-002_task-findTarget_run-05_events.tsv
│ ├── sub-002_task-findTarget_run-06_bold.json
│ ├── sub-002_task-findTarget_run-06_bold.nii.gz
│ ├── sub-002_task-findTarget_run-06_events.tsv
│ ├── sub-002_task-rest_run-01_bold.json
│ ├── sub-002_task-rest_run-01_bold.nii.gz
│ └── sub-002_task-rest_run-01_events.tsv
└── sub-002_scans.tsv
Here is what I have tried
I have followed the Troubleshooting section in (https://fmriprep.org/en/1.5.1/singularity.html) and was able to successfully check my access to the input data folder and BIDS validity, check my access to the output data folder and write permissions (also verified by the error files being written), checked my access and permissions to my $HOME folder.