This is how I run fmriprep on Sherlock (using job arrays):
#!/bin/bash
#
#SBATCH -J fmriprep
#SBATCH --array=1-13%5
#SBATCH --time=48:00:00
#SBATCH -n 1
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-cpu=4G
#SBATCH -p russpold,owners,normal
# Outputs ----------------------------------
#SBATCH -o log/%x-%A-%a.out
#SBATCH -e log/%x-%A-%a.err
#SBATCH --mail-user=%u@stanford.edu
#SBATCH --mail-type=ALL
# ------------------------------------------
BIDS_DIR="$OAK/data/openfmri/ds000003"
export TEMPLATEFLOW_HOME="/oak/stanford/groups/russpold/data/templateflow"
SINGULARITY_CMD="singularity run -B $OAK:$OAK -B $L_SCRATCH:/tmp $SINGULARITY_BIN/poldracklab_fmriprep_1.3.2-2019-03-18-7883f530f81f.simg"
OUTPUT_DIR="${OAK}/data/openfmri/derivatives/ds000003/fmriprep_1.3.2"
unset PYTHONPATH
export FS_LICENSE=$HOME/.freesurfer.txt
subject=$( find ${BIDS_DIR} -maxdepth 1 -type d -name "sub-*" | sort | sed "${SLURM_ARRAY_TASK_ID}q;d" | sed "s/sub-//g" | sed 's!.*/!!' )
cmd="${SINGULARITY_CMD} ${BIDS_DIR} ${OUTPUT_DIR} participant --participant-label $subject -w /tmp/work/ --skip_bids_validation --omp-nthreads 8 --nthreads 12 --mem_mb 30000 --output-space T1w template fsnative fsaverage5 --template-resampling-grid 2mm --use-syn-sdc --notrack --cifti-output"
echo Running task ${SLURM_ARRAY_TASK_ID}
echo Commandline: $cmd
eval $cmd
exitcode=$?
if [ "$exitcode" -ne "0" ]; then
echo "$subject" >> failed_subjects.${SLURM_ARRAY_JOB_ID}
echo "${SLURM_ARRAY_TASK_ID}" >> failed_taskids.${SLURM_ARRAY_JOB_ID}
fi
echo Finished tasks ${SLURM_ARRAY_TASK_ID} with exit code $exitcode
The magic happens in this line:
subject=$( find ${BIDS_DIR} -maxdepth 1 -type d -name "sub-*" | sort | sed "${SLURM_ARRAY_TASK_ID}q;d" | sed "s/sub-//g" | sed 's!.*/!!' )
All the folders that look like valid subject subfolders are searched and the current array ID is extracted from the list.
You can do what you suggested above, but that would imply that your subjects have an index as participant label, and that you don’t have more than 1000 different indices… But still will require some massaging, so my suggestion will be more flexible.
Regarding fMRIPrep version - yes, please rerun everything with the new version. You can keep your freesurfer for reuse though. What I usually have is:
- Dataset dir:
/some/path/to/BIDS/studyname
- Outputs are directed to:
/some/path/to/BIDS/studyname/derivatives/fmriprep-<version>.
Then, once I have run all my subjects through some fmriprep version, I extract the freesufer folder and make it available to new versions this way:
mv /some/path/to/BIDS/studyname/derivatives/fmriprep-<version> /some/path/to/BIDS/studyname/derivatives/freesurfer-6.0.1
ln -s /some/path/to/BIDS/studyname/derivatives/freesurfer-6.0.1 /some/path/to/BIDS/studyname/derivatives/fmriprep-<version>/freesurfer
Then, to start working on a new version of fMRIPrep:
mkdir -p /some/path/to/BIDS/studyname/derivatives/fmriprep-<new_version>/
ln -s /some/path/to/BIDS/studyname/derivatives/freesurfer-6.0.1 /some/path/to/BIDS/studyname/derivatives/fmriprep-<new_version>/freesurfer
HTH