FMRIPREP confounds error (copied columns for multiple subjects)

I have since reprocessed my data with fmriprep v1.2.6, but I came across something very strange in the confounds.tsv files for two subjects preprocessed with v1.0.12.

I am using fmriprep on Sherlock, and processed these subjects in jobs that were submitted a couple of minutes apart. The dataset is longitudinal and each subject has 3 scan sessions (4 runs each session, for a total of 12 runs). I noticed in the confounds files that for runs 1-3 of session 3, the framewise displacement, cosine, an motion parameter columns are the same for these two subjects. Other column values differ, and this only happened for the 1st three runs of the third session. I’ve checked the raw data and there are no issues there. I’m not sure how this could have happened or if the issue has been resolved, but wanted to flag it in case others have encountered similar problems.

I tried to attach the confound files but cannot upload attachments because I am a new user.

Can you attach your submission script? It seems to me that you are running into race conditions due to parallelization.

EDIT: by attach I meant copy and paste

Here is the script for 1 subject (the other is identical, except for the sub label and directory in the singularity call):

#!/bin/bash
#
#all commands that start with SBATCH contain commands that are just used by SLURM for scheduling
#################
#set a job name  
#SBATCH --job-name=Framing
#################  
#a file for job output, you can check job progress, append the job ID with %j to make it unique
#SBATCH --output=Framing.%j.out
#################
# a file for errors from the job
#SBATCH --error=Framing.%j.err
#################
#time you think you need; default is 2 hours
#format could be dd-hh:mm:ss, hh:mm:ss, mm:ss, or mm
#SBATCH --time=47:00:00
#################
#Quality of Service (QOS); think of it as job priority, there is also --qos=long for with a max job length of 7 days, qos normal is 48 hours.
# REMOVE "normal" and set to "long" if you want your job to run longer than 48 hours,  
# NOTE- in the hns partition the default max run time is 7 days , so you wont need to include qos
#SBATCH --qos=normal
# We are submitting to the dev partition, there are several on sherlock: normal, gpu, owners, hns, bigmem (jobs requiring >64Gigs RAM) 
# The more partitions you can submit to the less time you will wait, you can submit to multiple partitions with -p at once in comma separated format.
#SBATCH -p awagner,normal,owners 
#################
#number of nodes you are requesting, the more you ask for the longer you wait
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=16
#################
# --mem is memory per node; default is 4000 MB per CPU, remember to ask for enough mem to match your CPU request, since 
# sherlock automatically allocates 8 Gigs of RAM/CPU, if you ask for 8 CPUs you will need 32 Gigs of RAM, so either 
# leave --mem commented out or request >= to the RAM needed for your CPU request.
#SBATCH --mem=64000
#################
# Have SLURM send you an email when the job ends or fails, careful, the email could end up in your clutter folder
# Also, if you submit hundreds of jobs at once you will get hundreds of emails.
#SBATCH --mail-type=END,FAIL # notifications for job done & fail
# Remember to change this to your email
#SBATCH --mail-user=%u@stanford.edu
#now run normal batch commands
# note the "CMD BATCH is an  specific command
module purge
module load system
module load singularity
export FS_LICENSE=$SCRATCH/license.txt
# You can use srun if your job is parallel

singularity run --cleanenv /$SCRATCH/fmriprep-1.0.12.simg $SCRATCH/HSR $SCRATCH/HSR/preprocessed participant --participant_label sub-05 --output-space T1w template --fs-license-file $SCRATCH/license.txt -w $SCRATCH/work/sub-05 --nthreads 10 --omp-nthreads 1 --longitudinal -vvv

Hi @corey,

For Sherlock:

#!/bin/bash
#SBATCH --job-name=Framing
#SBATCH --output=Framing.%j.out
#SBATCH --error=Framing.%j.err
#SBATCH --time=47:00:00
#SBATCH --qos=normal
#SBATCH -p awagner,normal,owners 
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=16
#SBATCH --mem=64000
#SBATCH --mail-type=END,FAIL # notifications for job done & fail
#SBATCH --mail-user=%u@stanford.edu

export FS_LICENSE=$SCRATCH/license.txt

singularity run -B $L_SCRATCH:/tmp --cleanenv /home/groups/russpold/singularity_images/poldracklab_fmriprep_1.3.2-2019-03-18-7883f530f81f.simg $SCRATCH/HSR $SCRATCH/HSR/preprocessed participant --participant_label sub-05 --output-space T1w template --fs-license-file $SCRATCH/license.txt -w /tmp/work --nthreads 10 --omp-nthreads 8 --longitudinal -vvv

Several thoughts:

  • I would recommend using local scratch for temporary directories
  • I would deeply recommend you to update to fmriprep-1.3.2 (I have added our image, you may have access to it).
  • I would recommend --omp-nthreads 8 unless you need exact run-to-run reproducibility in registration.
  • I would discard the --longitudinal flag, unless the anatomy of the brains changes a lot between sessions.
  • I would recommend #SBATCH --array=1-2 to run in parallel (instead of submitting several jobs).

Let me know if the configuration I suggested worked for you

Thank you so much for your helpful comments. I have rerun the first few subjects with version 1.2.6, would you recommend using 1.3.2 and reruning these 5 subjects again vs proceeding with 1.2.6?

For the --array flag: if I were to run participants 4,5, and 6 at once (for example), would this read “–array=4-6”? I wasn’t able to find this in the fMRIPREP documentation.

Thanks!

This is how I run fmriprep on Sherlock (using job arrays):

#!/bin/bash
#
#SBATCH -J fmriprep
#SBATCH --array=1-13%5
#SBATCH --time=48:00:00
#SBATCH -n 1
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-cpu=4G
#SBATCH -p russpold,owners,normal
# Outputs ----------------------------------
#SBATCH -o log/%x-%A-%a.out
#SBATCH -e log/%x-%A-%a.err
#SBATCH --mail-user=%u@stanford.edu
#SBATCH --mail-type=ALL
# ------------------------------------------
BIDS_DIR="$OAK/data/openfmri/ds000003"
export TEMPLATEFLOW_HOME="/oak/stanford/groups/russpold/data/templateflow"
SINGULARITY_CMD="singularity run -B $OAK:$OAK -B $L_SCRATCH:/tmp $SINGULARITY_BIN/poldracklab_fmriprep_1.3.2-2019-03-18-7883f530f81f.simg"
OUTPUT_DIR="${OAK}/data/openfmri/derivatives/ds000003/fmriprep_1.3.2"
unset PYTHONPATH
export FS_LICENSE=$HOME/.freesurfer.txt
subject=$( find ${BIDS_DIR} -maxdepth 1 -type d -name "sub-*"  | sort | sed "${SLURM_ARRAY_TASK_ID}q;d" | sed "s/sub-//g" | sed 's!.*/!!' )
cmd="${SINGULARITY_CMD} ${BIDS_DIR} ${OUTPUT_DIR} participant --participant-label $subject -w /tmp/work/ --skip_bids_validation --omp-nthreads 8 --nthreads 12 --mem_mb 30000 --output-space T1w template fsnative fsaverage5 --template-resampling-grid 2mm --use-syn-sdc --notrack --cifti-output"
echo Running task ${SLURM_ARRAY_TASK_ID}
echo Commandline: $cmd
eval $cmd
exitcode=$?
if [ "$exitcode" -ne "0" ]; then
    echo "$subject" >> failed_subjects.${SLURM_ARRAY_JOB_ID}
    echo "${SLURM_ARRAY_TASK_ID}" >> failed_taskids.${SLURM_ARRAY_JOB_ID}
fi
echo Finished tasks ${SLURM_ARRAY_TASK_ID} with exit code $exitcode

The magic happens in this line:

subject=$( find ${BIDS_DIR} -maxdepth 1 -type d -name "sub-*"  | sort | sed "${SLURM_ARRAY_TASK_ID}q;d" | sed "s/sub-//g" | sed 's!.*/!!' )

All the folders that look like valid subject subfolders are searched and the current array ID is extracted from the list.

You can do what you suggested above, but that would imply that your subjects have an index as participant label, and that you don’t have more than 1000 different indices… But still will require some massaging, so my suggestion will be more flexible.

Regarding fMRIPrep version - yes, please rerun everything with the new version. You can keep your freesurfer for reuse though. What I usually have is:

  • Dataset dir: /some/path/to/BIDS/studyname
  • Outputs are directed to: /some/path/to/BIDS/studyname/derivatives/fmriprep-<version>.

Then, once I have run all my subjects through some fmriprep version, I extract the freesufer folder and make it available to new versions this way:

mv /some/path/to/BIDS/studyname/derivatives/fmriprep-<version> /some/path/to/BIDS/studyname/derivatives/freesurfer-6.0.1
ln -s /some/path/to/BIDS/studyname/derivatives/freesurfer-6.0.1 /some/path/to/BIDS/studyname/derivatives/fmriprep-<version>/freesurfer

Then, to start working on a new version of fMRIPrep:

mkdir -p /some/path/to/BIDS/studyname/derivatives/fmriprep-<new_version>/
ln -s /some/path/to/BIDS/studyname/derivatives/freesurfer-6.0.1 /some/path/to/BIDS/studyname/derivatives/fmriprep-<new_version>/freesurfer

HTH

2 posts were split to a new topic: FileExistsError: [Errno 17] File exists: ‘/out/freesurfer’