Fmriprep with singularity stuck

1

Hi all,

I’m trying to figure out what I’m doing wrong because I have run multiple tests but fmriprep seems to not finish and also doesn’t throw any errors. I am using the spartan hpc cluster (melbourne) with this script:

Singularity script

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --time=0-25:00:00
#SBATCH --mem=30G

myproject=$'/data/cephfs/punim0752/fmriprep'

module load Singularity/2.5.0-GCC-6.2.0

cd $myproject/fmriprep_test_10sub

SUBJECT=`sed -n -e "${SLURM_ARRAY_TASK_ID}p" list_subjects.txt`

# launch processing for each subject
singularity run --cleanenv -B "${myproject}":"${myproject}" "${myproject}/fmriprep_image/fmriprep-1.2.5.simg" \
"${myproject}/fmriprep_test_10subs/fmriprep_spartan_10subjects" "${myproject}/fmriprep_10subs_output/" \
participant \
--participant-label "sub-${SUBJECT}" --nthreads 1 --n_cpus 8 --fs-license-file "${myproject}/fmriprep_test_10subs/license.txt" --use-aroma

The step that it appears to get stuck on for >12 hours before it gets aborted at 25 hours is this:

190120-09:15:30,930 nipype.workflow INFO:
	 [Node] Finished "fmriprep_wf.single_subject_01_wf.anat_preproc_wf.anat_derivatives_wf.lta_2_itk".

Output file:

I also tried running it on a local machine (with 8CPUS and 16GB memory) and it didn’t finish for 48 hours. I tried with no reconall and it finished in ~1.5 hours.

2

As per the last line of the log:

 [Node] Finished "fmriprep_wf.single_subject_01_wf.anat_preproc_wf.anat_derivatives_wf.lta_2_itk".

it seems to me that this node is not the actuall problem.

The problem has the looks of a memory issue, however the fact that you were running with --nthreads 1 makes me doubtful.

Could you run this one subject exactly the same way but adding the -vvv flag, so we get extra verbose output? Please make sure you don’t reuse the work directory (otherwise the process will likely end up finishing).

3

Hi! So I tried several times and it didn’t finish, not even when I used the old working directory. I uploaded the verbose output files on github as well.

I ended up running freesurfer separately and then running fmriprep and it worked! So I’m running my entire dataset with pre-done freesurfer now.

4

Seems worth linking to the github issue where this was discussed more thoroughly: https://github.com/poldracklab/fmriprep/issues/1478