Freesurfer CPU time limit of 2 hours with fMRIPrep 24.0.0

Software Support
, ,
1

Summary of what happened:

I am trying to preprocess the Midnight Scan Club dataset with fMRIPrep version 24.0.0 on my institution’s cluster. I am using BABS to manage the SLURM job submission and output wrangling. For some reason, even though I set the job’s time limit to 10 days, the Freesurfer CPU time limit is being set to 2 hours, and all of the jobs are failing due to that CPU time limit.

I’m not sure if the problem is with how Freesurfer is called in fMRIPrep, or if there’s an environment variable in the fMRIPrep container that controls this, or something else entirely (e.g., some setting on the cluster I’m using).

Command used:

BABS config file

# This is an example config yaml file for:
#   BIDS App:         fMRIPrep ("fmriprep")
#   BIDS App version: 24.0.0
#   Task:             regular use
#   Which system:     Slurm

# WARNING!!!
#   This is only an example, which may not necessarily fit your purpose,
#   or be an optimized solution for your case,
#   or be compatible to the BIDS App version you're using.
#   Therefore, please change and tailor it for your case before use it!!!

# Arguments in `singularity run`:
singularity_run:
    -w: "$BABS_TMPDIR"   # this is a placeholder recognized by BABS.
    --n_cpus: '1'
    --omp-nthreads: '1'
    --stop-on-first-crash: ""
    --fs-license-file: "/cbica/home/salot/datasets/mobile-phenomics/freesurfer_license.txt"
    --skip-bids-validation: ""
    --output-spaces: "MNI152NLin6Asym:res-2"
    --cifti-output: 91k
    -v: '-v'
    --aggregate-session-reports: "1"
    --project-goodvoxels: ""
    --level: "full"

# Output foldername(s) to be zipped, and the BIDS App version to be included in the zip filename(s):
#   As fMRIPrep will use BIDS output layout, we need to ask BABS to create a folder 'fmriprep' to wrap all derivatives:
zip_foldernames:
    $TO_CREATE_FOLDER: "true"
    fmriprep: "24-0-0"   # folder 'fmriprep' will be zipped into 'sub-xx_(ses-yy_)fmriprep-23-1-3.zip'

# How much cluster resources it needs:
cluster_resources:
    interpreting_shell: "/bin/bash"
    hard_memory_limit: 200G
    temporary_disk_space: 200G
    hard_runtime_limit: "10-00:00:00"

# Necessary commands to be run first:
script_preamble: |
    conda activate babsenv

# Where to run the jobs:
job_compute_space: "/cbica/comp_space/salot/msc-work"

# Below is to filter out subjects (or sessions). Only those with required files will be kept.
required_files:
    $INPUT_DATASET_#1:
        - "func/*_bold.nii*"
        - "anat/*_T1w.nii*"

# Alert messages that might be found in log files of failed jobs:
#   These messages may be helpful for debugging errors in failed jobs.
alert_log_messages:
    stdout:
        - "Excessive topologic defect encountered"
        - "Cannot allocate memory"
        - "mris_curvature_stats: Could not open file"
        - "Numerical result out of range"
        - "fMRIPrep failed"

fMRIPrep job

#!/bin/bash
set -e -u -x

subid="$1"


mkdir -p ${PWD}/.git/tmp/wkdir
singularity run --cleanenv \
        -B ${PWD} \
        -B /cbica/home/salot/datasets/mobile-phenomics/freesurfer_license.txt:/SGLR/FREESURFER_HOME/license.txt \
        containers/.datalad/environments/fmriprep-24/image \
        inputs/data/BIDS \
        outputs/fmriprep \
        participant \
        -w ${PWD}/.git/tmp/wkdir \
        --n_cpus 1 \
        --omp-nthreads 1 \
        --stop-on-first-crash \
        --fs-license-file /SGLR/FREESURFER_HOME/license.txt \
        --skip-bids-validation \
        --output-spaces MNI152NLin6Asym:res-2 \
        --cifti-output 91k \
        -v -v \
        --aggregate-session-reports 1 \
        --project-goodvoxels \
        --level full \
        --participant-label "${subid}"

cd outputs
7z a ../${subid}_fmriprep-24-0-0.zip fmriprep
cd ..
rm -rf outputs .git/tmp/wkdir

Version:

fMRIPrep version 24.0.0
babs version 0.0.8

Environment (Docker, Singularity / Apptainer, custom installation):

apptainer version 1.1.8-1.el7

Data formatted according to a validatable standard? Please provide the output of the validator:

Yes. I am using the Midnight Scan Club data from OpenNeuro.

Relevant log outputs (up to 20 lines):

from the job’s stderr file

[INFO] == Command start (output follows) ===== 
+ subid=sub-MSC04
+ mkdir -p /cbica/comp_space/salot/msc-work/job-258350-sub-MSC04/ds/.git/tmp/wkdir
+ singularity run --cleanenv -B /cbica/comp_space/salot/msc-work/job-258350-sub-MSC04/ds -B /cbica/home/salot/datasets/mobile-phenomics/freesurfer_license.txt:/SGLR/FREESURFER_HOME/license.txt containers/.datalad/environments/fmriprep-24/image inputs/data/BIDS outputs/fmriprep participant -w /cbica/comp_space/salot/msc-work/job-258350-sub-MSC04/ds/.git/tmp/wkdir --n_cpus 1 --omp-nthreads 1 --stop-on-first-crash --fs-license-file /SGLR/FREESURFER_HOME/license.txt --skip-bids-validation --output-spaces MNI152NLin6Asym:res-2 --cifti-output 91k -v -v --aggregate-session-reports 1 --project-goodvoxels --level full --participant-label sub-MSC04
./code/fmriprep-24_zip.sh: line 27: 107818 CPU time limit exceeded singularity run --cleanenv -B ${PWD} -B /cbica/home/salot/datasets/mobile-phenomics/freesurfer_license.txt:/SGLR/FREESURFER_HOME/license.txt containers/.datalad/environments/fmriprep-24/image inputs/data/BIDS outputs/fmriprep participant -w ${PWD}/.git/tmp/wkdir --n_cpus 1 --omp-nthreads 1 --stop-on-first-crash --fs-license-file /SGLR/FREESURFER_HOME/license.txt --skip-bids-validation --output-spaces MNI152NLin6Asym:res-2 --cifti-output 91k -v -v --aggregate-session-reports 1 --project-goodvoxels --level full --participant-label "${subid}"
[INFO] == Command exit (modification check follows) =====