Running fMRIPrep through bsub

Summary of what happened:

We recently transitioned to a new computing environment that uses bsub batch submission system to run HPC jobs which rely a lot on Docker, but it does not seem to cooperate well with fmriprep Docker images (at least with how I’m calling it now). The problem I’m running into with bsub is that it has a unique docker wrapper system that does not seem to call fMRIprep to run. I’m able to have the system load the docker image and our storage but when it comes to actually calling fMRIprep, it doesn’t seem to recognize or call to the fMRIprep command and pass in the options I’m calling. Does anyone have any suggestions running fMRIprep with bsub?

Thanks for your help!

Command used (and if a helper script was used, a link to the helper script or the command generated):

pre_bsub.sh

#!/bin/sh

SUBJECT=$1

STORAGE_ALLOCATION=/storage1/fs1/Active/
BIDS_DIR=${STORAGE_ALLOCATION}moochie/study_data/CARE/ME_MRI_data/
SCRATCH_DIR=${STORAGE_ALLOCATION}moochie/analysis/CARE/ME_scratch/
OUTPUT_DIR=${STORAGE_ALLOCATION}moochie/analysis/CARE/ME_fMRIPrep_data/
LICENSE_DIR=${STORAGE_ALLOCATION}moochie/github/LCBDtools/scripts/MRI/fmriprep/

# export OUTPUT_DIRS
export LSF_DOCKER_VOLUMES="/storage1/fs1/:/storage1/fs1/ ${BIDS_DIR}:/input ${SCRATCH_DIR}:/scratch ${OUTPUT_DIR}:/output ${LICENSE_DIR}:/freesurfer"
export LSF_DOCKER_PRESERVE_ENVIRONMENT=true

bsub < fmriprep_bsub.sh                         

fmriprep_bsub.sh

#!/bin/bash 
#BSUB -G compute
#BSUB -q general-interactive
#BSUB -m general-interactive
#BSUB -a 'docker(nipreps/fmriprep)'
#BSUB -J bsub-fmriprep
#BSUB -n 8
#BSUB -R "select[mem>30000] span[hosts=1]"
#BSUB -oo fmriprep-output.txt

$BIDS_DIR $OUTPUT_DIR --participant --participant-label $SUBJECT  -w ${OUTPUT_DIR}scratch --fs-license-file /freesurfer/license.txt --level resampling --me-output-echos --output-spaces MNI152NLin6Asym MNI152NLin2009cAsym MNIPediatricAsym:cohort-2:res-2 --n_cpus 8 --low-mem

Version:

nipreps/fmriprep:latest(23.22.2)

Environment (Docker, Singularity / Apptainer, custom installation):

fMRIprep is being run through a batch submission (bsub) system using nipreps/fmriprep docker image to load and run fMRIprep

Data formatted according to a validatable standard? Please provide the output of the validator:

I don’t believe the bids validator is running yet within this new environment but in our previous environment the bids validator was running successfully and starting fMRIprep with ease. Happy to provide output if y’all think it would be useful!

Relevant log outputs (up to 20 lines):

Your job looked like:

Exited with exit code 1.

The output (if any) is above this job summary.

Screenshots / relevant information:

Troubleshooting wise I’ve done a couple of things to try and resolve the issue…

  1. Called directly to fmriprep from the final line of our bsub script (i.e. fmriprep $BIDS_DIR $OUTPUT_DIR --participant). Outcome was that fmriprep was not a recognized command
  2. Called to the fmriprep docker image entry point (i.e. /opt/conda/envs/fmriprep/bin/fmriprep $BIDS_DIR $OUTPUT_DIR --participant)

Thanks for your time and help with this!