I think I was able to figure out how to run fmriprep off bsub. It looks like bsub’s Docker wrapper is not ideal for fmriprep but it is still possible.
The first challenge is that, with at least our environment, when you load into a batch node through bsub you erase all of the docker images environmental variables. This overwrites fMRIpreps variables with variables you had set in the node you launched said batch node from, as a result making fMRIprep unaware of where it’s tools are. To resolve this issue we can break our script into two parts, one script that defines the environmental variables telling it where it’s tools are and then a bsub file that actually run’s fMRIprep itself. There is two things you can do to define environmental variables in the first .sh file, the first is to set the following variables to not preserve the launching nodes variables (which didn’t seem to work for us but might for you)…
export LSF_DOCKER_PRESERVE_ENVIRONMENT=false
Our environment didn’t seem to utilize this variable appropriately and was still overwriting at least some environmental variables. In this case you can write all of the environmental variables yourself! To find these environmental variables you need to look through the fMRIPrep Dockerfile but just to save you some time here’s the variables needed for at least nipreps/fmriprep:23.0.2 when I first made this…
# -------------- Set fMRIPrep Environmental -------- #
# This section is used to set the fMRIprep environmental variables based off
# nipreps/fmriprep:23.0.2. Normally this wouldn't be necessary when running
# docker but due to RIS's environment all environmental variables
# are wiped when we load the docker image. This is all based off of fMRIprep's
# Dockerfile where they set ENV varaibles.
#
# NOTE: If fMRIprep stops working when updating to a new version, it could be
# due to one of these environmental variables changes. Consider looking at the
# newer fMRIprep Dockerfile and comparing the ENV variables set to environmental
# variables listed below.
export FREESURFER_HOME="/opt/freesurfer/bin/"
export PATH="$FREESURFER_HOME:$PATH"
export MAMBA_ROOT_PREFIX="/opt/conda"
export PATH="/opt/conda/envs/fmriprep/bin:$PATH"
export UV_USE_IO_URING=0
export DEBIAN_FRONTEND="noninteractive"
export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"
export OS="Linux"
export FS_OVERRIDE=0
export FIX_VERTEX_AREA=""
export FSF_OUTPUT_FORMAT="nii.gz"
export FREESURFER_HOME="/opt/freesurfer"
export SUBJECTS_DIR="$FREESURFER_HOME/subjects"
export FUNCTIONALS_DIR="$FREESURFER_HOME/sessions"
export MNI_DIR="$FREESURFER_HOME/mni"
export LOCAL_DIR="$FREESURFER_HOME/local"
export MINC_BIN_DIR="$FREESURFER_HOME/mni/bin"
export MINC_LIB_DIR="$FREESURFER_HOME/mni/lib"
export MNI_DATAPATH="$FREESURFER_HOME/mni/data"
export PERL5LIB="$MINC_LIB_DIR/perl5/5.8.5"
export MNI_PERL5LIB="$MINC_LIB_DIR/perl5/5.8.5"
export PATH="$FREESURFER_HOME/bin:$FREESURFER_HOME/tktools:$MINC_BIN_DIR:$PATH"
export PATH="/opt/afni-latest:$PATH"
export AFNI_IMSAVE_WARNINGS="NO"
export AFNI_PLUGINPATH="/opt/afni-latest"
export MAMBA_ROOT_PREFIX="/opt/conda"
export PATH="/opt/conda/envs/fmriprep/bin:$PATH"
export CPATH="/opt/conda/envs/fmriprep/include:$CPATH"
export LD_LIBRARY_PATH="/opt/conda/envs/fmriprep/lib:$LD_LIBRARY_PATH"
export MKL_NUM_THREADS=1
export OMP_NUM_THREADS=1
export LANG="C.UTF-8"
export LC_ALL="C.UTF-8"
export PYTHONNOUSERSITE=1
export FSLDIR="/opt/conda/envs/fmriprep"
export FSLOUTPUTTYPE="NIFTI_GZ"
export FSLMULTIFILEQUIT="TRUE"
export FSLLOCKDIR=""
export FSLMACHINELIST=""
export FSLREMOTECALL=""
export FSLGECUDAQ="cuda.q"
export IS_DOCKER_8395080871=1
export PATH="/opt/workbench/bin_linux64:$PATH"
export LD_LIBRARY_PATH="/opt/workbench/lib_linux64:$LD_LIBRARY_PATH"
export LD_LIBRARY_PATH="/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH"
export PATH="/usr/bin/c3d_affine_tool:$PATH"
#export HOME="/home/fmriprep"
The second challenge with running fmriprep is that, at least in the system we are using, we no longer can map folders into the docker image using our -v variables. To map our data we need to using the following variable…
export LSF_DOCKER_VOLUMES="${BIDS_DIR}:/input ${OUTPUT_DIR}:/output ${LICENSE_DIR}:/freesurfer"
NOTE: You may need to import different folders based on your own system configuration, like a network file system or your home directory.
The third challenge with running fmriprep is that bsub requires docker images entry point to be set to /bin/bash but fMRIPrep’s docker image has an entry point that points to fMRIPrep itself like this which can be found in the Dockerfile…
ENTRYPOINT="/opt/conda/envs/fmriprep/bin/fmriprep"
We can fix this by overriding the Dockerfile’s entry point using the following variable in our pre-bsub file and call to our bsub file fmriprep_bsub.sh within the same folder…
export LSF_DOCKER_ENTRYPOINT=/bin/bash
bsub < fmriprep_bsub.sh
Then when we create our bsub file giving all the parameters we need, and maybe a little extra for good measure. For the job we need to create a scratch and output directory directly on the node your running on to ensure fMRIprep can save and access temporary files quick enough. Then once we have set up our environment we can finally call to fMRIprep to run by calling to the Dockerfile entrypoint fMRIprep originally set /opt/conda/envs/fmriprep/bin/fmriprep…
#!/bin/bash
#BSUB -q general-interactive
#BSUB -m general-interactive
#BSUB -a 'docker(nipreps/fmriprep)'
#BSUB -J bsub-fmriprep
#BSUB -n 16
#BSUB -R "select[mem>16000 && tmp>50]"
#BSUB -oo fmriprep-output.txt
mkdir scratch
mkdir output
/opt/conda/envs/fmriprep/bin/fmriprep /input output participant --participant-label sub-100 -w scratch --fs-license-file /freesurfer/license.txt --me-output-echos --output-spaces MNI152NLin6Asym MNI152NLin2009cAsym MNIPediatricAsym:cohort-2:res-2 --n_cpus 16
cp -r output/ Z:/filesystem/study/output/
NOTE: Based on how your system is set up, you may need to alter the HOME before calling to fMRIprep
Then finally on your computing system just call the pre-bsub file you created through a bash call and it should launch the job! I’m still working on getting fMRIprep running completely, I can get it to the point where fMRIprep is running but the scratch files fMRIprep is creating are saving too slowely for fMRIprep to load in time. Will update though once I figure out the remaining issue with our computing environment and run a full run sucessfully!