XCP-D Environmental Variables Overwritten

Software Support
1

Summary of what happened:

We’re in a bit of a weird computing environment that utilize bsub to manage their batch submission computing. A weird limitation of the system is that it wipes the environmental variables set within the docker image with the environmental variables set in the computing environment the computing node was launched from. To fix this issue previously we’ve taken the environmental variables set through the Dockerfile and set them within our computing environment before launching a given job. It appears though that most of the environmental variables for the XCP_D docker image are set within the xcp_d_build base docker image which does not appear to have a public Dockerfile or maybe was reimportant from another private base image pennlinc uses.

TL;DR - My environmental variables are being overwritten for the XCP_D docker image but I can’t seem to find them to reset them, @tsalo do you happen to have those variables on hand that I could copy over to my submission script?

Command used (and if a helper script was used, a link to the helper script or the command generated):

Script to set up the docker image needing XCP_D environmental variables…

#!/bin/sh

# ----------- System Environment Variables-------------- #

# Grab passed in subject
export SUBJECT=$1

# Define storage allocations areas of interest
export STORAGE_ALLOCATION=/storage1/fs1/perlmansusan/Active/
export FMRIPREP_DIR=${STORAGE_ALLOCATION}moochie/analysis/CARE/ME_fMRIPrep_data/derivates/fmriprep/
export SCRATCH_DIR=${STORAGE_ALLOCATION}moochie/analysis/CARE/ME_scratch/
export LICENSE_DIR=${STORAGE_ALLOCATION}moochie/github/LCBDtools/scripts/MRI/fmriprep/
export CONFOUND_DIR=${STORAGE_ALLOCATION}moochie/analysis/CARE/custom_confounds/

# Alter the XCP_D entry point to be /bin/bash so it cooperated with bsub and XCP_D
# is not automatically called upon entering the docker image without options.
export LSF_DOCKER_ENTRYPOINT=/bin/bash

# Potentially unimportant variable, used to make sure the environmental variables 
# in the current shell transfer to the docker image being loaded but does not appear
# to work.
export LSF_DOCKER_PRESERVE_ENVIRONMENT=true

export HOME="/scratch1/fs1/perlmansusan"

# -------- Call to bsub to run the bsub file ------------ #
export OUTPUT_DIR=${STORAGE_ALLOCATION}moochie/analysis/CARE/ME_XCP_khalil_data/
export LSF_DOCKER_VOLUMES="/storage1/fs1/perlmansusan/Active:/storage1/fs1/perlmansusan/Active /scratch1/fs1/perlmansusan:/scratch1/fs1/perlmansusan ${FMRIPREP_DIR}:/fmriprep ${OUTPUT_DIR}:/output ${LICENSE_DIR}:/freesurfer ${CONFOUND_DIR}:/custom_confounds /home/$USER:/home/$USER"
bsub -J $SUBJECT-xcp_d -oo logs/xcp_d/$SUBJECT-xcp_d.log -g /$USER/preprocessing < xcp-d_bsub.sh

Script to run XCP_D within the docker image…

#!/bin/sh 
#BSUB -G compute-perlmansusan
#BSUB -q general
#BSUB -m general
#BSUB -a 'docker(pennlinc/xcp_d:latest)'
#BSUB -n 16
#BSUB -R 'rusage[mem=26GB] select[mem>25000 && tmp>250]'

TMP_DIR=`ls /tmp/`

mkdir /tmp/$TMP_DIR/scratch

mkdir /tmp/$TMP_DIR/output

/usr/local/miniconda/bin/xcp_d /fmriprep /tmp/$TMP_DIR/output participant \
    --participant-label $SUBJECT \
    -w /tmp/$TMP_DIR/scratch \
    --low-mem \
    --fs-license-file $LICENSE_DIR\license.txt \
    --mode none \
    --input-type fmriprep \
    --file-format nifti \
    --despike y \
    -p none \
    -c /custom_confounds \
    --smoothing 0 \
    --motion-filter-type none \
    -r 40 \
    -f 0.3 \
    --min-time 0 \
    --output-type censored \
    --min-coverage 0.25 \
    --abcc-qc y \
    --combine-runs n\
    --linc-qc y \
    --warp-surfaces-native2std n

cp -r /tmp/$TMP_DIR/output/ /output/

Version:

XCP_D version 0.0.12

Environment (Docker, Singularity / Apptainer, custom installation):

Docker through LSF bsub

Relevant log outputs (up to 20 lines):

latest: Pulling from pennlinc/xcp_d
09db6f815738: Pulling fs layer
bbf6c0d5a635: Pulling fs layer
cc0d5765e2ac: Pulling fs layer
f3ed34a8f348: Pulling fs layer
886a90d8482d: Pulling fs layer
62a1a0967c60: Pulling fs layer
b94f174efd64: Pulling fs layer
00b62200211a: Pulling fs layer
ebc3d0abc32d: Pulling fs layer
8190a249acbf: Pulling fs layer
f3ed34a8f348: Waiting
71366ae47cf1: Pulling fs layer
886a90d8482d: Waiting
2cb2b2033c34: Pulling fs layer
d9202e6a9c4c: Pulling fs layer
d29d881a5264: Pulling fs layer
0859b3b3be1f: Pulling fs layer
c835a029766a: Pulling fs layer
ebc3d0abc32d: Waiting
62a1a0967c60: Waiting
257f05a2d884: Pulling fs layer
00b62200211a: Waiting
b94f174efd64: Waiting
f0399f54a3c5: Pulling fs layer
8190a249acbf: Waiting
d29d881a5264: Waiting
cc2dcfc5fa29: Pulling fs layer
71366ae47cf1: Waiting
d9202e6a9c4c: Waiting
0859b3b3be1f: Waiting
ebf38c798b19: Pulling fs layer
2cb2b2033c34: Waiting
393b90a8df47: Pulling fs layer
257f05a2d884: Waiting
c835a029766a: Waiting
107d3ecef608: Pulling fs layer
f0399f54a3c5: Waiting
ebf38c798b19: Waiting
cc2dcfc5fa29: Waiting
48c18c59c65c: Pulling fs layer
393b90a8df47: Waiting
ac4fa6c27f81: Pulling fs layer
107d3ecef608: Waiting
24ef8b27b1a7: Pulling fs layer
48c18c59c65c: Waiting
ac4fa6c27f81: Waiting
f07aaf596b5f: Pulling fs layer
24ef8b27b1a7: Waiting
a23100ee9050: Pulling fs layer
3a99c9ac11c0: Pulling fs layer
2baeb5b5a333: Pulling fs layer
fccec2131deb: Pulling fs layer
3a99c9ac11c0: Waiting
4b495d28ad66: Pulling fs layer
a23100ee9050: Waiting
18f868fcf1ae: Pulling fs layer
f07aaf596b5f: Waiting
fccec2131deb: Waiting
2baeb5b5a333: Waiting
5e8e43a05ae3: Pulling fs layer
4b495d28ad66: Waiting
18f868fcf1ae: Waiting
75e29d109d54: Pulling fs layer
3c704c19862b: Pulling fs layer
75e29d109d54: Waiting
dba0fde36498: Pulling fs layer
bc0ffe24b2bc: Pulling fs layer
686a99242ad3: Pulling fs layer
4f4fb700ef54: Pulling fs layer
ce82c0475275: Pulling fs layer
43002f3c1a6c: Pulling fs layer
4f4fb700ef54: Waiting
686a99242ad3: Waiting
066e1f460a6d: Pulling fs layer
bc0ffe24b2bc: Waiting
ce82c0475275: Waiting
2999aea03e7f: Pulling fs layer
43002f3c1a6c: Waiting
2650b0f08a84: Pulling fs layer
066e1f460a6d: Waiting
3f41c1c5b900: Pulling fs layer
dba0fde36498: Waiting
2650b0f08a84: Waiting
afe7bb8e07bf: Pulling fs layer
07ef81948195: Pulling fs layer
2999aea03e7f: Waiting
afe7bb8e07bf: Waiting
6d4b8e6b3966: Pulling fs layer
07ef81948195: Waiting
6d4b8e6b3966: Waiting
cc0d5765e2ac: Verifying Checksum
cc0d5765e2ac: Download complete
bbf6c0d5a635: Verifying Checksum
bbf6c0d5a635: Download complete
09db6f815738: Verifying Checksum
09db6f815738: Download complete
886a90d8482d: Verifying Checksum
886a90d8482d: Download complete
f3ed34a8f348: Verifying Checksum
f3ed34a8f348: Download complete
62a1a0967c60: Download complete
b94f174efd64: Download complete
09db6f815738: Pull complete
bbf6c0d5a635: Pull complete
cc0d5765e2ac: Pull complete
f3ed34a8f348: Pull complete
886a90d8482d: Pull complete
62a1a0967c60: Pull complete
b94f174efd64: Pull complete
00b62200211a: Verifying Checksum
00b62200211a: Download complete
71366ae47cf1: Verifying Checksum
71366ae47cf1: Download complete
ebc3d0abc32d: Verifying Checksum
ebc3d0abc32d: Download complete
2cb2b2033c34: Verifying Checksum
2cb2b2033c34: Download complete
d9202e6a9c4c: Verifying Checksum
d9202e6a9c4c: Download complete
d29d881a5264: Verifying Checksum
d29d881a5264: Download complete
c835a029766a: Download complete
257f05a2d884: Verifying Checksum
257f05a2d884: Download complete
f0399f54a3c5: Verifying Checksum
f0399f54a3c5: Download complete
cc2dcfc5fa29: Download complete
ebf38c798b19: Download complete
0859b3b3be1f: Download complete
393b90a8df47: Download complete
107d3ecef608: Verifying Checksum
107d3ecef608: Download complete
48c18c59c65c: Download complete
ac4fa6c27f81: Verifying Checksum
ac4fa6c27f81: Download complete
24ef8b27b1a7: Download complete
f07aaf596b5f: Download complete
a23100ee9050: Verifying Checksum
a23100ee9050: Download complete
3a99c9ac11c0: Verifying Checksum
3a99c9ac11c0: Download complete
2baeb5b5a333: Download complete
fccec2131deb: Download complete
8190a249acbf: Verifying Checksum
8190a249acbf: Download complete
5e8e43a05ae3: Verifying Checksum
5e8e43a05ae3: Download complete
4b495d28ad66: Download complete
18f868fcf1ae: Verifying Checksum
18f868fcf1ae: Download complete
dba0fde36498: Verifying Checksum
dba0fde36498: Download complete
00b62200211a: Pull complete
bc0ffe24b2bc: Verifying Checksum
bc0ffe24b2bc: Download complete
686a99242ad3: Verifying Checksum
686a99242ad3: Download complete
4f4fb700ef54: Verifying Checksum
4f4fb700ef54: Download complete
ce82c0475275: Verifying Checksum
ce82c0475275: Download complete
43002f3c1a6c: Verifying Checksum
43002f3c1a6c: Download complete
066e1f460a6d: Download complete
ebc3d0abc32d: Pull complete
2999aea03e7f: Verifying Checksum
2999aea03e7f: Download complete
2650b0f08a84: Verifying Checksum
2650b0f08a84: Download complete
3c704c19862b: Verifying Checksum
3c704c19862b: Download complete
afe7bb8e07bf: Verifying Checksum
afe7bb8e07bf: Download complete
3f41c1c5b900: Verifying Checksum
3f41c1c5b900: Download complete
6d4b8e6b3966: Verifying Checksum
6d4b8e6b3966: Download complete
07ef81948195: Verifying Checksum
07ef81948195: Download complete
8190a249acbf: Pull complete
71366ae47cf1: Pull complete
2cb2b2033c34: Pull complete
d9202e6a9c4c: Pull complete
75e29d109d54: Verifying Checksum
75e29d109d54: Download complete
d29d881a5264: Pull complete
0859b3b3be1f: Pull complete
c835a029766a: Pull complete
257f05a2d884: Pull complete
f0399f54a3c5: Pull complete
cc2dcfc5fa29: Pull complete
ebf38c798b19: Pull complete
393b90a8df47: Pull complete
107d3ecef608: Pull complete
48c18c59c65c: Pull complete
ac4fa6c27f81: Pull complete
24ef8b27b1a7: Pull complete
f07aaf596b5f: Pull complete
a23100ee9050: Pull complete
3a99c9ac11c0: Pull complete
2baeb5b5a333: Pull complete
fccec2131deb: Pull complete
4b495d28ad66: Pull complete
18f868fcf1ae: Pull complete
5e8e43a05ae3: Pull complete
75e29d109d54: Pull complete
3c704c19862b: Pull complete
dba0fde36498: Pull complete
bc0ffe24b2bc: Pull complete
686a99242ad3: Pull complete
4f4fb700ef54: Pull complete
ce82c0475275: Pull complete
43002f3c1a6c: Pull complete
066e1f460a6d: Pull complete
2999aea03e7f: Pull complete
2650b0f08a84: Pull complete
3f41c1c5b900: Pull complete
afe7bb8e07bf: Pull complete
07ef81948195: Pull complete
6d4b8e6b3966: Pull complete
Digest: sha256:c7a9396fe87348fa73632d8302b4eb2c1ee85d637703b9337e249a0c2f766d06
Status: Downloaded newer image for pennlinc/xcp_d:latest
docker.io/pennlinc/xcp_d:latest
Atlas 'MIDB' requires CIFTI processing. Skipping atlas.
Atlas 'MyersLabonte' requires CIFTI processing. Skipping atlas.
Custom confounds were provided, but --nuisance-regressors was set to none. Overriding the 'none' value and setting to 'custom'.
240916-20:29:56,61 nipype.workflow IMPORTANT:
	 Running XCP-D version 0.9.1
240916-20:30:29,692 nipype.workflow IMPORTANT:
	 Building XCP-D's workflow:
           * Preprocessing derivatives path: /fmriprep.
           * Participant list: ['50001', '50032', '50071', '50081', '50091', '50092', '50111', '50121', '50131', '50171', '50181', '50191', '50211', '50221', '50231', '50241', '50251', '50291', '50301', '50321', '50322', '50331', '50341', '50342', '50351', '50371', '50381', '50391', '50401', '50421', '50422', '50431', '50441', '50451', '50461', '50481', '50491', '50501', '50502', '50511', '50512', '50521', '50591', '50592', '50601', '50602', '50611', '50621', '50631', '50701', '50711', '50731', '50751', '50771', '50801', '50802', '50831', '50832', '50851', '50861', '50871', '50911', '50921', '50941', '50942', '50961', '50962', '50971', '50981', '50991', '50992', '51001', '51011', '51021', '51041', '51051', '51081', '51101', '51121', '51131', '51132', '51141', '51161', '51171', '51201', '51211', '51221', '51231', '51261', '51271', '51281', '51301', '51331', '51341', '51361', '51382', '5143', '51441', '51461', '51481', '51491', '51501', '51511', '51571', '51601', '51631', '51651', '51652', '51661', '51662', '51701', '51702', '51741', '51751', '51771', '51781', '51841', '51851', '51881', '51891', '51921', '51931', '52002', '52011', '52031', '52071', '52121', '52141', '52151', '52161', '52181', '52191', '52192', '52281', '52291', '52292', '52301', '52321', '52351', '52371', '52431', '52511', '52512', '52521'].
           * Run identifier: 20240916-202721_9381d0ae-b2f6-4de1-b94f-c401b8eca73f.
240916-20:30:34,786 nipype.utils IMPORTANT:
	 Collected data:
anat_brainmask: /fmriprep/sub-50001/anat/sub-50001_space-MNI152NLin6Asym_desc-brain_mask.nii.gz
anat_to_template_xfm: /fmriprep/sub-50001/anat/sub-50001_from-T1w_to-MNI152NLin6Asym_mode-image_xfm.h5
bold:
- /fmriprep/sub-50001/ses-0/func/sub-50001_ses-0_task-movieA_space-MNI152NLin6Asym_desc-preproc_bold.nii.gz
- /fmriprep/sub-50001/ses-1/func/sub-50001_ses-1_task-movieB_space-MNI152NLin6Asym_desc-preproc_bold.nii.gz
- /fmriprep/sub-50001/ses-2/func/sub-50001_ses-2_task-movieC_space-MNI152NLin6Asym_desc-preproc_bold.nii.gz
t1w: /fmriprep/sub-50001/anat/sub-50001_desc-preproc_T1w.nii.gz
t2w: null
template_to_anat_xfm: /fmriprep/sub-50001/anat/sub-50001_from-MNI152NLin6Asym_to-T1w_mode-image_xfm.h5

240916-20:30:35,193 nipype.utils INFO:
	 No standard-space surfaces found.
240916-20:30:35,900 nipype.utils IMPORTANT:
	 Collected mesh files:
lh_pial_surf: /fmriprep/sub-50001/anat/sub-50001_hemi-L_pial.surf.gii
lh_subject_sphere: /fmriprep/sub-50001/anat/sub-50001_hemi-L_desc-reg_sphere.surf.gii
lh_wm_surf: /fmriprep/sub-50001/anat/sub-50001_hemi-L_white.surf.gii
rh_pial_surf: /fmriprep/sub-50001/anat/sub-50001_hemi-R_pial.surf.gii
rh_subject_sphere: /fmriprep/sub-50001/anat/sub-50001_hemi-R_desc-reg_sphere.surf.gii
rh_wm_surf: /fmriprep/sub-50001/anat/sub-50001_hemi-R_white.surf.gii

240916-20:30:35,900 nipype.utils IMPORTANT:
	 Assuming segmentation was performed with FreeSurfer.
240916-20:30:36,516 nipype.utils IMPORTANT:
	 Collected morphometry files:
cortical_thickness: null
cortical_thickness_corr: null
myelin: null
myelin_smoothed: null
sulcal_curv: null
sulcal_depth: null

Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.10/site-packages/xcp_d/cli/workflow.py", line 99, in build_workflow
    retval["workflow"] = init_xcpd_wf()
  File "/usr/local/miniconda/lib/python3.10/site-packages/xcp_d/workflows/base.py", line 78, in init_xcpd_wf
    single_subject_wf = init_single_subject_wf(subject_id)
  File "/usr/local/miniconda/lib/python3.10/site-packages/xcp_d/workflows/base.py", line 375, in init_single_subject_wf
    warp_brainmask_results = warp_brainmask.run(
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 397, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.10/site-packages/xcp_d/interfaces/ants.py", line 199, in _run_interface
    runtime = super(ApplyTransforms, self)._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.10/site-packages/niworkflows/interfaces/fixes.py", line 64, in _run_interface
    runtime = super()._run_interface(
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 752, in _run_interface
    raise IOError(
OSError: No command "antsApplyTransforms" found on host compute1-exec-75.ris.wustl.edu. Please check that the corresponding package is installed.
2

What happens if this is set to false? Usually you do not want your entire environment to go into the docker container, and only mount important paths, such that you ensure the software is using the versions bundled with the container.

Best,
Steven

3

Hey Steven! Unfortunately when I do set it to False it crashes almost immediately without logs. I’ve reached out to our computing support group but they’ve not been able to fix the issue yet and I think they’re at the point they think it’s just a limitation with system unfortunately.

4

Oh wait think I figured another way to get the environmental variables! I ended up pulling the xcp_d image locally on my own computer and then running the docker image but switching out the entry point with a call to the linux printenv to get the environmental variables from docker image using this commands…

docker run --entrypoint 'printenv' -p 4000:80 --name XCP_D pennlinc/xcp_d:latest

In case anyone else needed them in the future here’s the environmental variables I pulled from the latest version of XCP_D…

PATH=$PATH:/usr/lib/ants:/opt/freesurfer/bin:/opt/freesurfer/tktools:/opt/freesurfer/mni/bin:/opt/fsl/lib:/opt/fsl/bin:/opt/c3d/bin:/usr/lib/afni/bin:/usr/local/miniconda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export HOSTNAME=104d06f00d8b
export OS=Linux
export FIX_VERTEX_AREA=
export CPATH=/usr/local/miniconda/include:
export LANG=C.UTF-8
export LC_ALL=C.UTF-8
export PYTHONNOUSERSITE=1
export AFNI_MODELPATH=/usr/lib/afni/models
export AFNI_IMSAVE_WARNINGS=NO
export AFNI_TTATLAS_DATASET=/usr/share/afni/atlases
export AFNI_PLUGINPATH=/usr/lib/afni/plugins
export C3DPATH=/opt/c3d/bin
export FSLDIR=/opt/fsl
export FSLOUTPUTTYPE=NIFTI_GZ
export FSLMULTIFILEQUIT=TRUE
export FSLLOCKDIR=
export FSLMACHINELIST=
export FSLREMOTECALL=
export FSLGECUDAQ=cuda.q
export LD_LIBRARY_PATH=/opt/fsl/lib:
export FSL_DEPS='libquadmath0;libnewimage.so;libmiscmaths.so'
export FREESURFER_HOME=/opt/freesurfer
export FSF_OUTPUT_FORMAT=nii.gz
export FUNCTIONALS_DIR=/opt/freesurfer/sessions
export LOCAL_DIR=/opt/freesurfer/local
export MINC_BIN_DIR=/opt/freesurfer/mni/bin
export MINC_LIB_DIR=/opt/freesurfer/mni/lib
export MNI_DIR=/opt/freesurfer/mni
export MNI_DATAPATH=/opt/freesurfer/mni/data
export MNI_PERL5LIB=/opt/freesurfer/mni/lib/perl5/5.8.5
export PERL5LIB=/opt/freesurfer/mni/lib/perl5/5.8.5
export SUBJECTS_DIR=/opt/freesurfer/subjects
export ANTSPATH=/usr/lib/ants
export MKL_NUM_THREADS=1
export OMP_NUM_THREADS=1
export HOME=/home/xcp_d