Issue with XCP_D on HPC and TemplateFlow

jsein · October 3, 2022, 5:40pm

Hi,

I am using: xcp_d version 0.1.3+0.g9fec4d8.dirty with singularity on a HPC.

Here is the command I used:

singularity run -B /scratch/jsein/BIDS:/work,$HOME/.templateflow:$HOME/.cache/templateflow  \
   /scratch/jsein/my_images/xcp_d-0.1.3.simg \
   /work/$study/derivatives/fmriprep /work/$study/derivatives/xcpd \
   participant --participant-label $sub \
   -w /work/temp_data_${study}\
   --mem_gb 50 --omp-nthreads 10 --nthreads 12 --smoothing 4  \
   --nuissance-regressors acompcor --despike --lower-bpf 0.01 --upper-bpf 0.08

I am interested in testing XCP_D on our data but I am stuck right at the beginning with this error which I believe is linked to the fact that the singularity container does not have access to the internet on the HPC I am using it:

Downloading https://templateflow.s3.amazonaws.com/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz

  0%|          | 0.00/28.0 [00:00<?, ?B/s]
29.0B [00:00, 324B/s]                     
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/cli/run.py", line 617, in build_workflow
    retval['workflow'] = init_xcpd_wf(
  File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/workflow/base.py", line 148, in init_xcpd_wf
    single_subj_wf = init_subject_wf(
  File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/workflow/base.py", line 395, in init_subject_wf
    bold_postproc_wf = init_boldpostprocess_wf(
  File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/workflow/bold.py", line 341, in init_boldpostprocess_wf
    bold2MNI_trans, bold2T1w_trans = get_transformfilex(
  File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/utils/utils.py", line 87, in get_transformfilex
    t1w_to_mni = glob.glob(mnisf + 'from-T1w_to-' + template +
UnboundLocalError: local variable 'template' referenced before assignment

I know how to deal with it with FMRIPREP:
https://fmriprep.org/en/stable/faq.html#how-do-you-use-templateflow-in-the-absence-of-access-to-the-internet

But for the moment I don’t know how to deal with this issue with XCP_D. Can anyone help me?

Thank you!

Julien

Steven · October 3, 2022, 7:02pm

Hi Julien,

What output spaces did you use in fMRIPrep? Would you mind sharing your full command?

Best,
Steven

jsein · October 3, 2022, 7:20pm

Hi Steven,

Thank you for passing by!

Sure, here is the fmriprep command I used:

singularity run -B /scratch/jsein/BIDS:/work,$HOME/.templateflow:/opt/templateflow --cleanenv /scratch/jsein/my_images/fmriprep-21.0.2.simg \
		 --fs-license-file /work/freesurfer/license.txt /work/$study /work/$study/derivatives/fmriprep_nosbref  \
		 participant --participant-label $sub \
		 -w /work/temp_data_${study}\
		 --mem-mb 50000 --omp-nthreads 10 --nthreads 12  \
		 --fd-spike-threshold 0.5 --dvars-spike-threshold 2.0 --bold2t1w-dof 9 --cifti-output 91k   \
		 --output-spaces MNI152NLin2009cAsym MNI152NLin6Asym T1w --ignore slicetiming  --fs-subjects-dir /work/$study/derivatives/fmriprep/sourcedata/freesurfer

Steven · October 3, 2022, 7:26pm

Can you print the contents of a subject’s anatomical fmriprep outputs?

jsein · October 3, 2022, 7:29pm

Here it is:

anat/
├── sub-01_desc-aparcaseg_dseg.nii.gz
├── sub-01_desc-aseg_dseg.nii.gz
├── sub-01_desc-brain_mask.json
├── sub-01_desc-brain_mask.nii.gz
├── sub-01_desc-preproc_T1w.json
├── sub-01_desc-preproc_T1w.nii.gz
├── sub-01_dseg.nii.gz
├── sub-01_from-MNI152NLin2009cAsym_to-T1w_mode-image_xfm.h5
├── sub-01_from-MNI152NLin6Asym_to-T1w_mode-image_xfm.h5
├── sub-01_from-T1w_to-MNI152NLin2009cAsym_mode-image_xfm.h5
├── sub-01_from-T1w_to-MNI152NLin6Asym_mode-image_xfm.h5
├── sub-01_from-T1w_to-fsnative_mode-image_xfm.txt
├── sub-01_from-fsnative_to-T1w_mode-image_xfm.txt
├── sub-01_hemi-L_inflated.surf.gii
├── sub-01_hemi-L_midthickness.surf.gii
├── sub-01_hemi-L_pial.surf.gii
├── sub-01_hemi-L_smoothwm.surf.gii
├── sub-01_hemi-R_inflated.surf.gii
├── sub-01_hemi-R_midthickness.surf.gii
├── sub-01_hemi-R_pial.surf.gii
├── sub-01_hemi-R_smoothwm.surf.gii
├── sub-01_label-CSF_probseg.nii.gz
├── sub-01_label-GM_probseg.nii.gz
├── sub-01_label-WM_probseg.nii.gz
├── sub-01_space-MNI152NLin2009cAsym_desc-brain_mask.json
├── sub-01_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz
├── sub-01_space-MNI152NLin2009cAsym_desc-preproc_T1w.json
├── sub-01_space-MNI152NLin2009cAsym_desc-preproc_T1w.nii.gz
├── sub-01_space-MNI152NLin2009cAsym_dseg.nii.gz
├── sub-01_space-MNI152NLin2009cAsym_label-CSF_probseg.nii.gz
├── sub-01_space-MNI152NLin2009cAsym_label-GM_probseg.nii.gz
├── sub-01_space-MNI152NLin2009cAsym_label-WM_probseg.nii.gz
├── sub-01_space-MNI152NLin6Asym_desc-brain_mask.json
├── sub-01_space-MNI152NLin6Asym_desc-brain_mask.nii.gz
├── sub-01_space-MNI152NLin6Asym_desc-preproc_T1w.json
├── sub-01_space-MNI152NLin6Asym_desc-preproc_T1w.nii.gz
├── sub-01_space-MNI152NLin6Asym_dseg.nii.gz
├── sub-01_space-MNI152NLin6Asym_label-CSF_probseg.nii.gz
├── sub-01_space-MNI152NLin6Asym_label-GM_probseg.nii.gz
└── sub-01_space-MNI152NLin6Asym_label-WM_probseg.nii.gz

Steven · October 3, 2022, 7:29pm

I do not think this issue is related to templateflow, but rather to the naming convention of the .h5 transformation files. That is, looking at the code here (xcp_d/utils.py at 8aa79b9ba4612fbf38231886ee9ac71215ec88e3 · PennLINC/xcp_d · GitHub), there are different conditions that they use to try to figure out what template is being used based on the transformation name, but somehow, it is not satisfying any of them, leading to the undefined variable error.

Steven · October 3, 2022, 7:32pm

Hmm, the h5 files look fine, I guess it would help to know what h5 file is being checked, which would mean introducing print statements in the utils command linked above. You could try building the singularity container as a sandbox and making the changes yourself, but I wish the error would be more obvious

jsein · October 3, 2022, 7:33pm

Thank you for this feedback.

I had an issue for another study where only MNI152NLin2009cAsym was asked as an output for fmriprep so I tried for another when where the MNI152NLin2009cAsym MNI152NLin6Asym T1w were called for the output.

I am puzzled still by this output in the error log:

Downloading https://templateflow.s3.amazonaws.com/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz

  0%|          | 0.00/28.0 [00:00<?, ?B/s]
29.0B [00:00, 324B/s]

which seems to indicate that XCP_D tried to download a template from TemplateFlow.

Steven · October 3, 2022, 7:35pm

I think this case warrants a github issues post, as it is may be due to the code not filtering the transform files properly as opposed to any user error. Hope you can get to the bottom of this!

jsein · October 3, 2022, 7:40pm

Thank you again!
ok I will dig deeper into this, I will let you know in this thread if I find a solution.

I agree, I will post an issue on GitHub. As it was my first trial with the XCP_D toolbox for which I read good recommendations, I was wondering if I was missing something obvious in my command line.

jsein · October 3, 2022, 7:47pm

For information, here is the output for a subject for which only MNI152NLin2009cAsym was asked for as output-spaces in fmriprep (v21.0.2):

Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/cli/run.py", line 617, in build_workflow
    retval['workflow'] = init_xcpd_wf(
  File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/workflow/base.py", line 148, in init_xcpd_wf
    single_subj_wf = init_subject_wf(
  File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/workflow/base.py", line 329, in init_subject_wf
    anatomical_wf = init_anatomical_wf(
  File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/workflow/anatomical.py", line 364, in init_anatomical_wf
    h5_file = fnmatch.filter(
IndexError: list index out of range

Steven · October 3, 2022, 8:37pm

At the very least this seems to add evidence that this is a H5 finding problem as opposed to a templateflow issue.

jsein · October 3, 2022, 9:24pm

I created a Github issue on this topic:

github.com/PennLINC/xcp_d

error in finding template space of the input

opened 09:22PM - 03 Oct 22 UTC

julfou81

bug

## Summary Hi, thank you for providing and supporting this nice tool. I am an h…eavy user of FMRIPREP and new user of XCP_D. My first try with XCP_D didn't go through, it crashes almost immediately with two kind of errors, depending which output spaces were asked to FMRIPREP: - if `MNI152NLin2009cAsym` only was requested in fmriprep, the error is the following: ``` Process Process-2: Traceback (most recent call last): File "/usr/local/miniconda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/local/miniconda/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/cli/run.py", line 617, in build_workflow retval['workflow'] = init_xcpd_wf( File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/workflow/base.py", line 148, in init_xcpd_wf single_subj_wf = init_subject_wf( File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/workflow/base.py", line 329, in init_subject_wf anatomical_wf = init_anatomical_wf( File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/workflow/anatomical.py", line 364, in init_anatomical_wf h5_file = fnmatch.filter( IndexError: list index out of range ``` - if ```MNI152NLin2009cAsym MNI152NLin6Asym T1w```were requested to FMRIPREP, the error in XCP_D is the following: ``` Downloading https://templateflow.s3.amazonaws.com/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz 0%| | 0.00/28.0 [00:00<?, ?B/s] 29.0B [00:00, 324B/s] Process Process-2: Traceback (most recent call last): File "/usr/local/miniconda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/local/miniconda/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/cli/run.py", line 617, in build_workflow retval['workflow'] = init_xcpd_wf( File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/workflow/base.py", line 148, in init_xcpd_wf single_subj_wf = init_subject_wf( File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/workflow/base.py", line 395, in init_subject_wf bold_postproc_wf = init_boldpostprocess_wf( File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/workflow/bold.py", line 341, in init_boldpostprocess_wf bold2MNI_trans, bold2T1w_trans = get_transformfilex( File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/utils/utils.py", line 87, in get_transformfilex t1w_to_mni = glob.glob(mnisf + 'from-T1w_to-' + template + UnboundLocalError: local variable 'template' referenced before assignment ``` ## Additional details - xcp_d version 0.1.3+0.g9fec4d8.dirty with Singularity on a HPC. - fmriprep version : 21.0.2 with Singularity on a HPC. ### What actually happened? Immediate crash ## Reproducing the bug Here is my XCP_D command: ``` singularity run -B /scratch/jsein/BIDS:/work,$HOME/.templateflow:$HOME/.cache/templateflow \ /scratch/jsein/my_images/xcp_d-0.1.3.simg \ /work/$study/derivatives/fmriprep /work/$study/derivatives/xcpd \ participant --participant-label $sub \ -w /work/temp_data_${study}\ --mem_gb 50 --omp-nthreads 10 --nthreads 12 --smoothing 4 \ --nuissance-regressors acompcor --despike --lower-bpf 0.01 --upper-bpf 0.08 ``` And a typical fmriprep command: ``` singularity run -B /scratch/jsein/BIDS:/work,$HOME/.templateflow:/opt/templateflow --cleanenv /scratch/jsein/my_images/fmriprep-21.0.2.simg \ --fs-license-file /work/freesurfer/license.txt /work/$study /work/$study/derivatives/fmriprep_nosbref \ participant --participant-label $sub \ -w /work/temp_data_${study}\ --mem-mb 50000 --omp-nthreads 10 --nthreads 12 \ --fd-spike-threshold 0.5 --dvars-spike-threshold 2.0 --bold2t1w-dof 9 --cifti-output 91k \ --output-spaces MNI152NLin2009cAsym MNI152NLin6Asym T1w --ignore slicetiming --fs-subjects-dir /work/$study/derivatives/fmriprep/sourcedata/freesurfer ``` Thank you for your support!