Xcp-d cannot be run with hcp preprocessed data

Yuan_Cao · June 25, 2024, 4:06pm

Summary of what happened:

My data was preprocessed using hcppipeline and of course when running xcp-d he reported the following error：

Command used (and if a helper script was used, a link to the helper script or the command generated):

sudo docker run -it --rm -v /media/ubuntu/NCU/deng/Rawdata:/data -v /media/ubuntu/NCU/out:/out -v /opt/freesurfer/license.txt:/opt/freesurfer/license.txt pennlinc/xcp_d:latest /data /out --despike --head_radius 40 -w /media/ubuntu/NCU/deng/Rawdata --smoothing 6 --fs-license-file /opt/freesurfer/license.txt participant --participant-label 559 --input-type hcp

Version: xcp_d:latest

Environment (Docker, Singularity / Apptainer, custom installation): Docker

Installed by docker

Data formatted according to a validatable standard? Please provide the output of the validator:

bids

Relevant log outputs (up to 20 lines):

With input_type hcp, cifti processing (--cifti) will be enabled automatically.dia/ubuntu/NCU/out:/out -v /opt/freesurfer/license.txt:/opt/freesurfer/license.txt pennlinc/xcp_d:latest /data /out --despike With input_type hcp, surface normalization (--warp-surfaces-native2std) will be enabled automatically.
240625-15:57:01,80 nipype.utils WARNING:
	 convert_hcp2bids is an experimental function.
240625-15:57:01,80 nipype.utils INFO:
	 Converting 559
240625-15:57:01,81 nipype.utils WARNING:
	 File DNE: /data/559/MNINonLinear/T1w.nii.gz
240625-15:57:01,81 nipype.utils WARNING:
	 File DNE: /data/559/MNINonLinear/brainmask_fs.nii.gz
240625-15:57:01,81 nipype.utils WARNING:
	 File DNE: /data/559/MNINonLinear/ribbon.nii.gz
240625-15:57:01,81 nipype.utils WARNING:
	 File DNE: /data/559/MNINonLinear/aparc+aseg.nii.gz
240625-15:57:01,81 nipype.utils WARNING:
	 File DNE: /data/559/MNINonLinear/fsaverage_LR32k/559.L.pial.32k_fs_LR.surf.gii
240625-15:57:01,81 nipype.utils WARNING:
	 File DNE: /data/559/MNINonLinear/fsaverage_LR32k/559.R.pial.32k_fs_LR.surf.gii
240625-15:57:01,81 nipype.utils WARNING:
	 File DNE: /data/559/MNINonLinear/fsaverage_LR32k/559.L.white.32k_fs_LR.surf.gii
240625-15:57:01,81 nipype.utils WARNING:
	 File DNE: /data/559/MNINonLinear/fsaverage_LR32k/559.R.white.32k_fs_LR.surf.gii
240625-15:57:01,81 nipype.utils WARNING:
	 File(s) DNE:
	/data/559/MNINonLinear/fsaverage_LR32k/559.L.thickness.32k_fs_LR.shape.gii
	/data/559/MNINonLinear/fsaverage_LR32k/559.R.thickness.32k_fs_LR.shape.gii
240625-15:57:01,82 nipype.utils WARNING:
	 File(s) DNE:
	/data/559/MNINonLinear/fsaverage_LR32k/559.L.corrThickness.32k_fs_LR.shape.gii
	/data/559/MNINonLinear/fsaverage_LR32k/559.R.corrThickness.32k_fs_LR.shape.gii
240625-15:57:01,82 nipype.utils WARNING:
	 File(s) DNE:
	/data/559/MNINonLinear/fsaverage_LR32k/559.L.curvature.32k_fs_LR.shape.gii
	/data/559/MNINonLinear/fsaverage_LR32k/559.R.curvature.32k_fs_LR.shape.gii
240625-15:57:01,82 nipype.utils WARNING:
	 File(s) DNE:
	/data/559/MNINonLinear/fsaverage_LR32k/559.L.sulc.32k_fs_LR.shape.gii
	/data/559/MNINonLinear/fsaverage_LR32k/559.R.sulc.32k_fs_LR.shape.gii
240625-15:57:01,82 nipype.utils WARNING:
	 File(s) DNE:
	/data/559/MNINonLinear/fsaverage_LR32k/559.L.MyelinMap.32k_fs_LR.func.gii
	/data/559/MNINonLinear/fsaverage_LR32k/559.R.MyelinMap.32k_fs_LR.func.gii
240625-15:57:01,82 nipype.utils WARNING:
	 File(s) DNE:
	/data/559/MNINonLinear/fsaverage_LR32k/559.L.SmoothedMyelinMap.32k_fs_LR.func.gii
	/data/559/MNINonLinear/fsaverage_LR32k/559.R.SmoothedMyelinMap.32k_fs_LR.func.gii
240625-15:57:01,82 nipype.utils INFO:
	 Finished collecting anatomical files
240625-15:57:01,82 nipype.utils INFO:
	 Finished collecting functional files
240625-15:57:01,82 nipype.utils INFO:
	 Copying files
240625-15:57:01,82 nipype.utils WARNING:
	 File used for more than one output: /usr/local/miniconda/lib/python3.10/site-packages/xcp_d/data/transform/itkIdentityTransform.txt
240625-15:57:01,83 nipype.utils INFO:
	 Finished copying files
240625-15:57:01,88 nipype.utils INFO:
	 Conversion completed
240625-15:57:09,515 nipype.workflow IMPORTANT:
	 Running XCP-D version 0.7.4
240625-15:57:09,521 nipype.workflow IMPORTANT:
	 Building XCP-D's workflow:
           * Preprocessing derivatives path: /media/ubuntu/NCU/deng/Rawdata/dset_bids/derivatives/hcp.
           * Participant list: ['559'].
           * Run identifier: 20240625-155657_74e129a7-4a08-46da-9739-656571ad91f5.
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.10/site-packages/xcp_d/cli/workflow.py", line 98, in build_workflow
    retval["workflow"] = init_xcpd_wf()
  File "/usr/local/miniconda/lib/python3.10/site-packages/xcp_d/workflows/base.py", line 80, in init_xcpd_wf
    single_subject_wf = init_single_subject_wf(subject_id)
  File "/usr/local/miniconda/lib/python3.10/site-packages/xcp_d/workflows/base.py", line 126, in init_single_subject_wf
    subj_data = collect_data(
  File "/usr/local/miniconda/lib/python3.10/site-packages/xcp_d/utils/bids.py", line 255, in collect_data
    raise FileNotFoundError(
FileNotFoundError: No BOLD data found in allowed spaces (fsLR).

Query: {'datatype': 'func', 'suffix': 'bold', 'desc': ['preproc', None], 'extension': '.dtseries.nii', 'space': 'fsLR'}

Found files

Screenshots / relevant information:

tsalo · June 25, 2024, 4:34pm

Since the HCP derivatives folder has sub-559 instead of 559, please try running XCP-D with --participant-label sub-559. No guarantee that’ll work (XCP-D might automatically strip the sub- out of the participant label).

Yuan_Cao · June 25, 2024, 4:52pm

Thanks for the quick reply. It still doesn’t work, and you are right the xcp-d can detect sub- by itself.
If I change to sub-559, it will be showed like this: xcp_d: error: One or more participant labels were not found in the BIDS directory: sub-559

Yuan_Cao · June 25, 2024, 5:04pm

Hi Taylor,
I may find the problem. The hcppipeline will generate the fsLR space data, but name it as task-rest_dir-RL_LR_Atlas.dtseries.nii. However, the Xcp-d seems to find the file name with space_fsLR (but not Atlas) as input. Do you know how to solve it?

Best,
Yuan

tsalo · June 26, 2024, 2:06pm

That’s a really tough one. I don’t have an easy solution- I can try to modify XCP-D’s ingression code to support this organization, but there will be a delay before that is usable.

One (albeit suboptimal) solution might be to reorganize the data yourself to match the organization XCP-D expects. You can see the full list of files XCP-D expects for HCP derivatives here. There might be an easy way to mock up that structure with symlinks from your actual HCP derivatives.

What version of HCPpipelines did you run? Also, can you share the command you used to run it?

Yuan_Cao · June 26, 2024, 3:26pm

Hi Taylor,
Thank you so much for your quick reply.
I tried to mock up a link and re-named it to what xcp-d can detect, however, it still showed no bold file could be found.
I just started the debug mode of xcp-d and tried to figure out which specific file the xcp-d needs from hcp output.
In hcpya.py, from the line 274:

    bold_cifti_orig = os.path.join(
        task_dir_orig,
        f"{base_task_name}_Atlas_MSMAll.dtseries.nii",
    )
    bold_cifti_fmriprep = os.path.join(
        func_dir_bids,
        f"{func_prefix}_space-fsLR_den-91k_bold.dtseries.nii",
    )
    copy_dictionary[bold_cifti_orig] = [bold_cifti_fmriprep]

It indicates the xcp-d need a file named {base_task_name}_Atlas_MSMAll.dtseries.nii file. I already generated this file last night, but when re-run the docker xcp-d, it still says No BOLD file.

The hcppipeline I used should be v4.2.0, because I started preprocessing my data in 2021. I can, of course, share my hcppipeline code, I just uploaded it into github:hcppreprocessing.

Thank you again.
Yuan