Missing "brain_mask_trans" file for XCP

Summary of what happened:

Trying to run XCP on a single scan preprocessed with fMRIprep 23.2.0. Running into an error about a missing “brain_mask_trans” file in the work directory. I have not seen this file name before and cannot find much about it online. I previously ran XCP 0.7.? on this dataset with no issues.

Command used (and if a helper script was used, a link to the helper script or the command generated):

+ export FREESURFER_HOME=/Users/ahalab/Documents
+ FREESURFER_HOME=/Users/ahalab/Documents
+ export HOME_DIR=/Volumes/Seagate/
+ HOME_DIR=/Volumes/Seagate/
+ for SES in '{1..1}'
+ for RUN in '{1..1}'
+ docker run --rm -it -v /Volumes/Seagate/BIDS/derivatives:/fmriprep:ro -v /Volumes/Seagate/work_xcp:/work:rw -v /Volumes/Seagate/BIDS/derivatives/xcp_d:/out:rw -v /Users/ahalab/Documents/license.txt:/out/fs_license:rw -v /Volumes/Seagate/BIDS/bids_filters/bids_filter_ses1_run1.json:/out/bids_filter.json:rw pennlinc/xcp_d:latest /fmriprep /out participant -w /work --fs-license-file /out/fs_license --mem-gb 15 --low-mem --bids-filter-file /out/bids_filter.json --participant_label AEON101 --mode abcd --smoothing 6 --combine-runs n --create-matrices all --motion-filter-type lp --band-stop-min 6 --file-format nifti --warp-surfaces-native2std n --min-time 0 --abcc-qc n

Version:

0.10.1

Environment (Docker, Singularity / Apptainer, custom installation):

Docker

Data formatted according to a validatable standard? Please provide the output of the validator:

BIDS valid

Relevant log outputs (up to 20 lines):

2024-12-11 14:21:42,110 [    INFO] Loading atlas: 4S756Parcels
241211-14:21:42,110 nipype.workflow INFO:
	 Loading atlas: 4S856Parcels
2024-12-11 14:21:42,110 [    INFO] Loading atlas: 4S856Parcels
241211-14:21:42,110 nipype.workflow INFO:
	 Loading atlas: 4S956Parcels
2024-12-11 14:21:42,110 [    INFO] Loading atlas: 4S956Parcels
241211-14:21:42,209 nipype.interface INFO:
	 stderr 2024-12-11T14:21:42.209365:Illegal instruction
2024-12-11 14:21:42,209 [    INFO] stderr 2024-12-11T14:21:42.209365:Illegal instruction
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.10/site-packages/nibabel/loadsave.py", line 101, in load
    stat_result = os.stat(filename)
FileNotFoundError: [Errno 2] No such file or directory: '/work/sub_AEON101_wf/sub-AEON101_ses-001_space-MNI152NLin2009cAsym_desc-brain_mask_trans.nii.gz'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.10/site-packages/xcp_d/cli/workflow.py", line 100, in build_workflow
    retval['workflow'] = init_xcpd_wf()
  File "/usr/local/miniconda/lib/python3.10/site-packages/xcp_d/workflows/base.py", line 81, in init_xcpd_wf
    single_subject_wf = init_single_subject_wf(subject_id)
  File "/usr/local/miniconda/lib/python3.10/site-packages/xcp_d/workflows/base.py", line 376, in init_single_subject_wf
    warp_brainmask_results = warp_brainmask.run(
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 397, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.10/site-packages/xcp_d/interfaces/ants.py", line 216, in _run_interface
    runtime = super()._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.10/site-packages/niworkflows/interfaces/fixes.py", line 68, in _run_interface
    _copyxform(
  File "/usr/local/miniconda/lib/python3.10/site-packages/niworkflows/utils/images.py", line 79, in _copyxform
    resampled = nb.load(out_image, mmap=False)
  File "/usr/local/miniconda/lib/python3.10/site-packages/nibabel/loadsave.py", line 103, in load
    raise FileNotFoundError(f"No such file or no access: '{filename}'")
FileNotFoundError: No such file or no access: '/work/sub_AEON101_wf/sub-AEON101_ses-001_space-MNI152NLin2009cAsym_desc-brain_mask_trans.nii.gz'

Screenshots / relevant information:

I’m running this on an external hard drive. I want to use abcd or hcbd mode so that I can retain the original TR indices of a complete time series and discard interpolated volumes manually. The censored output of linc mode won’t give me what I need for subsequent analysis. However, I tried running linc and hbcd mode and got the same error. I also copied a limited portion of my BIDS directory onto my internal HD and got the same error.


Hi @ejcorn,

Do you have MNI152NLin2009cAsym outputs from fMRIPrep? And does the error persist when using a fresh working directory? Is it for all subjects or just specific ones?

Best,
Steven

Yes - I just tried on another subject, another session from the same subject, after deleting the work directory, and got the same error. Here are the contents of my fMRIprep anatomical scan output:

sub-AEON101_ses-001_desc-brain_mask.json
sub-AEON101_ses-001_desc-brain_mask.nii.gz
sub-AEON101_ses-001_desc-preproc_T1w.json
sub-AEON101_ses-001_desc-preproc_T1w.nii.gz
sub-AEON101_ses-001_desc-ribbon_mask.json
sub-AEON101_ses-001_desc-ribbon_mask.nii.gz
sub-AEON101_ses-001_dseg.nii.gz
sub-AEON101_ses-001_from-MNI152NLin2009cAsym_to-T1w_mode-image_xfm.h5
sub-AEON101_ses-001_from-T1w_to-MNI152NLin2009cAsym_mode-image_xfm.h5
sub-AEON101_ses-001_from-T1w_to-fsnative_mode-image_xfm.txt
sub-AEON101_ses-001_from-fsnative_to-T1w_mode-image_xfm.txt
sub-AEON101_ses-001_hemi-L_desc-reg_sphere.surf.gii
sub-AEON101_ses-001_hemi-L_midthickness.surf.gii
sub-AEON101_ses-001_hemi-L_pial.surf.gii
sub-AEON101_ses-001_hemi-L_space-fsLR_desc-msmsulc_sphere.surf.gii
sub-AEON101_ses-001_hemi-L_space-fsLR_desc-reg_sphere.surf.gii
sub-AEON101_ses-001_hemi-L_sphere.surf.gii
sub-AEON101_ses-001_hemi-L_sulc.shape.gii
sub-AEON101_ses-001_hemi-L_thickness.shape.gii
sub-AEON101_ses-001_hemi-L_white.surf.gii
sub-AEON101_ses-001_hemi-R_desc-reg_sphere.surf.gii
sub-AEON101_ses-001_hemi-R_midthickness.surf.gii
sub-AEON101_ses-001_hemi-R_pial.surf.gii
sub-AEON101_ses-001_hemi-R_space-fsLR_desc-msmsulc_sphere.surf.gii
sub-AEON101_ses-001_hemi-R_space-fsLR_desc-reg_sphere.surf.gii
sub-AEON101_ses-001_hemi-R_sphere.surf.gii
sub-AEON101_ses-001_hemi-R_sulc.shape.gii
sub-AEON101_ses-001_hemi-R_thickness.shape.gii
sub-AEON101_ses-001_hemi-R_white.surf.gii
sub-AEON101_ses-001_label-CSF_probseg.nii.gz
sub-AEON101_ses-001_label-GM_probseg.nii.gz
sub-AEON101_ses-001_label-WM_probseg.nii.gz
sub-AEON101_ses-001_space-MNI152NLin2009cAsym_desc-brain_mask.json
sub-AEON101_ses-001_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz
sub-AEON101_ses-001_space-MNI152NLin2009cAsym_desc-preproc_T1w.json
sub-AEON101_ses-001_space-MNI152NLin2009cAsym_desc-preproc_T1w.nii.gz
sub-AEON101_ses-001_space-MNI152NLin2009cAsym_dseg.nii.gz
sub-AEON101_ses-001_space-MNI152NLin2009cAsym_label-CSF_probseg.nii.gz
sub-AEON101_ses-001_space-MNI152NLin2009cAsym_label-GM_probseg.nii.gz
sub-AEON101_ses-001_space-MNI152NLin2009cAsym_label-WM_probseg.nii.gz

Hi @Steven - hope you had a good weekend. Just wanted to check in and see if you were able to come up with any solutions for this!

Thanks,
Eli

The step that appears to be failing is an antsApplyTransforms to warp the anatomical brain mask from MNI space to T1w/T2w space. The folder /work/sub_AEON101_wf might have a report.rst file with information on why that node failed. Can you check the folder to see if that file exists?

Hi @tsalo, thanks for your help! The pipeline doesn’t even seem to be progressing to that point. the sub_AEON101_wf folder is empty. I tried with another subject as well, and on my hard disk instead of the external hard drive. The xcp derivatives /out folder is also empty, and nothing is in the logs folder.

There should be a message in your log starting with “Collected data:”. Can you share that?

A log file isn’t being generated. There is a config.toml file but that’s it. The logs folder in derivatives/xcp_d/logs is empty.

This won’t go in the logs folder. I’m referring to the stdout file from you job submission. The thing you copied from in your initial post, with things like:

Ah sorry:

	 Collected data:
anat_brainmask: /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz
anat_to_template_xfm: /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_from-T1w_to-MNI152NLin2009cAsym_mode-image_xfm.h5
bold:
- /fmriprep/sub-AEON101/ses-001/func/sub-AEON101_ses-001_task-rest_run-01_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz
t1w: /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_desc-preproc_T1w.nii.gz
t2w: null
template_to_anat_xfm: /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_from-MNI152NLin2009cAsym_to-T1w_mode-image_xfm.h5

2024-12-20 20:48:03,360 [IMPORTANT] Collected data:
anat_brainmask: /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz
anat_to_template_xfm: /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_from-T1w_to-MNI152NLin2009cAsym_mode-image_xfm.h5
bold:
- /fmriprep/sub-AEON101/ses-001/func/sub-AEON101_ses-001_task-rest_run-01_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz
t1w: /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_desc-preproc_T1w.nii.gz
t2w: null
template_to_anat_xfm: /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_from-MNI152NLin2009cAsym_to-T1w_mode-image_xfm.h5

241220-20:48:03,441 nipype.utils INFO:
	 No standard-space surfaces found.
2024-12-20 20:48:03,441 [    INFO] No standard-space surfaces found.
241220-20:48:03,752 nipype.utils IMPORTANT:
	 Collected mesh files:
lh_pial_surf: /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_hemi-L_pial.surf.gii
lh_subject_sphere: /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_hemi-L_desc-reg_sphere.surf.gii
lh_wm_surf: /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_hemi-L_white.surf.gii
rh_pial_surf: /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_hemi-R_pial.surf.gii
rh_subject_sphere: /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_hemi-R_desc-reg_sphere.surf.gii
rh_wm_surf: /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_hemi-R_white.surf.gii

2024-12-20 20:48:03,752 [IMPORTANT] Collected mesh files:
lh_pial_surf: /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_hemi-L_pial.surf.gii
lh_subject_sphere: /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_hemi-L_desc-reg_sphere.surf.gii
lh_wm_surf: /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_hemi-L_white.surf.gii
rh_pial_surf: /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_hemi-R_pial.surf.gii
rh_subject_sphere: /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_hemi-R_desc-reg_sphere.surf.gii
rh_wm_surf: /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_hemi-R_white.surf.gii

241220-20:48:03,752 nipype.utils IMPORTANT:
	 Assuming segmentation was performed with FreeSurfer.
2024-12-20 20:48:03,752 [IMPORTANT] Assuming segmentation was performed with FreeSurfer.
241220-20:48:03,865 nipype.utils IMPORTANT:
	 Collected morphometry files:
cortical_thickness: null
cortical_thickness_corr: null
myelin: null
myelin_smoothed: null
sulcal_curv: null
sulcal_depth: null

2024-12-20 20:48:03,865 [IMPORTANT] Collected morphometry files:
cortical_thickness: null
cortical_thickness_corr: null
myelin: null
myelin_smoothed: null
sulcal_curv: null
sulcal_depth: null
111

Perfect, thank you! Are you comfortable with Docker? If so, you could try calling the ApplyTransforms command within the XCP-D Docker image. That should produce a more informative error message:

To enter the Docker image:

docker run --rm -it \
    -v /Volumes/Seagate/BIDS/derivatives:/fmriprep:ro \
    -v /Volumes/Seagate/work_xcp:/work:rw \
    --entrypoint=/bin/bash \
    pennlinc/xcp_d:latest

Then you can start a Python session and do the following:

from xcp_d.interfaces.ants import ApplyTransforms

anat_brainmask = "/fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz"
template_to_anat_xfm = "/fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_from-MNI152NLin2009cAsym_to-T1w_mode-image_xfm.h5"
anat = "/fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_desc-preproc_T1w.nii.gz"

warp_brainmask = ApplyTransforms(
    input_image=anat_brainmask,
    transforms=[template_to_anat_xfm],
    reference_image=anat,
    num_threads=2,
    interpolation='GenericLabel',
    input_image_type=3,
    dimension=3,
)
warp_brainmask_results = warp_brainmask.run()
# if that doesn't give you a useful error message, you could try
warp_brainmask.cmdline
# then call that from the shell within the Docker image

Thank you so much - have never entered the docker image like this before, very useful. Here’s the initial output, not that helpful, basically same error as before:

>>> warp_brainmask_results = warp_brainmask.run()

241221-14:26:46,726 nipype.interface INFO:
	 stderr 2024-12-21T14:26:46.725897:Illegal instruction
2024-12-21 14:26:46,726 [    INFO] stderr 2024-12-21T14:26:46.725897:Illegal instruction
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.10/site-packages/nibabel/loadsave.py", line 101, in load
    stat_result = os.stat(filename)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/sub-AEON101_ses-001_space-MNI152NLin2009cAsym_desc-brain_mask_trans.nii.gz'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 397, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.10/site-packages/xcp_d/interfaces/ants.py", line 216, in _run_interface
    runtime = super()._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.10/site-packages/niworkflows/interfaces/fixes.py", line 68, in _run_interface
    _copyxform(
  File "/usr/local/miniconda/lib/python3.10/site-packages/niworkflows/utils/images.py", line 79, in _copyxform
    resampled = nb.load(out_image, mmap=False)
  File "/usr/local/miniconda/lib/python3.10/site-packages/nibabel/loadsave.py", line 103, in load
    raise FileNotFoundError(f"No such file or no access: '{filename}'")
FileNotFoundError: No such file or no access: '/tmp/sub-AEON101_ses-001_space-MNI152NLin2009cAsym_desc-brain_mask_trans.nii.gz'

Then through the shell it just doesn’t work. I confirmed that all of the input values here should be valid based on the help for antsApplyTransforms, and all of the file paths point to locations that exist. I also tried removing all of the options and just using the files and I got the same error:

antsApplyTransforms --default-value 0 --dimensionality 3 --float 0 --input /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz --input-image-type 3 --interpolation GenericLabel --output /tmp/sub-AEON101_ses-001_space-MNI152NLin2009cAsym_desc-brain_mask_trans.nii.gz --reference-image /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_desc-preproc_T1w.nii.gz --transform /fmriprep/sub-AEON101/ses-001/anat/sub-AEON101_ses-001_from-MNI152NLin2009cAsym_to-T1w_mode-image_xfm.h5
Illegal instruction

I am using a laptop with an Apple M2 chip. Per this issue it can occur when the compiler instruction set does not match the CPU - do you think that’s what’s happening here?

I know there are (or at least were) issues with running some of the steps in XCP-D on M2 chips, but I don’t actually know what those problems were.

If that issue is the problem, then I might need to build an M-series-specific Docker image, but testing will be difficult. @Steven @mattcieslak do you have any ideas?

EDIT: Did fMRIPrep successfully run through Docker on your laptop?

Hi @tsalo - thanks for explaining. I saw in the documentation there were issues with M1 chips but was hoping M2 would be ok :slight_smile:. I ran fMRIprep on a different laptop with an Intel chip. I can try fMRIprep on this laptop if it’s helpful. I’ll see if I can just run it on my Intel laptop, which I was trying to avoid. If the testing isn’t too involved I’m happy to help and try to run it here.

Confirmed that is the issue - this runs successfully on an Intel machine. Let me know if you want help troubleshooting this on M2 chips. I would certainly use it, but I know you all are very busy and probably have other priorities.

If the only issue is ANTs, then it seems like you may be able to fix it by adjusting your cmake call when building ANTs: