Crash when running xcp_d 0.1.3 (nodes left_sphere_raw_mris and right_sphere_raw_mris)

Software Support
1

Hi xcp-d experts:
I am running xcp 0.1.3, and running into some errors related to left_sphere_raw_mris and right_sphere_raw_mris, that I am hoping you can help me with.

I am running the following command line:

singularity run --cleanenv -B /ix/dhafeman:/home/xcp_d /ix/dhafeman/singularity/xcp_d-0.1.3.simg /ix/dhafeman/BDLONG/Preproc/fmriprep /ix/dhafeman/BDLONG/Preproc participant -f 0.1 -w /ix/dhafeman/BDLONG/Work --smoothing 8 --despike -p 36P --lower-bpf .009 --upper-bpf .08 --motion-filter-type notch --band-stop-min 12 --band-stop-max 20 --participant_label 400

I am getting multiple crash reports (2 for each side):

Node: xcpd_wf.single_subject_400_wf.anatomical_wf.left_sphere_raw_mris
Working directory: /ix/dhafeman/BDLONG/Work/xcpd_wf/single_subject_400_wf/anatomical_wf/left_sphere_raw_mris

Node inputs:

annot_file =
args =
dataarray_num =
environ = {‘SUBJECTS_DIR’: ‘/opt/freesurfer/subjects’}
functional_file =
in_file = /ix/dhafeman/BDLONG/Preproc/freesurfer/sub-400/surf/lh.sphere.reg
label_file =
labelstats_outfile =
normal =
origname =
out_datatype = gii
out_file =
parcstats_file =
patch =
rescale =
scalarcurv_file =
scale =
subjects_dir = /opt/freesurfer/subjects
talairachxfm_subjid =
to_scanner =
to_tkr =
vertex =
xyz_ascii =

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py”, line 67, in run_node
result[“result”] = node.run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py”, line 524, in run
result = self._run_interface(execute=True)
File “/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py”, line 642, in _run_interface
return self._run_command(execute)
File “/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py”, line 750, in _run_command
raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node left_sphere_raw_mris.

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py”, line 454, in aggregate_outputs
setattr(outputs, key, val)
File “/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py”, line 330, in validate
value = super(File, self).validate(objekt, name, value, return_pathlike=True)
File “/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py”, line 135, in validate
self.error(objekt, name, str(value))
File “/usr/local/miniconda/lib/python3.8/site-packages/traits/base_trait_handler.py”, line 74, in error
raise TraitError(
traits.trait_errors.TraitError: The ‘converted’ trait of a MRIsConvertOutputSpec instance must be a pathlike object or string representing an existing file, but a value of ‘/ix/dhafeman/BDLONG/Work/xcpd_wf/single_subject_400_wf/anatomical_wf/left_sphere_raw_mris/lh.sphere.reg_converted.gii’ <class ‘str’> was specified.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py”, line 401, in run
outputs = self.aggregate_outputs(runtime)
File “/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py”, line 461, in aggregate_outputs
raise FileNotFoundError(msg)
FileNotFoundError: No such file or directory ‘/ix/dhafeman/BDLONG/Work/xcpd_wf/single_subject_400_wf/anatomical_wf/left_sphere_raw_mris/lh.sphere.reg_converted.gii’ for output ‘converted’ of a MRIsConvert interface

xcp-d continues to run; however, I think this is slowing down analyses, and I also don’t know whether results will be valid. Thanks so much for your help with this!

Best,
Danella

2

Hello,

Can you please elaborate on what outputs (fMRIPrep, Freesurfer) you are working with? Is this error subject specific or for each subject? Did fMRIPrep have any errors?

Best,
Steven

3

Hi Steven,
Thanks in advance for your help with this!
I ran fmriprep on flywheel.

fMRIPrep version: 20.2.6
fMRIPrep command: /usr/local/miniconda/bin/fmriprep /flywheel/v0/work/bids /flywheel/v0/output/633e2471ac3f8a275e4ca1ab participant --aroma-melodic-dimensionality=-200 --bold2t1w-dof=6 --bold2t1w-init=register --dvars-spike-threshold=1.5 --fd-spike-threshold=0.5 --n_cpus=8 --omp-nthreads=8 --output-spaces MNI152NLin2009cAsym MNI152NLin6Asym --skull-strip-t1w=force --skull-strip-template=OASIS30ANTs --mem=51216

There were no errors, and all of the outputs look reasonable. I haven’t yet tried another subject, but can do that if there is some reason this might be subject-specific. Thanks!

4

Do you have FreeSurfer outputs? Usually “sphere” refers to a kind of surface object. Would it be possible to try with a more recent fmriprep version, and/or rerunning fmriprep to produce cifti outputs?

5

Yes, the freesurfer outputs seem to be there, including the inputs to these nodes:
/ix/dhafeman/BDLONG/Preproc/freesurfer/sub-400/surf/lh.sphere.reg
/ix/dhafeman/BDLONG/Preproc/freesurfer/sub-400/surf/rh.sphere.reg

I am not sure how running cifti would help - doesn’t this just move BOLD images to the surface? But I am willing to try if there is good rationale. Also, this fmriprep version is the most recent available on our flywheel instance - I can ask the flywheel help desk for a more updated version, if you think it might help.

Thanks!
Best,
Danella

6

It may make fmriprep produce anatomical files that XCP_D expects to see.

It is worth a shot! (edit, I see flywheel only supports up to 20.2.7, with an additional branch for v21).

Best,
Steven

7

Oh, you might need to bind the freesurfer drive in your singularity command.

It looks like the FreeSurfer path is assumed based on the location of the fMRIPrep directory (xcp_d/anatomical.py at 6071cc7920da90eec6337bde3bcc53356574f985 · PennLINC/xcp_d · GitHub). Is your freesurfer directory where it should be relative to where fmriprep usually puts it?

8

Can you also see what kind of files are in the scratch directory mentioned in the error report?

9

Yes - I believe so - and xcp-d continues to run even after this crash, so this is somehow not a fatal error. This is the node/command that seems to fail:

mris_convert /ix/dhafeman/BDLONG/Preproc/freesurfer/sub-400/surf/rh.sphere.reg /ix/dhafeman/BDLONG/Work/xcpd_wf/single_subject_400_wf/anatomical_wf/right_sphere_raw_mris/rh.sphere.reg_converted.gii

…and the input file (/ix/dhafeman/BDLONG/Preproc/freesurfer/sub-400/surf/rh.sphere.reg ) seems to be in the right place.

Apologies, I am not totally sure what you mean. Do you mean the Work directory?

ls /ix/dhafeman/BDLONG/Work/xcpd_wf/single_subject_400_wf/anatomical_wf/left_sphere_raw_mris
command.txt _node.pklz result_left_sphere_raw_mris.pklz
_inputs.pklz _report

I just sent the flywheel folks a request for a 21x or 22x version of fmriprep; and am rerunning fmriprep with the cifti option. If you have any other ideas, I appreciate it.

Thanks!
Danella

10

I am a bit confused with your Bind string, as you bind /ix/dhafeman as /home/xcp_d, but then do not reference /home/xcp_d in your command, and instead point xcp_d to paths in /ix/dhafeman. If everything you need is in /ix/dhafeman and you want to continue pointing singularity to directories named /ix/dhafeman, you should not rename it when you bind it in singularity.

Best,
Steven

11

Right - I didn’t have the bind command previously, and was getting the error message that /home/xcp_d was a read-only directory. Now that I have this binding command, I am no longer getting this error message.

This was the error previously:

Matplotlib created a temporary config/cache directory at /tmp/matplotlib-bywxs5ja because the default path (/home/xcp_d/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
221006-19:24:35,812 nipype.workflow IMPORTANT:

Running xcp_d version 0.1.3+0.g9fec4d8.dirty:
  * fMRI directory path: /ix/dhafeman/BDLONG/Preproc/fmriprep.
  * Participant list: ['400'].
  * Run identifier: 20221006-192430_0de302ed-1833-474f-94e6-402045b34555.

221006-19:24:41,733 nipype.utils WARNING:
A newer version (1.8.4) of nipy/nipype is available. You are using 1.8.3
Downloading https://templateflow.s3.amazonaws.com/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz
Process Process-2:
Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.8/multiprocessing/process.py”, line 315, in _bootstrap
self.run()
File “/usr/local/miniconda/lib/python3.8/multiprocessing/process.py”, line 108, in run
self._target(*self._args, **self._kwargs)
File “/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/cli/run.py”, line 617, in build_workflow
retval[‘workflow’] = init_xcpd_wf(
File “/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/workflow/base.py”, line 148, in init_xcpd_wf
single_subj_wf = init_subject_wf(
File “/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/workflow/base.py”, line 395, in init_subject_wf
bold_postproc_wf = init_boldpostprocess_wf(
File “/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/workflow/bold.py”, line 374, in init_boldpostprocess_wf
get_template(‘MNI152NLin2009cAsym’,
File “/usr/local/miniconda/lib/python3.8/site-packages/templateflow/api.py”, line 68, in get
_s3_get(filepath)
File “/usr/local/miniconda/lib/python3.8/site-packages/templateflow/api.py”, line 217, in _s3_get
with filepath.open(“wb”) as f:
File “/usr/local/miniconda/lib/python3.8/pathlib.py”, line 1221, in open
return io.open(self, mode, buffering, encoding, errors, newline,
File “/usr/local/miniconda/lib/python3.8/pathlib.py”, line 1077, in _opener
return self._accessor.open(self, flags, mode)
OSError: [Errno 30] Read-only file system: ‘/home/xcp_d/.cache/templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz’

Is there any way that this “fix” could actually be contributing to other problems? Thanks!

Best,
Danella

12

How about you keep the bind string as it was, but change the paths in the XCP_D command, but change the input paths to reflect this change, e.g.:

singularity run --cleanenv -B /ix/dhafeman:/home/xcp_d /ix/dhafeman/singularity/xcp_d-0.1.3.simg /home/xcp_d/BDLONG/Preproc/fmriprep /home/xcp_d/BDLONG/Preproc participant -f 0.1 -w /home/xcp_d/BDLONG/Work --smoothing 8 --despike -p 36P --lower-bpf .009 --upper-bpf .08 --motion-filter-type notch --band-stop-min 12 --band-stop-max 20 --participant_label 400

13

Ok - I am running another subject using the input paths you suggested. I also moved the freesurfer directory so that it is in /fmriprep/sourcedata/freesurfer, since that is the standard location. I am still getting the same crash report though.

Interestingly, my initial subject ran to completion, and in the .html output file does not list any errors. (There were two additional crash reports, related to rh_surface_apply_warpfield lh_surface_apply_warpfield, with very similar errors:

TraitError: The ‘out_file’ trait of an ApplyWarpfieldOutputSpec instance must be a pathlike object or string representing an existing file, but a value of ‘/ix/dhafeman/BDLONG/Work/xcpd_wf/single_subject_400_wf/anatomical_wf/in_file…ix…dhafeman…BDLONG…Preproc…fmriprep…sub-400…ses-01…anat…sub-400_ses-01_acq-mprage_hemi-R_pial.surf.gii/rh_surface_apply_warpfield/sub-400_ses-01_acq-mprage_hemi-R_pial.surf-MNIwarped.surf.gii’ <class ‘str’> was specified.

I will try with cifti output enabled, and also a new version of fmriprep, but I am also wondering if this output is ok, despite the crash reports? Thanks!

Best,
Danella

14

Update: I ran fmriprep 21.0.2 on the cluster for subject 400, and the same crash errors resulted - so it is not a version issue. Also, I ran on another subject (401) and the same crash occurred. (Of note, even though I get a slurm message that the process failed, all of the files seem to be there.) I have run fmriprep on flywheel with the cifti option, so I will see if this works. Thanks!

15

I haven’t been able to replicate this, but it might be worth opening an issue on the XCP_D github repo.