_autorecon_surfs0 MRISread rh.white: could not open file

I’m running fmriprep on some BIDS validated data, and our pipeline successfully completes on other subjects/sessions, howeverI’m receiving the same error over and over again on one particular session. The source of the error seems to be:

MRISread(/flywheel/v0/output/5de9524af99936001ce1a19f/freesurfer/sub-TOME3024/surf/rh.white): could not open file

This occurs in the node _autorecon_surfs0

New users can’t attach files here, however I’ve linked to the following files in pastebin:

The file “rh.white” is present in the outputs, and seems to be correctly registered with the brain, as it can be loaded in fsleyes along with “lh.white”, which match with the subject’s T1.

looking at the “recon-all-lh.cmd” log, it seems to error on the command:
bbregister --s sub-TOME3024 --mov /flywheel/v0/output/5de9524af99936001ce1a19f/freesurfer/sub-TOME3024/mri/orig/T2raw.mgz --lta /flywheel/v0/output/5de9524af99936001ce1a19f/freesurfer/sub-TOME3024/mri/transforms/T2raw.auto.lta --init-coreg --T2

This causes recon-all-lh to crash (or at least have an error in its log that prevents it from continuing. ). Later, recon-all-rh is run, and an identical command is called. This time, it completes successfully. In the logs, we see that “rh.white” is loaded and used successfully. This log is timestamped AFTER recon-all-lh.log.

I saw a similar error posted, but it was for DWI and tracula. The way we run our scripts, it’s not feasible for us to wait for the crash, fix the error, and continue running fmriprep from there. Does anyone have any suggestions for how to avoid this?

The full fmriprep call is as follows:
usr/local/miniconda/bin/fmriprep --output-spaces=MNI152NLin2009cAsym --bold2t1w-dof=6 --aroma-melodic-dimensionality=200 --fd-spike-threshold=2 --dvars-spike-threshold=2 --skull-strip-template=OASIS30ANTs --fmap-no-demean --write-graph --sloppy --fs-license-file=/flywheel/v0/input/freesurfer-license/license.txt /flywheel/v0/work/bids /flywheel/v0/output/5de951ebf999360029e1a376 participant

What version of fMRIPrep are you using? This looks like an issue that should be resolved in fMRIPrep 1.5.1+.

This error occurred in version 1.5.2, which I’ve verified from the logs. We use the provided poldracklab/fmriprep:1.5.2 dockerfile for our environment. We’re in the process of updating our pipeline, and have iterated over 1.5.0 and 1.5.2. I believe we also received this error with 1.5.0, so I don’t think 1.5.2 broke anything.

Thanks for the information. It looks like our FreeSurfer parallelization does not account for some T2 preprocessing steps that require synchronization. (FreeSurfer’s internal parallelization runs all jobs in lock-step, while we permit a bit more freedom.)

I’ll try to track down the root of this.