Freesurfer failing during fmriprep - _parcstats0 error

katseitz · May 1, 2024, 9:29pm

Summary of what happened:

A freesurfer error is causing fmriprep to fail. The rh and the left half errors slightly differently, but both seem related.

LH error:

ERROR: Label BA1_exvivo does not exist in SUBJECTS_DIR fsaverage! The fsaverage link probably points to an older freesurfer version

RH error:

error: mri_label2label: could not open label file /out/freesurfer/fsaverage/label/rh.BA2_exvivo.thresh.label ERROR reading /out/freesurfer/fsaverage/label/rh.BA2_exvivo.thresh.label Linux qnode0402 3.10.0-1160.95.1.el7.x86_64 #1 SMP Fri Jun 23 08:44:55 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

Command used (and if a helper script was used, a link to the helper script or the command generated):’

list_subs.txt is a text file container a list of subjects.

#!/usr/bin/bash

#SBATCH -A p99999 
#SBATCH -p normal
#SBATCH -t 16:00:00
#SBATCH --array=0-4%10 
#SBATCH --job-name="fmriprep_TEAM_\${SLURM_ARRAY_TASK_ID}"
#SBATCH --output=fmriprep.%A_%a.out
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4
#SBATCH --mem=15G

module purge
module load singularity/latest

IFS=$'\n' read -d '' -r -a input_args < list_subs.txt

#WITHOUT FMAP
singularity run --cleanenv -B /projects/b1108:/projects/b1108 \
-B /projects/b1108/studies/TEAM/data/processed/neuroimaging/fmriprep_23_2:/out \
-B /projects/b1108/studies/TEAM/data/raw/neuroimaging/bids:/data \
-B /projects/b1108/studies/TEAM/data/processed/neuroimaging/fmriprep_23_2/work:/work \
/projects/b1108/software/singularity_images/fmriprep_23.2.0.sif \
/data /out participant --nthreads 4 --omp-nthreads 3 --mem_mb 30000 \
--participant-label ${input_args[$SLURM_ARRAY_TASK_ID]} \
--fs-license-file /projects/b1108/software/freesurfer_license/license.txt \
--output-spaces MNI152NLin6Asym \
--fs-subjects-dir /out/freesurfer \
-w /work --ignore fieldmaps

Version:

23.2.0 for fmriprep

Environment (Docker, Singularity / Apptainer, custom installation):

Singularity

Data formatted according to a validatable standard? Please provide the output of the validator:

bids-validator@1.9.3
bids-specification@disable

e[36m	Please visit https://neurostars.org/search?q=INCONSISTENT_SUBJECTS for existing conversations about this issue.e[39m

	e[33m3: [WARN] Not all subjects/sessions/runs have the same scanning parameters. (code: 39 - INCONSISTENT_PARAMETERS)e[39m
		./sub-09001/ses-1/func/sub-09001_ses-1_task-mid_run-1_bold.nii.gz
			 The most common set of dimensions is: 64,64,35,292 (voxels), This file has the dimensions: 64,64,35,368 (voxels).
		./sub-09002/ses-1/func/sub-09002_ses-1_task-mid_run-1_bold.nii.gz
			 The most common set of dimensions is: 64,64,35,292 (voxels), This file has the dimensions: 64,64,35,368 (voxels).
		./sub-09993/ses-1/func/sub-09993_ses-1_task-mid_run-1_bold.nii.gz
			 The most common set of dimensions is: 64,64,35,292 (voxels), This file has the dimensions: 64,64,35,368 (voxels).
		./sub-09993/ses-1/func/sub-09993_ses-1_task-rest_bold.nii.gz
			 The most common set of dimensions is: 64,64,35,300 (voxels), This file has the dimensions: 64,64,35,247 (voxels).
		./sub-15305/ses-1/func/sub-15305_ses-1_task-mid_run-1_bold.nii.gz
			 The most common set of dimensions is: 64,64,35,292 (voxels), This file has the dimensions: 64,64,35,43 (voxels).
		./sub-15416/ses-1/func/sub-15416_ses-1_task-mid_run-1_bold.nii.gz
			 The most common set of dimensions is: 64,64,35,292 (voxels), This file has the dimensions: 64,64,35,368 (voxels).
		./sub-15427/ses-1/func/sub-15427_ses-1_task-mid_run-1_bold.nii.gz
			 The most common set of dimensions is: 64,64,35,292 (voxels), This file has the dimensions: 64,64,35,547 (voxels).
		./sub-15658/ses-1/fmap/sub-15658_ses-1_magnitude1.nii.gz
			 The most common set of dimensions is: 78,78,35 (voxels), This file has the dimensions: 78,78,39 (voxels).
		./sub-15658/ses-1/fmap/sub-15658_ses-1_magnitude2.nii.gz
			 The most common set of dimensions is: 78,78,35 (voxels), This file has the dimensions: 78,78,39 (voxels).
		./sub-15658/ses-1/fmap/sub-15658_ses-1_phasediff.nii.gz
			 The most common set of dimensions is: 78,78,35 (voxels), This file has the dimensions: 78,78,39 (voxels).
		./sub-15658/ses-1/func/sub-15658_ses-1_task-mid_run-1_bold.nii.gz
			 The most common set of dimensions is: 64,64,35,292 (voxels), This file has the dimensions: 64,64,34,292 (voxels).
		./sub-15909/ses-1/func/sub-15909_ses-1_task-mid_run-2_bold.nii.gz
			 The most common set of dimensions is: 64,64,35,292 (voxels), This file has the dimensions: 64,64,35,259 (voxels).
		./sub-18215/ses-1/anat/sub-18215_ses-1_T1w.nii.gz
			 The most common set of dimensions is: 176,512,512 (voxels), This file has the dimensions: 160,512,512 (voxels).

e[36m	Please visit https://neurostars.org/search?q=INCONSISTENT_PARAMETERS for existing conversations about this issue.e[39m

	e[33m4: [WARN] The recommended file /README is missing. See Section 03 (Modality agnostic files) of the BIDS specification. (code: 101 - README_FILE_MISSING)e[39m

e[36m	Please visit https://neurostars.org/search?q=README_FILE_MISSING for existing conversations about this issue.e[39m

        e[34me[4mSummary:e[24me[39m                   e[34me[4mAvailable Tasks:e[24me[39m        e[34me[4mAvailable Modalities:e[24me[39m 
        2136 Files, 26.95GB        gng                     MRI                   
        137 - Subjects             mid                                           
        1 - Session                rest                                          


e[36m	If you have any questions, please post on https://neurostars.org/tags/bids.e[39m

Relevant log outputs (up to 20 lines):

LH log output:

 9315   6354  20345  2.783 0.693     0.106     0.020       78     8.0  superiortemporal
 6100   4309  11768  2.577 0.467     0.117     0.022       57     5.7  supramarginal
  855    537   1335  2.285 0.384     0.119     0.023        9     0.7  transversetemporal
 3204   2135   6764  3.133 0.724     0.106     0.023       25     2.9  insula
/out/freesurfer/sub-01270/label
#--------------------------------------------
#@# BA_exvivo Labels lh Wed May  1 14:42:01 CDT 2024
-------------------------------------------------------------------------
ERROR: Label BA1_exvivo does not exist in SUBJECTS_DIR fsaverage!
       The fsaverage link probably points to an older freesurfer version
-------------------------------------------------------------------------
Linux qnode0402 3.10.0-1160.95.1.el7.x86_64 #1 SMP Fri Jun 23 08:44:55 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

recon-all -s sub-01270 exited with ERRORS at Wed May  1 14:42:01 CDT 2024

To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting

RH Log output:

 mri_label2label --srcsubject fsaverage --srclabel /out/freesurfer/fsaverage/label/rh.BA2_exvivo.thresh.label --trgsubject sub-01270 --trglabel ./rh.BA2_exvivo.thresh.label --hemi rh --regmethod surface 

No such file or directory

srclabel = /out/freesurfer/fsaverage/label/rh.BA2_exvivo.thresh.label
srcsubject = fsaverage
trgsubject = sub-01270
trglabel = ./rh.BA2_exvivo.thresh.label
regmethod = surface

srchemi = rh
trghemi = rh
trgsurface = white
srcsurfreg = sphere.reg
trgsurfreg = sphere.reg
usehash = 1
Use ProjAbs  = 0, 0
Use ProjFrac = 0, 0
DoPaint 0

SUBJECTS_DIR    /out/freesurfer
FREESURFER_HOME /opt/freesurfer
Loading source label.
error: Invalid argument
error: mri_label2label: could not open label file /out/freesurfer/fsaverage/label/rh.BA2_exvivo.thresh.label
ERROR reading /out/freesurfer/fsaverage/label/rh.BA2_exvivo.thresh.label
Linux qnode0402 3.10.0-1160.95.1.el7.x86_64 #1 SMP Fri Jun 23 08:44:55 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

recon-all -s sub-01270 exited with ERRORS at Wed May  1 14:46:21 CDT 2024

To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting

Screenshots / relevant information:

Looks like I’m experiencing the same issue as these folks: ERROR: Label BA1_exvivo does not exist in SUBJECTS_DIR fsaverage! The fsaverage link probably points to an older freesurfer version · Issue #3258 · nipreps/fmriprep · GitHub. I added --fs-subjects-dir /output/freesurfer to my fmriprep call and still encountered the same error. I have tested with multiple participants.

Steven · May 1, 2024, 9:56pm

Hi @katseitz,

Can you confirm that the fsaverage folder exists in your fs-subjects-dir and that it is not a symlink that might cause trouble for containerized applications? If it is there, was it created by freesurfer as part of fmriprep or by a separate process?

Best,
Steven

katseitz · May 1, 2024, 10:04pm

Hi @Steven,

Yes – looks like it’s all there and folders are populated.

Steven · May 1, 2024, 10:27pm

Hi @katseitz ,

Did you try running with —contain-all (singularity argument) as suggested in the repo issue?

katseitz · May 2, 2024, 12:46am

Hi Steven,

Just tried – I"m getting an OS error:

OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/conda/envs/fmriprep/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/fmriprep/cli/workflow.py", line 115, in build_workflow
    retval["workflow"] = init_fmriprep_wf()
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/fmriprep/workflows/base.py", line 94, in init_fmriprep_wf
    single_subject_wf = init_single_subject_wf(subject_id)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/fmriprep/workflows/base.py", line 322, in init_single_subject_wf
    anat_fit_wf = init_anat_fit_wf(
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/smriprep/workflows/anatomical.py", line 809, in init_anat_fit_wf
    brain_extraction_wf = init_brain_extraction_wf(
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/niworkflows/anat/ants.py", line 198, in init_brain_extraction_wf
    tpl_target_path, common_spec = get_template_specs(
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/niworkflows/utils/misc.py", line 115, in get_template_specs
    tpl_target_path = tf.get(in_template, **template_spec)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/templateflow/conf/__init__.py", line 69, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/templateflow/api.py", line 145, in get
    _s3_get(filepath)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/templateflow/api.py", line 308, in _s3_get
    with filepath.open("wb") as f:
OSError: [Errno 28] No space left on device

Here was my call:

#WITHOUT FMAP
singularity run --cleanenv --containall -B /projects/b1108:/projects/b1108 \
-B /projects/b1108/studies/TEAM/data/processed/neuroimaging/fmriprep_23_2:/out \
-B /projects/b1108/studies/TEAM/data/raw/neuroimaging/bids:/data \
-B /projects/b1108/studies/TEAM/data/processed/neuroimaging/fmriprep_23_2/work:/work \
/projects/b1108/software/singularity_images/fmriprep_23.2.0.sif \
/data /out participant --nthreads 4 --omp-nthreads 3 --mem_mb 30000 \
--participant-label ${input_args[$SLURM_ARRAY_TASK_ID]} \
--fs-license-file /projects/b1108/software/freesurfer_license/license.txt \
--fs-subjects-dir /out/freesurfer \
--output-spaces MNI152NLin6Asym \
-w /work --ignore fieldmaps --skip_bids_validation

We have loads of storage available ~30TB. Is --containall different than --contain-all? Thanks for your help!!

Steven · May 2, 2024, 1:14am

Hi @katseitz

No, that was just a typo on my end

In regards to the error, templateflow might be trying to download something. Unless you changed the TEMPLATEFLOW_HOME environment variable, the default will be to install in your home directory, which on computing clusters is typically very space limited. You can export SINGULARITYENV_TEMPLATEFLOW_HOME=/somewhere/else, just remember that it will also need to be mounted to the container.

Unrelated, but I do not recommend putting your working directory in the fmriprep output directory. fmriprep outputs are designed to be BIDS valid derivatives. The presence of the work directory might mess up softwares that assume BIDS-validity (e.g., Nilearn, XCP-D, Fitlins). I suppose the same goes for your freesurfer subjects dir.

If all else fails, you can just make subject-specific freesurfer output folders to avoid the race conditions, and then move them somewhere centralized after processing.

Best,
Steven

katseitz · May 2, 2024, 6:22pm

Hey Steven,

Sounds good and thanks for the pointers – I’ve moved the freesurfer and work directories out of the fmriprep directory to keep everything bids valid.

I exported the SINGULARITYENV_TEMPLATEFLOW_HOME as recommended. Fmriprep ran ‘successfully’ without errors but also without complete outputs. None of the functional scans were preprocessed and it doesn’t look like freesurfer ran either because the freesurfer/subject/surf directory is empty. This is what the .html output looks like.

The dataset is bids valid so it should be recognizing the functional scans and fmriprep is generating some files in the fmriprep_ouput/subject/ses-1/func folder, but I don’t think it really prepocessed the images. It also doesn’t note any functional scans in the fmiprep .html output. Below is a screenshot of the files that were generated for this subject.

What are some next troubleshooting steps? We’ve used this exact fmriprep call on other datasets with no issues. I’ve visually confirmed that the bids data we’re currently trting to fmriprep looks as expected…

Steven · May 2, 2024, 6:24pm

What is the new command?

katseitz · May 2, 2024, 6:33pm

#WITHOUT FMAP
singularity run --cleanenv --containall -B /projects/b1108:/projects/b1108 \
-B /projects/b1108/studies/TEAM/data/processed/neuroimaging:/base \
-B /projects/b1108/studies/TEAM/data/processed/neuroimaging/fmriprep_23_2:/out \
-B /projects/b1108/templateflow:/projects/b1108/templateflow \
-B /projects/b1108/studies/TEAM/data/raw/neuroimaging/bids:/data \
-B /projects/b1108/studies/TEAM/data/processed/neuroimaging/fmriprep_23_2_work:/work \
/projects/b1108/software/singularity_images/fmriprep_23.2.0.sif \
/data /out participant --nthreads 4 --omp-nthreads 3 --mem_mb 30000 \
--participant-label ${input_args[$SLURM_ARRAY_TASK_ID]} \
--fs-license-file /projects/b1108/software/freesurfer_license/license.txt \
--fs-subjects-dir /base/freesurfer \
--output-spaces MNI152NLin6Asym \
-w /work --ignore fieldmaps --skip_bids_validation

Steven · May 2, 2024, 6:38pm

What happens if you use a fresh working directory?

katseitz · May 2, 2024, 6:39pm

Whoops, I copied over the wrong call…


#WITHOUT FMAP
singularity run --cleanenv --containall -B /projects/b1108:/projects/b1108 \
-B /projects/b1108/studies/TEAM/data/processed/neuroimaging:/base \
-B /projects/b1108/studies/TEAM/data/processed/neuroimaging/fmriprep_23_2:/out \
-B /projects/b1108/templateflow:/projects/b1108/templateflow \
-B /projects/b1108/studies/TEAM/data/raw/neuroimaging/bids:/data \
/projects/b1108/software/singularity_images/fmriprep_23.2.0.sif \
/data /out participant --nthreads 4 --omp-nthreads 3 --mem_mb 30000 \
--participant-label ${input_args[$SLURM_ARRAY_TASK_ID]} \
--fs-license-file /projects/b1108/software/freesurfer_license/license.txt \
--fs-subjects-dir /base/freesurfer \
--output-spaces MNI152NLin6Asym \
-w /base/work --ignore fieldmaps

katseitz · May 2, 2024, 6:40pm

Let me try again, but working directory (and other output directories) were emptied out before the last call.

Steven · May 2, 2024, 6:41pm

Also, you might want to upgrade to 23.2.1 to take advantage of the resampling fix.

katseitz · May 8, 2024, 4:16pm

The issue was resolved when upgrading to 23.2.1! Thanks, @Steven!