Fmriprep: numpy.linalg.linalg.LinAlgError: SVD did not converge

When running fmriprep on some of my subjects fail to run due to SVD not converging for tcompcor, acompcor, and crowncompcor: numpy.linalg.LinAlgError: SVD did not converge in Linear Least Squares.The subjects that fail all fail for one specific task and have no problems in the other tasks. The error I keep receiving for tcompcor is below, I get the exact same error for acompcor and crowncompcor.

Previous posts here have suggested resubmitting and increasing memory. I have tried resubmitting the jobs for the failed subjects over 10 times and some subjects cleared but some consistently are showing the same errors. We’re running this on a high performance computing system and it seems to be only using about 35% of the allocated memory, so I’m not sure increasing memory will help in this case.

Has anybody else found alternative ways to fix this issue?

Alternatively, we’re thinking if we could skip SVD convergence associated with computing noise components, this can allow us run all subjects without errors. Any suggestions on how to skip this procedure?

Command used:

Used singularity to run fmriprep as a batch job with slurm.

export SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow
singularity run --cleanenv \
    --bind ${FMRIPREP_RESOURCES_PATH}:/resources \
    --bind ${BIDS_DIR}:/data \
    --bind ${WORK_DIR}:/work \
    --bind ${OUTPUT_DIR}:/output \
    --bind ${FMRIPREP_RESOURCES_PATH}/.cache/fmriprep:/home/fmriprep/.cache/fmriprep \
    ${FMRIPREP_RESOURCES_PATH}/fmriprep-23.2.0.simg /data /output \
    participant --participant_label $PARTICIPANT_LABEL \
    -w /work \
    -vvv --debug all \
    --notrack \
    --n_cpus 4 --omp-nthreads 1 --mem-mb 22000 \
    --write-graph \
    --ignore slicetiming \
    --output-spaces MNI152NLin6Asym MNI152NLin2009cAsym \
    --fs-license-file /resources/license.txt \
    --skip-bids-validation \
    --fs-no-reconall # if skipping surface reconstruction





Error output for tcompcor

Getting the same error for acompcor and crowncompcor

Node Name: fmriprep_23_2_wf.sub_052_wf.bold_ses_1_task_Video_run_2_wf.bold_confounds_wf.tcompcor

File: /output/sub-052/log/20240521-194119_e6b07c14-dfb0-4056-a899-263120a9c792/crash-20240521-201945-babur-tcompcor-4cfbbe61-15d1-4fac-9fdf-3b7d2bd2afb0.txt
Working Directory: /work/fmriprep_23_2_wf/sub_052_wf/bold_ses_1_task_Video_run_2_wf/bold_confounds_wf/tcompcor
components_file: tcompcor.tsv
failure_mode: NaN
header_prefix: t_comp_cor_
high_pass_cutoff: 128.0
ignore_initial_volumes: 1
mask_files: ['/work/fmriprep_23_2_wf/sub_052_wf/bold_ses_1_task_Video_run_2_wf/bold_fit_wf/unwarp_wf/brainextraction_wf/masker/clipped_mask.nii.gz']
percentile_threshold: 0.02
pre_filter: cosine
realigned_file: /work/fmriprep_23_2_wf/sub_052_wf/bold_ses_1_task_Video_run_2_wf/bold_native_wf/boldref_bold/sub-052_ses-1_task-Video_run-2_boldresampled.nii.gz
regress_poly_degree: 1
repetition_time: 1000.0
save_metadata: True
save_pre_filter: True
variance_threshold: 0.5
Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/", line 47, in run
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node tcompcor.

	Traceback (most recent call last):
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/", line 397, in run
	    runtime = self._run_interface(runtime)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/niworkflows/interfaces/", line 66, in _run_interface
	    runtime = super()._run_interface(runtime)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/algorithms/", line 669, in _run_interface
	    components, filter_basis, metadata = compute_noise_components(
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/algorithms/", line 1423, in compute_noise_components
	    voxel_timecourses, basis = cosine_filter(
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/algorithms/", line 1203, in cosine_filter
	    betas = np.linalg.lstsq(X, data.T)[0]
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/numpy/linalg/", line 2326, in lstsq
	    x, resids, rank, s = gufunc(a, b, rcond, signature=signature, extobj=extobj)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/numpy/linalg/", line 124, in _raise_linalgerror_lstsq
	    raise LinAlgError("SVD did not converge in Linear Least Squares")
	numpy.linalg.LinAlgError: SVD did not converge in Linear Least Squares

Hi @Begum_Babur and welcome to neurostars!

What if you do not single thread all the processes (raise omp-nrhreads).

Also unrelated but —fs-no-reconall is generally not recommended.

Are your data BIDS valid?


This is pretty surprising, given the simplicity of the regressors. np.linalg.lstsq(X, data.T)[0] should be equivalent to:

betas = np.linalg.pinv(X) @ data.T

Perhaps nipype should update its detrending so as not to be dependent on numpy’s implementation of lstsq.

Do you have the MRIQC output for these BOLD series? I would be curious to see the carpet plots.