When running fmriprep on some of my subjects fail to run due to SVD not converging for tcompcor, acompcor, and crowncompcor: numpy.linalg.LinAlgError: SVD did not converge in Linear Least Squares.The subjects that fail all fail for one specific task and have no problems in the other tasks. The error I keep receiving for tcompcor is below, I get the exact same error for acompcor and crowncompcor.
Previous posts here have suggested resubmitting and increasing memory. I have tried resubmitting the jobs for the failed subjects over 10 times and some subjects cleared but some consistently are showing the same errors. We’re running this on a high performance computing system and it seems to be only using about 35% of the allocated memory, so I’m not sure increasing memory will help in this case.
Has anybody else found alternative ways to fix this issue?
Alternatively, we’re thinking if we could skip SVD convergence associated with computing noise components, this can allow us run all subjects without errors. Any suggestions on how to skip this procedure?
Command used:
Used singularity to run fmriprep as a batch job with slurm.
...
export SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow
...
singularity run --cleanenv \
--bind ${FMRIPREP_RESOURCES_PATH}:/resources \
--bind ${BIDS_DIR}:/data \
--bind ${WORK_DIR}:/work \
--bind ${OUTPUT_DIR}:/output \
--bind ${FMRIPREP_RESOURCES_PATH}/.cache/fmriprep:/home/fmriprep/.cache/fmriprep \
${FMRIPREP_RESOURCES_PATH}/fmriprep-23.2.0.simg /data /output \
participant --participant_label $PARTICIPANT_LABEL \
-w /work \
-vvv --debug all \
--notrack \
--n_cpus 4 --omp-nthreads 1 --mem-mb 22000 \
--write-graph \
--ignore slicetiming \
--output-spaces MNI152NLin6Asym MNI152NLin2009cAsym \
--fs-license-file /resources/license.txt \
--skip-bids-validation \
--fs-no-reconall # if skipping surface reconstruction
Version:
fmriprep-23.2.0
Environment:
Singularity
Error output for tcompcor
Getting the same error for acompcor and crowncompcor
Node Name: fmriprep_23_2_wf.sub_052_wf.bold_ses_1_task_Video_run_2_wf.bold_confounds_wf.tcompcor
File: /output/sub-052/log/20240521-194119_e6b07c14-dfb0-4056-a899-263120a9c792/crash-20240521-201945-babur-tcompcor-4cfbbe61-15d1-4fac-9fdf-3b7d2bd2afb0.txt
Working Directory: /work/fmriprep_23_2_wf/sub_052_wf/bold_ses_1_task_Video_run_2_wf/bold_confounds_wf/tcompcor
Inputs:
components_file: tcompcor.tsv
failure_mode: NaN
header_prefix: t_comp_cor_
high_pass_cutoff: 128.0
ignore_initial_volumes: 1
mask_files: ['/work/fmriprep_23_2_wf/sub_052_wf/bold_ses_1_task_Video_run_2_wf/bold_fit_wf/unwarp_wf/brainextraction_wf/masker/clipped_mask.nii.gz']
mask_index:
mask_names:
merge_method:
num_components:
percentile_threshold: 0.02
pre_filter: cosine
realigned_file: /work/fmriprep_23_2_wf/sub_052_wf/bold_ses_1_task_Video_run_2_wf/bold_native_wf/boldref_bold/sub-052_ses-1_task-Video_run-2_boldresampled.nii.gz
regress_poly_degree: 1
repetition_time: 1000.0
save_metadata: True
save_pre_filter: True
use_regress_poly:
variance_threshold: 0.5
Traceback (most recent call last):
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/linear.py", line 47, in run
node.run(updatehash=updatehash)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
result = self._run_interface(execute=True)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
return self._run_command(execute)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node tcompcor.
Traceback:
Traceback (most recent call last):
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 397, in run
runtime = self._run_interface(runtime)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/niworkflows/interfaces/patches.py", line 66, in _run_interface
runtime = super()._run_interface(runtime)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/algorithms/confounds.py", line 669, in _run_interface
components, filter_basis, metadata = compute_noise_components(
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/algorithms/confounds.py", line 1423, in compute_noise_components
voxel_timecourses, basis = cosine_filter(
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/algorithms/confounds.py", line 1203, in cosine_filter
betas = np.linalg.lstsq(X, data.T)[0]
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/numpy/linalg/linalg.py", line 2326, in lstsq
x, resids, rank, s = gufunc(a, b, rcond, signature=signature, extobj=extobj)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/numpy/linalg/linalg.py", line 124, in _raise_linalgerror_lstsq
raise LinAlgError("SVD did not converge in Linear Least Squares")
numpy.linalg.LinAlgError: SVD did not converge in Linear Least Squares