Summary of what happened:
Hi all, I am using Tedana to process multi-echo functional images and I’m running into an issue that I haven’t encountered before concerning SVD convergence in a Python subscript.
I am using fmriprep to preprocess and motion correct the images, and passing the ACPC-aligned, echo-separated images to Tedana.
The error I am getting is “numpy.linalg.LinAlgError: SVD did not converge” about 1.5 hours into the processing. I have seen from other posts that this failure often occurs when there is a memory/resource issue, but I am running this via an LPC computing cluster and have not experienced this before.
Command used (and if a helper script was used, a link to the helper script or the command generated):
This is the Tedana call:
MEPCA=kundu
MaxIterations=500
MaxRestarts=10
tedana -d "$Subdir"/func/rest/"$DataDirs"/Rest_E*_acpc.nii.gz -e $(cat "$Subdir"/func/rest/"$DataDirs"/TE.txt) --out-dir "$Subdir"/func/rest/"$DataDirs"/Tedana/ \
--tedpca "$MEPCA" --fittype curvefit --mask "$Subdir"/func/rest/"$DataDirs"/brain_mask.nii.gz --maxit "$MaxIterations" --maxrestart "$MaxRestarts" --seed 42 --verbose # specify more iterations / restarts to increase likelihood of ICA convergence (also increases possible runtime).
and this is the submission command for our bsub cluster:
bsub -q bsc_long -e /project/oathes_analysis2/R61/logs/HCP/sub-$(echo $i)_func3.e -o /project/oathes_analysis2/R61/logs/HCP/sub-$(echo $i)_func3.o -n 16 -M 50000 -R "rusage [mem=50000] span[hosts=1]" -N sh adjusted_func_denoise_meica.sh /project/oathes_analysis2/R61/ $i
Version:
tedana v23.0.2
Environment (Docker, Singularity / Apptainer, custom installation):
Running via a conda environment:
module load miniconda/3-22.11
eval "$(/appl/miniconda3-22.11/bin/conda shell.bash hook)"
conda activate /project/oathes_analysis2/R61/tedanacondaenv
export PYTHONPATH=/project/oathes_analysis2/R61/tedanacondaenv/lib/python3.9/site-packages:$PYTHONPATH
export PATH=/project/oathes_analysis2/R61/tedanacondaenv/bin:$PATH
export PATH=/project/oathes_analysis2/R61/tedanacondaenv/lib:$PATH
Relevant log outputs (up to 20 lines):
202 Traceback (most recent call last):
203 File "/project/oathes_analysis2/R61/tedanacondaenv/bin/tedana", line 8, in <module>
204 sys.exit(_main())
205 File "/project/oathes_analysis2/R61/tedanacondaenv/lib/python3.9/site-packages/tedana/workflows/tedana.py", line 918, in _main
206 tedana_workflow(**kwargs, tedana_command=tedana_command)
207 File "/project/oathes_analysis2/R61/tedanacondaenv/lib/python3.9/site-packages/tedana/workflows/tedana.py", line 662, in tedana_workflow
208 mmix, seed = decomposition.tedica(
209 File "/project/oathes_analysis2/R61/tedanacondaenv/lib/python3.9/site-packages/tedana/decomposition/ica.py", line 67, in tedica
210 ica.fit(data)
211 File "/project/oathes_analysis2/R61/tedanacondaenv/lib/python3.9/site-packages/sklearn/base.py", line 1474, in wrapper
212 return fit_method(estimator, *args, **kwargs)
213 File "/project/oathes_analysis2/R61/tedanacondaenv/lib/python3.9/site-packages/sklearn/decomposition/_fastica.py", line 733, in fit
214 self._fit_transform(X, compute_sources=False)
215 File "/project/oathes_analysis2/R61/tedanacondaenv/lib/python3.9/site-packages/sklearn/decomposition/_fastica.py", line 621, in _fit_transform
216 u, d = linalg.svd(XT, full_matrices=False, check_finite=False)[:2]
217 File "/appl/miniconda3-22.11/lib/python3.9/site-packages/scipy/linalg/_decomp_svd.py", line 145, in svd
218 raise LinAlgError("SVD did not converge")
219 numpy.linalg.LinAlgError: SVD did not converge
I am happy to provide more context or figures/logs as needed.
If anyone has any suggestions or recommendations to get this to work, please let me know.