Tedana pca error: Mean of empty slice

Summary of what happened:

Hi, I am a new user of tedana. I have just successfullly used tedana to denoise my multi-echo data in the first run. However, when I started to use it to denoise the second run, an error occurred:

/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/tedana/decomposition/pca.py:209: RuntimeWarning: Mean of empty slice.
data_z = (data_z - data_z.mean()) / data_z.std() # var normalize everything

I have checked the adaptive mask but haven’t found anything special. Are there any possible solutions for this problem? Thank you for your help.

Command used (and if a helper script was used, a link to the helper script or the command generated):

My pipeline started with using afni_proc to realign the images and then combine and denoise using tedana. Here is my code.

afni_proc.py -subj_id $sub -blocks tshift volreg mask -copy_anat "T1.nii" -dsets_me_echo "Task_BOLD1/echo1.nii" "Task_BOLD2/echo1.nii" "Task_BOLD3/echo1.nii" "Task_BOLD4/echo1.nii"  -dsets_me_echo "Task_BOLD1/echo2.nii" "Task_BOLD2/echo2.nii" "Task_BOLD3/echo2.nii" "Task_BOLD4/echo2.nii" -dsets_me_echo "Task_BOLD1/echo3.nii" "Task_BOLD2/echo3.nii" "Task_BOLD3/echo3.nii" "Task_BOLD4/echo3.nii"  -reg_echo 2 -tcat_remove_first_trs 2 -volreg_align_to MIN_OUTLIER 

tedana -d "task_bold_run$run""_echo1.nii" "task_bold_run$run""_echo2.nii" "task_bold_run$run""_echo3.nii" -e 14.8 35.02 55.24 --out-dir $outputdir

Version:

tedana v24.0.2

Relevant log outputs (up to 20 lines):

INFO     pca:tedpca:203 Computing PCA of optimally combined multi-echo data with selection criteria: aic
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/tedana/decomposition/pca.py:209: RuntimeWarning: Mean of empty slice.
  data_z = (data_z - data_z.mean()) / data_z.std()  # var normalize everything
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/_core/_methods.py:138: RuntimeWarning: invalid value encountered in scalar divide
  ret = ret.dtype.type(ret / rcount)
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/_core/_methods.py:218: RuntimeWarning: Degrees of freedom <= 0 for slice
  ret = _var(a, axis=axis, dtype=dtype, out=out, ddof=ddof,
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/_core/_methods.py:175: RuntimeWarning: invalid value encountered in divide
  arrmean = um.true_divide(arrmean, div, out=arrmean,
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/_core/_methods.py:210: RuntimeWarning: invalid value encountered in scalar divide
  ret = ret.dtype.type(ret / rcount)
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/tedana/io.py:833: UserWarning: Data array used to create a new image contains 64-bit ints. This is likely due to creating the array with numpy and passing `int` as the `dtype`. Many tools such as FSL and SPM cannot deal with int64 in Nifti images, so for compatibility the data has been converted to int32.
  nii = new_img_like(ref_img, newdata, affine=affine, copy_header=copy_header)
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/bin/tedana", line 8, in <module>
    sys.exit(_main())
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/tedana/workflows/tedana.py", line 1077, in _main
    tedana_workflow(**kwargs, tedana_command=tedana_command)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/tedana/workflows/tedana.py", line 762, in tedana_workflow
    dd, n_components = decomposition.tedpca(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/tedana/decomposition/pca.py", line 215, in tedpca
    _ = ma_pca.fit_transform(data_img, mask_img)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/mapca/mapca.py", line 479, in fit_transform
    self._fit(img, mask, subsample_depth=subsample_depth)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/mapca/mapca.py", line 156, in _fit
    x = self.scaler_.fit_transform(x.T).T  # This was x_sc
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sklearn/utils/_set_output.py", line 316, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sklearn/base.py", line 1098, in fit_transform
    return self.fit(X, **fit_params).transform(X)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sklearn/preprocessing/_data.py", line 878, in fit
    return self.partial_fit(X, y, sample_weight)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sklearn/base.py", line 1473, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sklearn/preprocessing/_data.py", line 914, in partial_fit
    X = self._validate_data(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sklearn/base.py", line 633, in _validate_data
    out = check_array(X, input_name="X", **check_params)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sklearn/utils/validation.py", line 1096, in check_array
    raise ValueError(
ValueError: Found array with 0 feature(s) (shape=(352, 0)) while a minimum of 1 is required by StandardScaler.

This looks like an issue in PCA, but I’m not sure what’s causing it. My best guess is you are inputting empty or corrupted data, but I’m not sure where that would be happening. You note the adaptive mask looks reasonable. Have you opened the full volumes in the AFNI viewer to confirm nothing looks odd in space or time series?

Tedana also writes an output log in a tsv file. Could you share that here? Maybe it contains some into on where things broke down.

Best

Dan

Hi, Thanks for you reply!
I have just checked the data, but found nothing odd. Here is the log file in the tsv file:

2024-11-21T10:16:38 tedana.tedana_workflow INFO Using output directory: /Volumes/OneTouch/Grid_cell_dataset/results/preprocess/sub-01/tedana_out/run-01
2024-11-21T10:16:38 tedana.tedana_workflow INFO Initializing and validating component selection tree
2024-11-21T10:16:38 component_selector.validate_tree WARNING Decision tree includes fields that are not used or logged [‘_comment’]
2024-11-21T10:16:38 component_selector.init INFO Performing component selection with tedana_orig_decision_tree
2024-11-21T10:16:38 component_selector.init INFO Very similar to the decision tree designed by Prantik Kundu
2024-11-21T10:16:38 tedana.tedana_workflow INFO Loading input data: [‘task_bold_run1_echo1.nii’, ‘task_bold_run1_echo2.nii’, ‘task_bold_run1_echo3.nii’]
2024-11-21T10:18:08 tedana.tedana_workflow INFO Computing EPI mask from first echo
2024-11-21T10:18:24 utils.make_adaptive_mask INFO Echo-wise intensity thresholds for adaptive mask: [4277.32031065 2780.40773796 1774.95821635]
2024-11-21T10:18:24 utils.make_adaptive_mask WARNING 25 voxels in user-defined mask do not have good signal. Removing voxels from mask.
2024-11-21T10:18:24 tedana.tedana_workflow INFO Computing T2* map
2024-11-21T10:18:54 combine.make_optcom INFO Optimally combining data with voxel-wise T2* estimates
2024-11-21T10:19:13 tedana.tedana_workflow INFO Writing optimally combined data set: /Volumes/OneTouch/Grid_cell_dataset/results/preprocess/sub-01/tedana_out/run-01/desc-optcom_bold.nii.gz
2024-11-21T10:19:13 pca.tedpca INFO Computing PCA of optimally combined multi-echo data with selection criteria: aic

Best

May Wei

Hi, I skipped the realignment process and successfully used tedana on the raw fMRI data. Do I have to realign the images before tedana? Can I realign the images after tedana?

Best,
May Wei.

I currently recommend running tedana after realignment.

I looked more carefully at your afni_proc statement and I see that you are inputting the multiple echoes into AFNI, but not including the -combine_method and -echo_times options. You then run tedana outside of afni_proc. I’ve never done that before, so I’m not completely sure what it would do in practice.

Maybe try running tedana through afni_proc to see if that solves your above issue. The following should match your functionality from above exact each of the runs will have a separate tedana sub-directory within AFNI’s output and the files will use tedana’s “orig” file naming conventions.

afni_proc.py -subj_id $sub -blocks tshift volreg mask combine \
-copy_anat "T1.nii" \
-dsets_me_echo "Task_BOLD1/echo1.nii" "Task_BOLD2/echo1.nii" "Task_BOLD3/echo1.nii" "Task_BOLD4/echo1.nii"  \
-dsets_me_echo "Task_BOLD1/echo2.nii" "Task_BOLD2/echo2.nii" "Task_BOLD3/echo2.nii" "Task_BOLD4/echo2.nii" \
-dsets_me_echo "Task_BOLD1/echo3.nii" "Task_BOLD2/echo3.nii" "Task_BOLD3/echo3.nii" "Task_BOLD4/echo3.nii"  \
-combine_method m_tedana \
-echo_times 14.8 35.02 55.24 \
-reg_echo 2 -tcat_remove_first_trs 2 -volreg_align_to MIN_OUTLIER 
1 Like

Hi, I tried to run tedana through afni_proc but still got the value error. :sob:

INFO pca:tedpca:203 Computing PCA of optimally combined multi-echo data with selection criteria: aic
/opt/anaconda3/lib/python3.9/site-packages/tedana/decomposition/pca.py:209: RuntimeWarning: Mean of empty slice.
data_z = (data_z - data_z.mean()) / data_z.std() # var normalize everything
/opt/anaconda3/lib/python3.9/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)
/opt/anaconda3/lib/python3.9/site-packages/numpy/core/_methods.py:264: RuntimeWarning: Degrees of freedom <= 0 for slice
ret = _var(a, axis=axis, dtype=dtype, out=out, ddof=ddof,
/opt/anaconda3/lib/python3.9/site-packages/numpy/core/_methods.py:222: RuntimeWarning: invalid value encountered in true_divide
arrmean = um.true_divide(arrmean, div, out=arrmean, casting=‘unsafe’,
/opt/anaconda3/lib/python3.9/site-packages/numpy/core/_methods.py:256: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)
Traceback (most recent call last):
File “/opt/anaconda3/bin/tedana”, line 8, in
sys.exit(_main())
File “/opt/anaconda3/lib/python3.9/site-packages/tedana/workflows/tedana.py”, line 1077, in _main
tedana_workflow(**kwargs, tedana_command=tedana_command)
File “/opt/anaconda3/lib/python3.9/site-packages/tedana/workflows/tedana.py”, line 762, in tedana_workflow
dd, n_components = decomposition.tedpca(
File “/opt/anaconda3/lib/python3.9/site-packages/tedana/decomposition/pca.py”, line 215, in tedpca
_ = ma_pca.fit_transform(data_img, mask_img)
File “/opt/anaconda3/lib/python3.9/site-packages/mapca/mapca.py”, line 479, in fit_transform
self._fit(img, mask, subsample_depth=subsample_depth)
File “/opt/anaconda3/lib/python3.9/site-packages/mapca/mapca.py”, line 156, in fit
x = self.scaler
.fit_transform(x.T).T # This was x_sc
File “/opt/anaconda3/lib/python3.9/site-packages/sklearn/utils/_set_output.py”, line 316, in wrapped
data_to_wrap = f(self, X, *args, **kwargs)
File “/opt/anaconda3/lib/python3.9/site-packages/sklearn/base.py”, line 1098, in fit_transform
return self.fit(X, **fit_params).transform(X)
File “/opt/anaconda3/lib/python3.9/site-packages/sklearn/preprocessing/_data.py”, line 878, in fit
return self.partial_fit(X, y, sample_weight)
File “/opt/anaconda3/lib/python3.9/site-packages/sklearn/base.py”, line 1473, in wrapper
return fit_method(estimator, *args, **kwargs)
File “/opt/anaconda3/lib/python3.9/site-packages/sklearn/preprocessing/_data.py”, line 914, in partial_fit
X = self._validate_data(
File “/opt/anaconda3/lib/python3.9/site-packages/sklearn/base.py”, line 633, in _validate_data
out = check_array(X, input_name=“X”, **check_params)
File “/opt/anaconda3/lib/python3.9/site-packages/sklearn/utils/validation.py”, line 1096, in check_array
raise ValueError(
ValueError: Found array with 0 feature(s) (shape=(346, 0)) while a minimum of 1 is required by StandardScaler.

Any chance you can share this dataset somewhere I can access it and try to process it myself? This feels like something is getting corrupted in the dataset, but it might be subtle.

Could you give me your email address? Maybe I can share the dataset via WeTransfer.
Really appreciate your help!!!

Howdy-

I just have a couple comments on the afni_proc.py side of things.

This is the command you ran (exactly what you entered above, but I have spaced it out into multiple lines):

afni_proc.py                                                                 \
    -subj_id                $sub                                             \
    -blocks                 tshift volreg mask                               \
    -copy_anat              "T1.nii"                                         \
    -dsets_me_echo          "Task_BOLD1/echo1.nii" "Task_BOLD2/echo1.nii"    \
                            "Task_BOLD3/echo1.nii" "Task_BOLD4/echo1.nii"    \
    -dsets_me_echo          "Task_BOLD1/echo2.nii" "Task_BOLD2/echo2.nii"    \
                            "Task_BOLD3/echo2.nii" "Task_BOLD4/echo2.nii"    \
    -dsets_me_echo          "Task_BOLD1/echo3.nii" "Task_BOLD2/echo3.nii"    \
                            "Task_BOLD3/echo3.nii" "Task_BOLD4/echo3.nii"    \
    -reg_echo               2                                                \
    -tcat_remove_first_trs  2                                                \
    -volreg_align_to        MIN_OUTLIER

… and then you run tedana separately. But you can actually run tedana within afni_proc.py. While this might not resolve the issue here, I feel like this would likely be a much easier way to go in general.

This afni_proc.py+tedana integration is discussed in a couple places for examples of how to do this:

  • some examples in the afni_proc.py help
  • the afni_proc.py ME-FMRI demo, described in this OHBM poster and whose data+code can be installed by running:
      @Install_APMULTI_Demo1_rest
    
  • the hot-off-the-presses afni_proc.py description paper, which contains examples of running tedana within afni_proc.py for both surface and volumetric analyses. There is a GitHub repo with all the example processing commands for both desktop or HPC computing setups.
    • the main text’s Ex #4 shows how to run tedana for MEICA within afni_proc.py for a surface-based analysis (desktop version of GitHub script here)
    • supplementary Ex #9 shows running basic volumetric analysis with ME-FMRI in afni_proc.py using AFNI’s optimal combination routine (desktop version of GitHub script here), but to switch that combination method to using tedana’s MEICA, all you would have to do would be to change this option:
         combine_method   OC
      
      … to this one:
         combine_method   m_tedana
      

So, once the current data issue is resolved, you might want to consider adding a combine block to your afni_proc.py command. Note you could also add a regress block for modeling, and then you would get the afni_proc.py QC HTML report, too (which even contains tedana’s MEICA QC report, if you have used that method for combining echos.

–pt

Thank you for your advice! I will try it once the data issue is resolved.

Best,
May Wei

If you want to share the data directly, my email is on my profile page: Daniel A Handwerker, Ph.D. | Section on Functional Imaging Methods

Dan

Hi, I have sent you my data. It includes two runs. I will really appreciate that if you can process it and find possible causes of the issues. Thank you!
Best,
May Wei.

I just ran the dataset fully through with tedana. You sent me two runs and the following command worked:

afni_proc.py -subj_id testsbj -blocks tshift volreg mask combine \
-dsets_me_echo r1-echo1.nii r2-echo1.nii \
-dsets_me_echo r1-echo2.nii r2-echo2.nii \
-dsets_me_echo r1-echo3.nii r2-echo3.nii \
-combine_method m_tedana \
-echo_times 14.8 35.02 55.24 \
-reg_echo 2 -tcat_remove_first_trs 2 -volreg_align_to MIN_OUTLIER 

In case software versions matter, I used the following python and python library versions. (Note I’m using a slightly newer version of tedana, but nothing in my version should affect this part of the code.

Mac OS 14.7.1
Python 3.11.5
Tedana version: |24.0.3.dev14+gdddf251d|
|Other library versions: |{'bokeh': '3.4.1', 'mapca': '0.0.5', 'matplotlib': '3.8.0', 'nibabel': '5.2.0', 'nilearn': '0.10.4', 'numpy': '1.26.3', 'pandas': '2.1.4', 'robustica': '0.1.4', 'scikit-learn': '1.2.2', 'scipy': '1.11.4', 'threadpoolctl': '2.2.0', 'tqdm': '4.66.2'}|

Here’s the full tedana output log from one of the runs.
tedana_2024-11-23T192655.txt (38.9 KB)

One thing I am noticing is that you have severe dropout for much of the frontal cortex. The image below is the mask for the data with green being voxels with at least one used echo and magenta being voxels with at least 3 good echoes. This may or may not be part of your study design, but, for your original code you are aligning to a T1 image. If the dropout causes an extreme alignment failure it might results in significantly fewer voxels and some problems with the PCA. Maybe try your original afni_proc statement without the -copy_anat "T1.nii" line to see if that changes anything. That said, you’re have the copy_anat line, but no align block so I’m not sure if you’re doing any alignment to an anatomical.

Let me know if the above command works on your system. If not, try to send me your python and python library version numbers and I’ll try to replicate with something closer to what you’re using.

Best

Dan

Hi,

Thank you for your help! I tried to run afni_proc.py with the “-combine” option on another Mac and it worked well! I think it might have something to do with the afni on my computer.

Best

May Wei

1 Like

Hi, May Wei-

That’s great that the error messages have been hurdled.

If you would like to discuss more processing-specific considerations and option usage, perhaps starting from a demo example above, I’d be happy to. (Not sure if that should be in a different thread, which could reference back to this one.)

–pt

Hi pt,

Thanks for your advice! I’ll start with the demo example as you suggested.

Regarding the afni_proc process, I’m having some trouble co-registering my T1 image with my EPI images. Could this issue be related to the specific scanning orientation of my EPI images? Are there any additional options I should include?

Thanks for your help!
May Wei.

Hi, May Wei-

I don’t see an “align” block in your afni_proc.py command. That is the name of the block wherein EPI and anatomical data are aligned. Without it, you would only be relying on how the raw data set—that is typically not optimal alignment, even if they data came from a single session, because subjects can move within or between scans.

I think it would be good to put together a full afni_proc.py command, for example starting from one of the sources listed above:

This afni_proc.py+tedana integration is discussed in a couple places for examples of how to do this:

  • some examples in the afni_proc.py help
  • the afni_proc.py ME-FMRI demo, described in this OHBM poster and whose data+code can be installed by running:
  @Install_APMULTI_Demo1_rest
  • the hot-off-the-presses afni_proc.py description paper , which contains examples of running tedana within afni_proc.py for both surface and volumetric analyses. There is a GitHub repo with all the example processing commands for both desktop or HPC computing setups.

    • the main text’s Ex #4 shows how to run tedana for MEICA within afni_proc.py for a surface-based analysis (desktop version of GitHub script here)
    • supplementary Ex #9 shows running basic volumetric analysis with ME-FMRI in afni_proc.py using AFNI’s optimal combination routine (desktop version of GitHub script here), but to switch that combination method to using tedana’s MEICA, all you would have to do would be to change this option:
       combine_method   OC
    

    … to this one:

       combine_method   m_tedana
    

–pt