nilearn.regions.Parcellations and masking

Hello,

I’m hoping to apply nilearn’s Parcellations tool to a small mask within a high-resolution dataset with many timepoints, however, I’m running into a memory error.

kmeans = Parcellations(method='kmeans', n_parcels=2,
    mask = mask_image)
kmeans.fit(concat_data)

Instead of passing the entire, high-resolution 4-D dataset (concat_data here), is there any sort of workaround here? E.g., where I only need to pass the np array data within the mask? Any help would be greatly appreciated.

Best,

Tom

Can you paste the full traceback, it will help us know where the exact problem is.
Thanks!

If you want to use directly a numpy array, you can apply scikit-learn kmean’s algorithm on it.

Here is the full trace back:

File "parcellation_script.py", line 101, in <module>
    kmeans.fit(concat_data)
  File "/home/tvanasse/nilearn/nilearn/decomposition/base.py", line 411, in fit
    n_jobs=self.n_jobs)
  File "/home/tvanasse/nilearn/nilearn/decomposition/base.py", line 175, in mask_and_reduce
    ) for img, confound in zip(imgs, confounds))
  File "/home/tvanasse/miniconda/envs/nsddata/lib/python3.7/site-packages/joblib/parallel.py", line 1003, in __call__
    if self.dispatch_one_batch(iterator):
  File "/home/tvanasse/miniconda/envs/nsddata/lib/python3.7/site-packages/joblib/parallel.py", line 834, in dispatch_one_batch
    self._dispatch(tasks)
  File "/home/tvanasse/miniconda/envs/nsddata/lib/python3.7/site-packages/joblib/parallel.py", line 753, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "/home/tvanasse/miniconda/envs/nsddata/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 201, in apply_async
    result = ImmediateResult(func)
  File "/home/tvanasse/miniconda/envs/nsddata/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 582, in __init__
    self.results = batch()
  File "/home/tvanasse/miniconda/envs/nsddata/lib/python3.7/site-packages/joblib/parallel.py", line 256, in __call__
    for func, args, kwargs in self.items]
  File "/home/tvanasse/miniconda/envs/nsddata/lib/python3.7/site-packages/joblib/parallel.py", line 256, in <listcomp>
    for func, args, kwargs in self.items]
  File "/home/tvanasse/nilearn/nilearn/decomposition/base.py", line 205, in _mask_and_reduce_single
    this_data = masker.transform(img, confound)
  File "/home/tvanasse/nilearn/nilearn/input_data/multi_nifti_masker.py", line 326, in transform
    return self.transform_single_imgs(imgs)
  File "/home/tvanasse/nilearn/nilearn/input_data/nifti_masker.py", line 405, in transform_single_imgs
    dtype=self.dtype
  File "/home/tvanasse/miniconda/envs/nsddata/lib/python3.7/site-packages/joblib/memory.py", line 355, in __call__
    return self.func(*args, **kwargs)
  File "/home/tvanasse/nilearn/nilearn/input_data/nifti_masker.py", line 59, in filter_and_mask
    dtype=dtype)
  File "/home/tvanasse/nilearn/nilearn/input_data/base_masker.py", line 94, in filter_and_extract
    imgs, parameters['smoothing_fwhm'])
  File "/home/tvanasse/miniconda/envs/nsddata/lib/python3.7/site-packages/joblib/memory.py", line 355, in __call__
    return self.func(*args, **kwargs)
  File "/home/tvanasse/nilearn/nilearn/image/image.py", line 287, in smooth_img
    ensure_finite=True, copy=True)
  File "/home/tvanasse/nilearn/nilearn/image/image.py", line 225, in _smooth_array
    arr[np.logical_not(np.isfinite(arr))] = 0
MemoryError: Unable to allocate array with shape (182, 218, 182, 7500) and data type bool

And using scikit-learn kmean’s algorithm directly seems like a fine idea… thanks for the suggestion.

OK, it breaks in the smoothing. If you want smoothing (which is important for k-means) you cannot simply use scikit-learn. But maybe you smoothed before.

I am smoothing with the following code before:

masker = NiftiMasker(mask_img = mask_image, smoothing_fwhm=2, standardize=False)
roi_data = masker.fit_transform(all_data)
from sklearn.cluster import KMeans
kmeans = KMeans(n_clusters=2, random_state=0).fit(np.transpose(masked_data))
parcel_image = masker.inverse_transform(kmeans.labels_ + 1) # +1 change 0 to 1, and 1 to 2

Thanks for your help, and your all your work in creating these tools!

-Tom