Issue: fitting nimare.decode.continuous.CorrelationDecoder to Neurosynth dataset

Software Support
5

Hi Taylor,

Thanks for following up on this and for the clarification!

A couple related questions below:

I am running into some roadblocks with the required walltime to fit the CorrelationDecoder on the full Neurosynth dataset. I have tried to submit the following script on the Niagara cluster requesting the max walltime (24h) on 1 node (~200 GB RAM, 80 logical CPUs):

import os
os.environ['NUMEXPR_MAX_THREADS'] = '64'
import nimare
import sys

out_dir = os.path.abspath("nimare_ns_data/")
neurosynth_dset = nimare.dataset.Dataset.load(os.path.join(out_dir, "neurosynth_dataset.pkl.gz"))
neurosynth_dset.update_path(os.path.join(out_dir, "meta-analyses"))

from nimare.decode import continuous
decoder = continuous.CorrelationDecoder(feature_group=None, features=None, memory_limit='3gb')
decoder.fit(neurosynth_dset)

# Save decoder to avoid having to refit/regenerate MA maps for Neurosynth dataset for future target images
decoder.save(os.path.join(out_dir, "CorrelationDecoder_neurosynth_dset_fitted.pkl.gz"))

decoded_df = decoder.transform("cortex-c1.nii.gz")
decoded_df.to_csv("nimare_results/neurosynth_cortex-c1_decoded.csv")

The issue is that after 24h, only ~45/3228 terms have been decoded. Some things I’ve tried to speed this up include 1) adding os.environ['NUMEXPR_MAX_THREADS'] = '64', since I was initially getting the following warning when running the script:

# Previous warning
INFO:numexpr.utils:Note: detected 80 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
INFO:numexpr.utils:Note: NumExpr detected 80 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
INFO:numexpr.utils:NumExpr defaulting to 8 threads.

However, I’m not 100% sure if the CorrelationDecoder currently supports parallelization?
I’ve also tried 2) setting memory_limit='3gb' (anything much larger seems to result in a MemoryError) when initializing the decoder, but again have not seen much reduction in runtime.

Based on this thread, it seems like this is a known limitation of the current version. But since I am limited also by the max walltime per node, do you have any additional recommendations for speeding up the fitting stage (e.g., decoding the dataset in ‘chunks’ and then concatenating the resultant feature*r dataframes; running it over multiple nodes)?

I would also like to re-run the decoder using a subset of terms of interest (<200) rather than all features in the Neurosynth dataset. Based on the docs and the sample code you provided above, it seems that there are two potential ways I could do this, given a list of features myfeatures:

  1. Reduce features in neurosynth_dset.annotations to those of interest, e.g.
    neurosynth_dset.annotations = neurosynth_dset.annotations[["id", "study_id", "contrast_id", myfeatures[0], myfeatures[1], myfeatures[2] ... ]]
  2. Use the features parameter in the decoder initialization, e.g.
    decoder = continuous.CorrelationDecoder(feature_group=None, features=myfeatures)
    (Assuming all terms in myfeatures have prefix “terms_abstract_tfidf”)

Are 1) and 2) above essentially equivalent in their memory usage when it comes to fitting the decoder? Also, to further cut down on memory/time needed, would you recommend first reducing the studies in the Dataset by looping over myfeatures and using get_studies_by_label() to create a dataset including only studies with at least one of my labels of interest?

Let me know if any clarification is needed, and thank you again kindly,

Best,
Alyssa