Searchlight Cluster-Level Significance

1

Hi All,

I recently ran a searchlight and am attempting to identify significant clusters that reliably predict above chance and above behavior (accuracy of ~63%).

I’m a bit unsure how to check for significance in either situation. I’ve been using Nilearn and understand how to look for clusters above a certain accuracy and cluster size threshold, but am unsure what threshold to place.

I believe the best route would be to run a permutation test but should that be wholebrain or only on clusters I believe to be interesting (how do I determine that)?

Right now I’m using a simple cluster threshold of 10 and an accuracy threshold of 60% to distinguish my clusters and then intend to report the accuracy of the peak of each cluster, is this reasonable?

2

Your procedure may provide meaningful results to your eyes, but you cannot claim significance.
To claim significance, you need to run a formal permutation test on the whole brain or use some kind of Bonferroni correction.
I know that this is not great, but this is a core issue with searchlight approaches…
Best,
Bertrand

1 Like
3

Is there a reason I can’t do a permutation test only using the areas of interest?

4

If your ROIs were really selected independently from your current data, indeed you can restrict the test to these regions.
Best,
Bertrand

1 Like
5

I meant after I run my searchlight only some voxels are predictive above some threshold. Can I only run my permutation tests on the predictive voxels rather than whole brain?

I think I’ve seen this done in papers before and would make the computational cost tractable for me. Otherwise we’d be looking at 10’s of thousands of cpu hours, which just won’t work.

6

No, that’s double-dipping and you will get many false positives. I’d suggest having a look at:

1 Like
7

Ah so I’d end up assuming results are more significant than they truly are?

Is the idea that I should be trying to see how often clusters of this size + this threshold occurs in every voxel not just the probability of this occurring in THESE direct voxels?

Is that right?

8

Basically, yes. You want to know how unlikely your result is to occur if there is no signal to detect and the method is simply producing numbers out of its null distribution. The more complicated the statistics, generally, the harder it is to predict what that null distribution looks like; hence permutation tests where you destroy the connection between the signal you’re trying to find correlates for and just crunch the numbers to see what they do.

1 Like