ComBat and cross-validation

neuroimaging
mvpa
#1

Dear Neurostars,

Recently a few papers came out on ComBat; a tool that can be used for removing site effects from your multi-site datasets. According to these papers; it does a pretty good job removing unwanted scanner artifacts from modalities like DTI, functional connectivity (ref) and cortical thickness measurements ref.

In short, ComBat is an extension of a linear regression model that uses empirical Bayes to improve the estimation of the site parameters. I wanted to give this a try myself; and luckily ComBat code is available for Matlab; R and Python

Now I want to try out ComBat for my classification study, in which I try to separate patients from controls using cortical features of >4000 subjects, coming from 46 unique sites around the world. My model optimization, training and testing are performed in separate (inner- and outer-) cross-validation loops. The problem however is that ComBat seems to be a “one-shot” approach; in the sense that it is run only once on the entire data set instead of fitting the harmonization model’s parameters on the training data only; and apply them on both training and test data like you would typically do.

Is anyone more familiar with these kind of harmonization techniques, or has some recommendations on how to get rid of site-specific effects in multi-site data-sets in a cross-validated manner?

Many thanks,

Best,

Willem

1 Like
#2

For anyone interested; here are the github repo’s for ComBat inplemented in Matlab, R and Python:


#3

I am not an expert in these kind of approaches.
However, I also coded a python version of Combat and applied to DWI, as Fortin’s paper.
On my experience, there are two key parameters (cant recall them now), computed during the harmonization process, that you could export and re-use in new data.
However, I would not recommend to do so, since these algorithms are very sample-dependent. I have tried to harmonize the who sample or just subsets of patients, obtaining different results. In my case, I decided to harmonize my data in an analysis-specifc case.

#4

Hi Williem,

Could you please share with us what you did to apply Combat in a cross-validation?

Cheers,
Walter

#5

I’ve found that training combat on the training set and applying it to the test set can actually induce site effects in the test set. I’ve had the most luck applying it separately to the training and test sets. Assuming that your data is sufficiently large, that’s what I’d do.

Here’s some example code where I do that:

#6

@WillemB2104 neuroCombat seems to be abandoned, do you have any other channels for contacting the author and seeing if they’d be interested in reviving the project?

@slieped would you mind linking to your implementation? Did you manage to implement the non-parametric approach as well as the parametric one?

#7

Hi there, sorry for the delay. I found the neuroCombat author’s email address through his github repo and will ask around there. Our colleagues from neurofedora have contacted him this week as well and they had the same questions regarding the project status. Might be worth to follow as well, see here: https://github.com/ncullen93/neuroCombat/issues/3

Will keep you posted!