I’d like to perform RSA comparing the neural RDM with a behavioral RDM built from individual ratings, where each subject in our fMRI study rated each stimulus for a certain attribute on a scale from 1 to 6. Now, I’m wondering what the best way is to compute the pairwise (dis)similarity of these stimulus-ratings for RSA. Can one simply take the difference between two ratings? Or should the ratings be z-scored before taking the difference? Or should one not use the rating-values at all but take the difference of the ranks of the ratings (cf. https://www.sciencedirect.com/science/article/pii/S1053811916304591)?

I think it’s not entirely clear how a rating would translate to dissimilarities in the mental representation. I think first ranking the results and then calculating the difference is a little weird and intuitively doesn’t make much sense, but of course you could argue for it. For example, if you have values of 1, 2, 3, and 7, then the rank difference between 1 and 2 is the same as between 3 and 7. Therefore, by first ranking results, the difference would be treated as the same. I don’t think this makes sense. I would think that the difference between 3 and 7 is at least bigger than the difference between 1 and 2. But if you think they should be treated the same, then you can rank them. If you want to account for different use of the scale, I would rather try using a log-transform.

What I would do is calculate the Spearman rank correlation coefficient when comparing dissimilarity matrices. This is equivalent to a Pearson correlation on ranked dissimilarities. By ranking dissimilarities, you de-emphasize large absolute differences that may be related to just different use of the rating scale (i.e. you don’t think that the difference between 3 and 7 is 4x as large as the difference between 1 and 2, but you think that at least it is larger than it). Hope this makes sense.

Thanks a lot, Martin!
You make a very good case that ranking the ratings makes little sense - I completely agree.

A few follow-up questions:

If I want to standardize individual subjects’ ratings (“account for different use of the scale”), isn’t a z-transform more appropriate than a log-transform?

Would you simply take the difference between the transformed rating-values to create the RDM?

The connection between your two ideas is unclear to me. Your idea about log-transformation is about first-order similarity (i.e. creating the ratings RDM), whereas your idea about Spearman rank correlation is about second-order similarity (i.e. comparing the ratings RDM to the neural RDM), correct? If so, would you compute the difference of log-transformed ratings to create the RDM and compare the ratings and neural RDMs using Spearman rank correlation, or are these ideas completely independent/unrelated?

Assuming you want to interpret results at ratio scale (which i guess you don’t), I think you want to bring the data into a format that makes intuitive sense and that makes differences in use of the scale less extreme. If you assume that the histogram of responses is not natural but biased, and you assume a normal distribution to be more natural, then a log-transform often helps. You can still z-transform after that to unify the mean and std between different scales, but that all comes with assumptions.