Hi, recently, i met a question that how to evaluate the classification performance on SpaceNet or FREM evaluator. I have read carefully about your manuals in webside, but havn’t found my answer.
I know we can transform the 4D files into 2D by using masker.fit_transform function, and after that, f1 score, auc score and p-value of permutation test score could be easily gotten. But it seems that the above way can only work for the model imported from scikit-learn, like svc.
For the spacial model which imported from nilearn.decoding, like SpaceNet or FREM. The only score we can get is cv_score, which is far away from meeting our article.
Thus, i want to know, how to compute f1 score, auc score and p-value of permutation test score in SpaceNet model or FREMmodel? If possible, please show the code.
Hope for your reply.
Yours sincerely,
clancy
Hello, a possibility is always to just generate the predictions from the model and to score them separately with any function you’d like. Here you have to be careful on :
- the data you use
- how you split data between train and test
- how many split you perform
If we take the example of the f1_score, and FREMClassifier the pseudo code would be:
model = FREMClassifier()
model.fit(X_train, X_train_labels)
X_test_pred = model.transform(X_test)
sklearn.metrics.f1_score(*X_test_labels* , *X_test_pred*)
Specifically to FREM(), you can also change the scoring parameter to change the metric used to evaluate each model and the final model cv_scores_. It can be set to ‘accuracy’, ‘f1’, ‘precision’, ‘recall’ or ‘roc_auc’.