Is $R^2$ an appropriate evaluation metric for k-Nearest Neighbors?

3

1

I found a source that stated that $R^2$ is the ”percentage of the response variable variation that is explained by a linear model.” (Source)

Since kNN is not a linear model (it is nonparametric), is it appropriate to use $R^2$ to evaluate it? If not, what error metric would be appropriate? I'm using the kNN classifier provided by scikit-learn in python.

covfefe

Posted 2015-12-09T04:10:02.643

Reputation: 283

Answers

2

$R^{2}$ is a cursory measure of goodness of fit of a linear model to the data and it is used in regression analysis. It appears to be a popular choice when deciding between linear and non-linear regression models.

It seems you intend to use kNN for classification, which has different evaluation metrics than regression. Scikit-learn provides 'accuracy', 'true-positive', 'false-positive', etc (TP,FP,TN,FN), 'precision', 'recall', 'F1 score', etc. for evaluating performance of a classifier.

bonus: kNN are simple classifiers that are used when data shows local structure but not global structure, i.e. data with same label are found in multiple localized clusters in the data space. The key parameter to tune is 'k', which should be based on the size of these localized clusters as you would stand to achieve considerably different results with different values of 'k'. You can do a grid-search (also in scikit-learn) to tune your parameters (including 'k' of course).

evaluation metric: accuracy or ROC or mean Average Precision or F1-score: There are several metrics to choose from and the choice is based on the task you are performing and what your priorities are. (you can look up these metrics for more details) but you can begin with accuracy for starters.

Dynamic Stardust

Posted 2015-12-09T04:10:02.643

Reputation: 1 163

Great answer! I would also add to your "bonus" section that if you are going to use an analogue based classification method, your best results will come from a support vector machine. kNN is more of a historical vestige since SVM's are much more powerful and tunable. If you really understand how to cross validate, you can tune your SVM very quickly to provide excellent results. – AN6U5 – 2015-12-20T07:34:19.187

Cheers AN6U5. Yes, of course, thanks for adding that. You might be interested in this paper which discusses the benefits of kNN in the age of SVMs: http://www.wisdom.weizmann.ac.il/~irani/PAPERS/InDefenceOfNN_CVPR08.pdf

– Dynamic Stardust – 2015-12-21T00:15:09.423

If I was doing kNN regression, would $R^2$ metric be appropriate? – covfefe – 2015-12-22T01:44:17.383

short answer is 'Yes'. I did a bit of reading on nonparametric regression and evaluation, and so far the list of evaluation metrics is MSE, RMSE, MAE, and $R^2$. https://spark.apache.org/docs/latest/mllib-evaluation-metrics.html#regression-model-evaluation and https://www.kaggle.com/wiki/Metrics . Have a look at http://www.stat.cmu.edu/~larry/=sml/nonpar.pdf . Conclusions: If $R^2$ is stable, i.e. low variance for multiple random sub-samples of data, then I'd say that is good enough. (full disclosure: at this point I am still not convinced and I'll update this thread when I learn more.)

– Dynamic Stardust – 2015-12-22T16:54:34.980

0

$R^2$ is for data with a numerical range. Classification by k Nearest Neighbours assigns class labels that are just labels (even if you choose them to be numbers, they aren't like real numbers).

You use kNN in a supervised setting, typical quality assessment consists in splitting up your data in training and test sets (n-fold cross validation) and determining precision, recall, and F-score.

user10169

Posted 2015-12-09T04:10:02.643

Reputation: