$R^{2}$ is a cursory measure of goodness of fit of a linear model to the data and it is used in regression analysis. It appears to be a popular choice when deciding between linear and non-linear regression models.

It seems you intend to use kNN for classification, which has different evaluation metrics than regression. Scikit-learn provides 'accuracy', 'true-positive', 'false-positive', etc (TP,FP,TN,FN), 'precision', 'recall', 'F1 score', etc. for evaluating performance of a classifier.

bonus: kNN are simple classifiers that are used when data shows local structure but not global structure, i.e. data with same label are found in multiple localized clusters in the data space. The key parameter to tune is 'k', which should be based on the size of these localized clusters as you would stand to achieve considerably different results with different values of 'k'. You can do a grid-search (also in scikit-learn) to tune your parameters (including 'k' of course).

evaluation metric: accuracy or ROC or mean Average Precision or F1-score: There are several metrics to choose from and the choice is based on the task you are performing and what your priorities are. (you can look up these metrics for more details) but you can begin with accuracy for starters.

Great answer! I would also add to your "bonus" section that if you are going to use an analogue based classification method, your best results will come from a support vector machine. kNN is more of a historical vestige since SVM's are much more powerful and tunable. If you really understand how to cross validate, you can tune your SVM very quickly to provide excellent results. – AN6U5 – 2015-12-20T07:34:19.187

Cheers AN6U5. Yes, of course, thanks for adding that. You might be interested in this paper which discusses the benefits of kNN in the age of SVMs: http://www.wisdom.weizmann.ac.il/~irani/PAPERS/InDefenceOfNN_CVPR08.pdf

– Dynamic Stardust – 2015-12-21T00:15:09.423If I was doing kNN regression, would $R^2$ metric be appropriate? – covfefe – 2015-12-22T01:44:17.383

short answer is 'Yes'. I did a bit of reading on nonparametric regression and evaluation, and so far the list of evaluation metrics is MSE, RMSE, MAE, and $R^2$. https://spark.apache.org/docs/latest/mllib-evaluation-metrics.html#regression-model-evaluation and https://www.kaggle.com/wiki/Metrics . Have a look at http://www.stat.cmu.edu/~larry/=sml/nonpar.pdf . Conclusions: If $R^2$ is stable, i.e. low variance for multiple random sub-samples of data, then I'd say that is good enough. (full disclosure: at this point I am still not convinced and I'll update this thread when I learn more.)

– Dynamic Stardust – 2015-12-22T16:54:34.980