Performance Evaluation Metrics used in Training, Validation and Testing

2

2

Which specific performance evaluation metrics are used in training, validation and testing and why? I am thinking error metrics (RMSE, MAE, MSE) are used in validation, and testing should use a wide variety of metrics? I don't think performance is evaluated in training, but not 100% sure.

Specifically I am actually after deciding when to use (i.e. in training, validation or testing) correlation coefficient, RMSE, MAE and others for numeric data (e.g. Willmott's Index of Agreement, Nash-Sutcliffe coefficient, etc.)

Sorry about this being broad - I have actually been asked to define it generally (i.e. not for a specific dataset). But datasets I have been using have all numeric continuous values with supervised learning situations.

Generally I am using performance evaluation for environmental data where I am using ANN to model. I have continuous features and am predicting a continuous variable.

user9645302

Posted 2018-04-14T11:58:29.497

Reputation: 53

Welcome to ai.se...it indeed is too broad a question...choice of evaluation metrics solely depend on the problem and requirements at hand..also it depends whether you are using it to train the network...F-score is a measurement metric not used to train network..so I suggest you to narrow down the scope by editing the question a little bit and giving it some context – DuttaA – 2018-04-14T14:01:04.193

Thanks for that. Sorry about that - I will try to explain it more. I am actually after deciding when to use specifically correlation coefficient, RMSE, MAE and others for numeric data (e.g. Willmott's Index of Agreement, Nash-Sutcliffe coefficient, etc.) – user9645302 – 2018-04-14T14:04:47.123

Since I actually need to explore all ways performance evaluation is used in training, validation and testing, I would be happy to know how it used to train the network. Thank you – user9645302 – 2018-04-14T14:11:38.543

Actually there is so many number of optimizations you can't possibly track them all...like i said any1 can invent a new metric according to his choice...so it is quite difficult to answer your question unless you provide some more info..also you are asking some very mathematical questions whose details might not fit in a single answer – DuttaA – 2018-04-15T09:32:16.770

@DuttaA Thank you for your reply and sorry that this is not clear. Actually in my course I am doing it is all about using ANN to model data. The question I have been asked to answer is about discussing which stages out of training, validation and testing should the performance metrics correlation coefficient, RMSE, MAE, Willmott's Index, Nash-Sutcliffe coefficient, Legate and McCabe's Index and Percentage Peak Deviation be used in? We haven't been given any more information or data. I think what I would like to know is probably when do we use error metrics and when to use non-error metrics? – user9645302 – 2018-04-15T11:56:42.330

No answers