Train, Validation and Test Split for Reporting Accuracy of Neural Model and BOW


I need to report accuracies of my neural model in a conference paper as compared to various baselines. What are the accepted standards for reporting accuracies in a fair manner?

Neural Model: To be specific, I'm using 60% as train set, 20% as validation set and 20% as test set to report the accuracy of my neural model.

  1. Should I take an average or highest of 3 runs accuracy where in each run I randomly sample 60% as training data from the total 80% train + validation set.
  2. My neural model is computationally intensive and therefore it is not feasible to perform a k-fold cross validation. Will my accuracy results be accepted by the academic community without a k-fold cross validation? Since my data set is large, I assume using 20% of it used solely for testing should be a fair indicator of accuracy.

Bag-of-words (BOW):

  1. How do I report the accuracy for this model so as to perform a fair comparison?

  2. Should I train BOW on only 60% of data (same as which my neural model is being trained on) or should I train BOW on 80% of data (train + validation for my neural model)? Which is the accepted way? I then test BOW on the same remaining 20% of data (as in neural model) in either of the above case.

  3. The other approach is to perform k-fold cross validation but the test set will not be the same 20% as the test set on which my neural model is being evaluated. Is this approach recommended though?

Any other information on how to report accuracy results in a research paper comparing neural models (train, val, test) with linear models-BOW,SVM (train,test) is welcome. Please help.


Posted 2017-06-26T10:41:15.883

Reputation: 29

No answers