I need to report accuracies of my neural model in a conference paper as compared to various baselines. What are the accepted standards for reporting accuracies in a fair manner?
Neural Model: To be specific, I'm using 60% as train set, 20% as validation set and 20% as test set to report the accuracy of my neural model.
- Should I take an average or highest of 3 runs accuracy where in each run I randomly sample 60% as training data from the total 80% train + validation set.
- My neural model is computationally intensive and therefore it is not feasible to perform a k-fold cross validation. Will my accuracy results be accepted by the academic community without a k-fold cross validation? Since my data set is large, I assume using 20% of it used solely for testing should be a fair indicator of accuracy.
How do I report the accuracy for this model so as to perform a fair comparison?
Should I train BOW on only 60% of data (same as which my neural model is being trained on) or should I train BOW on 80% of data (train + validation for my neural model)? Which is the accepted way? I then test BOW on the same remaining 20% of data (as in neural model) in either of the above case.
- The other approach is to perform k-fold cross validation but the test set will not be the same 20% as the test set on which my neural model is being evaluated. Is this approach recommended though?
Any other information on how to report accuracy results in a research paper comparing neural models (train, val, test) with linear models-BOW,SVM (train,test) is welcome. Please help.