How to interpret my neural network with high accuracy but low probability in test results


I have built a classical ANN using keras which provides probability (using sigmoid function) of the outcomes (0 or 1). While the accuracy of the model is high when the model is fit ~90%, the outcome probability of the test set results is very poor. How can I interpret this?

Build ANN

classifier = Sequential()
classifier.add(Dense(activation="relu",input_dim=7,kernel_initializer="uniform", units = 4))
classifier.add(Dense(activation="relu",kernel_initializer="uniform", units = 4))
classifier.add(Dense(activation="sigmoid", kernel_initializer="uniform", units = 1))
classifier.compile(optimizer="adam", loss="binary_crossentropy",metrics=['accuracy']),y_train, batch_size=10,epochs=100)

Predict results:

y_pred = classifier.predict(X_test)

I have attached the test set results where result probabilities can be seen with respect to frequency. blue shows probabilities for 1 and orange for 0 test results

Is this a case of overfitting? How can I tweak an ann to avoid overfitting?

Ankur Bansal

Posted 2018-09-14T08:15:52.753

Reputation: 33



First possibility:

How well is your dataset balanced? Are there mostly 0s? If I understand your graph well, it would seem that is the case.

If you in fact have 90% 0 and 10% 1 in your dataset, then your results would indicate the model is actually just performing random selection. This means it hasn't really learny anything about the data.

Second possibility:

If your training accuracy is high, but your test accuracy is low, this generally does point to overfitting, as you suspect.

Your model is essentially memorising the training data set, making it able to simply recall the correct result quite reliably, but without learning the underlying function that describes the dataset.

Here are a few things that might be behind this:

  • training for too many epochs
  • using a model that too big and complex
  • not using enough regularisation

I would recommend first checking the class balances of your dataset. If it highly imbalanced, you could look into performing stratified sampling to create your train/test split. This ensures that each batch has roughly the same number of 0 and 1 samples.

Secondly, you might consider using a slightly larger network. In the first instance, this could be as simple as increasing the number of neurons in each of your Dense layers. I would suggest something like (100, 50, 1) to begin with. That will immediately tell you if the size of the model is restricting learning.


Posted 2018-09-14T08:15:52.753

Reputation: 12 573

Thank you for the detailed explanation. You have correctly identified that the 0 and 1 dataset are not balanced and are in the ratio of 80%-20%. I will try out the suggestions you have made. – Ankur Bansal – 2018-09-14T09:31:15.897

I tried balancing the dataset for 0 and 1. This has resulted in much better test results and seems like the it is trained on the data. I will keep on tweaking it using other methods, but this is great. A big thank you @n1k31t4 ! – Ankur Bansal – 2018-09-14T09:52:10.810

@AnkurBansal - you're welcome! :-) – n1k31t4 – 2018-09-14T10:17:52.757