How to improve accuracy of deep neural networks


I am using Tensorflow to predict whether the given sentence is positive and negative. I have take 5000 samples of positive sentences and 5000 samples of negative sentences. 90% of the data I used it for training the neural network and rest 10% for testing.

Below is the parameter initialisation. Batch size = 100 Epochs = 30 Number of hidden layers = 3 Nodes in each hidden layer = 100 epochs = 30

I could see in each epoch the cost function is getting reduced reasonably. However the accuracy of the model on test set is poor (only 56%)

 Epoch  1 completed out of  30 loss :  22611.10902404785
 Epoch  2 completed out of  30 loss :  12377.467597961426
 Epoch  3 completed out of  30 loss :  8659.753067016602
 Epoch  4 completed out of  30 loss :  6678.618850708008
 Epoch  5 completed out of  30 loss :  5391.995906829834
 Epoch  6 completed out of  30 loss :  4476.406986236572
 Epoch  7 completed out of  30 loss :  3776.497922897339
 Epoch  25 completed out of  30 loss :  478.93606185913086
 Epoch  26 completed out of  30 loss :  450.8017848730087
 Epoch  27 completed out of  30 loss :  435.0913710594177
 Epoch  28 completed out of  30 loss :  452.10553523898125
 Epoch  29 completed out of  30 loss :  539.5199084281921
 Epoch  30 completed out of  30 loss :  685.9198244810104
 Accuracy of Train : 0.88155556
 Accuracy of Test : 0.524

Is there any parameter that can be tuned to increase the accuracy of the model considering the same number of data set.


Posted 2018-05-05T10:38:48.163

Reputation: 1 271

How about to do regularization to simplify your fitting model? I believe it can improve your accuracy. – MinSama – 2018-05-05T10:46:15.897

@MinSama I will try to insert the regularization parameter and update here. Also I am looking for more samples of data as the below answers say. – deepguy – 2018-05-11T02:20:54.503



You need much more data.

Deep NN shines when you have excessive amounts of data.

With only a little bit if data it can easily overfit. The big difference between training and test performance shows that your network is overfitting badly. This is likely also because your network model has too much capacity (variables, nodes) compared to the amount of training data. A smaller network (fewer nodes) may overfit less.

Has QUIT--Anony-Mousse

Posted 2018-05-05T10:38:48.163

Reputation: 7 331

I tried to implement the same with 1 hidden layer of 5 nodes, but still I got accuracy of 1 for both training and test. Have been doing lot of debug, wondering why 100% accuracy.! – deepguy – 2018-05-06T08:30:13.543

any suggestions ? – deepguy – 2018-05-10T19:49:47.967

Get much more data. 5000 is just way too little for text. – Has QUIT--Anony-Mousse – 2018-05-10T20:21:16.507


For increasng your accuracy the simplest thing to do in tensorflow is using Dropout technique. Try to use tf.nn.dropout. between your hidden layers. Do not use it for your first and last layers. For applying that, you can take a look at How to apply Drop Out in Tensorflow to improve the accuracy of neural network.


Posted 2018-05-05T10:38:48.163

Reputation: 12 077

1Thanks for the suggestion and link. I will try this method and update you. – deepguy – 2018-05-05T15:47:56.033


Whatever regularization technique you're using, if you keep training long enough, you will eventually overfit the training data, you need to keep track of the validation loss each epoch. at least use early stopping to stop the training process when the validation loss stops decreasing.

Fadi Bakoura

Posted 2018-05-05T10:38:48.163

Reputation: 848

Thanks for the reply. I could see at epoch 27 loss has reached the minimum and started raising again. I think it needs to be stopped at epoch 27 right? – deepguy – 2018-05-11T02:23:01.827

The loss on the log is it from validation or training? if it's from training you have to further split the data into validation set. – Fadi Bakoura – 2018-05-11T03:31:01.080

The loss mentioned in the log is from the training data set generated in each epoch while training. – deepguy – 2018-05-11T04:06:09.037

When the training loss increases, it means the model has a divergence caused by a large learning rate. the thing is, when doing SGD, we are estimating the gradient. therefore when a noisy update is repeated (training too many epochs) the weights will be in a bad position far from any good local minimum. and the non-linearity activation functions are saturated. my suggestion to you is to read this paper:

– Fadi Bakoura – 2018-05-11T08:12:08.733


This is simple case of overfitting.

To improve accuracy, I would suggest to do the following changes:

Since your 'x' variable are sentences, you can try to use Sequential model with one Embedding Layer and one LSTM layer:

from tensorflow.keras.layers import Dense, Embedding, LSTM
from tensorflow.keras.models import Sequential

model = Sequential()

model.add(Embedding(max_features, 32))

Add the last layer as Dense Layer, and the loss function as binary_crossentropy:

model.add(Dense(1, activation='sigmoid'))

model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'])

Fit the model and store the history variable to find the optimal epochs:

history =, y_train, epochs=20, batch_size=128, validation_split=0.2)

Plot the following 2 curves: i) Validation loss vs epochs ii) Validation accuracy vs epochs

# Plotting Results
import matplotlib.pyplot as plt

acc = history.history['accuracy']
val_acc = history.history['val_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']
epochs = range(1, len(acc) + 1)

plt.plot(epochs, acc, 'b', label='Training acc')
plt.plot(epochs, val_acc, 'g', label='Validation acc')

plt.title('Training and validation accuracy')
fig = plt.figure()

plt.plot(epochs, loss, 'b', label='Training loss')
plt.plot(epochs, val_loss, 'g', label='Validation loss')
plt.title('Training and validation loss')


After which you will get a graph something like these: enter image description here

enter image description here

For this specific case, we see that the optimal epochs is occurring at 12, So we need to again train with 12 epochs and test on test data., y_train, epochs=12, batch_size=128)
model.evaluate(x_test, y_test)

Saurabh Verma

Posted 2018-05-05T10:38:48.163

Reputation: 131