## 3D - CNN. Why my cost function decreases, but the accuracy does not increase?

2

I'm implementing a C3D-inspired neural network for human emotion recognition, the problem I'm facing is that altough the cost function is decreasing, for both training and validation sets, I do not appreciate any improvement in terms of accuracy, for neither of boths sets.

My cost function is the cross-entropy between the logits (output of the last layer) and the correct prediction

def tower_loss(name_scope, logit, labels):
xent = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logit,labels=labels)
cross_entropy_mean = tf.reduce_mean(xent)
return cross_entropy_mean


Then, the optimizer uses the ADAM algorithm for minimizing the cost function as follows

loss = tower_loss(scope, logit, labels_placeholder)


Although I'm seing the cost function decreasing, I haven't seen any improvement in the classification.