Recurrent neural network producing same predictions



I am trying to train a recurrent neural network that I built in keras on timeseries data to predict number of sales for next 10 days. For this, I've created my dataset as -

var(t) -> var(t+1)
var(t+1) -> var(t+2)
var(t+2) -> var(t+3)
var(t+3) -> var(t+4) and so on 

I did Min-Max scaling on this data and the RNN code is as follows -

model = Sequential()
model.add(LSTM(20, input_shape=(1, look_back),activation='tanh',bias_initializer='ones'))
model.add(Dense(1, activation='linear',bias_initializer='ones'))
model.compile(loss='mean_squared_error',optimizer=opt), ytrain, epochs=100, batch_size=1, verbose=2)

But the plot I am getting is when I did predictions on xtrain (green = ytrain, blue = ypred) -

Observed vs expected signals

The rnn isn't learning anything at all. Its producing same results for each dataset. I've tried adding hidden layers, increasing number of neurons, changing parameters (learning rate, momentum), optimizers (sgd, adam, adagrad, rmsprop), lstm activation fxn (tanh, softsign). I got little fluctuations in some cases in the graph. But the output is mostly constant. Also, I've only 200 datasets.

Can someone please guide me what I am doing wrong here. What else I can try. Will small sized data not work using RNN at all ? If so, is there any other way to solve this problem (except ARIMA model) ?

EDIT - Increased batch size to 100 and epochs to 1000. Received some better results. Also, I did mean normalization [(x-mean)/std_dev] instead of MinMax scaling.

enter image description here


Posted 2017-07-14T19:53:45.710

Reputation: 153

input shape is suspicious. what value as look_back? did you import Adam optimizer as such keras.optimizers.Adam as adam? learning rate of 0.1 is quite high. batch_size of 1 looks small. – tagoma – 2017-07-14T20:04:54.187

@edouard look_back is 1. yes, adam optimizer is from keras.optimizers.adam I've tried learning rate values - 0.01, 0.001, 0.03, 0.3 - none worked. – Ishan – 2017-07-14T20:12:52.923

@edouard I've edited the question with some changes that you mentioned. Decreasing learning rate to 0.01 didn't help. – Ishan – 2017-07-14T20:27:24.843



Try increasing your batch size. If your batch size is small, the gradients are a rough approximation of the true gradients.

Yash Katariya

Posted 2017-07-14T19:53:45.710

Reputation: 421