Improving Performance of LSTM for time series prediction

1

My data consists of two features and a set of time series data labeled as "bookings". I have 1056 data point in the times series, for which I have two features for each. The data size is 1056x3.

My aim is to: given previous 21 data points and two features, predict the coming 7 data points.

I have prepared my data accordingly to make it suitable for LSTM and then trained my network.

My code is below:

import numpy as np
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
data= pd.read_excel("data.xlsx", 'Sheet3')

n_in=28
n_out=7


for obs in range(1,n_in):
    data["T_" + str(obs)] = data.bookings.shift(obs)
data.fillna(0.00,inplace=True)

X=data
Y=data.get(["bookings"])
for obs in range(1,n_out+1):
    Y["T+" + str(obs)] = data.bookings.shift(-obs)
Y.fillna(0.00,inplace=True)
Y=Y.drop(["bookings"], axis=1)

X=X.values
Y=Y.values


data = np.concatenate((X,Y),axis=1)

Train=data[:740,:] 
Test=data[740:,:]

scaler = MinMaxScaler()
Train = scaler.fit_transform(Train)
Test = scaler.transform(Test)

Xtrain,Xtest=Train[:,:n_in+2], Test[:,:n_in+2]
Ytrain, Ytest=Train[:,-n_out:] , Test[:,-n_out:]


Xtrain = np.array(Xtrain).reshape(Xtrain.shape[0], 1, n_in+2)
Ytrain = np.array(Ytrain).reshape(Ytrain.shape[0], 1, n_out)
Xtest = np.array(Xtest).reshape(Xtest.shape[0], 1, n_in+2)
Ytest = np.array(Ytest).reshape(Ytest.shape[0], 1, n_out)

Then I create the LSTM Network and train it:

from keras.layers import RepeatVector
from keras.layers import TimeDistributed
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM,Bidirectional
model = Sequential()

model.add(Bidirectional(LSTM(80, activation= 'relu', input_shape=(1, n_in+2))))
model.add(RepeatVector(1))
model.add(Bidirectional(LSTM(80, activation='relu', return_sequences=True)))
model.add(TimeDistributed(Dense(n_out)))
model.compile(optimizer='adam', loss='mse')

history = model.fit(Xtrain, Ytrain, epochs=25, validation_split=0.2, verbose=1, batch_size=2)

Verbose of the training:

Train on 592 samples, validate on 148 samples
Epoch 1/25
592/592 [==============================] - 24s 41ms/step - loss: 0.0327 - val_loss: 0.0146
Epoch 2/25
592/592 [==============================] - 4s 7ms/step - loss: 0.0115 - val_loss: 0.0119
Epoch 3/25
592/592 [==============================] - 4s 6ms/step - loss: 0.0094 - val_loss: 0.0115
Epoch 4/25
592/592 [==============================] - 4s 7ms/step - loss: 0.0084 - val_loss: 0.0108
Epoch 5/25
592/592 [==============================] - 4s 7ms/step - loss: 0.0077 - val_loss: 0.0101
Epoch 6/25
592/592 [==============================] - 4s 7ms/step - loss: 0.0073 - val_loss: 0.0104
Epoch 7/25
592/592 [==============================] - 4s 7ms/step - loss: 0.0070 - val_loss: 0.0119
Epoch 8/25
592/592 [==============================] - 4s 7ms/step - loss: 0.0070 - val_loss: 0.0097
Epoch 9/25
592/592 [==============================] - 4s 7ms/step - loss: 0.0067 - val_loss: 0.0099
Epoch 10/25
592/592 [==============================] - 4s 6ms/step - loss: 0.0066 - val_loss: 0.0097
Epoch 11/25
592/592 [==============================] - 4s 7ms/step - loss: 0.0067 - val_loss: 0.0097
Epoch 12/25
592/592 [==============================] - 4s 7ms/step - loss: 0.0064 - val_loss: 0.0095
Epoch 13/25
592/592 [==============================] - 4s 7ms/step - loss: 0.0063 - val_loss: 0.0100
Epoch 14/25
592/592 [==============================] - 4s 7ms/step - loss: 0.0062 - val_loss: 0.0095
Epoch 15/25
592/592 [==============================] - 4s 7ms/step - loss: 0.0060 - val_loss: 0.0092
Epoch 16/25
592/592 [==============================] - 4s 7ms/step - loss: 0.0059 - val_loss: 0.0098
Epoch 17/25
592/592 [==============================] - 4s 7ms/step - loss: 0.0059 - val_loss: 0.0098
Epoch 18/25
592/592 [==============================] - 4s 7ms/step - loss: 0.0058 - val_loss: 0.0093
Epoch 19/25
592/592 [==============================] - 4s 7ms/step - loss: 0.0060 - val_loss: 0.0091
Epoch 20/25
592/592 [==============================] - 4s 7ms/step - loss: 0.0058 - val_loss: 0.0095
Epoch 21/25
592/592 [==============================] - 4s 7ms/step - loss: 0.0056 - val_loss: 0.0091
Epoch 22/25
592/592 [==============================] - 4s 6ms/step - loss: 0.0056 - val_loss: 0.0093
Epoch 23/25
592/592 [==============================] - 4s 7ms/step - loss: 0.0056 - val_loss: 0.0091
Epoch 24/25
592/592 [==============================] - 4s 7ms/step - loss: 0.0057 - val_loss: 0.0091
Epoch 25/25
592/592 [==============================] - 4s 7ms/step - loss: 0.0055 - val_loss: 0.0095

The loss gets stuck there and doesn't reduce with increasing the number of epochs.

Then I use the trained network to predict and test the performance. I scale back and reshape as well:

from sklearn.metrics import mean_squared_error

yhat = model.predict(Xtest, verbose=0)
yhat=np.array(yhat).reshape(yhat.shape[0],yhat.shape[2])
Ytest=np.array(Ytest).reshape(Ytest.shape[0],Ytest.shape[2])
Xtest = np.array(Xtest).reshape(Xtest.shape[0],Xtest.shape[2])

temp=np.concatenate((Xtest,yhat),axis=1)
yhat= scaler.inverse_transform(temp)
yhat=yhat[:,-n_out:]

temp=np.concatenate((Xtest,Ytest),axis=1)
Ytest= scaler.inverse_transform(temp)
Ytest=Ytest[:,-n_out:]

rmse = np.sqrt(mean_squared_error(Ytest, yhat))
print('Test RMSE: %.3f' % rmse)

Some examples of my output, where the actual is in blue and prediction is in orange. enter image description here enter image description here enter image description here

My questions are:

  1. Is there any flaw in the steps above?

  2. Any idea how can I improve the performance of the network?

HaneenSu

Posted 2019-11-26T09:57:12.800

Reputation: 55

No answers