How to use the same minmaxscaler used on the training data with new data?

4

2

Im using the keras LSTM model to make prediction, and the code above is to scale the data: inputs are shaped like (n, 11, 1) and the label is 1D
DailyDemand.py

#scaling data
scaler_x = preprocessing.MinMaxScaler(feature_range =(-1, 1))
x = np.array(x).reshape ((len(x),11 ))
x = scaler_x.fit_transform(x)
scaler_y = preprocessing.MinMaxScaler(feature_range =(-1, 1))
y = np.array(y).reshape ((len(y), 1))
y = scaler_y.fit_transform(y)

# Split train and test data
x_train=x[0: train_end ,]
x_test=x[train_end +1: ,]
y_train=y[0: train_end]
y_test=y[train_end +1:] 
x_train=x_train.reshape(x_train.shape +(1,))
x_test=x_test.reshape(x_test.shape + (1,))
# Train and save the Model named fit1 in a json and h5 files 
[....]
# serialize model to JSON
model_json = fit1.to_json()
with open("model.json", "w") as json_file:
    json_file.write(model_json)
# serialize weights to HDF5
fit1.save_weights("model.h5")
print(">>>> Model saved to model.h5 in the disk")

and Now im trying to predict a new values of a new data with this trained model. so i loaded the model from the files:

predict.py

from DailyDemand import scaler_y
from DailyDemand import scaler_x
[...]
# load json and create model
json_file = open('model.json', 'r')
loaded_model_json = json_file.read()
json_file.close()
loaded_model = model_from_json(loaded_model_json)
# load weights into new model
loaded_model.load_weights("model.h5")
print("Loaded model from disk")

########################################
# make prediction with the loaded model

FeaturesTest = [267,61200,695,677,70600,116700,130200,768,659,741,419300]
xaa = np.array(FeaturesTest).reshape ((1,11 )).astype(float)
print(xaa)
xaa = scaler_x.fit_transform(xaa) 
xaa = xaa.reshape(xaa.shape +(1,))
print("print FeaturesTest scalled: ")
print(xaa) # incorrect scalled value, always returns -1 ones 
xaa = [[[-1.]
  [-1.]
  [-1.]
  [-1.]
  [-1.]
  [-1.]
  [-1.]
  [-1.]
  [-1.]
  [-1.]
  [-1.]]]
tomorrowDemand = loaded_model.predict(xaa)
print("tomorrowDemand scalled: ", tomorrowDemand)
prediction = scaler_y.inverse_transform(np.array(tomorrowDemand).reshape ((len(tomorrowDemand), 1))).astype(int)
print ("the real demand is 95900 and the prediction is: ", prediction)

The problem is how i can use the same scaler used in the training on the new data? i want to know if i made a mistake in this code to use the same scaller on the new data?

Nbenz

Posted 2018-04-25T11:42:10.643

Reputation: 263

Answers

5

You are refitting scaler_x on your test set, which you don't want. Change this line:

xaa = scaler_x.fit_transform(xaa)

to

xaa = scaler_x.transform(xaa)

You are getting [-1, -1, ..., -1] because with one sample, each feature is equal to the minimum.

Bert Kellerman

Posted 2018-04-25T11:42:10.643

Reputation: 220

1Thank you so much for your answer, it works very well, you have saved my day =D. – Nbenz – 2018-04-25T15:22:53.153

Awesome! I'm so glad to help. Can you click the check mark to accept the answer? – Bert Kellerman – 2018-04-25T16:20:43.217

Done, sorry i was busy with the problem and i forgot to check the answer mark :) – Nbenz – 2018-04-25T16:23:39.363