Is this a problem for a Seq2Seq model?

3

I'm struggling to find a tutorial/example which covers using an seq2seq model for sequential inputs other then text/translation.

I have a multivariate dataset with n number of input variables each composed of sequences, and a single output sequence which is unrelated to any of the input variables (e.g. using weekly wind speed and humidity to predict temperature).

I've converted the features and label to batches with fixed time steps and n dimensions, however I'm confused which model should be used. Ideally the output should be a single sequence (e.g. annual temperature) however which model would achieve this? Is this something an LSTM could achieve, or is this a problem for a seq2seq model?

Any suggestions/insightful would be appreciated.

Ellio

Posted 2018-08-19T19:45:13.700

Reputation: 53

Answers

0

The type of your problem is time series regression. The common way of dealing with such problems is using ARMA models and Kalman filters with input (control vector). Look at this links:

https://www.sas.upenn.edu/~fdiebold/Teaching104/Ch14_slides.pdf

https://en.m.wikipedia.org/wiki/Kalman_filter

For time series regression using LSTM, check Multivariate LSTM Forecast Model part from this link:

https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras

pythinker

Posted 2018-08-19T19:45:13.700

Reputation: 1 162

What about using a Sequential model in Keras? Would that also be suited to this problem? I've read that ARMA models might need a lot of tuning... – Ellio – 2018-08-20T15:55:52.143

@Elliot I updated my answer. Please check it. – pythinker – 2018-08-20T16:44:19.737

0

To use a seq2seq neural network for timeseries regression (and not forecasting, as everybody seems to be doing), a simple Keras model could be

inputs = Input(shape=(n_timesteps, n_features))
x = LSTM(n_units, return_sequences = True)(inputs)
outputs = Dense(1, activation="linear")(x)

where you output a single value in the final layer, with linear activation, so that no scaling is applied. Then a common choice is to minimize MSE with your favorite optimizer - usually RMSprop for timeseries.

Then, if you add convolutions and pooling layers, be sure to set padding = "same", to maintain the sequence length throughout the layers, or fix downsampled timesteps with RepeatVector to match the input sequence length. Specific choice of layers may be problem-dependent.

komodovaran_

Posted 2018-08-19T19:45:13.700

Reputation: 143