Training Encoder-Decoder using Decoder Outputs

2

1

I am trying to build an encoder-decoder model for a text style transfer problem. The problem is I don't have parallel data between the two styles so I need to train the model in an unsupervised setting.

Some papers I have seen use an auto-encoder to train the encoder and decoder components separately. By setting the problem as an auto-encoder, they can train the decoder by passing the target sequence (equal to the input sequence) into the decoder. (Here are some examples, https://arxiv.org/pdf/1711.06861.pdf, https://arxiv.org/pdf/1804.04003.pdf)

Instead of an auto-encoder, I would like to know if it's possible to train a decoder by feeding its predictions at time, t-1, into the input at time-step t. I would pass the generated output into a classifier to check the style and to obtain a training signal. Is this sensible and what are the pros / cons of doing so? Thanks.

Physbox

Posted 2018-07-07T14:20:51.903

Reputation: 207

Answers

2

I would like to know if it's possible to train a decoder by feeding its predictions at time, t-1, into the input at time-step t.

Yes, it is possible to do it.

But I don't see why you would do it.

Υou will have accumulated error propagated and amplified in every new prediction, making your prediction to diverge from the ground truth sooner or later.

pcko1

Posted 2018-07-07T14:20:51.903

Reputation: 3 275

Thanks for your answer. I understand your point about why the error would be propagated. The reason I am considering it is because I don't have a ground truth to input to the decoder. – Physbox – 2018-07-08T22:39:48.800

it is like an open-loop control system :) – pcko1 – 2018-07-08T22:41:54.557