## Does the output of the Sequence-to-Sequence encoder model exist in the same semantic space as the inputs (Word2vec)?

2

Does the output generated from the LSTM encoder module exist in the same semantic space as the original word vectors? If so, say for example we have a sentence and we pass it through the encoder to get an encoded output and then we also calculate the average of word vectors for the same sentence separately, will the two new vectors (encoded and average) be comparable? Will their euclidean distance be relatively small?

Question was closed 2020-10-20T20:26:16.020

No, assuming your input vectors are one-hot encodings. These input one-hot encodings are in an $$n$$-dimensional Euclidean vector space. The last hidden layer of an LSTM is not due to the non-linear activation functions across the encoder. Therefore, an average of the inputs will not necessarily align well in a vector space with the model output, nor are you guaranteed any similarity in cosine/Euclidean distance.