Keras autoencoder not converging

5

1

Could someone please explain to me why the autoencoder is not converging? To me the results of the two networks below should be the same. However, the autoencoder below is not converging, whereas, the network beneath it is.

autoencoder implementation, does not converge

autoencoder = Sequential()
encoder = containers.Sequential([Dense(32,16,activation='tanh')]) 
decoder = containers.Sequential([Dense(16,32)])
autoencoder.add(AutoEncoder(encoder=encoder, decoder=decoder, 
                        output_reconstruction=True))
rms = RMSprop()
autoencoder.compile(loss='mean_squared_error', optimizer=rms)

autoencoder.fit(trainData,trainData, nb_epoch=20, batch_size=64,
            validation_data=(testData, testData), show_accuracy=False)

non-autoencoder implementation, converges

model = Sequential()
model.add(Dense(32,16,activation='tanh')) 
model.add(Dense(16,32))
model.compile(loss='mean_squared_error', optimizer=rms)

model.fit(trainData,trainData, nb_epoch=numEpochs, batch_size=batch_size,
            validation_data=(testData, testData), show_accuracy=False)

Fritz O'Connor

Posted 2017-10-13T00:02:32.203

Reputation: 95

Answers

3

The new version (0.3.0) of Keras no longer has tied weights in AutoEncoder, and it still shows different convergence. This is because weights are initialized differently.

In the non-AE example, Dense(32,16) weights are initialized first, followed by Dense(16,32). In the AE example, Dense(32,16) weights are initialized first, followed by Dense(16,32), and then when you create the AutoEncoder instance, Dense(32,16) weights are initialized again (self.encoder.set_previous(node) will call build() to initialize weights).

Now the following two NNs converge exactly the same:

autoencoder = Sequential()
encoder = containers.Sequential([Dense(32,16,activation='tanh')]) 
decoder = containers.Sequential([Dense(16,32)])
autoencoder.add(AutoEncoder(encoder=encoder, decoder=decoder, 
                        output_reconstruction=True))
rms = RMSprop()
autoencoder.compile(loss='mean_squared_error', optimizer=rms)
np.random.seed(0)
autoencoder.fit(trainData,trainData, nb_epoch=20, batch_size=64,
            validation_data=(testData, testData), show_accuracy=False)

Alejandro Rodriguez

Posted 2017-10-13T00:02:32.203

Reputation: 66

Could you please explain why this works? – aneesh joshi – 2017-10-18T05:45:10.377