Can a neural network whose output is uniformly equal to zero learn its way out of it?


I am performing a regression task on sparse images. The images are a result of a physical process with meaningful parameters (actually, they are a superposition of cone-like shapes), and I am trying to train a regressor for these parameters.

Here, sparse images mean that the data, and thus the expected output, is made of 2D square tensors, with only one channel, and that it is expectd that roughly 90% of the image is equal to zero. However, in my system, the data is represented as dense tensors.

I built a neural network with an encoder mapping the image on an output for which I have chosen activation and shape such that it corresponds with those meaningful parameters.

I then use custom layers to build an image from these parameters in a way that matches closely the physical process, and train the network by using the L2 distance between the input image and the output image.

However, for a large set of parameters, the output image will be equal to zero, since these are sparse images. This is the case in general for the initial network.

Is it possible that, through training, the neural network will learn its way out of this all-zero parameterization ?

My intuition is that, in the beginning, the loss will be equal to the L2 norm of the input image, and the gradient will be uniformly zero, hence, no learning.

Can anyone confirm ?


Posted 2020-04-14T11:52:33.677

Reputation: 21

Hi Alex and welcome to AI SE! Can you clarify what you mean by "sparse images"? – nbro – 2020-04-14T12:44:55.970

Just edited the question. Thank you by the way :-) – Alex – 2020-04-14T12:58:42.213

So, are you using some kind of autoencoder? – nbro – 2020-04-14T16:59:23.690

Yes, except for the fact that the decoder is deterministic. – Alex – 2020-04-14T17:31:26.817

No answers