23

10

I'm working through a question from the online book.

I can understand that if the additional output layer is of 5 output neurons, I could probably set bias at 0.5 and weight of 0.5 each for the previous layer. But the question now ask for a new layer of four output neurons - which is more than enough to represent 10 possible outputs at $2^{4}$.

Can someone walk me through the steps involved in understanding and solving this problem?

The exercise question:

There is a way of determining the bitwise representation of a digit by adding an extra layer to the three-layer network above. The extra layer converts the output from the previous layer into a binary representation, as illustrated in the figure below. Find a set of weights and biases for the new output layer. Assume that the first 3 layers of neurons are such that the correct output in the third layer (i.e., the old output layer) has activation at least 0.99, and incorrect outputs have activation less than 0.01.

@NeilSlater, I can see how (1 * 10) * 1 = +10 which is given in your explanation, however I cannot understand how you get to -10 instead of 0. I would have thought that: (0 * 10) * 1 = 0 would be correct. Please excuse the basic nature of this question as I have just begun going through the book mentioned. – rrz0 – 2018-02-09T16:04:03.447

1@Rrz0: Because I am assuming a sigmoid layer on output, as it is a binary classification - the bit is either on or off. So in your example you get

`sigmoid((0 * 10) * 1)`

which is 0.5. By choosing suitably large numbers, you ensure either a very high or low output before the sigmoid, which will then output very close to 0 or 1. This is more robust IMO than the linear output assumed in FullStack's answer, but ignoring that, essentially our two answers are the same. – Neil Slater – 2018-02-09T19:13:06.033@NeilSlater, thanks for getting back, and for the explanation. I think that I have understood. So, when for example 3 is activated, assuming

`+-10`

weights you get:`(+10*1)`

or`(-10*1)`

depending whether you want to output a 0 or 1 after the`sigmoid()`

. In this case we would do:`(-10*1), (-10*1), (+10*1), (+10*1)`

which represents 0011. Is the reasoning here correct or am I missing something? Thanks again – rrz0 – 2018-02-09T20:03:43.423@Rrz0: That's correct. – Neil Slater – 2018-02-09T20:33:17.630

@NeilSlater would this solution by Sebastian Bensusan also be acceptable: https://s33.postimg.cc/p6r2f6uof/Screen_Shot_2018-06-06_at_10.25.39_PM.png

– Pablo – 2018-06-07T02:26:34.467@Pablo: Yes that looks almost identical to FullStack's answer here, with the bit order reversed. – Neil Slater – 2018-06-07T06:50:33.333

Thank you. I couldn't quite follow this part, would you mind elaborating further please? -

"I might have weights going from activation of old output i=3, AOld3 to logit of new outputs ZNewj, where ZNewj=Σi=9i=0Wij∗AOldi as follows:

W3,0=−10 W3,1=−10 W3,2=+10 W3,3=+10" – Victor Yip – 2015-08-02T12:10:38.450

1@VictorYip: The equation is just the normal feed-forward network equation, but to use it I had to define my terms carefully (since you have no reference maths in your question). The "logit" Z value is the value calculated at the neuron

beforeactivation functions have been applied (and generally $A_i = f( Z_i )$ where $f$ is e.g. sigmoid function). The example weights are the values I would use for connecting new output layer neurons to old ones, but just the ones that connect the 4 neurons in the new output layer to one of the neurons in old output layer (the one for output "3") – Neil Slater – 2015-08-02T15:20:18.047So if you want to see this working in Octave / Matlab, you can do something like this: – learningMachine – 2020-03-06T06:26:17.020

@NeilSlater - will your example weights work for the outputs that are not 3? I don't see that they will. Please elaborate. Thanks. – FullStack – 2015-10-23T06:59:18.913

@FullStack: Yes it will work, because if $A_3^{old}$ is not active (activation 0), then none of the weights in the example have any impact. You have to construct similar maps for connections from each output neuron in the old layer - each one is associated very simply with its binary representation in the new layer, and they are all independent. – Neil Slater – 2015-10-23T08:31:29.337