C++ Neural Networks and Fuzzy Logic by Valluru B. Rao M&T Books, IDG Books Worldwide, Inc. ISBN: 1558515526   Pub Date: 06/01/95

You may think that the last column of W being all zeros presents a problem in that when the input is X₁ and whatever the output, when this output is presented back in the backward direction, it does not produce X₁, which does not have a zero in the last component. There is a thresholding function needed here. The thresholding function is transparent when the activations are all either +1 or –1. It then just looks like you do an inverse mapping from bipolar values to binary, which is done by replacing each –1 by a 0. When the activation is zero, you simply leave the output of that neuron as it was in the previous cycle or iteration. We now present the thresholding function for BAM outputs.

     1  if  yj > 0   1   if  xi > 0
bj|t+1 =  bj|t    if  yj = 0   and   ai|t+1 =   ai|t  if  xi = 0
    0   if  yj < 0                                     0   if  xi < 0

where x_i and y_j are the activations of neurons i and j in the input layer and output layer, respectively, and b_j|_t refers to the output of the jth neuron in the output layer in the cycle t, while a_i|_t refers to the output of the ith neuron in the input layer in the cycle t. Note that at the start, the a_i and b_j values are the same as the corresponding components in the exemplar pair being used.

If X₁ = (1, 0, 0, 1) is presented to the input neurons, their activations are given by the vector (–4, 4, 0). The output vector, after using the threshold function just described is (0, 1, 1). The last component here is supposed to be the same as the output in the previous cycle, since the corresponding activation value is 0. Since X₁ and Y₁ are one exemplar pair, the third component of Y₁ is what we need as the third component of the output in the current cycle of operation; therefore, we fed X₁ and received Y₁. If we feed Y₁ at the other end (B field) , the activations in the A field will be (2, –2, –2, 2), and the output vector will be (1, 0, 0, 1), which is X₁.

With A field input X₂ you get B field activations (4, -4, 0), giving the output vector as (1, 0, 1), which is Y₂. Thus X₂ and Y₂ are heteroassociated with each other.

Let us modify our X₁ to be (1, 0, 1, 1). Then the weight matrix W becomes

        1  [-1  1  1]       -1  [1  -1  1]       -2    2    0
 W =   -1              + 1               =    2   -2    0
        1                    1                    0    0    2
        1                   -1                   -2    2    0

and

                  -2     2     0    -2
WT =    2    -2     0     2
                   0     0     2     0

Now this is a different set of two exemplar vector pairs. The pairs are X₁ = (1, 0, 1, 1), ₁ = (0, 1, 1), and X₂ = (0, 1, 1, 0), Y₂ = (1, 0, 1). Naturally, the weight matrix is different, as is its transpose, correspondingly. As stated before, the weights do not change during the operation of the network with whatever inputs presented to it. The results are as shown in Table 8.1 below.

You may think that you will encounter a problem when you input a new vector and one of the neurons has activation 0. In the original example, you did find this situation when you got the third output neuron’s activation as 0. The thresholding function asked you to use the same output for this neuron as existed in the earlier time cycle. So you took it to be 1, the third component in (0, 1, 1). But if your input vector is a new X vector for which you are trying to find an associated Y vector, then you do not have a Y component to fall back on when the activation turns out to be 0. How then can you use the thresholding function as stated? What guidance do you have in this situation? If you keep track of the inputs used and outputs received thus far, you realize that the Field B (where you get your Y vector) neurons are in some state, meaning that they had some outputs perhaps with some training vector. If you use that output component as the one existing in the previous cycle, you have no problem in using the thresholding function.

As an example, consider the input vector X₃ = ( 1, 0, 0, 0), with which the activations of neurons in Field B would be (-2, 2, 0). The first component of the output vector is clearly 0, and the second clearly 1. The third component is what is in doubt. Considering the last row of the table where Y₂ gives the state of the neurons in Field B, you can accept 1, the last component of Y₂, as the value you get from the thresholding function corresponding to the activation value 0. So the output would be the vector (0, 1, 1), which is Y₁. But Y₁ is heteroassociated with X₁. Well, it means that X₃ = ( 1, 0, 0, 0) is not heteroassociated with any X vector.