C++ Neural Networks and Fuzzy Logic by Valluru B. Rao M&T Books, IDG Books Worldwide, Inc. ISBN: 1558515526   Pub Date: 06/01/95

Chapter 6
Learning and Training

In the last chapter, we presented an overview of different neural network models. In this chapter, we continue the broad discussion of neural networks with two important topics: Learning and Training. Here are key questions that we would like to answer:

•  How do neural networks learn?

•  What does it mean for a network to learn ?

•  What differences are there between supervised and unsupervised learning ?

•  What training regimens are in common use for neural networks?

Objective of Learning

There are many varieties of neural networks. In the final analysis, as we have discussed briefly in Chapter 4 on network modeling, all neural networks do one or more of the following :

•  Pattern classification

•  Pattern completion

•  Optimization

•  Data clustering

•  Approximation

•  Function evaluation

A neural network, in any of the previous tasks, maps a set of inputs to a set of outputs. This nonlinear mapping can be thought of as a multidimensional mapping surface. The objective of learning is to mold the mapping surface according to a desired response, either with or without an explicit training process.

Learning and Training

A network can learn when training is used, or the network can learn also in the absence of training. The difference between supervised and unsupervised training is that, in the former case, external prototypes are used as target outputs for specific inputs, and the network is given a learning algorithm to follow and calculate new connection weights that bring the output closer to the target output. Unsupervised learning is the sort of learning that takes place without a teacher. For example, when you are finding your way out of a labyrinth, no teacher is present. You learn from the responses or events that develop as you try to feel your way through the maze. For neural networks, in the unsupervised case, a learning algorithm may be given but target outputs are not given. In such a case, data input to the network gets clustered together; similar input stimuli cause similar responses.

When a neural network model is developed and an appropriate learning algorithm is proposed, it would be based on the theory supporting the model. Since the dynamics of the operation of the neural network is under study, the learning equations are initially formulated in terms of differential equations. After solving the differential equations, and using any initial conditions that are available, the algorithm could be simplified to consist of an algebraic equation for the changes in the weights. These simple forms of learning equations are available for your neural networks.

At this point of our discussion you need to know what learning algorithms are available, and what they look like. We will now discuss two main rules for learning—Hebbian learning, used with unsupervised learning and the delta rule, used with supervised learning. Adaptations of these by simple modifications to suit a particular context generate many other learning rules in use today. Following the discussion of these two rules, we present variations for each of the two classes of learning: supervised learning and unsupervised learning.

Hebb’s Rule

Learning algorithms are usually referred to as learning rules. The foremost such rule is due to Donald Hebb. Hebb’s rule is a statement about how the firing of one neuron, which has a role in the determination of the activation of another neuron, affects the first neuron’s influence on the activation of the second neuron, especially if it is done in a repetitive manner. As a learning rule, Hebb’s observation translates into a formula for the difference in a connection weight between two neurons from one iteration to the next, as a constant [mu] times the product of activations of the two neurons. How a connection weight is to be modified is what the learning rule suggests. In the case of Hebb’s rule, it is adding the quantity [mu]a_ia_j, where a_i is the activation of the ith neuron, and a_j is the activation of the jth neuron to the connection weight between the ith and jth neurons. The constant [mu] itself is referred to as the learning rate. The following equation using the notation just described, states it succinctly:

[Delta]wij = [mu]aiaj

As you can see, the learning rule derived from Hebb’s rule is quite simple and is used in both simple and more involved networks. Some modify this rule by replacing the quantity a_i with its deviation from the average of all as and, similarly, replacing a_j by a corresponding quantity. Such rule variations can yield rules better suited to different situations.

For example, the output of a neural network being the activations of its output layer neurons, the Hebbian learning rule in the case of a perceptron takes the form of adjusting the weights by adding [mu] times the difference between the output and the target. Sometimes a situation arises where some unlearning is required for some neurons. In this case a reverse Hebbian rule is used in which the quantity [mu]a_ia_j is subtracted from the connection weight under question, which in effect is employing a negative learning rate.

In the Hopfield network of Chapter 1, there is a single layer with all neurons fully interconnected. Suppose each neuron’s output is either a + 1 or a – 1. If we take [mu] = 1 in the Hebbian rule, the resulting modification of the connection weights can be described as follows: add 1 to the weight, if both neuron outputs match, that is, both are +1 or –1. And if they do not match (meaning one of them has output +1 and the other has –1), then subtract 1 from the weight.