ARTICLE
Communicated by David Zipser
Conversion of Temporal Correlations Between Stimuli to Spatial Correlations Between Attractors M. Griniasty M.V. TsodyW Racah lnstitute of Physics and Center for Neural Computation, Hebrew University, Jerusalem
Daniel J. Amit' INFN, Sezione di Roma, lstituto di Fisica, Universita di Roma, La Sapienza, Ple Aldo Moro, Roma, Italy
It is shown that a simple modification of synaptic structures (of the Hopfield type) constructed to produce autoassociative attractors, produces neural networks whose attractors are correlated with several (learned) patterns used in the construction of the matrix. The modification stores in the matrix a fixed sequence of uncorrelated patterns. The network then has correlated attractors, provoked by the uncorrelated stimuli. Thus, the network converts the temporal order (or temporal correlation) expressed by the sequence of patterns, into spatial correlations expressed in the distributions of neural activities in attractors. The model captures phenomena observed in single electrode recordings in performing monkeys by Miyashita et al. The correspondence is as close as to reproduce the fact that given uncorrelated patterns as sequentially leamed stimuli, the attractors produced are significantly correlated up to a separation of 5 (five) in the sequence. This number 5 is universal in a range of parameters, and requires essentially no tuning. We then discuss leaming scenarios that could lead to this synaptic structure as well as experimentalpredictions following from it. Finally, we speculate on the cognitive utility of such an arrangement. 1 Introduction
1.1 Temporal to Spatial Correlations in Monkey Cortex. The remarkable sequence of neurocognitive experiments by Miyashita (1988), Miyashita and Chang (19881, and Sakai and Miyashita (1991) is the most direct evidence of the relevance of attractor dynamics in cortical cognitive processing. It is at the same time detailed and structured enough 'On leave of absence from The Institute of Higher Nervous Activity, Moscow. +Onleave of absence from Racah Institute of Physics, Hebrew University, Jerusalem.
Neural Computation 5,1-17 (1993)
@ 1993 Massachusetts Institute of Technology
2
M. Griniasty, M. V,Tsodyks, and D. J. Amit
to guide and confront attractor neural network (ANN) modeling. In the first experiment (Miyashita and Chang 1988), the monkey is trained to recognize and match a set of visual patterns. As a result, one observes selective enhancement of neural spike activity, which persists for 16 sec after the removal of the stimulus. The fact that selective, stimulus-related, enhancement of neural activity persists for 16 sec in the absence of the provoking stimulus is evidence of nonergodic attractor dynamics (see, e.g., Amit 1992). The same encouraging evidence has forced a confrontation on the question of activity rates on retrieval by attractors. The rates in the Miyashita attractors were many times lower than what models of the Hopfield type (Hopfield 1982; Amit 1989) predicted. This fruitful confrontation led to a study (Amit and Tsodyks 1991) that showed that when neural description is taken in greater detail, as well as the conditions prevailing in cortex, attractors can appear having stochastic behavior and low rates. The second study (Miyashita 1988) went further to provide information about coding in the particular module of the anterior ventral temporal cortex of the monkey: It was discovered that despite extreme precaution in producing visual stimuli uncorrelated in their spatial form, spatial correlations appeared in the patterns of sustained activities, evoked by the stimuli, during the delay period. These persistent activities we interpret as the structure of the attractors. There was one kind of correlation that was preserved in the stimuli, the temporal order of their presentation was maintained fixed during training. What the monkey’s brain appears to be doing, is to convert the temporal correlation into a spatial one. Namely, spatial correlations were observed among the attractors, corresponding to the stimuli that were close temporally in the training session. These attractors are the result of retrieval dynamics. The spatial correlations between the activities of the neurons investigated persisted to a fifth neighbor in the temporal sequence. The correlation figure of Miyashita (1988) is reproduced in Figure 1. 1.2 Modeling Correlation Conversion. The main result of the Hopfield program has been to connect the intuitive call for selective (stimulus dependent) attractor dynamics (associative memory) with specific constructions of synaptic matrices, and therefore a bridge to unsupervised learning. The program was limited by the requirement that the attractors be as close as possible to the patterns of which the matrices were constructed, that is, the presumed items in the learning process. This went under the name of autoassociation. Here we shall show that a simple modification of the synaptic matrices used for autoassociation in ANNs leads to a relaxation dynamics that associates with stimuli near one of the random, uncorrelated underlying patterns, an attractor that is correlated with several patterns. The patterns that have the largest correlations with a given attractor are the neighbors of the stimulus leading to the attractor, in the sequence of stored patterns.
Conversion of Temporal to Spatial Correlations
c
3
0.4t
L.
c
.g -
-
0.3 -
F
6 0 0
-6
0.2
-
0.1
-
0
CI
g
0
0
-1
5
10th neighbour
Figure 1: Spatial correlations between attractors, in monkey's anterior ventral temporal cortex, corresponding to structurallyuncomlated patterns, as a function of the difference in the position of the learned stimuli in the fixed training sequence. From Miyashita (1988).
It then follows that attractors are correlated among themselves. Again, the attractors that are correlated are the neighbors in the sequence of the underlying patterns. These are just the type of correlations observed by Miyashita. In fact, the number of attractors that are found to be correlated significantly in the model is the same as in the experiment. The extended model is discussed in two different variants: one is the original formulation of k1 neurons, with the artificial symmetry between active and passive states of neurons; the second is a 0-1 formulation (Tsodyks and Feigel'man 1988; Buhmann et al. 19891, in which this symmetry is removed and that can naturally be interpreted in terms of high and low activity rates of neurons in attractors. The results differ in detail, but the main qualitative features, of converting sequential order among uncorrelated patterns to a set of correlated attractors, is present in both. Both models have symmetric synaptic matrices, which are unrealistic but convenient. The study of autoassociative ANNs, over the last several years, has made it clear that most of the attractor properties of these extensively connected networks are rather robust to the introduction of synaptic asymmetry (see, e.g., h i t 1989). We then proceed to interpret the proposed synaptic matrices in terms of learning dynamics. It is argued that rather plausible synaptic dynamics, accompanying the relaxation in the ANN, may produce a synaptic
M. Griniasty, M. V. Tsodyks, and D. J. Amit
4
matrix with correlated attractors for uncorrelated external stimuli. Within such learning scenarios, one is led to predict that the presentation of uncorrelated patterns in a random sequence would produce attractors that are uncorrelated, and are each close to the representation of the original patterns, as would be the case in the Hopfield model. Finally, we discuss the potential utility of such conversions of temporal correlations to spatial correlations in modeling several aspects of cognitive behavior. 2 The Model with fl Neurons
The original way of pursuing the Hopfield ANN program was to choose the variables describing the instantaneous states of each neuron S;(t)=fl, where i labels the neuron (i = I , . . . ,N). The patterns, to be stored in an N neuron network, are N-bit words of fls, the value of each bit chosen independently, with probability 0.5. Denoting the components of the activity of neuron number i in pattern number p, tr, the proposed synaptic matrix is written as:
where p is the total number of patterns stored in the connections. The patterns, p, are considered to form an ordered sequence, which corresponds to the order of presentation in the training phase. For simplicity, the sequence is taken to be cyclic. Each pattern in the construction of the matrix is connected to one preceding pattern. Note, in particular, that this extended matrix still preserves the symmetry of the Hopfield matrix, which implies that all attractors will be fixed points, and makes analysis so much simpler. How this relates to a learning scenario is discussed in Section 4. The matrix of equation 2.1, for a = 0, reduces to the original Hopfield matrix. This matrix is accompanied, as usual, by a schematic spike emission dynamics that, in the noiseless case, determines the new state, Si(t), of the neuron according to Si(t
+ 6t) = sign[hi(t + bt)]
(2.2)
where
h; mimics the value of the postsynaptic potential, relative to the threshold, on neuron i. The linear superposition of bilinear terms in the neural activities of the stored patterns is sometimes referred to as a Hebbian learning from a “tabula rasa.” We shall return to the question of learning later.
Conversion of Temporal to Spatial Correlations
5
The natural variables for the description of the nonergodic asymptotic behavior of the network are the “overlaps” m”(t) of the current state of the network, Si(t), with the stored pattern p. They are defined as
See, for example, Amit et al. (1985) and Amit (1989). The value of overlap
measures how close is the state of the network to the stored pattern mp = 1, the state is identical, as a binary word to the pattern p, that is, Si = #‘ for all i. If mp = 1 for the asymptotic state, the attractor, then the corresponding pattern is retrieved perfectly. With the matrix, equation 2.1, the “field hi can be expressed in terms of the overlaps (Amit et al. 19851, which implies that so can the dynamics of the network as well as its attractors. Namely, we can write equation 2.3 as mp
p. If
from which one derives the mean-field equations determining the attractors. For a symmetric matrix, those are simple fixed points. They read, in the limit of a network of a large number of neurons with a relatively low number of stored patterns:
The double angular brackets imply an averaging over the distribution of the bits in the patterns (see, e.g., Amit et al. 1985; Amit 1989). Autoassociation was the interpretation of the fact that, in the absence of noise, for low loading, the equations 2.6 had solutions with one single m p # 0, which attracted a wide set of initial states in the neighborhood of each pattern. Away from these large basins, ‘,spurious states” were found to exist (Amit et al. 1985). Moreover, the artificial symmetry of the +1 and -1 states produced attractors of the sign reversed states of each pattern. These, retrieval properties of the Hopfield ANN have been found very robust to extensive noise and synaptic disruption, including asymmetric disruption. If one tries a pure pattern solution for equations 2.6, with p = 2 for example, one has1:
m2 = ( ( t 2 w(m2[t2 + a(tl
+ t3)i)))
(2.7)
For a < 0.5, it is the first term in the square brackets that dominates the sign of the argument of the sign-function and m2 = 1 is a fixed point solution, as in the case a = 0. For a > 0.5, this is no longer the case. For ‘The 2s are superscripts not squares.
M. Griniasty, M. V. Tsodyks, and D. J. Amit
6
25% of the sites t3= t1= -t2 and the argument has the sign opposite to that of Starting from a state with m2 = 1, and all other overlaps 0,
c2,
one arrives, after one step to a state with m2 = m1 = m3 = 0.5. This is no fixed point either. The solutions of equations 2.6 have several overlaps different from zero. The previous discussion suggests a numerical procedure for arriving at the solution: start from a pure pattern state and iterate until convergence. This is what the network would do, if given one of the pure patterns it learned, Si = t’, as an initial state, until it relaxes to a fixed point. The symmetry of the dynamics under pattern permutations implies that this has to be done for one pattern only The equations were solved in this way. One finds that starting from a pure pattern, one arrives at a stable solution after several iterations. The solution reached is a state with nonzero overlaps with several stored patterns, symmetrically distributed around the pattern, which served as the stimulus. Only a small number, actually 5, of these overlaps are significantly large, provided u < 1.2 This distribution of overlaps in an attractor, corresponding to one of the underlying patterns, is shown in Figure 2. In this case p = 13 patterns are stored, and u = 0.7. It is remarkable that the structure of the attractor does not depend on the number of patterns p, nor on the value of a, in the entire range 0.5 < u < 1. For a > 1, the network develops attractors that have overlaps with all stored patterns. The values of the overlaps decrease as the number of patterns increases. This means that after learning sufficiently many patterns, the network loses its ability to associate attractors with the stimuli. One can read from Figure 2 that the retrieval attractor has substantial overlaps with several patterns, symmetrically disposed, before and after, in the sequence relative to the pattern corresponding to the stimulus. Clearly, if each attractor is correlated with several patterns, then the attractors corresponding to different patterns must themselves be correlated. These correlations would correspond to the correlations measured by Miyashita and Chang (1988) (Fig. 1). The correlation of activities in two attractors, u p and u’, is defined as i
N
C(ui”- a)(aiv- a ) 1‘1 j=l
C(p,v) = i
(2.8)
where 8 is the average activity in a given attractor and the normalization constant ICI is chosen so that C ( p ,p ) = 1. In the present case, T = 0, and (C( = N. Hence, the correlation of attractors p and v can be written as
1
= -
C sign(ha)sign(hr)= ((sign(hj’)sign(h”)))
N i
(2.9)
2Noteadded in proof L. Cugliandolo has recently proved that beyond 5 all overlaps are exactly zero.
Conversion of Temporal to Spatial Correlations
7
pattern
Figure 2 Overlaps between an attractor and stored patterns, as a function of the separation of the pattern in the sequence from the pattern underlying the attractor. where hy is the local field on neuron i when the network is in attractor number p. The last equality is an expression of self-averaging, giving an average over the distribution of patterns. Finally, substituting the fields from equation 2.5 in equation 2.9, we arrive at the correlation coefficient:
(2.10)
where m; is the overlap of the attractor corresponding to stimulus number p with pattern number p. These attractor overlaps are illustrated in Figure 3, where we plot the correlations among different pairs of attractors vs. the distance between their corresponding patterns. Figure 3 clearly demonstrates that while stored patterns are completely random, and hence uncorrelated, the states reached by the network on presentation of these same patterns, have a substantial degree of correlations, which decreases with the separation of the patterns in the sequence. Note that while an attractor "sees" two-three patterns on each side, it sees five attractors on each
8
M. Griniasty, M. V. Tsodyks, and D. J. Amit
Figure 3: Correlations between attractors as a function of the separation in the sequence of the patterns to which the attractors belong. side. The qualitative form of the correlations captures the experimental trend (Fig. 1). The absolute values of the correlations differ. This can be due to a different normalization used in Miyashita and Chang (1988), where the normalization is not given explicitly. It may also be that the absolute values will be different in more realistic networks. Finally, the analysis given above is based on the exact symmetry of the matrix 2.1, in which case the system has only fixed point attractors. Asymmetry can enter in two ways: either as a local, random disruption of the synaptic elements, or as a coherent asymmetry of the two transition terms in the symmetric matrix. The first type of asymmetry is in the realm of the robustness of the attractor dynamics of the network. Concerning the second type: The mean-field equations 2.6 hold even if the coefficients of the two transition terms in Jq are not equal. We have found that the behavior of the network is robust against some amount of asymmetry between the two coefficients. However, if the asymmetry becomes too large, equations 2.6 first acquire another solution, with the maximum overlap shifted to another pattern. At still higher asymmetry, the fixed point solution is lost, in favor of a time-dependent attractor. In this attractor the network moves from one pattern to another, in a direction determined by the major nondiagonal term in the matrix 2.1.
Conversion of Temporal to Spatial Correlations
9
3 ANN with Discrete 0-1 Neurons
The above description was extended to deal with a 0-1 representation of information, allowing for the removal of the symmetry between active and refractory states of the neurons (Tsodyks and Feigel’man 1988; Buhmann ef aI. 1989). This description has several further advantages: its terms are very close to a representation in terms of spike rates, which are positive analog variables; moreover it allows for very efficient storage of patterns as the coding rate, that is, the fraction of Is in the patterns, becomes very low-sparse coding. Since it is important to show that the correlation effects to be discussed take place also for this case, we shall recall that formulation as well. The dynamics is described in terms of instantaneous neural variables, V;(t), which take the values (0,U as V;(t
+ St) = 0
,]jjVj(t) [ j
I
-0
(3.1)
where Q ( x ) = 1 for x > 0, and 0 otherwise, and 0 is a neural threshold. These variables can be directly interpreted as high and low activity (spike rates) of each neuron. Such would be a description in terms of analog rates, in a high gain limit. In this case the patterns to be stored by a learning dynamics, q”, are chosen as N-bit words of independently chosen O,ls, that is, q;=o;1
p=ll...lp
(3.2)
where the probability for a 1,O-bit is f l (1- f ) , respectively. An extension of the symmetric synaptic matrix, appropriate for autoassociation (Tsodyks and Feigel’man 1988; Buhmann et al. 1989), to our requirements would be
(3.3)
and the corresponding overlaps are generalized to (3.4)
With the couplings 3.3, the dynamics can be expressed in terms of the above overlaps and so can the fixed points of the retrieval attractors. The latter have the form
M. Griniasty, M. V. Tsodyks, and D. J. Amit
10
When a = 0, the system of equations reduces to that of Tsodyks and Feigel'man (1988) and Buhmann et al. (1989). In this case, at low loading, the exact stored patterns are the retrieval attractors of the network, that is, the equations admit solutions with a single nonvanishing overlap, which in turn is equal to 1. These attractors persist until a reaches the critical value of
Above this value of a, the pure patterns are unstable and the network, having a symmetric synaptic matrix, finds new fixed points. The equations 3.5 have to be solved numerically for the values of the overlaps in the retrieval attractors. This we do, again following the network, as was explained in the previous section. A typical solution is shown in Figure 4, for parameter values: f = 0.01, 6 = 0.2, p = 11, and a = 0.25. Figure 4a represents the overlaps vs. the pattern number in the sequence, relative to the pattern of the stimulus. Figure 4b is the correlationbetween the attractors. Note that in distinction to the fl case, the significant overlaps here, of which there are five in total, are all equal. They are all unity, up to terms of OU).This implies that the attractor is approximately the union of the 1-bits in the five patterns centered around the stimulus. In particular, the mean spatial activity level in the attractors is higher than in the pure patterns. A fact that can be tested experimentally. The correlation Figure 4b may seem somewhat simple compared with the experimental one of Figure 1. Clearly, the experimental correlations are not a straight line going to zero at a separation of five patterns. We find the appearance of the correlations as well as their clear trend to decrease with the separation in the training sequence, down to very small values at a separation of five, very significant. All that was put in was the synaptic structure connecting successive patterns in the sequence. The remaining differences may be attributed to several factors, all of which are under study. These factors are a
The neurons in the experiment are analog neurons, represented by spike rates, and not discrete 0-1 neurons.
a
In the experiment the neurons operate in the presence of noise, while here for simplicity we dealt with a noiseless situation.
a
The matrix we chose is surely not the matrix in the monkey's cortex. One consequence is that all our attractors are identical.
a
In the experiment the sample groups of neurons are small and are chosen in special ways. This leads inter alia to large fluctuations. O u r correlations are ideal in that they take into account an infinite number of neurons.
Conversion of Temporal to Spatial Correlations
11
All these effects can be studied either by an extension of the above reasoning to analog neurons with noise, or by simulations. These studies are under way. The attractor to attractor correlations are computed according to equation 2.8. What remains is to determine r and ICI for this case. If the mean proportion of 1-bits in the attractors is g, then 8 = g and ICI = Ng(1 -g). It is rewarding to see that the correlations between neighboring attractors are monotonically decreasing with the separation in the sequence, and are disappearing after the fifth neighbor, as in the experimental data. In the present case, the number of condensed patterns, those having large values of the overlap with the attractor corresponding to a stimulus, depends on the value of u. This variation leaves finite intervals of u in which the attractors are invariant. Increasing u, we observe a sequence of bifurcations, where the number of condensed patterns increases by two. Correspondingly, the number of significantly correlated attractors increases by four on crossing a bifurcation value of a. Between any two bifurcation points the solution does not change, that is, the number of significantly correlated attractors as well as the magnitude of the correlations remain invariant.
4 Learning
In this section we will try to discuss possible learning scenarios, which could lead to a synaptic structure of the type considered in the previous sections. At the present time there is not enough information about the learning mechanism and memory preservation in the cortex, and our discussion can at best be tentative. We feel though that such a discussion may not be completely premature, just because of the level of specific detail provided by the experiments of Miyashita et al., and the ability of theory to approach a similar level of detail. Moreover, it is our feeling that a discussion of the implications to learning, of such findings, may lead to experiments which may shed additional light on constraints on learning through neurophysiological correlates of behavior. It is plausible to describe the synaptic dynamics as
(4.1)
where y is the rate of decay of the synaptic value and Ki,Lj are, respectively, the post- and pre-synaptic contributions to the synaptic efficacy. Both K and L depend on the activity of the corresponding neuron. A simple mechanism that would lead to the matrix 2.1 could be to apply a usual Hebbian modification rule, with both pre- and post-synaptic
M. Griniasty, M. V. Tsodyks, and D. J- Amit
12
a
0
2
4
6
pattern
8
10
12
b
Figure 4: Correlations between attractors (a) and overlaps of attractor with patterns (b) vs. the separation in the sequence of the patterns to which the attractors belong.
Conversion of Temporal to Spatial Correlations
13
terms as linear combinations of the current and preceding patterns, for example,
(4.2)
This form may result from two different scenarios. In both we assume that strong presentations of the individual, uncorrelated patterns create attractors for those patterns themselves. Then, during training, which consists of many repeated presentations, the network, which remains in an attractor between presentations, is made to move to the next attractor by a new presentation. It should be emphasized that in this description the role of the attractors is quite crucial. Before the patterns themselves are stored in the synapses as attractors, at the presentation of a consecutive pattern (to be learned) in the sequence during training, there is no memory of the previous pattern. This is especially true if the time between presentations of consecutive patterns is as long as in the experiments of Miyashita et al. The difference between the two scenarios is in the way we view the origin of the source terms, K and L, for the synaptic change. In the first we assume that the values of the neuronal spike frequencies represent, in an analog way, the transition between the two attractors. In this picture, K l ( t ) = L l ( t ) = 0, before the transition starts, which is about when the next pattern is presented. When the network is well established in the new attractor, K 2 ( t ) and L*(t) tend to zero. In the second scenario, one assumes that it is the local synaptic variable that remembers some short history of the pre- or postsynaptic activity of the corresponding neuron. For example, it may be the case that the synaptic mechanism modifies its effectiveness depending on the mean of the neuron's activity in some prior time window 7. The pre- or postsynaptic change may be enhanced, or suppressed, by a history of high average mean activity. Since the moving mean of the activity history is a linear combination of the activity in two consecutive patterns, while the network is moving from one attractor to the next, the end result is the same, provided, of course, that the averaging window is short in comparison with the time spent in each attractor. But this does not seem to be a strong requirement, given that the network stays in these attractors for many seconds. As a simple assumption we can take K = L, as functions of the p r e and post-synaptic activities, which implies KI = LI, K2 = L2. Relaxing this constraint would lead us to asymmetric synaptic transition terms, of the type discussed in Section 2. If the resulting asymmetry is not large we expect the performance of the network to be robust. In the symmetric case, the contribution to the synaptic dynamics is
14
M. Griniasty, M. V. Tsodyks, and D. J. Amit
As one pattern follows the other, these contributions sum up, when equation 4.1 is integrated. If we neglect the exponential decay, y, the summation is direct and after a long time, when all patterns have been presented many times in a fixed order, the resulting matrix would be proportional to
where the time integration is over an interval 7, in which synaptic modification is taking place. This matrix has the same form as the one we introduced in the previous sections. It corresponds though to a case in which u 5 0.5. This fact should not be considered too adverse. Synaptic decay, for example, is sufficient to raise u above 0.5. In the final analysis one should consider analog neurons, toward which the 0-1 neurons are an intermediate stage. Even for the discrete 0-1 neurons, the critical value of u is much lower than 0.5, while the heuristic learning mechanism can remain essentially the same. Finally, if the patterns are presented in a random order during training, one can expect every pattern to be followed by any other one, given that a large number of presentations is required for satisfactory learning. This implies that the transition terms in equation 4.4, containing any particular pattern, will be multiplied by a sum over all other patterns. That sum vanishes on the average and the transition terms become negligibly small. No correlations are then generated by the netwprk, from uncorrelated patterns. 5 Experimental Predictions and Some Speculations
Given that a synaptic matrix, which can be learned without supervision, is able to convert temporal correlations into spatial ones, one is tempted to make some preliminary speculations about the computational and behavioral utility of such synaptic development. One directly measurable application had been pointed out in Sakai and Miyashita (1991). In this experiment the monkeys are trained to recognize 24 visual patterns, organized in 12 fixed pairs. The pairs are presented in a random order. Correlations are generated among the two members of each pair only. Those correlations are then shown to be correlated with the ability of the monkey to retrieve the second member of a pair, after being presented with the first. The basic nature of this type of association for the construction of cognitive behavioral patterns is quite immediate. What is special about this particular experiment is that the associative retrieval of the paired member is directly connected to the presence of the correlations in the representation of the pairs of attractors in the part of cortex under electrophysiological observation.
Conversion of Temporal to Spatial Correlations
15
The interpretation of this experiment does not require speculation. To
go one step beyond, one can expect the generation of such correlations to underlie the effect of priming (Besner and Humphreys 1990). In other words, if the network is in one of its attractors, and a new stimulus is presented, the transition between two attractors that are highly correlated (i.e., have a particularly large number of active neurons common to their representations) is much faster than the transition between less correlated attractors. This effect was observed in a simulation with realistic neurons (Amit et al. 1991), when the pure patterns, involved in the construction of the synaptic matrix, included as explicit correlations (Rubin 1991). This effect can be directly measured in a Miyashita (1988) type experiment. One would expect that the transition time between different attractors would increase with the distance of the two patterns in the sequence of presentation. In cognitive psychology the effect is familiar in experiments in which the reaction time is measured for the recognition of distorted words or other images. This reaction time is significantly shortened if the pattern to be recognized is preceded by a cognitively correlated pattern (Besner and Humphreys 1990). In the language of the model we would say that the "priming" image leads the network into its corresponding attractor. That attractor is correlated with the attractor corresponding to the test stimulus. Hence, the transition between the two is faster than the transition from some other state in which the network may find itself in otherwise. Complementing this scenario with the suggestion that at least part of our basic cognitive correlations is related to temporal contiguity of afferent stimuli completes this speculation. This interpretation can be extended one small step further. As attractors get increasingly correlated, there is an increase in the probability that noise would cause transitions between them, transitions of the BuhmannSchulten type (Buhmann and Schulten 1987). This opens the way for the scenario in which such transitions can be provoked in a cortical network by random afferent activation of the module. The transitions will tend to take place between correlated attractors, which in the present model are related to temporal proximity during learning. Note that this process can also be observed in the experiments of the Miyashita type, though their cognitive content is more difficult to investigate. One could hope to be able to investigate the process of learning the matrix that generates the correlations. We have argued in Section 4 that the process will go through the intermediate stage of learning the pure pattern attractors first. This was based on the assumption that there is autonomous learning in the particular module under observation. This is not self-evident, and it may be that the pure patterns are quickly learned as attractors in a different area, as hippocampus, for example, and those attractors then assist in learning the correlated attractors. Since the question is open, one could attempt to clarify it by presenting different parts of the training sequence, in an experiment such as Miyashita (19881, with
16
M. Griniasty, M. V. Tsodyks, and D. J. Amit
different frequencies. Then, if learning actually first goes through the creation of individual attractors for the pure patterns, one should observe lower correlations in the parts shown less frequently, as well as lower coding rates. In other words, pure patterns are expected to use fewer neurons than the composite patterns correlated by the dynamics (see, e.g., Section 3). On the other hand, if the module learns the correlated attractors directly, no group of patterns should show the appearance of uncorrelated attractors. Acknowledgments
The authors acknowledge useful discussions with N. Rubin, S. Seung, and H. Sompolinsky DJA has benefited from many useful discussions of the learning mechanisms with Stefan0 Fusi. We are indebted to N. Rubin for information concerning priming effects. MVT's research is supported in part by the Ministry of Science and Technology. References Amit, D. J. 1989. Modeling Brain Function. Cambridge University Press, New York. Amit, D. J. 1992. In defence of single electrode recordings. NETWORK 4(4). Amit, D. J., Gutfreund, H., and Sompolinsky, H. 1985. Spin-glass models of neural networks. Phys. Rev. A32,1007. Amit, D. J.,Evans, M. R., and Abeles, M. 1991. Attrador neural networks with biological probe neurons. NETWORK 1,381. Amit, D. J., and Tsodyks, M. V. 1991. Quantitative study of attractor neural network retrieving at low spike rates I Substrate-spikes, rates and neuronal gain. NETWORK 2, 259; and Low-rate retrieval in symmetric networks. NETWORK 2,275. Besner, D., Humphreys, G. eds. 1990. Basic Processes in Reading: Visual word recognition. Hillside, NJ: Erlbaum; Tweney, R. D., Heiman, G. H., and Hoemann, H. W. 1977. Effects of visual disruption on sign intelligibility. J. Exp. Psycol. Gen.106,255. Buhmann, J., Divko, R., and Schulten, K. 1989. Associative memory with high information content. Phys. Rev. A39,2689. Buhmann, J.,and Schulten, K. 1987. Noise driven associations in neural networks. Europhys. Lett. 4, 1205. Hopfield, J. J. 1982. Neural networks and physical systems with emergent selective computational abilities. Proc. Natl. A d . Sci. USA. 79,2554. Miyashita, Y. 1988. Neuronal correlate of visual associative long-term memory in the primate temporal cortex. Nature (London) 335,817. Miyashita, Y.,and Chang, H. S. 1988. Neuronal correlate of pictorial short-term memory in the primate temporal cortex. Nature (London) 331,68. Rubin, N. 1991. Private communication.
Conversion of Temporal to Spatial Correlations
17
Sakai, K., and Miyashita, Y. 1991. Neural organization for the long-term memory of paired associates. Nature (London) 354, 152. Tsodyks, M. V., and Feigel'man, M. V. 1988. The enhanced storage capacity in neural networks with low activity level. Europhys. Lett. 46, 101.
Received 21 January 1992; accepted 5 May 1992.
This article has been cited by: 1. Valentina Daelli, Alessandro Treves. 2010. Neural attractor dynamics in object recognition. Experimental Brain Research 203:2, 241-248. [CrossRef] 2. Nicolas Brunel, Frédéric Lavigne. 2009. Semantic Priming in a Cortical Network ModelSemantic Priming in a Cortical Network Model. Journal of Cognitive Neuroscience 21:12, 2300-2319. [Abstract] [Full Text] [PDF] [PDF Plus] 3. F L Metz, W K Theumann. 2009. Symmetric sequence processing in a recurrent neural network model with a synchronous dynamics. Journal of Physics A: Mathematical and Theoretical 42:38, 385001. [CrossRef] 4. Tomoyuki Kimoto, Tatsuya Uezu, Masato Okada. 2008. Multiple Stability of a Sparsely Encoded Attractor Neural Network Model for the Inferior Temporal Cortex. Journal of the Physical Society of Japan 77:12, 124002. [CrossRef] 5. F. Metz, W. Theumann. 2007. Period-two cycles in a feedforward layered neural network model with symmetric sequence processing. Physical Review E 75:4. . [CrossRef] 6. Sawako Tanimoto, Masato Okada, Tomoyuki Kimoto, Tatsuya Uezu. 2006. Distinction of Coexistent Attractors in an Attractor Neural Network Model Using a Relaxation Process of Fluctuations in Firing Rates -- Analysis with Statistical Mechanics --. Journal of the Physics Society Japan 75:10, 104004. [CrossRef] 7. Masato Okada. 2006. Part 3: Brain science, information science and associative memory model. New Generation Computing 24:2, 185-201. [CrossRef] 8. F. Metz, W. Theumann. 2005. Pattern reconstruction and sequence processing in feed-forward layered neural networks near saturation. Physical Review E 72:2. . [CrossRef] 9. Tatsuya Uezu, Aya Hirano, Masato Okada. 2004. Retrieval Properties of Hopfield and Correlated Attractors in an Associative Memory Model. Journal of the Physics Society Japan 73:4, 867-874. [CrossRef] 10. Gianluigi Mongillo, Daniel J. Amit, Nicolas Brunel. 2003. Retrospective and prospective persistent activity induced by Hebbian learning in a recurrent cortical network. European Journal of Neuroscience 18:7, 2011-2024. [CrossRef] 11. Kazushi Mimura, Tomoyuki Kimoto, Masato Okada. 2003. Synapse efficiency diverges due to synaptic pruning following overgrowth. Physical Review E 68:3. . [CrossRef] 12. Masahiko Yoshioka, Masatoshi Shiino. 2000. Associative memory storing an extensive number of patterns based on a network of oscillators with distributed natural frequencies in the presence of external white noise. Physical Review E 61:5, 4732-4744. [CrossRef] 13. Kaname Toya, Kunihiko Fukushima, Yoshiyuki Kabashima, Masato Okada. 2000. Journal of Physics A: Mathematical and General 33:14, 2725-2737. [CrossRef]
14. Tomoki Fukai, Tomoyuki Kimoto, Makoto Doi, Masato Okada. 1999. Journal of Physics A: Mathematical and General 32:30, 5551-5562. [CrossRef] 15. Masahiko Yoshioka, Masatoshi Shiino. 1998. Associative memory based on synchronized firing of spiking neurons with time-delayed interactions. Physical Review E 58:3, 3628-3639. [CrossRef] 16. Néstor Parga , Edmund Rolls . 1998. Transform-Invariant Recognition by Association in a Recurrent NetworkTransform-Invariant Recognition by Association in a Recurrent Network. Neural Computation 10:6, 1507-1525. [Abstract] [PDF] [PDF Plus] 17. Asohan Amarasingham , William B. Levy . 1998. Predicting the Distribution of Synaptic Strengths and Cell Firing Correlations in a Self-Organizing, Sequence Prediction ModelPredicting the Distribution of Synaptic Strengths and Cell Firing Correlations in a Self-Organizing, Sequence Prediction Model. Neural Computation 10:1, 25-57. [Abstract] [PDF] [PDF Plus] 18. Daniel J. Amit, Stefano Fusi, Volodya Yakovlev. 1997. Paradigmatic Working Memory (Attractor) Cell in IT CortexParadigmatic Working Memory (Attractor) Cell in IT Cortex. Neural Computation 9:5, 1071-1092. [Abstract] [PDF] [PDF Plus] 19. Nicolas Brunel. 1996. Hebbian Learning of Context in Recurrent Neural NetworksHebbian Learning of Context in Recurrent Neural Networks. Neural Computation 8:8, 1677-1710. [Abstract] [PDF] [PDF Plus] 20. Hans Liljenström. 1996. Neuromodulation can significantly change the dynamical state of cortical networks. Behavioral and Brain Sciences 19:02, 303. [CrossRef] 21. Paul L. Nunez. 1996. Multiscale neocortical dynamics, experimental EEG measures, and global facilitation of local cell assemblies. Behavioral and Brain Sciences 19:02, 305. [CrossRef] 22. J. J. Wright, D. T. J. Liley. 1996. Dynamics of the brain at global and microscopic scales: Neural networks and the EEG. Behavioral and Brain Sciences 19:02, 285. [CrossRef] 23. Ichiro Tsuda. 1996. The form of chaos in the noisy brain can manifest function. Behavioral and Brain Sciences 19:02, 309. [CrossRef] 24. Valerie Gray Hardcastle. 1996. Modeling for modeling's sake?. Behavioral and Brain Sciences 19:02, 299. [CrossRef] 25. Andrew Oliver. 1996. Dynamics of the brain — from the statistical properties of neural signals to the development of representations. Behavioral and Brain Sciences 19:02, 306. [CrossRef] 26. Hubert Preissl, Werner Lutzenberger, Friedemann Pulvermüller. 1996. Is there chaos in the brain?. Behavioral and Brain Sciences 19:02, 307. [CrossRef] 27. Márk Molnár. 1996. Chaos in induced rhythms of the brain – the value of ERP studies. Behavioral and Brain Sciences 19:02, 305. [CrossRef]
28. Harry R. Erwin. 1996. Multiscale modeling of the brain should be validated in more detail against the biological data. Behavioral and Brain Sciences 19:02, 297. [CrossRef] 29. M. N. Zhadin. 1996. Rhythmicity in the EEG and global stabilization of the average level of excitation in the cerebral cortex. Behavioral and Brain Sciences 19:02, 309. [CrossRef] 30. Zbigniew J. Kowalik, Andrzej Wrobel, Andrzej Rydz. 1996. Why does the human brain need to be a nonlinear system?. Behavioral and Brain Sciences 19:02, 302. [CrossRef] 31. Robert Miller. 1996. Empirical data base for simulation: Firing rates and axonal conduction velocity for cortical neurones. Behavioral and Brain Sciences 19:02, 304. [CrossRef] 32. J. J. Wright, D. T. J. Liley. 1996. Multiscale modeling of brain dynamics depends upon approximations at each scale. Behavioral and Brain Sciences 19:02, 310. [CrossRef] 33. Walter S. Pritchard. 1996. The EEG data indicate stochastic nonlinearity. Behavioral and Brain Sciences 19:02, 308. [CrossRef] 34. Edgar Koerner. 1996. Comparative reduction of theories — or over-simplification?. Behavioral and Brain Sciences 19:02, 301. [CrossRef] 35. Lester Ingber. 1996. Nonlinear nonequilibrium nonquantum nonchaotic statistical mechanics of neocortical interactions. Behavioral and Brain Sciences 19:02, 300. [CrossRef] 36. Theodore H. Bullock. 1996. Is the distribution of coherence a test of the model?. Behavioral and Brain Sciences 19:02, 296. [CrossRef] 37. Péter Érdi. 1996. Levels, models, and brain activities: Neurodynamics is pluralistic. Behavioral and Brain Sciences 19:02, 296. [CrossRef] 38. Daniel J. Amit. 1996. Is the time ripe for integration of scales?. Behavioral and Brain Sciences 19:02, 295. [CrossRef] 39. Walter J. Freeman. 1996. Neural system stability. Behavioral and Brain Sciences 19:02, 298. [CrossRef] 40. Xiangbao Wu, Robert A. Baxter, William B. Levy. 1996. Context codes and the effect of noisy learning on a simplified hippocampal CA3 model. Biological Cybernetics 74:2, 159-165. [CrossRef] 41. Ehud Ahissar. 1995. Are single-cell data sufficient for testing neural network models?. Behavioral and Brain Sciences 18:04, 626. [CrossRef] 42. Masahiko Morita. 1995. Another ANN model for the Miyashita experiments. Behavioral and Brain Sciences 18:04, 639. [CrossRef] 43. Ralph E. Hoffman. 1995. Additional tests of Amit's attractor neural networks. Behavioral and Brain Sciences 18:04, 634. [CrossRef]
44. Shimon Edelman. 1995. How representation works is more important than what representations are. Behavioral and Brain Sciences 18:04, 630. [CrossRef] 45. Maartje E. J. Raijmakers, Peter C. M. Molenaar. 1995. How to decide whether a neural representation is a cognitive concept?. Behavioral and Brain Sciences 18:04, 641. [CrossRef] 46. Daniel J. Amit. 1995. The Hebbian paradigm reintegrated: Local reverberations as internal representations. Behavioral and Brain Sciences 18:04, 617. [CrossRef] 47. Walter J. Freeman. 1995. The Hebbian paradigm reintegrated: Local reverberations as internal representations. Behavioral and Brain Sciences 18:04, 631. [CrossRef] 48. David C. Krakauer, Alasdair I. Houston. 1995. An evolutionary perspective on Hebb's reverberatory representations. Behavioral and Brain Sciences 18:04, 636. [CrossRef] 49. Joaquin M. Fuster. 1995. Not the module does memory make – but the network. Behavioral and Brain Sciences 18:04, 631. [CrossRef] 50. Peter M. Milner. 1995. Attractors – don't get sucked in. Behavioral and Brain Sciences 18:04, 638. [CrossRef] 51. Friedemann Pulvermüller, Hubert Preissl. 1995. Local or transcortical assemblies? Some evidence from cognitive neuroscience. Behavioral and Brain Sciences 18:04, 640. [CrossRef] 52. J. J. Wright. 1995. How do local reverberations achieve global integration?. Behavioral and Brain Sciences 18:04, 644. [CrossRef] 53. Elie Bienenstock, Stuart Geman. 1995. Where the adventure is. Behavioral and Brain Sciences 18:04, 627. [CrossRef] 54. Frank der van Velde. 1995. Association and computation with cell assemblies. Behavioral and Brain Sciences 18:04, 643. [CrossRef] 55. Wolfgang Klimesch. 1995. The functional meaning of reverberations for sensoric and contextual encoding. Behavioral and Brain Sciences 18:04, 636. [CrossRef] 56. Eric Chown. 1995. Reverberation reconsidered: On the path to cognitive theory. Behavioral and Brain Sciences 18:04, 628. [CrossRef] 57. Jean Petitot. 1995. The problems of cognitive dynamical models. Behavioral and Brain Sciences 18:04, 640. [CrossRef] 58. G. J. Dalenoort, P. H. de Vries. 1995. What's in a cell assembly?. Behavioral and Brain Sciences 18:04, 629. [CrossRef] 59. Josef P. Rauschecker. 1995. Reverberations of Hebbian thinking. Behavioral and Brain Sciences 18:04, 642. [CrossRef] 60. Michael Hucka, Mark Weaver, Stephen Kaplan. 1995. Hebb's accomplishments misunderstood. Behavioral and Brain Sciences 18:04, 635. [CrossRef] 61. Anders Lansner, Erik Fransén. 1995. Distributed cell assemblies and detailed cell models. Behavioral and Brain Sciences 18:04, 637. [CrossRef]
62. Morris W. Hirsch. 1995. Mathematics of Hebbian attractors. Behavioral and Brain Sciences 18:04, 633. [CrossRef] 63. Daniel J. Amit. 1995. Empirical and theoretical active memory: The proper context. Behavioral and Brain Sciences 18:04, 645. [CrossRef] 64. W Whyte, D Sherrington, A C C Coolen. 1995. Journal of Physics A: Mathematical and General 28:12, 3421-3437. [CrossRef] 65. Michel Kerszberg, Claudine Masson. 1995. Signal-induced selection among spontaneous oscillatory patterns in a model of honeybee olfactory glomeruli. Biological Cybernetics 72:6, 487-495. [CrossRef] 66. Tomoki Fukai. 1995. A model cortical circuit for the storage of temporal sequences. Biological Cybernetics 72:4, 321-328. [CrossRef] 67. Iris Ginzburg, Haim Sompolinsky. 1994. Theory of correlations in stochastic neural networks. Physical Review E 50:4, 3171-3191. [CrossRef] 68. L. F. Cugliandolo . 1994. Correlated Attractors from Uncorrelated StimuliCorrelated Attractors from Uncorrelated Stimuli. Neural Computation 6:2, 220-224. [Abstract] [PDF] [PDF Plus] 69. L F Cugliandolo, M V Tsodyks. 1994. Journal of Physics A: Mathematical and General 27:3, 741-756. [CrossRef]
NOTE
Communicated by Hal White
On the Realization of a Kolmogorov Network Ji-Nan Lin Rolf Unbehauen LehrstuhlfUr Allgemeine und TheoretischeElektrotechnik, Universitdt Erlangen-Niirnberg,Cauerstrasse 7, D-8520 Erlangen, Germany
It has been suggested that the theorem by Kolmogorov (1957) about the multivariate function representation in the form (0.1)
with Q 2 2N + 1 provides theoretical support for neural networks that implement multivariate mappings (Hecht-Nielson 1987; Lippmann 1987). Girosi and Poggio (1989) criticized Kolmogorov's theorem as irrelevant. They based their criticism mainly on the fact that the inner functions p,,. are highly nonsmooth and the output functions gq are not in a parameterized form. However, this criticism was not convincing: Kurkova (1991) argued that highly nonsmooth functions can be regarded as limits or sums of infinite series of smooth functions, and the problems in realizing a Kolmogorov network can be eliminated through approximately implementing (~4. and g, with known networks. In this note we present our view on the discussion from a more essential point of view. Since P(., in equation 0.1 should be universal, Kolmogorov's theorem can be regarded as a proof of a transformation of representation of multivariate functions in terms of the Q univariate output functions g,. [In some improved versions of Kolmogorov's theorem it is proved that only one g in equation 0.1 is necessary (Lorentz 19661.1 Such a strategy is embedded in the network structure as shown in Figure 1. (Note that the block T is independent off.) If Figure 1 is thought of as a general network structure for approximation of multivariate functions, a question is whether an arbitrarily given multivariate function f can be (approximately) implemented through an (approximate) implementation of the corresponding Q univariate functions g,. To this question we have an answer as stated below: Proposition. In Figure 1, an approximate implementation of gq does not in general deliver an approximate implementation of the original function f, unless g, can be exactly implemented. Neural Computation 5,18-20 (1993)
@ 1993 Massachusetts Institute of Technology
Realization of a Kolmogorov Network
19
: IT1 Figure 1: The basic strategy in Kolmogorov’s theorem is embedded in a network structure where a universal transformation T maps the multidimensional Euclidean space (the domain of multivariate functions) into one or several unidimensional ones. Here we mean by function approximation a mechanism that provides an estimation of the corresponding output (function value) at an arbitrary point in the input space (domain), which is meaningful in some sense (e.g., an interpolation of some given samples). Such a procedure is closely related to the “spatial relation’’ (e.g., the Euclidean distance) defined in the input space. For instance, from the viewpoint of interpolation, an estimation of the function value at a point depends on its position relative to that of the sample points in the domain, as well as on the sample values. As we know, there does not exist a homeomorphism between a multidimensional Euclidean space and an interval in the real line, i.e., there does not exist a way to map points from the former into the latter while preserving the spatial relations between them. (It is due to this nonhomeomorphism that the inner functions vqnmust be highly nonsmooth;) That means, approximating gq in the real line may not have an equivalent meaning to that of approximating the original function f in a multidimensional space. For instance, a ”reasonable” interpolation in an interval in the real line between two samples of a gq may lose its meaning in the corresponding regions in the multidimensional domain off. In fact, there exist various methods of one-to-one mapping from a multidimensional space to a unidimensional one. (The construction of the inner functions vqnin Kolmogorov‘s theorem is only one of them.) Therefore, it is not difficult to understand that the structure in Figure 1
20
Ji-Nan Lin and Rolf Unbehauen
is, theoretically, for an exact representation of multivariate functions. If the uni-input subnetwork in Figure 1 are assumed to be adjustable for each input point, that is, they are able to provide for each input point an arbitrarily desired output independent of the other points, then g, and thus f can be exactly implemented by the network. However, such an assumption has little significance in practice. On the other hand, from the viewpoint of information theory, the implementation of the univariate functions g, does not mean a simplification of that of the original multivariate one f , since the universality of the inner structure T implies that all the information describing f should be carried by 8,. Based on the above discussion, we believe that Kolmogorov’s theorem is irrelevant to neural networks for mapping approximation. The discussion is illustrated by a network model with an example of function approximation (Lin and Unbehauen 1992). The consequence of our discussion is not encouraging for efforts toward constructing mapping approximation networks along the lines of Kolmogorov’s theorem and its proofs. However, if we take equation 0.1 as a general approximation representation of multivariate functions in terms of summation and superposition of univariate functions, it is closely relevant to mapping networks. Some useful neural networks (e.g., the perceptron type network) can be represented by equation 0.1.
References Girosi, F., and Poggio, T. 1989. Representation properties of networks: Kolmogorov’s theorem is irrelevant. Neural Comp. 1,465469. Hecht-Nielson,R. 1987. Kolmogorov‘s mapping neural network existence theorem. In Proceedings of the International Conference on Neural Networks, pp. 111, 11-14. IEEE. Kolmogorov, A. N. 1957. On the representation of continuous functions of several variables in the form of a superposition of continuous functions of one variable and addition. Dokl. Akad. Nauk USSR 114(5),953-956. Kurkova, V. 1991. Kolmogorov’s theorem is relevant. Neural Comp. 3, 617-622. Lin, J.-N., and Unbehauen, R. 1992. A simplified model for understanding the Kolmogorov network. In Proceedings URSl Int. Symp. on Signals, Systems and Electronics, ISSSE 92, Paris, 11-14. Lippmann, R. P. 1987. An introduction to computing with neural nets. I E E E ASSP Mag. 4-22. Lorentz, G. G. 1966. Approximation offunctions, Holt, Rinehart & Winston, New York. Received 6 February 1992; accepted 5 June 1992.
This article has been cited by: 1. Allan Pinkus. 1999. Approximation theory of the MLP model in neural networks. Acta Numerica 8, 143. [CrossRef] 2. Yu G. Smetanin. 1998. Neural networks as systems for recognizing patterns. Journal of Mathematical Sciences 89:4, 1406-1457. [CrossRef] 3. Mohammad Bahrami. 1995. Issues on representational capabilities of artificial neural networks and their implementation. International Journal of Intelligent Systems 10:6, 571-579. [CrossRef]
Communicated by Lawrence Abbott
Statistical Mechanics for a Network of Spiking Neurons Leonid Kruglyak' William Bialek Department of Physics, and Department of Molecular and Cell Biology, University of California at Berkeley, Berkeley, CA 94720 USA and NEC Research Institute, 4 Independence Way, Princeton, N]08540 USA
We show that a simple statistical mechanics model can capture the collective behavior of large networks of spiking neurons. Qualitative arguments suggest that regularly firing neurons should be described by a planar "spin" of unit length. We extract these spins from spike trains and then measure the interaction Hamiltonian using simulations of small clusters of cells. Correlations among spike trains obtained from simulations of large arrays of cells are in quantitative agreement with the predictions from these Hamiltonians. We comment on the novel computational abilities of these "XY networks."
1 Introduction Understanding the computations performed by biological nervous systems requires methods for describing the collective behavior of large networks of neurons. For a physicist, the natural tool is statistical mechanics. The development of the Hopfield model (Hopfield 1982) and its relation to king spin glasses have resulted in the many recent efforts along these lines (Amit 1989). While these neural network studies have produced a number of interesting results, they have been based on extremely simplified models of neurons-often just two-state devices. The hope has been that most microscopic biological details are not important for the collective computational behavior of the system as a whole. Renormalization group ideas have taught us that microscopic details are often irrelevant, but in the neural case it is unclear exactly where to draw the line between important features and incidental details. Most real neurons produce trains of identical action potentials, or spikes, and it is the timing of these spikes that carries information; in many cases significant computations are carried out on time scales comparable to the interspike intervals (de Ruyter van Steveninck and Bialek 1988; Bialek et al. 1991). 'Present address: Theoretical Physics, Oxford University, 1 Keble Rd., Oxford OX1 3NP.
Neural Computation 5,21-31 (1993)
@ 1993 Massachusetts Institute of Technology
22
Leonid Kruglyak and William Bialek
How do we relate the observable sequences of spikes to the local spins or fields in a statistical mechanics model? In this paper we begin with a semirealistic model for spiking neurons in a network and systematically construct a corresponding statistical mechanics model for interacting spins. This model is successful in that it accurately describes the correlations among spike trains observed in simulations of large arrays of interconnected neurons. The construction is largely numerical, and while we can offer analytical explanations only for some of its successes, we believe that the numerical results themselves are of considerable interest. We emphasize that the goal of this work is not to model a particular part of the nervous system but rather to show that an explicit reduction of a network of spiking cells to a statistical mechanics spin system is possible.
2 A Model for Spiking Cells
We use the Fitzhugh-Nagumo (FN) model (Fitzhugh 1961; Nagumo et al. 1962) to describe the electrical dynamics of an individual neuron. This model demonstrates a threshold for firing action potentials, a refractory period, and single-shot as well as repetitive firing-in short, all the qualitative properties of neural firing. It is also known to provide a reasonable quantitative description of several cell types (Rinzel and Ermentrout 1989; Fitzhugh 1969). To be realistic it is essential to inject into each cell a noise current bin(t ) , which we take to be gaussian, spectrally white, and independent in each cell n. We model a synapse between two neurons by exponentiating the voltage from one and injecting it as current into the other. Our choice is motivated by the fact that the number of transmitter vesicles released at a synapse is exponential in the presynaptic voltage (Aidley 1971); other synaptic transfer characteristics, including small delays, give results qualitatively similar to those described here. We emphasize that all of our methods for analyzing the FN model with exponential synapses have been applied to other models of spike generation with essentially identical results (Kruglyak 1990). In particular, both a simpler "integrate and fire" model (Gerstner 1991) and a more complicated channel model (Rinzel and Ermentrout 1989; Morris and Lecar 1981) in which regular bursts of spikes play the same role as the spikes do in the FN model were examined in detail. The equations of motion for a network of interacting noisy FN neurons are
(2.1)
Network of Spiking Neurons
23
where V , is the transmembrane voltage in cell n, l o is the dc bias current, and the W,,are auxiliary variables; VOsets the scale of voltage sensitivity in the synapse. Voltages and currents are dimensionless, and the parameters of the system are expressed in terms of the time constants T~ and T~ and a dimensionless ratio a. The FN model with noise has the standard sigmoidal input/output relation if one plots the firing rate vs. dc injected current in a single cell.' Most neural network models take this simple relation to be the central feature of neural firing. However, the sigmoidal i/o relation hides the experimentally salient distinction between regular and irregular patterns of firing. Regular firing is characterized by a tightly bunched interspike interval distribution; in the irregular case the distribution is approximately Poisson. These different regimes correspond to different parameter values in the model, and experiments on sensory neurons suggest that many cells are confined to one regime or the other under natural conditions (Teich and Khanna 1985; Goldberg and Fernandez 1971). Networks of neurons in the two regimes should exhibit very different collective properties. We study in detail the case of regular firing. If the intervals between spikes cluster tightly about a mean value, the firing is nearly periodic, and we expect that it should be describable by an underlying oscillation. Note that we are not talking about a perturbed oscillation of the noiseless FN equations; in fact, the simulations described below are carried out in a parameter region where those equations do not oscillate-the oscillations we see are noise-driven. Also, when cells are coupled together the interactions cannot be treated as a small effect since a single spike from one cell can trigger a spike in another cell. Hence the standard analytic methods for reducing complex nonlinear oscillations to phase equations (Kuramoto 1984) do not apply, and we have to follow our intuition and attempt to extract the underlying oscillation numerically.
3 From Spike Trains to Spins
We assume that spikes occur at times ti when the total phase of an oscillation crosses zero (mod27r). Then this oscillation has a mean frequency wo that is simply related to the mean interspike interval and a slowly varying phase 4(t) that describes the deviations from perfect periodicity. If this description is correct, the spike train s ( t ) = xi 6 ( t - t i ) should have a power spectrum with well resolved peaks at kwo, f2w0,.. .. This is what we observe in simulations of the FN model for one isolated cell. 'The noiseless behavior is quite different-there is a threshold current below which the rate is zem and above which it jumps to a finite value. There is also a second threshold current above which the cell ceases to fire.
Leonid Kruglyak and William Bialek
24
We then low-pass filter s ( t ) to keep only the f w o peaks, obtaining a phase and amplitude modulated cosine,
[Fs](t)x A ( t )C O S [ U ~+~ 4(t)]
(3.1)
where [Fs](t)denotes the filtered spike train, and the amplitude A ( t ) and the phase 4(t) vary slowly with time. Plotting the filtered spike train against its time derivative results in a phase portrait characteristic of a noisy sinusoidal oscillation, confirming equation 3.1. As indicated earlier, very similar phase portraits are obtained for the Gerstner and bursting Morris-Lccar models (Kruglyak 1990). Hence the spike train can be described by a two-vector in the phase plane. We do not expect the magnitude of this vector to matter since A(t) is related to the unimportant details of the filtering process and to the biologically irrelevant differences among spikes. The orientation of the vector, now assumed to be of unit length, gives us the phase. Using the phase portrait it is thus possible to process the spike train from a neuron and recover a time-dependent, planar unit spin S ( t ) . We now want to see how these spins interact when we connect two cells via synapses. 4 From Synapses to Spin-Spin Interactions
We characterize the two-neuron interaction by accumulating a histogram of the phase differences between two neurons connected via a model synapse? A variety of synaptic interactions have been examined; the results below though not the exact form of the interaction hold in every case. The probability distribution of the phase difference defines an effective Hamiltonian, P(&,&) o( exp[-H($l - 42)]. Note that this Hamiltonian is simply another way to characterize the equilibrium distributionit is not meant to describe the time evolution of the system. Hence it remains a useful concept even when the standard notion of an energy function breaks down. This is in contrast with the usual statistical mechanics approach to neural networks, which assumes a Liapunov dynamics for the noiseless case and then treats all noise by promoting the Liapunov function to the role of a true Hamiltonian and placing it at finite temperature. The assumption of Liapunov dynamics cannot be justified for biological networks, and the noise can be more complicated (Crair and Bialek 1990). Figure 1 shows the effective Hamiltonian for a pair of symmetrically connected cells. We see that with excitatory synapses (1 > 0) the interaction is ferromagnetic, as expected. Once again, the Hamiltonians for the Gerstner and bursting Morris-Lecar models show only minor variations from the FN Hamiltonian (Kruglyak 1990). *An interaction that depends only on the phase difference is only the simplest case; there could in principle be a dependence on the absolute phase of one of the cells as well. As mentioned below and described in detail elsewhere, no such dependence is seen when we look at probability distributions of phases in small clusters of cells.
Network of Spiking Neurons
25
Effective Hamiltontans 6.0
4.0
I
*r^2, $0
8.0
'
V
-0.0
0.0
P
2.0
4.0
h o(llmne.
Figure 1: Effective Hamiltonians for three values of the coupling strength I. From simulations of equation 3.1 for two cells with 71 = 0.1, Q = 10,Io = -0.25, Vo = 0.3,a = 1.1, and 61, of spectral density S, = 1.25 x With arrays of more than two neurons it is possible that the effective Hamiltonian includes more than just a simple nearest-neighbor interaction. We have searched for these effects in simulations on small clusters of cells, and always find that the observed phase histograms can be accounted for by appropriate convolutions of the histograms found in the two-neuron simulations. This leads us to predict that the statistical mechanics of an entire network will be described by the effective Hamiltonian H = CijHij(4i - 4j), where Hq(+i - 4j) is the effective Hamiltonian measured for a pair of connected cells i , j as in Figure 1. 5 Correlation Functions
One crucial consequence of equation 3.1 is that correlations of the filtered spike trains are exactly proportional to the spin-spin correlations that are the natural objects in statistical mechanics. Specifically, if we have two cells n and rn,
(Sn . S m ) = (cos(4n - 4m)) = ~,-~([FsnI(t)[FsmI(t))
(5.1)
This relation shows us how the statistical description of the network can be tested in experiments that monitor actual neural spike trains.
Lmnid Kruglyak and William Bialek
26
5.1 One Dimension. It is well known that when planar spins are connected in a one-dimensional chain with nearest-neighbor interactions, correlations between spins drop off exponentially with distance. The mapping from spike trains to spins predicts that this exponential decay will also be observed in the spike train correlation functions. To test this prediction we have run simulations on chains of 32 Fitzhugh-Nagumo neurons connected to their nearest neighbors. Correlations computed directly from the filtered spike trains as indicated above indeed decay exponentially, as seen in Figure 2. More complicated correlation functions are predicted and observed with, for example, delayed or asymmetric synaptic interactions (Kruglyak 1990). In Figure 3 we compare the correlation lengths predicted from the effective Hamiltonians determined as described above with the correlation lengths extracted from the realistic simulations; the agreement is excellent. We emphasize that while the “theoretical” correlation lengths are based on the statistical mechanics of a simple spin model, the “experimental” correlation lengths are based on measured correlations among spike trains generated by realistic neurons. There are no free parameters.
Correlation us. Distance i n 1 Damsns-ion
0.0
10.0
6.0 M - e
(n &tUw
16.0
zo.0
Spdnoa
Figure 2: Correlation function in 1 dimension for three values of the coupling strength (lowest curve, J = 0.0005, middle curve, J = 0.001, highest curve, J = 0.0015). The points are obtained from simulations of chains of neurons; the lines are best-fit exponentials.
Network of Spiking Neurons
27
6.0
4.0
!
j
3.0
E
i
2.0 1.0
0.0
0.0
1.0
2.0
3.0
4.0
6.0
Ihdlcted C m l o l l a Length
Figure 3: Correlation length obtained from fits to the simulation data vs. correlation length predicted from the Hamiltonians. 5.2 ’Itvo Dimensions. In the twodimensional case we connect each neuron to its four nearest neighbors on a square lattice. The corresponding spin model is essentially the XY model, but from Figure 1 we see that the interaction potential between neighbor spins has a somewhat different form, with the energy rapidly reaching a plateau as the spins tilt apart. The XY model itself is of course well understood (Kosterlitz and Thouless 1973; Nelson 1983), and models with potentials as in Figure 1 should be in the XY universality class. To get a feeling for the relation between the Hamiltonian in Figure 1 and the XY model we follow Jose et al. (1977) and carry out a MigdalKadanoff renormalization. We recall that for the XY model itself this approximation leads to an “almost” fixed line at low temperatures, while above the transition the flow is to infinite temperature. We find the same results in our model; the “almost” fixed line is the same and flow to infinite temperature is asymptotically along the same curve as in the XY case. The similarity of flows in the XY model and in our model is so great that we feel confident in predicting the conditions under which the network should exhibit algebraic or exponential decay of correlations. We then check these predictions directly by simulating two-dimensional arrays of neurons with toroidal boundary conditions. Figure 4 shows
Leonid Kruglyak and William Bialek
28
a
0.0
-2.0
I
-4.0
0.0
6.0
10.0
16.0
DLttonce tn LcJt(ca Spoclns.
b
..
-0.4
I
I 0.
_."
"tttt
-n
0.0
1.0
2.0
8.0
4.0
Figure 4: Conelation function in 2 dimensions. (a) Above the transition, loglinear plot; linear behavior indicates exponential fall-off. (b) Below the transition, log-log plot; linear behavior indicates algebraic fall-off. the spike train correlations as a function of distance for two coupling strengths, one below and one above the transition point suggested by the renonnalization calculation. The low coupling (high T ) data are taken on a 32 x 32 lattice and are well described by exponential fall-off, at least until the correlations are so small as to be lost in the statistical noise. In the strong coupling (low T ) case we study a 128 x 128 lattice and find that
Network of Spiking Neurons
29
the correlations first decay algebraically and then plateau and become irregular due to finite size effects. In the algebraic phase the long-distance behavior of the system should be describable in terms of spin waves (Kosterlitz and Thouless 1973). We can fit the data of Figure 4b, except at the shortest distances, with an exact lattice spin wave calculation in which the critical exponent r] x 0.2. This is reasonable since we are just below the apparent phase transition, 0.25. We have also looked at the effects of several kinds of where r] disorder (deleting a fraction of connections, randomly choosing coupling strengths, randomly assigning the inherent firing frequencies of the cells) and find at most a shift in the transition coupling with no apparent qualitative changes when the disorder is small (Kruglyak 1990; Kruglyak and Bialek 1991b). In particular, the phase with algebraic decay of correlations is preserved, albeit at stronger couplings. This result also holds for the other neural models described in Section 2. This robustness to changes in connectivity, connection strengths, and internal dynamics of neurons gives us confidence that the model discussed in this paper is applicable to real biological systems. 6 Discussion
To summarize, we have found a systematic procedure for extracting spin variables from spike trains for a particular class of realistic model neurons. We then measure an effective interaction Hamiltonian by simulating small clusters of cells. This allows us to formulate a statistical mechanics model for a network of spiking cells by directly referring to a more microscopic model rather than by simply postulating a coarsegrained description. We use correlations between spike trains to test the collective properties of networks predicted from the model and find that the predictions are both qualitatively and quantitatively correct. The fact that our particular network is described by an XY-like model is especially interesting. There are many computational problems, especially in vision, where one would like to make comparisons among signals carried by neurons separated by large physical distances in a given layer of the brain. It has been traditionally assumed that the range of possible comparisons is limited by the range of physical interconnections, which is typically quite short. Physically we know that this is wrong, since we can have long-range correlations near a critical point even when the microscopic interactions are short ranged. The difficulty is that the system must be carefully poised at the critical point, but this is not a problem in XY systems, which have a critical line and hence an entire regime of long-range correlations. Reasonably regular two-dimensional architectures are common in regions of the nervous system devoted to sensory information processing. In many of these systems one can observe neural responses to stim-
30
Leonid Kruglyak and William Bialek
uli that provide direct input only to very distant neurons (Allman et al. 1985);such responses are described as coming from ”beyond the classical receptive field.” The power-law decay of correlations in XY-like models may provide an apt description of the gradual decrease in responsiveness to more distant stimuli found in these experiments. If the XY scenario is applicable, we expect that these neurons should be regularly firing, and there is recent evidence for such oscillatory behavior in cortical cells that exhibit long-range responses (Gray and Singer 1989; Gray et al. 1989; Eckhorn et al. 1988). Simulations show that local temporal correlations in such networks can indeed carry information about large-scale spatial properties of the stimulus (Kruglyak 1990; Kruglyak and Bialek 1991a).
Acknowledgments We thank 0. Alvarez, D. Arovas, A. B. Bonds, K. Brueckner, M. Crair, E. Knobloch, and H. Lecar for helpful discussions. The work at Berkeley was supported in part by the National Science Foundation through a Presidential Young Investigator Award (to W. B.), supplemented by funds from Cray Research, Sun Microsystems, and the NEC Research Institute, by the Fannie and John Hertz Foundation through a Graduate Fellowship (to L. K.), and by the USPHS through a Biomedical Research Support Grant.
References Aidley, D. J. 1971. The Physiology of Excitable Cells. Cambridge University Press, Cambridge. Amit, D. J. 1989. Modeling Brain Function. Cambridge Universisty Press, Cambridge. Allman, J., Meizin, F., and McGuiness, E. 19851 Stimulus specific responses from beyond the classical receptive field: Neurophysiological mechanisms for local-global comparisons in visual neurons. Annu. Rev.Neurosci. 8,407. Bialek, W., Rieke, F., de Ruyter van Steveninck, R. R., and Warland, D. 1991. Reading a neural code. Science 252,1854. Crair, M. C., and Bialek, W. 1990. Non-Boltzmann dynamics in networks of spiking neurons. In Advances in Neural lnformation Processing Systems, 2, D. Touretzky, ed. Morgan Kaufmann, San Mateo, CA. de Ruyter van Steveninck, R. R., and Bialek, W. 1988. Real-time performance of a movement-sensitive neuron in the blowfly visual system: Coding and information transfer in short spike sequences. Proc. R. SOC.Lond. B 234,379. Eckhorn, R., Bauer, R., Jordan, W., Brosch, M., Kruse, W., Munk, M., and Reitboeck, H. J. 1988. Coherent oscillations: A mechanism of feature linking in the visual cortex? Biol. Cybern. 60,121. Fitzhugh, R. 1961. Impulses and physiological states in theoretical models of nerve membrane. Biophys. J. 1,445-466. Fitzhugh, R. 1969. Mathematical models of excitation and propagation in nerve. In Biologicul Engineering, H. P. Schwan, ed., Chap. 1. McGraw Hill, New York.
Network of Spiking Neurons
31
Gerstner, W. 1991. Associative memory in a network of “biological” neurons. In Advances in Neurallnformation Processing Systems, 3, D. Touretzky,ed. Morgan Kaufmann, San Mateo, CA. Goldberg, J. M., and Fernandez, C. 1971. Physiology of peripheral neurons innervating semicircularcanals of the squirrel monkey. 111: Variations among units in their discharge properties. 1.Neurophys. 34, 676. Gray, C. M., KONg, P., Engel, A. K., and Singer, W. 1989. Oscillatory responses in cat visual cortex exhibit inter-columnar synchronization which reflects global stimulus properties. Nature (London) 338,334. Gray, C. M., and Singer, W. 1989. Stimulus-specific neuronal oscillations in orientation columns of cat visual cortex. Proc. Natl. Acad. Sci. U.S.A. 86, 1698. Hopfield, J. J. 1982. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. U.S.A. 79,2554. Josh,J. V., Kadanoff, L. P., Kirkpatrick, S., and Nelson, D. R. 1977. Renormalization, vortices, and symmetry-breakingperturbations in the two-dimensional planar model. Phys. Rev.B 16,1217. Kruglyak; L. 1990. From biological reality to simple physical models: Networks of oscillating neurons and the XY model. Ph.D. thesis, University of California at Berkeley, Berkeley, CA. Kruglyak, L., and Bialek, W. 1991a. Analog computation at a critical point: A novel function for neuronal oscillations? In Advances in Neural Information Processing Systems, 3, D. Touretzky, ed. Morgan Kaufmann, San Mateo, CA. Kruglyak, L., and Bialek, W. 1991b. From biological reality to simple physical models: Networks of oscillating neurons and the XY model. In Neural Networks: From Biology to High Energy Physics, 0. Benhan, C. Bosio, P. Del Giudice, and E. Tabet, eds., ETS Editrice, Pisa, 1992. Kosterlitz, J. M., and Thouless, D. J. 1973. Ordering, metastability, and phase transitions in two-dimensional systems. J. Phys. C: Solid State Phys. 6,1181. Kuramoto, Y. 1984. Chemical Oscillations, Waves, and Turbulence. Springer, Berlin. Morris, C., and Lecar, H. 1981. Voltage oscillations in the barnacle giant muscle fiber. Biophys. J. 35, 193-213. Nagumo, J. S.,Arimoto, S., and Yoshizawa, S. 1962. An active pulse transmission line simulating a nerve axon. Proc. I. R. E . 50, 2061. Nelson, D. R. 1983. Defect-mediated phase transitions. In Phase Transitions and Critical Phenomena, C. Domb and J. L. Lebowitz, eds., Vol. 7, Chap. 1. Academic Press, London. Rinzel, J., and Ermentrout, G. B. 1989. Analysis of neural excitability and oscillations. In Methods in Neuronul Modeling, C. Koch and I. Segev, eds., Chap. 5. The MIT Press, Cambridge, MA. Teich, M. C., and Khanna, S. M. 1985. Pulse-number distribution for the neural spike train in the cat‘s auditory nerve. J.Acoust. SOC.Amer. 77, 1110. Received 3 September 1991; accepted 29 May 1992.
This article has been cited by: 2. Henry Tuckwell, Laurent Toubiana, Jean-Francois Vibert. 2000. Enhancement of epidemic spread by noise and stochastic resonance in spatial network models with viral dynamics. Physical Review E 61:5, 5611-5619. [CrossRef] 3. Henry Tuckwell, Laurent Toubiana, Jean-Francois Vibert. 1998. Spatial epidemic network models with viral dynamics. Physical Review E 57:2, 2163-2169. [CrossRef] 4. Roger Rodriguez, Henry Tuckwell. 1996. Statistical properties of stochastic nonlinear dynamical models of single spiking neurons and neural networks. Physical Review E 54:5, 5585-5590. [CrossRef]
Communicated by James Anderson
Acetylcholine and Learning in a Cortical Associative Memory Michael E. Hasselmo Department of Psychology, Harvard University, Cambridge, MA 02138 USA
Implementing associativememory function in biologically realistic networks raises difficulties not dealt with in previous associative memory models. In particular, during learning of overlapping input patterns, recall of previously stored patterns can interfere with the learning of new patterns. Most associative memory models avoid this difficulty by ignoring the effect of previously modified connections during learning, thereby clamping activity to the patterns to be learned. Here I propose that the effects of acetylcholine in cortical structures may provide a neurophysiological mechanism for this clamping. Recent brain slice experiments have shown that acetylcholine selectively suppresses excitatory intrinsic fiber synaptic transmission within the olfactory cortex, while leaving excitatory afferent input unaffected. In a computational model of olfactory cortex, this selective suppression, applied during learning, prevents interference from previously stored patterns during the learning of new patterns. Analysis of the model shows that the amount of suppression necessary to prevent interference depends on cortical parameters such as inhibition and the threshold of synaptic modification, as well as input parameters such as the amount of ovelc lap between the patterns being stored. 1 Introduction
A wide range of neural network models have suggested that associative memory function may depend on excitatory intrinsic connections within the cortex. These include both linear associative matrix memories (Anderson 1983; Kohonen 1984) and models related to spin glass systems (Hopfield 1982). However, the majority of these models have focused on network dynamics during the recall of previously stored memories. During learning of new memories, most associative memory models ignore the effect of intrinsic connections within the network by clamping the activity of units to the desired pattern, by computing synaptic modification independently of network dynamics, or by applying learning before the spread of activation. This allows use of the Hebb rule for computation Neural Computation 5 , 3 2 4 (1993)
@ 1993 Massachusetts Institute of Technology
Acetylcholine and Learning
33
of a clean outer product for each input pattern being stored. The sum of these outer products is computed for m different patterns and stored as an intrinsic excitatory connectivity matrix By as follows, where A!p) represents element i of pattern p:
Despite the common use of clamping in associative memory models, no neurophysiological mechanism has previously been presented describing how the brain might suppress normal network recall dynamics during learning. Without clamping, and with learning rates that are similar to or slower than the update of activation, recall of previously stored patterns will interfere with the learning of new patterns. This paper presents a neurophysiological mechanism that may prevent interference during learning in a cortical associative memory. 2 The Problem: Interference during
Learning
The learning rule in associative memory models is taken to be analogous to the phenomenon of long-term potentiation within cortical structures. However, biological evidence suggests that long-term potentiation cannot occur until the presynaptic activity (aj) has reached the terminal bouton and influenced postsynaptic activity (ai). Thus, associative memory models should not apply a learning rule until activity has propagated across the synapse being modified. Rather than applying the learning rule immediately, before the spread of activity, as in ABij(t) = ai(t)aj(t), learning should be applied only after synaptic transmission has had time to influence the postsynaptic activity, as in ABij(t 1) = ai(t l)aj(t). This presents a difficulty for associative memory models. Unless activity is clamped to the desired pattern during learning, activity will be influenced by intrinsic connections modified by previously stored patterns. Thus, when a new pattern is presented to the network, the activity will depend partly on previously learned patterns that overlap with the new pattern. For instance, after learning of one pattern A ( ] ) ,the connectivity matrix Bfi = A&')&(*).In this case, presentation of a second pattern A@)will result in postsynaptic activity ai combining the input pattern and the spread of activation along excitatory intrinsic connections:
+
+
+ where n = the number of neurons in the network. If this activity is taken as the postsynaptic activity in the Hebbian learning rule, and presynaptic
Michael E. Hasselmo
34
activity is taken as U j = A:), the modification of synaptic strength takes the form: ABij(t + 1) =
Ui(t
+ l ) ~ j ( =t )
Thus, in addition to the outer product of the pattern with itself, Ai@)Aj@), the connectivity matrix also contains the outer product between the two patterns stored, Ai(1)Aj(2), scaled to the dot product between these two patterns. For orthogonal patterns, the dot product equals zero, and no interference during learning occurs. However, for nonorthogonal patterns, interference during learning will occur between each new pattern p and all previously stored patterns q according to their direct overlap, and also according to the overlap between these patterns and intervening patterns stored within the network. This adds an interference term to the learning rule as follows:
[
AB!@ '1 = A!P)A!P) ' I + ql=l
91
( 2 A p ) A p ) )( 2 A p ) A p ) ) k=l
k=l
Coupled with the effect of each connection enhancing its own growth, this can lead to a positive feedback cycle, resulting in runaway synaptic modification. In simulations, if there is no set of patterns completely orthogonal to other sets of patterns, if all patterns are trained in parallel, and if the strength of connections saturates at some value, this interference can ultimately result in a connectivity matrix more closely resembling p=l q=l
This is useless for associative memory storage, since presentation of any one memory will recall elements of all memories stored within the network. While described here for linear associative memories, the effect of interference during learning also appears in associative memory models based on the spin-glass analogy (Hasselmo et al. 1991). In particular, interference will be even more severe if activation is updated for several time steps during learning. Modifications of the Hebbian learning rule have been used to decrease the level of interference between memories during recall, but unless they change the normal dynamics of the network during learning, they cannot prevent initial interference between
Acetylcholine and Learning
35
: ’ No cholinergic suppression Afferent fibers
/
Intrinsic fibers
L
Conml
IOOpM Carbschol
Conml
1O@MCubschol
Figure 1: Experimental results showing selective cholinergic suppression of intrinsic fiber synaptic transmission within the piriform cortex. Cholinergic agonists such as carbachol have little effect on synaptic potentials elicited by stimulation of afferent fibers arriving from the olfactory bulb, but strongly suppress synaptic potentials elicited by stimulation of intrinsic fibers arising from other pyramidal cells within the cortex. overlapping memories during learning. Thus, the covariance learning rule (Sejnowski and Stanton 19901, or learning rules incorporating decay (Kohonen 1984) can gradually decrease interference during parallel learning of all memories, but during sequential learning of each memory independently, interference will occur at least in the initial stages of learning a new pattern. Thus, some neurophysiological mechanism for clamping cortical activity to the input pattern would be necessary to completely prevent interference during learning in a cortical associative memory. 3 A Solution: Suppression of Synaptic Transmission by
Acetylcholine
I have recently developed a model of a neurophysiological mechanism for the clamping of network activity to the input pattern during learning. The initial motivation for this model arose from experiments in brain slice preparations of piriform (olfactory) cortex (Hasselmo and Bower, 19921, as illustrated in Figure 1. Piriform cortex contains clear laminar segrega-
36
Michael E. Hasselmo
tion of excitatory afferent and intrinsic fiber synapses, allowing selective study of synaptic transmission at these two sets of synapses. I have found that acetylcholine and cholinergic agonists such as carbachol selectively suppress synaptic potentials evoked by stimulation of the intrinsic fibers arising from other cortical pyramidal cells, while having almost no effect on synaptic potentials elicited by stimulation of afferent fibers from the olfactory bulb. This effect appears to be due to activation of presynaptic muscarinic cholinergic receptors (Hasselmo and Bower 1992). If applied during the learning of new patterns, this selective suppression of intrinsic fiber synaptic transmission can prevent interference between memories during learning. In the computational framework presented above, cholinergic suppression during learning would be comparable to homogeneously decreasing the strength of the intrinsic connectivity matrix B. In the following equations, this suppression of intrinsic fiber synaptic transmission will be represented by the coefficient c. Setting c = 1.O indicates maximal suppression of synaptic transmission and c = 0 indicates no suppression. Thus, the learning rule takes the form
In the above equation, setting c = 1.0 prevents interference during learning, and allows the clean computation of an outer product of the input pattern with itself, AF’A,?’. However, physiological evidence argues against a complete suppression of synaptic transmission during learning. First, even at high doses of cholinergic agonists, the suppression of synaptic transmission has a mean value of about 70% (Hasselmo and Bower 1992). Second, any associative memory function in this model depends on synaptic modification at the intrinsic fiber synapses that are being suppressed. As noted before, most theories of synaptic modification assume dependence on synaptic transmission at the synapses being modified. Thus, it appears unlikely that synaptic modification could occur if synaptic transmission were completely suppressed at intrinsic fiber synapses. However, as described in the next section, simulations of associative memory function in a model of olfactory cortex show that interference during learning can be prevented with submaximal suppression of synaptic transmission. This work is summarized below. 4 Computational Model of Olfactory Cortex
I have studied the effect of this cholinergic modulation in a model of piriform cortex as an associative memory network, as shown in Figure 2 (Hasselmo et al. 1991, 1992). In the model of piriform cortex, the initial input of pattern p sets the neuron activity a k ( t ) = Ak(P),and the activation
Acetylcholine and Learning
37
Figure 2: Schematic diagram of the computational model of piriform cortex. The model incorporates afferent fiber input from the olfactory bulb as well as excitatory intrinsic connections arising from other pyramidal cells. In addition, the model incorporates the effects of inhibitory interneurons mediating feedback inhibition. The output of the model is shown during recall in response to a degraded version of a previously learned input pattern (larger black boxes and darker shading of neurons represent greater activity). Neuron activity spreads along previously strengthened connections to complete missing elements of the input pattern.
of each neuron i is synchronously updated for one time step according to the equation n
Ui(f
I ) = A?’
n
+ k=lC(1- C)Bikg[&(t) - P] - k=lCHikg[ak(t)- P]
where u = neuron activation, A = the afferent input pattern, B = the intrinsic connectivity matrix, H = the feedback inhibition connectivity
38
Michael E. Hasselmo
matrix, p = the output threshold, c = the level of cholinergic suppression, and the input/output function g[ak(f)-p] = tanh[ak(t)-p]. For activation levels below p, output was zero. For levels above p, output increased to an asymptotic value usually set at 1.0. During recall, c is set at 0 (no suppression), while during learning c is set to values between 0 and 1.0, with 1.0 representing maximal suppression (no synaptic transmission). Learning occurs after one time step using a common adaptation of the Hebb rule reflecting direct dependence on postsynaptic depolarization, with fl = the postsynaptic threshold of synaptic modification, and 7 = the coefficient of learning:
In the computational model of piriform cortex, the selective suppression of intrinsic fiber synaptic transmission, applied during learning, prevents interference between overlapping input patterns being stored (Hasselmo et ul. 1991, 1992). The effects of acetylcholine can be seen clearly if we consider the recall of the model at different stages of learning, as shown in Figure 3. This figure shows the recall of the model in response to a degraded version (R)of a learned pattern (L) across 50 cycles of learning. Five overlapping input patterns are being stored within the network, but the figure shows recall of only one pattern. As seen on the top, without cholinergic suppression during learning, previously stored patterns interfere with the storage of new patterns. This interference has a positive feedback effect, which very rapidly causes the network to respond the same to all input patterns, with a broad, homogeneous level of activity containing components of all learned patterns. On the other hand, with 70% cholinergic suppression implemented in the model, previously stored patterns do not interfere with the storage of other patterns. Thus, the model learns to respond to the degraded input (R)with the full learned version of that input (L), completing the missing input lines. Some interference does appear during recall, but this is far more limited than what occurs without the effects of acetylcholine. 5 Cholinergic Suppression in a Linear Associative Memory
The amount of cholinergic suppression of intrinsic fiber synaptic transmission necessary to prevent interference during learning can be determined with an analytical description assuming linear associative memory properties. The starting point of this description is obtained by combining the activation equation and the learning rule for the model of olfactory cortex. In this case, the synaptic modification after one time step will take the form
Acetylcholine and Learning
39
Figure 3: The recall of the network is shown at different stages of learning without cholinergic suppression (0% suppression) and with 70% cholinergic suppression. Five overlapping patterns are being stored within the network, but recall is shown for only one pattern. The learned version of this pattern is shown above the letter L, with six active input lines (black boxes). The version of the pattern presented during recall is shown above the letter R, with only four active input lines. The output of the network in response to the recall pattern is shown after 0 to 50 learning cycles on the right (larger black boxes represent greater output). Without cholinergic suppression, the network eventually starts to respond to the recall pattern with elements of all learned patterns. With 70% cholinergic suppression, the network eventually responds to the recall pattern with completion of the missing components of the learned pattern, showing little interference from other learned patterns.
Michael E. Hasselmo
40
For purposes of simplification, we will take the input/output function g( ) as linear with slope = I, and threshold p = 0, and set ak(t) = Ak@)and 77 = 1. In addition, we will assume feedback inhibition is homogeneous and the input vectors all have the same length, so we can represent xi=,H&zk(t) - p] as the constant H. Thus, we obtain
ABv = A?)
+ e(1- c)BikAf”- H - a] A,?’) k=l
To represent learning of previous memories A(q)in a simplified form, we set the intrinsic connectivity matrix Bik = Cy=;=, Ai(q)Ak(q).Thus, for learning of a new memory p, the intrinsic matrix will be adjusted according to
This contains the clean outer product of pattern p with itself, plus an interference term scaled to the overlap with previously stored patterns 9. If we consider a network where the weights grow to a saturating value, sufficient learning will prevent the interference term from distorting storage of individual patterns when it takes values less than zero. However, for values greater than zero, interference will spread between patterns. Thus, the spread of interference during learning between patterns is prevented if
For a maximum value of A?) = 1, this condition is satisfied as long as cholinergic suppression is strong enough that
Thus, the level of suppression necessary to prevent interference between two memories decreases with higher levels of inhibition and a higher threshold of synaptic modification. Though this was derived using considerable simplifications of the model, the nonlinear computer simulation of piriform cortex shows qualitatively similar results. As shown in Figure 4, changing the strength of inhibition in the simulation changes the level of cholinergic suppression necessary to prevent interference. Notice from the equations, however, that as the values of inhibition and the threshold of synaptic modification increase, in addition to preventing interference between patterns, they can also directly impede the learning of the new pattern. Thus, the strength of these different parameters must be balanced. Fortunately, the simulation shows that interference can be prevented within a physiologically realistic range for the values of cholinergic suppression, inhibition and the threshold of synaptic modification.
Acetylcholine and Learning
41
Figure 4 Increasing the strength of inhibition within the model decreases the level of suppression necessary to prevent interference during learning. The recall of the model is shown with the same format as in Figure 3. On the left, the recall of the model is shown across 50 learning cycles at three different levels of cholinergic suppression (c = 0.30,0.50 and 0.701,with inhibition maintained at a strength of 0.05. On the right, the recall of the model is shown across 50 learning cycles at the same three levels of cholinergic modulation, with inhibition maintained at a strength of 0.10. Note that with stronger inhibition less cholinergic suppression is required to prevent interference during learning. The analysis presented here also shows that the level of cholinergic suppression necessary to prevent interference increases when there is a greater amount of overlap (larger sum of dot products) between the new pattern p being stored and all other patterns q previously stored in the network. This could be interpreted as a measure of the capacity of the network, beyond which interference during learning will occur. 6 Discussion
In a model of piriform cortex, cholinergic suppression of intrinsic fiber synaptic transmission (Hasselmo and Bower 1992) prevents previously
42
Michael E. Hasselmo
learned patterns from interfering with the storage of new overlapping patterns. This effect is similar to clamping the activity of the neurons to the desired pattern during learning. Thus, this provides the fist description of a neurophysiological mechanism for what has been a standard feature of learning in the majority of associative memory models (Anderson 1983; Kohonen 1984; Hopfield 1982). While the model presented here considers modification of excitatory intrinsic feedback connections within one cortical region, since the learning rule involves an association between presynaptic activity at time t and postsynaptic activity at time t + 1, the results apply equally well to modification of associational connections between different cortical regions receiving separate afferent input. Numerous studies have shown that muscarinic cholinergic antagonists impair memory function in a broad range of behavioral tasks (for a review, see Hagan and Moms 1989). In particular, much evidence supports the suggestion of the model that cholinergic blockade selectively impairs learning, while having less effect on the recall of previously learned information (Ghonheim and Mewaldt 1975). In addition, the prediction of the model that loss of cholinergic suppression should more strongly affect learning of overlapping input patterns is supported by results showing that muscarinic cholinergic antagonists cause greater difficulty in learning tasks with multiple or irrelevant cues, or tasks with less discriminable stimuli (Hagan and Morris 1989). The model presented here generates the prediction that loss of the cholinergic modulation of cortical structures should more strongly affect the learning of memories with overlapping components than the learning of memories without overlapping components. Experimental evidence on other neuropharmacological effects of acetylcholine supports the possibility that acetylcholine clamps cortical activity to the input pattern during learning. Acetylcholine has postsynaptic effects on cortical neurons, causing increased spiking response to synaptic stimulation and current injection (ffrench-Mullen et af. 1983; Hasselmo and Bower 1992; Hasselmo and Barkai 1992). These effects could enhance the response of cortical neurons to afferent input during the suppression of intrinsic fiber synaptic transmission, helping to clamp neurons to afferent input patterns. In the model presented here, insufficient suppression of synaptic transmission allows interference during learning to cause runaway synaptic modification. This pattern of breakdown may prove a useful model for the initiation and progression of cortical degeneration found in Alzheimer’s disease. In Alzheimer’s disease, the first regions to show degeneration in the form of neurofibrillary tangles are layer I1 of entorhinal cortex, region CA1 of the hippocampus, and the subiculum (Hyman et al. 1984). These regions have characteristics which would make them particularly vulnerable to runaway synaptic modification. In particular, the projection from layer I1 of the entorhinal cortex to the molecular
Acetylcholine and Learning
43
layer of the dentate gyrus shows strong associative long-term potentiation (McNaughton et al. 19781, but the outer molecular layer does not show cholinergic suppression of synaptic transmission (Kahle and Cotman 1989). In Alzheimer’s disease, tangles first appear at the cells of origin of this projection, and strong plaque deposits appear in the terminal field. While the specific causality of the neuropathology found in Alzheimer’s disease has not been determined, it is plausible that it could be due to excessive metabolic demands produced by the type of runaway synaptic modification found within the model, or to excitotoxic effects accompanying the excessive strengthening of excitatory synaptic connections. In this case, the analysis described here could explain the selective vulnerability of specific cortical subregions in terms of an imbalance of a range of cortical parameters, including inhibition, the threshold of synaptic modification, cholinergic modulation, and the relative overlap of input patterns arriving from other cortical regions. Acknowledgments This work was supported by a grant from the French Foundation for Alzheimer Research. I thank Matt Wilson for development of the simulation graphics, Brooke Anderson for early development of the olfactory cortex simulation, Ross Bergman for current programming support’ and Alan Yuille and Fred Waugh for comments on the manuscript. References Anderson, J. A. 1983. Cognitive and psychological computation with neural models. IEEE Trans. Systems, Man, Cybern. SMC-13,799-815. ffrench-Mullen, J. M. H., Hori, N., Nakanishi, H., Slater, N. T., and Carpenter, D. 0. 1983. Asymmetric distribution of acetylcholine receptors and M channels on prepyriform neurons. Cell. Mol. Neumbiol. 3, 163-182. Ghonheim, M.M.,and Mewaldt, S. I? 1975. Effects of diazepam and scopolamine on storage, retrieval, and organizational processes in memory. PsyChOpha~COlOgia44,257-262. Hagan, J. J., and Morris, R. G. M.1989. The cholinergic hypothesis of memory: A review of animal experiments. In Psychopharmacologyof the Aging Nervous System, L. L. Iversen, S. D. Iversen, and S. H. Snyder, eds., pp. 237-324. Plenum Press, New York. Hasselmo, M. E., and Barkai, E. 1992. Cholinergic modulation of the input/output function of rat piriform cortex pyramidal cells. SOC.Neurosci. Abstr. 18, 521. Hasselmo, M. E., and Bower, J. M. 1992. Cholinergic suppression specific to intrinsic not afferent fiber synapses in rat piriform (olfactory) cortex. 1. Neurophysiol. 67(5), 1222-1229.
44
Michael E. Hasselmo
Hasselmo, M. E., Anderson, B. P., and Bower, J. M. 1991. Cholinergic modulation may enhance cortical associative memory function. In Advances in Neurul 1nformutionProcessing Systems3, R. P.Lippman, J. Moody, and D. S. Touretsky, eds., pp. 46-52. Morgan Kaufmann, San Mateo, CA. Hasselmo, M. E., Anderson, B. P., and Bower, J. M. 1992. Cholinergic modulation of cortical associative memory function. 1.Neurophysiol. 67(5), 12301246. Hopfield, J. J. 1982. Neural networks and physical systems with emergent selective computational abilities. Proc. Nutl. Acud. Sci. U.S.A. 79,2554-2559. Hyman, B. T., Damasio, A. R., Van Hoesen, G. W., and Barnes, C. L. 1984. Cell specific pathology isolates the hippocampal formation in Alzheimer’s disease. Science 225, 1168-1170. Kahle, J. S., and Cotman, C. W. 1989. Carbachol depresses the synaptic responses in the medial but not the lateral perforant path. Bruin Res. 482, 159-163. Kohonen, T. 1984. Self-organitation and Associative Memory. Springer-Verlag, Berlin. McNaughton, B. L., Douglas, R. M., and Goddard, G. V. 1978. Synaptic enhancement in fascia dentata: Cooperativity among coactive afferents. Bruin Res. 157,277-293. Sejnowski, T. J.,and Stanton, P.K. 1990. Covariance storage in the hippocampus. In An Introduction to Neurul and Electronic Networks, S. Zornetzer, J. Davis, and C. Lau, eds., pp. 365-375. Academic Press, San Diego. Received 10 December 1991; accepted 12 May 1992.
This article has been cited by: 2. A. David Redish, Steve Jensen, Adam Johnson. 2008. A unified framework for addiction: Vulnerabilities in the decision process. Behavioral and Brain Sciences 31:04. . [CrossRef] 3. X. Zhang, A.A. Minai. 2004. Temporally Sequenced Intelligent Block-Matching and Motion-Segmentation Using Locally Coupled Networks. IEEE Transactions on Neural Networks 15:5, 1202-1214. [CrossRef] 4. Friedrich T. Sommer, Thomas Wennekers. 2003. Models of distributed associative memory networks in the brain. Theory in Biosciences 122:1, 55-69. [CrossRef] 5. Silvia Scarpetta , L. Zhaoping , John Hertz . 2002. Hebbian Imprinting and Retrieval in Oscillatory Neural NetworksHebbian Imprinting and Retrieval in Oscillatory Neural Networks. Neural Computation 14:10, 2371-2396. [Abstract] [PDF] [PDF Plus] 6. Gayle M. Wittenberg, Megan R. Sullivan, Joe Z. Tsien. 2002. Synaptic reentry reinforcement based network model for long-term memory consolidation. Hippocampus 12:5, 637-647. [CrossRef] 7. Manuel A. Sánchez-Montañés , Paul F. M. J. Verschure , Peter König . 2000. Local and Global Gating of Synaptic PlasticityLocal and Global Gating of Synaptic Plasticity. Neural Computation 12:3, 519-529. [Abstract] [PDF] [PDF Plus] 8. Jean-Marc Fellous, Christiane Linster. 1998. Computational Models of NeuromodulationComputational Models of Neuromodulation. Neural Computation 10:4, 771-805. [Abstract] [PDF] [PDF Plus] 9. Asnat Greenstein-Messica , Eytan Ruppin . 1998. Synaptic Runaway in Associative Networks and the Pathogenesis of SchizophreniaSynaptic Runaway in Associative Networks and the Pathogenesis of Schizophrenia. Neural Computation 10:2, 451-465. [Abstract] [PDF] [PDF Plus] 10. David Horn , Nir Levy , Eytan Ruppin . 1998. Memory Maintenance via Neuronal RegulationMemory Maintenance via Neuronal Regulation. Neural Computation 10:1, 1-18. [Abstract] [PDF] [PDF Plus] 11. A. David Redish, David S. Touretzky. 1997. Cognitive maps beyond the hippocampus. Hippocampus 7:1, 15-35. [CrossRef] 12. Christiane Linster, Claudine Masson . 1996. A Neural Model of Olfactory Sensory Memory in the Honeybee's Antennal LobeA Neural Model of Olfactory Sensory Memory in the Honeybee's Antennal Lobe. Neural Computation 8:1, 94-114. [Abstract] [PDF] [PDF Plus] 13. Paul E. Patton, Bruce McNaughton. 1995. Connection matrix of the hippocampal formation: I. The dentate gyrus. Hippocampus 5:4, 245-286. [CrossRef] 14. Hans Liljenström. 1995. Autonomous learning with complex dynamics. International Journal of Intelligent Systems 10:1, 119-153. [CrossRef]
15. James M. Bower. 1993. The modulation of learning state in a biological associative memory: an in vitro, in vivo, and in computo study of object recognition in mammalian olfactory cortex. Artificial Intelligence Review 7:5, 261-269. [CrossRef] 16. Barry Horwitz, Olaf Sporns. 1993. Neural modeling and functional neuroimaging. Human Brain Mapping 1:4, 269-283. [CrossRef]
Communicated by David Field
Convergent Algorithm for Sensory Receptive Field Development Joseph J. Atick The Rockefeller University, 1230 York Avenue, New York, NY 10021 U S A
A. Norman Redlich School of Natural Sciences, Institute for Advanced Study, Princeton, NJ 08540 U S A
An unsupervised developmental algorithm for linear maps is derived which reduces the pixel-entropy (using the measure introduced in previous work) at every update and thus removes pairwise correlations between pixels. Since the measure of pixel-entropy has only a global minimum the algorithm is guaranteed to converge to the minimum entropy map. Such optimal maps have recently been shown to possess cognitively desirable properties and are likely to be used by the nervous system to organize sensory information. The algorithm derived here turns out to be one proposed by Goodall for pairwise decorrelation. It is biologically plausible since in a neural network implementation it requires only data available locally to a neuron. In training over ensembles of two-dimensional input signals with the same spatial power spectrum as natural scenes, networks develop output neurons with centersurround receptive fields similar to those of ganglion cells in the retina. Some technical issues pertinent to developmental algorithms of this sort, such as "symmetry fixing," are also discussed. 1 Introduction
Recent theoretical results on neural processing support the idea that efficiency of information representation in the sensory pathways could have cognitive advantages [Barlow 1989 (+REFS); Linsker 1988; Atick and Redlich 19901. This is a predictive idea since it leads to the hypothesis that much of the processing in the early stages is geared toward recoding incoming sensory signals into a more efficient form. Starting with natural signals one can assess the efficiency of the sampled representation formed by the array of sensory cells and mathematically derive recodings that would improve the efficiency. These recodings can then be compared with the multistages of neural processing observed. Neural Computation
5,45-60 (1993)
@ 1993 Massachusetts Institute of Technology
46
Joseph J. Atick and A. Norman Redlich
One form of efficient representations that figures prominently in the recent literature is the so called "minimum entropy'' one,' where the sum of the individual entropies for the elements of the representation (e.g., pixels) is minimal for the ensemble of natural signals (Field 1987;Barlow 1989; Barlow et al. 1989;Atick and Redlich 1990,1992). This minimum is achieved when there is the least possible statistical dependence between elements. The idea that the nervous system could be engaged in trying to build such a minimum entropy representation of the environment has been tested in the limited context of retinal processing (Atick and Redlich 1992). There it was assumed that the retina, being the first stage in the visual pathway, could reduce pixel-entropy by eliminating no higher than two-point correlations (pairwise correlations). The linear transform on the photoreceptor activities needed to achieve pixel-pixel decorrelations was shown to agree with observed retinal filters-after being careful to take noise into account. In general, the problem of finding entropy reducing maps is very difficult. It is also very unlikely that one will be able to analytically solve for the explicit form of these maps as was done for the pairwise decorrelating map in the retina. An alternate approach is to use neural networks to try to compute these maps. What one needs are developmental algorithms that iteratively reduce statistical dependence among the elements of the representation as a network is trained over more sensory inputs, and preferably algorithms that are guaranteed to converge to the optimal maps. In this paper we derive a simple developmental algorithm, for the linear class of maps, which we prove lowers pixel-entropy at each learning stage. For this class of maps lowering pixel-entropy is equivalent to decreasing pairwise correlations at each step. The algorithm turns out to be identical to one originally introduced by Goodall (1960)who was interested in decorrelation in a different context. When introduced, Goodall's algorithm was proven-without reference to an entropy measureto converge to the solution that pairwise decorrelates. For us this old proof is an independent check of convergence since our entropy measure has no local minima. Therefore our demonstration that the Goodall algorithm successively lowers the entropy is sufficient in itself to prove it converges to the global minimum. This minimum entropy solution is the one that we have previously shown predicts the linear processing observed in the retina-after incorporating noise filtering. One major purpose of this paper is to demonstrate that these ganglion cell receptive fields can also be developed by applying the Goodall algorithm to an ensemble with the same second-order statistical properties (same power spectrum) as natural scenes. Also we expect insight gained from this simple algorithm to be helpful in discovering more complex algorithms capable of produc'In previous papers we have referred to this as minimum redundancy representation, since it eliminates the part of redundancy that is due to statistical dependence among the elements.
Algorithm for Sensory Receptive Field Development
47
ing minimum entropy representations for nonlinear maps that reduce statistical dependence beyond pairwise decorrelation. We should point out that there are several other pairwise decorrelating algorithms in the literature (Kohonen and Oja 1976; Oja 1982; Linsker 1986; Barlow and Foldiak 1989; Foldiak 1989; Sanger 1989; Rubner and Schulten 1990). However, the algorithm that we are presenting here is of particular interest to us since it is proven to reduce our previously introduced entropy measure. Another nice property of this algorithm is its locality in the sense that all of the data needed by a synapse to modify its strength is available at the input to the neuron. This means that the algorithm is at least plausibly one that might be implemented biologically. By actually attempting here to derive receptive fields based on a "natural'' input ensemble we are also forced to face an important issue that also arises in implementing similar decorrelation algorithms, but has not to our knowledge been resolved. The problem is that decorrelation itself does not guarantee a unique solution for the receptive fields, and most of the solutions are not localized. This problem can be traced to a large symmetry under which any decorrelating solution remains decorrelating. Here we theoretically analyze the requirements for fixing this symmetry and we find a simple way to do so that leads to localized receptive fields. 2 Entropy Reduction and Convergence
We begin by assuming that the sensory input signal {Si}, representing the set of photoreceptor responses, is recoded through a linear map Kq to the set of outputs { Oi}, which in the retina are the ganglion cell responses:
As in our previous work (Atick and Redlich 1990,19921, we introduce an "entropy" measure E{K}, that grades different recodings K according to how well they minimize the sum of pixel entropies, without overall loss of information:
E{K}
= Tr(K. R
. KT) - logdet(KT.K)
(2.2)
In 2.2 Rij = (SiSj) is the autocorrelator for the input ensemble, with brackets denoting ensemble average; boldface denotes matrices. Minimizing the first term was shown to be equivalent to minimizing the sum of the individual pixel entropies, while the second term acts to enforce reversibility of the map (no information loss). When the measure 2.2 is minimized, 6E/6K = 0, one gets a decorrelating solution K satisfying K . R.K~ = I
(2.3)
JosephJ. Atick and A. Norman Redlich
48
which when convoluted with a noise filter was shown to reproduce ganglion cell receptive fields (Atick and Redlich 1992). Although mathematically it is straightforward to find a decorrelating solution K satisfymg 2.3, it is not clear how a network of neurons can arrive at such a K. What is needed is a biologically plausible develop mental algorithm that can be proven to converge to a K satisfymg 2.3. One way to do this is to find a small update 6K for the map K that is guaranteed to lower E{K} at each step. This requires that the change in E{K} due to 6K must be negative:
:(
)
6E{K} = Tr - .6KT < 0
(2.4)
One obvious possibility is to use gradient descent 6KT = - (6E/6K)T. However this leads to a nonlocal algorithm that at each update stage requires the computation of an inverse matrix. We propose instead the update (2.5)
where from 2.2 6E - = 2 [K - R - (KT)-']
(2.6)
6K
Like gradient descent this update always reduces E{K} since by 2.4 and 2.5 6E{K} = - T r ( [ K . R . K T - 1 ] [ K . R . K T - 1 I T )
(2.7)
which is always negative (or zero upon convergence) since Tr(M.MT)> 0 for any matrix M # 0. (For the mathematically minded reader this shows that E can be thought of as a Lyapunov functional.) The next step is to show that the update 2.5 can be implemented as an algorithm requiring no nonlocal calculations. For this we rewrite 2.5 as an update for K-I instead of KT, using the fact 6K-' = -K-'.bK.K-'. For later convenience, we also change notation now and define W = K-I; the variables Wv will designate the actual synaptic strengths in the feedback network introduced in the next section. Then in terms of W the algorithm 2.5 becomes (2.8)
where T is a time constant setting the update rate. Although W-I does appear in 2.8, it appears in the combination (W-l)T, which as will be discussed in the next section is equal to (SiOj),and thus in the network Re
Algorithm for Sensory Receptive Field Development
49
implementation no nonlocal computations will be needed. The algorithm 2.8 can be recognized as the ensemble averaged form of the Goodall (1960) algorithm. Since direct minimization of E{K} in 2.2 gives the global minimum 2.3 with no local minima, our demonstrating that 2.8 or 2.5 always reduces E{K} is sufficient to prove convergence. However, for completeness we shall also give the old proof of convergence (Goodall 1960) that does not require a minimization measure. One reason we give this proof here is that it leads us to rewrite the update algorithm 2.8 in yet another form that will be useful in our later discussion of symmetry fixing and locality. To start the proof we need, in addition to 2.8, the transpose equation (2.9) using R = RT. By multiplying 2.8 from the right by WT and 2.9 from the left by W and then adding the resulting equations, we arrive at the integrable differential equation r dt d(W.WT)=2(R-W.WT)
(2.10)
which has the solution
where C is a constant matrix determined by initial conditions. In the limit where t / r becomes large We WT = R, which is just the minimum E{K} solution 2.3 since W = K-'. 3 Network Implementation
In this section we discuss a network implementation of the above algorithm. As shown in Figure 1 the network has one nontrivial layer, with feedforward connections directly from the input Si and lateral feedback connections among the neurons. It is assumed that the input Si is directly connected only to the ith neuron, with link strength unity. The plastic links in this network are the lateral feedback links that connect the output of the jth neuron back to the input of the ith neuron with link strength Wq. The neurons are assumed to be linear, thus the dynamics of this network can be written as
where Oi is the output of the ith neuron, Sj is its feedforward input, and
50
Joseph J. Atick and A. Norman Redlich
01
Figure 1: The architecture of the network used for decorrelation. It is assumed that the ith neuron receives direct input with synaptic weight unity from Si and feedback input from the outputs of all neurons with weights W,. Only the feedback links Wv are updated during learning.
T is a time constant. Below we shall assume that T << T for the update algorithm 2.8 so we can safely approximate the system by its equilibrium solution, dOi/dt = 0, si
=
c
wijoj
(3.2)
i
This shows that the matrix W, can be identified with the inverse of the retinal kernel Kij introduced above, so Wij-l corresponds to the ith ganglion cell’s receptive field. In terms of the network variables Si, Oj, Wi,, the algorithm 2.8 can be written as (3.3)
Algorithm for Sensory Receptive Field Development
51
where the brackets denote an ensemble average. To see that this is the same as 2.8 note that (SiOj) = Ck(SiSk)yil and (SiSj) = Rv. AS long as 7 is much greater than the characteristic time scale of the ensemble we can remove the averaging brackets in 3.3 and finally arrive at the algorithm that requires no a priori knowledge of any ensemble properties (no prior knowledge of R)
(3.4) Another significant fact about 3.4is that it is biologically plausible in the sense that all of the data, Si, Oj, Wv needed for the ith neuron to update its synaptic links, Wij, is available locally to that neuron (see Fig. 1). In Section 6 we simulate 3.4 for an ensemble of inputs with the same autocorrelator as natural scenes. But before we can do that we need to discuss the important technical issue of the nonuniqueness of the decorrelating solutions, and the related problem of nonlocalized receptive fields. 4 Finding Localized Receptive Fields
We would like next to apply the learning algorithm 3.4 to generate linear receptive fields for an ensemble of "natural" input images. However, if one naively employs this learning algorithm, the receptive fields very often turn out to be nonlocalized, and thus do not resemble those of retinal ganglion cells. The problem is that the algorithm 3.4or 2.8, though guaranteed to converge to a painvise decorrelating solution, W . W T= R, does not find a unique solution W, and most of the solutions are not localized. The source of the nonuniqueness is that W . WTpossesses a large symmetry, since any transform W -, W U of a solution W by any orthogonal matrix U (U. UT = 1) is also a solution. This means that there is a whole class of acceptable solutions parameterized by U. However, from a biological point of view, most of these solutions are unacceptable since they correspond to nonlocalized receptive fields. So the problem is to eliminate the extra symmetry U in a way that guarantees localization. In other words, we wish to remove the extra degrees of freedom in W due to this symmetry by constraining the form of W,while preserving its decorrelating property: W . WT= R. One type of constraint that does this is
-
W=WT
(4.1)
since there is no longer any freedom to multiply W by U without violating this condition. That this condition removes all the extra freedom follows from the fact that the number of independent components of the matrix
Joseph J. Atick and A. Norman Redlich
52
W that are eliminated is N(N - 1)/2, which is precisely the number of independent components in U. The reason for our choice of constraint 4.1 is that, for all autocorrelators R that we consider, 4.1 leads to localized, translation invariant receptive fields. To see this note that 4.1 implies that W satisfies the more restricted equation W W = R, which has (up to sign) a unique solution since the U symmetry has been eliminated. This means that if R is translation invariant so is W.2 Furthermore, translation invariance of W allows us to go to frequency space and write w ( f ) = R(f). Recall then that W(f) is the inverse of the receptive field kernel K(f), so K(f) = This means that K(f) will be localized so long as R(f) has the property that it is a sufficiently smooth function of frequency f. One can show that for the receptive field to be localized to an a p proximate size D, the smoothness criterion for K(f) [and thus implicitly for R(f)] is [aK(f)/df]/[DK(f)] << 1. This smoothness condition, for the appropriate D, is satisfied by the autocorrelators R(f) that we use here which are based on the properties of natural scenes. Now that we know what condition 4.1 we wish the solution W to satisfy once it converges, the next step is to find a way to guarantee that the learning algorithm 3.4 will actually converge to the specific W satisfying 4.1, rather than to one of the many other W related to it through multiplication by some U. In the next section we introduce a procedure for doing so that is well known to physicists as “gaugefixing” and that we call here “symmetry-fixing.” The discussion in that section is somewhat technical, so for the reader who is not interested in the details we state the conclusion here so that the next section can be skipped without loss of continuity: To guarantee that the system converges to a solution that satisfies 4.1 (i.e., localized solution) it turns out that it is suficient to start the algorithm at time zero with the initial condition W(t = 0) = 1 (this is sufficient but may not be necessary). While this is a very simple boundary condition, it is not obvious that it alone ensures convergence to configurations satisfying both W WT= R and W = WT,which is what we prove in the next section.
fdm.
5 Symmetry Fixing
Our approach here is to ask how the algorithm 3.4 can in general be forced to converge to a solution that satisfies some symmetry fixing constraint, say 4.1 that we argued in the last section ensures locality. To *This follows since the solution to W .W = R is unique. Therefore if a translation invariant solution is found it must be the only solution. 3An alternative way to fix the symmetry is to include in the developmental algorithm terms that attempt to produce statistical inde ndence beyond pairwise decorrelation (there is no proof yet that any development algorithm with higher order terms converges). This is assuming, of course, that higher order correlations break the U symmetry, but this is often the case (see, e.g., Hopfield 1991).
ape
Algorithm for Sensory Receptive Field Development
53
accomplish this we study both the possibility of using special initial conditions [such as W(t = 0) = 11 as well as the possibility of modlfylng the algorithm itself in a way that does not spoil its decorrelating or convergence properties. It will turn out that for the special case of the condition W = WT, no modification of the algorithm is needed if one uses the initial condition W(t = 0) = 1. However, to prove this requires that we look at the general symmetry fixing problem, where it will be necessary to modify the algorithm. One reason, also, that we explore the more general symmetry fixing problem is that the U symmetry is ubiquitous in the literature on decorrelating algorithms. Therefore, it is important to learn tools for handling this symmetry. Generally speaking, symmetry fixing is a procedure that introduces extra control over the dynamics for W in a way that automatically forces the system to converge to the W satisfying the desired constraint. The extra control as we explain next is introduced by modifying 2.8 in a way that does not interfere with the convergence proof. The key to the modification of the dynamics comes from recognizing that not only the solution W WT = R possesses the U symmetry, but also the dynamic equation 2.10, which is the fundamental equation ensuring convergence to the decorrelation solution, is invariant under the transformation W + W U(t) for any time-dependent orthogonal matrix U(t). This means that multiplication of W(t) by a time-dependent orthogonal transformation does not interfere with the convergence proof. However, it does modify the dynamics in 2.8 since that equation is not invariant under this transformation. In fact under W + W U(t) 2.8 transforms to m
dw
T-
dt
.
= R (W_')T- W + W
. dU - . UT dt
(5.1)
To see that the last term indeed drops out from the dynamic equation 2.10 for W . WT we need only to note that U . (dUT/dt) = - (dU/dt) .UT,valid since U . UT= 1. The next step is to show how the freedom to choose U(t) can be used to control the dynamics such that the system will converge to a W with a particular property, say WT = W. One might attempt to satisfy this constraint 4.1 on the solution W by starting with initial conditions that satisfy it. However, the original dynamic equation 2.8 very quickly would evolve W(t) to configurations that violate 4.1. On the other hand, the modified equation 5.1 gives us the extra freedom to pick U(t) such that the property 4.1 is maintained at all time t. For completeness we explicitly give the symmetry-fixing term in 5.1 that does maintain 4.1 for all time t, so long as Wr(0) = W(0) at time t = 0. We give it in a basis where W is diagonal with eigenvalues Ai: (5.2)
Joseph J. Atick and A. Norman Redlich
54
where the brackets denote the commutator defined for any two matrices A and B as [A,B] = A . B - Be A. One could try to simulate the dynamics in 5.1 with the explicit expression 5.2 included, however, this term appears too complex to be biologically plausible. However, if we choose the more restrictive initial condition for W(0) to satisfy [R, W-’(O)] = 0, in addition to W(0) = WT(0),then the necessary symmetry fixing term 5.2 vanishes on the first iteration, and can be shown to remain zero thereafter. So by going through the symmetry-fixing exercise we have identified a set of special initial conditions on W that do guarantee that 4.1 is maintained at all times, and moreover do so without modifying the original equation 2.8. One choice of initial conditions that very simply satisfies both [R, W-’1 = 0 and W = WT is W(0) = 1, which is the condition we use in our simulations in the next section. Without the above analysis we would not have known a priori that this initial condition is sufficient to ensure that the solution the system converges to will satisfy W = WT. This particular initial condition may be biologically plausible since it means that initially the neurons have no lateral links and that these links are developed as needed. Also, we have tested for the stability of this initial condition by starting the runs from configurations where the lateral links are nonvanishing but small, and we find small nonvanishing links do not disturb the convergence to the unique solution.
6 Simulations
To test the learning algorithm 3.4we first applied it to a one-dimensional (1D)ensemble of inputs Si, i = I, . . . , n with n = 64. For simulation purposes, we imposed periodic boundary conditions on the 1D space and generated an ensemble of Si with an autocorrelator Rii whose Fourier transform (power spectrum) was RCf) l/v14 but that had no higher order correlations. This particular RCf) in one dimension was an arbitrary choice that happened to give clear results. By itself, however, this ensemble is not realistic since unlike real sensory signals it is noise free. So we added to the signal Si noise with flat power spectrum IN(f)l’ = V . This type of noise signal Ni is already spatially decorrelated having autocorrelator ( N j N j ) 6i,p Adding the random noise signal Ni also happens to serve a significant function by increasing the stability of the algorithm 3.4, although this might seem anti-intuitive. The source of the instability is that W-’ appearing in 2.8 (even though we implement 3.4)can eventually blow up if there are any very small eigenvalue modes in R, since ultimately (W . WT)-’ -+ R-’ according to 2.11. This small eigenvalue problem can be accentuated during the learning process while W . WT is only approximately equal to R, so during learning W may have even smaller
-
-
Algorithm for Sensory Receptive Field Development
55
eigenvalues than it does once it converges. Adding noise with a flat spectrum eliminates this small eigenvalue problem by adding a constant to all the eigenvalues. In frequency space Rcf) + Rcf) + N2 so the eigenvalues can never get too small. In practice any nonvanishing noise N2 assures stabilization. Thus the algorithm 3.4 applied to any realistic problem-which always has nonvanishing noise-will continue to be convergent. In our one-dimensional simulations we chose a signal to noise of four. The parameter 7 in 3.4 was chosen to be much larger than the characteristic time scale of the ensemble so that 3.4 could approximate the convergent algorithm 3.3. A general lower bound on 7 is not possible since it is ensemble dependent and hence requires experimentation. We find that smooth convergence does require a relatively large T compared to the size of the terms on the right-hand side of 3.4. In our onedimensional simulations we chose T = 5000. Finally, we used the initial condition W, = S, that as discussed in Section 3 is sufficient to eliminate the U symmetry. This then turns out to produce spatially local receptive fields W-I as shown in Figure 2, which exhibits the converged solution for the 32nd neuron, W ,;; after 20,000 iterations. Having demonstrated the viability of 3.4 through a one-dimensional example, the next step is to apply 3.4 to a two-dimensional ensemble with statistical properties close to those of a natural visual environment. Field (1987) argued, based on some experimentation with "natural" scenes, that natural environments have an approximately scale invariant spatial autocorrelator, which is equivalent to having a spatial power spectrum proportional to lfl-2 for two-dimensional frequencies f. We have shown previously (Atick and Redlich 1992) that the retinal ganglion cell kernel W-' acts to decorrelate the environmental signal-at least at low frequencies where the signal to noise is high-which in frequency space means W-'(f) If1 so that I W(f)I2 = R(f). Here, the goal of the two-dimensional simulation is to demonstrate that this decorrelating kernel can be learned by a network implementing the developmental rule 3.4. In two dimensions (2D) we generated an ensemble with power spectrum and without higher order statistics. We again added random noise for a SIN = 4, chose the initial condition W = 1, and found that T = 20,000 was sufficiently large to ensure smooth convergence. There was one major difference, however, between our one- and twodimensional simulations that was necessitated by the need to reduce computation time in the two-dimensional case. In 2 0 it was computationally necessary to assume translation invariance of the solution WV, meaning assuming in advance that W,will take the form Wij = W( li - j l ) after convergence [note that the index i in 2 0 now denotes the twodimensional vector i = (&, i,,)]. This is a valid assumption since, as discussed in Sections four and five, the initial condition W = 1 implies W = WT,which in turn implies translation invariance of W, as long as the ensemble is chosen to be translation invariant. Restricting W to N
56
Joseph J. Atick and A. Norman Redlich
Figure 2: The one-dimensional receptive field, WG;,for the 32nd neuron after the algorithm converges Cj running along the horizontal axis with 64 the total number of neurons). The receptive fields of other neurons are very similar. The dashed line is the autocorrelator (SiS32) averaged over the training data, while the solid curve is (W wT)j32,which is very close to the actual autocorrelator. The training was done with 20,000 iterations, T = 5000 and S I N = 4 starting with the initial condition Wij(f= 0) = 60. Figre 3: Facing page. Typical two-dimensional images used in training the network in 20. Image B differs from image A only by the addition of a small amount of noise. These images have the property that their power spectrum is identical to that of natural scenes, namely l/(fI2. We have trained networks using these images instead of actual scenes for convenienceonly. The algorithm should converge to similar solutions if trained on actual natural scenes as long as the power spectrum is the same.
Algorithm for Sensory Receptive Field Development
b
57
58
Joseph J. Atick and A. Norman Redlich
be translationally invariant in advance is computationallyvaluable since it reduces the number of computations from order n2 to n, where n is the number of input signals (number of photoreceptors), which for our simulation was n = 16 x 16. We should emphasize, though, that this restriction was purely for simulation purposes, and in an actual neural implementation of 3.4 there would be no need for it. The result of the two-dimensional simulation following convergence, which took about 200,000 steps, is shown in Figure 4. It has the generic center-surround property of a ganglion cell, but does not correspond exactly to a ganglion cell receptive field. That is because it is a purely decorreluting kernel, whereas the true ganglion cell kernel both decorrelates and filters noise, as shown in Atick and Redlich (1992). In that paper we demonstrated that the total retinal kernel can be obtained by first low pass filtering the noisy input signal and then decorrelating the result. Here we have demonstrated that 3.4 can perform the decorrelation step. Therefore if one first low pass filters in exactly the same way as in Atick and Redlich (1992) (where we were careful to include the appropriate transmission noise following the lowpass stage) and then applies 3.4, one must arrive at the same realistic ganglion cell kernels found in that paper, since this effectively reproduces the steps outlined there. We also expect that applying 3.4 to natural scenes, since they have the same spectrum as our simulation scenes, would likewise produce results close to those shown here. In a two-stage process in which noise is filtered and then the signal is decorrelated, it is also possible to derive the noise filter separately using a different learning algorithm. For example in Atick and Redlich (1991) we gave a convergent developmental rule for learning least mean square noise smoothing. (Such noise filtering algorithms differ from decorrelation algorithms such as 3.4 in that they require sufficient supervision to distinguish signal from noise.) Therefore, both noise filtering and decorrelation can be achieved developmentally in a two-step process in which first noise filtering and then decorrelation are learned. There are also information theoretic formalisms in which both noise filtering and decorrelation can be achieved through minimizing (or maximizing) a single information theoretic quantity (Atick and Redlich 1990; Linsker 1988). If a developmental algorithm could be found to perform this minimization (or maximization) then both goals, noise filtering and decorrelation, could be achieved through one learning stage. Linsker (1991) has made some progress in this direction, though his learning algorithm still requires a two-phase process. We recently became aware of the Ph.D. thesis of Mark Plumbley (Engineering Dept., Cambridge University), where a proof of convergence for another decorrelating algorithm is presented, using a similar approach.
Algorithm for Sensory Receptive Field Development
59
Figure 4: The receptive field found by the algorithm for the central neuron in two-dimensional 16 x 16 array of neurons. The receptive fields of other neurons are very similar. The training was done with 200,OOO iterations, T = 20, OOO, SIN = 4, and W = 1 using images like those shown in Figure 3.
Acknowledgments We would like to thank K. Miller for useful discussions and the Seaver Institute for its support. References Atick, J. J., and Redlich, A. N. 1990. Towards a theory of early visual processing. Neural Comp. 2, 308-320. Atick, J. J.,and Redlich, A. N. 1991. Predicting ganglion and simple cell receptive field organizations. Int. J. Neural Syst. 1, 305-315. Atick, J. J.,and Redlich, A. N. 1992. What does the retina know about natural scenes? Neural Comp. 4, 196-210. Barlow, H.B. 1989. Unsupervised learning. Neural Comp. 1,295-311. Barlow, H . B., and Foldiak P. 1989. The Computing Neuron. Addison-Wesley, New York. Barlow, H. B., Kaushal, T. P., and Mitchison, G. J. 1989. Finding minimum entropy codes. Neural Comp. 1,412423.
60
Joseph J. Atick and A. Norman Redlich
Field, D. J. 1987. Relations between the statistics of natural images and the response properties of cortical cells. J. Opt. Soc.Am. A 4,2379-2394. Foldiak, P. 1989. Adaptive network for optimal linear feature extraction. Proc. IEEEIAVNS Int. Joint Conf. Neural Networks, Washington, DC,Vol. 1,401405. Goodall, M. C. 1960. Performance of stochastic net. Nature (London) 185, 557558. Hopfield, J. J. 1991. Olfactory computation and object perception. Proc. Natl. Acad. Sci. U.S.A. 00, 64624466. Kohonen, T., and Oja, E. 1976. Fast adaptive formation of orthogonalizing filters and associativememory in recurrent networks of neuron-like elements. Biol. Cybem. 21,85-95. Linsker, R. 1986. From basic network principles to neural architecture: Emergence of spatial-opponent cells. Proc. Natl. Acad. Sci. U.S.A. 83,508-512. Linsker, R. 1988. Self-organization in a perceptual network. Computer 21, (March), 105-117. Linsker, R. 1991. Talk at the 1991 meeting for the Society for Neuroscience. Oja, E. 1982. A simplified neuron model as a principal component analyzer. J. Math. Biol. 15, 267-273. Rubner, J., and Schulten, K. 1990. Development of feature detectors by selforganization. Biol. Cybern. 62,193-199. Sanger, T. D. 1989. Optimal unsupervised learning in a single-layer linear feedforward neural network. Neural Networks 2,459-473. Received 21 January 1992; accepted 12 May 1992.
This article has been cited by: 2. Martin T Wiechert, Benjamin Judkewitz, Hermann Riecke, Rainer W Friedrich. 2010. Mechanisms of pattern decorrelation by recurrent neuronal circuits. Nature Neuroscience 13:8, 1003-1010. [CrossRef] 3. Stuart D. Wick, Martin T. Wiechert, Rainer W. Friedrich, Hermann Riecke. 2010. Pattern orthogonalization via channel decorrelation by adaptive networks. Journal of Computational Neuroscience 28:1, 29-45. [CrossRef] 4. Simone Fiori . 2005. Nonlinear Complex-Valued Extensions of Hebbian Learning: An EssayNonlinear Complex-Valued Extensions of Hebbian Learning: An Essay. Neural Computation 17:4, 779-838. [Abstract] [PDF] [PDF Plus] 5. Y.C. Eldar, A.V. Oppenheim. 2003. MMSE whitening and subspace whitening. IEEE Transactions on Information Theory 49:7, 1846-1851. [CrossRef] 6. Eizaburo Doi , Toshio Inui , Te-Won Lee , Thomas Wachtler , Terrence J. Sejnowski . 2003. Spatiochromatic Receptive Field Properties Derived from Information-Theoretic Analyses of Cone Mosaic Responses to Natural ScenesSpatiochromatic Receptive Field Properties Derived from Information-Theoretic Analyses of Cone Mosaic Responses to Natural Scenes. Neural Computation 15:2, 397-417. [Abstract] [PDF] [PDF Plus] 7. Michael Schmitt . 2002. Neural Networks with Local Receptive Fields and Superlinear VC DimensionNeural Networks with Local Receptive Fields and Superlinear VC Dimension. Neural Computation 14:4, 919-956. [Abstract] [PDF] [PDF Plus] 8. Alexander Dimitrov , Jack D. Cowan . 1998. Spatial Decorrelation in Orientation-Selective Cortical CellsSpatial Decorrelation in Orientation-Selective Cortical Cells. Neural Computation 10:7, 1779-1795. [Abstract] [PDF] [PDF Plus] 9. Jim Kay, W. A. Phillips. 1997. Activation Functions, Computational Goals, and Learning Rules for Local Processors with Contextual GuidanceActivation Functions, Computational Goals, and Learning Rules for Local Processors with Contextual Guidance. Neural Computation 9:4, 895-910. [Abstract] [PDF] [PDF Plus] 10. S.C. Douglas, A. Cichocki. 1997. Neural networks for blind decorrelation of signals. IEEE Transactions on Signal Processing 45:11, 2829-2842. [CrossRef] 11. Anthony J. Bell , Terrence J. Sejnowski . 1995. An Information-Maximization Approach to Blind Separation and Blind DeconvolutionAn Information-Maximization Approach to Blind Separation and Blind Deconvolution. Neural Computation 7:6, 1129-1159. [Abstract] [PDF] [PDF Plus] 12. A. Norman Redlich . 1993. Supervised Factorial LearningSupervised Factorial Learning. Neural Computation 5:5, 750-766. [Abstract] [PDF] [PDF Plus]
Communicated by Dana Ballad
Three-Dimensional Object Recognition Using an Unsupervised BCM Network The Usefulness of Distinguishing Features Nathan Intrator Joshua I. Gold B m University, Providence, RI 02912 USA We propose an object recognition scheme based on a method for feature extraction from gray level images that corresponds to recent statistical theory, called pmjection pursuit, and is derived from a biologically motivated feature extracting neuron. To evaluate the performance of this method we use a set of very detailed psychophysical three-dimensional object recognition experiments (Biilthoff and Edelman 1992). 1 Introduction A system that performs recognition of three-dimensional (3D)objects in visual space must transform a complex pattern of visual inputs to an appropriate categorization. Such recognition is possible, for example, by template matching once the object and its templates are brought into register (Ullman 1989). Other similar schemes (Lowe 1986; Thompson and Mundy 1987)base the recognition on viewpoint consistency, which relate projected locations of key features of a model to its 3D structure given a hypothesized viewpoint. The regularization network or HyperBF interpolation scheme (Poggio and Edelman 1990; Poggio and Girosi 1990) represents 3D objects by sets of two-dimensional (2D)views using vectors of key-feature locations and regards generalization from familiar to novel views as a problem of nonlinear hypersurface interpolation in the space of all possible views. All these methods rely on the ability to find key features in the objects and, in some cases, to solve the correspondence problem between them.' However, sometimes these tasks can be as difficult as the recognition itself. In this paper, we propose an object recognition method that does not rely on finding such key features a priori. Instead, a transformation is sought that reduces the pixel image representations into a lowdimensional space from which nonlinear hypersurface interpolation can 'Edelman and Weinshall (1991) used the vertices without solving the correspondence problem between them.
Neural Computation 5,61-74 (1993)
@ 1993 Massachusetts Institute of Technology
64
Nathan Intrator and Joshua I. Gold
a priori an ordered list of vertices from the image and using a generalized radial basis function classification scheme (Moody and Darken 1989; Poggio and Girosi 1990, GRBF). This method classified lists of vertices based on their orientation within a vector space defined by the vertex sets of known objects; it achieved close to human performance in generalizing to novel views of the wires. The performance reflected a strong focus on the classification technique, and assumed a deterministic, a priori feature extraction. We, on the other hand, want to concentrate on the examination of the properties of our proposed feature extraction method and therefore in this study have chosen to use a classical, well-known classifier, based on the k-nearest-neighbor-rule5 (see for example, Duda and Hart 1973). In addition to the type of classifier used, the recognition paradigm with which the system is tested is a vital component in determining the usefulness of the features extracted. In the following sections we present an application of the BCM model to a set of specific 3D object recognition problems. The experiments chosen fulfill two important criteria: (1) they test the model's abilities to both recognize and generalize across a wide range of difficulties, and (2) these same studies have been used to test the abilities of not only computational models, but also human subjects; the psychophysical results in fact serve as benchmarks for this study. 3.1 Previous Studies. Bulthoff and Edelman (1992) developed and used wire-like objects in their experiments, in an effort to simplify the problem for the feature-extractorby providing little or no occlusion of the key features from any viewpoint. The wires consisted of seven connected segments, each pointed in a random direction but with its vertices distributed normally around the origin. Each experiment consisted of two phases, training and testing. In the training phase subjects were shown the target object from two standard views, located 75" apart along the equator of the viewing sphere. The target oscillated around each of the two standard orientations with an amplitude of 335" about a fixed vertical axis, with views spaced at 3" increments (see Fig. 1). Test views were located either along the equator-n the minor arc bounded by the two standard views (INTER condition) or on the corresponding major arc (EXTRA condition)--or on the meridian passing through one of the standard views (ORTHOcondition). Testing was conducted according to a two-alternative forced choice (2AFC) paradigm, in which subjects were asked to indicate whether the displayed image constituted a view of the target object shown during the preceding training session. Test images were either unfamiliar views of the training object or random views of a distractor (one of a distinct set of objects generated by the same procedure). 5Very similar classification results were obtained using a backpropagation classifier. In a forthcoming article, performance of backpropagationand radial basis function (RBF) classifiers will be compared using features extracted by the above feature extraction method.
Object Recognition Using a BCM Network
65
Figure 1: The training and testing experimental paradigm. A number of interesting characteristics of human visual object recognition abilities emerged from the psychophysical experiments. Generalization over orientations lying between two sets of known views-the INTERcondition-resulted in, on average, significantly fewer errors than with the other two extrapolation conditions. In addition, error rates increased steadily as the testing views moved farther away from the learned views, until recognition was near chance levels at large displacements. Finally, there were indications for a “horizontal bias,” so that error rates were lower when generalization was required along the horizontal, as opposed to the vertical, plane.
3.2 Experimental Paradigm. In the first part of the study, the network was tested on a 63 by 63 array of 8-bit gray-scale values with a paradigm nearly identical to the one used in the psychophysical investigation (Edelman and Bulthoff 1991). The procedure was modified slightly in that training was performed with two wires, since the k-NN classifier would yield meaningless results if trained on only a single wire. In the second part of the study, simple yes/no recognition was upgraded to a more difficult classification task involving six separate wires. The modification was necessary to fully test thFBCM model’s ability to extract the most salient rotation-invariant features from the images. Specifically, since BCM neurons explicitly search for differentiating features (due to the search for multimodality in the projected distribution),
Nathan Intrator and Joshua I. Gold
66
many cases involving only two distinct sets of inputs can be solved yith ”features” corresponding to prototypical views of each wire. In these cases, the two sets of wire-views, corresponding to the two wires, would form two distinct clusters in feature space. However, such differentiation would be much more difficult with a larger number of wires, and therefore the BCM network neurons would be forced to find projections that correspond to individual, rotation-invariant features, not prototypical views of individual wires. In addition, the model was modified in an attempt to account for the asymmetic psychophysical results. In the psychophysical experiments, the horizontal bias was found when humans were given the exact same paradigm as described above, except the objects were rotated 90” so that the training axis was aligned vertically, not horizontally. One possible explanation of such asymmetry is in increased resolution at the object representation level, namely, due to the fact that behaviorally, humans spend more time rotating around a vertical axis (i.e., rotation in a horizontal plane). This is experimentally equivalent to having more patterns rotated in a horizontal than in a vertical plane. This possibility has been eliminated in the careful psychophysical experiment performed by Edelman and Biilthoff (19911, in which subjects are provided identical experience with horizontal and vertical training. The continued existence of the bias under such conditions implicates an internal mechanism. We hypothesized greater a riori resolution in the infernal representation along the horizontal plane. More specifically, we set the ratio between the resolution in the horizontal plane and that in the vertical plane (the aspect ratio) to be 2/1 for “normal” training in the horizontal plane; conversely, training in the vertical plane was, from the point of view of the network, equivalent to setting the aspect ratio to be 1/2. Prediction of simulation performance due to this asymmetrical resolution is not straightforward since there are two contradictory effects. On the one hand, decreased resolution in the vertical plane means reduced disparity from rotations along that plane and therefore possibly better performance. On the other hand, there may also be improved performance in the horizontal axis since higher resolution will emphasize features that are rotation invariant along that direction. !
F
4 Results
The six wires used in the experiments are depicted in Figure 2. Different views of three of the wires are depicted in Figure 3. When only two wires were used (experiment one) the features extracted correspond almost exclusively to a single view of a whole image of one of the wires. “Thereis, in fact, limited evidence for visual field elongation in the horizontal plane (Hughes 19m.
Object Recognition Using a BCM Network
67
Figure 2: The six wires from a single view.
Figure 3 Different views (15' apart) of a single wire; top-to-bottom are INTER, EXTRA,and ORTHO. In contrast, when the task was recognition of six wires the extracted features emphasized small patches of several images or views, namely, areas that either remain relatively invariant under the rotation performed during training or represented a major differentiating characteristic of a specific wire (Fig. 4). A typical result is a set of weights that may correspond to a single wire but emphasizes small patches of the object and selectively inhibits selected areas which correspond to invariant locations of adjacent wires. For example, the top left image of Figure 4 largely r e p resents object number 5 in Figure 2 with additional inhibition from other objects, while the top right image or the bottom second from the right image exhibits weights related to several imagedviews. Classification results demonstrate the usefulness of the extracted features: generalization in the INTERorientations resulted in consistently
Nathan Intrator and Joshua I. Gold
68
Figure 4 Rotation invariant features for tublike objects extracted using a network of seven BCM neurons trained on six t u b l i k e objects. White areas ~ p resent strong synaptic weights, black areas represent negative synaptic weights (inhibition).
low error rates-around 15%(in which the chance error rate on this six wire experiment is 83.3%&which indicates that the features extracted by the BCM network could generalize well in those new views.' Furthermore, the results are comparable to those obtained in the psychophysical experiments. First, INTER recognition resulted in, on average, significantly fewer errors than with the other two extrapolation conditions. Second, error rates increased steadily as the testing views moved farther away from the learned views, until recognition was near chance levels at large displacements. These results are analogous to the ones shown in Figure 5 in which the aspect ratio is 2/1. Taken together, Figures 5 and 6 demonstrate a horizontal bias as seen in the psychophysical studies. When aspect ratio is 0.5, which corresponds in our model to training on rotations in the vertical plane, INTER performance is worse. This result suggests that finding specific rotation invariant features was harder in that case, given its lower resolution. On the other hand, there is no significant change in the performance of ExTRA and ORTHOorientations, suggesting that the extracted features were in both situations equally useful for EXTRAand ORTHOorientations. ~~
'Additional supporttothe usefulness of the extracted features to rotation invariant recognition is shown in sub uent work (Intratoret al. 1991; War et al. 1991) in which the extracted features are u a to occlude parts of the images and another network is trained to recognize the occluded images.
Object Recognition Using a BCM Network
69
-
0.8
cr
Inter
-
Extra
Q)
d
0.6
Ortho
&
0 &
____.
.........
0.4
w 0.2 0
10
0
20 30 40 Distance [Degl
50
60
Figun? 5 Fraction of misclassiflcation performance for wires trained on the horizontal plane.
o.8 4J
2
Oe6
-
0.4
-
0.2
-
&
0 &
.........
Ortho
.... .... ........
....
W
0
.......
Extra
a,
,/'
......... __.. ._..
F'
Figure 6: Fraction of misclassification performance for wires trained on the vertical plane. Note the degradation in performance in the INTERorientations.
Figures 7 and 8 show the results of the experimental paradigm testing the effect of additional experience during training in the horizontal
Nathan Intrator and JoshuaI. Gold
70
1 -
-
o.8
-
Extra
0)
c,
2
Inter
OS6
-
Ortho
0
10
+E+
U 0 U U
w
20 30 40 Distance [Degl
50
60
Figure 7 Fraction of dsclassification performance for wires trained on the horizontal plane with no asymmetry.
1
1
0.8 0.6
0.4 0.2 n
0
10
20 30 40 Distance [Deg]
50
60
Figure 8 Fraction of misclassification performance for wires trained with reduced training experience (views). plane! Both figures show results on training with an aspect ratio of 1, that is, no resolution asymmetry was used between the horizontal and 8Testing in both cases used the same number of patterns as in the previous experiments.
Object Recognition Using a BCM Network
71
vertical plane. In the experiments summarized in Figure 7, the same number of training views (experience) as in the previous set of experiments were used. In the experiments summarized in Figure 8, half as many training views were used. A number of interesting observations can be made. Results on the INTERcondition for an aspect ratio of 1 behave as can be predicted from the previous set of experiments; specifically, error rates were in between those of aspect ratios 2 and 0.5. EXTRAand ORTHOresults, however, were less noticeably affected, indicating that object resolution primarily affected the discovery of rotation invariant features to be used for recognition in the INTERcondition, as opposed to reducing overall recognition ability. Results from Figure 8, however, demonstrate a different effect. Reducing the number of training patterns, analogous to reducing the experience of vertical training, does not lead to an asymmetry in specific recognition conditions, but instead to a general decline in overall recognition ability. This suggests that reducing the number of training views in a model (without reducing the overall training angle rotation) does not simply affect the ability to extract rotation-invariant features for a particular recognition task. Instead, it degrades the ability of the model in overall feature extraction performance.
5 Discussion
This paper touches on issues of object representation. It is assumed that an object is internally represented by a particular combination of features. The nature of these features and the means for binding together the most important combination of features are still undetermined (Sejnowski 1986). We presented an unsupervised method for extracting features directly from gray-level pixel images, and we showed that a surprisingly small number of features is needed for a complex classification task. A comparison of our results to similar psychophysical experiments gives some indication that these features possess desired invariance properties that allow for overall classification performance that compares favorably with human performance. Extracting features from these gray-level images is a highly nontrivial statistical task. The dimensionality of this problem is 63 x 63 pixels; therefore, the curse of dimensionality implies that the number of training patterns should be immense, and yet from a small training set of 132 wires useful directions (projections)were extracted corresponding to features that were especially useful for rotation invariant recognition. This suggests that the BCM network may be a practical tool for gray-level image recognition in which internal low-dimensional feature representation emerges as a result of unsupervised training.
72
Nathan Intrator and Joshua I. Gold
Acknowledgments We wish to thank Heinrich Biilthoff, Shimon Edelman, and Leon Cooper for the encouragement and many fruitful conversations that have led to this paper. Dave Sheinberg, Phillipe Schyns, and Eric Sklar were invaluable for their help in using the AVSW system for getting the gray-level images. Finally, the excellent computational facilities of the Cognitive Science Department at Brown University allowed us to complete the simulations required for this project. Research was supported by the National Science Foundation, the Army Research Office, and the Office of Naval Research.
References Bear, M. F., and Cooper, L. N. 1988. Molecular mechanisms for synaptic modification in the visual cortex: Interaction between theory and experiment. In Neuroscience and Connectionist Theory, M. Gluck and D. Rumelhart, eds., pp. 65-94. Lawrence Erlbaum, Hillsdale, NJ. Bear, M. F., Cooper, L. N., and Ebner, F. F. 1987. A physiological basis for a theory of synapse modification. Science 237,4248. Bellman, R. E. 1961. Adaptive Control Processes. Princeton University Press, Princeton, NJ. Bienenstock, E. L., Cooper, L. N., and Munro, P. W. 1982. Theory for the development of neuron selectivity: Orientation specificity and binocular interaction in visual cortex. J , Neurosci. 2, 32-48. Bulthoff, H. H., and Edelman, S. 1992. Psychophysical support for a 2-D view interpolation theory of object recognition. Proc. Natl. Acad. U.S.A. 89,6044. Clothiaux, E. E., Cooper, L. N., and Bear, M. F. 1991. Synaptic plasticity in visual cortex: Comparison of theory with experiment. I. Neurophysiol. 66, 17si804. Duda, R. O., and Hart, P. E. 1973. Pattern Classification and Scene Analysis. John Wiley, New York. Edelman, S. 1991. Features of recognition. CSTR 10, Weizmann Institute of Science. Edelman, S.,and Bulthoff, H. H. 1992. Orientation dependence in the recognition of familiar and novel views of 3D objects. Vision Res., in press. Edelman, S., and Poggio, T. 1992. Bringing the Grandmother back into the picture: A memory-based view of object recognition. J. Pattern Recog. Artif lntell. 6, 37-62. Edelman, S., and Weinshall, D. 1991. A self-organizingmultiple-view representation of 3D objects. Biol. Cybern. 64,209-219. Fisher, R. A. 1936. The use of multiple measurements in taxonomic problems. Ann. Eugen. 7,179-188. Friedman, J. H. 1987. Exploratory projection pursuit. J. Am. Stat. Assoc. 82, 249-266.
Object Recognition Using a BCM Network
73
Friedman, J. H., and Tukey, J. W. 1974. A projection pursuit algorithm for exploratory data analysis. ZEEE Transact. Comput. C(231, 881-889. Gold, J. I. 1991. A model of dendritic spine head [Ca++]:Exploring the biological mechanisms underlying a theory for synaptic plasticity. Unpublished honors thesis, Brown University. Harman, H. H. 1967. Modern Factor Analysis, 2nd ed. University of Chicago Press, Chicago. Huber, P. J. 1985. Projection pursuit (with discussion). Ann. Statist. 13,435-475. Hughes, A. 1977. The topography of vision in mammals of contrasting live style: Comparative optics and retinal organisation. In The Visual System in Vertebrates,Handbookof Sensory Physiology VZI/5,F. Crescitelli,ed., pp. 613-756. Springer-Verlag, Berlin. Intrator, N. 1990. A neural network for feature extraction. In Advances in Neural Information Processing Systems, D. S. Touretzky and R. I? Lippmann, eds., Vol. 2, pp. 719-726. Morgan Kaufmann, San Mateo, CA. Intrator, N. 1992. Feature extraction using an unsupervised neural network. Neural Comp. 4,98-107. Intrator, N., and Cooper, L. N. 1992. Objective function formulation of the BCM theory of visual cortical plasticity: Statistical connections, stability conditions. Neural Networks 5,3-17. Intrator, N., Gold, J. I., Bulthoff, H. H., and Edelman, S. 1991. Three-dimensional object recognition using an unsupervised neural network Understanding the distinguishing features. In Proceedings ofthe 8th Israeli Conferenceon AICV, Y. Feldman and A. Bruckstein, eds., pp. 113-123. Elsevier, Amsterdam. Jones, M. C., and Sibson, R. 1987. What is projection pursuit? (with discussion). J. R. Statist. Soc. A(150), 1-36. Kruskal, J. B. 1969. Toward a practical method which helps uncover the structure of the set of multivariate observations by finding the linear transformation which optimizes a new 'index of condensation.' In Statistical Computation, R. C. Milton and J. A. Nelder, eds., pp. 427-440. Academic Press, New York. Lowe, D. G. 1986. Perceptual Organization and Visual Recognition. Kluwer Academic Publishers, Boston, MA. Moody, J., and Darken, C. 1989. Fast learning in networks of locally tuned processing units. Neural Comp. 1,281-289. Poggio, T., and Edelman, S. 1990. A network that learns to recognize threedimensional objects. Nature (London) 343,263-266. Poggio, T., and Girosi, F. 1990. Networks for approximation and learning. IEEE PYOC.78(9), 1481-1497. Sebestyen, G. 1962. Decision Ahking Processes in Pattern Recognition. Macmillan, New York. Sejnowski, T. J. 1986. Open questions about computation in Cerebral Cortex. In Parallel Distributed Processing, J. L. McClelland and D. E. Rumelhart, eds., Vol. 2, pp. 372-389. MIT Press, Cambridge, MA. Sklar,E., Intrator, N., Gold, J. J., Edelman, S. Y., and Biilthoff, H. H. 1991. A hierarchical model for 3D object recognition based on 2D visual representation. Neurosci. Soc.Abstr.
74
Nathan Intrator and Joshua I. Gold
Thompson, D. W.,and Mundy, J. L. 1987. Three-dimensional model matching from an unconstrained viewpoint. In Proceedings of lEEE Conference on Robotics and Automation, pp. 208-220. Raleigh, NC. Ullman, S. 1989. Aligning pictoral descriptions: An approach to object recognition. Cognition 13, 13-254. Received 11 July 1991; accepted 14 May 1992.
This article has been cited by: 2. Eric E. Cooper, Brian E. Brooks. 2004. Qualitative Differences in the Representation of Spatial Relations for Different Object Classes. Journal of Experimental Psychology: Human Perception and Performance 30:2, 243-256. [CrossRef] 3. Y. Dotan, N. Intrator. 1998. Multimodality exploration by an unsupervised projection pursuit neural network. IEEE Transactions on Neural Networks 9:3, 464-472. [CrossRef] 4. Q.Q. Huynh, L.N. Cooper, N. Intrator, H. Shouval. 1998. Classification of underwater mammals using feature extraction based on time-frequency analysis and BCM theory. IEEE Transactions on Signal Processing 46:5, 1202-1207. [CrossRef] 5. Shimon Edelman . 1995. Representation of Similarity in Three-Dimensional Object DiscriminationRepresentation of Similarity in Three-Dimensional Object Discrimination. Neural Computation 7:2, 408-423. [Abstract] [PDF] [PDF Plus] 6. Nathan Intrator . 1993. Combining Exploratory Projection Pursuit and Projection Pursuit Regression with Application to Neural NetworksCombining Exploratory Projection Pursuit and Projection Pursuit Regression with Application to Neural Networks. Neural Computation 5:3, 443-455. [Abstract] [PDF] [PDF Plus]
Communicated by John Moody
Complexity Optimized Data Clustering by Competitive Neural Networks Joachim Buhmann' Lawrence Livermore National Laboratory, Computational Physics Division, P. 0.Box 808, L-270, Livermore, C A 94550 U S A
Hans Kiihnel Physik Department, T35, Technische Universitiit Miinchen, Boltzmannstrafle, 0-8046 Garching, Germany
Data clustering is a complex optimization problem with applications ranging from vision and speech processing to data transmission and data storage in technical as well as in biological systems. We discuss a clustering strategy that explicitly reflects the tradeoff between simplicity and precision of a data representation. The resulting clustering algorithm jointly optimizes distortion errors and complexity costs. A maximum entropy estimation of the clustering cost function yields an optimal number of clusters, their positions, and their cluster probabilities. Our approach establishes a unifying framework for different clustering methods like K-means clustering, fuzzy clustering, entropy constrained vector quantization, or topological feature maps and competitive neural networks. 1 Introduction
Natural and artificial information processing systems in vision, speech recognition and telecommunication rely on data compression and data recoding techniques either to process large amounts of data efficiently or to discard noise and to reveal the underlying structure in a data set (Linde et al. 1980; Bezdek 1980; Rose et al. 1990; Chou et al. 1989;Kohonen 1984; Rumelhart and Zipser 1985). In situations where we know the nature of the data source or at least a set of promising models for it, the data clustering task amounts to an estimation of the underlying probability density with Bayesian learning (Duda and Hart 1973) being the appropriate method. Bayesian classification theory (Hansen et al. 1991) as implemented in the autoclass system is a successful example of this approach. In the absence of a parametric model for the data, however, the 'Present address: Rheinische Friedrich-Wilhelms-Universitat, Institut fiir Informatik 11, Romerstrafe 164, D-5300 BOM 1, Germany.
NeurnI Computation 5, 75-88 (1993)
@ 1993 Massachusetts Institute of Technology
76
Joachim Buhmann and Hans Kuhnel
data clustering problem can be formulated as an optimization problem of a suitable objective function which preserves the original data as complete as possible. Distortion errors that result from the clustering process have to be limited to a minimum. Furthermore, optimizing a clustering cost function is the strategy of choice if we have to observe additional constraints imposed on the clustering solution by the information processing application considered. Such constraints are often not related to the data source and, therefore, would not influence a clustering solution based on the probability density estimation of the data source. Clustering strategies derived from an optimization principle are also known as vector quantization (Linde et al. 1980). The most prominent algorithms are the MacQueen K-means clustering algorithm (MacQueen 1967) and the LBG vector quantization algorithm by Linde et al. (1980). One central issue of this paper is to supplement the usual distortion measure with a complexity term that penalizes too complex clustering solutions. We discuss an objective function for data clustering that compromises between distortion errors and the complexity of a reduced data representation. The joint optimization of these two cost terms determines an optimal number of cluster centers. A maximum entropy estimation of the cluster assignments allows us to derive batch and online versions of the resulting algorithms for a number of different distortion and complexity measures. The close analogy of complexity optimized clustering with winner-take-all neural networks suggests a neural-like implementation resembling topological feature maps.* 2 The Objective Function-Choices
for Distortion and Complexity
costs Approaches to represent a data set { x i I xi E @; i = 1 , . . . ,N}by a reduced set of data prototypes {y, I y, E @; a = 1,. . . ,K} based on a distortion cost function arise in a large variety of data processing applications. The primary objective in these applications is to determine an appropriate number K of clusters and to find their centers y, and their cluster probabilities pa. The naming convention for the parameters y, is not unique; therefore, we will interchangeably use the terms “cluster center,” “prototype,” or “reference vector” for y,. The assignments {Mi, I a = 1,. . . ,K;i = 1,.. . ,N},Mi, € (0, I} of data point X i to cluster a are chosen such that the residual distortion error ’ D i , ( X i r y,) between data point xi and cluster center y, is minimized and that the resulting complexity of the cluster set is limited. Mi, = 1 denotes that data point X i is uniquely assigned to cluster a, which implies the uniqueness constraint Cfj=lMi, = 1. The cluster probability of a cluster cr is d e ZTheidea to implement clustering solutions by winner-take4 networks has been discussed by various authors [e.g.,see Kohonen (1984) and Rumelhart and Zipser (1985) for a historical perspective].
Complexity Optimized Data Clustering
77
xzl
fined as p , = Mi,/N. The most frequently used distortion measures are powers of the Euclidean distance between xi and y,, for example, Di, = [[Xi - y,1['. However, a Minkowski lp metric
might be preferable for application with a %oxy" data structure, that is, data sets generated by distributions with sharp edges. Rate distortion theory (see Cover 1991) specifies the optimal choice of y,, that is, it has to be the centroid y, of a cluster a, y, being defined by the centroid condition ciMi,(d/dy,)Di,(xi,Ya) = 0. The complexity C, of cluster a depends on the specific information processing application at hand, in particular, we assume that C, is a function of only the cluster probability pa. The proposed clustering cost function N
X
(2.1)
compromises between distortion costs and complexity costs, thereby determining an optimal number K of clusters. Variation of {Mi,} implicit1 determines the cluster centroid y, and the cluster probability p , = Mi,/N. X is a weighting parameter required to adjust the complexity cost to the scale of the distortion costs. The cost function 2.1 has to be optimized in an iterative fashion: (1) vary the assignment variables Mi, for a fixed number K of clusters such that the costs &K({M~,}) decrease; (2) increment the number of clusters K -+ K + 1 and optimize Mi, again. The optimized cluster parameters y,, pa, and K are determined by the configuration with minimal costs. Note that configurations with degenerate clusters, that is, y, = yp for a # /3 have to be rejected since they cause an overestimation of the "true" complexity of a cluster set. The well-known K-means clustering algorithm (Linde et al. 1980) corresponds to the case X = 0 and K fixed a priori. An application-dependentcomplexity could be the mass storage space of a storage medium, the processing hardware in electronics, the number of neurons in brains, or the channel bandwidth in communication systems. Complexity costs that penalize small, sparsely populated clusters, that is, C, = I/& s = 1,2, . . . , favor equal cluster probabilities, thereby emphasizing the hardware aspect of a clustering solution. The special case s = 1 with complexity costs strictly proportional to the number of clusters (C, = l/p, + Xi C, Mi,/p, = NK) defines a clustering principle which is functionally equivalent to the K-means clustering although we follow the heuristics to increase the number of clusters incrementally. Biological systems certainly favor such a load-balancing strategy for data processing since they have to employ neurons for information processing, which creates energetic and physiological costs for living brains.
x
Joachim Buhmann and Hans Kiihnel
78
A complexity measure that provides a lower bound on encoding costs in the context of data compression and bandwidth limited data transmission is the Shannon entropy of a cluster set (C) = - C, pa logp, (C, = - logp,) and was used by Chou et al. (1989) for the design of an entropy constrained vector quantizer. (C) measures the minimum average number of bits necessary to uniquely encode a given cluster set. The choice of an appropriate distortion measure DiU(xi,y,) is another degree of freedom in the design of a clustering algorithm. The most commonly used distortion costs are distance measures Di, = IIXi-yoll', which preserve the permutation symmetry of 2.1 with respect to the cluster index a. We will now discuss the relationship of our approach to a class of clustering algorithms with a preference for topological arrangements of clusters. Luttrell(1989) discovered in his studies of hierarchical vector quantizers that a fictitious noise process in the communication channel between sender and receiver can break the permutation symmetry and favors a particular topology over other possible cluster arrangements. This picture that is sketched in the block diagram below establishes a connection to a class of topological vector quantization algorithms known as self-organizing feature maps (Kohonen 1984; Ritter et al. 1992). The same idea of coding schemes that are robust against channel noise has been discussed in the literature on vector quantization as source-channel-coding (Farvardin 1990).
I noise T,, I {Xi}+
1
1-
-+a+
pGEq
+
y+
1-1
+
{y,}
At the encoding stage data point xi is assigned to cluster a. The index a, however, gets corrupted by channel noise and arrives as index y at the receiver's side of the communication channel. Let us denote the transition probability from index a to index y by T a y , C!=, T,, = 1. The receiver, consequently, reconstructs the "incorrect" cluster center y, as a representation of xi instead of the "correct" reference vector y,. The additional distortions due to the channel noise have to be taken into account when we estimate the most likely assignment variables and, thereby, the centroid positions and the cluster probabilities. The average clustering costs of the data set {xi} are
The average distortion error ((Dia)= C, TuyDi,(xi,y,) between data point xi and cluster center yU quantifies the intrinsic quantization errors and the noise induced errors. The centroids y, are defined by N
K
n
Complexity Optimized Data Clustering
79
The noise process with its transition probabilities T, induces a topology onto the set of clusters, for example, a tridiagonal matrix ,T = 1 - v;T,,,il = v/2; T, = 0 Vla - yI > 1 (see Fig. 2) defines a linear chain with nearest-neighbor transitions. Note, that the structure of the clustering cost functions 2.1 and 2.2 is the same. We have introduced only a generalized distortion measure ((Dial in the topological case. Topology preserving clustering reduces to nontopological clustering if we identify ,T = ,S with ,S being the Kronecker symbol (S, = 1 if (I: = 7; ,S = 0 if a # y). The clustering schemes based on 2.1 or 2.2 do not take any a priori knowledge about data classes into account. Many applications, however, provide information how to partition the training data into a set of classes y. Given such a priori class information for the training data, it is desirable that clusters do not overlap with class boundaries, for example, data belonging to different classes should not be assigned to the same cluster. This knowledge about the class membership Fir E [0,1], C, Fir = 1Vi of data point i in class y allows us to supervise a clustering process. The costs associated with supervised clustering depend on the joint probability pra = CiFi,Mi,/N of a data point out of class y being assigned to cluster a. The uniqueness of class membership implies the relation pa = &p,,. We introduce a new cost term S,, called supemision costs, that penalizes assignments of data from different classes to the same cluster. S, should be positive and it should vanish if the conditional probability p,la, that a data point assigned to cluster a is member of class y, approached certainty, that is, p,la = p,,/p, = 1. A supervision cost term that satisfies this condition is defined by the ”conditional class entropy” S, = - C, p,l, log(p,l,). Adding the supervision costs S, to the logarithmic complexity costs C, = -log@,) both cost terms being weighted equally (AS = AC = A) we arrive at the “supervised complexity costs” C; = - C, p , ~ , log(p,,) which penalizes too complex clustering solutions and includes a priori class information. As discussed in Bichsel and Seitz (1989) the ”conditional class entropy” as a penalty term favors a partitioning of the input space with clusters of unique class adherence, a very valuable property for any subsequent classification stage. The simplest functional form of S, that is polynomial in p,, with a root at P,l, = 1 is S, = C,P,l,(l - p,la). 3 Maximum Entropy Estimation of y, and p ,
Different combinations of complexity terms, topology constraints, and class information for supervised clustering define a variety of clustering algorithms that are relevant in very different information processing contexts. To derive robust, preferably parallel algorithms for these data clustering cases, we study the optimization problem 2.2 in a probabilistic framework (see Rose et al. 1990). This design philosophy for
Joachim Buhmann and Hans Kiihnel
80
optimization algorithms is motivated by the success of simulating annealing (Kirkpatrick et al. 1983) and of neural optimization algorithms (Durbin and Willshaw 1987; Yuille 1990; Simic 1990). The most likely distribution of assignment variables Mi, can be determined by the maximum entro y principle (Jaynes 1957), and equals the Gibbs distribution P ( { M } ) = exp(-PEK). The "computational temperature" T = 1/P, which plays the role of a Lagrange parameter for the average clustering costs, controls the randomness in the assignment process. The partition function Z normalizes the Gibbs distribution. Statistical physics (see, e.g., Amit 1989; Rose et al. 1990) states that maximizing the entropy at a fixed temperature is equivalent to minimizing the free energy
P
K
i
N
with respect to the variables paryurparyu (see Buhmann and Kuhnel 1992c for details). The auxiliary variables p a , ya are Lagrange parameters that enforce the constraints pa = Ezl Mi,/N and Ci ,&Mi7Tya(a/?a)Dia = 0, respectively. It can be shown that all minima of FK have ya I 0 and pa = Xpa(aCa/apa). The fact that ye vanishes3 is implied by the definition of ya as the centroid of cluster a. The resulting reestimation equations for the expected cluster probabilities and the expected centroid positions are (3.2) (3.3)
Equations 3.2 and 3.3 are necessary conditions of FKbeing minimal. We have identified pa, y. in 3.1-3.4 with their expectation values, which is 31n self-organizing feature maps yo is usually defined as the center of mass of the Voronoi cell a instead of the generalized centroid C i C 7 M i Y T f u ~ Z ) i o ( X i , y=u )0, which produces a nonvanishingconjugate field ya.
Complexity Optimized Data Clustering
81
justified in the large N limit. The expectation value (Mi,)of the assignment variable Mi, can be interpreted as a fuzzy membership of data point xi in cluster a. The fuzziness that is controlled by the temperature T sets a resolution limit below which different clusters cannot be discriminated. In the case of supervised clustering with the cost function 2.2 supplemented by supervision costs, the free energy generalizes to
&
The term Firpya = A c p a ( X a / d p a )+ Aspa C,"=Ir i y ( a & / @ y a ) enforces the constraint pya = & I'iyMi,/N. Equation 3.2 has to be generalized to a reestimation of the average joint probabilities pay, that is,
The reestimation equation for the cluster centroids has the same form as in 3.3 with (Mi,) given by 3.7. See Buhmann and Kuhnel (1992a) for a more detailed derivation of the equations 3.5-3.7. The global minimum of the free energy 3.1 with respect to p,,y, (y,, jj, already inserted) determines the maximum entropy solution of the cost function 2.1. Note that the optimization problem 2.1 or 2.2 of a KN state space has been reduced to a K(d+l)-dimensional minimization of the free energy FK(3.1 or 3.5). To find the optimal parameters p a , y, and the number of clusters K that minimize the free energy we start with one cluster located at the centroid of the data distribution, split that cluster, and reestimate p,, ya using equations 3.2 and 3.3. The new configuration is accepted as an improved solution if the free energy 3.1 has been decreased. This splitting and reestimation loop is continued until we fail to find a new configuration with lower free energy. An iterative splitting strategy is required because the cost function exhibits numerous local minima, especially since we compare solutions with different numbers of clusters. Cluster splitting in the topological case should be performed such that a new cluster does not violate the topology constraint severely,
Joachim Buhmann and Hans Kuhnel
82
~~
I
* *
+
*
* ' * * I * ** 1 . * . * * * *+* * c ' * 1 . * . * . * I * * +** * * * ' t
Figure 1: A data distribution (4000 data points) (a), generated by four normally distributed sources, is clustered with the complexity measure C, = -logpa. Two zero temperature solutions for X = 2.5,0.4 are shown in (b,c) where plus signs (+) denote the positions of the gaussians and stars (*) denote cluster centers.
that is, it should not give rise to a topological defect. The resulting configuration is stable under single cluster splitting, but we are not assured to find the global minimum. Note the difference between our approach to determine the number of clusters and the suggestion by Rose et al. (1990) to use the temperature as a control parameter. The temperature determines the fuzziness of a clustering solution, whereas the complexity term penalizes excessively many clusters. In the case of hard clustering (T + 0) we still like to limit the number of clusters by an application dependent complexity measure. Nontopological (T,? = 6,J clustering results at zero temperature for the logarithmic complexity measure (C, = - log p,) are shown in Figure 1. At high complexity costs (b) the algorithm finds four clusters located at the centers of the gaussians. In the limit of very small complexity costs (c) the best clustering solution densely covers the data distribution. The specific choice of logarithmic complexity costs causes an almost homogeneous density of cluster centers, a phenomenon that is explained by the vanishing average complexity costs (C,) = -pa logp, of very sparsely occupied clusters (see Gish and Pierce 1968; Buhmann and Kuhnel 1992~). Analytical results of the asymptotic (K large) cluster density for different complexity measures are discussed in Buhmann and Kiihnel (1992~). Figure 2 shows a clustering configuration assuming a onedimensional topology in index space with nearest-neighbor transitions. The short links between neighboring nodes of the neural chain indicate that the distortions due to external noise have also been optimized. Note that complexity optimized clustering determines the length of the chain or, for a more general noise distribution, an optimal size of the cluster
Complexity Optimized Data Clustering
83
Figure 2: Topology presetving clustering with C, = l/p,: a chain of 50 clusters covers the data distribution of Figure 1. The short average distances between consecutive clusters demonstrate that the total distortion error has been minimized, including the distortions induced by external noise (17 = 0.05).
set. This stopping criterion for adding new cluster nodes generalizes selforganizing feature maps (Kohonen 1984)and removes arbitrariness in the design of topological mappings. Furthermore, our algorithm is derived from an energy minimization principle in contrast to self-organizing feature maps, which “cannot be derived as a stochastic gradient on any energy function” (Erwin et al. 1992). The complexity optimized clustering scheme has been tested on the real-world task of image compression (Buhmann and Kuhnel 1992b). Entropy optimized vector quantization of wavelet decomposed images has reduced the reconstruction error of the compressed images up to 30%. Furthermore, the tendency of entropy optimized clustering to represent outlier regions with low data density yields psychophysicallymore pleasing image reconstructions since rare image features like edges are more faithfully encoded than by K-means clustering (Buhmann and Kuhnel 1992~). Figure 3 summarizes our experiments with supervised clustering controlled by the supervision costs Ci. To test the performance of supervised clustering compared to an unsupervised scheme as K-means clustering
84
Joachim Buhmann and Hans Kuhnel
Number of Clusters h’
Figure 3: Comparison of K-means clustering results (4 with supervised clustering ( 0 ) of a data set generated by a homogeneous two-dimensional data distribution with a disk shape ( ( ( x i / (_< 1). The data set (800data points) is divided into two classes of eight equally sized segments per class which alternate. Supervised clustering outperforms K-means clustering significantly for K 216 as indicated by the decrease of the average “conditional class entropy” (So).
we generated a disk-shaped data set with homogeneous density inside the disk llxill 5 1 and zero density outside. The data set was divided into two classes of eight segments per class with segments of different classes adjacent to each other. This artificial data set of two strongly intermingled classes allows us to benchmark the supervised clustering scheme with K-means clustering. Figure 3 shows that supervised clustering outperforms K-means clustering significantly for K 216 clusters as indicated by the decrease of the average “conditional class entropy” (So).
4 Online Clustering -
The described clustering procedures are butch algorithms since they require all data points {xi} to be available at the same time, a very restrictive constraint for applications dealing with a large data volume. Online clustering algorithms process a data stream sequentially by updating y, and pa iteratively. Such an algorithm is implemented by the first two layers of a competitive neural network with an architecture depicted in Figure 4. The third layer composed of classification units assigns data points to classes provided that class labels are available for the training data. The third layer is absent in unsupervised clustering schemes. The synaptic vectors y, define localized receptive fields of the clustering units
Complexity Optimized Data Clustering
85
a, which can be interpreted as radial basis function units. Learning after N - 1 data points have been processed results in the incremental changes
(4.1) (4.2)
The upper indices (N) and (N - 1) denote the estimates of pa, yo after XN and XN-~ have been processed, respectively, To derive equations 4.14.3we have expanded the equations 3.2, 3.6, and 3.3 u p to linear terms in AporApyo,Aya,keeping ( M N , ~fixed. ) All terms proportional to derivatives of ( M N , ~vanish ) in the hard clustering limit exponentially fast IO[exp(-c/T)/q, c > 0 as T + 01.
Figure 4 A three layer competitive neural network for data clustering with d units in the input layer, K units in the clustering layer, and G units in the classification layer implements the complexity optimized clustering algorithm. Unit a in the clustering layer receives activity from the input units weighted by the synaptic vector yo. The units in the clustering layer are connected by a winner-take-all network. Due to mutual competition each data point XN is assigned to exactly one cluster a in the case of hard clustering and to a small number of clusters for fuzzy clustering. The third layer calculates the conditional class probability of a data point i being in class 7. This layer is absent for unsupervised clustering.
Joachim Buhmann and Hans Kiihnel
86
In the case of squared Euclidean distortion costs (Dia = equation 4.3 reduces to the form
I1Xi
-~
~ 1 1 ~ ) (4.4)
which corresponds to MacQueen's update rule for K-means clustering (MacQueen 1967) and which is similar to learning in competitive neural networks. yiN-') is moved toward the most recent data point XN proportionally to the mismatch (XN - yLN-'))and proportional to XN'S effective membership C, T,,(MN,,) in cluster a. In addition to that, the update formula 4.4 weights any change in yiN)by the effective number of data points NC, T,,piN), which are already assigned to cluster a. The learning rate 1/(NC, T,,piN)) treats different clusters according to their history. That generalizes conventional topological feature maps which suggest the same learning rate for all clusters (Kohonen 1984). Is the learning rate c/(NC,T,,piN)) with c = 1 the fastest rate that guarantees the correct maximum entropy estimation of the cluster centers? This question of optimally efficient stochastic approximation of the clustering solution has received considerable attention recently, especially since there exists evidence from numerical simulation that fast adaptive K-means clustering (Darken and Moody 1990) and "search then converge" clustering schedules (Darken and Moody 1992) outperform the original MacQueen's update rule in speed. According to numerical and theoretical results there exists a critical c* below which convergence in the asymptotic limit N + 00 is dramatically slowed down (Darken and Moody 1992). Unfortunately, c* is unknown for clustering and the numerically supported conjecture that the optimal learning rate c"p' = c* > 1 has to be proved. Online optimization of the number of clusters K relies on a heuristics for cluster merging and cluster creation. We have explored the following heuristics for cluster creation: A data point xi initializes a new cluster K 1 with YK+I = Xi if the clustering costs by assigning xi to an already existing cluster exceeds the complexity costs of the new cluster K + 1, that is, CK+I < min,( llxi - y,ll' + XC,). We found in a series of clustering experiments that this strategy causes a slight overestimation of the number of clusters but the resulting cluster configurations have comparable cluster costs to configurations found in batch runs.
+
5 Conclusion
Complexity optimized clustering is a maximum entropy approach to unsupervised and supervised data clustering that determines the optimal number of clusters as a compromise between distortion errors and the complexity of a cluster set. The complexity term turns out to be as important for the design of a cluster set as the distortion measure. We have de-
Complexity Optimized Data Clustering
87
rived a batch and an online version of the proposed clustering algorithms. Complexity optimized clustering maps onto a winner-take-all network, which suggests hardware implementations in analog VLSI (Andreou et al. 1991; Lazzaro et al. 1989). Topology preserving clustering suggests a cost function based approach to limit the size of self-organizing maps. The proposed framework for data clustering unifies traditional clustering techniques like K-means clustering, entropy constraint clustering or fuzzy clustering with neural network approaches as topological vector quantizers. The network size and the cluster parameters are determined by a problem adapted complexity function, which removes considerable arbitrariness present in other nonparametric clustering methods. A related approach to clustering (Wong 1992) was recently brought to our attention. The clustering algorithm proposed by Wong is complementary to ours since it relies on cluster melting rather than cluster splitting.
Acknowledgments It is a pleasure to thank H. Sompolinsky, T. Tishby, and P.Tavan for helpful discussions. JB has been supported by the German Federal Ministry of Science and Technology and by the Air Force Office of Scientific Research (C. von der Malsburg, PI) while working at the Center for Neural Engineering of the University of Southern California. HK is a recipient of a graduate fellowship, Technical University Munich.
References Amit, D. 1989. Modelling Brain Function. Cambridge University Press, Cambridge. Andreou, A. G., Boahen, K. A., Pouliquen, P. O., Pavasovii., A,, Jenkins, R. E., and Strohbehn, K. 1991. Current mode subthreshold MOS circuits for analog VLSI neural systems. I E E E Transact. Neural Networks 2, 205-213. Bezdek, J. C. 1980. A convergence theorem for the fuzzy isodata clustering algorithms. I E E E Transact. Pattern Anal. Machine Intelligence 2(1), 1-8. Bichsel, M., and Seitz, P. 1989. Minimum class entropy: A maximum information approach to layered networks. Neural Networks 2, 133-141. Buhmann, J., and Kiihnel, H. 1992a. Unsupervised and supervised data clustering with competitive neural networks. IJCNN International Conference on Neural Networks, Baltimore, pp. N796-IV801. IEEE. Buhmann, J., and Kiihnel, H. 1992b. Complexity optimized vector quantization: A neural network approach. In Data Compression Conference, J. Storer, ed., pp. 12-21. IEEE Computer Society Press, Los Alamitos, CA. Buhmann, J., and Kiihnel, H. 1992c. Vector quantization with complexity costs. I E E E Transact. I n f o r m . Theory, in press. Chou, P. A., Lookabaugh, T., and Gray, R. M. 1989. Entropy-constrained vector quantization. I E E E Transact. Acoust. Speech Signal Process. 37, 31-42.
88
Joachim Buhmann and Hans Kuhnel
Cover, T. M.1991. Elements of lnformation Theory. Wiley, New York. Darken, C., and Moody, J. 1990. Fast adaptive k-means clustering: Some empirical results. lnternational Joint Conference on Neural Networks, Sun Diego, pp. I1 233-11 238. IEEE. Darken, C., and Moody, J. 1992. Towards faster stochastic gradient search. In Neural Information Processing Systems 4 . Morgan Kaufmann, San Mateo, CA. Duda, R. O., and Hart, P E. 1973. Pattern Classification and Scene Analysis. Wiley, New York. Durbin, R., and Willshaw, D. 1987. An analogue approach to the travelling salesman problem using an elastic net method. Nature (London) 326, 689691. Erwin, W., Obermayer, K., and Schulten, K. 1992. Self-organizingmaps: Ordering, convergence properties, and energy functions. Biol. Cybern. 67,47-55. Farvardin, N. 1990. A study of vector quantization for noisy channels. IEEE Transact.Inform. Theory 36(4), 799-809. Gish, H., and Pierce, J. N. 1968. Asymptotically efficient quantizing. IEEE Transact.Inform. Theory IT 14,676-683. Hanson, R., Stutz, J., and Cheeseman, I? 1991. Bayesian classification theory. Tech. Rep. FIA-90-12-7-01, NASA Ames Research Center. Japes, E. T. 1957. Information theory and statistical mechanics. Phys. Rev. 106, 620-630. Kirkpatrick, S., Gelatt, C., and Vecchi, M. 1983. Optimization by simulated annealing. Science 220, 671-680. Kohonen, T.1984. Self-Organization and Associative Memory. Springer, Berlin. Lazzaro, J., Ryckebusch, R., Mahowald, M. A., and Mead, C. A. 1989. Winnertake-all networks of o(n) complexity. In Neural Information Processing Systems I, pp. 703-711. Morgan Kaufmann, San Mateo, CA. Linde, Y., Buzo, A., and Gray, R. M. 1980. An algorithm for vector quantizer design. l E E E Transact. Commun. COM 28, 84-95. Luttrell, S. P.1989. Hierarchical vector quantisation. IEE Proc. 136,405-413. MacQueen, J. 1967. Some methods for classification and analysis of multivariate observations. Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability, pp. 281-297. Ritter, H., Martinetz, T., and Schulten, K. 1992. Neural Computation and Selforganizing Maps. Addison-Wesley, New York. Rose, K., Gurewitz, E., and Fox, G. 1990. Statistical mechanics and phase transitions in clustering. Pkys. Rev. Lett. 65(8), 945-948. Rumelhart, D. E., and Zipser, D. 1985. Feature discovery by competitive learning. Cog. Sci. 9,75-112. Simic, P. 1990. Statistical mechanics as the underlying theory of "elastic" and "neural" optimizations. Network 1,89-103. Wong, Y. 1992. Clustering data by melting. Neural Cornp. 5(1), 89-104. Yuille, A. L. 1990. Generalized deformable models, statistical physics, and matching problems. Neural Comp. 2(1), 1-24. Received 4 December 1991; accepted 18 August 1992.
This article has been cited by: 2. Ya-Jun Zhang, Zhi-Qiang Liu. 2002. Self-splitting competitive learning: a new on-line clustering paradigm. IEEE Transactions on Neural Networks 13:2, 369-380. [CrossRef] 3. J. Vesanto, E. Alhoniemi. 2000. Clustering of the self-organizing map. IEEE Transactions on Neural Networks 11:3, 586-600. [CrossRef] 4. Isaac David Guedalia , Mickey London , Michael Werman . 1999. An On-Line Agglomerative Clustering Method for Nonstationary DataAn On-Line Agglomerative Clustering Method for Nonstationary Data. Neural Computation 11:2, 521-540. [Abstract] [PDF] [PDF Plus] 5. T. Hofmann, J.M. Buhmann. 1998. Competitive learning algorithms for robust vector quantization. IEEE Transactions on Signal Processing 46:6, 1665-1675. [CrossRef] 6. Chi-Sing Leung, Lai-Wan Chan. 1998. An error control scheme for transmission of vector quantization data over noisy channels. IEEE Transactions on Signal Processing 46:10, 2767-2780. [CrossRef] 7. T. Hofmann, J.M. Buhmann. 1997. Pairwise data clustering by deterministic annealing. IEEE Transactions on Pattern Analysis and Machine Intelligence 19:1, 1-14. [CrossRef] 8. Ibrahim H. Osman, Gilbert Laporte. 1996. Metaheuristics: A bibliography. Annals of Operations Research 63:5, 511-623. [CrossRef] 9. Steven Gold, Anand Rangarajan, Eric Mjolsness. 1996. Learning with Preknowledge: Clustering with Point and Graph Matching Distance MeasuresLearning with Preknowledge: Clustering with Point and Graph Matching Distance Measures. Neural Computation 8:4, 787-804. [Abstract] [PDF] [PDF Plus] 10. David Miller , Kenneth Rose . 1996. Hierarchical, Unsupervised Learning with Growing via Phase TransitionsHierarchical, Unsupervised Learning with Growing via Phase Transitions. Neural Computation 8:2, 425-450. [Abstract] [PDF] [PDF Plus] 11. Jianchang Mao, A.K. Jain. 1996. A self-organizing network for hyperellipsoidal clustering (HEC). IEEE Transactions on Neural Networks 7:1, 16-29. [CrossRef] 12. N. Barkai, H. Sompolinsky. 1994. Statistical mechanics of the maximum-likelihood density estimation. Physical Review E 50:3, 1766-1769. [CrossRef] 13. T L H Watkin, J -P Nadal. 1994. Journal of Physics A: Mathematical and General 27:6, 1899-1915. [CrossRef]
Communicated by John Moody and Steve Zucker
Clustering Data by Melting Yiu-fai Wong Department of Electrical Engineering, 116-81, California Institute of Technology,Pasadena, CA 91125 USA We derive a new clustering algorithm based on information theory and statistical mechanics, which is the only algorithm that incorporates scale. It also introduces a new concept into clustering: cluster independence. The cluster centers correspond to the local minima of a thermodynamic free energy, which are identified as the fixed points of a one-parameter nonlinear map. The algorithm works by melting the system to produce a tree of clusters in the scale space. Melting is also insensitive to variability in cluster densities, cluster sizes, and ellipsoidal shapes and orientations. We tested the algorithm successfully on both simulated data and a Synthetic Aperture Radar image of an agricultural site with 12 attributes for crop identification. 1 Introduction
Clustering is an important problem that can be found in many applications where a priori knowledge about the distribution of the observed data is not available (Duda and Hart 1973; Jain and Dubes 1988). Simply stated, the goal is to partition a given data set into several compact groups. Each group indicates the presence of a distinct category in the measurements. It is widely used for exploratory data analysis in diverse disciplines. The literature is therefore spread among many different fields over many years. It is almost impossible to cite each contribution individually. One of the early algorithms was invented by Lloyd (1982), which was later extended by Linde et al. (1980) for vector quantization. In pattern recognition, the ISODATA algorithm (Ball and Hall 1967) and its sequential version, the k-means clustering algorithm, have been extensively used. Other algorithms include the fuzzy techniques (Ruspini 1969; Bezdek 1981; Gath and Geva 1989; Rose et al. 1990) and the hierarchical techniques such as the agglomerative and divisive methods (see Wishart 1969). These algorithms, however, suffer from several difficulties: (a) they are highly sensitive to the initialization; (b) they perform poorly if the data contain overlapping clusters; and (c) they also suffer from the inability to handle variabilities in cluster shapes, cluster densities, and cluster Neural Computation 5,89404 (1993)
@ 1993 Massachusetts Institute of Technology
90
Yiu-fai Wong
sizes. The most urgent problem is the lack of cluster validity criteria (Ekzdek 1981). All the algorithms tend to create clusters even when no natural clusters exist in the data. In this paper, we examine a fundamental way of looking at the problem of clustering, and derive a new algorithm based on information theory and statistical mechanics. We identify clustering with heating up a thermodynamic system, giving rise to hierarchical clustering in the scale space. Melting can also account for variability in cluster densities, cluster sizes, and cluster shapes (ellipsoids). The algorithm was tested successfully on both simulated data and a SyntheticAperture Radar (SAR) image of an agricultural land with 12 attributes for crop identification (Wong et al. 1992;Wong and Posner 1992). A main contribution of this paper is that this interdisciplinary approach from information theory, thermodynamics, and nonlinear dynamics can provide a proper formulation for effective clustering and related optimization problems.
2 Scale and Cluster Independence
Intuition tells us that the number of clusters depends on the scale we look at the data. At a very coarse scale, the whole data set is a cluster, whereas at a very fine scale, every datum is itself a cluster. Scale has not been exploited by the other clustering techniques, though the idea of scale space has been around for a long time (Gabor 1946;Koenderink 1984). Wong (1992)introduced a concept called “cluster independence.” To explain it, consider the situation where several people are given the same data and the same rule about clusters. Each is told to stop once a cluster is found. If they do not communicate, it is clear that the assignments of the clusters are independent. If clusters indeed exist, the information should be present in the data itself. The notion of scale implies that the data points near the cluster centers should give more information while the data points far away should give less. This can be implemented by assigning a cost of having a data point reveal the cluster locations. To make a cluster robust, the information should be spread among the data. If we treat the contributions to the determination of a cluster from all the data points as a probability distribution, this means that this probability distribution should be chosen such that its entropy is maximized subject to a linear cost constraint Uaynes 1957). Cluster independence allows us to consider one cluster at a time. Suppose the cost function is e ( x ) = ( x - Y ) ~where x is a datum and y is a cluster center. This means that we use the squared distance as a measure of the compactness of a cluster. Let P ( x ) denote the contribution
Clustering Data by Melting
91
of datum x to y . Maximizing the entropy
-
c
P(X)logP(x)
(2.1)
X
subject to the constraint
C P ( x ) e ( x )= C
(2.2)
X
one obtains
p( x) = e-P(X-Y)’ /z
(2.3)
where Z = Cxe-fl(X-y)2.To make the connection with thermodynamics, we define the “free energy”
F=--
;
l0gZ
(2.4)
At equilibrium, it is known that a thermodynamic system settles into configurations that minimize its free energy. That is, we want a F / a y = 0, or equivalently, (2.5) the weighted mean of the data. We point out that equation 2.5 is very different from that obtained by the maximum-likehood estimate of a Gaussian mixture (Wolfe 1970; Cheeseman et al. 1988). Unlike these Bayesian approaches, our method does not assume any particular data distribution. Without loss of generality, we restrict the notation and the exposition to the case of one-dimensional data. The case of higher dimensional data was treated in Wong (1992). The good news is that the dynamics are essentially the same. Definition 1. A nominal cluster is centered at y if and only if y is u local minimum of the free energy of the thermodynamic system described above. Equation 2.5 is only a necessary condition for y to be a cluster center. The sufficient conditions will become clearer as the “melting” process is explained. The details can be found in Wong (1992). Because of that, we will use “cluster” instead of “nominal cluster.” Without worrying whether nominal clusters are real clusters, one can ask the following questions: 1. Do clusters exist? This depends on whether the equation has any solutions. 2. How many clusters are there? This depends on the number of
solutions the equation has.
Yiu-fai Wong
92
3. How do the clusters evolve? The answer is given by the trajectories of the solutions of equation 2.5 as p varies.
The above list could have been longer but it suffices to illustrate the importance of equation 2.5. Since we are concerned only with local minima, this is a great advantage over other applications of physical optimization where global optima are sought. 3 Melting and Its Dynamics
Solutionsof equation 2.5 cannot be computed analytically. However, they are identical to the fixed points of the following one-parameter map':
This connects our problem with nonlinear dynamics. Figure 1 is a plot of the map for 20 data points along a unit interval with a large p. It is clear that the free energy F acts as the Lyapunov function (Wiggins 1990) for the mapping. The difference between successive ys is -$3F/&j. Thus, the ys march down the surface of the free energy and settle down in some local minimum. The mapping 3.1 exhibits no chaotic behavior. Hence, solutions can always be computed iteratively and the convergence is exponentially fast. One can see that /3 truly captures the notion of scale. At a very large p, every datum is itself a cluster, while at a very small p, the whole data set is a cluster. The essence of the algorithm is thus as follows. Start with a huge p ( h e scale); initialize every datum as a cluster. As /3 is gradually decreased, the number of clusters decreases due to the merging of the clusters. When two clusters merge, the associated data points are merged as well. Eventually the whole data set is a cluster. Specifically, the Melting Procedure is as follows:
pmXis a number related to the dynamic range and an assumed noise in the observations;
1. Choose p-; 2. set i = 1,
=
3. let every data point be a cluster; 4. iterate according to the mapping 3.1 N times or until the clusters converge. In our simulations, N = 200;
5. record the new cluster centers; 'We could have instead established a similar equivalence with the differential equa- y)e-o(x-Y)', but the analysis is very similar and the results are the tion dyldt = same.
cx(x
Clustering Data by Melting
93
c
I 0
0.2
0.6
0.4
0.8
1
Y
Figure 1: The map for 20 data points along the unit interval. 6. if more than two clusters that previously are distinct share the same center, the set of data associated with the new cluster is the union of those with the original clusters; 7. i = i
+ 1, pi = &1/1.05;
8. if there is more than one cluster, go to 4; else Melting is complete.
It is clear that the Melting Procedure generates a strict tree structure in the scale space, analogous to a dendrogrum. Figure 2 shows an example of the Melting Procedure for a set of onedimensional data that has two clusters. The graphs are obtained by computing the fixed points of equation 3.1 as scale increases. The horizontal axis indexes i in the Melting Procedure. We merely identify scale with i, which is plotted logarithmically because of the exponential terms in equation 2.5. The original data are plotted as *s at i = 0. The dynamics involved in the merging process can be studied using local bifurcation theory (Wiggins 1990). The necessary condition for bifurcation to occur is af/tIy = 1. That is,
Yiu-fai Wong
94
-1.5' 0
20
40
60
80
100
I
I20
Figure 2: The fixed points versus scale. The leftmost points are the data.
In Wong (1992), two types of bifurcations were identified: pitchfork and saddle-node. Their bifurcation diagrams are shown in Figure 3, which show the trajectories of the fixed points as the parameter ,f3 is varied around its critical value. In a pitchfork bifurcation, two clusters continuously merge into a cluster while in a saddle-node bifurcation, a cluster becomes unstable and is siphoned into another cluster. Such bifurcations can be seen in Figure 2. The interpretation of these two bifurcations for cluster analysis is as follows (Wong 1992): A pitchfork bifurcation indicates (1) uniformly spaced or nonclustered data, or (2) clustered data but with a high degree of symmetry at certain scales. A saddle-node bifurcation indicates the inhomogeneous spatial distribution present in the data. As one expects, in clustering data, saddle-node bifurcations will be most frequently observed. It is now clear why we choose to "melt" the system starting from a low temperature, as contrasted with annealing (Kirkpatrick et al. 1983). Annealing would fail since saddle-node bifurcation implies that we do not know how much hill-climbing is needed to reach the other local minimum. We can also find an information-theoreticbasis for condition 3.2. The rate distortion function deals with the question of the minimum number of bits needed to encode a source symbol subject to an expected distortion
Clustering Data by Melting
a
Y
b
Y
95
stable branch unstable branch
5
1n=p
Figure 3 (a) Pitchfork bifurcation in our clustering scheme. bifurcationin OUT clustering scheme.
(b)
Saddle-node
constraint (Pierce and Posner 1980). For a Gaussian source with variance u2 and the average distortion 5 6,
(3.3) Thus, when equation 3.2 becomes an equality/ R(6) = 0 signifies that there is no need to waste bits to encode the source. The cluster should either disappear or be merged. 4 What Is a Good Cluster and
How Many Are There?
One needs a criterion to decide the good clusters among all the clusters in the scale space. We will briefly outline the ideas in Wong (1992). Recall that p ( x ) is the contribution of a data point to a cluster. Thus the quantity fractional free energy (FFE) of a nominal cluster (4.1)
Yiu-fai Wong
96
I
0
0.2
0.4
0.6
0.8
1
Figure 4 (a) Data and the computed clusters illustratingability to handle many
clusters.
is a measure how good a cluster Q is. A large FFE indicates that most of the contributions come from the data belonging to the cluster itself and vice versa. What is large or small is set by a threshold MT,which expresses a degree of confidence. Hence, by keeping track of the fixed points and their FFE values, a criterion for deciding "good clusters" was defined in Wong (1992). We need to select the real clusters among the good clusters. It is very difficult to define a universally accepted criterion because clusters really need to be interpreted in the context of the specific applications. Nonetheless, an attempt to define a scale-based criterion was carried out in Wong (19921, which has found to be applicable in the radar application (Wong and Posner 1992). , If distinct good clusters exist in the data, their FFEs should remain good over a large range of logarithmic scale in p even though the fixed points may vary their positions slightly. In addition, the FFEs of these clusters should start out with very high value, only to drop quickly when they are about to bifurcate, hence, for a good cluster, the longer its FFE remains high, the more robust it is.
Clustering Data by Melting
'
97
O t
-0.2 0
10
20
30
40
50
60
70
80
90
0
Figure 4 (b) x-components of the trajectories of the cluster centers versus scale.
1-
0.9 0.8 -
0.7 0.6 -, 0.5
<
0.4 -;
0.3 -j 0.2 -; 0.1
-;
00
Figure 4 (c) Plots of fractional free energies for the data points in a.
Yiu-fai Wong
98
Figure 4b shows the x-component of the trajectories of the clusters for the data shown in Figure 4a.2 Figure 4c is a plot of the FFEs of the clusters versus scale. One sees that there is a range of scale over which three clusters exist. But there is a longer range of scale over which four clusters exist, which is the correct answer. Here is how we formally define the robustness of a good cluster: Definition 2. The robustness ofa good cluster is defined as the range of logarithmic scales over which its FFE remains above MT.
The rule to decide the number of clusters is to pick out the most robust ones until there are no more good clusters left. Here is the Melting Algorithm (Wong 1992): 1. Perform the Melting Procedure; 2. decide the good clusters among the nominal clusters; denote the set
by 7 = {Tl, T2,.. .
3. compute the robustness of the good clusters;
4. initialize U to an empty set; 5. while 7 is nonempty, do the following:
a. pick the element Tk in 7 with the biggest robustness measure; b. put this element into U; c. remove Tk and the elements in 7 that either are contained in or contain T’; 6. collect the data points that do not belong to one of the clusters in U into a set N, which we hope is empty.
Several remarks about the above algorithm: (1)The algorithm actually consists of two parts: melting and determination of the clusters. (2) The Melting Procedure is governed by the scale parameter ,f3 only. (3),,8, should be chosen such that the number of clusters at i = 2 is not significantly less than the number of data points to start with. Otherwise, the initial temperature is too high, which might cause premature partitioning of the data. p0 can be obtained easily by simple preprocessing. (4) The determination of the clusters is camed out in steps 2-5. We also note that step 5 can be modified to further study the finer structure of the data, such as clusters within clusters. *The data were generated from normal distributions. A “cross” denotes the center of the distribution as seen by the computer. A “circle”denotes the representative of a cluster that is just the arithmetic mean of the data in a given cluster. The horizontal axis indexes scales with fewer than 10 clusters in the Melting Procedure (to avoid too many curves). The same explanations apply to Figure 5.
Clustering Data by Melting
99
5 Ability to Handle Clusters of Oriented Ellipsoidal Shapes
Figure 5a shows a data set consisting of four clusters with various orientations and ellipsoidal shapes. Figure 5b shows the trajectories of the clusters. Figure 5c is the plots of the FFEs; it clearly shows that there are four clusters. Figure 5d shows the partition obtained by the algorithm. Note the few data points marked by Os, which get grouped into clusters different from that generated by the computer. Since they are far away from the originating cluster, such grouping is acceptable. Even without a norm which is biased in the different directions, there is a built-in dynamics in the formulation to handle oriented ellipsoidal shapes with a single p. This was also demonstrated in the radar application (Wong and Posner 1992). Here is a brief explanation. Obviously, the dynamics of the mapping 3.1 is invariant to the rotation of the coordinate system. Suppose, as is reasonable, that each cluster consists of data coming from a source corrupted with unimodal noise. Due to insufficientsampling or pure random fluctuation, the local density is not monotonically decreasing. However, at a coarser scale, it will be monotone. This implies that sooner or later, a cluster will see that its center cannot be balanced due to the monotonicity. It will try to "swim" toward the gradient until balance is reached, which is possible only in the neighborhood around the true signal.
0.8
0.60.4 -
0.2 0-0.2
-
-0.4 -
-0.6 -0.8 -1-
.0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Figure 5: (a) Data illustrating ability to handle many clusters of different shapes and sizes.
Mu-fai Wong
100
0.1
0
-0.1
-0.1
-0.3
-0.4
.---.______
-0.5
10
20
30
40
so
3
Figure 5: (b) y-components of the trajectories of the cluster centers versus scale.
1
0.9 0.8
0.7 0.6 0.5 0.4
0.3 0.1 0.1 (1
Figure 5: (c) Plots of fractional free energies for the data points in a.
Clustering Data by Melting
0.4
101
1 e
0.2 -
.
,
.
.
. .. .. . .. . .... .. . . . . . . , O ' , .. , .
0-
e
e B
*+
. . . . '0 . . ........... ....
-0.2 -
t +
2
0 '
-0.4 -
-0.6 -
-1
'
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Figure 5: (d) Clustering of the data shown in a. 0, misclassified data points. 6 Computational Complexity and Other Observations
Instead of artificial data, we will illustrate the timing on a real application to the clustering and classification of a 12-dimensional SAR image of an agricultural area (Wong and Posner 1992). Due to the attracting dynamics, the convergence is exponentially fast. It was observed that convergence to a fixed point took an average of 15 iterations at each p. The exact rate of convergence, however, depends on the Jacobian of the map (3.11, which cannot be known a priori, making it impossible to give an upper bound. For the application, the computation took 26 minutes of SPARC-I1 cpu time to find the clusters. This is very intensive compared to ISODATA (Ball and Hall 19671, which takes 3 sec provided, however, it is given the right initialization. To compare ISODATA with Melting Algorithm, we note the key point that if we initialize ISODATA wrongly, it will never find the correct clusters. Here, "wrongly" means putting more than one initial cluster in a real cluster. For the application, we have 1397 data points. There are 13 clusters, each with about 106 data points. Suppose that the initial cluster centers are assigned randomly. There are
102
Yiu-fai Wong
choices. Of these, only 10613= 2.13 x 1p6 initializations give the correct partition; even this is an overestimate since some data are noisy. Hence, the probability of a correct initialization is at most 1.82 x Since ISODATA is 520 times faster than our algorithm, its probability of getting the correct answer is 0.0095 in 26 min, which would have been lower had it not been given the number of clusters. This simple calculation shows that in an obvious sense the Melting Algorithm is at least 105 times (1/0.0095)better than ISODATA. Furthermore, one has to weigh the quality and assurance of the solution obtained by our Melting Algorithm. The current implementation of the Melting Algorithm does not include any heuristics to speed up its computation though it did use a lookup table of the exponential function. We note the following along these lines: 1. Initially, there are a huge number of clusters. Most clusters will merge quickly since they exist simply because it is too ”cold.” This effect can be seen in Figure 2. Some preprocessing such as simple grouping would reduce the complexity dramatically. 2. The purpose of the Melting Procedure is to track the trajectories of the clusters in the scale space. Instead of decreasing p by a constant factor, we can also utilize numerical techniques such as continuation (Doebel1986)and adaptive step size selection to track the bifurcation points more accurately and faster.
3. The algorithm is ideal for parallel implementation because of local calculations and cluster independence. In addition, it is possible that some partial a priori information will allow one to perform melting over a range of scales. Thus, the computational complexity of the algorithm can be improved significantly with the techniques outlined above and other heuristics that we have yet to investigate. Some preliminary work is reported in Tam (1992). 7 Summary
Clustering is a hard problem. The traditional clustering algorithms suffer from several difficulties. The willingness of existing algorithms to partition any set of data suggests that they may more suitably be named “partitioning” algorithms rather than “clustering” algorithms. In this work, we have devised a new clustering algorithm that properly exploits the notion of scale. We also introduced the notion of cluster independency, which has not been formally recognized by prior researchers. It permits the natural application of the maximum entropy principle? Cluster centers correspond to the local minima of a thermo3For related results on clustering using maximum entropy principle, see Rose et al. (1990)and the work by J. Buhmann and H. Kiihnel in this issue.
Clustering Data by Melting
103
dynamic free energy. The system is identical to a one-parameter nonlinear map, which can be rigorously analyzed using bifurcation techniques. Melting the system produces a tree of clusters in the scale space. Melting can also account for variabilities in cluster densities, sizes and shapes (ellipsoidal). We further tested this algorithm on the clustering and classification of a Synthetic Radar Aperture image of an agricultural site with 12 attributes. Since clustering is a form of unsupervised learning, we expect this work should provide some new insights for neural network research and optimization theory, too, but we will not discuss that here. Acknowledgments I am deeply indebted to my thesis advisor Edward C. Posner for his patience, encouragement, and advice. I benefited much from him. I thank Professor Steve Wiggins at Caltech for clarifying some concepts on the local bifurcations. This work is supported by Pacific Bell through a grant to the California Institute of Technology and by NASA through the Caltech Jet Propulsion Laboratory, as well as a Charles Lee Powell Foundation Graduate Fellowship at Caltech. References Ball, G., and Hall, D. 1967. A Clustering Technique for Summarizing Multivariate Data. Behuv. Sci. 12, 153-155. Bezdek, J. C. 1981. Pattern Recognition with Fuzzy Objective Function Algorithms. Plenum, New York. Cheeseman, P. et al. 1988. Autoclass: A Bayesian classificationsystem. Proceedings of the 1988 Machine LRarning Workshop. Doedel, E. 1986. AUTO: Softwure for Continuation and Bifurcation Problems in Ordinary Differential Equations. Tech. Rep., Applied Mathematics, Caltech. Duda, R. O., and Hart, I? E. 1973. Pattern Cluss$cutionand Scene Anulysis. Wiley, New York. Gabor, D. 1946. Theory of communication. 1.IEE 93,429-457. Gath, I., and Geva, A. B. 1989. Unsupervised optimal fuzzy clustering. IEEE Trans. Pattern Anal. Machine Intell. PAMI-11,773-781. Jain, A. K., and Dubes, R. C. 1988. Algorithms for Clustering Data. Prentice Hall, Englewood Cliffs, NJ. Jaynes, E. T. 1957. Information theory and statistical mechanics I. Phy. Rev. 106, 620-630. Kirkpatrick, S., Gelatt, C. D., and Vecchi, M. P. 1983. Optimizationby simulated annealing. Science 220,671480. Koenderink, J. J. 1984. The structure of images. Biol. Cybern. 50,363-370. Linde, Y., Buzo, A., and Gray, R. M. 1980. An algorithm for vector quantization. IEEE Trans. Commun. COM-28, 84-95.
104
Yiu-fai Wong
Lloyd, S. P. 1982. Least squares quantization in PCM. IEEE Trans. Inform. Theory 28(129), 137 (reprint of the 1957 paper). Pierce, J. R., and Posner, E. C. 1980. Introduction to Communication Science and Systems. Plenum Press, New York. Rose, K., Gurewitz, E., and Fox, G. C. 1990. A deterministic annealing approach to clustering. Pattern Recog. Lett. 11, 589-594. Ruspini, E. 1969. A new approach to clustering. Inform. Contr. 15,22-32. Tam, T. K. 1992. Fast and Parallel Implementation of Melting Algorithm for Clustering. Caltech Summer Undergraduate Research Fellowship Report. Wiggins, S. 1990. Introductions to Applied Nonlinear Dynamical Systems and Chaos. Springer-Verlag,New York. Wishart, D. 1969. Mode analysis: A generalization of nearest neighbor which reduces chaining effects. In Numerical Taxonomy, A. J. Cole, ed., pp. 282-308. Academic Press, London. Wolfe, J. H. 1970. Pattern clustering by multivariate mixture analysis. Multivariate Behavioral Research 5, 329-350. Wong, Yiu-fai, Peters, K. J., and Posner, E. C. 1992. Unsupervised and hierarchical cluster analysis and classificationof SAR images. Proceedings International Space Year Conference on Earth and Space Science Information Systems, Pasadena, February, to be published by AIR Wong, Yiu-fai, and Posner, E. C. 1992. A new clustering algorithm applicable to multispectral and polarimetric SAR images. IEEE Trans. Geosci. Remote Sensing, in press. Wong, Y. F., and Posner, E. C. 1992. Scale-space clustering and classification of SAR images with numerous attributes and classes. To be presented at 1992 IEEE Workshop on Applicationsof Computer Vision, November, Palm Springs. Wong, Yiu-fai 1992. T'ards a Simple and Fast Learning and Classification System. Ph.D. Thesis, Caltech, Electrical Engineering. Received 16 January 1992; accepted 12 May 1992.
This article has been cited by: 2. Miguel A. Carreira-Perpinan. 2007. Gaussian Mean-Shift Is an EM Algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence 29:5, 767-776. [CrossRef] 3. Min Wang, Yee Leung, Chenhu Zhou, Tao Pei, Jiancheng Luo. 2006. A Mathematical Morphology Based Scale Space Method for the Mining of Linear Features in Geographic Data. Data Mining and Knowledge Discovery 12:1, 97-118. [CrossRef] 4. Fahui Wang. 2005. Job Access and Homicide Patterns in Chicago: An Analysis at Multiple Geographic Levels Based on Scale-Space Theory. Journal of Quantitative Criminology 21:2, 195-217. [CrossRef] 5. Koji Okuhara, Koji Sasaki, Shunji Osaki. 2000. Reproductive and competitive radial basis function networks adaptable to dynamical environments. Systems and Computers in Japan 31:13, 65-75. [CrossRef] 6. Yee Leung, Jiang-She Zhang, Zong-Ben Xu. 2000. Clustering by scale-space filtering. IEEE Transactions on Pattern Analysis and Machine Intelligence 22:12, 1396-1410. [CrossRef] 7. Isaac David Guedalia , Mickey London , Michael Werman . 1999. An On-Line Agglomerative Clustering Method for Nonstationary DataAn On-Line Agglomerative Clustering Method for Nonstationary Data. Neural Computation 11:2, 521-540. [Abstract] [PDF] [PDF Plus] 8. A.N. Srivastava, R. Su, A.S. Weigend. 1999. Data mining for features using scale-sensitive gated experts. IEEE Transactions on Pattern Analysis and Machine Intelligence 21:12, 1268-1279. [CrossRef] 9. K. Rose. 1998. Deterministic annealing for clustering, compression, classification, regression, and related optimization problems. Proceedings of the IEEE 86:11, 2210-2239. [CrossRef] 10. Marcelo Blatt , Shai Wiseman , Eytan Domany . 1997. Data Clustering Using a Model Granular MagnetData Clustering Using a Model Granular Magnet. Neural Computation 9:8, 1805-1842. [Abstract] [PDF] [PDF Plus] 11. Koji Tsuda, Shuji Senda, Michihiko Minoh, Katsuo Ikeda. 1997. Sequential fuzzy cluster extraction and its robustness against noise. Systems and Computers in Japan 28:6, 10-17. [CrossRef] 12. S.V. Chakravarthy, J. Ghosh. 1996. Scale-based clustering using the radial basis function network. IEEE Transactions on Neural Networks 7:5, 1250-1261. [CrossRef] 13. N. Barkai, H. Sompolinsky. 1994. Statistical mechanics of the maximum-likelihood density estimation. Physical Review E 50:3, 1766-1769. [CrossRef] 14. Joachim Buhmann , Hans Kühnel . 1993. Complexity Optimized Data Clustering by Competitive Neural NetworksComplexity Optimized Data Clustering by
Competitive Neural Networks. Neural Computation 5:1, 75-88. [Abstract] [PDF] [PDF Plus] 15. Ravi KothariData Mining . [CrossRef]
Communicated by John Platt
Coarse Coding Resource-Allocating Network Gustavo Deco Jiirgen Ebmeyer Siernans AG, SI AT 3, Landshuter Strusse 26, 8044 Miinchen-Unterschleissheim, Miinchen, Germany In recent years localized receptive fields have been the subject of intensive research, due to their learning speed and efficient reconstruction of hypersurfaces. A very efficient implementation for such a network was proposed recently by Platt (1991). This resource-allocating network (RAN)allocates a new neuron whenever an unknown pattern is presented at its input layer. In this paper we introduce a new network architecture and learning paradigm. The aim of our approach is to incorporate "coarse coding" to the resource-allocatingnetwork. The network presented here provides for each input coordinate a separate layer, which consists of one-dimensional, locally tuned gaussian neurons. In the following layer multidimensional receptive fields are built by using pi-neurons. Linear neurons aggregate the outputs of the pineurons in order to approximate the required input-output mapping. The learning process follows the ideas of the resource-allocating network of Platt but due to the extended architecture of our network other improvements of the learning process had to be defined. Compared to the resource-allocating network a more compact network with comparable accuracy is obtained. 1 Introduction
Several authors have analyzed localized receptive fields for solving the problem of hypersurface reconstruction. Layered neural networks with locally tuned units (Moody and Darken 1989) or hyper-basis functions (Poggio and Girosi 1990a,b) have been formulated and successfully applied. The use of such gaussian neurons was proposed due to their relation with the theory of approximation. Lapedes and Farber (1987) suggested further that multilayer neurons, with sigmoidal functions are effective because they build gaussian-like bumps, which perform the approximation of the hypersurface. The use of gaussian neurons has brought improved accuracy and faster learning. The traditional pattern recognition methods, such as the k-nearest neighbors or Parzen windows, suffer from the same disadvantage as localized receptive field networks: they allocate too many units Neural Computution 5,10S114 (1993) @ 1993 Massachusetts Institute of Technology
106
Gustavo Deco and Jiirgen Ebmeyer
during the learning phase. This may be problematic for the network's capability to generalize, as pointed out by Platt (1991). Recently, a very ingenious architecture and learning paradigm, called a resourceallocating network (RAN), was proposed by Platt (1991). The basic idea of this network is the construction of an architecture capable of adjusting efficiently the number of units to obtain a compact and very accurate network. Poggio (1990) has pointed out that radial basis functions are factorizable and that their synthesis is much easier when factorized. This method is called "coarse coding." Poggio suggested that the model that could explain how 3D objects are learned, by the cortical cells involved in visual face recognition. He also discussed the relation of this model to theories of cerebellum and motor control (Poggio 1990). The aim of the present work is to formulate a RAN-like self-building architecture which uses coarse coding to improve the accuracy and reduce the complexity of the network. We call this a coarse coding resourceallocating network, henceforth referred to as CC-RAN. In the first section of the paper the architecture of this network is described and in the second section the learning algorithm is presented. This network was implemented in C++ on a Unix-Workstation and tested with the standard benchmark Mackey-Glass chaotic time series. Results and discussion are given in the final section. 2 Architecture of the Network
In this section the architecture of CC-RAN is described. To simpl* the theoretical description we will discuss in this paper the network architecture for approximation of functions of R" to R. The generalization for functions of R" to R" follows trivially. Figure 1 shows the graphic representation of CC-RAN. Each input variable Xi is connected with a separate layer R F k of one-dimensional, gaussian receptive fields. The second layer consists of pi-neurons (Rumelhart etal. 1986)that synthesize the factorized receptive fields by selecting one local unit per lW layer and multiplying them together. As output layer, one linear neuron performs the linear combination of the multidimensional functions. The neurons of the first layers R F k (receptive field k) are locally tuned units. The neuron m of this layer R F k responds active only when the coordinate xk of the presented input x is in the neighborhood of the center cr. We implemented these neurons following Platt (1991) and Moody and Darken (1989) but now, as one-dimensional gaussian fields, (2.1)
where C T ~ is the width and c r the center of the gaussian function.
Coarse Coding Resource-Allocating Network
x
,
x
*
x3
107
.......
X k
.......
Xn
Figure 1: Graphic representation of the coarse coding resource-allocating network introduced in this paper. Each input coordinate is connected with onedimensional locally tuned neuron in a separate RF layer. The synthesis of the one-dimensional receptive fields is performed by pi-neurons. The height and combination of this multidimensional gaussian function is calculated in the linear output neuron to achieve a best approximation to the given hypersurface.
The pi-neurons of the second layer perform the following operation (see also Durbin and Rumelhart 1989),
p' -
n
i p p
k = 0,. . . ,dimension of input vector
(2.2)
' - ( k
The k-index extends to all RF layers. In equation 2.2pi is the output of the pi-neuron i. The connection of pi-neuron i with the layer R F k [indicated by n(k,i)l is determined during the learning process. The
Gustavo Deco and Jurgen Ebmeyer
108
linear output neuron evaluates the outputs of the pi-neurons by
i = 0, . . . ,number of pi-units
The y gives the output when all pi-units are deactivated. Wi are the synapses between the pi-neurons of the second layer and the output neuron. These weights represents the height of the synthesized gaussian function. This structurewill be constructed and adjusted optimally by the leaming algorithm. This is described in the next section. The coarse coding was introduced by using a layer of one-dimensional, locally tuned units for each coordinate of the input vector. It is interesting to remark that this network satisfies the Stoneweirstrass theorem. This means that this network describes a universal approximator for real-valued maps defined on convex, compact sets of R" (see Hartman et al. 1990). We define the set,
It is easy to prove that N is an algebra of real continuous functions on a compact set Q, and that it separates points of Q and that it does not vanish at any point on Q; these are the preconditions of the Stone theorem. So a uniform closure of N contains all real valued continuous functions of Q. 3 Learning Paradigm for CC-RAN
We follow 'the idea of Platt (1991) for the implementation of the allocation and learning rules for CC-RAN.The network generation process starts initially without any neuron. Each layer of the one-dimensional receptive fields (RF)has an allocator. This allocator creates a new locally tuned neuron in layer R F k if two novelty conditions are satisfied. The first condition considers a pattern as new if the minimal distance of the coordinate xk relative to the centers c? is greater than a determined barrier d, i.e.,
The second condition is given by llTP - OPll
'
E
(3.2)
which means that new neurons will not be created for smaller corrections.
Coarse Coding Resource-Allocating Network
109
The barrier d ( t ) shrinks each training epoch following the exponential decay,
d ( t ) = max(Minima1d, e-f’r Maximal d )
(3.3)
In equation 3.3 T is the decay factor. The width and the center of the created neurons are given by
(3.4) (3.5) The second layer of pi-neurons has its own allocator of pi-neurons. This allocator creates a new pi-neuron in two different cases. The first case requires the satisfaction of two conditions. A new pi-neuron is created if at least one of the RF layers has created a new one-dimensional gaussian unit. The second condition demands equation 3.2 to be satisfied. The new created pi-neuron will be connected to the new one-dimensional gaussian units. In the others RF layers where no new gaussian unit has been created, the gaussian unit with the nearest center to the respective input will be connected to the new pi-neuron. The second case to create a pi-neuron is present, if no new gaussian neurons are created and no pi-neuron exists that connects the gaussian neurons in such a way that the distance between the input and the center of the multidimensional receptive field (built by all RF layers) is smaller than d ( t ) . In this case the new neuron will connect the neurons of the different RF layers with the nearest center to each input. The new connections, Wi, of the newly created pi-neuron with the output layer are given by
After the coarse representation of the function, new units with smaller widths are successively inserted until the network has learned the examples with the desired accuracy and resolution. This resolution is essentially given by the minimal value of d ( t ) . In the case that neither gaussian neurons nor pi-neurons are created the traditional gradient method for smaller corrections is used. The correction equations are
(3.8)
Gustavo Deco and Jiirgen Ebmeyer
110
In equation 3.9 the summation is extended over all ,'i so that n ( K , i') = i, in other words all pi-units that are connected with the gaussian unit i of layer k. As pointed out by Platt, the two novelty conditions are necessary for creating a compact network. In this way new neurons represent exactly the novelty patterns, and the small aberrations are corrected using the gradient method. Due to the use of pi-neuron and the independent allocation of RF neurons at each input dimension, CC-RAN forms more compact networks than RAN does, because the storage of each dimension happens separately, so that one particular RF center can be used in many pi-neurons without having to be restored in a new neuron every time. 4 Results
The standard test case for regularization networks is the prediction of the chaotic time series defined by Mackey and Glass (1977). The delay difference equation of Mackey-Glass can be expressed as x(t
T) + 1) = (1- b)x(t)+ 1u+ xx (( tt -- T)'O
(4.1)
We tested CC-RAN with u = 0.2, b = 0.1, and T = 17. To compare with other neural and nonneural models the training set contained four points: x ( t ) , x ( t - 6), x ( t - 12), x(t - 18) and the network had to predict the output x ( t + 85). The used parameters for CC-RAN are = 0.02, 7 = 25, dmax = 0.7, dmin = 0.07 or 0.03, E = 0.05, IC = 0.87, and 400 learning epochs. The test set consists of 500 points of the output of the Mackey-Glass equation at t = 4000. Figure 2 shows the normalized error (rms error divided by the square root of the variance of the output of equation 4.1 as a function of the training set size). A normalized error equal to one indicates that the prediction is not better than the mean value. For comparison the curves of other approximation methods are also given. A better generalization capability can be observed when a training set with 100 points is used. The measurement of the complexity of the network is shown in Figure 3. This figure relates the dependence of the number of weights (height, width, and centers of the gaussian functions) of the networks to the size of the training set. If the size of the training set grows, the increase in complexity is very small for the RAN and CC-RAN network. Figure 4 shows the normalized error as a function of the number of weights. In this figure it can be seen that CC-RAN is a remarkably compact network (small number of weights) achieving a similar accuracy as the other ones. Even in those cases where CC-RAN (dmin = 0.03) performs more accurately than RAN, its complexity is still smaller than that of RAN. The essential fact that contributes to this compactness is the use of the same one-dimensional receptive field by many different pineurons. The improvement of the independent creation of RF layers lies
Coarse Coding Resource-Allocating Network
5
1
100
1
I
I I I I I I
1
I
111
I
1 I I I I I
lo00
I
I
l
l
10000
Slze of Training Set
Figure 2 Normalized error as a function of the size of training set. ( 0 )CC-RAN (dmin = 0.07); (*I CC-RAN (dmin = 0.03); (0)RAN ( E = 0.05, of Platt 1991); (*I Backpropagation; (A)hashing B-spline; (0)standard RBF; (m) K-means RBF. All results of the other approximationsare taken from Platt (1991). in the fact that they permit us to learn and adapt compactly and efficient each component of a multidimensional input that can be strongly asymmetric and/or of different range and dispersion. This was the reason for the implementation of coarse coding as a resource-allocating network. 5 Conclusions
A new architecture and learning algorithm, called coarse coding resourceallocating network is proposed for a self-building neural network. The network learns by allocating new neurons to the receptive field layers as well as to the pi-neuron layer, if determined novelty conditions are
112
Gustavo Deco and Jiirgen Ebmeyer
She of Training Set
Figure 3: Number of weights of the networks or algorithms as a function of the size of training set. ( 0 )CC-RAN (dmin = 0.07); ( a )CC-RAN ( d m i n = 0.03); (0) RAN ( E = 0.05,of Platt 1991); (*) Backpropagation; (A) hashing B-spline; (Q) standard RBF; (B) K-means RBF. All results of the other approximations are taken from Platt (1991).
satisfied (i.e./ if substantial differences to already learned patterns are found). If only small aberrations between expected and actual output are observed, only small adjustments are made by applying of the gradient correction method to the centers of each one-dimensional receptive field and on the height of the synthesized multidimensional gaussian function by the pi-neurons. More compact networks are obtained by separate allocation of RF centers for each dimension, which may be used in many pi-neurons without having to be restored in a new neuron every time. The compactnessachieved by the final network in learning the chaotic series of Mackey-Glass was important for improved generalization.
Coarse Coding Resource-AllocatingNetwork
113
E
100
lo00
1woo
Number of Weighta
Figure 4 Normalized e m r as a function of the number of weights of the networks or algorithms. ( 0 )CC-RAN (dmin = 0.07); (*)CC-RAN (dmin = 0.03); (0) RAN ( E = 0.05, of Platt 1991); (*)Backpropagation; (A) hashing B-spline; (0) standard RBF; (a)K-means RBF. All results of the other approximations are taken from Platt (1991). Acknowledgments We gratefully acknowledge helpful comments from the editor and J. Cuellar.
References Durbin, R., and Rumelhart, D. 1989. Product units: A computationally powerful and biologically plausible extension to backpropagation networks. Neural Comp. 1, 133-142.
114
Gustavo Deco and Jurgen Ebmeyer
Hartman, E., Keeler, J., and Kowalski, J. 1990. Layered neural networks with gaussian hidden units as universal approximations. Neural Comp. 2, 210215. Lapedes, A., and Farber, R. 1987. Nonlinear signal processing using neural networks: Prediction and system modeling. Tech. Rep. LA-UR-87-2662, Los Alamos National Laboratory, Los Alamos, NM. Mackey, M., and Glass, L. 1977. Oscillation and chaos in physiological control systems. Science 197, 287. Moody, J., and Darken, C. 1989. Fast learning in networks of locally-tuned processing units. Neural Comp. 3, 281-294. Platt, J. 1991. A resource-allocating network for function interpolation. Neural Comp. 3,213-225. Poggio, T. 1990. A theory of how the brain might work. A.I. Memo No. 1253, Artificial Intelligence Laboratory, Massachusetts Institute of Technology. Poggio, T., and Girosi, F. 1990a. Networks for approximation and learning. Proc. IEEE 78(9), 1481-1497. Poggio, T., and Girosi, F. 1990b. A theory of networks for approximation and learning. A.I. Memo No. 1140, Artificial Intelligence Laboratory, Massachusetts Institute of Technology. Rumelhart, D., Hinton, G., and Williams, R. 1986. Learning representations by back-propagating errors. Nature (London) 323,533. Received 29 January 1992; accepted 27 April 1992.
This article has been cited by: 2. Tin-Yau Kwok, Dit-Yan Yeung. 1997. Constructive algorithms for structure learning in feedforward neural networks for regression problems. IEEE Transactions on Neural Networks 8:3, 630-645. [CrossRef] 3. G. Deco , W. Finnoff , H. G. Zimmermann . 1995. Unsupervised Mutual Information Criterion for Elimination of Overtraining in Supervised Multilayer NetworksUnsupervised Mutual Information Criterion for Elimination of Overtraining in Supervised Multilayer Networks. Neural Computation 7:1, 86-107. [Abstract] [PDF] [PDF Plus] 4. Bjørn Lillekjendlie, Dimitris Kugiumtzis, Nils Christophersen. 1994. Chaotic time series. Part II. System Identification and Prediction. Modeling, Identification and Control: A Norwegian Research Bulletin 15:4, 225-245. [CrossRef]
Communicated by Michael Jordan
Training Periodic Sequences Using Fourier Series Error Criterion James A. Kottas* h4IT EECS Department, Massachusetts Institute of Technology, Cambridge, MA 02139 USA Training a network to learn a set of periodic input/output sequences effectively makes the network learn a mapping between amplitudes and phases in Fourier space. The spectral backpropagation (SBP) training algorithm is a different way of doing this task. It measures the Fourier series components of the output error sequences and minimizes the total spectral energy as an adaptation criterion. This approach can train not only the weights but also time delays associated with the interconnects. Furthermore, the cells can have finite bandwidth via a first-order low-pass filter. Having adaptable time delays gives the SBP algorithm a powerful way to control the phase characteristics of the network. 1 Introduction In many areas such as adaptive controls, speech, and signal processing, the signals of interest are functions of time rather than vectors to be classified. Many researchers have investigated processing temporal sequences using neural networks (for example, see Jordan 1989; Pearlmutter 1989; and Pineda 1989). Consider the special case when the sequences are periodic. This can occur, for example, when trying to recognize and distinguish between the characteristic limit cycles produced by a neural model of the olfactory bulb (Yao and Freeman 1990). One way to do this task is to use a timedelay neural network (Waibel et al. 1989). This type of network essentially converts a temporal sequence into a vector and then uses a parallel mapping network to perform any recognition or classification. The network can recognize a sequence after one period and is not constrained to use periodic sequences. However, the serial input buffer (the time-delay part) must be as long as the minimum sequence period. For long sequences, the buffer could become quite large. Another way to solve this type of problem is to use a network for performing amplitude and phase processing on the sequence to transform it 'Current address: Syrnbus Technology, 1330 Beacon Street, Suite 249, Brookline, MA 02146 USA.
Neural Computation 5,115-131 (1993) @ 1993 Massachusetts Institute of Technology
JamesA. Kottas
116
into a form that can be recognized more easily. This approach involves training a network to learn a set of periodic input/output sequences. Although algorithms such as Pearlmutter’s (1989) continuous-time formulation of the conventionalbackpropagationequations (Rumelhartet d. 1986) and Pineda’s (1989) recurrent backpropagation algorithm could be used here, the technique described herein called the spectral backpropagation (SBP) algorithm offers definite advantages. The SBP algorithm uses the Fourier series spectrum of the desired output sequences to form a spectral error criterion that is then backpropagated through the network. Essentially, the SBP algorithm is an extension of the conventional recurrent backpropagation algorithm for descending an error surface in the Fourier domain. Although more computationally intensive, the SBP algorithm allows not only the weights but also time delays associated with the interconnects to be trained. This ability allows the training process direct access to the phase components of the sequence relationships to be learned. Furthermore, the cells in the network can have finite bandwidth via a first-order low-pass filter. When the cells have nonlinear output functions, the resulting spreading of spectral energy can be handled without difficulty. However, having adaptable time delays can permit simple linear networks to be used for processing temporal sequences rather than more complex nonlinear networks. In this paper, the basic structure of the SBP training algorithm is described along with representative simulation results. Throughout the discussion, continuous-time functions are denoted by f (t) and discrete-time functions by f [t]. 2 Spectral Backpropagation ’Raining Algorithm
Consider a network of cells that are described by the continuous-time equations, (2.1)
(2.2) (2.3) where ui(t) is the weighted input sum for the ith cell at time t, xi(t) is the external input, vi(t) is the filtered cell input with time constant rv, and yi(t) is the cell output generated from the filtered input via the function Si. Each interconnect to the ith cell from cell j has weight wi, and time delay Tg. Assuming yi(t) is smooth, periodic, and “slowly varying,” it can be approximated by the truncated Fourier series, K
yj(t) z
C [Ykcos(k~ot) + yksin(kwot)]
k=O
(2.4)
Training Periodic Sequences
117
where wo = 2s/T0 with TObeing the fundamental period of the oscillation and K the highest harmonic to consider. The coefficients Yfk and y k represent the real-valued Fourier series amplitudes for the cosine and sine components, respectively. They can be computed using the transform,
yk
=
7 7
bk yi(f) COS(kkJ0t)dt
(2.5)
To 0 q k
= pk
lji(t)Sin(kLdof)dt
(2.6)
To 0 where
1 for k = O
(2.7)
2 fork > 0 Using the vector notation yik
=
[t]
(2.8)
the cell equations can be transformed into the Fourier domain, resulting in CoS(k~0Tij) - sin(kw0Tij) ’ yjk (2.9) ujk = x i k -k wjj sin(kwoTij) cos(kw0Tjj) i
[
vik
=
I
1
(2.10)
(2.11)
For linear cells with yi(t) = mivi(t)
(2.12)
where mi is a gain constant, the spectral cell output is simply Yjk
= miVjk
(2.13)
Note that the transformation into the Fourier domain causes the time delays Tij to become a simple quadrature phase matrix. Similarly, the cell time constant ru becomes an amplitude scaling factor that depends on the spectral hequency component kwo. The spectral error criterion, which is the basis for the SBP algorithm, is obtained by comparing the actual output sequence yi(t) to the desired
James A. Kottas
118
output sequence yjd'(t). In the time domain, the error as a function of time is (2.14) ei(t) = yjd'(t) - yi(t) where i refers only to a network output cell here, The total error for the current output sequence over all output cells (No) is given by (2.15) In the same way, the total error over all output sequences is simply the sum of E for each sequence. Since this is a linear operation, the derivation below will be done as if there was only one desired output sequence. The results are then summed over all output sequences to obtain the complete error criterion. Using Parseval's theorem, the error in equation 2.15 can be approximated by (2.16) where Ef, and E:k are the Fourier series coefficients of the temporal error sequence defined in equation 2.14. As in the conventional backpropagation algorithm, the weights and time delays are adapted according to the gradient descent driving term, (2.17) where zi, can be either a weight w, or a time delay T, and ~a(r) is the correspondingadaptation time constant with units of numbers of training epochs. Using an approach analogous to that given for the conventional backpropagation algorithm in Rumelhart et al. (19861,expressions for Awij and AT, can be derived. For output cells, the spectral cell errors for the ith cell at the kth frequency index (k = 0,1,2, . .. ,K) can be defined as r aE 1
;1
av,F,
(2.18)
6ik = -
allowing the driving term to be written as (2.19) using the chain rule. Note that the spectral cell errors 6ik are independent of z,, the weight or time delay being adapted.
Training Periodic Sequences
119
The term that is dependent on Zij, dVik/azij VZijVik, can be derived from the spectral cell equations in 2.9 and 2.10. When zij is the weight wij, d V i k / h i j = VwijVik is given by (2.20) where
and
These coefficients incorporate the effects of both the cell filter and the interconnect time delais. When zi, is the time delay Tq, aVik/aTV = VTijVik is given by2 Alk (Tij)
8Tij
(2.23)
- -wij
AdTij)
where (2.24)
-av,c,8Tij
avik
-
- aTij -
= -Wijkwo
[
A2k(Tij)
1 [J
Alk(Tij)
-A~k(Tij) Aa(Tij)
y k
James A. Kottas
120
Given the expressions for aVik/dzij,only the spectral cell errors need to be determined in order to compute the adaptation driving term Azij. For an output cell, (2.27)
in general. If the output cell is linear so yi(t) = mivi(t), the spectral cell error simplifies to
6ik = miEjk
(2.28)
where E i k is the set of Fourier series components of the error sequence defined in equation 2.14. However, if the output cell has a nonlinear output function Si, the spectral components of vi(t) are spread across the frequency spectrum. This effect is captured by the term,
(2.29)
The components of this matrix can be calculated using (2.30) (2.31)
(2.32)
(2.33)
where (2.34)
and Si(v) = dSi(v)/dv. Clearly, the nonlinear case involves much more computation than when the cells are linear.
Training Periodic Sequences
121
For a hidden cell (one whose output is not an output of the network), the spectral cell errors can be computed using the recursive backpropagation relationship, T
(2.35)
When the hidden ce is linear, the backpropagation expression can be simplified to (2.36)
Using these expressions, the adaptation driving terms Awij and ATq can be computed for each adaptable interconnect. 3 The Training Process
A training epoch consists of the set of desired training sequences, each repeated n, times, in succession. An example epoch for two training cycles is shown in Figure 1. At the beginning of an epoch, the error gradients Azij(t) are initialized to 0. The first training sequence, having period TI (so TO= TI),is presented for ne - 1 cycles to allow transients to decay away. During the neth cycle, the output error El for the first sequence is computed along with the spectral Coefficients Yik, dYik, and Yik. At the end of this cycle, the gradient information Azij(t) is computed and saved. Next, the second training sequence with period 'T (so TOnow is set to Tz) is applied to the network. After n, - 1 transient cycles, the output error E2 and a new set of spectral coefficients are measured during the n,th cycle. The gradient information for this sequence is calculated and added to the previous Azij(t)from the first training sequence. If there was a third training sequence, the above procedure used for the second sequence would be applied again using this third sequence. After all training sequences have been cycled f i e times and the resulting gradient terms Azij(t ) accumulated, the weights and/or time delays are updated and the total training error is found by summing the individual output errors, Ci Ei. Alternatively, the weights and/or time delays can be adapted immediately after each training sequence. In this case, AzV(t)is not integrated over all sequences but used after each sequence. While this method is not a true gradient descent, it usually converges faster in practice than when the interconnect updates are done only at the end of the epoch.
James A. Kottas
122
One Training Epoch
Figure 1: An example epoch with two training cycles. Note that each cycle is repeated n, = 6 times to allow transients to decay. 4 Training Considerations 4.1 Time Delay Wrap-Around. If there is only one training sequence with period TI,or if all the training sequences have the same period (TI= T2 = ... = TN),Ti, can wrap around the period if the gradient information in ATi,(t) tries to make Tij negative. Then, if Tu does become negative, it can be changed to Tij TO(where TO= TI).
+
4.2 Zero Weights. During the training process, it is quite possible that a weight may have to change sign. However, such a transformation can cause the weight to become zero and stay there for the remainder of the adaptation. This condition can introduce an undesirable local minimum into the error surface by effectively reducing the number of degrees of freedom in the network. The interconnects at risk are those associated with the outputs of the hidden cells. From equations 2.35 and 2.36, the hidden cell errors 6, are proportional to wi,. As wi, + 0, 6, also approaches 0, causing the driving term Awii(t) to become very small. The solution wi, = 0 is the
Training Periodic Sequences
123
trivial solution to the adaptation, since 6, =. Aw,(t) = 0 in this case. A further consequence is that the backpropagation process effectively stops because the cell error is zero. The solution to the problem is to monitor w, and Aw,. When wij is within toof 0 in magnitude and the sign of Awij is such that w, will be adapted toward 0, it is necessary to force wq to "jump" over 0. Assuming to> 0, this can be done using the following algorithm: If 0 5 W, 5 €0 and Awij < 0, set W , = - E O . If
-60
5 Wij 5 0 and AWV> 0, set W ,
= €0.
4.3 Finite Cell Bandwidth. The overall effect of having cells with finite bandwidth is to limit the spectral content of the input/output training sequences. Nonlinear networks have a mechanism for counteracting the attenuation that is unavailable in linear networks. Nonlinear sigmoidal output functions Si can mix the spectral energy in different frequencies via the driving term '7VitYik2 given in equations 2.29-2.33. For frequencies much greater than the cutoff frequency fc = 1/(27rrv), the attenuation is too great for this coupling mechanism to overcome. However, for frequencies around fc, a nonlinear network can learn to compensate for the attenuation via the SBP training algorithm. Since there is no corresponding mechanism in a linear network, it cannot be expected to learn a set of input/output sequences that contain frequencies around or above fc. For reliable learning to occur, the bandwidth of the output sequences must be less than fc, preferably by at least half a decade M 3.16). Another effect of a positive rv is that a phase delay is introduced between the input and output signals of a cell. This delay can be thought of as a form of time delay for propagating signals through the cell. Its influence.in the training process is maintained by the coefficients Alk(t) and AZ(t) defined in equations 2.21 and 2.22, respectively. The amount of the effective delay cannot be controlled because it is determined by rv and the frequency content of the input sequence. However, the interconnect time delays can be used to compensate accordingly. Multiple interconnect time delays must be adapted, though, because the amount of compensation available from any single interconnect is limited. The limit is imposed not by the interconnects but by the spectral content of the input signal-the propagation delay varies between frequencies of the input signal. This situation reveals another benefit of having adaptable time delays in the interconnects.
5 Discrete-Time Considerations
In the continuous-time formulation, the time delays are continuous. However, in a discrete-time simulation of a network, the delays must become
James A. Kottas
124
discrete in some form because of storage limitations. To approximate continuous delays, the function value at a nonintegral delay can be interpolated from neighboring function values at integral delays. For example, suppose Tv = 5.3 time steps and y,[t - Tij] is needed (where t is the current time step). Using linear interpolation,
yj[t - 5.31 x (0.7)yj[t - 51 + (0.3)yj[t - 61
(5.1)
For a general linear interpolation function, Tq can be broken down into an integral part and a fractional part,
+
Tij = LTvJ 6Tij
(5.2)
where LTij) is the largest integer less than or equal to Tij (i.e., the truncation funchon) and 6Tjj is the fractional offset (0 5 6Tij < 1). The form for the general linear interpolator is
The discrete-time equations for the continuous-time cells governed by equations 2.1-2.3 are (5.4) (5.5) (5.6) where a, = ed1Iruand y,[t - Tij] is computed using the above approximation. Note that if all the time delays are 0, each interconnect effectively has a minimum delay of 1 time step. In the remaining discussions, the time delays Tij will refer to the amount of additional time delay in the interconnects. The change in yi(t) over time can be approximated using the backward difference formula,
@ !!
M Ayi[t] = yi[t] -yi[t - 11 (5.7) dt The corresponding Fourier series coefficients for yi[t] are approximated by
(5.9) in which there is an implied At = 1 in the summation. Note that the approximations in these formulas go beyond the integral-to-sum conversion. Depending on TOand K, the sine/cosine basis set of functions
Training Periodic Sequences
125
may not be approximately orthogonal in discrete time because of too few samples in the waveforms. The highest sampling frequency for the sinusoids, KIT0 where TOis the duration of the shortest training sequence, should be larger than the Nyquist rate, 2/To. A factor of 5 is sufficient in practice. The changes in the weights and time delays, governed by equation 2.17, are implemented in discrete time using ~ij[t] = Z q [ t - 11
where
+ (1 - ~$')A~ij[t]
(5.10)
= e-1/7?).
6 Choosing Parameters
There are two parameters that must be chosen a priori: K, the maximum spectral frequency, and n,, the number of cycles per sequence in an epoch. The criteria for choosing K are varied and depend upon the available computation power as well as the duration of the shortest training sequence. The argument for making K large is to capture the details of the shapes for all the training sequences. However, larger values for K require more computation, especially if nonlinear cells are used so the VV*Yik, terms in equations 2.30-2.33 must be calculated. Furthermore, when training with short sequences, K cannot be too large or else the sine/cosine set of basis functions in the Fourier series decomposition ceases to be orthogonal. The net effect is akin to aliasing whereby the spectral coefficients become inaccurate, usually too big. A reasonable guideline is to have at least 10 samples per period of the sine and cosine functions with the highest harmonic. Therefore, if TOis the duration of the shortest training sequence, K should be at most T0/10. The criteria for choosing n, is based on making sure the network is at a steady state (limit cycle or fixed point) in order for the Fourier analysis to be valid. Therefore, the number of transient cycles per sequence in an epoch, n, - 1, must be large enough for all transients to have decayed away. However, as n, increases, the training run time will lengthen because of the increased computational load. The best way to choose n, is with a dynamic algorithm that monitors the network outputs and signals when a steady state has been reached. However, this algorithm could become rather complicated when trying to detect limit cycles. For a fixed value of n, and an arbitrary network topology, the best way to determine n, is empirically. Using reasonable guesses for the weights and time delays, the network can be simulated in the normal runtime mode and its transient response measured. Then, n, can be set to the smallest number of periods that will provide this amount of transient time. In general, it is a good idea to increase this value for n, by 1 or 2 for a safety margin.
James A. Kottas
126
However, the training process still can fail due to insufficient transient time because the exact path of the evolution in weight and time delay space is not known. For example, when training recurrent linear networks, the poles could temporarily migrate close to the imaginary axis, resulting in longer transient times. If n, is not large enough, the adaptation could fail or converge to a false "solution" that incorporates the long transient effects. 7 Sample Simulation Results
The simple recurrent network shown in Figure 2 was used to test the SBP algorithm on an infinite impulse response (IIR) network. Both the weights wI2and wzl and the time delays T12 and 7'21 were allowed to be adaptable. The initial weights were set to 0.5 (with no random perturbations) and the time delays to 0. The input interconnect was fixed with wl0 = 1 and TIO = 0. A single input/output pair of sequences was used for training. The input sequence was the one-dimensional limit cycle defined by x1it] = sin
(T)
(7.1)
where TO = 200 time steps. The desired output sequence was generated by the network by setting rf,= 49.498 time steps (so a, = 0.98), w10 = WZI = 1, w12 = -1, TIO = 0, and T21 = 7'12 = 30 time steps. Both cell output functions were linear with a gain mi = 1. Although using the network to generate the desired output sequence is not indicative of a practical problem, it does ensure that the network has a chance of learning the input/output sequence relationship.
~
Figure 2: A simple linear filter/oscillator with an infinite impulse response.
Training Periodic Sequences
127
A total of five spectral components were computed, so K = 4. Strictly speaking, K could have been 1 since the input sequence contains only one frequency component. The number of cycles per epoch was set to n, = 2, allowing for one transient cycle. To speed convergence, the SuperSAB adaptive gain algorithm (Tollenaere 1990) was employed whereby the common learning gain q in equation 2.17 becomes the set of time-varying gains $)[t]. In this algorithm, the gain for each weight or time delay is increased as long as the error gradient for the particular weight or time delay does not change sign. If it does, the gain is penalized and the gradient driving term is temporarily reset to 0. The resulting evolution of the training error E during adaptation is shown in Figure 3a. The corresponding evolutions of the weights and time delays are shown in Figure 3b and c. The initial and final limit cycles are illustrated in Figure 4. These plots show the successful a d a p tation performed by the SBP algorithm. The oscillations apparent in the evolutions of the total training error and the weights are induced by the gradient resets made by the SuperSAB algorithm when the gradient AT,, [f] changes sign at about t = 270 epochs. Since only one training sequence was used here, the time delays were allowed to wrap around 0 to TO.The delay T12 takes advantage of this ability as illustrated in Figure 3c. If the wrap-around is disabled, the adaptation settles into an unsatisfactory local minimum, thus preventing the training from completing successfully. Other simulations showed that various combinations of the weights and time delays can be adapted. However, if the input interconnect (wl0,Tlo) is allowed to vary along with both sets of weights and time delays, the adaptation path is such that the network develops an eigenvalue that is very close to 1 in magnitude. The resulting long time constant prevents the network from reaching a steady state within the allotted transient time. Coincidentally, the adaptation converges to a solution, but this solution is not the desired one because it cancels out the long transient response. For the correct solution to be obtained, ne would have to be increased significantly (two orders of magnitude in this particular case). 8 Discussion
The same network in Figure 2 was trained on two input/output limit cycles with different fundamental periods. As before, the network was used to generate the output cycles given two input cycles. The SBP successfully trained the network to learn the new set of weights and time delays. In this case, the time delays could not wrap around TOsince TOwas not the same for both cycles. Several other cases have been tested with the SBP algorithm (Kottas 1991). It has been used to train a nonperiodic impulse response h[t]
128
JamesA. Kottas
Figure 3 The evolution of the weights and time delays for the network shown in Figure 2. (a) The total training error E . (b)The weights. (c) The time delays. The negative delays indicate wraparound has occurred. The actual delay is given by T12 + TOwhen T12 < 0. TO= 200 time steps here.
Training Periodic Sequences
129
Figure 4 (a) The initial (before training) and desired (final)output limit cycles. (b) Snapshots of 8 output cycles taken during the training process with 100 epochs between cycles. into a conventional finite-impulse-response (FIR) filter network. Sharp transitions in h [ f ]can only be approximated because of the finite number of Fourier components used in the training. The SBP algorithm also can train the time delays in a FIR filter network. If all weights are held constant at 1, the resulting network realizes a phase-only FIR filter. If the time delays are permitted to wrap around To in this case, the SBP algorithm will not necessarily find the optimal solution, that is, one with a minimum of time delay. However, the time delays can be modified easily by subtracting off unnecessary delays such that the relative delays between the FIR taps are preserved. Finally, the SBP algorithm can train vector patterns and not just sequences into a network. For example, it successfully trained a feedforward network with one hidden layer to learn the exclusive-OR operation between two binary inputs. In this case, only the weights were adaptable and the time delays were set to 0. Furthermore, only the bias component (K = 0) was used for training. The number of cycles per sequence (a binary vector here) per epoch was set to n, = 3 to allow the discrete time simulation to completely propagate an input pattern through the network and produce the corresponding output pattern. In all these cases, the SBP algorithm requires that a steady state solution exists for each input. If the network is inherently unstable such as with a recurrent linear network with eigenvalues greater than one in magnitude, the SBP will not work and thus will not be able to restore stability to the network.
130
James A. Kottas
Adaptable gains (@[ t]) can significantly decrease the convergence time, especially when the weights and time delays select a point near a quadratic well in the error surface. However, the gradient-reset step in the SuperSAB algorithm often induces oscillations in the adaptation. A simple solution is to ignore this part of the SuperSAB algorithm and let the SBP algorithm continue from the current point (as opposed to the previous point) on the error surface. Compared with the conventional recurrent backpropagation algorithm, the SBP algorithm requires more computational resources. However, the added computational complexity is small compared with the ease in generating networks for solving certain problems. Consider the task of recognizing limit cycles. Using the SBP algorithm, it is possible to train very quickly two linear networks, both with no hidden cells, to recognize arbitrarily shaped limit cycles in less than three periods with a high degree of phase sensitivity. This application is discussed more fully in the context of a finite state machine based on limit cycles in Kottas (1991). 9 Conclusions
The spectral backpropagation (SBP) algorithm can train the weights and time delays in a network given a set of periodic input/output sequences. Its adaptation criterion is based on minimizing the spectral energy in a Fourier series decomposition of the output error sequences. A useful property of the SBP algorithm is that it allows the cells in the network to have finite bandwidth. The SBP algorithm has been demonstrated successfully with continuous-time limit cycles and recurrent networks (varying combinations of the weights and time delays), nonperiodic discrete-time sequences and FIR networks (varying the weights and time delays independently), and static vector patterns in feedforward networks (weights only). Since it requires steady-state behavior to exist in the network, the SBP algorithm cannot train inherently unstable networks. The main advantage of having trainable time delays is that it allows the phase characteristics of the network to be controlled more easily than with the conventional recurrent backpropagation algorithm. This feature is desirable when designing networks to process periodic signals. Acknowledgments This work was sponsored in part by DARF'A through Grant AFOSR-860301. I would like to thank Vernon Shrauger, Thomas McNamara, and the reviewer for their useful comments on the manuscript. I also would like to acknowledge several helpful discussions with Cardinal Warde and Michael Jordan.
Training Periodic Sequences
131
References Jordan, M. I. 1989. Supervised learning and systems with excess degrees of freedom. In Proceedings of the 1988 Connectionist Models Summer School, G . Hinton, D. Touretzky, and T. Sejnowski, eds., pp. 62-75. Morgan Kaufmann, San Mateo, CA. Kottas, J. A. 1991. Limit cycles in neural networks for information processing. Ph.D. thesis, MIT. Pearlmutter, B. A. 1989. Learning state space trajectories in recurrent neural networks. Neural Comp. 1,263-269. Pineda, F. 1989. Recurrent backpropagation and the dynamical approach to adaptive neural computation. Neural Comp. 1, 161-172. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing. Vol. 1: Foundations, D. E.Rumelhart, J. L. McClelland, and the PDP Research Group, eds., pp. 318-362. The MIT Press, Cambridge, MA. Tollenaere, T. 1990. SuperSAB Fast adaptive back propagation with good scaling properties. Neural Nehvorks 3,561-573. Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., and Lang, K. 1989.Phoneme recognition using time-delay neural networks. IEEE Trans.ASSP 37,328-339. Yao, Y.,and Freeman, W. J. 1990. Model of biological pattern recognition with spatially chaotic dynamics. Neural Networks 3, 153-170. Received 3 September 1991; accepted 5 May 1992.
This article has been cited by:
Communicated by Halbert White
Generalization and Approximation Capabilities of Multilayer Networks Yoshikane Takahashi NTT Network Information Systems Laboratories, 1-2356 Take Yokosuka-shi,Kanagawa 283-03, Japan
This paper develops a theory for constructing 3-layered networks. The theory allows one to specify a finite discrete set of training data and a network structure (minimum intermediate units, synaptic weights and biases) that generalizes and approximates any given continuous mapping between sets of contours on a plane within any given permissible error.
1 Introduction A real object world usually consists of a continuum each point of which also consists of a continuum, for example, objects such as pictures and hand-written characters. Humans perceive the whole object world continuously both inter-object and intra-object from a certain finite discrete set of perceived data. These human perception capabilities (generalization and approximation) are represented by a continuous mapping between the object worlds. Multilayer networks are expected to simulate these generalization and approximation capabilities in an effective manner. Ji et al. (1990) modified the backpropagation learning algorithm for 3layered networks to generalize continuous curves from the set of integer points. However, they do not further investigate specific construction methods of optimal training data and network structures (minimum intermediate units, synaptic weights and biases) to generate neural mappings that generalize continuous curves, which is left as an open problem (the generalization problem). It has been proven that any continuous mapping from an m-dimensional real space to an n-dimensional real space can be approximated within any given permissible distortion by the 3-layered network with enough intermediate units (Hornik et al. 1989; Funahashi 1989). However, this result assures just the network existence and provides no construction methods for optimal network structures to generate neural mappings Neural Computation 5,132-139 (1993) @ 1993 Massachusetts Institute of Technology
Capabilities of Multilayer Networks
133
that approximate the continuous mapping; this is also left as another open problem (the approximation problem). This paper gives a theoretical solution to both the generalization and the approximation problems. That is, it develops a theory to construct a set of training data and 3-layered network structure that generates a neural mapping which generalizes and approximates any given continuous mapping between sets of contours on a plane within any given permissible distortion. 2 Problem Formulation
2.1 Definitions. 2.1.1 Metric Spaces of Contours. Let x denote a contour on the plane.
where I = [O, 11. Set C ( I )= { x I x : contour}. C(I) is a metric space with the distance d: d ( f , x b ) = max I f ( t ) - xb(t)l for V f , V xb E C(I) tE1
t EI
(2.2)
Ie(t)
for V t E I. where IP(f)- Xb(t)l = Ci - $(t)I Assume that X and Y denote any compact subspaces of C(I). 2.1.2 Continuous Mappings. Define a continuous mapping f from X
onto Y f : X Y such that V ( x ) ] ( t ) = y ( t ) and f is a restriction of a mapping F : C(I) + C(I) that satisfies these double continuity conditions: (2.3) a. For each x in C(I),F ( x ) and F-'(x) are continuous on I. b. F and F-' are continuous on C(I). Examples off include smooth deformations on contours such as rotation and similarity extension. Any one continuous mapping f is fixed and discussed. 2.2.3 Three-layered Networks. Each constituent of the 3-layered network 52 is defined as follows. 1. Inputs and outputs are values of contours x in X and y in Y respec-
tively.
Yoshikane Takahashi
134
2. The input layer has two units. Outputs from unit i are represented by the set { x , ( t ) I t E I} (i = 1,2). 3. The intermediate layer has r units with r in 1 5 r 5 YM where YM is assumed to be a fixed natural number. The mapping function of unit j is defined as: r
1
(2.4) where cp is any bounded, monotone increasing and continuously differentiable function. Aij is a synaptic weight from unit i to unit j with the condition [Aij[ 5 A (A: positive constant). 0, is a bias at unit j with the condition 0 5 j 5 0 (0: positive constant) (1 5 i 5 2, 1 5 j 5 r). All these mapping functions altogether are expressed as z ( t ) = cp[Ax(t) - 191. 4. The output layer has two units. The mapping function of unit k is defined as
(2.5) where c,k is a synaptic weight from unit j to unit k with the condition lcpl 5 C (C: positive constant) (1 5 j 5 r, 1 I k 5 2).
5. Assume that YM and cp are known and fixed while T, A, 19, and c are unknown variables. r = (r, A, 6, c) is called a structure of 52. 2.2 Generalization and Approximation. Let any structure r be given. Then r generates a neural mapping [ defined as [: C(I) + C(I) such that
y(t) = [ [ ( x ) ] ( t )= cp[A x ( t ) - 291 for V x
E C(I),y E
2.2.1 Generalization. Assume any nonnegative real number given. Define XN x YN as a finite discrete subspace of X x Y
XN x YN n
=
(2.6)
C(I) E
1 0 is
{(x",y") I x" E X,y" E Y,y" =f(x"),
= 1,2,..., A!}
(2.7)
Also define A as a partitioning of I: A = {P E O = t O < t' t9"
r 1 4 = 0, I , . . . , K }
< ... < P < ... < p-' < t K = I - P = 1 / K for 9 = O , l , . . . , K - 1
(2.8)
Yoshikane Takahashi
136
3 Solving GAP
GAP is solved in three steps, each of which is described in each section. 3.1 Tkaining Data Construction. As it is assumed that X and Y are compact in C(Z) and f is continuous, the subspace XN x YN of X x Y in 2.7 is constructed as V x E X , 3n : d ( x ,x") < &/40r~Ch M,
and d [ f ( x ) , f ( x " )< ] &/5 (3.1)
As f is continuous, a positive real number 6 > 0 is found such that
d V ( P ) , F ( x b ) ]< &/loif d(x@,xb)< 6 where V P E XN,Vxb E C(Z)
(3.2)
Applying the Weierstrass plynomial approximation theorem to each continuous function x" E X ,a polynomial x p E C(Z) is constructed with M reselected if necessary so that
d(x",XP) < &/40r~CA M <6
(3.3)
Set X p = {x" I xp: A polynomial that satisfies 3.3. x" E XN} c C(Z). Thus, 3.2 and 3.3 lead to
Applying again the Weierstrass polynomial approximation theorem to each continuous function F(xp) E Y, a polynomial Fp(x") E C(Z) is constructed such that
d[F(A?),FP(g)] < &/lo
(3.5)
Thus, 3.4 and 3.5 lead to dV(X"),FP(XP)I< 4 5
(3.6)
As XN x YN is a finite subspace of X x Y,by selecting an appropriately fine partitioning of Z in 2.8, a set of training data expressed in 2.9 is constructed for any x" E XN so that
n
V t E I,3 3 E Xp,IFp($) E C(Z), 3q : IA?(t) - A?(fl)I < &/4oY~Ch M for v t E [t4,fl+']
(3.7)
IIFP(?)](t) - [Fp(A?)](fl)l< &/5 for V t E [ f l , f l + l ]
(3.8)
Capabilities of Multilayer Networks
137
3.2 Structure Construction. This section will construct a structure r0 with minimum units ro to realize within the distortion &/lONK. It is a necessary condition for a structure realizing withixi the distortion &/lONKto have the minimum units ro that the structure generates a mapping from the input layer to the output layer that extracts the minimum information from II with the distortion E/lONK (Bichsel and Seitz 1989). Let denote the data set that extracts the minimum information from with the distortion &/lONK.In general, the rate distortion theorem in the information theory establishes the information extraction method that preserves the minimum information from any information space (typically a source information space) to any other information space (typically a compressed information space) within any given distortion between the two information spaces (e.g., Berger 1971). Thus, is constructed by the aid of the rate distortion theorem, and is represented as follows:
n
n
n' n
n'
n'
= {(xW,w"9) I (x",w")E
xN x YN,x"9 = x"(t9),
w"'? = w"(P),fqE A
n Here
=
1,2,..., N ; q = O , l ,
...)K}
(3.9)
n' satisfies the condition that Iw"(f4) -yn(tq)l 5 &/lo for V y" E YN,VP E A
(3.10)
n'.
Next, let roexactly realize Then, the components A, 6, and c of l?o must satisfy the following equation system: r
1
(3.11)
w;(P)= c c j k z ; ( P ) (k = 1,2)
(3.12)
i
The structure r0 is constructed by solving the equation system 3.11,3.12 with a sequence of three theoretical procedures briefly described as follows.
Procedure I. The equation system 3.12 is solved. Set AI(C) = (6 1 6: A real number such that 161 IC}. Then, for any fixed c E AI(C)~'Mthe system 3.12 comes to a linear equation system consisting of 2 N ( K 1) equations with the 2 N ( K + 1 ) known constants wi(ts) and the r ~ N ( K f 1 ) unknown variables zy(F). This linear equation system is solvable in zy(trl) under appropriate parametric constants c E A,s(C) L AI(C)~'M.Solving it, with arbitrary constants assigned to some zy(ts)'s if necessary, produces the set Z ( c ) = { z ( c ) 1 z(c): A solution of 3.12 under c E A,s(C)}.
+
Procedure 2. The equation system 3.11 is solved (a similar solution procedure is proposed by Shepansky 1988). Set A,(@) = ( 6 I 6: A real number such that 0 I6 5 8).Then using the inverse function cp-' of
Yoshikane Takahashi
138
cp, for any fixed 29 E Az(Q)'M the system 3.11 comes to a linear equation system as follows consisting of rMN(K + 1) equations with the known constants xr(t4) of 2 N ( K l), cp-'[27(t4)] of rMN(K 1) and dj of r M and the 2rMunknown variables Xi,. (3.13) C ~ i j x y ( t s=) cp-'[zy(tq)I dj (j = 172,.. * rM)
+
i
+
+
9
The system 3.13 is solvable in Xij under appropriate parametric constants c E Ais(C) & A,s(C), Z ( C ) E Z(c)' C Z(c), and 19 E A2.40) C Solving it produces the set Ass(A) = {X(c,d) 1 X(c,d): A solution of 3.13 under c E A&(c)/ z(c) E Z(c)', and 19 E A ~ s ( 0 ) n ) A,(A) where A3(A) = { A I A: A real number such that 1x1 5 A}. Procedure 3. The intermediate unit number rM is minimized. Pick out all combinations of X(c,d) E AJS(A),6 E Azs(0) and c E A;,(C) that at the same time include the subvectors (Xlj, Xzj) = (O,O), dj = 0, and (cj1,cjz) = (0,0), respectively for some j . Eliminating such a null intermediate unit j or units j's from each combination [X(c, d ) , 8, c] produces a structure r*= [r, X(c, d)*, d*,c'] that includes no null intermediate unit any longer. Then, select any one structure ro = (ro, X(c, d ) * ,d*, c*] that has the minimum number ro of the intermediate units among all these structures r*. 3.3 Conditions Satisfaction. It is assured that the pair (fl,ro)constructed in Sections 3.1 and 3.2 in fact satisfies the conditions (CON11 and (CON21 in GAP. It is evident from the construction of r0 in Section 3.2 that (CON2) is satisfied. Consider next (CONI). rogenerates the neural mapping [ in 2.6 that satisfies the conditions:
3.3.1 Generalization Condition (2.10). As [ maps x"(t4) to w"(t4) where [xn(t9),w"(t4)]satisfies 3.14, it holds that [ [ ( x " ) ] ( P= ) w"(fl) for n = 1,2,. . . ,N and q = 0,1,. . . ,K (3.15) Using 3.15, the left-hand side of 2.10 is evaluated as I[E(x)l(t)- Y"(P)l 5 1[5(x)l(t)- [t(X")l(fl)l + Ilw"(fl) - Y"(fl)I (3.16) Owing to 2.14,3.1,3.3, and 3.7, the first term of the right-hand side of 3.16 is evaluated as
I[<(x)](t)- [r(x")l(fl)I 5 rMCA Mlx(t) - X " ( f l ) l 5 rMCA M [ d ( x ,x") + d(x",?) + IXP(t) - fl(P)l +d(?,x")] < (1/40)[~+ E + E f E ] = &/I0
(3.17)
Capabilities of Multilayer Networks
139
Thus, 2.10 is obtained from 3.10, 3.16, and 3.17. 3.3.2 Approximation Condition (2.11). The left-hand side of 2.11 is evaluated as
IV(x)l(t) - K(x)l(t)l I dV(x),f(x”)l + dV(x”)7FP(fl)I + I[FP(2)l(t)- [FP(fl>l(fl)1 + d [ F P ( g ) , f ( ~ ”+) ]Iy”(t4)- [t(x)](t)I (3.18) Thus, 2.11 is obtained by applying the evaluations 3.1, 3.6, 3.8, and 2.10 to the right-hand side of 3.18. 4 Conclusion
This paper develops the theory to solve GAP, the generalization and approximation problem of the 3-layered networks in the application of contour perception on the plane. Compared to the works such as Ji et al. (1990), and Hornik et al. (19891, the theory in this paper is much more elaborate and constructive and thus applicable to the real world objects perception. Further, the theory provides more specific and direct insights into the training data and structure to GAP, which elucidates the generalization and approximation capabilities of the 3-layered networks. The theory needs to be extended to cover more complex cases such as general multilayer networks applied to the 3-dimensional objects perception. As for the practical aspect of GAP, the theory in this paper is expected to stimulate the development of effective algorithmic solutions to GAP together with their computational complexity estimation. References Berger, T. 1971. Rate Distortion Theory. Prentice-Hall, Englewood Cliffs, NJ. Bichsel, M., and Seitz, I? 1989. Minimum class entropy: A maximum information approach to layered networks. Neural Networks 2, 133-141. Funahashi, K. 1989. On the approximate realization of continuous mappings by neural networks. Neural Networks 2, 183-192. Hornik, K., Stinchcombe, M., and White, H. 1989. Multilayer feedforward networks are universal approximators. Neural Networks 2, 359-366. Ji, C., Snapp, R., and Psaltis, D. 1990. Generalizingsmoothnessconstraints from discrete samples. Neural Cornp. 2, 188-197. Shepansky, J. 1988. Fast learning in artificial neural systems: Multilayer percep tron training using optimal estimation. In J. R. Johnson, ed., Proceedings of the Aerospace Applications of Art$cial Intelligence Conference, Netrologic, Vol. 1, pp. 85-93. Dayton, OH. Received 25 February 1992; accepted 5 May 1992.
This article has been cited by: 2. G. Wolfe, R. Vemuri. 2003. Extraction and use of neural network models in automated synthesis of operational amplifiers. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 22:2, 198-212. [CrossRef] 3. Kyu-Cheoul Shim, Darrell G. Fontane, John W. Labadie. 2002. Spatial Decision Support System for Integrated River Basin Flood Control. Journal of Water Resources Planning and Management 128:3, 190. [CrossRef] 4. Allan Pinkus. 1999. Approximation theory of the MLP model in neural networks. Acta Numerica 8, 143. [CrossRef] 5. Kuo-lin Hsu, Hoshin Vijai Gupta, Soroosh Sorooshian. 1995. Artificial Neural Network Modeling of the Rainfall-Runoff Process. Water Resources Research 31:10, 2517. [CrossRef]
Communicated by Gerald Tesauro
Statistical Theory of Learning Curves under Entropic Loss Criterion Shun-ichi Amari Noboru Murata Deparfmenf of Mathemafical Engineering and Infomafion Physics, University of Tokyo, Bunkyo-ku, Tokyo 113, Japan
The present paper elucidates a universal property of learning curves, which shows how the generalization error, training error, and the complexity of the underlying stochastic machine are related and how the behavior of a stochastic machine is improved as the number of training examples increases. The error is measured by the entropic loss. It is proved that the generalization error converges to Ho,the entropy of the conditional distribution of the true machine, as HO+ m * / ( 2 t ) , while the training error converges as HO- m'/(2t), where t is the number of examples and m' shows the complexity of the network. When the model is faithful, implying that the true machine is in the model, m* is reduced to m, the number of modifiable parameters. This is a universal law because it holds for any regular machine irrespective of its structure under the maximum likelihood estimator. Similar relations are obtained for the Bayes and Gibbs learning algorithms. These learning curves show the relation among the accuracy of learning, the complexity of a model, and the number of training examples. 1 Introduction
It is an important subject of research of neural networks and machine learning to study general characteristics of learning curves, which represent how fast the behavior of a learning machine is improved by learning from examples. It is also important to evaluate the performance of a trained machine in terms of that for the old training examples. This is given by the relation between the generalization error and the training error, in terms of the complexity of the network. This is an interdisciplinary problem related to neural networks, machine learning, algorithms, statistical inference, etc. There are a number of approaches to learning machines. One is the stochastic descent learning algorithm (see, e.g., Widrow 1966;Amari 1967; Rumelhart, Hinton, and Williams 1986;White 1989). Even in an old paper by Amari (1967) where the stochastic descent method was proposed for Neural Computation 5,140-153 (1993) @ 1993 Massachusetts Institute of Technology
StatisticalTheory of Learning Curves
141
general layered neural networks, the asymptotic dynamic behavior of learning curves was discussed, and the trade-off between the learning speed and the accuracy was studied [see Heskes and Kappen (1991) for recent developments]. Another approach is a computational one (Valiant 1984) in which the learning performance was evaluated stochastically under computational complexity constraints on algorithms. This approach was successfully applied to neural networks (Baum and Haussler 1989). Haussler et al. (1988) studied the convergence rate of general learning curves by relaxing algorithmic constraints. See also Haussler et al. (1991) for recent developments. Here, the VC dimension plays a major role. Yamanishi (1990,1991) extended the framework to noisy or stochastic machines. The third approach is statistical-mechanical. Levin et al. (1990) presented a Bayesian statistical-physical approach to study learning curves, where behaviors of generalization errors, predictive-entropic errors, and stochastic complexity of Rissanen (1986) were discussed. There are also a number of papers using a statistical-mechanical approach to this problem [see, for example, Hansel and Sompolinsky (1990); Gyorgyi and Tishby (1990); S u n g et al. (1991); +per and Haussler (1991)l. The statisticalmechanical approach can give some deep theory for specific simple models such as the simple perceptron, in which the replica method is typically used in the "thermodynamic limit" situation. The present paper uses the fourth approach of statistical inference to elucidate the asymptotic learning behavior of a general stochastic learning dichotomy machine. The predictive entropic loss is used for evaluating the machine performance, where the maximum likelihood estimator of the Bayes and the Gibbs algorithms is used to choose a candidate machine based on training examples. The statistical approach is based on the asymptotic expansion of estimators [see, e.g., Amari (1985) for the higher order asymptotic expansion]. Before comparing the results of the present paper with others, we state the problems treated here and the main results. We consider a stochastic machine or stochastic multilayer neural network parameterized by a vector parameter w,which, when an input x is given, emits a binary ouput y with probability p(y I x, w). Suppose we are given t examples = { (yl, X I ) , . . . , (yt, x,)}, where xi is randomly generated from a fixed but unknown probability distribution p(x) and yi is a corresponding output generated by the true machine that has parameter WO.The maximum likelihood estimator wf is calculated as a candidate machine in the beginning. This machine predicts an output y for given x by the predictive distribution p(y I x,wf). There are two different methods of evaluating the behavior of a machine. One is the average error rate at which the candidate machine predicts an output different from that of the true machine. The other is the average predictive entropy evaluated by the expectation of -logp(y I x,wf)for an input-output pair (x,y), which is zero if the prediction is 100% correct. We use this entropic
<,
142
Shun-ichi Amari and Noboru Murata
loss to evaluate the learning behavior of a machine (see also Yamanishi 1991).
The generalization error is the average entropic loss, or average predictive entropy, of a trained machine for a new example (yr+1,xt+l). It is proved that the average predictive entropy for the generalization error (e(t))eenconverges to the entropy HOof the true machine asymptotically as in the following Theorems, where ( ) denotes the expectation and rn is the number of parameters in w. This is in agreement with Yamanishi’s result (Yamanishi 1991). On the other hand, the training error (e(t))bh is the average entropic loss of the candidate machine for the training examples (yi, xi), i = 1, . . . ,t, which are used to estimate wt. It is proved that the training error also converges as in the Theorems. Theorem. Universal Convergence Theorem.
Since HOis unknown, we can obtain (e(f))genby the relation
This is in good agreement with the AIC approach (Akaike 1974). Instead of using the maximum likelihood estimator wt, we can use the Bayes approach. When the behavior of a trained machine is evaluated by the Bayes posterior distribution (the Bayes algorithm), the learning curves are exactly the same as the previous Theorem. When we choose a candidate machine from the posterior distribution [the Gibbs learning algorithm (Opper and Haussler 1991)1, we obtain the following result. Theorem. Bayesian Convergence Theorem.
for the generalization error, and
for the training error. The above results hold under the assumption that there exists woby which the true machine is specified. However, in many cases there is no wo that specifies the true machine. The model is said to be unfaithful in this case. Let w; be the best approximation to the true machine in
Statistical Theory of Learning Curves
143
the sense of Kullback divergence and let H,' be its entropy By using the maximum likelihood estimator, we prove the following theorem, where
m' = tr(K'-'G') to be defined later plays a role of the effective dimensions. Theorem. Convergence Theoremfor Unfaithful Model.
m* ( e ( t ) ) b h = H* - 2t Now we compare our methods and results with others. The l / t convergence law was first proved by Haussler et al. (1988).However, its coefficients were not exactly known. Their exact values are still unknown even for the simple perceptron in the case of the error rate loss (Haussler et al. 1991). By using the entropic loss, the Theorem proves the universal coefficient of the convergence rate. This is universal in the sense that the theorem holds irrespective of the machine architecture. This implies that the VC dimension seems to be irrelevant for stochastic machines. The statistical-mechanical approach is useful for determining the coefficient for the l / t convergence. However, it uses the replica method which is unjustified. Moreover, it is applicable only to simple models like the simple perceptron and only in the case of the thermodynamic limit, implying that both t and m tends to infinity with a fixed ratio a: = t/m. Our method does not use the statistical-mechanical assumptions such as the replica method, annealed approximation, and the thermodynamic limit. Instead, we use the standard technique of asymptotic statistical infednce, which is valid under the regularity conditions such as the existence of the moments of random variables and the existence of the Fisher information. The statistical technique is not applicable to deterministic machines, because they violate the regularity conditions. Therefore, the present paper complements the result by Amari et al. (1992)where learning curves are obtained for deterministic machines under the annealed approximation. Amari (1992)succeeded in obtaining a similar result for deterministic machines without the annealed approximation. The present results are closely related to the model selection by AIC (Akaike 1974)and its generalization to general nonlinear neural networks (Murata et al. 1991;Moody 1992). The first Theorem can be regarded as a detailed version of the original AIC, while the third Theorem corresponds to its generalization. Moody (1992)proposes a similar generalization of AIC under a more general loss criterion in an unfaithful model. This approach is more general in the sense that it includes a regularization term, but is less general than Murata et al. (1991)in the sense that the latter treats a more general nonlinear model including non-additive noises. It should be pointed out that these papers give essentially the same effective number m* of parameters, although they are different in their expressions.
Shun-ichi Amari and Noboru Murata
144
2 Statistical Theory of Stochastic Machines
Let us consider a machine which receives an n- dimensional input signal x E R" and emits a binary output y = 1 or -1. A machine is stochastic when y is not a function of x but y takes on 1 and -1 subject to a probability p(y I x) specified by x. Let us consider a parametric family of machines where a machine is specified by an m-dimensional parameter w E R" such that the probability of output y, given an input x, is specified by p(y I x, w). A typical form of p(y I x, w) is as follows. A machine first calculates a smooth function f(x,w) and then specifies the probabilities by
where
k ( f )=
1 1+e-A
(2.2)
When f(x, w) > 0, it is more likely that the output of the machine is y = 1, and when f(x, w) < 0, it is more likely that the output is y = -1. The parameter l / p is the so-called "temperature" parameter. When p = 00, the machine is deterministic, emitting y = 1 when f(x, w) > 0 and y = -1 when f(x, w) < 0. Let us consider the case where the true machine that generates examples is specified by WO. More specifically, let p(x) be a nonsingular probability distribution of input signals x, and let x l , . .. ,xt be t randomly and independently chosen input signals subject to p(x). The true machine generates answers yl, . . . ,yt using the probability distribution p(yi I xi,wo), i = I , . . . , t. Let be t pairs of examples thus generated,
. . .,(xt,yt)}
(2.3)
from which we guess the true machine. Let wtbe the maximum likelihood estimator from the observed data It. Since the probability of obtaining & from a machine specified by w is t
I W) = nP(xi)P(Yi I xi, W)
~ ( < t
i=l
by taking the logarithm,
should be maximized by the maximum likelihood estimator wt, where
l(Y I x, w) = logP(Y I x, w)
(2.4)
StatisticalTheory of Learning Curves
145
3 Generalization Error and Training Error in Terms of the Predictive
Distribution Given t examples &, we estimate the true parameter with wt.The behavior of the estimated machine is given by the conditional probability p(y I x, wt).Given the next example xt+] randomly chosen subject to p(x), the next output yt+l is predicted with the probability p(yt+l 1 x t + l , wt). The best prediction in the sense of the minimum expected error is that the predicted output y;+] is 1 when p(1
I xt+r,*t)
> p(-l
I Xt+l,Wt)
and is -1 otherwise. The prediction error is given by ut = 0.51yt+l-y;+]I.
This is a random variable depending on the t training examples tt and Xt+l*
Its expectation (uJgen with respect to tt and xt+l is called the generalization error, because it denotes the average error when the machine trained with t examples predicts the output of a new example. On the other hand, the training error is evaluated by the average of ui(i = 1,. . .,t), which are the errors when the machine wtpredicts the past outputs yi for the training inputs xi retrospectively, using the distribution p(yi I xi, wt), that is
This error never converges to 0 when a machine is stochastic, because even when wtconverges to the true parameter wothe machine cannot be free from stochastic errors. The prediction error can also be measured by the logarithm of the predictive probability for the new input-output pair (yt+l,xt+l),
e(t) = - logp(yt+l I
xt+l, f i t )
(3.1)
This is called the entropic loss, log loss or stochastic complexity (Rissanen 1986; Yamanishi 1991). The generalizationentropic error is its expectation over the randomly generated training examples ttand new input-output pair (xt+~,YI+I 1, (e(t))gen
= -(logp(yt+1 I xt+l,wt))
(3.2)
Since the expectation of - logp(y I x) is the conditional entropy,
the generalization entropic loss is the expectation of the conditional entropy H(Y 1 X;wt) over the estimator wt. The entropic error of the true
Shun-ichi Amari and Noboru Murata
146
machine, specified by WO,is given by the conditional entropy,
Ho = H(Y I XWO)= El- logp(y I X,WO)l
(3.3)
Similarly, the training entropic error is the average of the entropic loss over the past examples (yjl Xi) that are used to obtain wt, (3.4) Obviously, the training error is smaller than the generalization error. It is interesting to know the difference between the two errors. The following theorem gives the universal behaviors of the training and generalization entropic errors in a faithful model, that is, when there is a wo specifying the true machine. Theorem 1. Universal Convergence Theorem for Training and Generalization Errors. Theasymptotic learning curve for the entropic training error is given by
and for the entropic generalization error by
where m is the number of parameters in w. The result of l / t convergence is in good agreement with the results obtained for another model by the statistical-mechanical approach (e.g., Seung et al. 1991). It is possible to compare our result with Yamanishi (19911, where the cumulative log loss,
is used. Here wjis the maximum likelihood estimator based on the i observations ti. From (3.61, we easily have
in agreement with Yamanishi (19911, because of
cI 71
= log t
+ o(l0g t)
i=l
The proof of Theorem 1 uses the standard techniques of asymptotic statistics and is given in the Appendix.
Statistical Theory of Learning Curves
147
4 Learning Curves for Unfaithful Model
It has so far been assumed that there exists wosuch that the true distribution p(y I x) is written as
P(Y I 4
I
= P(Y %WO)
(4.1)
This implies that the model M = {p(y I x,w)} of the distribution parameterized by w is faithful. When the true distribution is not in M, that is, there exisits no wosatisfying (4.11, the model M is said to be unfaithful. We can obtain learning curves in the case of unfaithful models, in a quite similar manner as in the faithful case. Let p(y I x,w:) be the best approximation of the true distribution p(y I x) in the sense that w; minimizes the Kullback-Leibler divergence
where the expectation E is taken with respect to the true distribution
p(x)p(y 1 x). We define the following quantities: H,' = E[- logp(y I x,will G* = E[{Vl(Y I x,w,')l{vr(y I X,Wi)}T1 K* = -E[VVl(y I X, w:)]
(4.2)
(4.3) (4.4)
where V is the gradient operator, V implying the column vector
vl=
(&)
the suffix T denotes the transposition of a vector, and VVl is the Hessian matrix. In the faithful case, w: = wo,H,'= Ho,and G' = K' = G is the Fisher information matrix. However, in general, G' # K* in the unfaithful case.
Theorem 2. Convergence Theorem for Learning Curves : Lhfiithful Case. The asymptotic learning cume for the entropic training error is given by m* (e(t))@&= H,'- (4.5) 2t and for the entropic generalization error by m* (e(t))@&= H,'+ (4.6) 2t where m' = tr(K*-'G') is the trace ofK*-'G'. See the Appendix for the proof. It is easy to see that m* = m in the faithful case, because of K' = G*.The above relations can be used for selecting an adequate model (see Murata et al. 1991; Moody 1992).
148
Shun-ichi Amari and Noboru Murata
5 Bayesian Approach
The Bayesian approach uses a prior distribution 9(w),and then calculates the posterior probability distribution Q(w I 5') based on t observations (training examples). The predictive distribution based on Ct is defined by (5.1) P(Y I x;ttt) = JP(Y I x,w)Q(w I W w One idea is to use this predictive distribution for predicting the output. Another idea is to choose one candidate parameter w; from the posterior distribution Q(w I &) and to use p(y I x; w;) for predicting the output. The former one is called the Bayes algorithm and the latter is called the Gibbs algorithm (Opper and Haussler 1991). The entropic generalization loss is evaluated by the expectation of - log p(y I x; &) for a new example (y, x) in the Bayes algorithm case and the expectation of - log p(y I x; w;) in the Gibbs algorithm case. The entropic training loss is given, correspondingly, by
We first study the case of using the predictive distribution p(y I x; 4). By putting (5.2)
the predictive distribution is written as (5.3) p(yt+1 I X'+1, 5') = Zt+l/Z' [Amari et al. (1992); see also the statistical-mechanical approach, for example, Levin et al. (1990); S u n g et al. (1991); Opper and Haussler 1991)l. Therefore, (5.4)
We can evaluate these quantities by statistical techniques (see the Appendix). Theorem 3. The learning curves for the Bayesian predictive distribution are the same as those for the maximum likelihood estimation. We can perform similar calculations in the case of the Gibbs algorithm. Theorem 4. The learning curves for the Gibbs algorithm is for the training error (e(f1)tIain = Ho and for the generalization error
(5.5)
(5.6)
Statistical Theory of Learning Curves
149
Conclusions We have presented a statistical theory of learning curves. The characteristics of learning curves for stochastic machines can easily be analyzed by the ordinary asymptotic method of statistics. We have shown a universal l / t convergence rule for the faithful and unfaithful statistical models. The difference between the training error and the generalization error is also given in detail. These results are in terms of the entropic loss, which fits very well with the maximum likelihood estimator. The present theory is closely related with the AIC approach (Akaike 1974; Murata et al. 1991; Moody 1992) and the MDL approach (Rissanen 1986). Our statistical method cannot be applied to deterministic machines, because the statistical model is nonregular in this case, where the Fisher information diverges to infinity. However, we can prove
for the entropic loss without using the annealed approximation (Amari 1992). But this does not hold for the expected error ut. Appendix: Mathematical Proofs In order to prove Theorem 1, we use the following fundamental lemma in statistics. Lemma. The maximum likelihood estimator wt based on t observations ttis asymptotically normally distributed with mean woand covariance matrix (tG)-',
C )
W , N N wo,-G-'
(A11
where wois the true parameter and G = (gij) is the Fisher information matrix defined by
where E denotes the expectation with respect to the distribution p(x)p(y I x, wo).
When the probability distribution is of the form (Z.l), the Fisher information matrix can be calculated to be
(see Amari 1991). This shows that G diverges to 00 as the temperature tends to 0, the estimator wtbecoming more and more accurate.
Shun-ichi Amari and Noboru Murata
150
Proof of Theorem 1. In order to calculate ( W g e n = -E[logp(y
I x,Wt)l
we expand
I(y I x, W:)= logp(y I x, Wt) at WO,giving
I(y I x,Wt) = qy I x,wo) + Vl(y I x,wo)(Wt - wo) 1 +i(Wt - wo)TVVl(y I x, wo)(W, - wo) + * * *
(A3)
where VI is the gradient with respect to w, VVI = ($l/aW&j) is the Hessian matrix, and the superscript T denotes the transposition of a column vector. By taking the expectation with respect to the new input-output pair (y,x), we have
because of the identity
-W"I(y
I x,wo)l = E[(Vl)(VOT1
Taking the expectation with respect to Wt, we have
E[Wt - WO] = O(l/t) 1 E[(Wt - WO)(W: - W O ) ~ ]= -G-' -I-O(l/$) t and hence
m E[(Wt - W O ) ~ G ( W - WO)] ~ = - + O(l/P) t
Statistical Theory of Learning Curves
151
and substituting this in (A9),and then summing over i, we have t
C l(yi I X, i=l
fit)
=
C l(yi I xi, wo)
because the maximum likelihood estimator wtsatisfies t
Since the xis are independently generated, by the law of large numbers, we have N
-H0
N
E[VVl(y I X, WO)]= -G
I
- C VVl(yi 1 Xi, wo) t i=l
Since (wt- WO)/& is normally distributed with mean 0 and covariance matrix G-I, (Wf - WO)~G(W' - WO)
can be expressed as a s u m of squares of m independent normal random variables with mean 0 and variance 1, implying that it is subject to the X2-distribution of degree m. Therefore, we have
where xi is a random variable subject to the Xz-distribution of degree m. Since its expectation is m,
This proves Theorem 1. In order to prove Theorem 2, we use the following lemma. Lemma. The maximum likelihood estimator wtunder an unfaithful model is asymptotically normally distributed with mean w; and covariance matrix t-'K*-'GK'-',
Shun-ichi Amari and Noboru Murata
152
We do not give the proof of the lemma, because it is too technical. Refer to Murata et al. (1991).The proof of the theorem is almost parallel to the faithful case, if we replace woby wC; and taking account that K' # G*. The Bayesian case can be proved by using the relations p(w I 6 )
t
N
q(w)tm/21G1'/2exp{--(w 2 - WJTG(w - W t ) }
However, the proof is much more complicated and we omit it. One can complete it by using the asymptotic statistical techniques. Acknowledgments The authors would like to thank Dr. K. Judd for comments on the manuscript. The present research is supported by the Japanese Ministry of Education, Science and Culture under Grant-in-Aid on Special Priority Area of Higher Order Brain Functioning. References Akaike, H. 1974. A new look at the statistical model identification. IEEE Trans. AC-19, 716-723.
Amari, S. 1967. Theory of adaptive pattern classifiers. IEEE Vans. EC-16(3), 299-307.
Amari, S. 1985. Differential-Geometrical Methods in Statistics. Springer Lecture Notes in Statistics, 28, Springer, New York. Amari, S. 1991. Dualistic geometry of the manifold of higher-order neurons. Neural Networks 4, 443445. Amari, S. 1992. Universal property of learning curves. METR92-03, Univ. of Tokyo. Amari, S., Fujita, N., and Shinomoto, S. 1992. Four types of learning curves. Neural Comp. 4(4), 605-618. Baum, E. B., and Haussler, D. 1989. What size net gives valid generalization? Neural Comp. 1,151-160. Gyorgyi, G., and Tishby, N. 1990. Statistical theory of learning a rule. In Neural Networks and Spin Glasses, K. Thuemann and R. Koeberle, eds., pp. 3-36. World Scientific, Singapore. Haussler, D., Kearns, M., and Shapire, R. 1991. Bounds on the sample complexity and the VC dimension. Proc. 4th Ann. Workshopon Computational Learning Theory, pp. 61-73. Morgan Kaufmann, San Mateo, CA. Haussler, D., Littlestone, N., and Warmuth, K. 1988. Predicting (0,l) functions on randomly drawn points. Proc. COLT'BB, pp. 280-295. Morgan Kaufmann, San Mateo, CA.
StatisticalTheory of Learning Curves
153
Hansel, D., and Sompolinsky,H. 1990. Learning from examples in a single-layer neural network. Europhys. Lett. 11, 687-692. Heskes, T. M., and Kappen, B. 1991. Learning processes in neural networks. Phys. Rev.A 440,2718-2726. Levin, E., Tishby, N., and Solla, S. A. 1990. A statistical approach to learning and generalization in layered neural networks. Proc. IEEE 78(10), 1568-1574. Moody, J. E. 1992. The effective number of parameters: An analysis of generalization and regularization in nonlinear systems. In Advances in Neural Information Processing Systems, J . E. Moody, S. J. Hanson, and R. P. Lippmann, eds. Morgan Kaufmann, San Mateo, CA. Murata, N., Yoshizawa, S., and Amari, S. 1991. A criterion for determining the number of parameters in an artificial neural networks model. In Artificial Neural Networks, T. Kohonen, K. Makisara, 0. Simula, and J. Kangas, eds. Elsevier Science Publishers 8. V., North-Holland. Opper, M., and Haussler, D. 1991. Calculation of the learning curve of Bayes optimal classfication algorithm for learning a perceptron with noise. Proc. 4th Ann. Workshopon Computational Learning Theory. Morgan Kaufmann, San Mateo, CA. Rissanen, J. 1986. Stochastic complexity and modeling. Ann. Statist. 14, 10801100.
Rosenblatt, F. 1961. Principles of Neurodynamics. Spartan, New York. Rumelhart, D., Hinton, G. E., and Williams, R. J. 1986. Learning internal representationsby error propagation. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1. Foundations. MIT Press, Cambridge, MA. Seung, S., Sompolinsky, H., and Tishby, N. 1991. Learning from examples in large neural networks. To be published. Valiant, L. G. 1984. A theory of the learnable. Comm. ACM. 27(11), 11341142. White, H. 1989. Learning in artificial neural networks: A statistical perspective. Neural Comp. 1,42!j-464. Widrow, B. 1966. A Statistical Theory of Adaptation. Pergamon Press, Oxford. Yamanishi, K. 1990. A learning criterion for stochastic rules. Proc. 3rd Ann. Workshop on Computational Learning Theory, pp. 67-81. Morgan-Kaufmann, San Mateo, CA. Yamanishi, K. 1991. A loss bound model for on-line stochastic prediction strategies. Proc. 4th Ann. Workshop on Computational Learning Theory. MorganKaufmann, San Mateo, CA. Received 14 November 1991; accepted 18 August 1992.
This article has been cited by: 2. Yu Nishiyama, Sumio Watanabe. 2007. Stochastic complexity of complete bipartite graph-type Boltzmann machines in mean field approximation. Electronics and Communications in Japan (Part III: Fundamental Electronic Science) 90:9, 1-9. [CrossRef] 3. Shun-ichi Amari , Hyeyoung Park , Tomoko Ozeki . 2006. Singularities Affect Dynamics of Learning in NeuromanifoldsSingularities Affect Dynamics of Learning in Neuromanifolds. Neural Computation 18:5, 1007-1065. [Abstract] [PDF] [PDF Plus] 4. Koichiro Nishiue, Sumio Watanabe. 2005. Effects of priors in model selection problem of learning machines with singularities. Electronics and Communications in Japan (Part II: Electronics) 88:2, 47-58. [CrossRef] 5. Kazushi Ikeda. 2004. An Asymptotic Statistical Theory of Polynomial Kernel MethodsAn Asymptotic Statistical Theory of Polynomial Kernel Methods. Neural Computation 16:8, 1705-1719. [Abstract] [PDF] [PDF Plus] 6. Koji Tsuda, Shotaro Akaho, Motoaki Kawanabe, Klaus-Robert Müller. 2004. Asymptotic Properties of the Fisher KernelAsymptotic Properties of the Fisher Kernel. Neural Computation 16:1, 115-137. [Abstract] [PDF] [PDF Plus] 7. Toshiaki Aida. 2001. Reparametrization-covariant theory for on-line learning of probability distributions. Physical Review E 64:5. . [CrossRef] 8. Sumio Watanabe . 2001. Algebraic Analysis for Nonidentifiable Learning MachinesAlgebraic Analysis for Nonidentifiable Learning Machines. Neural Computation 13:4, 899-933. [Abstract] [PDF] [PDF Plus] 9. Didier Herschkowitz, Manfred Opper. 2001. Retarded Learning: Rigorous Results from Statistical Mechanics. Physical Review Letters 86:10, 2174-2177. [CrossRef] 10. Wenxin Jiang, M.A. Tanner. 2000. On the asymptotic normality of hierarchical mixtures-of-experts for generalized linear models. IEEE Transactions on Information Theory 46:3, 1005-1013. [CrossRef] 11. Toshiaki Aida. 1999. Field Theoretical Analysis of On-Line Learning of Probability Distributions. Physical Review Letters 83:17, 3554-3557. [CrossRef] 12. Silvia Scarpetta, Magnus Rattray, David Saad. 1999. Journal of Physics A: Mathematical and General 32:22, 4047-4059. [CrossRef] 13. S. Guarnieri, F. Piazza, A. Uncini. 1999. Multilayer feedforward networks with adaptive spline activation function. IEEE Transactions on Neural Networks 10:3, 672-683. [CrossRef] 14. Terrence L. Fine , Sayandev Mukherjee . 1999. Parameter Convergence and Learning Curves for Neural NetworksParameter Convergence and Learning Curves for Neural Networks. Neural Computation 11:3, 747-769. [Abstract] [PDF] [PDF Plus]
15. Magnus Rattray, David Saad. 1999. Analysis of natural gradient descent for multilayer neural networks. Physical Review E 59:4, 4523-4532. [CrossRef] 16. Didier Herschkowitz, Jean-Pierre Nadal. 1999. Unsupervised and supervised learning: Mutual information between parameters and observations. Physical Review E 59:3, 3344-3360. [CrossRef] 17. A. Uncini, L. Vecci, P. Campolucci, F. Piazza. 1999. Complex-valued neural networks with adaptive spline activation function for digital-radio-links nonlinear equalization. IEEE Transactions on Signal Processing 47:2, 505-514. [CrossRef] 18. Magnus Rattray, David Saad, Shun-ichi Amari. 1998. Natural Gradient Descent for On-Line Learning. Physical Review Letters 81:24, 5461-5464. [CrossRef] 19. Jianfeng Feng. 1998. Journal of Physics A: Mathematical and General 31:17, 4037-4048. [CrossRef] 20. A.J. Zeevi, R. Meir, V. Maiorov. 1998. Error bounds for functional approximation and estimation using mixtures of experts. IEEE Transactions on Information Theory 44:3, 1010-1025. [CrossRef] 21. Shun-ichi Amari . 1998. Natural Gradient Works Efficiently in LearningNatural Gradient Works Efficiently in Learning. Neural Computation 10:2, 251-276. [Abstract] [PDF] [PDF Plus] 22. S. Raudys. 1997. On dimensionality, sample size, and classification error of nonparametric linear classification algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence 19:6, 667-671. [CrossRef] 23. A. Atiya, Chuanyi Ji. 1997. How initial conditions affect generalization performance in large networks. IEEE Transactions on Neural Networks 8:2, 448-451. [CrossRef] 24. Sepp Hochreiter, Jürgen Schmidhuber. 1997. Flat MinimaFlat Minima. Neural Computation 9:1, 1-42. [Abstract] [PDF] [PDF Plus] 25. S. Amari, N. Murata, K.-R. Muller, M. Finke, H.H. Yang. 1997. Asymptotic statistical theory of overtraining and cross-validation. IEEE Transactions on Neural Networks 8:5, 985-996. [CrossRef] 26. Manfred Opper. 1996. On-line versus Off-line Learning from Random Examples: General Results. Physical Review Letters 77:22, 4671-4674. [CrossRef] 27. K.-R. Müller, M. Finke, N. Murata, K. Schulten, S. Amari. 1996. A Numerical Study on Learning Curves in Stochastic Multilayer Feedforward NetworksA Numerical Study on Learning Curves in Stochastic Multilayer Feedforward Networks. Neural Computation 8:5, 1085-1106. [Abstract] [PDF] [PDF Plus] 28. Manfred Opper, David Haussler. 1995. Bounds for Predictive Errors in the Statistical Mechanics of Supervised Learning. Physical Review Letters 75:20, 3772-3775. [CrossRef] 29. Florence d'Alché-Buc, Jean-Pierre Nadal. 1995. Asymptotic performances of a constructive algorithm. Neural Processing Letters 2:2, 1-4. [CrossRef]
30. M. B Gordon, D. R Grempel. 1995. Learning with a Temperature-Dependent Algorithm. Europhysics Letters (EPL) 29:3, 257-262. [CrossRef] 31. Peter Sollich. 1994. Query construction, entropy, and generalization in neural-network models. Physical Review E 49:5, 4637-4651. [CrossRef]
Communicated by Haim Sompolinsky
Learning in the Recurrent Random Neural Network Erol Gelenbe Ecole des Hautes Etudes en Informatique, Universitt Rent Descartes (Paris V),45 rue des Saints-P$res, 75006 Paris, France The capacity to learn from examples is one of the most desirable features of neural network models. We present a learning algorithm for the recurrent random network model (Gelenbe 1989, 1990) using gradient descent of a quadratic error function. The analytical properties of the model lead to a "backpropagation" type algorithm that requires the solution of a system of n linear and n nonlinear equations each time the n-neuron network "learns" a new input-output pair. 1 Introduction
The capability to learn from examples is one of the most desirable features of neural network models. Therefore this issue has been at the center of much research in neural network theory and applications (Ackley et al. 1985; Le Cun 1985; Rumelhart et al. 1986). Learning theory in general is of major interest because of its numerous implications in machine intelligence, as well as its ability to provide a better understanding of the relationship between natural and artificial intelligence. In the area of artificial neural networks, learning has been extensively studied in the context of feedfonvard networks, primarily on the basis of the backpropagation algorithm (Rumelhart et al. 1986). Designing effective learning algorithms for general (i.e., recurrent) networks is a current and legitimate scientific concern in neural network theory. There are numerous examples where recurrent networks constitute a natural approach to problems. Such examples include, in particular, image processing and pattern analysis and recognition (see, for instance, Atalay et al. 19911, where local interactions between picture elements lead to mutual interactions between neighboring neurons, which are naturally represented by recurrent networks. In such cases, it is clear that effective learning algorithms for recurrent networks can enhance the value of neural network methodology. Another area where recurrent networks are indispensable is in combinatorial optimization, and it would be interesting to explore further the relationship between the application Neural Computation 5,154-164 (1993) @ 1993 Massachusetts Institute of Technology
Learning in the Recurrent Random Neural Network
155
of neural networks to control and optimization (Gelenbe and Batty 1992) and network learning. Several authors have considered learning algorithms for recurrent connectionist networks (Almeida 1987; Pineda 1987, 1989; Pearlmutter 1989; Behrens et al. 1991). These are based on neural network dynamics, which exhibit a fixed-point behavior. The work presented in this paper extends this approach to the random network model (Gelenbe 1989, 1990), which has the advantage of possessing well-defined fixed-point equations representing the stationary solution of the stochastic network equations. Applications of the random network model to image texture generation, associative memory, pattern recognition, and combinatorial optimization have been described elsewhere (Atalay et al. 1991; Gelenbe et al. 1991; Mokhtari 1991; Gelenbe and Batty 1992). In this paper we present a “backpropagation”type learning algorithm for the recurrent random network model (Gelenbe 1989, 1990), using gradient descent of a quadratic error function when a set of input-output pairs is presented to the network. Both the excitation and inhibition weights of the random network model must be learned by the algorithm. Thus, it requires the solution of a system of 2n linear and n nonlinear equations each time the n-neuron network ”learns” a new input-output pair. The system of nonlinear equations describes the networks fixedpoint, while the linear equations are obtained from the partial derivatives of these equations with respect to the network weights. To justify the use of the algorithm, we prove (in the Appendix) a general theorem concerning necessary and sufficient conditions for the existence of the stationary or fixed-point solution to the network. This general result completes the work presented in Gelenbe (1990) where only more restrictive sufficient conditions were given. Note that for our network existence implies uniqueness of the solution, due to the fact that the random network model is characterized by Chapman-Kolmogorov equations. Furthermore existence implies stability, since all moments of the state distribution can be explicitly computed from the model’s product-form property. 2
The Random Network Model
In the random network model (RN), n neurons exchange positive and negative impulse signals. Each neuron accumulates signals as they arrive, and fires if its total signal count at a given instant of time is positive. Firing then occurs at random according to an exponential distribution of constant rate, and signals are sent out to other neurons or to the outside of the network. Each neuron i of the network is represented at time t by its input signal potential ki(t), constituted only by positive signals that have accumulated, which have not yet been cancelled by negative signals,
Erol Gelenbe
156
and which have not yet been sent out by the neuron as it fires. Positive signals represent excitation, while negative signals represent inhibition. A negative signal reduces by 2 the potential of the neuron to which it arrives (i.e., it “cancels” an existing signal) or has no effect on the signal potential if it is already zero, while an arriving positive signal adds 2 to the neuron potential. This is a simplified representation of biophysical neural behavior (Kandel and Schwartz 1985). In the RN, signals arrive at a neuron from the outside of the network (exogenous signals) or from other neurons. Each time a neuron fires, a signal leaves it depleting its total input potential. A signal leaving neuron i heads for neuron j with probability p+(i,j) as a positive (or normal) signal, or as a negative signal with probability p-(i, j) or it departs from the network with probability d(i). p ( i , j ) = p+(i,j ) + p-(i, j) is the transition probability of a Markov chain representing the movement of signals between neurons. We have j) + d(i) = 1 for 1 5 i 5 n. External (or exogenous) inputs to each neuron i of the network are provided by stationary Poisson processes of rate A(i), and X(i). A neuron is capable of firing and emitting signals if its potential is strictly positive, and firing times are modeled by iid exponential neuron firing times with rate r(i), at neuron i. In Gelenbe (1989) it was shown that this network has a product form solution. That is, the network‘s stationary probability distribution can be written as the product of the marginal probabilities of the state of each neuron. This does not imply that the neurons have a behavior that is independent of each other. Indeed the probabilities that each neuron is excited are obtained from the coupled nonlinear signal flow equations (2) below, which yield the rate of signal arrival and hence the rate of firing of each neuron in steady state. The RN has a number of interesting features:
cip(i,
1. It represents more closely the manner in which signals are trans-
mitted in a biophysical neural network where they travel as spikes rather than as fixed analog signals. 2. It is computationally efficient.
3. It is easy to simulate, since each neuron is simply represented by a
counter; this may lead to a simple hardware implementation. 4. It represents neuron potential and therefore the level of excitation
as an integer, rather than as a binary variable, which leads to more detailed information on system state; a neuron is interpreted as being in the “firing state” if its potential is positive. Let k(t) = [k,(t), . . . ,kn(t)] be the vector of signal potentials at time t, and k = (kl,.. . ,kn) be a particular value of the vector. p(k) denotes the stationary probability distribution p(k) = limt,, Prob[k(t)= k] if it exists. Since { k ( t ) : t 2 0) is a continuous time Markov chain it satisfies the
Learning in the Recurrent Random Neural Network
157
usual Chapman-Kolmogorov equations; thus in steady state p(k) must satisfy the global balance equations:
+ ~ ( i ) ] l [ >k i 0 ] ]= C [ p ( k ; ) r ( i ) d ( i )+ p(kr)A(i)l[ki> 01
p(k) x [ A ( i ) + [A(i) i
i
+ p(kr)X(i)+ E { p ( k ; - ) r ( i ) p + ( i , j ) l [ k j > 01 i
+ ~ ( k $ + ) r ( i ) ~ - ( i ,+j )p(ki+)r(i)p-(i,j)l[kj ) = O]}] where the vectors used are
k; k17 klk$+
( k i , . . . , k i + l ,...,kfl) ( k l , .. . ,ki - 1,. . . , k f l ) = (kl, . . . ,ki + 1,. . . ,kj - 1,.. . ,k f l ) = (k1,. . . ,ki + 1,. . . ,kj 1,.. . ,kn) = =
+
and 1[X]is the usual characteristic function which takes the value 1 if X is true and 0 otherwise.
Theorem (Gelenbe 1989). Let
qi = X+(i)/[r(i)
+ X-(i)]
(2.1)
where the X+(i), A-(i) for i = 1,.. . ,n satisfy the following system of nonlinear simulataneous equations: X+(i) = x q j r ( j ) p + ( j ,i) i
+ A(i),
X-(i)
= x q j r ( j ) p - ( j , i)
i
+ X(i)
(2.2)
I f a unique nonnegative solution {A+(i), X-(i)} exists to equations 2.1 and 2.2 such that each qi < 1, then n
~ ( k=)
n[l-qiI&
i=l
As a consequence of this result, whenever the qi < 1 can be found, the network is stable in the sense that all moments (marginal or joint) of the neural network state can be found from the above formula, and all moments are finite. For instance, the average potential at a neuron i is simply qi/[l- q i ] . The rate (frequency) of the emission of spikes from neuron i in steady state is then 9ir(i). Furthermore because the underlying model is described by Chapman-Kolmogorov equations, whenever there is a solution, it is necessarily unique and given by the above product form formula. If for some neuron, we have X+(i) > [r(i) X-(i)], we say that the neuron is unstable or saturated. This implies that it is constantly excited in steady state: lim,,,Prob[ki(t) > 01 = 1. Its rate of spike emission
+
Em1 Gelenbe
158
is then r ( i ) : to another neuron j of the network its output appears as a constant source of positive or negative signals of rates r ( i ) p + ( i , j ) and Y(Op-(i,j). For notational convenience let us write w+(j, i ) = r ( i ) p + ( i , j ) 2 0, w-v, i) = r ( i ) p - ( i , j ) 2 0 N ( i ) = Cqjw+o,i)+ A(i), and i
D(i)
r(i)
=
+ C qjw-u, i) + x ( i ) i
Then 1 becomes qi
= N(i)/D(i)
(2.3)
and r ( i ) = Ci(w+(j,i) + w-v, i)]. 2.1 The Role of the Parameters w+U, i ) and w-v, i). The weight parameters w+(j,i) and w-u, i) have a somewhat different effect in the RN model than the weights w(/,i)in the connectionist model. In the RN model, all the w+(j,i) and w-(j,i) are nonnegative since they represent rates at which positive and negative signals are sent out from any neuron i to neuron j . Furthermore, in the RN model, for a given pair ( i , j ) it is possible that both w+(i,j) > 0 and w-(i,j) > 0; in general, it is not possible to transform an RN into an equivalent network in which certain connections are only excitatory, while others are only inhibitory, as would be the case in the usual connectionist model. Therefore, in the RN, for each pair 0, i) it will be necessary to learn both w+(i,j ) and w-(i, j ) . 3 Learning with the Recurrent Random Network Model
We now present an algorithm for choosing the set of network parameters W in order to learn a given set of K input-output pairs ( L , Y) where the set of successive inputs is denoted L = ( ~ 1 ,... ,L K } / and Lk = (Ak, A,) are pairs of positive and negative signal flow rates entering each neuron: Ak = [Ak(l), . . . ,Adn)],
xk
= [Xk(l), - .. ,Ak(n)1
The successive desired outputs are the vectors Y = {yl, . . . ,y ~ } ,where each vector yk = (ylk, . . . ,yn& whose elements yik E [O,I] correspond to the desired values of each neuron. The network approximates the set of desired output vectors in a manner that minimizes a cost function E k : n
Ek = (1/2)Cai(qi - yik)’, i=l
Ui
20
Learning in the Recurrent Random Neural Network
159
Without loss of generality, we treat each of the n neurons of the network as an output neuron; if we wish to remove some neuron j from network output it suffices to set Uj = 0 in the cost function, and to disconsider qj when constructing the output of the network. O u r algorithm lets the network learn both n by n weight matrices Wz = {wk+(i,j)}and W; = {w;(i, j ) } by computing for each input Lk = (Ak, Xk), a new value Wt and W; of the weight matrices, using gradient descent. Clearly, we seek only solutions for which all these weights are positive. Let us denote by the generic term w(u,v) either w(u,v) = w-(u,v), or w(u, v) = w+(u,v). The rule for weight update may be written as n
~ ( u , v=)w - I ( u , ~ ) - vCai(qik - ~ i k ) [ a q r / W ~ , v ) ] k
(3.1)
i=l
where 77 > 0 is some constant, and 1. q& is calculated using the input Lk and w(u,v) = ~ k - ~ ( u , vin) ,equation 3. 2. [aqi/bw(u,v)]kis evaluated at the values 9i wk-1
=
qik and w(u,v) =
(u,v).
To compute [aqi/aW(u,u)]kwe turn to the expression 3, from which we derive the following equation:
a q i / h ( u ,v) =
C a q j / h ( u ,v)[w+(j,i ) - w-(i, i)qil/D(i) i
l[u = i]qi/D(i) l[w(u,v) = w+(u,i)]qu/D(i)- l[w(u,u) = w-(u,i)lquqi/D(i) Let q = (ql,.. . ,qn),and define the n x n matrix .. W={[w+(i,j)- w - ( i , j ) q j ] / D ( i ) } i , 1 = 1 , ...,n We can now write the vector equations: -
+
= aq/&+(u,vw+ 7+(u,v)qu 8q/h-(u, = aq/aw- (u,v) W 7 - (u,v)qu where the elements of the n-vectors -y+(u, v ) = [-$(u, v), . . . ,$ ( u , v)], Y ( u , ~= ) [ n ( ~ , v ). ., ., - ~ ( ~are ,41 ^li+(u,v)= -l/D(i) if u = i,v # i,
aq/&+(u,v)
= =
~,:(u,v) = = =
=
+
+l/D(i) ifu#i,v=i, 0 for all other values of (u,v), -(1 + q i ) / D ( i ) if u = i,v = i, -l/D(i) i f u = i , v # i , -9i/D(i) if u # i,v = i, 0 for all other values of (u,v)
Em1 Gelenbe
160
Notice that
(3.2) where I denotes the n by n identity matrix. Hence the main computational effort in solving 3.2 is simply to obtain [I which can be done in time complexity O(n3), or O(mnz)if an m-step relaxation method is used. Since the solution of 3 is necessary for the learning algorithm, in the Appendix we derive necessary and sufficient conditions for the existence of the q j . We now have the information to specify the complete learning algorithm for the network Initiate the matrices W$ and W; in some appropriate manner. This initiation will be made at random (among nonnegative matrices) if no better information is available; in some cases it may be possible to choose these initial values by using a Hebbian learning rule. Choose a value of in 4.
v-’,
1. For each successive value of k, starting with k = 1 proceed as follows. Set the input values to Lk = (Ak, Xk).
2. Solve the system of nonlinear equations 3 with these values. 3. Solve the system of linear equations 3.2 with the results of (2). 4. Using equation 3.1 and the results of (2) and (31, update the matrices Wt and Wi.Since we seek the “best” matrices (in terms of gradient descent of the quadratic cost function) that satisfy the nonnegutivity constraint, in any step k of the algorithm, if the iteration yields a negative value of a term, we have two alternatives:
a. set the term to zero, and stop the iteration for this term in this step k; in the next step k + 1 we will iterate on this term with the same rule starting from its current null value; b. go back to the previous value of the term and iterate with a smaller value of 77. In our implementation we have used (a). Note that we may either proceed with a complete gradient descent [iterating on Steps (21, (3), and (4) until the change in the cost function or in the new values of the weights is smaller than some predetermined value], or only one iteration can be carried out for all the weights for each successive value of k (new input).
Learning in the Recurrent Random Neural Network
161
Clearly, one may either update the weight matrices separately for each successive value of k (i.e., successive input) as suggested, or sum the updates for all inputs at each iteration of the algorithm. 3.1 Complexity. Several authors have examined the complexity of neural network learning algorithms (Pineda 1989; Baum 1991). One viewpoint (Pineda 1989) is to consider the complexity of each network weight update, while another is to consider the complexity of learning a given family of input-output functions (Baum 1991). In the latter approach, it is known that learning even elementary boolean functions using the backpropagation algorithm is NP complete. In fact, the complexity of our learning algorithm is of the same order as that of the algorithms described in Pineda (1989). Here we merely discuss the complexity of weight update for the algorithm we have presented. Notice that the algorithm requires that for each (u,v) and for each input (successive k) we solve the nonlinear system of equations 3, and the linear system 3.2. Equations 3.2 have to be solved for each (u,v ) . [I, is obtained in time complexity O(n3),or in time complexity O(rnn2)if a relaxation method with rn iterations is adopted as suggested, for instance in Pineda (1989). The remaining computations for 3.2 are trivial. Similarly for 3, which is a nonlinear system of e uations (to be solved once for each step), the complexity will be O(mn ).
w-'
9
Appendix: Existence and Uniqueness of Network Solutions As with most neural network models, the signal flow equations 1 and 2,
which describe the manner in which each neuron receives inhibitory or excitatory signals from other neurons or from the outside world, are nonlinear. These equations are essential to the construction of the learning algorithm described above. Yet only sufficient conditions for the existence (and uniqueness) of their solution had previously been established for feedforward networks, or for so-called hyperstable networks (Gelenbe 1989, 1990). Thus in order to implement the learning algorithm it is useful to have necessary and sufficient conditions for their existence. This is precisely what we do in this appendix. Rewrite 1 and 2 as follows:
+
+ h(i), = C X+(j)p-(j, i)r(i)/[r(i)+ ~ ( i )+]X(i)
X+(i) = cX+(j)p+(j, i)r(i)/[r(i) X-(i)] i
(A.1) i where the qi have disappeared from the equations. The X+(i) and X-(i) represent the total arrival rates of positive and negative signals to neuron i.
X-(i)
Em1 Gelenbe
162
Define the following vectors: A' with elements X+(i) X- with elements X-(i)
A with elements A(i) X with elements X(i)
Let F be the diagonal matrix with elements fi = r(i)/[r(i) + X-(i)] I 1. Then A.l may be written as
+ A,
A' = X'FP'
X'(1-
X- = X'FP-
+X
FP') = A
(A.2)
+X
(A.3)
A- = X'FP-
Proposition 1. Equations A.2 and A.3 have a solution (A', A-). Proof. Since the series CEo(FP+)n is geometrically convergent (Kemeny and Snell 1960, p. 43 ff), we can write A.2 as
n=O
so that A.3 becomes (A.4)
Now define y = X- - A, and call the vector function m
where the dependence of G on y comes from F, which depends on A-. Notice that G is continuous. Therefore by Brouwer's fixed-point theorem,
Learning in the Recurrent Random Neural Network
163
has a fixed-point y*. This fixed point will in turn yield the solution of A.2 and A.3: m
n=O
completing the proof. 0 If this computation yields a fixed-point y* such that for any neuron i such that X+(i) 2 [r(i) X-(i)], the stationary solution for neuron i does not exist; this simply means that in steady state neuron i is constantly excited, and we set qi(y*)= 1. If on the other hand we obtain X+(i) < [r(i) X-(i)], then we set qi(y*) = A+(i)/[r(i)+ X-(i)]. Since p ( k ) is a probability distribution it must s u m to 1, which is the case if qi(y*)< 1, for all i and hence p ( k ) exists. Let us insist on the fact that p ( k ) is indeed unique, and that 9 4 ~ ’ ) < 1 for all i implies stability (in the sense of finiteness of all moments of the state).
+
+
Remark. This reduces the problem of determining existence and uniqueness of the steady-state distributions of a random network to that of computing y’ from A.5, which always exists by Proposition 1, and then of verifying the condition qi(y*)< 1, for each i = 1,.. . ,YZ. Acknowledgments The author acknowledges the support of Pale Algorithmique Rbpartie, C3 CNRS, the French National Program in Distributed Computing, and of a Grant from the Minist6re de la Recherche et de la Technologie(Paris, France).
References Ackley, D. H., Hinton, G. E., and Sejnowski, T. J. 1985. A learning algorithm for Boltzmann machines. Cog. Sci. 9, 147-169. Almeida, L. B. 1987. A learning rule for asynchronous perceptrons with feedback in a combinatorial environment. Proc. IEEE First International Conf. Neural Networks, San Diego, CA, Vol. 11, pp. 609-618. Atalay, V., Gelenbe, E., and Yalabik, N. 1991. Texture generation with the random neural network model. In Artificial Neural Networks, Vol. I, T. Kohonen, ed., pp. 111-117. North-Holland, Amsterdam. Baum, E. B. 1991. Neural net algorithms that learn in polynomial time from examples and queries. Draft Paper May 11 (private communication). Behrens, H., Gawronska, D., Hollatz, J., and Schurmann, B. 1991. Recurrent and feedforward backpropagation: Performance studies. In Artqcial Neural Networks, Vol. 11, T. Kohonen et al., eds., pp. 1511-1514. North-Holland, Amsterdam. Gelenbe, E. 1990. Stability of the random neural network model. Neural Cornp. 2(2), 239-247.
164
Erol Gelenbe
Gelenbe, E. 1989. Random neural networks with negative and positive signals and product form solution. Neural Comp. 1(4),502-510. Gelenbe, E.,and Batty, F. 1992. Minimum cost graph covering with the random network model. ORSA TC on Computer Science Conference, Williamsburg, VA, January. Pergamon Press, Oxford. Gelenbe, E., Stafilopatis, A., and Likas, A. 1991. In Artificial Neural Networks, Vol. I, T. Kohonen, ed., pp. 307-315. North-Holland, Amsterdam. Kandel, E. C., and Schwartz, J. H. 1985. Principles of Neural Science, Elsevier, Amsterdam. Kemeny, J. G., and Snell, J. L. 1965.FiniteMarkov Chins. Van Nostrand, Princeton, NJ. Le Cun, Y. 1985.A learning procedure for asymmetric threshold networks. Proc. Cognitiva 85, 599-604. Mokhtari, M. 1992. Recognition of typed images with the random network model. lnt. 1.Artificial Intelligence Pattern Recognition, in press. Pearlmutter, 8. A. 1989. Learning state space trajectories in recurrent neural networks. Neural Comp. 1(2), 263-269. Pineda, F. J. 1987. Generalization of backpropagation to recurrent and higher order neural networks. In Neural Information Processing Systems, D. Z. Anderson, ed., p. 602. American Institute of Physics. Pineda, F. J. 1989. Recurrent backpropagation and the dynamical approach to adaptive neural computation. Neural Comp. 1(2), 161-172. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing, D. E. Rumelhart, J. L. McClelland, and the PDP Research Group, eds., Vols. I and II, Bradford Books and MIT Press, Cambridge, MA. Received 3 September 1991;accepted 27 May 1992.
This article has been cited by: 2. Zhihao Guo, Shaya Sheikh, Camelia Al-Najjar, Hyun Kim, Behnam Malakooti. 2010. Mobile ad hoc network proactive routing with delay prediction using neural network. Wireless Networks 16:6, 1601-1620. [CrossRef] 3. Samar Samir Mohamed, J. M. Li, M. M. A. Salama, G. H. Freeman, H. R. Tizhoosh, A. Fenster, K. Rizkalla. 2010. An Automated Neural-Fuzzy Approach to Malignant Tumor Localization in 2D Ultrasonic Images of the Prostate. Journal of Digital Imaging . [CrossRef] 4. Jin-ting Wang, Peng Zhang. 2009. A single-server discrete-time retrial G-queue with server breakdowns and repairs. Acta Mathematicae Applicatae Sinica, English Series 25:4, 675-684. [CrossRef] 5. Erol Gelenbe, Peixiang Liu, Boleslaw K. Szymanski, Christopher Morrell. 2009. Cognitive and self-selective routing for sensor networks. Computational Management Science . [CrossRef] 6. Erol Gelenbe, Stelios Timotheou. 2008. Random Neural Networks with Synchronized InteractionsRandom Neural Networks with Synchronized Interactions. Neural Computation 20:9, 2308-2324. [Abstract] [PDF] [PDF Plus] 7. P. G. Harrison, T. Kocak, E. Gelenbe. 2008. Discussant Contributions for the Computer Journal Lecture by Erol Gelenbe. The Computer Journal 51:6, 731-734. [CrossRef] 8. E. Gelenbe, S. Timotheou. 2008. Synchronized Interactions in Spiked Neuronal Networks. The Computer Journal 51:6, 723-730. [CrossRef] 9. Erol Gelenbe. 2007. Steady-state solution of probabilistic gene regulatory networks. Physical Review E 76:3. . [CrossRef] 10. Erol Gelenbe. 2006. Introduction. Computational Management Science 3:3, 175-176. [CrossRef] 11. Pavel Bocharov, Ciro D’Apice, Alexandre Pechinkin. 2006. Product form solution for exponential G-networks with dependent service and completion of service of killed customers. Computational Management Science 3:3, 177-192. [CrossRef] 12. Alper Teke, Volkan Atalay. 2006. Texture Classification and Retrieval Using the Random Neural Network Model. Computational Management Science 3:3, 193-205. [CrossRef] 13. M. A. Karkoub. 2006. Prediction of Hydroforming Characteristics using Random Neural Networks. Journal of Intelligent Manufacturing 17:3, 321-330. [CrossRef] 14. Erol Gelenbe, Peixiang Liu, Jeremy LainLaine. 2006. Genetic Algorithms for Route Discovery. IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) 36:6, 1247-1254. [CrossRef] 15. Ricardo Lent. 2006. Linear QoS Goals of Additive and Concave Metrics in <emphasis emphasistype="italic">Ad Hoc Cognitive Packet Routing.
IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) 36:6, 1255-1260. [CrossRef] 16. T. Kocak, J. Seeber, H. Terzioglu. 2003. Design and implementation of a random neural network routing engine. IEEE Transactions on Neural Networks 14:5, 1128-1143. [CrossRef] 17. S. Mohamed, G. Rubino. 2002. A study of real-time packet video quality using random neural networks. IEEE Transactions on Circuits and Systems for Video Technology 12:12, 1071-1083. [CrossRef] 18. E. Gelenbe, K.F. Hussain. 2002. Learning in the multiple class random neural network. IEEE Transactions on Neural Networks 13:6, 1257-1267. [CrossRef] 19. E. Gelenbe, E. Seref, Z. Xu. 2001. Simulation with learning agents. Proceedings of the IEEE 89:2, 148-157. [CrossRef] 20. C.E. Cramer, E. Gelenbe. 2000. Video quality and traffic QoS in learning-based subsampled and receiver-interpolated video sequences. IEEE Journal on Selected Areas in Communications 18:2, 150-167. [CrossRef] 21. E. Gelenbe, Zhi-Hong Mao, Yan-Da Li. 1999. Function approximation with spiked random networks. IEEE Transactions on Neural Networks 10:1, 3-9. [CrossRef] 22. E. Gelenbe, A. Ghanwani, V. Srinivasan. 1997. Improved neural heuristics for multicast routing. IEEE Journal on Selected Areas in Communications 15:2, 147-155. [CrossRef] 23. E. Gelenbe, Yutao Feng, K.R.R. Krishnan. 1996. Neural network methods for volumetric magnetic resonance imaging of the human brain. Proceedings of the IEEE 84:10, 1488-1496. [CrossRef] 24. C. Cramer, E. Gelenbe, H. Bakircloglu. 1996. Low bit-rate video compression with neural networks and temporal subsampling. Proceedings of the IEEE 84:10, 1529-1543. [CrossRef]
REVIEW
Communicated by Steven Nowlan
Neural Networks and Nonlinear Adaptive Filtering: Unifying Concepts and New Algorithms 0. Nerrand
P. Roussel-Ragot L. Personnaz G. Dreyfus Ecole Supirieure de Physique et de Chimie Industrielles de la Ville de Paris, 10, rue Vauquelin, 75005 Paris, France
S . Marcos Laboratoire des Signaux et S y s t h e s , Ecole Supirieure d’Electricit6, Plateau de Moulon, 91192 Gif sur Yvette, France
The paper proposes a general framework that encompasses the training of neural networks and the adaptation of filters. We show that neural networks can be considered as general nonlinear filters that can be trained adaptively, that is, that can undergo continual training with a possibly infinite number of time-ordered examples. We introduce the canonical form of a neural network. This canonical form permits a unified presentation of network architectures and of gradient-based training algorithms for both feedforward networks (transversal filters) and feedback networks (recursive filters). We show that several algorithms used classically in linear adaptive filtering, and some algorithms suggested by other authors for training neural networks, are special cases in a general classification of training algorithms for feedback networks. 1 Introduction
The recent development of neural networks has made comparisons between “neural” approaches and classical ones an absolute necessity, to assess unambiguously the potential benefits of using neural nets to perform specific tasks. These comparisons can be performed either on the basis of simulations-which are necessarily limited in scope to the systems that are simulated--or on a conceptual basis-ndeavoring to put into perspective the methods and algorithms related to various approaches. The present paper belongs to the second category. It proposes a general framework that encompasses algorithms used for the training of neural networks and algorithms used for the estimation of the parameters of filters. Specifically, we show that neural networks can be used Neural Computation 5,165-199 (1993) @ 1993 Massachusetts Institute of Technology
166
0.Nerrand et al.
adaptively, that is, can undergo continual training with a possibly infinite number of time-ordered examples-in contradistinction to the traditional training of neural networks with a finite number of examples presented in an arbitrary order; therefore, neural networks can be regarded as a class of nonlinear adaptive filters, either transversal or recursive, which are quite general because of the ability of feedforward nets to approximate nonlinear functions. We further show that algorithms that can be used for the adaptive training of feedback neural networks fall into four broad classes; these classes include, as special instances, the methods that have been proposed in the recent past for training neural networks adaptively, as well as algorithms that have been in current use in linear adaptive filtering. Furthermore, this framework allows us to propose a number of new algorithms that may be used for nonlinear adaptive filtering and for nonlinear adaptive control. The first part of the paper is a short presentation of adaptive,filters and neural networks. In the second part, we define the architectures of neural networks for nonlinear filtering, either transversal or recursive; we introduce the concept of canonicalform of a network. The third part is devoted to the adaptive training of neural networks; we first consider transversal filters, whose training is relatively straightforward; we subsequently consider the training of feedback networks for nonlinear recursive adaptive filtering, which is a much richer problem; we introduce undirected, semidirected, and directed algorithms, and put them into the perspective of standard approaches in adaptive filtering (output error and equation error approaches) and adaptive control (parallel and series-parallel approaches), as well as of algorithms suggested earlier for the training of neural networks.
2 Scopes of Adaptive Filters and of Neural Networks 2.1 Adaptive Filters. Adaptive filtering is of central importance in many applications of signal processing, such as the modeling, estimation, and detection of signals. Adaptive filters also play a crucial role in system modeling and control. These applications are related to communications, radar, sonar, biomedical electronics, geophysics, etc. A general discretetime filter defines a relationship between an input time sequence {u(n),u(n - l),. . .} and an output time sequence { y ( n ) , y(n - l),. . .}, u(n) and y ( n ) being either uni- or multidimensional signals. In the following, we consider filters having one input and one output. The generalization to multidimensional signals is straightforward. There are two types of filters: (1) trunsversuljilters (termed finite impulse response or FIR filters in linear filtering) whose outputs are functions of the input signals only; and (2) recursive filters (termed infinite impulse response or IIR filters in linear filtering) whose outputs are func-
Neural Networks and Adaptive Filtering
167
tions both of the input signals and of a delayed version of the output signals. Hence, a transversal filter is defined by y(n) = @[u(n),~ ( -nl ) ,. . . , u(n - M
+ l)]
(1)
where M is the length of the finite memory of the filter, and a recursive filter is defined by y(n) = @ [ u ( n ) , u ( n - l,..., ) u(n-M+l),y(n-l),y(n-2) ,...,y(n-N)](2)
where N is the order of the filter. The ability of a filter to perform the desired task is expressed by a criterion; this criterion may be either quantitative, for example, maximizing the signal-to-noise ratio for spatial filtering (see for instance Applebaum and Chapman 1976), minimizing the bit error rate in data transmission (see for instance Proakis 19831, or qualitative, for example, listening for speech prediction (see for instance Jayant and No11 1984). In practice, the criterion is usually expressed as a weighted sum of squared differences between the output of the filter and the desired output (e.g., LS criterion). An adaptive filter is a system whose parameters are continually updated, without explicit control by the user. The interest in adaptive filters stems from two facts: (1) tailoring a filter of given architecture to perform a specific task requires a priori knowledge of the characteristics of the input signal; since this knowledge may be absent or partial, systems that can learn the characteristics of the signal are desirable; and (2) filtering nonstationary signals necessitates systems that are capable of tracking the variations of the characteristics of the signal. The bulk of adaptive filtering theory is devoted to linear adaptive filters, defined by relations (1)and (21, where @ is a linear function. Linear filters have been extensively studied, and are appropriate for many purposes in signal processing. A family of particularly efficient adaptation algorithms has been specially designed in the case of transversal linear filtering; they are referred to as the recursive least square (RLS) algorithms and their fast (FRLS) versions (Bellanger 1987; Haykin 1991). Linear adaptive filters are widely used for system and signal modeling, due to their simplicity, and due to the fact that in many cases (such as the estimation of gaussian signals) they are optimal. Despite their popularity, they remain inappropriate in many cases, especially for modeling nonlinear systems; investigations along these lines have been performed for adaptive detection (see for instance Picinbono 19881, prediction, and estimation (see for instance McCannon et al. 1982). Unfortunately, when dealing with nonlinear filters, no general adaptation algorithm is available, so that heuristic approaches are used. By contrast, general methods for training neural networks are available; furthermore, neural networks are known to be universal approximants (see for instance Hornik et al. 19891, so that they can be used to approximate any smooth nonlinear function. Since both the adaptation of filters (Haykin 1991; Widrow and Stearns 1985) and the training of
0. Nerrand et al.
168
neural networks involve gradient techniques, we propose to build on this algorithmic similarity a general framework that encompasses neural networks and filters. We do this in such a way as to clanfy how neural networks can be applied to adaptive filtering problems. 2.2 Neural Networks. The reader is assumed to be familiar with the scope and principles of the operation of neural networks; to help clarlfy the relations between neural nets and filters, the present section presents a broad classification of neural network architectures and functions, restricted to networks with supervised training.
2.2.1 Functions of Neural Networks. The functions of neural networks depend on the network architectures and on the nature of the input data: 0
0
Networkarchitectures: neural networks can have either a feedforward structure or a feedback structure; Input data: the succession of input data can be either time-ordered or arbitrarily ordered.
Feedback networks (also termed recurrent networks) have been used as associative memories, which store and retrieve either fixed points or trajectories in state space. The present paper stands in a completely different context we investigate feedback neural networks that are never left to evolve under their own dynamics, but that are continually fed with new input data. In this context, the purpose of using neural networks is not that of storing and retrieving data, but that of capturing the (possibly nonstationary) characteristics of a signal or of a system. Feedforzuard neural networks have been used basically as classifiers for patterns whose sequence of presentation is not significant and carries no information, although the ordering of components within an input vector may be significant. In contrast, the time ordering of the sequence of input data is of fundamental importance for filters: the input vectors can be, for instance, the sequence of values of a sampled signal. At time n, the network is presented with a window of the last M vaIues of the sampled signal {u(n),u(n- l),. . . , u(n - M + l)}, and, at time n + 1, the input is shifted by one time period {u(n + l),u(n),. . . ,u(n - M + 2 ) ) . In this context, feedfonoard networks are used as transversal filters, and feedback networks are used as recursive filters. A very large number of examples of feedforward networks for classification can be found in the literature. Neural network associative memories have also been very widely investigated (Hopfield 1982; Personnaz et al. 1986; Pineda 1987). Feedforward networks have been used for prediction (Lapedes and Farber 1988; Pearlmutter 1989; Weigend et al. 1990). Examples of feedback networks for filtering can be found in Robinson and Fallside (1989), Elman (1990), and Poddar and Unnikrishnan (1991).
Neural Networks and Adaptive Filtering
169
Note that the above classification is not meant to be rigid. For instance, Chen et al. (1990) encode a typical filtering problem (channel equalization)into a classificationproblem. Conversely, Waibel et al. (1989) use a typical transversal filter structure as a classifier.
2.2.2 Nonadaptive and Adaptive Training. At present, in the vast majority of cases, neural networks are not used adaptively: they are first trained with a finite number of training samples, and subsequently used, for example, for classification purposes. Similarly, nonadaptive filters are first trained with a finite number of time-ordered samples, and subsequently used with fixed coefficients. In contrast, adaptive systems are trained continually while being used with an infinite number of samples. The instances of neural networks being trained adaptively are quite few (Williams and Zipser 1989a,b; Williams and Peng 1990; Narendra and Parthasarathy 1990,1991). 3 Structure of Neural Networks for Nonlinear Filtering 3.1 Model of Discrete-TimeNeuron. The behavior of a discrete-time neuron is defined by relation 3:
where
f i is the activation function of neuron i vi is the potential of neuron i zj can be either the output of neuron j or the value of a network
input j
Pi is the set of indices of the afferent neurons and network inputs to neuron i ~ i j is , ~ the weight of the synapse that transfers information from neuron or network input j to neuron i with (discrete) delay T
qij is the maximal delay between neuron j and neuron i. It should be clear that several synapses can transfer information from neuron (or network input) j to neuron if each synapse having its own delay T and its own weight C Y , ~ .Obviously, one must have Cii,o = 0 V i for causality to be preserved. If neuron i is such that i 4 Pi and qi, = 0 V j E Pi, neuron i is said to be static.
0. Nerrand et al.
170
Output at time n
State variables at
k ....I
Feedforward network 1
.....
Unit [,I.*.[iJ I 3 delays
.......
Figure 1: General canonical form of a feedback neural network.
3.2 Structure of Neural Networks for Filtering. The architecture of a network, that is, the topology of the connections and the distribution of delays, may be fully or partially imposed by the problem that must be solved: the problem defines the sequence of input signal values and of desired outputs; in addition, a priori knowledge of the problem may give hints which help designing an efficient architecture [see for instance the design of the feedforward network described in Waibel et al. (1989)l. To clarify the presentation and to make the implementation of the training algorithms easier, the canonical form of the network is especially conve nient. We first introduce the canonical form of feedback networks; the canonical form of feedforward networks will appear as a special case.
3.2.1 The Canonical Form of Feedback Networks. The dynamics of a discretetime feedback network can be described by a finite-difference equation of order N , which can be expressed by a set of N first-order difference equations involving N variables (termed state variables) in addition to the M input variables. Thus,any feedback network can be cast into a canonical
form that consists of a feedforward (static) network whose outputs are the outputs of the neurons that have desired values, and the values of the state variables, whose inputs are the inputs of the network and the values of the state variables, the latter being delayed by one time unit (Fig. 1).
Neural Networks and Adaptive Filtering
171
Note that the choice of the set of state variables is not necessarily unique: therefore, a feedback network may have several canonical forms. The state of the network is the set of values of the state variables. In the following, all vectors will be denoted by uppercase letters. The behavior of a single-input-single-outputnetwork is described by the state equation 4 and output equation 4a:
s ( n + 1) = cp[S(n),W)]
(4)
where U ( n ) is the vector of the M last successive values of the external input u and S ( n ) is the vector of the N state variables (state vector). The output of the network may be a state variable. The transformation of a noncanonical feedback neural network filter to its canonical form requires the determination of M and of N . In the single-input-single-output case, the computation of the maximum number of external inputs E (M I E) is done as follows: construct the network graph whose nodes are the neurons and the input, and whose edges are the connections between neurons, weighted by the values of the delays; find the direct path of maximum weight D from input to output; one has E = D + 1. The determination of the order N of the network from the network graph is less straightforward; it is described in Appendix 1. If the task to be performed does not suggest or impose any structure for the filter, one may use either a multilayer perceptron, or the most general form of feedforward network in the canonical form, that is, a fully connected network; the number of neurons, of state variables and of delayed inputs must be found by trial and error. If we assume that the state variables are delayed values of the output, or if we assume that the state of the system can be reconstructed from values of the input and output, then all state variables have desired values. Such is the case for the NARMAX model (Chen and Billings 1989) and for the systems investigated in Narendra and Parthasarathy (1990).Figure 2 illustrates the most general form of the canonical form of a network having a single output y(n) and N state variables {y(n - l),. . .,y(n -N)}. It features M external inputs, N feedback inputs, and one output; it can implement a fairly large class of functions a; the nonrecursive part of the network (which implements function CP) is a fully connected feedforward net. More specific architectures are described in the literature, implementing various classes of functions cp and Q. Some examples of such architectures are presented in Appendix 2. 3.2.2 Special Case: The Canonical Form of Feedforward Networks. Similarly, any feedforward network with delays, with input signal u, can be cast into the form of a feedforward network of static neurons, whose inputs are the successive values u(n),u(n - l ) ,. . . ,u(n - M + 1);this puts
0. Nerrand et al.
172
. . . .....
......
Figure 2 Canonical form of a network with a fully connected feedforward net, whose state variables are delayed values of the output.
the network under the form of a transversal filter obeying relation 1:
y(n) = @[u(n),u(n - l),. . . ,u(n - M
+ l)]= @[U(n)]
The transformation of a noncanonical feedforward neural network filter to its canonical form requires the determination of the maximum value M, which is done as explained above in the case of feedback networks. An example described in Appendix 1 shows that this transformation may introduce the replication of some weights, known as “shared weights.”
4 Training Neural Networks for Adaptive Filtering 4.1 Criterion. The task to be performed by a neural network used as a filter is defined by a (possibly infinite) sequence of inputs u and of corresponding desired outputs d. At each sampling time n, an error e(n) is defined as the difference between the desired output d(n) and the actual output of the network y(n): e(n) = d(n) - y(n). For instance, in process identification, d(n) is the output of the process to be modeled; in a predictor, d(n) is the input signal at time n + 1. The training algorithms aim at finding the network coefficientsso as to satisfy a given quality criterion. For example, in the case of nonadaptive
Neural Networks and Adaptive Filtering
173
training (as defined in Section 2.2.2), the most popular criterion is the least squares (LS)criterion; the cost function to be minimized is
Thus, the coefficients minimizing J ( C ) are first computed with a finite number K of samples; the network is subsequently used with these fixed coefficients. In the context of adaptive training, taking into account all the errors since the beginning of the optimization does not make sense; thus, one can implement a forgetting mechanism. In the present paper, we use a rectangular "sliding window'' of length N,;hence the following cost function: 1 I(n,C) = 2
" C p=n-Nc+l
e(pI2
The choice of the length N, of the window is task-dependent, and is related to the typical time scale of the nonstationarity of the signal to be processed. In the following, the notation ](n) will be used instead of J ( n ,C). The computation of e(p) will be discussed in Sections 4.3 and 4.4.2. 4.2 Adaptive Training Algorithms. Adaptive algorithms compute, in real time, coefficient modifications based on past information. In the present paper, we consider only gradient-based algorithms, which require the estimation of the gradient of the cost function, V J ( n )and , possibly the estimation of I(n);these computations make use of data available at time n. In the simplest and most popular formulation, a single modification of the vector of coefficients A C ( n ) = C ( n ) - C(n - 1) is computed between time n and time n + 1; such a method, usual in adaptive filtering, is termed a purely recursive algorithm. The modification of the coefficients is often performed by the steepestdescent method, whereby A C ( n ) = - p V J ( n ) . To improve upon the steepest-descent method, quasi-Newton methods can be used (Press et al. 1986), whereby A C ( n ) = + p D ( n ) , where D ( n ) is a vector obtained by a linear transformation of the gradient. Purely recursive algorithms were introduced in order to avoid timeconsuming computations between the reception of two successive samples of the input signal. If the application under investigation does not have stringent time requirements, then other possibilities can be considered. For instance, if it is desired to get closer to the minimum of the cost function, several iterations of the gradient algorithm can be performed between time n and time n + 1. In that case, the coefficient-modification
0. Nerrand et al.
174
vector AC(n) is computed iteratively as AC(n) = CK,(n) - Co(n), where K, is the number of iterations at time n, with
Ck(n) = Ck-l(n) -k pk&-l(n)
(k = 1 to Kn)
where Dk-l(n) is obtained from the coefficients computed at iteration k - 1, and Co(n) = CKnvI(n- 1). If K, > 1, the tracking capabilities of the system in the nonstationary case, or the speed of convergence to a minimum in the stationary case, may be improved with respect to the purely recursive algorithm. The applicability of this method depends specifically on the ratio of the typical time scale of the nonstationarity to the sampling period. As a final variant, it may be possible to update the coefficients with a period T > 1 if the time scale of the nonstationarity is large with respect to the sampling period:
cO(n) = CK,,-,(n - T ) Whichever algorithm is chosen, the central problem is the estimation of the gradient, VJ(n):
At present, two techniques are available for this computation: the forward computation of the gradient and the popular backpropagation of the gradient. 1. The forward computation of the gradient is based on the following
relation:
The partial derivatives of the output at time n with respect to the coefficients appearing on the right-hand side are computed recursively in the forward direction, from the partial derivatives of the inputs to the partial derivatives of the outputs of the network. 2. In contrast, backpropagation uses a chain derivation rule to com-
pute the gradient of J(n). The required partial derivatives of the cost function J(n) with respect to the potentials are computed in the backward direction, from the output to the inputs. The advantages and disadvantages of these two techniques will be discussed in Sections 4.3 and 4.4.2.
Neural Networks and Adaptive Filtering
175
In the following, we show how to compute the coefficient modifications for feedforward and feedback neural networks, and we put into perspective the training algorithms developed recently for neural networks and the algorithms used classically in adaptive filtering. 4.3 Training Feedforward Neural Networks for Nonlinear 'Ransversal Adaptive Filtering. We consider purely recursive algorithms (i.e., T = 1 and K,,= 1). The extension to non-purely recursive algorithms is straightforward. As shown in Section 3.2.2, any discrete-time feedforward neural network can be cast into a canonical form in which all neurons are static. The output of such a network is computed from the M past values of the input, and the output at time n does not depend on the values of the output at previous times. Therefore, the cost function
is a sum of N, independent terms. Its gradient can be computed, from the N, + M + 1 past input data and the N, corresponding desired outputs, as a sum of N, independent terms: therefore, the modification of the coefficients, at time n, is the sum of N, elementary modifications computed from N, independent, identical elementary blocks [each of them with coefficients C ( n - l)],between time n and time n 1. We introduce the following notation, which will be used both for feedforward and for feedback networks: the blocks are numbered by m; all values computed from block m of the training network will be denoted with superscript m. For instance, y"(n) is the output value of the network computed by the mth block at time n: it is the value that the output of the filter would have taken on, at time n - N, m, if the vector of coefficients of the network at that time had been equal to C(n - 1). With this notation, the cost function taken into account for the modification of the coefficients at time n becomes
+
+
+
where em(n)= d(n - N, m ) - y"(n) is the error for block m computed at time n. As mentioned in Section 4.2, two techniques are available for computing the gradient of the cost function: the forward computation technique (used classically in adaptive filtering) and the backpropagation technique (used classically for neural networks) (Rumelhart et al. 1986). Thus, each
0. Nerrand et al.
176
block, from block m = 1 to block rn = N,, computes a partial modification AcF of the coefficients and the total modification, at time n, is NC
Ac,( n ) =
Acy (n) m=l
as illustrated in Figure 3. It was mentioned above that either the forward computation method or the backpropagation method can be used for the estimation of the gradient of the cost function. Both techniques lead to exactly the same numerical results; it has been shown (Pineda 1989) that backpropagation is less computationally expensive than forward computation. Therefore, for the training of feedforward networks operating as nonlinear transversal filters, backpropagation is the preferred technique for gradient estimation. However, as we shall see in the following, this is not always the case for the training of feedback networks. 4.4 Training Feedback Neural Networks for Nonlinear Recursive Adaptive Filtering. This section is devoted to the adaptive training of feedback networks operating as recursive filters. This problem is definitely richer, and more difficult, than the training of feedforward networks for adaptive transversal filtering. We present a wide variety of algorithms, and elucidate their relationships to adaptation algorithms used in linear adaptive filtering and to neural network training algorithms.
4.4.1 General Presentation ofthe Algorithmsfor Training Feedback Networks. Since the state variables and the output of the network at time n depend on the values of the state variables of the network at time n - 1, the computation of the gradient of the cost function requires the computation of partial derivatives from time n = 0 up to the present time n. This is clearly not practical, since (1) the amount of computation would grow without bound, and (2) in the case of nonstationary signals, taking into account the whole past history does not make sense. Therefore, the estimation of the gradient of the cost function is performed by truncating the computations to a fixed number of sampling periods N f into the past. Thus, one has to use N f computational blocks (defined below), numbered from rn = 1 to rn = N f : the outputs y"(n) are computed through Nt identical versions of the feedforward part of the canonical form of the network [each of them with coefficients C(n - 1)l. Clearly, N f must be larger than or equal to N, to compute the N, last errors P(n).Here again, we first consider the case where T = 1 and K, = 1. Figure 4 shows the mth computational block for the forward computation technique: the state input vector is denoted by Sg(n); the state output vector is denoted by Stut(n).The canonical feedforward (FF)net computes the output from the external inputs P ( n )and the state inputs
Figure 3 Computation of two successive coefficient modifications for a nonlinear transversal filter (N,= 3).
At time n
178
0. Nerrand et al.
Figure 4 Training block m at time n with a desired output value: computation of a partial coefficient modification using the forward computation of the gradient for a feedback neural network. If the output of block m has no desired value, it has no "products" part and does not contribute directlv to coefficient modifications: it just transmits the state variables and their derivatives to the next block.
Sg(n). The forward computation (FC) net computes the partial derivatives required for the coefficient modification, and the partial derivatives of the state vector which may be used by the next block. The N f blocks compute sequentially the Nt outputs {y"} and the partial derivatives {@"'/acij}, in the forward direction (rn = 1 to Nt). The N, errors { P } (computed from the outputs of the last Ncblocks) and the corresponding partial derivatives are used for the computation of the coefficient modifications, which is the sum of N, terms:
Details of the computations are to be found in Appendix 3. For the blocks to be able to perform the above computations, the values of the state inputs SE(n) and of their partial derivatives with respect to the weights must be determined. The choice of these values is of central importance; it gives rise to four families of algorithms.
Neural Networks and Adaptive Filtering
179
4.4.2 Choice of the State Inputs and of Their Partial Derivatives. Choice of thestateinputs: The most "natural" choice of the state inputs of block m is to take the values of the state variables computed by block m-1: S z ( n ) = St;'(n) with Sk(n) = S,!,,,(n - 1). Thus, the trajectory of the network in state space, computed at time n, is independent of the trajectory of the process: the input of block m is not directly related to the actual values of the state variables of the process to be modeled by the network, hence the name undirected algorithm. If the coefficients are mismatched, this choice may lead to large e m r s and to instabilities. Figure 5a shows pictorially the desired trajectory of the state of the network and the trajectory which is computed at time n when an undirected algorithm is used (Nt = 3, N, = 2). We show in the next section that in that case, one must use the forward computation technique to compute the coefficient modifications (Fig. 5b). This choice of the state inputs has been known as the output error approach in adaptive filtering and as the parallel approach in automatic control. It does not require that all state variables have desired values. In order to reduce the risks of instabilities, an alternative approach may be used, called a semidirected algorithm. In this approach, the state of the network is constrained to be identical to the desired state for m = 1:
Sk(n) = [d(n- N t ) l d ( n- Nt - l)l.. . , d ( n - Nt - N + l)] and Sz(n) = S?;'(n). This is possible only when the chosen model is such that desired values are available for all state variables; this is the case for the NARMAX model. Figure 6a shows pictorially the desired trajectory of the state of the network and the trajectory that is computed at time n when a semidirected algorithm is used (Nt = 4, N, = 2). We show in the next section that in that case, one can use the backpropagation technique to compute the coefficient modifications (Fig. 6b). The trajectory of the state of the network can be further constrained by choosing the state inputs ofall blocks to be equal to their desired values:
SE(n) = [d(n - Nt + m - l ) , d ( n - Nt + m - 2), . . . , d ( n - Nt + m
-N)]
for m = 1 to Nt. With this choice, the training is under control of the desired values, hence of the process to be modeled, at each step of the computations necessary for the adaptation (hence the name directed algorithm); therefore, it can be expected that the influence of the mismatch of the model to the process is less severe than in the previous cases. Figure 7a shows pictorially the desired trajectory of the state of the network and the trajectory that is computed at time n when a directed algorithm is used (Nt = N, = 3). We show in the next section that in that case, one can use the backpropagation technique to compute the coefficient modifications (Fig. %). In directed algorithms, all blocks are independent, just as in the case of the training of feedforward networks (Section 4.3); therefore, one has Nt = N,.
180
0. Nerrand et al.
Figure 5: Undirected algorithm (with Nt = 3 and Nc = 2). (a) Pictorial representation of the desired trajectory, and of the trajectory computed at time n, in state space; the trajectory at time n is computed by the blocks shown in b. (b)Computational system at time n. The detail of each block is shown in Figure 4. Note that the output of block 1 has no desired value.
This choice of the values of the state inputs has been known as the equation error approach in adaptive filtering and as the series-parallel approach in automatic control. It is an extension of the teacher forcing technique (Jordan 1985) used for neural network training. If some state inputs do not have desired values, hybrid versions of
Neural Networks and Adaptive Filtering
181
Figure 6: Semidirected algorithm (with Nr = 4 and Nc = 2). (a) Pictorial representation of the desired trajectory, and of the trajectory computed at time n, in state space; the trajectory at time n is computed by the blocks shown in b. (b) Computational system at time n. The detail of each block is shown on Figure 8. Note that the outputs of blocks 1 and 2 have no desired values, but do contribute an additive term to the coefficient modifications. the above algorithms can be used: those state inputs for which no desired values are available are taken equal to the corresponding computed state variables (as in a n undirected algorithm), whereas the other state inputs may be taken equal to their desired values (as in a directed or in a semidirected algorithm). Consistent choices of the partial derivatives of the state inputs: The choices of the state inputs lead to corresponding choices for the initialization
0. Nerrand et al.
182
b-l A d 1 (n)
Ad2(n)
Ad3(n)
(b)
Figure 7 Directed algorithm (with Nf= Nc = 3). (a) Pictorial representation of the desired trajectory, and of the trajectory computed at time n, in state space; the trajectory at time n is computed by the blocks shown in b. (b) Computational system at time n. The detail of each block is shown on Figure 8. Note that in a directed algorithm, each block is independent from the others and must have a desired output value.
of the partial derivatives, as illustrated in Figures 5a, 6a, and 7a. In the case of the undirected algorithm, one has S$(n) = SrG1(n);therefore, a consistent choice of the values of the partial derivatives of the state inputs consists in taking the values of the partial derivatives of the state outputs computed by the previous block:
Neural Networks and Adaptive Filtering
183
except for the first block where one has as!m ( n ) - W"An - 1) ac, dCij In the case of the semidirected algorithm, the state input values of the first block are taken equal to the corresponding desired values; the latter do not depend on the coefficients; therefore, their partial derivatives can consistently be taken equal to zero. The values of the partial derivatives of the state inputs of the other blocks are taken equal to the values of the partial derivatives of the state outputs computed by the previous block. In the case of the directed algorithm, one can consistently take the partial derivatives of the state inputs of all blocks equal to zero. The parameters T, K,, N f , Nc being fixed, the first three algorithms described above are summarized on the first line of each section of Table 1. The first part of the acronyms refers to the choice of the state inputs and the second part refers to the choice of the partial derivatives of the state inputs. They include algorithms which have been used previously by other authors: the real-time recurrent learning algorithm (Williams and Zipser 1989a) is an undirected algorithm (using the forward computation technique) with N f = N, = 1, This algorithm is known as the recursive prediction error algorithm, or IIR-LMS algorithm, in linear adaptive filtering (Widrow and Steams 1985). The teacher-forced real-time recurrent learning algorithm (Williams and Zipser 1989a) is a hybrid algorithm with Nt = Nc = 1. The above algorithms have been introduced in the framework of the fomard computation of the gradient of the cost function. However, the estimation of the gradient of the cost function by backpropagation is attractive with respect to computation time, as mentioned in Section 4.3.4. If this technique is used, the computation is performed with N f blocks, where each coefficient cij is replicated in each block m as cr. Therefore, one has
The training block m is shown in Figure 8: after computing the N, errors using the N f blocks in the forward direction, the N f blocks compute the derivatives of ](n)with respect to the potentials { v y } , in the backward direction. The modification of the coefficients is computed from the N f blocks as
It is important to notice that backpropagation assumes implicitly that the partial derivatives of the state inputs of the first copy are taken equal to zero. Therefore, the backpropagation technique will lead to the same coefficient modifications as the forward propagation technique if and only if it is used
Desired values
Desired values
Desired values
Desired values
Desired values
Desired values
Semidhcted (SD) algorithm
SDD algorithm
SDUD algorithm
Directed algorithm (D) (equationerror) (teacher forcing) (series parallel)
DSD algorithm
DUD algorithm
D e s i i values
Desired values
Desired values
S:;'(n)
S%%)
s:;'(n)
as:,,
acij
-(n as:,,
- 1)
Zero
zero
- 1)
zero
zero
zero
-(n acij
=
- 1)
zero
acij
-as:,, (n
OC,
-(n)
Initialization: partial derivatives for the first block
'In each section, the first line describes the algorithms with consistent choices of the state inputs.
SLAn - 1) s:ilw
SZ1(n)
1)
S:u,(n -
s:;W
S:dn - 1)
SE(n) =
sk(n) =
UDSD algorithm
Undirected (UD) algorithm (output error) (parallel) U D D algorithm
State input of a current block
Initialization: state input of the first blodc
Table 1: Three Families of Algorithms for the Training of Feedback Neural Networks'
as%;' -@acij I
zero
- (acij n)
=
as:;:;' acij ( n ) zero as:;:;'
- (acij n)
as:;'
zero
as:;' -@Iacij
-as2 (n) acij
Partial derivatives for a current blodc
Neural Networks and Adaptive Filtering
185
Figure 8: Training block m at time n with a desired output value: computation of a partial coefficient modification using the backpropagation technique for the estimation of the gradient for a feedback neural network. If block m has no desired value, then P = 0, but it does contribute an additive term to the coefficient modification. It should be noticed that forward propagation through all blocks must be performed before backpropagation.
within algorithms complying with this condition, that is, within directed or semidirected algorithms (Figs. 6b and 7b);backpropagationcannot be used consistently within undirected and hybrid algorithms. When both backpropagation and forward computation techniques can be used, backpropagation is the best choice because of its lower computational complexity. An example of the use of a directed algorithm for identification and control of nonlinear processes can be found in Narendra and Parthasarathy (1990). Other choices of the partial derivatives of the state inputs: Because adaptive neural networks require real-time operation, tradeoffs between consistency and computation time may be necessary: setting partial derivatives i3S;/i3cy equal to zero may save time by making the computation by backpropagation possible even for undirected algorithms (UD-Dor UD-SDalgorithms). The full variety of algorithms is shown on Table 1: in each group, the first line shows the characteristics of the fully con-
186
0. Nerrand et al.
sistent algorithm, whereas the other two lines show other possibilities which are not fully consistent, but which can nevertheless be used with advantage. The SD-UD, D-SD,and D-UD algorithms have been included for completeness: computation time permitting, the accuracy of the computation may be improved by setting the partial derivatives of the state inputs to nonzero values in the directed or semidirected case. Undirected algorithms have been in use in linear adaptive filtering: the extended L M S algorithm is a UD-D algorithm (see Table 1) with Nf = N, = 1 (Shynk 1989); the u posteriori error algorithm is also a UD-D algorithm with Nt = 2, N, = l (Shynk 1989). The truncated buckpropagution through time algorithm (Williams and Peng 1990) is a UD-D algorithm with N, = 1 and Nt > 1, with a special feature: to save computation time, the coefficients of the blocks 1 to Nf - 1 are the coefficients that were computed at the corresponding times. 5 Conclusion
The present paper provides a comprehensive framework for the adaptive training of neural networks, viewed as nonlinear filters, either transversal or recursive. We have introduced the concept of canonical form of a neural network, which provides a undying view of network architectures and allows a general description of training methods based on gradient estimation. We have shown that backpropagation is always advantageous for training feedforward networks adaptively, but that it is not necessarily the best method for training feedback networks. In the latter case, four families of training algorithms have been proposed; some of these algorithms have been in use in classical linear adaptive filtering or adaptive control, whereas others are original. The unifying concepts thus introduced are helpful in bridging the gap between neural networks and adaptive filters. Furthermore, they raise a number of challenging problems, both for basic and for applied research. From a fundamental point of view, general approaches to the convergence and stability of these algorithms are still lacking; a preliminary study along these lines has been presented (Dreyfus et ul. 1992); from the point of view of applications, the real-time operation of nonlinear adaptive systems requires specific silicon implementations, thereby raising the questions of the speed and accuracy required for the computations.
Appendix 1 We consider a discrete-time neural network with any arbitrary structure, and its associated network graph as defined in Section 3.2. The set of state variables is the minimal set of variables that must be initialized to allow the computation of the state of all neurons at any
Neural Networks and Adaptive Filtering
187
time n > 0, given the values of the external inputs at all times from 0 to n. The order of the network is the number of state variables. Clearly, the only neurons whose state must be initialized are the neurons which are within loops (i.e., within cycles in the network graph). Therefore, to determine the order of the network, the network graph should be pruned by suppressing all external inputs and all edges which are not within cycles (this may result in a disconnected graph). To determine the order, it is convenient to further simplify the network graph as follows: (1) merge parallel edges into a single edge whose delay is the maximum delay of the parallel edges; (2) if two edges of a loop are separated by a neuron that belongs to this loop only, suppress the neuron and merge the edges into a single edge whose delay is the sum of the delays of the edges. We now consider the neurons which are still represented by nodes in the simplified network graph. We denote by N the order of the network. If, for each node i of the simplified graph, we denote by Ai the delay of the synapse, afferent to neuron i, which has the largest delay (i.e., the weight of the edge directed toward i, which has the largest weight), then a simple upper bound for N is given by i
The state Xi of a neuron i, which has an afferent synapse of delay Aj, cannot be computed at times n < Ai; the computation of the states of the other neurons may require the values of xi at times 0,1,. . . ,Ai - 1; thus, the contribution of neuron i to the order of the network is smaller than or equal to Aj. Let the quantity wi be defined as wi = Ai - min(Aj - 7 j . j ) jER,
if Ai - min(A, - 7j.i) > 0 jERi
wi = 0 otherwise where Ri stands for the set of indices of the nodes that are linked to i by an edge directed from i to j (i.e., the set of neurons to which neuron i projects efferent synapses). Then the order of the network is given by
N=Cwj i
The necessity of imposing the state of neuron i at time k (0 < k < Ai - 1) depends on whether this value is necessary for the computation of the state of a neuron j to which neuron i sends its state: if k 7j.i is smaller than the maximum delay A, of the synapses afferent to j, it is not necessary to transmit the state of neuron i at time k to neuron j , since the latter does not have the information required to compute its state at time k + qi; the information on the state of neuron i at time k is necessary only
+
0. Nerrand et al.
188
if one has k 2 Aj - 7j.j. Therefore, the minimum number of successive values required for neuron i is equal to
Ai - min(Aj - 7j.J FRi
if Ai - @(A, 1ERi
- 7j.i)
> 0,
zero otherwise
Clearly, this result is in accord with the upper bound given above. The above results determine the number of state variables related to each neuron. The choice of the set of state variables is not unique. The presence of parallel edges within a loop, or the presence of feedforward connections between loops, may require the replication of some neurons and of some coefficients. Figure Al.la shows a feedback network and Figure Al.lb shows its canonical form; the order of the network is 6. The example shows that some weights are replicated.
Appendix 2 This appendix describes several architectures of feedback neural networks that have been proposed in the literature. We present their canonical form, so that they can be easily compared. The discrete-time mathematical model of a time-invariant dynamical process is of the form
where vector U is the input of the dynamical system, vector S denotes the state of the system, and vector Y is the output of the system. Since neural networks with hidden neurons are able to approximate a large class of nonlinear functions, they can be used for implementing functions cp and Q. The network proposed by Jordan (1986) is trained to produce a given sequence y(n) for a given constant input P ("plan"). Thus it is used as an associative memory. The network and its canonical form are shown in Figure A2.1. The representation of the network under its canonical form shows that the network is of order 2, although the representation used by Jordan exhibits four connections with unit delays. Note that the state variables are not delayed values of the output. The presence of hidden neurons allows this network to learn any function y(n) = Q[S(n),U(n)]. The network suggested by Elman (1988) is used as a nonlinear filter. Its canonical form is shown on Figure A2.2.
Figure Al.1: (a) Example of a feedback neural network. Numbers in rectangles are synapse delay values, u is the external input, and y is the output of the network. (b) Canonical form of the network ( E = 8, M = 2, N = 6). The cij,. notation of relation 3 is wed.
0. Nerrand et al.
190
Figure A2.1: (a) Network architecture proposed by Jordan. (b)Canonical form.
4
4
Figure A2.2 Canonical form of the network architectureproposed by Elman. Each state variable is computed as a fixed nonlinear function f of a weighted sum of the external inputs and state inputs. Therefore, the class of functions cp that can be implemented is restricted to the form:
v")W)l , = f[AS(fl) + W n ) l where A and B are the synaptic matrices. SimiIarly, the output is computed as a fixed nonlinear function f of a weighted sum of the state
Neural Networks and Adaptive Filtering
191
variables, so that the class of functions @ that can be implemented is restricted to
@[S(n),W)l = f[CS(n)l where C is the synaptic matrix. The network proposed in Williams and Zipser (1989a) and Williams and Peng (1990) is used as a nonlinear filter. The state of the network at time n 1 is computed as a weighted sum of the inputs and of the state values at time n, followed by a fixed nonlinearityf i . As a result, the network can only implement nonlinear functions of the form fi[AS(n)+ BU(n)l. The network used by Poddar and Unnikrishnan (1991) consists of a “feedforward network of pairs of neurons; each neuron, except the output neuron, and each external input is associated to a “memory neuron.” If xi(n) is the value of the output of neuron i and x,(n) the value of the output of the associated memory neuron j at time n, the output of the memory neuron at time n + 1 is
+
xj(n + 1) = aixi(n) + ( 1 - ai)xj(n),
0 < ai I 1
If ai = 0, the memory neurons introduce only delays, so that the network is a nonlinear transversal filter. If ai # 0, the memory neurons are linear low-pass first order filters, and the network is actually a feedback network. A state output is associated to each memory neuron. Figure A2.3a shows an example of such an architecture where neurons 3,4,7, and 8 are the memory neurons associated to the two inputs 1 and 2 and to the two neurons 5 and 6, respectively. The canonical form is shown in Figure A2.3b where x3, x4, x7, x8 are chosen as state variables. For process identification and control problems, the most general structure used by Narendra and Parthasarathy (1991) is a model of the specific form y(n) = @ I [ u ( n - l ) , u ( n - 2 ) ,
...I+
@p[y(n-l),y(n-2),
...I
where 91 and \Tr2 are implemented by MLP networks with 20 neurons in the first hidden layer and 10 neurons in the second hidden layer. Appendix 3
For simplicity, we present the training of the fully connected neural net of Figure 2 we denote the external inputs by z1 to ZM! the feedback inputs by Z M + ~ to Z M + N , and the outputs of the neurons by ZM+N+~ to ZM+N+“ (where v is the number of neurons). The neurons are ordered in the following way: the pth neuron receives the outputs of neurons indexed 4 < p (fully connected).
IE
F 4-
I
Neural Networks and Adaptive Filtering
193
At time n, we have to consider the following cost function:
J(n)= I
5
(ey
m=N,-N,+I
where Nt is the number of blocks used to compute the N, values e"' (Nt 2 Nc). In this appendix, we present the contribution of block m (1 5 m 5 Nf) to the gradient estimation. This contribution is computed from the external input vector, the desired value and the state input vector. We denote the available values of the coefficients at time n by { q } . The canonical FF net of the mth block, with coefficients { c y } = {cij}, computes the outputs zy = fi(v7) of all neurons and the state output vector Stut(n) from the external input vector
If"(#)
+
n Nt m ) , u ( n- Nf = [ ~ (= [z?, zz", . . . ,Z G ]
+ m - 'I), . . . ,u(n - Nt + m - M + l ) ]
and the state input vector S g ( n ) = [~;t;+~, Z Z + ~. ., . ,z;t;+N]
as follows:
1. For i = 1 to M (external inputs): zy
= u(n
2. For i = M
- Nt + m - i + 1)
+ 1 to M + N (state inputs):
zy is given by the chosen algorithm (Table 1)
3. For i = M
+ N + 1to M + N + v - 1 (hidden neurons):
4. For i = M
+ N + v (linear output neuron):
Thus, the state output vector is szut(n)
[ZG+N+vr Z G + N + v + l ,
*.
m ZM+N+v+N-ll
.?
= [Y"? z k + l , .. . I z ~ + N - l l
and, ifN,-N,+l 5 m 5 Nt, weobtain from the desired valued(n-Nt+m) and the output ym: em = d ( n - Nf + m ) - ym.
0.Nerrand et al.
194
In the following, we present two methods for the computation of the gradient of J(n): the forward computation and the backpropagation techniques. 1. Forward Computation (Fig.4): We consider the whole set of Nfblocks as a static network on which we perform the forward computation technique. It is based on the following relation:
The linear FC net of the mth block computes, with coefficients {q}and cf:(q)}l the set of partial derivatives of the state output (including y") with respect to all coefficients Cjj: aSk:,/acjj(n). For the v(M + N) + (v - l)v/2 coefficients cq (i > 17: 1. For p = 1 to M (external inputs):
2. For p = M
+ 1 to M + N (feedback inputs):
az; - is given by the chosen algorithm (Table 1 ) acij
3. F o r p = M + N + l t o M + N + v - l (hiddenneurons):
4. For p = M
+ N + v (linear output neuFon):
Thus the partial derivatives of the state output are given by:
Neural Networks and Adaptive Filtering
195
Once all partial derivatives of the output values y" are computed for the Nt blocks, the gradient of J ( n ) is obtained from
If the steepest-descent method is used, the coefficient modifications are given by Aci,(n) = -p- a'(n)
e"-w
=p m=Nt-N,+l
- C
-
Nt
Ac!(n)
m=Nt-N,+I
2. Backpropgufwn (Figure 8). Considering the effect of the coefficient ci, only, one has
thus
Then the gradient of J ( n )can be written as
where
This means that standard backpropagation can be applied to the whole set of NIblocks considered as a static network with replicated coefficients. The linear BP net of the mth block computes, with coefficients {cT} and { fi(q)},the set of partial derivatives of J(n) with respect to the potentials of all neurons. We define the following set of variables 97: 1. f o r i = M + N + v + N - 1 d o w n t o M + N + v + l : if m = Nt then qT = 0, otherwise L$" = 9L&+l
0. Nerrand et al.
196
2. for i = M
+ N + v (linear output neuron): +
if m = Nt then qy = e"', otherwise qy = em ; : ;4 note that qy = -3. for i = M
+ N + v - 1 down to M + N + 1 (hidden neurons):
where Ri is the set of indices of the neurons to which the ith neuron transmits its output
4. for i = M
+ N (last feedback input):
qr =
c c;qr
hERi
5. for i = M
q?
+ N - 1 down to M + 1 (other feedback inputs):
=
c chmi@
f qZN+v
hERi
Note that computation by backpropagation assumes implicitly that the derivatives of the feedback inputs of the first block (m = 1)with respect to the coefficients are equal to zero; this is in contrast to the forward computation of the gradient, where these values can be initialized arbitrarily. Note also that with the forward computation technique, the number of partial derivatives to compute for each block is v[vM (v - l)v/2] whereas with the backpropagation method this number is v. Once all partial derivatives of ](n) with respect to the potentials vy of all neurons are computed for the Nt blocks, the gradient of I(n) is obtained from
+
If the steepest-descent method is used, the coefficient modifications are given by
Neural Networks and Adaptive Filtering
197
Acknowledgments
The authors are very grateful to 0. Macchi for numerous discussions that have been very helpful in putting neural networks into the perspective of adaptive filtering. C. Vignat has been instrumental in formalizing some computational aspects of this work. We thank H. Gutowitz for his critical reading of the manuscript. This work was supported in part by EEC Contract ST2JO312C.
References Applebaum, S. P.,and Chapman, D. J. 1976. Adaptive arrays with main beam constraints. IEEE Trans. Antennas and Propagation, AP-24, 650-662. Bellanger, M. G. 1987. Adaptive Digital Filters and Signal Analysis: Marcel Dekker, New York. Chen, S., and Billings S. A. 1989. Representations of non-linear systems: the NARMAX model. Int. 1.Control 49,1013-1032. Chen, S., Gibson, G. J., Cowan, C. E N., and Grant, P. M. 1990. Adaptive equalization of finite nonlinear channels using multilayer perceptrons. Signal Process. 20, 107-119. D r e y h , G., Macchi, 0, Marcos, S., Personnaz, L., Roussel-Ragot, P., Urbani, D., and Vignat, C. 1992. Adaptive training of feedback neural networks for non-linear filtering and control. In: Neural Networks for Signal Processing 11, S. Y. Kung, F. Fallside, J. Aa. Sorenson, C. A. Kamm, eds., pp. 550-559. IEEE. Elman, J. L. 1990. Finding structure in time. Cog. Sci. 14, 179-211. Fallside, F. 1990. Analysis of linear predictive data as speech and of ARMA processes by a class of single-layer connectionistmodels. In Neurocomputing: Algorithms, Architectures and Applications, F. Fogelman-Soulie and J. Herault, eds., pp. 265-283, Springer. Haykin, S. 1991. Adaptive Filter Theory: Prentice-Hall International Editions, Englewood Cliffs,NJ. Hopfield, J. J. 1982. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. U.S.A. 79, 2554-2558. Hornik, K., Stinchcombe, M., and White, H. 1989. Multilayer feedforward networks are universal approximators. Neural Networks 2,359-366. Jayant, N. S., and Noll, P. 1984. Digital Coding of Waveforms. Principles and Applications to Speech and Video. Signal Processing Series, A. Oppenheim, Ed. Prentice-Hall, Englewood Cliffs, NJ. Jordan, M. I. 1985. The Learning of Representations for Sequential Performace. Doctoral Dissertation, University of California, San Diego. Jordan, M. I. 1989. Serial order: A parallel, distributed processing approach. Proc. Eighth Annu. Conf. Cog. Sci. SOC., 531-546. Lapedes, A., and Farber, R. 1988. How neural nets work. In Neural Information
198
0.Nerrand et al.
Processing Systems, D. Z. Anderson, ed., pp. 442-456, American Institute of Physics. McCannon, T. E., Gallagher, N. C., Minoo-Hamedani, D., and Wise, G. L. 1982. On the design of nonlinear discrete-time predictors. IEEE Trans. Inform. Theory 28,366-371. Narendra, K. S., and Parthasarathy, K. 1990. Identification and control of dynamical systems using neural networks. IEEE Trans. Neural Networks 1,4-27. Narendra, K.S.,and Parthasarathy,K. 1991. Gradient methods for the optimization of dynamical systems containing neural networks. IEEE Trans. Neural Networks 2, 252-262. Nicolau, E.,and Zaharia, D. 1989. Adaptive arrays. In Studies in Electrical and Electronic Engineering 35. Elsevier, Amsterdam. Pearlmutter, B. 1989. Learning state space trajectories in recurrent neural networks. Neural Comp. 1, 263-269. Personnaz, L., Guyon, I., and Dreyfus, G. 1986. Collective computational prop erties of neural networks: New learning mechanisms. Phys. Rev. A 34,42174228. Picinbono, B. 1988. Adaptive methods in temporal processing. In U n d e m t e r Acoustic Data Processing, Y. T. Chan, ed., pp. 313-327. Kluwer Academic Publishers, Dordrecht. Pineda, F. 1987. Generalization of backpropagation to recurrent neural networks. Phys. Rev. Lett. 59,2229-2232. Pineda, F. J. 1989. Recurrent backpropagation and the dynamical approach to adaptive neural computation. Neural Comp. 1,161-172. Poddar, P., and Unnikrishnan, K. P. 1991. Non-linear prediction of speech signals using memory neuron networks. In Neural Networks for Signal Processing, Proceedings of the 2991 IEEE Workshop, B. H. Juang, s. Y. Kung, and C. A. Kamm, eds., pp. 395-404. Press, W. H., Flannery, B. P., Teukolsky, S. A., and Vetterling, W. T. 1986. Numerical Recipes. Cambridge University Press, Cambridge. Proakis, J. G. 1983. Digital Communications. McGraw-Hill, New York. Robinson, A. J., and Fallside, F. 1989. A dynamic connectionist model for phoneme recognition. In Neural Networks from Models to Applications, L. Personnaz and G. Dreyfus, eds., pp. 541-550. IDSET, Paris. Rumelhart, D., Hinton, G., and Williams, R. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing, D. Rumelhart and J. McClelland, eds. MIT Press, Cambridge. Shynk, J. J. 1989. Adaptive IIR filtering. IEEE ASSP Mag. April, 4-21. Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., and Lang, K.1989. Phoneme recognition using time-delay neural networks. IEEE Trans. Acoustics, Speech, Signal Process. 37, 328-339. Weigand, A. S., Huberman, B. A., and Rumelhart, D. E. 1990. Predicting the future: A connectionist approach. Int. 1.Neural Syst. 1,193-209. Widrow, B., and Steams, S. D. 1985. Adaptiw Signal Processing. Prentice-Hall, Englewood Cliffs, NJ. Williams, R. J., and Zipser, D. 1989a. A learning algorithm for continually
Neural Networks and Adaptive Filtering
199
running fully recurrent neural networks. Neural Comp. 1,270-280. Williams, R. J., and Zipser, D. 1989b. Experimental analysis of the real-time recurrent learning algorithm. Connect. Sci. 1, 87-111. Williams, R. J., and Peng, J. 1990. An efficient gradient-based algorithm for on-line training of recurrent network trajectories. Neural Comp. 2,490-501. Received 30 January 1992; accepted 21 August 1992.
This article has been cited by: 1. L. Saad Saoud, A. Khellaf. 2010. Nonlinear dynamic systems identification based on dynamic wavelet neural units. Neural Computing and Applications 19:7, 997-1002. [CrossRef] 2. A. Yilmaz, K. E. Akdogan, M. Gurun. 2009. Regional TEC mapping using neural networks. Radio Science 44:3. . [CrossRef] 3. Chi Sing Leung, Ah Chung Tsoi. 2006. Combined learning and pruning for recurrent radial basis function networks based on recursive least square algorithms. Neural Computing and Applications 15:1, 62-78. [CrossRef] 4. Su Lee Goh , Danilo. P. Mandic . 2004. A Complex-Valued RTRL Algorithm for Recurrent Neural NetworksA Complex-Valued RTRL Algorithm for Recurrent Neural Networks. Neural Computation 16:12, 2699-2713. [Abstract] [PDF] [PDF Plus] 5. Shih-Lin Hung, C. S. Huang, C. M. Wen, Y. C. Hsu. 2003. Nonparametric Identification of a Building Structure from Experimental Data Using Wavelet Neural Network. Computer-Aided Civil and Infrastructure Engineering 18:5, 356-368. [CrossRef] 6. Danilo P. Mandic . 2002. Data-Reusing Recurrent Neural Adaptive FiltersData-Reusing Recurrent Neural Adaptive Filters. Neural Computation 14:11, 2693-2707. [Abstract] [PDF] [PDF Plus] 7. P.A. Mastorocostas, J.B. Theocharis. 2002. A recurrent fuzzy-neural model for dynamic system identification. IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) 32:2, 176-190. [CrossRef] 8. Stefan C. Kremer . 2001. Spatiotemporal Connectionist Networks: A Taxonomy and ReviewSpatiotemporal Connectionist Networks: A Taxonomy and Review. Neural Computation 13:2, 249-306. [Abstract] [PDF] [PDF Plus] 9. Paolo Campolucci , Aurelio Uncini , Francesco Piazza . 2000. A Signal-Flow-Graph Approach to On-line Gradient CalculationA Signal-Flow-Graph Approach to On-line Gradient Calculation. Neural Computation 12:8, 1901-1927. [Abstract] [PDF] [PDF Plus] 10. L.S.H. Ngia, J. Sjoberg. 2000. Efficient training of neural nets for nonlinear adaptive filtering using a recursive Levenberg-Marquardt algorithm. IEEE Transactions on Signal Processing 48:7, 1915-1927. [CrossRef] 11. I. Rivals, L. Personnaz. 2000. Nonlinear internal model control using neural networks: application to processes with delay and design issues. IEEE Transactions on Neural Networks 11:1, 80-90. [CrossRef] 12. P. Campolucci, A. Uncini, F. Piazza, B.D. Rao. 1999. On-line learning algorithms for locally recurrent neural networks. IEEE Transactions on Neural Networks 10:2, 253-271. [CrossRef]
13. D. P. Mandic, J. A. Chambers. 1999. A posteriori error learning in nonlinear adaptive filters. IEE Proceedings - Vision, Image, and Signal Processing 146:6, 293. [CrossRef] 14. Gérard Dreyfus , Yizhak Idan . 1998. The Canonical Form of Nonlinear Discrete-Time ModelsThe Canonical Form of Nonlinear Discrete-Time Models. Neural Computation 10:1, 133-164. [Abstract] [PDF] [PDF Plus] 15. G.C. Mouzouris, J.M. Mendel. 1997. Dynamic non-Singleton fuzzy logic systems for nonlinear modeling. IEEE Transactions on Fuzzy Systems 5:2, 199-208. [CrossRef] 16. I. Kamwa, R. Grondin, V.K. Sood, C. Gagnon, Van Thich Nguyen, J. Mereb. 1996. Recurrent neural networks for phasor detection and adaptive identification in power system control and protection. IEEE Transactions on Instrumentation and Measurement 45:2, 657-664. [CrossRef] 17. Eric A. Wan , Françoise Beaufays . 1996. Diagrammatic Derivation of Gradient Algorithms for Neural NetworksDiagrammatic Derivation of Gradient Algorithms for Neural Networks. Neural Computation 8:1, 182-201. [Abstract] [PDF] [PDF Plus] 18. Tsungnan Lin, B.G. Horne, P. Tino, C.L. Giles. 1996. Learning long-term dependencies in NARX recurrent neural networks. IEEE Transactions on Neural Networks 7:6, 1329-1338. [CrossRef] 19. C.W. Omlin, C.L. Giles. 1996. Rule revision with recurrent neural networks. IEEE Transactions on Knowledge and Data Engineering 8:1, 183-188. [CrossRef]
NOTE
Communicated by William W. Lytton
Fast Calculation of Synaptic Conductances Rajagopal Srinivasan Department of Electrical Engineering, Case Western Reserve University, Cleveland, OH 44206 USA Hillel J. Chiel, Departments of Biology and Neuroscience, Case Western Reserve University, Cleveland, OH 44106 USA Synaptic conductances are often modeled as sums of cr functions
where t is the current time, ti is the time of the ith spike in the presynaptic neuron, and T is the time constant of the synapse. If the time of decay of the synapse, TD, is not equal to its time of onset, TO, the conductance at time t after k spikes have occurred is
The drawback of these solutions is that one must keep track of the times of occurrences of each spike that initiated the synaptic potentials, and recalculate each exponential in the summation at each time step. This creates a large storage and computational overhead. Since both these equations represent the impulse response of a second-order differential equation, another approach is to numerically integrate additional differential equations for each synapse in the network (Wilson and Bower 1989). We have developed an improved method for computing synaptic conductances that separates equations 1 and 2 into two components: one that is a function of the current time of the simulation and one that accumulates the contributions of previous spike events to the synaptic conductance. We demonstrate that this method requires only the storage of two running sums and the time constants for each synapse, and that it is mathematically equivalent to equations 1 and 2. We will then demonstrate that it is also faster for a given level of precision than numerically integrating differential equations for each synapse. We will first describe our algorithm for equation 1, and then for equation 2. Neurul Computation 5,200-204 (1993) @ 1993 MassachusettsInstitute of Technology
Fast Calculation of Synaptic Conductances
201
Equation 1 can be rewritten as follows:
+
When the k 1st spike occurs at time tk+l, single terms can be added to each of the two summations in brackets to update them, eliminating the need to store spike times. To keep the exponentials inside the summation and outside the brackets from growing too large or small (respectively) as t increases over time, the exponentials can be rescaled. The left-hand term can be rewritten as
(4)
We will refer to the terms within the summation as Suml(tk). It can be updated once the k + 1st spike occurs as follows:
sum1( f k + l ) = e - ( f k + l - t k ) / T because after the k
Sum1(fk)
+I
(5)
+ 1st spike occurs, k
1
The right-hand term in equation 3 can also be rewritten as
I
i=l
+
Once the k 1st spike occurs, the new form of the terms within the summation [which we refer to as sud(tk+l)]would be
c
c k
k+l
tie(ti++l)/T
= e-(b+l-k)/T
tie-(tk-'g)/T
+ tkfl
(8)
i= 1
i=l
so that Sud(tk) is updated using the following rule: sud(tk+l) = e-(fk+l-tk)/T Sud(tk) + t k + l
(9)
Thus, at time t > f k + l , from equations 3-9, the synaptic conductance is equal to e-(t-fk+l)/r
r
[t(Suml(tk+l))- (sud(tk+l))]
(10)
Rajagopal Srinivasan and Hillel J. Chiel
202
The conductance needs to be evaluated at each time step of the simulation, that is, from time t to time t + At. This can be accomplished by which yields the term multiplying the term at time t, e-(r-tk+l)/+, by cAt/+, Thus, the conductances can be updated as at time t At, e-[('+A')-tk+ll/T. follows:
+
Suml(t + At) = e-*'/' Suml(t) + S Sum2(t + At) = e-At/r Sum2(t)+ St where S = 1 if a spike occurred at time t and S = 0 otherwise. The synaptic conductance can then be calculated at time t At from
+
1 g = -[(t 7
+ At) Suml(t + At) - Sum2(t + At)]
(13)
Equations 11, 12, and 13 summarize our algorithm for updating equation l, which requires storage of only Suml, Surd, and T . Since these equations are mathematically identical to equation 1, the accuracy of this method does not depend on the step size At, unless the step size becomes so large that spikes are missed. Of course, this constraint on step size is true for equation 1 as well. By the same logic, equation 2 can be updated as follows: Sum3(t + At) = e-A'"%un3(t) + S Sum4(t + At) = e-A'/roSum4(t)+ S where S = 1 if a spike occurred at time t and S = 0 otherwise, Sum3(t) = cb, e-(f-ti)/w,and Sum4(t) = e-(t-fl)/ro. The synaptic conductance can then be calculated at time t At from mTo [Suml(t + At) - Sum2(t At)] (16) g=
$, +
I, ,[
+
Determining the value of the conductance requires only that Suml,Sum2, TO, and TD be saved for each synapse. In addition, TDTO/(TD - TO) is a constant for a given synapse, and therefore can be precalculated for each synapse. This method requires far fewer exponentiations and additions than does the original closed-form solution (compare equations 1 and 2 to equations 11-16). Furthermore, the accuracy of our method is limited only by the machine precision. It also requires far less memory storage to maintain this accuracy. The number of spikes that must be stored to maintain the precision of equations 1 or 2 depends on (1) the spike frequency, (2) the synaptic time constant, and (3) the required precision level, E . To ensure that a spike that has occurred at some time to in the past will add less than E to equation 1, it must be true that ( ( t- to/t)e < E. It can be shown that, for E < setting (t - tO)/r equal to P ( E )= ln(l/E)
+ (1+ l / l n ( l / ~ ) ] [ ~ ( l n ( l / ~ ) ) ]
(17)
Fast Calculation of Synaptic Conductances
203
will always satisfy this constraint. For example, if E < the value of P ( E )would be 16.63, which implies that t-to must be equal to 16.637, that is, spike to must be stored until a time of 16.637 has elapsed, after which its contribution to equation 1 will be less than If the time constant T of this synapse is 50 msec, this requires that a spike be stored for 831.5 msec. The worst case size of the storage queue for a synapse would then be determined by this storage time, divided by the minimum period between spikes (which determines the maximum number of spikes that may occur during this storage time). If the input cell spikes with a minimum period of 100 msec between spikes, the queue would have to have room to store 831.5/100 M 9 spike times. In general, Max queue size = P(E)T/Tmin where Th,, is the minimum firing period of the input cell. Thus, storage requirements for equation 1 or 2 increase logarithmically with increasing precision [since, as E decreases, the fastest growing term in equation 17 is ln(l/~)I,increase linearly with the synaptic time constant 7, and increase inversely with the minimum firing period Tmh. If one chooses to implement a queue dynamically, one has the computational overhead of keeping track of which spikes have aged sufficiently to be dropped. Whether one uses a fixed size array or a dynamic array, one must use more storage as the precision, input firing frequency, or time constant of a synapse increases. How does our method compare to numerically integrating a secondorder differential equation, injecting new impulses each time an action potential occurs in the presynaptic neuron? A variety of techniques exist for numerically integrating differential equations (Press et al. 1988). One efficient, stable, and fairly accurate technique that is frequently employed is referred to as the exponential technique. For the second-order differential equation yielding equation 1 or 2, applying this technique yields the following finite difference equations (Wilson and Bower 1989, p. 328):
where x ( t ) is nonzero at the time a spike occurs, and zero otherwise. A drawback of this approach is that it is not inherently as precise as our method. One must trade off speed versus precision for equations 19-20. For example, if we choose to use a 1 msec time step for our method, in order to guarantee that deviations from it are smaller than we found that this numerical integration technique must be run with a step size 4 times smaller; to obtain deviations smaller than requires a step size 8 times smaller, and to obtain deviations smaller than requires a step size 16 times smaller. Of course, we could choose a larger step size for our method without loss of accuracy (see above), and the step sizes
204
Rajagopal Srinivasan and Hillel J. Chiel
for the numerical integration technique would then be proportionally smaller. We directly compared the time taken by the three methods by writing three benchmark programs in C (code listings available from the authors on request), and timing them on a Decstation 5000/200. As a reasonable precision limit for the methods, we chose For equation 1 and equations 11-13, a l msec step size was used. For equations 19-20, a l msec step size gave relatively poor precision (deviations were on the order of a step size of 0.125 msec was necessary to limit deviations to less than from the other two methods. The time constant for the synapse was 50 msec, the input spike frequency was 10 Hz, and we chose a queue size for the method of equation 1 that would maintain spike times until they had decayed to values smaller than (from equation 18, we determined that the queue should hold 9 spike times). Simulation time was 200 sec. Using these parameters, our method (equations 11-13) required only 2.1 sec of real time, whereas the method of equation 1 required 15.8 sec of real time, and the method of equations 19-20 required 18.1 sec of real time. These results suggest that our method is superior both in terms of speed and accuracy to previous methods. Acknowledgments
R. S. was supported by a Research Experience for Undergraduates Supplement to H. J. C.’s NSF Grant, BNS 88-10757.H. J. C.thanks the NSF for its support of this research. We are grateful for the comments of Dr.Randall Beer and two anonymous reviewers on an earlier draft of this manuscript. References Press,W. H., Flannery, B. P., Teukolsky, S. A,, and Vetterling, W. T. (eds.) 1988. Numerical Recipes in C . Cambridge University Press, Cambridge. Wilson, M. A., and Bower, J. M. 1989. The simulation of large-scale neural networks. In Methods in Neuronal Modeling, C. Koch and I. Segev, eds., pp. 291333. MIT Press, Cambridge. Received 11 February 1992; accepted 3 August 1992.
This article has been cited by: 1. Mark E. Nelson. 2010. Electrophysiological models of neural processing. Wiley Interdisciplinary Reviews: Systems Biology and Medicine n/a-n/a. [CrossRef] 2. Hans E. Plesser, Markus Diesmann. 2009. Simplicity and Efficiency of Integrate-and-Fire Neuron ModelsSimplicity and Efficiency of Integrate-and-Fire Neuron Models. Neural Computation 21:2, 353-359. [Abstract] [Full Text] [PDF] [PDF Plus] 3. Jan Reutimann , Michele Giugliano , Stefano Fusi . 2003. Event-Driven Simulation of Spiking Neurons with Stochastic DynamicsEvent-Driven Simulation of Spiking Neurons with Stochastic Dynamics. Neural Computation 15:4, 811-830. [Abstract] [PDF] [PDF Plus] 4. Maurizio Mattia , Paolo Del Giudice . 2000. Efficient Event-Driven Simulation of Large Networks of Spiking Neurons and Dynamical SynapsesEfficient Event-Driven Simulation of Large Networks of Spiking Neurons and Dynamical Synapses. Neural Computation 12:10, 2305-2329. [Abstract] [PDF] [PDF Plus] 5. Michele Giugliano . 2000. Synthesis of Generalized Algorithms for the Fast Computation of Synaptic Conductances with Markov Kinetic Models in Large Network SimulationsSynthesis of Generalized Algorithms for the Fast Computation of Synaptic Conductances with Markov Kinetic Models in Large Network Simulations. Neural Computation 12:4, 903-931. [Abstract] [PDF] [PDF Plus] 6. Michele Giugliano , Marco Bove , Massimo Grattarola . 1999. Fast Calculation of Short-Term Depressing Synaptic ConductancesFast Calculation of Short-Term Depressing Synaptic Conductances. Neural Computation 11:6, 1413-1426. [Abstract] [PDF] [PDF Plus] 7. J. Köhn , F. Wörgötter . 1998. Employing the Z-Transform to Optimize the Calculation of the Synaptic Conductance of NMDA and Other Synaptic Channels in Network SimulationsEmploying the Z-Transform to Optimize the Calculation of the Synaptic Conductance of NMDA and Other Synaptic Channels in Network Simulations. Neural Computation 10:7, 1639-1651. [Abstract] [PDF] [PDF Plus] 8. William W. Lytton . 1996. Optimizing Synaptic Conductance Calculation for Network SimulationsOptimizing Synaptic Conductance Calculation for Network Simulations. Neural Computation 8:3, 501-509. [Abstract] [PDF] [PDF Plus] 9. Alain Destexhe, Zachary F. Mainen, Terrence J. Sejnowski. 1994. Synthesis of models for excitable membranes, synaptic transmission and neuromodulation using a common kinetic formalism. Journal of Computational Neuroscience 1:3, 195-230. [CrossRef] 10. A. Destexhe , Z. F. Mainen , T. J. Sejnowski . 1994. An Efficient Method for Computing Synaptic Conductances Based on a Kinetic Model of Receptor BindingAn Efficient Method for Computing Synaptic Conductances Based on a Kinetic Model of Receptor Binding. Neural Computation 6:1, 14-18. [Citation] [PDF] [PDF Plus]
Communicated by David Willshaw
NOTE
The Variance of Covariance Rules for Associative Matrix Memories and Reinforcement Learning Peter Dayan Terrence J. Sejnowski Computational Neurobiology Laboratory, The Salk Institute, P. 0.Box 85800, San Diego, CA 92186-5800 USA Hebbian synapses lie at the heart of most associative matrix memories (Kohonen 1987; Hinton and Anderson 1981) and are also biologically plausible (Brown et al. 1990; Baudry and Davis 1991). Their analytical and computational tractability make these memories the best understood form of distributed information storage. A variety of Hebbian algorithms for estimating the covariance between input and output patterns has been proposed. This note points out that one class of these involves stochastic estimation of the covariance, shows that the signal-to-noise ratios of the rules are governed by the variances of their estimates, and considers some parallels in reinforcement learning. Associations are to be stored between R pairs [a(w),b(w)] of patterns, where a(w) E (0, and b(w) E (0,l}”,using the real-valued elements of an m x n matrix W . Elements of a(w) and b(w) are set independently with probabilities p and r, respectively, of being 1. A learning rule specifies how element Wi, changes in response to the input and output values of a particular pair-the model adopted here (from Palm 1988a,b) considers local rules with additive weight changes for which: n
Wij =
C Aij(w),
where Aij(w) = f [ai(w), bj(w))
W=l
and f can be represented as [a,p, y,61 based on
One way to measure the quality of a rule is the signal-to-noise ratio (S/N) of the output of a single “line” or element of the matrix, which is a measure of how well outputs that should be 0 can be discriminated from outputs that should be 1. The larger the S/N, the better the memory will perform (see Willshaw and Dayan, 1990 for a discussion). A wide variety of Hebbian learning rules has been proposed for heteroNeural Computation
5,205-209 (1993)
@ 1993 Massachusetts Institute of Technology
Peter Dayan and Terrence Sejnowski
206
and autoassociative networks (Kohonen 1987; Sejnowski 1977a; Hopfield 1982; Perez-Vincente and Amit 1989; Tsodyks and Feigel'man 1988). The covariance learning rule fcov = [PT, -p(l - T), -(1 - p ) ~(1, - p)(l - T)], has the highest S/N (Willshaw and Dayan 1990; Dayan and Willshaw 1991); however, both it and a related rule, fprd = [-pr, -pr, -pr, 1 - pr] (Sejnowski1977a),have the drawback that (Y # 0, that is, a weight should change even if both input and output are silent. Note the motivations behind these rules: fcov
-
fprd
(input - p ) x (output - T) input x output - pr
Alternative rules have been suggested that better model the physiological phenomena of long-term potentiation (LTP) and depression (LTD) in the visual cortex and hippocampus, including the heterosynaptic rule fhet = [0, -p, 0 , l - p] (Stent 1973; Rauschecker and Singer 1979), and the homosynaptic rule fhm = [O,O, -T, 1 - T] (Sejnowski 197%; Stanton and Sejnowski 1989), motivated as fhet fhom
N
(input - p) x output input x (output - T)
These have been shown to have lower S/Ns than the covariance rule (Willshaw and Dayan 1990); however, for the sparse patterns, that is, low values of p and T, this difference becomes small. The sparse limit is interesting theoretically, because many more patterns can be stored, and empirically, because the cortex has been thought to employ it (see, for example, Abeles et al. 1990). All of these rules are effectively stochastic ap roximations of the covariance between input and output ((ai(w)- Lii) F j ( U ) - bj)) where the averages Ow are taken over the distributions generating the pltterns; they all share this as their common mean.' If inputs and outputs are independent, as is typically the case for heteroassociative memories, or autoassoaative ones without the identity terms, then their common expected value is zero. However, the rules differ in their variances as estimates of the covariance. Since it is departures of this quantity from its expected value that mark the particular patterns the matrix has learned, one would expect that the lower the variance of the estimate the better the rule. This turns out to be true, and for independent inputs and outputs the S/N of the rules:
S/N
[fcovl
w!p* r(1-p
1-r)
Variance of Covariance Rules for Matrix Memories
207
are inversely proportional to their variances. fcov is the best, fprd the worst, but the ordering of the other two depends on p and r. Circumstances arise under which the optimal rule differs, as for instance if patterns are presented multiple times but input lines can fail to fire on particular occasions-this would favor the homosynaptic rule. Exactly the same effect underlies the differences in efficacy between various comparison rules for reinforcement learning. Sutton (1984) studied a variety of two-armed bandit problems, which are conventional tasks for stochastic learning automata. On trial w, a system emits action y(w) E (0,l) (i.e., pulls either the left or the right arm) and receives a probabilistic reward r(w) E (0,l) from its environment, where
In the supervised learning case above, the goal was to calculate the covariance between the input and output. Here, however, the agent has to measure the covariance between its output and the reward in order to work out which action it is best to emit (i.e., which arm it is best to pull). Sutton evaluated r(w)b(w)- (y(w))] and an approximation to [r(w)- (r(w))][y(w) - (~(w))], where (y(w)) averages over the stochastic process generating the outputs and
is the expected reinforcement given the stochastic choice of y(w). These are direct analogues of fhst orfhom [depending on whether y(w)is mapped to a(w) or b(w)l and fcov, respectively, and Sutton showed the latter significantly outperformed the former. There is, however, an even better estimator. In the previous case, a and b were independent; here, by contrast, r(w) is a stochastic function of y(w). The learning rule that minimizes the variance of the estimate of the covariance is actually
+
where f = P[y(w) = O]p1 Pb(w) = l ] p ~pairs the probability of emitting action 0 with the reinforcement for emitting action 1. Williams (personal communication)suggested ? on just these grounds and simulations (Dayan 1991)confirm that it does indeed afford an improvement. Four previously suggested Hebbian learning rules have been shown to be variants of stochastic covariance estimators. The differences between their performances in terms of the signal-to-noise ratio they produce in an assodative matrix memory may be attributed to the differences in the variance of their estimates of the covariance. The same effect underlies the performance of reinforcement comparison learning rules, albeit suggesting a different optimum.
208
Peter Dayan and Terrence Sejnowski
Acknowledgments
We are very grateful to Steve Nowlan and David Willshaw for helpful comments. Support was from the SERC and the Howard Hughes Medical Institute.
References Abeles, M., Vaadia, E., and Bergman, H. 1990. Firing patterns of single units in the prefrontal cortex and neural network models. Network 1, 13-25. Anderson, J. A., and Rosenfeld, E., eds. 1988. Neurocomputing: Foundations of Research. MIT Press, Cambridge, MA. Baudry, M., and Davis, J. L. 1991. Long-Term Potentiation: A Debate of Current Issues. MIT Press, Cambridge, MA. Brown, T. H., Kairiss, E. W., and Keenan, C. L. 1990. Hebbian synapses: Biophysical mechanisms and algorithms. Annu. Rev. Neurosci. 13, 475-512. Dayan, P. 1991+ Reinforcement comparison. In Connectionist Models: Proceedings of the 2990 Summer School, D. S . Touretzky, J. L. Elman, T. J. Sejnowski and G. E. Hinton, eds. Morgan Kaufmann, San Mateo, CA. Dayan, P., and Willshaw, D. J. 1991. Optimal synaptic learning rules in linear associative memories. Biol. Cybernet. 65, 253-265. Hinton, G. E., and Anderson, J. A., eds. 1981. Parallel Models of Associative Memory. Lawrence Erlbaum, Hillsdale, NJ. Hopfield, J. J. 1982. Neural networks and physical systems with emergent computational abilities. Proc. Natl. Acad. Sci. U.S.A. 79, 2554-2558. Kohonen, T. 1987. Content-addressableMemories, 2nd ed. Springer-Verlag,Berlin. Palm, G. 1988a. On the asymptotic information storage capacity of neural networks. In Neural Computers. NATO AS1 Series, R. Eckmiller and C. von der Malsburg, eds., Vol. F41, 271-280. Springer-Verlag,Berlin. Palm, G. 1988b. Local synaptic rules with maximal information storage capacity. In Neural b Synergetic Computers, Springer Series in Synergetics, H. Haken, ed., Vol. 42, 100-110. Springer-Verlag, Berlin. Perez-Vincente, C. j., and Amit, D. J. 1989. Optimised network for sparsely coded patterns. J. Phys. A: Math. General 22, 559-569. Rauschecker, J. P., and Singer, W. 1979. Changes in the circuitry of the kitten’s visual cortex are gated by postsynaptic activity. Nature (London) 280,58-60. Sejnowski, T. J. 1977a. Storing covariance with nonlinearly interacting neurons. J. Math. Biol. 4, 303-321. Sejnowski, T. J. 1977b. Statistical constraints on synaptic plasticity. J. Theoret. Biol. 69, 385-389. Stanton, P., and Sejnowski, T. J. 1989. Associative long-term depression in the hippocampus: Induction of synaptic plasticity by Hebbian covariance. Nature (London) 339, 215-218. Stent, G. S. 1973. A physiological mechanism for Hebb’s postulate of learning. Proc. Natl. Acad. Sci. 70, 997-1001.
Variance of Covariance Rules for Matrix Memories
209
Sutton, R. S. 1984. Temporal Credit Assignment in Reinforcement Learning. Ph.D. Thesis, University of Massachusetts, Amherst, MA. Tsodyks, M. V., and Feigel'man, M. V. 1988. The enhanced storage capacity in neural networks with low activity level. Europhys. Lett. 6, 101-105. Willshaw, D. J., and Dayan, P. 1990. Optimal plasticity in matrix memories: What goes up MUST come down. Neural Comp. 2,8593. Received 20 January 1992; accepted 14 September 1992.
This article has been cited by: 1. Max Garagnani, Thomas Wennekers, Friedemann Pulvermüller. 2009. Recruitment and Consolidation of Cell Assemblies for Words by Way of Hebbian Learning and Competition in a Multi-Layer Neural Network. Cognitive Computation 1:2, 160-176. [CrossRef]
Communicated by Geoffrey Hinton
NOTE
Optimal Network Construction by Minimum Description Length Gary D. Kendall Trevor J. Hall Departmentof Physics, King’s College London, Strand, London WC2R 2LS, UK
1 Introduction
It has been established that the generalization ability of an artificial neural network is strongly dependent on the number of hidden processing elements and weights (Baum and Haussler 1989). There have been several attempts to determine the optimal size of a neural network as part of the learning process. These typically alter the number of hidden nodes and/or connection weightings in a multilayer perceptron by either heuristic methods (Le Cun et al. 1990; Fahlman and Lebiere 1990) or inherently via some network size penalty (Chauvin 1989; Weigend et al. 1991; Nowlan and Hinton 1992). In this note an objective method for network optimization is proposed that eliminates the need for a network size penalty parameter. 2 Network Complexity
Rissanen has proposed a formalism of Ockham’s razor with the minimum description length (MDL) principle (Rissanen 1989). It is asserted that the smallest representation of an observed training set is the statistical process by which it was generated. Hence the most probable model 8 of any system S given a set of observed data is the model that minimizes the total description length L.
L = L(S I 8 ) + q e ) In a perfect coding system the description length of a set of symbols approaches the sum of the self information of the symbols. It is therefore straightforward to closely approximate the total description length of a neural network system by summing the self information of the parameters of the network itself and the input/output values of each training pair unaccounted for by the network.
L = z(s I e) + z(e) Neural Computation 5,210-212 (1993) @ 1993 Massachusetts Institute of Technology
Optimal Network Construction
211
To calculate the self information I associated with each of these parameters the sum of the self information I(Ek) of the constituent .bytes is taken.
where
Ek
are the individual events of each byte in each parameter and
P(Ek) is the a priori probability of Ek given the distribution of bytes in the corresponding set of parameters (k E (0, . . .,2551). Network optimization using this method can take place for any parameterized neural network model with supervised learning. In minimizing the total description length, both network learning and network construction take place simultaneously. An approach that can be used to optimize an adaptive network architecture by minimum description length is to include additional parameters indicating the status of each node/weight (i.e., used or unused) from a maximum network size. In then calculating the description length of the neural network model it is necessary to include only the network parameters used.
3 Discussion
It has been found experimentally that a genetic optimization algorithm (Goldberg 1988)is successful in minimizing the total description length over the discrete space involved in this method. Weights and training data parameters are coded as signed 16 bit words with each gene representing 8 bits. A population size of 500 with a mutation probability of has been found successful for a network with a maximum of fifteen hidden nodes. Results from network simulations have demonstrated that learning on a distribution about the XOR problem, with a general recurrent network, has resulted in the most simple feedforward design to prevail within 1000 generations: A multilayer perceptron with one hidden node and direct bottom to top connections. Simulations on line detection in 9 x 9 pixel images given 200 training examples, using a multilayer perceptron architecture, have taken 300,000 generations to converge to an optimal matched filter. The final network size and weight distribution produced are implicit in the quantity of and variation in the training data provided. Training by this method not only encourages weight elimination by the reduction of parameters as in Weigend et al. (1991)but also weight sharing through the minimization of the self information of the parameters as in Nowlan and Hinton (1992).
212
Gary D. Kendall and Trevor J. Hall
Acknowledgments The work of G. D. Kendall is supported by the UK Science and Engineering Research Council and Smith System Engineering Limited. References Baum, E. B., and Haussler, D. 1989. What size net gives valid generalization. Neural Comp. 1(1),151-160. Chauvin, Y. 1989. A back-propagation algorithm with optimal use of hidden units. In Advances in Neural Information Processing Systems I , D. S. Touretzky, ed., pp. 519-526. Morgan Kaufmann, San Mateo, CA. Fahlman, S. E., and Lebiere, C. 1990. The cascade-correlation learning architecture. In Advances in Neural Information Processing Systems 2, D. S. Touretzky, ed.,pp. 524-532. Morgan Kaufmann, San Mateo, CA. Goldberg, D. E. 1988. Genetic Algorithms in Search, Optimisation and Machine Learning. Addison-Wesley, Reading, MA. Le Cun, Y., Denker, J. S., and Solla, S. A. 1990. Optimal brain damage. In Advances in Neural Information Processing Systems 2, D. s. Touretzky, ed., pp. 598605. Morgan Kaufmann, San Mateo, CA. Nowlan, S. J., and Hinton, G. E. 1992. Simplifying neural networks by soft weight-sharing. Neural C O M ~4(4), . 473-493. Rissanen, J. 1989. Stochastic Complexity in Statistical Inquiry. World Scientific. Weigend, A. S., Rumelhart, D. E., and Huberman, 8. A. 1991. Generalization by weight-elimination with application to forecasting. In Advances in Neural Infomation Processing Systems 3, R. P. Lippmann, J. E. Moody, and D. S. Touretzky, eds., pp. 875-882. Morgan Kaufmann, San Mateo, CA. Received 10 April 1992; accepted 3 September 1992.
This article has been cited by: 1. X.M. Gao, X.Z. Gao, J.M.A. Tanskanen, S.J. Ovasaka. 1997. Power prediction in mobile communication systems using an optimal neural-network structure. IEEE Transactions on Neural Networks 8:6, 1446-1455. [CrossRef] 2. Guozhong An . 1996. The Effects of Adding Noise During Backpropagation Training on a Generalization PerformanceThe Effects of Adding Noise During Backpropagation Training on a Generalization Performance. Neural Computation 8:3, 643-674. [Abstract] [PDF] [PDF Plus] 3. Gary D. Kendall, Trevor J. Hall, Timothy J. Newton. 1993. An investigation of the generalisation performance of neural networks applied to lofargram classification. Neural Computing & Applications 1:2, 147-159. [CrossRef]
Communicated by Jack Byme
A Neural Network Model of Inhibitory Information Processing in Aplysia Diana E. J. Blazis Thomas M. Fischer Thomas J. Carew Departments of Psychology and Biology, Yale University,
New Haven, CT 06520 USA
Recent cellular studies have revealed a novel form of inhibitory information processing in the siphon withdrawal reflex of the marine mollusc ApZysia: Motorneuronal output is significantly reduced by activity-dependent potentiation of recurrent inhibition within the siphon withdrawal network (Fischer and Carew 1991, 1993). This inhibitory modulation is mediated by two types of identified interneurons, L29s and L30s. In an effort to describe and analyze this and other forms of inhibitory information processing in Aplysia, and to compare it with similar processing in other nervous systems, we have constructed a neural network model that incorporates many empirically observed features of these interneurons. The model generates important aspects of the interactions of cells L29 and L30, and with no further modification, exhibits many network level phenomena that were not explicitly incorporated into the model. 1 Introduction
Recurrent inhibitory circuits are a common architectural feature of nervous systems, providing a powerful mechanism for rapid and precise control of neural output (Pompeiano 1984). Moreover, intrinsic and extrinsic modulation of recurrent inhibitory circuitry endows those networks with a high degree of flexibility and an enhanced capacity for adaptive modification. This type of modulation has been described in several systems including spinal motor neurons (Pompeiano 1984; Fung et al. 1988) and hippocampal circuits (Miles 1991). Recent cellular studies in the marine mollusc Aplysia have identified a recurrent inhibitory circuit in the neural network underlying the siphon withdrawal reflex (SWR)(Frost et al. 1988; Hawkins and Schacher 1989; Fischer and Carew 1991, 1993). The circuit is formed between identified interneurons L29 and L30: L29 provides excitatory input to L30, which projects back on L29 with an inhibitory synapse. The output element of this circuit, cell Neural Computation 5, 213-227 (1993) @ 1993 Massachusetts Institute of Technology
214
D. E. J. Blazis, T. M. Fischer, and T. J. Carew
L29, provides substantial input to siphon motor neurons (MNs). In the L29/L30 circuit, recurrent inhibition of L29 can be modulated in at least two ways: (1)it is reduced by stimulation mimicking tail shock (Frost et al. 1988), and (2) it is increased by prolonged direct activation of either L29 or L30 (Fischer and Carew 1991, 1993). Thus, L29/L30 interactions provide an example of a recurrent inhibitory circuit that can be modified by both extrinsic and intrinsic inputs. Moreover, the L29/L30 circuit is of additional interest because it exhibits an intrinsic form of plasticity, activity-dependent potentiation of recurrent inhibition (described more fully below). This form of use-dependent regulation of recurrent inhibition endows the L29/L30 circuitry with the capacity for dynamic gain control in the SWR (Fischer and Carew 1993). The SWR of Aplysia has been widely used as a model system for studies of learning and memory. The siphon is located in the mantle cavity and is used as an exhalent funnel for respiration (Fig. 1A). In response to tactile stimuli delivered directly to the siphon or to other sites on the body the animal withdraws its siphon into the mantle cavity. The SWR and its circuitry show a variety of adaptive modifications (for review, see Carew and Sahley 1986). Large cells and restricted neural networks have facilitated the identification and analysis of neural circuitry underlying this response and its modification. Interneurons L29 and L30 are located in the abdominal ganglion (a central ganglion that contains most of the circuitry underlying the SWR) (Fig. IB). L29 interneurons (about 5 in number) were previously shown to play an important role in excitatory information processing in the SWR circuit (Hawkins et al. 1981; Frost et al. 1988; Hawkins and Schacher 1989). L29s are activated by siphon stimulation, provide substantial excitatory input to identified siphon motor neurons (MNs, e.g., the LFS MNs, Frost et al. 1988, see also Fischer and Carew 1993), and receive recurrent inhibition from L30 interneurons (about 2 in number). Recently, Fischer and Carew (1991,1993) have shown that direct activation of a single L29 produces transient inhibition of the total reflex input to the MNs. They further showed that this inhibition occurs because L29s recruit recurrent inhibition onto themselves from L30 interneurons. The net effect of this recruitment of inhibition is that the L29 response to siphon input is significantly reduced for up to 40 sec following intracellular activation, resulting in a smaller net EPSP in the MN. The mechanism subserving this inhibition appears to be use-dependent facilitation of the inhibitory post-synaptic potential (IPSP) from L30 to L29 (Fischer and Carew 1991, 1993). We have recently developed a neural network model to describe this form of inhibitory information processing in Aplysia both to quantitatively analyze this type of adaptive modification in the SWR and to examine the possible behavioral significance of the L29/L30 circuit in the SWR neural network. Our overall strategy is to first represent in computational form the principal features of each identified cell, and then to
Inhibitory Information Processing in Aplysia
A.
215
B.
C.
Figure 1: Schematicillustrationof the siphon withdrawal reflex of Aplysia examined at several levels of analysis. (a) Diagram of intact Aplysia, illustrating the position of the siphon (adapted from Kandel 1979). (b) Diagram of a reduced preparation used to analyze neuronal correlates of behavior, showing approximate locations of interneurons and motorneuronsdescribed in the present study. Reflex input can be elicited with a tactile stimulus to the siphon while simultaneously recording from identified elements in the circuit for siphon withdrawal. (c) Connectivity of neural network model. The asterisk denotes a synapse that exhibits use-dependent potentiation.
216
D. E. J. Blazis, T. M. Fischer, and T. J. Carew
progressively refine the model by continually adding empirically derived cellular parameters. Thus, a number of important cellular parameters and interactive mechanisms are not yet fully incorporated in the model. [Some of these features have been modeled by other investigators: sensorimotor facilitation and inhibition have been analyzed by Gingrich and Byrne (1985)and Byrne et al. (19891,and interneuronal contributions to MN response duration have been studied by Frost et al. (1991).1Nonetheless, our model describes key features of the relationship between L29 and L30 and generates network-level phenomena of the SWR, including activity-dependent potentiation of recurrent inhibition.
2 Methods
Biophysical and synaptic parameters in the model are derived or estimated from cellular studies in Aplysia and other invertebrates. Parameters of the model are listed in Table 1, and equations for selected features of the model are listed below. We have implemented our model with the network simulator GENESIS (Wilson et al. 19891, using an exponential Euler integration method and a time step of 0.1 msec.
2.1 Network Architecture. The network consists of a sensory module and four single compartment cells: three interneurons and a motor neuron (Fig. 1C).The sensory module consists of four receptors that mimic siphon sensory input. The number of sensory receptors selected was arbitrary. In Aplysia, sensory input from the siphon accelerates rapidly and then declines over the course of continued stimulation. The mechanisms underlying this accommodation are not yet incorporated into the model; instead, receptor firing rates are set so as to decline over stimulus presentation. Another feature of sensory neurons is synaptic depression, which occurs with repeated sensory neuron stimulation at relatively short interstimulus intervals (ISIs; Castelluci et al. 1970; Byrne et al. 1989). In our empirical studies, siphon stimulation was presented at an IS1 of 5 min, an interval that precludes habituation at both behavioral and synaptic levels. Therefore, the current model does not incorporate a representation of synaptic depression. The MN has only three synaptic inputs, one from the sensory module, one from a generic excitatory interneuron (denoted as E in Fig. 10, and one from a single L29. Three features of the biological circuit are incorporated in the network: (1) L29 makes an excitatory synapse onto L30; (2)L30, in turn, projects back upon L29 with an inhibitory synapse that shows marked use-dependent potentiation (indicated by an asterisk); and (3) there is a rectifying electrical synapse between L29 and L30.
Inhibitory Information Processing in Aplysia
217
Table 1: Parameters of the Model. Parameter
L29
L30
E
MN
c,
.63 -61 12
.63 -37 12
.63 -40 12
.63 -4s 12
1.7 +55 .0025 -65 2.85 -65
1.98 $55 0.61 -60
3.14 +55 0.28 -60
3.14 +55 0.61 -60
(nF) Em (mV) Rill
Active conductances G N (lo-' ~ S) (mV) GK, s) EK, (mV) G K ~ s) E m 9 (mV) Leak conductance G b , a (lO-'OS) E ~ e a k(mV) sensory inputs G (10-5 s) 71 (msec), 72 (msec) Em, (mV) output Target G (10-5 s) 71 (msec), (msec) E , (mV) Electrical s y n a p d Target
r Saturation (nA)
80 -75
1.7 13 55 L30/MN 0.11 1,3/ 1,7 55
L30 02-7
0.55
-
-
-
0.1
0.004
-75
-75
0.004 -75
5.5 13 55
7.5 L3 55
1.7 13 55
-
L29
MN
0.054-0.4ob
0.11
9,s -65
13
L29 0.05e-' 0.08
55
-
-
-
@Hyperpolarizedto -70 mV in all simulations. bConductancerange, computed as a function of WO activation frequency and time; see text equation 3. CCurrentacross electrical synapse: See equations 4a and 4b and accompanying text for explanation of terms.
2.2 Synaptic Conductances. The model incorporates time-dependent synaptic conductances. Ionic current resulting from activation of a synaptic conductance is calculated as follows:
where I = current, E,, reversal potential of the ion, and Emem= resting potential of the cell. G(t) is computed with the dual exponential form:
D.E.J. Blazis, T. M.Fischer, and T.J. Carew
218
where w is a weighting factor that is free to vary. For the EPSP and IPSP between L29 and L30 (respectively), the reversal potential and time constants of the simulated conductances reflect those empirically observed. The magnitudes of synaptic contributions from each of the three pathways converging onto the MN were specified in the following way. First, the strength of the sensory input was set to result in peak complex EPSP amplitude of 5-10 mV (in all cases, the MN was hyperpolarized to -70 mV, as in empirical studies). The strength of the L29 EPSP onto the MN was set to produce a depolarization of about 10 mV. The strength of the sensory input to L29 was set to yield an appropriate firing pattern in response to a short stimulus (that is, 4-6 action potentials to a 100 msec activation of the sensory receptors). Since the complex EPSP seen in the MN in response to sensory stimulation typically has a peak amplitude of 25-30 mV (Fischer and Carew 19921, the strength of the connection from interneuron E was then set to make up the difference between that amount and the contribution of the sensory array and the L29 input. A look-up function describing activity-dependent potentiation of the L30 IPSP has been incorporated in the model. This preliminary representation of potentiation of the L30 IPSP is based on a linear regression of data obtained by Fischer and Carew (1991,1992)and specifies that the inhibitory synaptic conductance from L30 onto L29 changes as a function both of L30 activation and of time:
+
+
~ ( t =) k ( O . 1 ~ 0.591~ I)
(3)
where x = spike count of L30 during activation, y = time after stimulation offset, and k = 3400 (a value which yields approximately a 1 mV IPSP in L29 at baseline). In the simulations presented here, activation of L30 by sensory stimulation is not taken into account. Thus the model slightly underestimates the weight of the inhibitory connection from L30 to L29. 2.3 Active Conductances. Active conductances are based on the Hodgkin and Huxley (1952)formulations. At present, each cell has a sodium conductance (GNJ, a delayed rectifier potassium conductance the implethat exhibits inactivation (GK"), and a leak conductance (GL); mentations of these conductances were drawn from existing GENESIS version 1.3 libraries. L29 has an additional, noninactivating potassium current (GKU~,derived from Byrne 1980). Parameters for the various conductances were set to approximate the firing patterns of each cell. In more detailed versions of the model, other conductances will be integrated as well (e.g., the Ca-dependent potassium conductance described for cell L30,Frost et al. 1988).
2.4 Electrical Synapse. The model includes a representation of the electrical synapse between L29 and L30. The current through the electri-
Inhibitory Information Processing in Aplysia
219
cal synapse at a given time is proportional to the differencebetween the potentials of cells L29 and L30:
where rl and r2 (values in Table 1) are coupling coefficients for the simulated electrical synapse between cells L29 and L30 that reflect the results of empirical studies (Fischer and Carew 1993). Like the biological synapse, the simulated electrical synapse is recbfying, such that current passes more readily from L29 to L30 than in the reverse direction. The current from one cell to another is also bounded (“saturation” term in Table 1)to reflect the fact that large voltage trajectories, such as those due to action potentials, are severely attenuated by the electrical synapse. Taken collectively, the above assumptions allow the model to generate salient features of the L29-L30 interaction. In the sections that follow, we compare our empirical results with our simulations of a variety of physiological manipulations of this simple network. 3 Results 3.1 Characteristic L29/L30 Interactions. Figure 2 shows that the model produces key features of the firing pattern of interneuron L29. The left column shows empirically obtained results (Fischer and Carew 1991,1993),and the right column shows the simulations. Traces A, through D1 of Figure 2 show characteristic features of cell L29: (1)L29 responds to siphon input with a brief burst of action potentials (Fig. 2A1). (2) L29 exhibits a characteristic response to maintained intracellular injection of depolarizing current, responding initially with a brisk burst of action potentials that becomes arrhythmic (Fig. 2B1). This %uttering” occurs because L29 activates L30, thereby recruiting inhibition onto itself. (3) Subthreshold depolarization of L29 recruits IPSPs back onto itself (Fig. 2C1) by virtue of current flow to L30 through the electrical synapse (L30has a lower voltage threshold for action potential initiation than L29). (4) L29 receives an IPSP from L30 that is potentiated by repetitive firing of L30 (Fig. 2D1).In the experiment shown in Figure 2D, cell L29 has been hyperpolarized to a membrane potential more negative than the reversal potential of the L30 IPSP, resulting in a depolarizing synaptic response. A single action potential from cell L30 (indicated by the arrow) results in a small IPSP in L29 (PRE).When cell L29 is then repeatedly activated, thereby activating L30 (not shown), dramatic usedependent facilitation results in a 2- to 3-fold increase in the L30 IPSP amplitude (POST). Our simulations of the L29-L30 interactions are qualitatively quite similar to the empirical data (Fig. 2, A2-41, but several aspects of the
D. E. J. Blazis, T. M. Fischer, and T. J. Carew
220
EMPIRICAL RESULTS
SIMULATIONS
B,
m
WO
Splke
Figure 2 Comparison of empirical results and simulations showing characteristic features of the firing pattern of cell L29 and interactions between cells L29 and L30. (Al, A2) Response of cell L29 to brief tactile stimulation. (B1, b)Response of cell L29 to intracellular current injection. (C1, Cz) Recruitment of IPSPs onto L29 from L30 resulting from subthreshold current injection into L29. (D1,D2)Use-dependent potentiation of the IPSP from L30 to L29 (the IPSP is depolarizing due to hyperpolarization [HYP]of L29, see text). In both the empirical and simulated experiments, L29 was activated for 5 sec by current sufficient to produce a firing rate of 30 Hz (approximately 3 4 nA),which generates sufficient activation of L30 (not shown) to potentiate the L30-L29 IPSP
simulations warrant comment. First, the frequency of firing to siphon input is somewhat lower for the simulated L29 (Fig. 2A2)than that empirically observed. Second, the simulated firing frequency of L29 to current injection does not completely match the empirical data (Fig. 2B2). Third, there is a reasonable match of the empirical data to simulations of subthreshold recruitment of inhibition onto L29 (Fig. 2C2)and facilitation of the IPSP from L30 to L29 (Fig. 2D2).
Inhibitory Information Processing in Aplysia
221
In summary, the model produces essential features of the interaction between L29 and L30. In addition, as will be discussed below, with no further changes, the model also generates important aspects of the synaptic input to the MNs as well as inhibitory modulation of that input. 3.2 Reduction of M N Output by Voltage Clamp of L29. Empirical results shown in Figure 3 (upper half) illustrate the effect on the complex EPSP in a siphon MN of voltage clamping a single L29 during siphon stimulation. On the left, synaptic input to the MN and L29 was elicited by tactile stimulation of the siphon (see Fig. 1B). On the right are shown the responses of these same cells to siphon tap when a single L29 is functionally removed from the circuit by voltage clamping it at its resting potential (approximately -60 mV). In the example shown in Figure 3, the evoked complex EPSP in the MN is diminished to 50% of control. The simulated data (Fig. 3, lower half) capture this result qualitatively, although on average the simulated complex EPSP is shorter in duration than the empirically measured EPSP (note differences in gain and time base in Fig. 3). 3.3 Inhibition of M N Output by Intracellular Activation of L29. As described above, Fischer and Carew (1991, 1993) found that intracellular activation of L29 results in sigruficant inhibition of the tap-evoked complex EPSP in the MN for about 40 sec following L29 activation. In the example shown in Figure 4A1, activation of the S W R network by a siphon tap resulted in a complex E S P of about 25 mV in the MN and a brisk burst of action potentials in L29. A single L29 was then activated with intracellular current for 5 sec (not shown). Twenty seconds after L29 activation, the complex EPSP elicited by an identical siphon tap was significantly reduced (to about 5 mV), as was the response of L29 itself. Both the EPSP and the response of L29 to siphon tap recovered 5 min later. As shown in Figure 4A2, our simulations of activity-dependent potentiation of recurrent inhibition induced by activation of L29 are qualitatively similar to the empirical results. Specifically, 20 sec after L29 activation (and the consequent activity-dependent potentiation of the L30 IPSP onto L29) the tap-evoked complex EPSP is reduced (although the magnitude of the inhibitory effect is smaller than that empirically observed, see Discussion), and the response of L29 itself is also markedly reduced. Both the complex EPSP and the response of L29 to siphon tap recovered 5 min later. The mechanism subserving the inhibitory effects of L29 appears to be that it recruits inhibition onto itself from L30. Figure 4B1 is an empirical record showing that injection of depolarizing current into L29 activates L30 through both chemical and electrical synapses (Fig. 1C). This empirical result is also produced by the model (Fig. 4B2). That L29 activation potentiates the L30 IPSP implies that direct activation of L30 should also produce inhibition of the complex EPSP at the MN; this is indeed the
D.E. J. Blazis, T. M. Fischer, and T. J. Carew
222
case (Fischer and Carew 1991,1993). Moreover, the magnitude and time course of IPSP potentiation by activation of L29 (Fig. 2D1)map onto the inhibition of the tap-evoked response of the MN (Fischer and Carew 1991, 1993).
EMPIRICAL RESULTS
SIMULATIONS Baseline -------
Clamp
LFS Motor Neuron
L29
Vm
1
000 Mln SlphonTap
I
500 Mln
J15 mV 5 o m
Inhibitory Information Processing in Aplysia
223
4 Discussion
We have presented a preliminary computational analysis of a neural network that exhibits activity-dependentpotentiation of recurrent inhibition. We are encouraged that the model captures some of the key features of the L29-L30 interaction and, without further modification, exhibits some of the empirically observed circuit interactions, including inhibitory modulation, that were not explicitly incorporated into the model. 4.1 Discrepancies between Empirical Results and Simulations. The simulations of activation (through current injection) and inactivation (through voltage clamp) of L29 (Figs. 3 and 4) show that the model describes the empirical results qualitatively, but, at least in some cases, not quantitatively. First, the EPSPs in the simulated MN are of shorter duration than those observed empirically; and in the case of activation of L29, the magnitude of inhibition observed is somewhat smaller than that shown in Figure 4A (although the magnitude of the simulated inhibition is quite close to the average L29-induced inhibition of 20% observed by Fischer and Carew 1993). It is likely that the discrepancies between simulated and empirical results due to the fact that the current model incorporates only a single L29 into the circuit, whereas there are at least 5 L29 cells in the actual reflex circuit. Most of the L29s appear to contribute to the EPSP recorded in the MN (Frost et aZ. 1988; Hawkins and Schacher 1989; Fischer and Carew 1993), and thus our model underestimates the summed contribution of the entire L29 class. Ongoing simulations incorporating multiple L29s can diredly test this hypothesis (Blazis ef aZ. 1992) and, in addition, may shed light on the unique contribution of a single L29 to various features of network behavior, a determination that would be difficult (if not impossible) to achieve in cellular experiments alone. Related questions of interest in this computational analysis will focus on the functional significanceof particular architectural and synaptic features of the network, such as the role of multiple L29s and W s , the contribution of the redifyhg electrical synapse between L29 and L30, and the dynamics of facilitation of L30 synapses under different activation conditions.
Figure 3 Facing p g e . Comparison of empirical results and simulations showing the response of L29 and a siphon MN to tactile stimulation before and during voltage clamp of L29. For both the empirical and simulated results, left-hand traces show the mponse of the MN and a single L29 to a siphon tap (arrow) before voltage clamp. Right-hand traces show the response of the h4N and L29 when L29 is voltage clamped at its resting potential. In this and subsequent figures, the MN is hyperpolarized to reveal underlying EPSPs.
224
D.E. J. Blazis, T.M. Fischer, and T.J. Carew
4.2 Role of the L29L30 Circuit in Information Processing in the SWR. Our current cellular and computational analyses are aimed at determining the functional relationships between activity-dependentpotentiation of recurrent inhibition and other forms of plasticity observed in
Inhibitory Information Processing in Aplysia
225
the SWR. For example, since the L30 synapse potentiates at relatively low rates of activation (Fischer and Carew 1993)one form of plasticity that the L29-L30 circuit could theoretically mediate is habituation (habituation is know to involve reduction of afferent input to MNs with repeated sensory stimulation, see Castelluci et al. 1970). Build-up of use-dependent potentiation of the L30 IPSP, achieved via repeated siphon stimulation at short ISIs known to produce habituation, could progressively remove the contribution of L29 interneurons to siphon MN output. Indeed, our ongoing simulations suggest that the L29/L30 circuit alone can mediate at least partial habituation of reflex output, depending on stimulus duration and IS1 (Blazis et al. 1992). Thus, inhibition created by the L29/L30 recurrent inhibitory circuit could augment other mechanisms, such as homosynaptic depression, thought to subserve habituation in Aplysia (Castelluci et al. 1970). In addition to a role in nonassociative learning, L29 and L30 might also contribute to associative processes. For example, as described by Fischer and Carew (19931, this interneuronal network could play a role in changes in response topography associated with classical conditioning of the SWR (Hawkins et al. 1989; Walters 1989). In conclusion, cellular and computational analyses of simple nervous systems like that of Aplysia can yield insights that are useful for understanding both natural and artificial intelligent systems (see Hawkins and Kandel 1984; Card and Moore 1990). To date, most studies of the SWR of Aplysia have focused on a single synapse, that between the siphon sensory neurons and motor neurons (for review, see Carew and Sahley 1986). However, several studies (Frost et al. 1988; Hawkins and Schacher 1989; Fischer and Carew 1991, 1992) as well as the present work show that the SWR network also contains functionally important recurrent inhibitory circuits that can be modulated by both extrinsic
Figure 4: Facing page. Comparison of empirical results and simulationsshowing inhibition of the complex EPSP in the MN and recruitment of L30 firing following intracellular activation of L29. (Al, A2) Empirical result (All and simulation (A2) of L29-induced inhibition of the complex ESP in the h4N. For each pair of traces, the top trace shows the response of the MN md the lower trace shows the response of L29. 0.00 min: response of MN and L29 to siphon tap (arrow). 5.00 min: response of MN and L29 to siphon tap 20 sec following intracellular activation of a single L29 (3-4 nA, 5 sec, as in Fig. 2D). The diminished EPSP amplitude of the MN and reduced responding of L29 occurs because activation of L29 recruits inhibition from L30 onto L29, effectively removing L29 from the circuit (see text). 10.00 min: response of the MN and L29 to siphon tap 5 minutes after L29 activation. (B1, B2) Empirical result (B1) and simulation (B2) of activation of L30 by intracellular current injection into L29. Note that, in response to maintained depolarization of L29, L30 continues to fire action potentials (even after L29 has been silenced by recurrent inhibition)due to the electrical synapse between L29 and L30 (see Fig. 2B,C).
226
D. E. J. Blazis, T. M. Fischer, and T. J. Carew
inputs and intrinsic activity. The ability of cellular studies in Aplysia to pmvide critical physiological constraints for a biologically realistic model greatly facilitates a computational analysis. In tun,a computational analysis can pmvide general insights into the principles of operation underlying different forms of information processing in an identified neural network. Acknowledgments
We thank Kent Fitzgerald, Edward Kairiss, and Emilie Marcus, and two anonymous reviewers for valuable comments on an earlier version of the manuscript. We are also grateful to David Berkowitz and Edward Kairiss for many helpful discussions. This research was supported by National Research Institute Service Award IF32-MH10134-02to D. E. J. B., Grant PHS T32-MH1839705to T.M. F., and AFOSR Award AF 89-0362to T.J. C. References Blazis, D. E. J., Fischer, T. M., and Carew, T. J. 1992. A neural network model of use-dependent gain control in the siphon withdrawal reflex of Aplysia. SOC. Neurosci. Abstr. 18,713. Byrne, J.H.1980. Quantitative aspects of ionic conductance mechanisms contributing to firing pattern of motor cells mediating inking behavior in Aplysia californica. J. Neurophysiol. 43, 651-668. Byrne, J. H., Gingrich, K. J., and Baxter, D. A. 1989. Computational capabilities of single neurons: Relationship to simple forms of associative and nonasd a t i v e learning in Aplysia. In Computational Models of Learning in Simple Neural Systems, R. D. Hawkins and G. H. Bower, eds., pp. 31-63.Academic Press, San Diego. Card, H.C., and Moore, W. R. 1990. Silicon models of associative learning in Aplysia. Neural Networks 3,333-346. Carew, T. J., and Sahley, C. L. 1986. Invertebrate learning and memory: From behavior to molecules. Annu. Rev. Neumci. 9,435487. Castelluci, V., Pinsker, H., Kupfermann, I., and Kandel, E. R. 1970. Neuronal mechanisms of habituation and dishabituation of the gill-withdrawal reflex of Aplysia. Science 167,1745-1748. Fischer, T. M.,and Carew, T. J. 1991. Activation of the facilitatory interneuron L29 produces inhibition of reflex input to siphon motor neurons in Aplysia. Soc. Neurosci. Abstr. 17,1302. Fischer, T. M.,and Carew, T. J. 1993. Activity dependent potentiation of recurrent inhibition: A mechanism for dynamic gain control in the siphon withdrawal reflex of Aplysia. I. Neurosc., in press. Frost, W. N., Clark, G. A., and Kandel, E. R. 1988. Parallel processing of shortterm memory for sensitization in Aplysia. J. Neurobiol. 19,297-334.
Inhibitory Information Processing in Aplysia
227
Frost, W. N., Wu., L. G., and Lieb, J. 1991. Simulation of the Aplysia siphon withdrawal circuik Slow components of internmnal synapses contribute to the mediation of reflex duration. SOC.Neurosci. Abstr. 17,1390. Fun& S. J., Pompeiano, O., and Barnes, C. D. 1988. Coerulospinal influence on recurrent inhibition of spinal motonuclei innervating antagonistic hindleg muscles of the cat. PflugmArch. 412,346-353. Gingrich, K. J., and Byme, J. H. 1985. Simulation of synaptic depression, posttetanic potentiation, and presynaptic facilitation of synaptic potentials from sensory neurons mediating gill-withdrawal reflex in Aplysia. J.Neurophysiol. 53,652-669. Hawkins, R. D., and Schacher, S. 1989. Identified facilitator neurons L29 and L28 are excited by cutaneous stimuli used in dishabituation, sensitization, and classical conditioning of Aplysia. J. Neurosci. 9,42364245. Hawkins, R. D., and Kandel, E. R. 1984. Is there a cell biological alphabet for simple forms of learning? Psychol. Rev. 91,375-391. Hawkins, R. D., Castelluci, j7. F., and Kandel, E. R 1981. Interneurons involved in mediation and modulation of gill-withdrawal reflex in Aplysia. I. Identification and characterization. J. Neumphysiol. 45,304-314. Hawkins, R. D., Lalevic, N., Clark, G. A., and Kandel, E. R. 1989. Classical conditioning of the Aplysia siphon-withdrawal reflex exhibits response specificity. Proc. Natl. Acad. Sci. U.S.A. 86,7620-7624. Hodgkin, A. L., and Huxley, A. F. 1952. A quantitativedescription of membrane current and its application to conduction and excitation in nerve. J. Physiol. (London) 138,253-281. Kandel, E. R. 1979. Behavioral Biology of Aplysia: A Contribution to the Comparative Study of OpisthobranchMolluscs. W. H. Freeman, San Francisco. Miles, R. 1991. Tetanic stimuli induce a short-term enhancement of recurrent inhibition in the CA3 region of guinea-pig hippocampus in vitro. J. Physiol. 443, 669-682. Pompeiano, 0. 1984. Recurrent inhibition. In Handbook of the Spinal Cord, N. Davidoff, ed., pp. 461-557. Marcel Dekker, New York. Walters, E. T. 1989. Transformation of siphon responses during conditioning of Aplysia suggests a model of primitive stimulus-response association. Proc. Natl. Acad. Sci. U.S.A. 86, 7616-7619. Wilson, M. A., Bhalla, U. S., Uhley, J. D., and Bower, J. M. 1989. GENESIS: A system for simulating neural networks. In Advances in Neural Information Processing Systems I, D. S. Touretzky, ed.,pp. 485492. Morgan Kaufmann, San Mateo, CA. Received 27 February 1992; accepted 21 August 1992.
This article has been cited by: 2. S. Yamada, M. Nakashima, S. Shiono. 1998. Reinforcement learning to train a cooperative network with both discrete and continuous output neurons. IEEE Transactions on Neural Networks 9:6, 1502-1508. [CrossRef] 3. Joe L. Martinez, Brian E. Derrick. 1996. LONG-TERM POTENTIATION AND LEARNING. Annual Review of Psychology 47:1, 173-203. [CrossRef]
Communicated by Gordon Shepherd
Computational Diversity in a Formal Model of the Insect Olfactory Macroglomerulus C . Linster Ecole Supkrieure de Physique et de Chimie Industrielles de la Ville de Paris, Luboratoire d'Electronique, 10, rue Vauquelin, 75005 Paris, France
C. Masson Luboratoire de Neurobiologie Comparie des Invertkhris, INRAICNRS (URA1190), 92140 Bures Sur Yvette, France
M. Kerszberg Biologie Molkulaire, CNRS (USA 1284), Institut Pasteur, 25, rue du Docteur Roux, 75015 Paris, France
L. Personnaz G . Dreyfus Ecole Suphieure de Physique et de Chimie Industrielles de la Ville de Paris, Luboratoire d'Electronique, 10, rue Vauquelin, 75005 Paris, France
We present a model of the specialist olfactory system of selected moth species and the cockroach. The model is built in a semirandom fashion, constrained by biological (physiological and anatomical) data. We propose a classification of the response patterns of individual neurons, based on the temporal aspects of the observed responses. Among the observations made in our simulations a number relate to data about olfactory information processing reported in the literature; others may serve as predictions and as guidelines for further investigations. We discuss the effect of the stochastic parameters of the model on the observed model behavior and on the ability of the model to extract features of the input stimulation. We conclude that a formal network, built with random connectivity, can suffice to reproduce and to explain many aspects of olfactory information processing at the first level of the specialist olfactory system of insects. 1 Introduction
We study the detection of sexual pheromones by insects, with a view to the more general modeling of the olfactory pathway. We use the known anatomical data of the olfactory system, and retain the level of detail we deem necessary to produce biologically relevant behavior. Thus, we do Neural Computation 5, 228-241 (1993) @ 1993 Massachusetts Institute of Technology
Model of the Insect Olfactory Macroglomerulus
229
not attempt to represent the particulars of dendritic passive propagation, or the precise local input-output functionalityof individual synapses. We model them by simple ingredients such as propagation delays and activation thresholds. Precise biological data on the wiring is not available, therefore we introduce randomness in the connectivity. A variety of approaches to the modeling of olfactory systems have been presented thus far: the pioneering work of Rall and Shepherd (1968) exploits a wealth of detail concerning the precise shape of mitral cell dendrites in order to compute electrical potentials; Wilson and Bower (1988, 1989) and Haberly and Bower (1989) replicate certain basic features of responses by extensive simulations of larger cell sets exploiting the same data; Lynch and Granger (Lynch et al. 1989; Lynch and Granger 1989) study associative memory and synaptic adaptation in piriform cortex, including considerable detail about synaptic processes, and a Hebb-type learning rule; Li and Hopfield (1989) attempt to abstract a set of relevant parameters from the biological details of the olfactory modular organization, with a highly simplified model: interneurons are lumped into single variables. In contrast, we study the individual and collective behavior of neurons whose dendrites make contacts within the so-called macroglomerulus (or macroglomerular complex, MGC), which is responsible for sexual pheromone recognition. The aim of our work is to analyze the emergence of the responses necessary for odor recognition and localization. 2 Biological Background
In the olfactory system of insects, two subsystems process behaviorally important odor classes: the specialist subsystem detects sexual pheromones, while the generalist subsystem recognizes food odors (for a review see Masson and Mustaparta 1990). In the following, we focus on the specialist subsystem. It receives information from sensory neumns, which are sensitive to non-overlapping molecule spectra ("labeled lines"). The axons of sensory neurons project onto the antennal lobe local interneurons, which possess no axons, and onto the antennal lobe projection or output neurons. The latter transfer signals to other centers for further integration with other sensory modalities. The huge convergence between pheromone-sensitive and projection neurons, which Ernst and Boeckh (1983) estimate as 5000:l in the cockroach, leads to a characteristic spatial organization of all synaptic connections in subassemblies termed gzomeruli, which are identifiable and species-specific. In the case of interest to us, (e.g., in certain moth species and in the cockroach),this reduces to a single MGC (Fig. 1). We use data pertaining to the moth species Manducu sexta and to the cockroach Penplaneta americana (for reviews see Christensen and Hildebrand 1987a; Boeckh and Ernst 1987). The complex responses to stim-
230
C. Linster et al.
Figure 1: schematic representation of the specialist oIfactory system. In the macmglomdus, receptor cell axons connect with local interneurons(restricted to the antennal lobe), and with pmjection n m n s , which convey information to higher centers.
ulation by pheromone blends, as observed intracellularly in projection newom, indicate that integrative pmesses take place in the MGC. In the moth, the depolarization of a local intemeuron can cause inhibition of background activity in a projection neuron. There is also evidence that local interneurons are responsible for much or all of the inhibitory synaptic activity (Christensen and Hildebrand 198%). Furthermore, the long-latency excitation exhibited by some projection neurons suggests that polysynaptic pathways are present between pheromone-responsive primary afferent axons and the projection neurons. In fact, it has been demonstrated, in the cockroach, that the receptor axons synapse mainly with local interneurons (Boeckh et al. 1989; Distler 1990). 3 The Formal Model
Neurons may be at rest ( x = 0) or above firing threshold ( x = 1). They are probabilistic neurons with memory: the probability P[Xi(t) = 11 that the state x i ( t ) of neuron i at time t is 1 is given by a sigmoid function of
Model of the Insect Olfactory Macroglomerulus
231
the neuron membrane potential q ( t ) at time t:
which is biased by a positive threshold 8j, and where T is a parameter, called temperature, which determines the amount of noise in the network (random fluctuations of the membrane potential). In discrete time, the fluctuation of the membrane potential around the resting potential, due to input e;(t) at its postsynaptic sites, is expressed as At * V j ( t - At) - .ej(t - At)
+ 7i
where 7; is the membrane time constant and At is the sampling interval, with e;(t) =
C[W - xj(t~- Q)]
w h m Wij is the weight of the synapse between neuron j and neuron i, and q is its delay. The weights are binary. The value of the transmission delay associated with each synapse is fixed but chosen randomly; it is meant to model all sources of delay, transduction, and deformation of the transmitted signal from the cell body or dendrodendritic terminal of neuron j to the receptor site of neuron i. The mean value of the delay distribution is longer for inhibition than for exatation: we thereby take into account approximately the fact that IPSCsusually have slower decay than EPSCs, and may accumulate to act later than actually applied. We consider three types of neurons: receptor, inhibitory and exatatory. Two types of receptor neurons (A and B ) are sensitive only to input A or B, where A and B represent two odor components. For all A (respectively B ) type receptor neurons, we have ei(t) = A(t), [respectively B(t)l, where A(t) is the concentration of component A. Receptor neurons may make axodendritic (YY = 0), exatatory synapses with both types of interneurons. Interneurons may make dendrodendritic synapses (Tij # 0 ) with any other intemeuron, but the connectivity c will be sparse.
4 Results
To analyze the behavior of such a network, we first introduce a classification of the possible response patterns of the neurons, which has been found useful for the analysis of olfactory response patterns (Meredith
232
C. Linster et al.
1986; Kauer 1974; Fonta et al. 1991). In the network under investigation,' which exhibits a typical distribution of response patterns, we observe three classes of patterns: purely excitatory, purely inhibitory, and mixed (both inhibitory and excitatory) responses. Excitation and inhibition are defined in relation to the neuron spontaneous activity. The mixed response patterns subdivide into three groups, according to the relative durations of the inhibition and excitation phases (Fig. 2A and 28). We analyze the behavior of the network in response to four characteristics of the input patterns (pure odors, A or B, and mixed odors, A and B), which are behaviorally important (see Kaissling and Kramer 1990): (1) amplitude, (2) stimulus shape, (3)frequency of stimulus presentation, and (4) ratio of the components in mixed odors. The behavior of the model network exhibits several characteristicsthat agree with biological data: selective neurons respond to only one of the two odor components and nonselective neurons respond to both components. The neurons exhibit a limited number of response patterns, most of them a combination of excitation and inhibition (Fig. 3A and 3B). The recognition of the concentration ratio of odor components is of behavioral importance, but it is not known whether the detection of a precise ratio is achieved at the level of the glomerulus or at higher olfactory centers. Here, we observe amplitude and temporal variations of the response patterns of individual interneurons as a function of the concentration ratio. Interneurons with oscillatory responses code, by temporal changes in their response patterns (Fig. 4A), for ratio variations of the input stimulation. In addition, pairs of neurons respond simultaneously to mixed input of a specific input ratio: in contrast, the first spikes of the responses to other ratios are separated by 25-50 msec; thus, the response latency could be one of the response parameters that indicate ratio detection (Fig. 5). The odor plume formed downwind from the calling female possesses a highly variable structure. Pulsed stimulation improves a male moth ability to orient toward an odor source (Baker et al. 1985; Kennedy 1983). We have therefore observed the behavior of the interneurons in response to pulsed stimulation. We find that some interneurons cannot follow pulsed stimulation beyond a specific cut-off frequency (Fig. 4B). The ability of these neurons to detect a certain frequency range depends on their response pattern; the cut-off frequency of each neuron depends on
'Fifty neurons; connectivity c = 10%(190 synapses); synaptic strength w,j = +l/-l; 30% receptor neurons, 30% excitatory intemeurons, 40%inhibitory intemeurons; sampling step At = 5 msec, which is enough to study the maximal physiological spiking frequencies (Christensen el al. 1989a); membrane time constant 7' - 25 msec; synaptic delays are chosen from a uniform distribution between 10 and for excitatory synapses, and between 10 and 100 msec for inhibitory synapses; the parameters of the sigmoids are T = 1 and 8 = 1 for receptor neurons, T = 0.375 and 8 = 1.5 for the others.
6-m
Model of the Insect Olfactory Macroglomerulus
233
20-40 At
!! 15-30 At
j Lnunu
2040 &
Rs:
1
R6:
15-30 At
I
Figure 2 (A) Response patterns: the amount of activation and inactivation is shown as a function of the stimulation (there At = 5 msec). R1, Activation for the duration of the stimulation. The spiking frequency varies as a function of the amplitude of the input. R2, The activation is followed by an inactive phase after the end of the stimulation. R3, Phasic burst, followed by an inactive phase of the same duration as the stimulus. R4, Phasic burst, followed by a tonic phase of diminished activation or by a phase of nonresponse, and by a short inactive phase after the end of the stimulation. R5, Phasic burst, followed by several phases of inactivation and activation (oscillatory response). R6, Inactivation during the application of the stimulation. The amplitude of the negative potential is a function of the amplitude of the stimulation.
234
C. Linster et al.
Figure 2 (B) Neurons responding with Rl-R6 (duration of each stimulation: 200 msec).
the duration of the stimulation and on the interstimulus interval. Neurons that respond with mixed exatation and inhibition show irregular responses and cannot follow high-frequency stimulation. Neurons that respond with exatation mostly respond continously to high-frequency stimulation. These behaviors depend mainly on the relations between the stimulation frequency, the interstimulus interval, and the temporal parameters of the model. Synaptic delays determine the behavior of mixed responses, while membrane time constants determine the behavior of excitatory responses. The stimulus profiles (rise and fall times of the odor signal) i n d i c a t e irrespective of the stimulus concentration-the distance between the lo-
Model of the Insect Olfactory Macroglomerulus
235
Figure 3 Responses of selective (A) and nonselective(B) neurons to stimulation with one and both odors (duration of each stimulation: 200 msec).
cation of odor perception and the odor source. We observe a number of interneurons that reflect the profile of the stimulation irrespective of its concentration. This depends again on the response patterns; neurons that exhibit purely excitatory responses reflect the input profile by response latency and response duration, whereas neurons that exhibit an oscillatory response have completely different temporal response patterns as a function of the input profile (Fig. 4C). In this section, we have shown that the response patterns of individual neurons reflect various characteristics of the input pattern. Selective
236
C. Linster et al.
neurons indicate the presence, amplitude, and stimulus profile of one component (depending on their response pattern); nonselective neurons indicate the presence, amplitude, and stimulus profile of the mixture of the two components. Some nonselective neurons also reflect the quality of the mixture, that is, the ratio of the components. 5 Influence of the Distribution of Neurons and Synapses
The number and the diversity of the response patterns depend on the total number of neurons, on the distribution of excitation and inhibition in the network, on the number of connections and feedback loops, and on the temporal parameters (i.e., synaptic delays, membrane time constants). The diversity of response patterns grows with the percentage of synapses in the network (all other parameters remaining unchanged). At connectivity c < 2%, afferent synapses cause purely excitatory responses (Rl); around c = 2%, simple mixed responses (R2) and inhibitory responses (R6) appear; at about c = 8%, the majority of the interneurons respond mainly with excitation (R1 and R2). The full diversity and distribution of response patterns described above are observed for most networks around c = 10%. With an increasing number of synapses, the number of response patterns decreases. Due to an increasing network activity, the response patterns tend to oscillate, and the network saturates. Similarly, increasing the inhibitory synapse number beyond 50% introduces oscillations; the total activity in the network decreases. Beyond 60% inhibition, only R3 responses (phasic burst followed by a long inhibitory period) survive. If there is too much excitation in the network (more than forty percent excitatory neurons or more than forty percent receptor neurons), the network becomes unstable and saturates.
Figure 4 Facing page. Responses of selective and nonselective neurons with varying response patterns to stimulation with varying input characteristics. (A) Stimulation with varying ratios of the input components,the sum of the amplitudes of the two components being constant. Several neurons respond with varying temporal response patterns to changing ratios (duration of each stimulation: 50 msec). (B)Neuron 7 responds with phasic bursts to stimulation at low frequencies,and responds continually to stimulation at the same frequency but with shorter interstimulus intervals, because the interstimulus interval approches the membrane constant of the neuron (upper diagram: stimulation duration 30 msec, interstimulus interval 20 msec; middle diagram: stimulation duration 40 msec, interstimulus interval 10 msec; bottom diagram: stimulation duration 20 msec, interstimulus interval 10 msec). (C) Stimulation by input with varying profiles, the rise and fall times vary from 10 msec to 50 msec (stimulationduration 100 msec).
Model of the Insect Olfactory Macroglomerulus
237
6 Discussion
In this section, we discuss the relevance of the results to the specialist system of insects. The model exhibits several behaviors that agree with biological data, and it allows us to state several predictive hypotheses about the processing of the pheromone blend. In the model, we observe two broad classes of interneurons: selective (to one odor component) and nonselective neurons. The fact that a distinct representation of pheromone components in parallel pathways
c.Linster et al.
238
Figure 5 Importance of response latenaes for ratio detection (stimulationduration 50 msec).
coming from the antenna is preserved by some antennal lobe neurons (local interneurons and projection neurons), but not all of them, has been reported in several species: in moths, Mmrducvr sexta (Christensen and 19831, and in the Hildebrand 1987a,b, 1989b) and Bombyx mMj (0cockmach, P e u a e (Boeckh 1976; Burrows etal. 1982; Boeckh and Selsam 1984; Hiisl1990). Selective and nonselective neurons exhibit a variety of mpnse patterns, which fall into three classes. inhibitory, excitatory, and mixed. Such a classification has indeed been pmlpposed for olfactory antennal lobe neurons (localinterneumns and pmjection neurons)in the specialist olfactory system in Mnduaa (Qlristensen et al. 1989a; Christensen and Hildebrand 1987a,b). Similar observations have been reported for Bombyx mon' (0% 1983)and for the cocboach (8urrows et d.1982; Boeckh and Ernst 1987). In our model we observe a number of local interneurons that cannot follow pulsed stimulationbeyond a neuroMIpeclficmt-off fmquency. This frequency depends on the neuron respanse pattern and on the duration of the interstimulusinterval. These d t s agree with data pertainins to antennal lobe neurons (intern-ns and prapction neurons) in Manduca serta (Christensen and Hildebrand 1988) and in Heliothis Oirescens (Christensen et al. 1989b). In both species, some antennal lobe neurons follow pulsed input with phasic bursts up to a cut-off fnquency. Physiological evidencein several species (ChxWemmand Hildebrand 198%; Burrows et al. 1982)has led to the hypothesis that some projection neurons (or local interneurons), may code for pheromone concentration and quality by measuring differences in response latency and duration, instantaneous spike frequency, and total number of spikes. Furthermore,
-
Model of the Insect Olfactory Macroglomerulus
239
the overall response to the correct blend of pheromones may be qualitatively different from the response to some other ratio of pheromones (Christensen and Hildebrand 198%). Our model exhibits characteristics (Figs. 4A and 5) that could substantiate these suggestions. They will be analyzed and discussed in more detail in a forthcoming publication. 7 Conclusion
We have presented an original model of olfactory information processing in the macroglomerulus of insects. This model incorporates very simple ingredients; its c o ~ e c t i v i t yis chosen randomly, from distributions which take into account complete, albeit approximate, biological knowledge. From these simple assumptions, a variety of neuronal responses emerge, some of them strongly resembling those observed in living systems. Our model performs feature extraction on the signal represented in separate input lines. A number of feahws concerning the single odor components as well as their blend are represented in parallel lines by the interneuron network. These results agree with the hypothesis that there are ‘,separate but parallel lines of olfactory information flow between the antenna1 lobe and the protocerebrum, each line carrying information about different aspects of a pheromonal stimulus” (Christensen et al. 1989a). The use of random c o ~ e d i ~ iand t y synaptic delays gives us a means to study the conditions under which such feature extraction can arise, and the diversity of output patterns that are thereby exhibited. Thus, a model built with random c o ~ e d i v i t ysuffices to explain, reproduce, and predict a number of signal processing properties on the olfactory specialist subsystem. The variation of random distribution parameters and delays gives insights into the means whereby natural neural nets may be modulated by higher control mechanisms, be they genetic, adaptive, or instructive.
Acknowledgments This work has been supported in part by EEC BRAIN contract ST2J-0416C and by Ministere de la Recherche et de la Technologie (Sciences de la Cognition).
References Baker, T. C., W i l l i s ,M. A., Haynes, K. F., and Phelan, P.L. 1985. A pulsed cloud of sex pheromones elicits upwind flight in male moths. Physiol. Entomol. 10, 257-265. Boeckh, J. 1976. Aspects of nervous coding of sensory quality in the olfactory pathway of insects. Proceedings of the XV International Congress of Entomology, Washington,19-27 August 1976.
240
C. Linster et al.
Boeckh, J., and Selsam, P.1984.Quantitative investigation of the odor specificity of central olfactory neurons in the American cockroach. Chem. Senses 9(4), 369-380. Boeckh, J., and Emst, K. D. 1987. Contribution of single unit analysis in insects to an understanding of olfactory function. I. Comp. Physiol. A161, 549-565. Boeckh, J.,Ernst, K. D., and Selsam, P. 1989. Double labelling reveals monosynaptic connections between antennal receptor cells and identified intemeurons of the deutocerebrum in the American cockroach. Zool. I&.Anut. 119, 303-312. Burrows, M., Boeckh, J., and Esslen, J. 1982. Physiological and morphological properties of interneurons in the deutocerebrum of male cockroaches which respond to female pheromone. I. Comp. Physiol. 145,447-457. Christensen, T. A., and Hildebrand, J. G. 1987a. Functions, organization, and physiology of the olfactory pathways in the lepidopteran brain. In Arthropod Bruin: Its Evolution, Development, StructureundFunctions, A. P. Gupta, ed. John Wiley, New York. Christensen, T. A., and Hildebrand, J. G. 1987b. Male-specific, sex pheromoneselective projection neurons in the antennal lobes of the moth Munducu sextu. I. Comp. Physiol. A 160, 553-569. Christensen, T. A., and Hildebrand, J. G. 1988. Frequency coding by central olfactory neurons in the sphinx moth Manducu sextu. Chem. Senses 13(1), 123-130. Christensen, T. A., Hildebrand, J. G., and Tomlinson, J. H. 1989a. Sex pheromone blend of Munducu sextu: Responses of central olfactory interneurons to antennal stimulation in male moths. Arch. Insect Biochem. Physiol. 10, 281-291. Christensen, T.A., Mustaparta, H., and Hildebrand, J. G. 1989b. Discrimination of sex pheromone blends in the olfactory system of the moth. Chem. Senses 14(3),463-477. Distler, P. 1990. GABA-immunohistochemistryas a label for identifying types of local intemeurons and their synaptic contacts in the antennal lobe of the American cockroach. Histochemistry 93,617-626. Emst, K. D., and Boeckh, J. 1983. A neuroanatomical study on the organization of the central antennal pathways in insects. Cell Tissue Res. 229, 1-22. Fonta, C.,Sun, X. J.,and Masson, C. 1991. Cellular analysis of odour integration in the honeybee antennal lobe. In The Behuviour and Physiology of Bees, L. J. Goodman and R. C. Fischer, eds., pp. 227-241. C.A.B. International, London. Haberly, L. B., and Bower, J. M. 1989.Olfactory cortex: Model circuit for study of associative memory? TINS 12(7), 133. Hod, M. 1990. Pheromone-sensitive neurons in the deutocerebrum of Periplunetu umericunu: Receptive fields on the antenna. I. Comp. Physiol. A 167,321-327. Kaissling, K-E., and Kramer, E. 1990. Sensory basis of pheromone-mediated orientation in moths. Verh. Dtsch. Zoolo. Ges. 83, 109-131. Kauer, J, S. 1974. Response patterns of amphibian olfactory bulb neurons to odor stimulation. I. Physiol. 243, 695-715. Kennedy, J. S. 1983. Zigzagging and casting as programmed response to windborne odor: A review. Physiol. Entomol. 8, 109-112.
Model of the Insect Olfactory Macroglomerulus
241
Li, Z., and Hopfield, J. J. 1989. Modeling the olfactory bulb and its neural oscillatory processings. Biol. Cybernet. 61,379-392. Lynch, G., and Granger, R. 1989. Simulation and analysis of a simple cortical network. In Computational Models of Learning in Simple Neural Systems, R. D. Hawkins and G. H. Bower, eds., pp. 205-238. Academic Press, New York. Lynch, G., Granger, R., and Larson, J. 1989. Some possible functions of simple cortical networks suggested by computer modeling. In Neural Models of Plasticity, J. H. Byrne and W. 0. Berry, eds., pp. 329-361. Academic Press, New York. Masson, C., and Mustaparta, H. 1990. Chemical information processing in the olfactory system of insects. Physiol. Rev. 70(1), 199-245. Meredith, M. 0.1986.Patterned response to odor in mammalian olfactory bulb: The influence of intensity. J. Neurophysiol. 56(3), 572-597. Olberg, R. M. 1983. Interneurons sensitive to female pheromone in the deutocerebrum of the male silkworm moth, Bombyx mori. Physiol. Entomol. 8, 419-428. Rall, W., and Shepherd, G. M. 1968. Theoreticalreconstructionof field potentials and dendrodendritic synapse interactions in olfactory bulb. J. Neurophysiol. 31,884-915. Wilson, M.A., and Bower, J. M. 1988. A computer simulation of olfactory cortex with functional implications for storage and retrieval of olfactory information. In Neural Information Processing Systems, D. Z . Anderson, ed., pp. 114-126. American Institute of Physics, New York. Wilson, M. A., and Bower, J. M. 1989. The simulation of large scale neural networks. In Methods in Neuronal Modelling: From Synapses to Networks, C . Koch and I. Segev, eds., pp. 291-334. MIT Press, Cambridge, MA. Received 6 December 1991; accepted 3 August 1992.
This article has been cited by:
Communicated by Fernando Pineda
Learning Competition and Cooperation Sungzoon Cho James A. Reggia Department of Computer Science, University of Maryland, College Park, MD 20742 USA
Competitive activation mechanisms introduce competitive or inhibitory interactions between units through functional mechanisms instead of inhibitory connections. A unit receives input from another unit proportional to its own activation as well as to that of the sending unit and the connection strength between the two. This, plus the finite output from any unit, induces competition among units that receive activation from the same unit. Here we present a backpropagation learning rule for use with competitive activation mechanisms and show empirically how this learning rule successfully trains networks to perform an exclusive-OR task and a diagnosis task. In particular, networks trained by this learning rule are found to outperform standard backpropagation networks with novel patterns in the diagnosis problem. The ability of competitive networks to bring about context-sensitive competition and cooperation among a set of units proved to be crucial in diagnosing multiple disorders. 1 Introduction
Competitiveactivation mechanisms have recently been proposed as a method for producing competitive or inhibitory interactions between units (nodes) through functional mechanisms instead of inhibitory connections (Reggia 1985). A unit sends output to another unit proportional to the receiving unit's activation as well as the connectionweight between them. Since the total output from a unit is finite, competition arises among the receiving units; a stronger receiving unit with a higher activation level gets more input, indirectly reducing the input to weaker (lower activation) units, often until a clear winnerb) emerges. This process has the same effect as inhibiting other competing units without having direct inhibitory connections. This approach brings about more flexible and context-sensitive information processing than more traditional methods: a set of units can ' A similar idea was independently described in Rumelhart and McClelland (1986).
Neural Computation 5,242-259 (1993) @ 1993 Massachusetts Institute of Technology
Learning Competition and Cooperation
243
compete or cooperate depending on the context (this will be elaborated on later). Applications including print-to-sound transformation, diagnosis, satellite communication scheduling, and models of cerebral cortex have demonstrated that competitive activation mechanisms can work successfully, in many cases with substantially fewer connections than standard approaches (Reggia et al. 1988; Bourret et al. 1989; Peng and Reggia 1990; Reggia et al. 1992; Cho and Reggia 1992). Existing learning methods, however, cannot be used with competitive activation mechanisms because even in feedforward networks there is an implicit recurrent flow of information (to guide the competitive process). This lack of learning methods has greatly limited the usability of competitive activation mechanisms in the past. Thus, we have derived an error backpropagation (EBP) learning rule for networks employing these mechanisms (Cho and Reggia 1991). This new learning rule, referred to as competitive EBP, can be applied to networks with an arbitrary connection structure, but we restrict our attention to simple architectures where one is interested in the final state of a competition, not its process? In other words, our learning rule could be described as training networks to learn a set of fixed points of a dynamic system, unlike some recurrent learning rules that are concerned with a set of trajectories. This paper describes the competitive EBP learning rule and two applications of it where context-dependent competition and cooperation among units play key roles. The first application involves training networks with three units to learn to perform an exclusive-OR (XOR) operation on their inputs. This application illustrates clearly how competition and cooperation among units can function effectively for this simple, linearly nonseparable problem. A second, diagnostic application involves localization of damage in the human central nervous system given the hdings on a patient’s neurological examination. Networks are initially trained to identify the location of single disorders (sites of damage) given a set of manifestations associated with each disorder. Then previously unseen sets of manifestations are presented to the trained networks. It is found that the networks trained with competitive EBP produce better diagnostic hypotheses than networks trained with a standard backpropagation learning rule when multiple disorders are present simultaneously. Solving such multimembership diagnosticproblems is widely recognized to be a difficult task (Peng and Reggia 1990). The following section briefly describes competitive activation mechanisms and the competitive EBP rule in an informal manner [refer to Cho and Reggia (1991) for more details]. Simulation results involving the two applications described above are then presented to demonstrate that this approach can work effectively.
20ur learning rule has also proven to be effective in learning continuous-valued functions (Cho and Reggia 1992).
Sungzoon Cho and James A. Reggia
244
2 Activation Mechanisms and Learning Rule
Given an arbitrarily connected network, let the activation level of unit k at time t, ak(t), be given as
where output O U t k j ( t ) from unit j to unit k is distributed competitively by r
1
(2.2) LIEN
J
The function f k denotes any differentiable activation function. The weight on the connection from unit k to unit j is denoted by wkj, which is assumed to be zero when there is no connection between the two units. The term E k denotes a constant external input to unit k. The network-wide constant parameters a, P, and y represent decay, input gain, and output gain, respectively. Values of Q and p control how fast the activation decays while that of y determines how much output a unit sends in terms of its activation level. The parameter p determines how much competition exists among the units. The larger the value of p, the more competitive the model's behavior. The output O U t k j ( f ) is proportional not only to the sender's activation level aj(t), but also to the receiver's activation level, &(t). Therefore, a stronger unit receives more activation. Another unit I, which also gets input from unit j, can be seen as competing against unit k for the output from unit j because the normalizing factor ClEN w p ; ( f )in the denominator of equation 2.2 constrains the sum of the outputs from unit j to be equal to its activation level, u,(t), when y = 1. The activation sent to unit k, therefore, depends not only on those of the units sending it activation such as unit j , but also on those of its competitors to which unit k has no explicit connections. This has the effect of introducing implicit "feedback (recurrent effects) into networks in locations where explicit connections do not exist. This is why conventional backpropagation and other learning methods are not directly applicable in models where equation 2.2 controls the spread of activation. Applying gradient descent to the usual sum-of-square error measure (Rumelhart et al. 19861, we derived the following weight updating rule (2.3)
where B, = ElEN wlSu;.Since is zero except for those units I that have connections from unit s (i.e., wjs # 0), the summation is actually over only
Learning Competition and Cooperation
245
those units to which unit s sends its activation. Therefore, learning can be carried out by a local computation. The bk values in equation 2.3 are the fixed point solutions of the dynamic system
(2.4) where Ak denotes the difference between the desired and the actual values at output unit k (zero at other units). Equation 2.4 appears to be complex but, in fact, is quite simple: all factors other than the &s in equation 2.4 are constants since this backward dynamic system is run after the forward system (equations 2.1-2.2) reaches equilibrium and the p becomes error signals at the output units have been c o m p ~ t e d When .~ zero (no competition), equation 2.4 reduces to the same backward dynamic system as derived in recurrent backpropagation (Almeida 1987; Pineda 1987) except for normalized weights. Equation 2.4 can be interpreted intuitively as follows. The first term, CjfN d j k b j ( t), represents ak's partial responsibility for the error signal at to unit j . This term also exists in a unit j since it influenced signal standard EBP learning rule (Rumelhart ef al. 1986) and in a recurrent EBP learning rule for noncompetitive activation mechanisms (Pineda 1987). The second term akAk is the error signal arising directly from external teaching signals and is nonzero only for output units. The third term, PCmeNoUth{bk(t) - (I/?) C I € N O U t f r n [ ~ f ( t ) / a r n ] } ,occurs only in Our cornpetitive EBP learning rule for competitive activation mechanisms. The quantity p CmEN o U t h b k ( f ) can be interpreted as accounting for the fact that unit k is indirectly responsible for its own error signal through its influence on the incoming activation signal outh from unit rn (the term outh has factor a k in it). However, unit k is not solely responsible for that error since the activation level of unit k itself is indirectly influenced by those of its competitors, that is, the other units to which unit m sends its output signals. The sum (I/?) C l E N o u t ~ , [ 6 / ( ( t ) / a m ] can be viewed as the amount that needs to be subtracted from unit k to compensate for this indirect influence by competitors. This term was derived from the denominator term B, of o u t h . Learning proceeds as follows. A training pattern is presented as input to the network, the forward dynamic system (equations 2.1-2.2) is run until it reaches equilibrium, the error signal A for each output unit is cal~~~
~~~
3Benaimand Samuelides proved convergence of feedforward networks employing a slightly different competitive activation mechanism (Benaim and Samuelides 1990). Empirically, we encountered neither oscillations nor divergence.
246
Sungzoon Cho and James A. Reggia
culated, the backward dynamic system (equation 2.4) is run to compute the error signals for all the units in the network, and then the weight on each connection is changed according to the learning rule (equation 2.3). These steps are taken for all training patterns and this is said to comprise one epoch. Learning continues until either all training patterns are correctly learned or a preset time limit of 400 epochs expires. Simulations were conducted using Maryland/MIRRORS 11, a simulation software environment that constructs and runs a neural network as specified by the user.
3 Exclusive-OR (XOR)
Exclusive-OR (XOR) has been a popular problem to test learning algorithms due to the historic fact that one-layer linear models and elementary perceptrons are unable to learn the task. Because of its linear nonseparability property, a hidden layer of units is required to learn the task with backpropagation algorithm (Rumelhart et al. 1986). Here we use a competitive activation mechanism and our competitive EBP learning rule to train a network with two input units and an output unit to learn XOR.4 Competition and cooperation among a set of units were found to play a crucial role in performing the task. Figure 1 depicts the three-unit network used in the simulation. Units 1 and 2 represent input units and unit 3 represents the output unit. A sigmoid function, f ( x ) = 1/(1+ e-s(x-0.5)),was chosen as f k ( x ) in equation 2.1 for all k. With larger values of s, the sigmoid function becomes steeper, producing clearer winners and losers during a simulation. However, large values of s also cause the derivatives of the activation functions to vanish, making error signals much less significant. Such "flat spots" have been recognized previously and various cures suggested (Fahlman 1988; Hinton 1989; Cho and Reggia 1991). Here we start with a small s value of 1 and gradually increase it to 10 (this process is not simulated annealing). All of the parameters a,P, and y and competition parameter p were set to 1 and fixed throughout learning. Four input patterns (l,O), (O,l), (l,l),and (0,O)were used with the corresponding target outputs l,l,O, and 0, respectively. A total of 45 different nonsolution initial weight states were used, 32 cases of which were randomly generated and the remaining 13 cases of which were handcrafted so that the initial weight state was located far away from a solution weight state. In all 45 cases, the competitive EBP learning rule successfully changed the initial nonsolution state into a solution state. The mean, median, minimum, and maximum number 40Ur input units are doing more than what ordinary input units do, but since they receive direct external inputs, they are not hidden units.
Learning Competition and Cooperation
247
I
I
El
E2
Figure 1: The three-unit network trained to learn XOR. Weight wi, represents the weight from unit j to unit i. External inputs E l and E2 are fed to input unit 1 and 2, respectively. The activation level at unit 3 at equilibrium is considered to be the output of the network.
of learning epochs taken were 145, 44, 33, and 840, respectively. The networks with handcrafted initial weights were found to take a much longer time than the networks with randomly generated initial weights as expected. All of the final solution states obtained share the property that the connections from input to output units are stronger than those between input units (i.e., wZ1< w31 and wI2< w32). How such sets of weights perform XOR with a competitive activation mechanism is as follows. When the pattern (1,O) is applied to the input units, input unit 2 and output unit 3 compete for the output from input unit 1. Since ~ 3 >1 W Z I ,unit 1 sends a larger amount of activation to unit 3 than unit 2. With a larger input from unit 1, the activation level of unit 3 surpasses that of unit 2. Then, with the help of a slightly larger activation level, unit 3 gets an even larger amount of input from unit 1than does unit 2 since the input amount to unit 3 is proportional not only to the connection weight but also to the activation level of unit 3. This process of increasing the activation level of unit 3 accelerates while unit 2 remains close to where it started, that is, near zero. The network finally reaches equilibrium with unit 3 as the winner and unit 2 as the loser, which means that the activation levels of unit 1 and unit 3 are close to one and that of unit 2 is close to zero. This is exactly what is desired. By symmetry, unit 3 wins with the input pattern (0,l) since ~ 3 >2 w12. With the pattern (l,l),initially two competitions occur at the same time: between unit 1 and unit 3 for
248
Sungzoon Cho and James A. Reggia
unit 2’s activation, and between unit 2 and unit 3 for unit 1’s activation. With both unit 1 and unit 2 receiving external inputs and feeding each other (cooperation),they both win over unit 3, overcoming the fact that w 3 1 > w 2 1 and ~ 3 > 2 ~ 1 2 . 5 Finally with pattern (O,O), unit 3 does not get any significant input from either of the input units so it stays at the initial activation level which is zero. Through competition and cooperation among units, a competitive activation mechanism implements XOR task. 4 Diagnostic Associations
Recently, backpropagation models have been applied to diagnostic problem solving with some success. However, these backpropagation models for diagnosis apply to small, circumscribed decision problems where it is usually assumed that at most a single disorder is present. For example, the system for diagnosing myocardial infarction was limited to determining the presence or absence of that single disorder (Baxt 1990). The previous backpropagation models thus typically perform a pattern classification task (selection of one diagnostic category out of several when given a problem description), and do not in any sense solve general diagnostic problems where one must construct a multiple disorder solution from individual disorders. This is a major limitation because in many diagnostic problems multiple disorders may be present simultaneously (referred to here as multiple disorder patterns) (Peng and Reggia 1990). Further, a diagnostician is often presented with manifestations that are a proper subset of the manifestations associated with a disorder (referred to here as partial manifestation patterns). What would be desirable is to train a network with “textbook examples” consisting of all manifestations associated with each individual disorder (referred to here as prototypical patterns), yet for the trained network to produce reasonably good diagnosis with nonprototypical multiple disorder patterns and partial manifestation patterns. Connectionist models based on backpropagation have not been demonstrated previously to perform well for multiple disorder problems. Our hypothesis was that networks trained by competitive backprop agation learning rule would produce better diagnoses given multiple disorder patterns and partial manifestation patterns since they do not employ inhibitory connections. To assess this, we trained networks using both standard EBP (referred to as standard networks) and competitive EBP (referred to as competitive networks), and then compared their respective performance with previously unseen multiple disorder patterns and partial manifestation patterns.
’This process does not work when w31 >> wZ1,and w32 B w12.
Learning Competition and Cooperation
249
4.1 The Localization Task and Training Networks. The medical information used for training is based on standard reference sources and personal knowledge of one of the authors OR) (Adams and Victor 1985; Brazis et al. 1985). We selected 29 different manifestations (Table 1) and 16 localizations of brain damage, so 16 associations between a disorder and a set of prototypical manifestations had to be learned (Table 2). Each disorder represents damage to a specific part of the brainstem, cerebellar hemispheres, or cerebral hemispheres. With each ,manifestation and disorder assigned a numbered index, these associations can be viewed as 16 binary 1 / 0 patterns (Table 3). Presence of a 1 at the ith row and jth column indicates a causal relation between the ith manifestation and the jth disorder. For all 16 patterns, during training only one output unit was to be turned on, representing the presence of the disorder whose prototypical manifestations were present. These prototypical patterns, created as described above, were found to be linearly separable since the 16 input patterns are linearly independent when viewed as a binary vector (Hertz et al. 1991). Because these prototypical patterns were linearly separable, we used networks with two layers of units, namely, input and output units only. However, this is not an easy problem to solve: we are not concerned here with just classdying input patterns into one of n categories. We are interested in the far more difficult task of identdymg a set of categories when a network has only been trained with singledisorder exemplars. Thus, for example, with diagnosis involving n disorders the cardinality of the output space is 2", not n. This kind of multimembership problem is widely recognized to
Table 1: Manifestations (Input Units) Nos. 1, 2
3,4 5, 6 7, 8 9, 10 11, 12 13, 14 15, 16
17 18, 19 20,21
22,23 24'25 26,27 28'29
Manifestations Left, right hemiparesis Left, right facial paresis Left, right tongue paresis Left, right gaze palsy (conjugate) Left, right intemuclear opthalmoplegia Left, right 3rd nerve palsy Left, right 6th nerve palsy Left, right Horner's syndrome Nystagmus Left, right hemiataxia Left, right touch-proprioceptionimpairment Left, right pain-temperature impairment Left, right facial sensory impairment Left, right hemianopsia Sensory, motor aphasia
Sungzoon Cho and James A. Reggia
250
Table 2 Disorders (outputunits) and Their Corresponding Manifestations No. Disorder
Left medial medulla Right medial medulla Left lateral medulla Right lateral medulla Left medial pons Right medial pons Left lateral pons Right lateral pons Left midbrain Right midbrain Left cerebellum Right cerebellum Left frontal lobe 14 Right frontal lobe 15 Left pareitotemporal lobe 16 Right pareitotemporal lobe
1 2 3 4 5 6 7 8 9 10 11 12 13
Manifestations (see Table 1) 2, 5, 21 1, 6, 20 15, 17, 18, 20, 23, 24 16, 17, 19, 21, 22, 25 2, 4, 6, 7, 9, 13, 17, 18, 21 1, 3, 5, 8, 10, 14, 17, 19, 20 3, 7, 15, 17, 18, 21, 23, 24 4, 8, 16, 17, 19, 20, 22, 25 2, 4, 6, 11, 17, 18, 21, 23 1, 3, 5, 12, 17, 19, 20, 22 17, I8 17,19 2, 4, 6, 8, 29 1, 3,5, 7 21, 23, 25,27, 28 20,22, 24, 26
be a very challenging problem in statistical pattern recognition and diagnostic problem solving (Peng and Reggia 1990). There is no generally recognized ideal solution to such problems at present, and any approach (neural network or otherwise) that can approximate solutions is worth investigation. The standard network we used is shown in Figure 2a and the competitive network is shown in Figure 2b. The standard network used a single bias unit that was always on. The multiple "bias units" in the competitive network were an experiment: they were introduced originally to develop as feature detectors. In other words, they were expected to selectively turn on for certain features in the input manifestation patterns. 4.2 Learning Prototypical Patterns (Single-Disorder Patterns). Both standard and competitive networks learned the prototypical patterns rather easily! First, we successfully trained standard networks with zero, four, and eight hidden units in around 90, 300, and 210 epochs, respectively. For each input pattern of a set of manifestations, the correct 6Parameters a, p, 7, and p in equations 2.1 and 2.2 were set to 1.0. Parameter s increased linearly from 3 to 8 over 200 epochs and then remained fixed. Constant 0.1 was added to the derivative of the activation function to accelerate learning (Fahlman 1988). For both forward and backward dynamic systems equilibrium was assumed if every dynamic variable changed less than 0.01.
Learning Competition and Cooperation
251
Table 3: Manifestation-Disorder Associations (I/O patterns) Manif.
No.
1
1
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
1 1
1
Disorder No. 2 3 4 5 6 7 8 9 10 11 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
12 13 14 15 16 1 1 1 1 1 1 1
1
1 1 1
1 1 1 1 1 1 1 1 1
disorder unit became strongly activated while others did not. Since the standard networks with no hidden units gave best generalization performance, we describe results only from the network with no hidden units (shown in Fig. 2a). We also trained 35 different competitive networks. Five different numbers of bias units were used, ranging from zero to four. For each case, seven different competitive networks were generated by randomly assigning initial weights. Any competitive network with at least one bias unit learned the training patterns successfully in between 21 and 45 epochs, with an average of 36 epochs? The bias units did not learn to become feature detectors, but functioned in a fashion equivalent to a single bias unit that always came on. Furthermore, the competitive networks with exactly one bias unit produced the best diagnosis for multiple disorder patterns and partial manifestation patterns. Thus, we 'Although competitive networks took fewer epochs in training than standard networks, each epoch with a Competitive network takes more real time due to the recurrent nature of competitive activation mechanisms.
Sungzoon Cho and James A. Reggia
252
b.
a.
OUTPUT
biaa
OUTPUT
I . . . ]
'1 000
... INPUT
0
000
...
0
INPUT
Figure 2: (a) Standard network and (b) competitive network. The standard network shown in (a) consists of 29 input units representing the manifestations listed in Table 1, 16 output units representing the disorders listed in Table 2, and a bias unit. Each input unit and the bias unit is connected to every output unit. Standard networks with a hidden layer of four and eight units were also trained, but did not give a better performance in either training or testing and so are not considered furthermore. Initially, weights were randomly chosen from a uniform distribution in the range of [-0.3,0.3]. The competitive network shown in (b)has the same set of input and output units and connections as the standard network, except that it has a set of zero to four bias units, receiving connections from input units (see text for explanation). A total of 35 competitive networks (seven for each number of bias units) were randomly generated with initial weights picked from a uniform distribution in the range of [0,1]. Input units are fully connected to bias units, which are in turn fully connected to output units. Both incoming and outgoing connections of the bias units were updated according to the learning rule.
present only results from one of the competitive networks with one bias unit. The weight values of the trained standard network with no hidden unit (but one bias unit) and those of the trained competitive network with one bias unit are shown in Figure 3a and b, respectively. Note that both networks developed a similar but not identical pattern of excitatory connections, but the competitive network does not have any inhibitory connections. If the trained networks were pruned, that is, if connections with very small valued weights were removed from the networks, the competitive network would become much smaller than the standard network. As we explain below, examination of these weight values provides insight as to why competitive networks produce a better diagnosis than standard networks with multiple disorder patterns and partial manifestation patterns.
Learning Competition and Cooperation
253
a.
b.
Figure 3: Connection weights of trained (a) standard and (b) competitive networks. Each box represents a connection from an input to an output unit. The size of a circle in the ith row and jth column is proportional to the size of the connection weight from input unit j - 1 to output unit i with the bias unit denoted as unit 0 (leftmost column). The filled circles represent positive weight values and the open circles represent negative weight values. Note that the trained standard network has a large number of inhibitory weights, and that the bias unit in the standard network inhibits output nodes while that in the competitive network excites them.
4.3 Testing Multiple Disorder Patterns. For multiple disorder patterns, the combined manifestations of two disorders were presented simultaneously to the trained networks. A total of 48 multiple disorder
254
Sungzoon
Cho and James A. Reggia
input pattern pairs were selected for testing (with Pi denoting the pattern of input manifestations associated with disorder i, PI, plus each of the preceding 15 patterns, PI, plus each of the preceding 13 patterns, PI, plus each of the preceding 11 patterns, and PI, plus each of the preceding 9 patterns were presented). It turns out that 4 of these 48 multiple disorder patterns are the same as one of the two combined original input patterns, so there are only 44 truly multiple disorder patterns. The trained networks would be performing ideally if they produced two clear winning disorder units corresponding to those disorders whose manifestations were presented together.8 The standard network activated the two correspondingdisorder units in only 16 cases, and only one disorder unit in the remaining 28 cases. In addition, most of the winners in a total of 44 cases were not fully activated (see Table 4). This weak activation in the output layer of the standard network, representing the failure to generate two clear correct winners, is attributed to the large number of inhibitory connections that are present (see negative weights in Fig. 3a). When manifestations are turned on that are associated with more than one disorder, the corresponding disorder units receive strong inhibitory input as well as excitatory input. Consider, for instance, the case of disorders 1 and 16. Disorder 1 is associated with manifestations 2, 5, and 21 while disorder 16 is associated with manifestations 20, 22, 24, and 26. When all seven of these manifestations were presented, however, neither disorder 1 nor disorder 16 turned on strongly (first row of Table 4). This is because disorder 1 has inhibitory connections from manifestations 20,22,24, and 26, all of which are associated with disorder 16 (see Fig. 3a), thus sending inhibition to disorder 1. Similarly, manifestations 2, 5, and 21 which are associated with disorder 1 send inhibitory activations to disorder 16, thus resulting in a weak activation for disorder 16 (see Fig. 3a). Manifestation units not only send excitatory signal to the associated disorder units, but also send inhibitory signal to the disorder units that are not associated with them. This prevents the standard network from producing clear-cut multiple winners when multiple disorders are present. The competitive network, on the other hand, turned on both of the two associated disorder units very strongly in 38 out of the 44 cases, clearly outperforming the standard network (Table 5). Use of only excitatory connections coupled with the competitive activation mechanism enabled the network to produce multiple winners in the disorder layer
*The actual situation is more complex than this as it is possible that the union of manifestations for two disorders could correspond to the manifestations of another single disorder.
Learning Competition and Cooperation
255
Table 4 Multiple-Disorder Testing with Standard NetworkQ Disorder
No.
1
116 2 16 3 16 4 16 5 16 6 16 7 16 8 16 9 16 10 16 11 16 12 16 13 16 14 16 15 16 114 2 14 3 14 4 14 5 14 6 14 7 14 8 14 9 14
.5
2
3
4
Activation Level at Each Output Unit 5 6 7 8 9 10 11 12 13
2 12 3 12 5 12 7 12 9 12 11 12 1 10 2 10 3 10 4 10 5 10
6 10 7 10 8 10 9 10
15
.4
.5
16 .6 .7 .3
.4
.6
.8 .4
.8
.5 .8 .6 .4
.7
.5 .5 .5 .5 .5
.6 .6
.4
.3 .2
.5 .6
.7 .5 .5 .4
.8
10 14 11 14 12 14 13 14
112
14
.3
.6 .6
.2
.6
.4
.2 .2
.6
.5 .9
.9 .9
.9
.5
.5
.7 .9 .6 .6
.5
.2
.8 .6
.5
'Entries less than 0.2 not shown.
when the two winners are necessary to account for the input manifestations (see Fig. 3b). Although there are some additional disorder units
Sungzoon Cho and JamesA. Reggia
256
Table 5 Multiple-Disorder Testing with Competitive Networka Disorder
No. 116 2 16 3 16 4 16 5 16 6 16 7 16 8 16 9 16 10 16 11 16 12 16 13 16 14 16 15 16 114 2 14 3 14 4 14 5 14 6 14 7 14 8 14 9 14 10 14 11 14 12 14 13 14 112 2 12 3 12 5 12 7 12 9 12 11 12 110 2 10 3 10 4 10 5 10 6 10 7 10 8 10 9 10
1 1
1
1
Activation Level at Each Output Unit 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
#Entriesless than 0.2not shown.
turned on, those "nonperfect" diagnoses are not necessarily undesirable in a clinical sense. For instance, when P,and PI,are presented, not only
Learning Competition and Cooperation
257
disorders 3 and 14, but also disorder 7 is activated. This is a reasonable diagnosis given the fact that seven out of eight manifestations associated with disorder 7 are present (Peng and Reggia 1990). 4.4 Testing Partial Manifestation Patterns. A total of 29 partial manifestation patterns, each involving only one input unit (manifestation) being activated, were presented to the standard network as well as to the competitive network. The majority (19) of manifestations are associated with more than one disorder while the rest (10) of them are associated with only one disorder (manifestation 9 through 14 and 26 through 29). In the majority of cases, both the standard and the competitive network activated multiple disorder units partially, producing no clear winners but a set of alternatives. These results are similar to a "differential diagnosis'' that a human diagnostician might give. It is with the remaining 10 cases where the input manifestation can be caused by only a single disorder that the two types of networks performed differently. The standard network did not produce a clear winner in any of these 10 cases; most of the disorder units were activated rather weakly. Input from a single manifestation unit was not enough to fully activate the disorder units? The competitive network, however, turned on the respective corresponding disorder units as clear winners since it is the relative amount of input to the disorder units, not the absolute amount, that determines the result of competition. Note that even with a small difference in the input, competitive networks can produce clear winners and losers.
5 Conclusion
Competitive activation mechanisms are an alternative way of controlling the spread of activation in neural networks. They have been shown to be effective in some applications, but until now lacked an effective supervised learning method. We therefore have derived a backpropagation learning rule for use with competitive activation mechanisms and have described it in this paper. This learning rule can be viewed as a generalization of a previous form of recurrent backpropagation with normalized weights (Pineda 1987). To demonstrate that our new learning rule can work effectively, we first applied it to the task of exclusive-OR. Competitive networks with three units were successfully trained to perform this operation on their 9This was studied further by independently training and testing standard networks with normalized input patterns. Such networks produced strong winners in the corresponding single disorder units. However, when the single manifestation presented was associated with more than one disorder, the standard networks trained with normalized patterns produced strong winners in all the associated disorder units, which is an unacceptable result. Using normalized input patterns also produced poor results with multiple-disorder patterns.
Sungzoon Cho and James A. Reggia
258
inputs. Having hidden units was unnecessary because the two input units and one output unit could compete to produce correct behavior by the network. The second application involved locating areas of brain damage given manifestations (neurological signs and symptoms) as input. It should be noted that a diagnosis problem like this is not just a pattern classification problem where the given set of manifestations is classified into one of several categories. Rather, it also involves constructing a hypothesis that includes more than one disorder. Standard backpropagation networks have not been demonstrated to handle multiple-disorder diagnosis problems effectively. The standard backpropagation network we studied developed many inhibitory connections when trained with just prototypical cases. When manifestations of multiple disorders were subsequently presented simultaneously, these inhibitory connections prevented the network from producing appropriate diagnostic hypotheses. Competitive networks, on the other hand, performed qualitatively better in such cases. In competitive networks, the disorder units were able to "cooperate" in producing multiple winners when appropriate even though they were only trained on prototypical single-disorder cases. Competitive networks also did better with partial manifestation patterns than standard networks. In summary, we have derived and demonstrated the effectiveness of a supervised learning rule for competitive activation mechanisms. Together with an unsupervised learning rule previously developed for competitive activation mechanisms (Sutton et al. 19901, this greatly increases the range of tasks to which this approach can be applied. Acknowledgments
This work was supported by N M awards NS29414 and NS16332. Dr.Reggia is also with the Institute for Advanced Computer Studies at the University of Maryland. References Adam, R., and Victor, M. 1985. Principles of Neurology. McGraw-Hill, New
York. Almeida, L. 1987. A learning rule for asynchronous perceptrons with feedback in a combinatorial environment. Proceedings of the IEEE First Annual International Conference on Neural Nehoorks, Vol. 11, pp. 609418,San Diego, CA. Baxt, W. 1990. Use of an artificial neural network for data analysis in clinical decision making. Neural Camp. 2,480-489. Benaim, M., and Samuelides, M. 1990. Dynamical properties of neural nets using competitive activation mechanisms. Proceedings of lnternational Joint Conference on Neural Networks, Vol. III, pp. 541546, San Diego, CA.
Learning Competition and Cooperation
259
Bourret, P., Goodall, S., and Samuelides, M. 1989. Optimal scheduling by competitive activation: Application to the satellite antennae scheduling problem. Proceedings of International Joint Conferenceon Neural Networks, Vol. I , pp. 565572, Washington, DC. Brazis, I?, Masdeu, J., and Biller, J. 1985. Localization in Clinical Neurology. Little, Brown, Boston. Cho, S., and Reggia, J. 1991. A recurrent error back-propagation rule for competitive activation mechanisms. Tech. Rep. CSTR-2661, Department of Computer Science, University of Maryland. Cho, S., and Reggia, J. 1992. Learning visual coordinate transformations with competition. Proceedings of International Joint Conference on Neural Networks, Baltimore, MD, Vol. IV,pp. 49-54. Fahlman, S. 1988. Faster-learningvariations on back-propagation: An empirical study. Proceedings of the 1988 Connectionist Models Summer School, pp. ,3845, Pittsburgh, PA. Hertz, J., Krogh, A., and Palmer, R. 1991. Introduction to the Theory of Neural Computation. Addison-Wesley, Redwood City, CA. Hinton, G. 1989. Connectionist learning procedures. Artificial Intelligence 40, 185-234. Peng, Y.,and Reggia, J. 1990. Abductive Inference Models for Diagnostic ProblemSolving. Springer-Verlag,Berlin. Pineda, F. 1987. Generalization of back-propagation to recurrent neural networks. Phys. Rev.Lett. 59(19), 2229-2232. Reggia, J. 1985. Virtual lateral inhibition in parallel activation models of associative memory. Proceedings of the 9th International Joint Conferenceon Artificial Intelligence, Vol. 1, pp. 244-248, Los Angeles, CA. Reggia, J., D’Autrechy, C. L., Sutton, G., and Weinrich, M. 1992. A competitive distribution theory of neocortical dynamics. Neural Comp. 4, 287-317. Reggia, J., Marsland, P., and Berndt, R. 1988. Competitive dynamics in a dualroute connectionist model of print-to-sound transformation. Complex Syst. 2, 509-547. Rumelhart, D., Hinton, J., and Williams, R. 1986. Learning representations by back-propagating errors. Nature (London) 323,533-536. Rumelhart, D., and McClelland, J. 1986. On learning the past tenses of English verbs. In Parallel Distributed Processing, Vol 2: Foundations, D. Rumelhart, J. McClelland, and the PDP Research Group, eds., pp. 216-271. MIT Press, Cambridge, MA. Sutton, G., Reggia, J., and Maisog, J. 1990. Competitive learning using competitive activation rules. Proceedings of International Joint Conference on Neural Networks, Vol. II, pp. 285-291, San Diego, CA. Received 16 May 1991; accepted 26 June 1992.
This article has been cited by: 2. Yu G. Smetanin. 1998. Neural networks as systems for recognizing patterns. Journal of Mathematical Sciences 89:4, 1406-1457. [CrossRef]
Communicated by Christof Koch
Constraints on Synchronizing Oscillator Networks David E. Cairns Roland J. Baddeley Leslie S. Smith Centrefor Cognitiveand Computational Neuroscience, University of Stirling, Stirling, Scotland, FK9 4LA
This paper investigates the constraints placed on some synchronized oscillator models by their underlying dynamics. Phase response graphs are used to determine the phase locking behaviors of three oscillator models. These results are compared with idealized phase response graphs for single phase and multiple phase systems. We find that all three oscillators studied are best suited to operate in a single phase system and that the requirements placed on oscillatory models for operation in a multiple phase system are not compatible with the underlying dynamics of oscillatory behavior for these types of oscillator mode1s
.
1 Introduction Following observations of oscillations and synchronization behavior in cat visual cortex (Eckhorn et al. 1989; Gray et al. 1989a) a number of interpretations have been put forward to explain these results (Gray et al. 1989b; Eckhorn et al. 1988; Grossberg and Somers 1991; Shastri 1989; Sompolinsky et al. 1990). It has been suggested that a possible interpretation of the observed synchronization behavior is that the brain could be using synchronized oscillations as a method of solving the binding problem (Von der Malsburg and Schneider 1986). If a cluster of nodes that share a common property are synchronized, they are thus labeled as belonging to one group. Other synchronized nodes that are in a different phase of an oscillatory cycle are effectively labeled as a separate group. By using this method, it can be seen that a number of different entities may be stored simultaneously, each represented by a different phase in an oscillatory cycle. A fundamental requirement behind these theories is that groups of nodes should be able to move into and remain in separate synchronized phases. A simple but effective architecture that enables synchronization to take place is lateral coupling. Lateral connections between node pairs Neural Compufufion 5,260-266 (1993) @ 1993 Massachusetts Institute of Technology
Synchronizing Oscillator Networks
261
transfer a measure of the activation state of one node to the other. This causes a change in the period of the receiving node and thereforea change in its phase. We investigate the response of three generic oscillator models to this type of effect and determine whether or not they are capable of supporting multiple phases as required by the above theories in order to perform binding. 2 Method
Three studies were performed, one for each of the oscillator models. To provide a small but general cross section, we chose one simple oscillator model and two biological models. For the simple oscillator, a leaky integrator model was chosen to illustrate the most basic phase response one can obtain from a nonlinear system (Appendix A.1). As an example of models of cellular oscillations or potential pacemaker cells, a reduced version of the Hodgkin-Huxley cell membrane model (the Morris-Lecar oscillator) was chosen [Appendix A.2 (Rinzel and Ermentrout 1989)l. At the multicellular level, an oscillator based on an original model of excitatory/inhibitory cell cluster interactions by Wilson and Cowan (1972) was used [Appendix A.3 (Wang et al. 199011. The technique for obtaining the phase response graphs was obtained from Rinzel and Ermentrout’s original study of the dynamics of the Morris-Lecar model (Rinzel and Ermentrout 1989). Each oscillator was driven by a constant input until the period of the oscillation had stabilized. This gave a base period Ab. Driving the oscillator by the constant input and starting from a point just after peak activation, trials were made for points across the phase of the oscillation. For each successive trial, the instant of delivery of an entraining signal to the oscillator was increased. Each entraining input was of a constant size and was delivered for a constant proportion (0.025) of the period of the oscillator. The entraining input caused a change in the period of the oscillator. A measure of the relative phase shift caused by the entraining input was calculated according to equation 2.1. The cumolative results of the trials allowed the production of phase response graphs for each oscillator. (2.1) where A0 is the phase shift, Ab is the normal period, and A, is the new period. 3 Discussion
The following discussion relates how the results of the study (shown in Fig. 1) compare with an idealized phase response behavior that one
262
D. E. Cairns, R. J. Baddeley, and L. S. Smith
would like in a single phase and a multiple phase system. For a single phase system where all nodes move toward a globally synchronized activation, the ideal phase response behavior can be represented by the graph in Figure 2a. An entraining signal causes the phase of a receiving node to move in the direction of the phase of the node producing the signal. The degree of phase shift is proportional to the difference between the two nodes and thus causes a steady convergence with minimal possibility of overshoot. The direction of phase shift is determined by the difference in phase, the phase shift being in the direction of the shortest "route" to synchrony. A node with this form of behavior will always attempt to synchronize with the originator of any signal and will remain unperturbed only when in synchrony. This represents an idealized behavior for a single phase system, however any system that has zero phase shift at 0 and 1 with a monotonic decrease in phase in the region 0-0.5 and a monotonic increase in phase in the region 0.5-1.0 (with a discontinuity at 0.5) will cause synchronization to occur (Niebur et al. 1991; Sompolinsky et al. 1990). An example of the phase response for an oscillator in a multiple phase system is shown in Figure 2b. The oscillator maintains the requirements for phase locking with entraining signals arriving close to the phase of a receiving node. However, if the entraining signals arrive further out of phase then no phase shift occurs. This "dead zone" allows for the coexistence of multiple phase groups where inputs arriving from each out-of-phase group do not perturb the receiving group. The above phase response behavior is atypical of most oscillatory dynamics. The frequency of a node is usually increased or decreased as a result of extra input. Only in cases where the "activation" of the node is saturated (for example, when it has reached its peak or is in a refractory period), will little phase shift occur. The extended region of low response required for multiple phases is unlikely to be present in the basic dynamics of most oscillator models. Comparing these requirements with the phase graphs of the oscillators under study, it can be seen that they are best suited to single phase/synchronized activation systems. All three models exhibit an almost linear positive phase convergence in the latter half of their phase. In the case of the two neurophysiologicallybased systems some negative movement is also observed in the first half of the phase plane. Although none of the phase responses is ideal, they are sufficient to allow all of the models to exhibit effective synchronization behavior. Conversely, the oscillators studied do not show the type of behavior necessary for a multiple phase system. They do not possess significant regions of low response to entraining input in their mid-phase region. Consequently these oscillators do not allow for separate phases to coexist stably in a system. They will always be attempting to cause global synchronization. This would favor a network where one population is synchronized against a background of incoherent activity (Sompolinsky et al. 1990; Niebur et al.
SynchronizingOscillator Networks
263
Phase Shift
' u a 4
0.2
0.6
0.4
0.8
1
Input 0.50 ._._Input 0.25
-0.2 0.2 (C)
0.4
0.6
0.E
1
Phase
Figure 1: Phase response graphs. (a) Leaky integrator model, (b) Morris-Lecar cell membrane model, and (c) Wang et al. cell cluster model. Each graph shows the change in phase that occurs when an oscillator is perturbed at a given point in its phase. The x axis gives the phase of the oscillator at the point it is perturbed and the y axis the degree of perturbation in terms of phase shift. The amount of entraining input I by which the oscillator is stimulated is given as a fraction of the driving input of the oscillator. 1991; Koch and Schuster 19921, thus allowing figure ground separation but not labeling of multiple objects by phase (Shastri 1989). 4 Conclusion
This paper indicates there is a limit to the number of stable phases one can expect a system of interacting oscillators to maintain and that this limit is low. The results give support to models that use similar oscillators to achieve low level synchronization for the purposes of coherent activation. For models that use synchronized oscillations and multiple phases
D. E. Cairns, R. J. Baddeley, and L. S. Smith
264
Figure 2: Idealized phase response graphs. (a)Single phase system; (b)multiple phase system. as a method to solve the binding problem, they show that the number of phases available is likely to be significantly less than the minimum required to perform useful computation.
Appendix A Oscillator Models A.l Leaky Integrator
where T is the tonic input (1.0) k = 0.95 E is the entraining input (0.5,0.25) O = 19.93
A.2 Morris-Lecar
dv dt = -iion(V,
+T+ E
W)
(A.2)
Synchronizing Oscillator Networks
265
where
gG, = 1.1
v is the voltage
Vl = -0.01
q5 = 0.2
w is the fraction of K+ channels open g~ = 2.0 T is the tonic input (0.28) g~ = 0.5 E is the entraining input (0.14,0.07)
~2 ~3 ~4
= 0.15 = 0.0 = 0.3
V K = -0.7 V L = -0.5
A.3 Wang et al.
dyi-- --Yi + G, dt
T,
F ( x ) = (1 - V ) X T~ = 0.9
6, = 0.4
+ Tyx;)
(-T
yyY
+ ~2
(0 I Q I 1)
T,, = 1.0
1.0 Oy = 0.6 TW = 1.9 f = 0.2 A, = 0.05 Tyx= 1.3 j = 0.2 A, = 0.05 Tw = 1.2 Q = 0.4 a! = 0.2 /3 = 0.14
T~ =
(A.9)
(A.12)
I; is the tonic input (0.3) Siis the entraining input (0.15,0.075)
Acknowledgments The authors would like to thank the members of CCCN for useful discussions in the preparation of this paper, in particular Peter Hancock and Mike Roberts for their helpful comments on the draft versions. Roland Baddeley and David Cairns are both funded by SERC and Leslie Smith is a member of staff in the Department of Computing Science and Mathematics at the University of Stirling. References Eckhorn, R., Bauer, R., Jordan, W., Brosch, M., Kruse, W., Munk, M., and Reitboeck, H. J. 1988. Coherent oscillations: A mechanism of feature linking in the visual cortex? B i d . Cybernet. 60, 121-130. Eckhorn, R., Reitboeck, H. J., Arndt, M., and Dicke, I? 1989. Feature linking via stimulus-evoked oscillations: Experimental results from cat visual cortex
266
D. E. Cairns, R. J. Baddeley, and L. S. Smith
and functional implications from a network model. Proc. Intl. Joint Conf. Neural Networks (Washington), pp. 723-730. Gray, C. M.,Konig, P., Engel, A. K., and Singer, W. 1989a.Oscillatory responses in cat visual cortex exhibit inter-columnar synchronisation which reflects global stimulus properties. Nature (London) 338, 1698-1702. Gray, C. M.,Konig, P., Engel, A. K., and Singer, W. 1989b. Synchronisation of oscillatory ~psponsesin visual cortex: A plausible mechanism for scene segmentation. Proc. lntl. Symp. Synergetics Cognition, Vol. 45,C24,pp. 82-98. Grossberg, S., and Somers, D. 1991. Synchronised oscillations during cooperative feature linking in a cortical model of visual perception. Neural Networks 4,453-466. Koch, C., and Schuster, H. 1992. A simple network showing burst synchronization without frequency locking. Neural Comp. 4(2), 211-223. Niebur, E., Schuster, H. G., Kammen, D. M.,and Koch, C. 1991. Oscillator-phase coupling for different two-dimensional network connectivities. Phys. Rev. A 44,6895-6904. Rinzel, J.,and Ermentrout, G. B. 1989. Analysis of neural excitability and oscillations. In Methods in Neuronal Modeling-From Synapses to Networks. C. Koch and I. Segev, eds. MIT Press, Cambridge. Shastri, L. 1989. From simple associntions to systematic reasoning: A connectionist representation of rules, variables and dynamic bindings. Tech.Rep. University of Pennsylvania. Sompolinsky, H., Golomb, D., and Kleinfeld, D. 1990. Global processing of visual stimuli in a neural network of coupled oscillators. Proc. Natl. Acad. Sci. U.S.A. 87, 7200-7204. von der Malsburg, C., and Schneider, W. 1986. A neural cocktail-partyprocessor. Biol. Cybernet. 54, 29-40. Wan& D.,Buhmann, J., and von der Malsburg, C. 1990. Pattern segmentation in associative memory. Neural Comp. 2,96106. Wilson, H.P.,and Cowan, J. D. 1972. Excitatory and inhibitory interactions in localized populations of model neurons. Biophys. 1 - 1 2 . Received 19 May 1992; accepted 3 September 1992.
This article has been cited by: 2. G. Frank, G. Hartmann, A. Jahnke, M. Schafer. 1999. An accelerator for neural networks with pulse-coded model neurons. IEEE Transactions on Neural Networks 10:3, 527-538. [CrossRef] 3. S. Campbell, DeLiang Wang. 1996. Synchronization and desynchronization in a network of locally coupled Wilson-Cowan oscillators. IEEE Transactions on Neural Networks 7:3, 541-554. [CrossRef] 4. Alfred Nischwitz, Helmut Glünder. 1995. Local lateral inhibition: a key to spike synchronization?. Biological Cybernetics 73:5, 389-400. [CrossRef]
Communicated by Ralph Liiker
Learning Mixture Models of Spatial Coherence Suzanna Becker Geoffrey E. Hinton Department of Computer Science, University of Toronto, Toronto, Ontario, Canada M5S 1A4
We have previously described an unsupervised learning procedure that discovers spatially coherent properties of the world by maximizing the information that parameters extracted from different parts of the sensory input convey about some common underlying cause. When given random dot stereogramsof curved surfaces, this procedure learns to extract surface depth because that is the property that is coherent across space. It also learns how to interpolate the depth at one location from the depths at nearby locations (Beckerand Hinton 1992b). In this paper, we propose two new models that handle surfaces with discontinuities. The first model attempts to detect cases of discontinuities and reject them. The second model develops a mixture of expert interpolators. It learns to detect the locations of discontinuities and to invoke specialized, asymmetric interpolators that do not cross the discontinuities. 1 Introduction
Standard backpropagation is implausible as a model of perceptual learning because it requires an external teacher to speclfy the desired output of the network. We have shown (Becker and Hinton 1992b) how the external teacher can be replaced by internally derived teaching signals. These signals are generated by using the assumption that different parts of the perceptual input have common causes in the external world. Small modules that look at separate but related parts of the perceptual input discover these common causes by striving to produce outputs that agree with each other (see Fig. la). The modules may look at different modalities (e.g., vision and touch), or the same modality at different times (e.g., the consecutive 2-D views of a rotating 3-D object), or even spatially adjacent parts of the same image. In previous work, we showed that when our learning procedure is applied to adjacent patches of images, it allows a neural network that has no prior knowledge of depth to discover stereo disparity in random dot stereograms of curved surfaces. A more general version of the method allows the network to discover the best way of interpolating the depth at one location from the depths at nearby locations. We first summaNeural Computation 5,267-277 (1993) @ 1993 Massachusetts Institute of Technology
268
Suzanna Becker and Geoffrey E. Hinton
rize this earlier work, and then introduce two new models that allow coherent predictions to be made in the presence of discontinuities. The first assumes a model of the world in which patterns are drawn from two possible classes: one which can be captured by a simple model of coherence, and one which is unpredictable. This allows the network to reject cases containing discontinuities. The second method allows the network to develop multiple models of coherence, by learning a mixture of depth interpolators for curved surfaces with discontinuities. Rather than rejecting cases containing discontinuities, the network develops a set of location-specific discontinuity detectors, and appropriate interpolators for each class of discontinuities. An alternative way of learning the same representation for this problem, using an unsupervised version of the competing experts algorithm described by Jacobs et al. (1991), is described in Becker and Hinton (1992a). 2 Learning Spatially Coherent Features in Images
Using a modular architecture as shown in Figure la, a network can learn to model a spatially coherent surface, by extracting mutually predictable features from neighboring image patches. The goal of the learning is to produce good agreement between the outputs of modules that receive input from neighboring patches. The simplest way to get the outputs of two modules to agree is to use the squared difference between the outputs as a cost function, and to adjust the weights in each module so as to minimize this cost. Unfortunately, this usually causes each module to produce the same constant output that is unaffected by the input to the module and therefore conveys no information about it. We would like the outputs of two modules to agree closely (i.e., to have a small expected squared difference) relative to how much they both vary as the input is varied. When this happens, the two modules must be responding to something that is common to their two inputs. In the special case when the outputs, da, db, of the two modules are scalars, a good measure of agreement is
where V is the variance over the training cases. Under the assumption that da and db are both versions of the same underlying gaussian signal that have been corrupted by independent gaussian noise, it can be shown that I is the mutual information (Shannon and Weaver 1964)between the underlying signal and the average of d, and db. By maximizing I we force the two modules to extract as pure a version as possible of the underlying common signal.
2.1 The Basic Stereo Net. We have shown how this principle can be applied to a multilayer network that learns to extract depth from ran-
Learning Mixture Models
269
max I
vu
Q Q hidden units
Figure 1: (a)Two modules that receive input from corresponding parts of stereo images. The first module receives input from stereo patch A, consisting of a horizontal strip from the left image (striped) and a corresponding strip from the right image (hatched). The second module receives input from an adjacent stereo patch B. The modules try to make their outputs, da and db, convey as much information as possible about some underlying signal (i.e., the depth) which is common to both patches. (b) The architecture of the interpolating network, consisting of multiple copies of modules like those in (a) plus a layer of interpolating units. The network tries to maximize the information that the locally extracted parameter dc and the contextually predicted parameter dc convey about some common underlying signal. We actually used 10 modules and the central 6 modules tried to maximize agreement between their outputs and contextually predicted values. We used weight averaging to constrain the interpolating function to be identical for all modules. dom dot stereograms (Becker and Hinton 1992b). Each network module received input from a patch of a left image and a corresponding patch of a right image, as shown in Figure la. Adjacent modules received input from adjacent stereo image patches, and learned to extract depth by trying to maximize agreement between their outputs. The real-valued depth (relative to the plane of fixation) of each patch of the surface gives rise to a disparity between features in the left and right images; because that disparity is the only property that is coherent across each stereo image, the output units of modules were able to learn to accurately detect relative depth. 2.2 The Interpolating Net. The basic stereo net uses a very simple model of coherence in which an underlying parameter at one location is assumed to be approximately equal to the parameter at a neighboring location. This model is fine for the depth of frontoparallel surfaces but it
270
Suzanna Becker and Geoffrey E. Hinton
is far from the best model of slanted or curved surfaces. Fortunately, we can use a far more general model of coherence in which the parameter at one location is assumed to be an unknown linear function of the parameters at nearby locations. The particular linear function that is appropriate can be learned by the network. We used a network of the type shown in Figure lb. The depth computed locally by a module, d,, was compared with the depth predicted by a linear combination d, of the outputs of nearby modules, and the network tried to maximize the agreement between d, and d,. The contextual prediction, d,, was produced by computing a weighted sum of the outputs of two adjacent modules on either side. The interpolating weights used in this sum, and all other weights in the network, were adjusted so as to maximize agreement between locally computed and contextually predicted depths. To speed the learning, we first trained the lower layers of the network as before, so that agreement was maximized between neighboring locally computed outputs. This made it easier to learn good interpolating weights. When the network was trained on stereograms of cubic surfaces, it learned interpolating weights of -0.147, 0.675, 0.656, -0.131 (Becker and Hinton 1992b). Given noise free estimates of local depth, the optimal linear interpolator for a cubic surface is -0.167, 0.667, 0.667, -0.167. 3 Mixture Models of Coherence
The models described above were based on the assumption of a single type of coherence in images. We assumed there was some parameter of the image that was either constant for nearby patches, or varied smoothly across space. In natural scenes, these simple models of coherence may not always hold. There may be widely varying amounts of curvature, from smooth surfaces, to highly curved spherical or cylindrical objects. There may be coherent structure at several spatial scales; for example, a rough surface like a brick wall is highly convoluted at a fine spatial scale, while at a coarser scale it is planar. And at boundaries between objects, or between different parts of the same object, there will be discontinuities in coherence. It would be better to have multiple models of coherence, which could account for a wider range of surfaces. One way to handle multiple models is to have a mixture of distributions (McLachlan and Basford 1988). In this section, we introduce a new way of employing mixture models to account for a greater variety of situations. We extend the learning procedure described in the previous section based on these models.
3.1 Throwing out Discontinuities. If the surface is continuous, the depth at one patch can be accurately predicted from the depths of two patches on either side. If, however, the training data contains cases in
Learning Mixture Models
271
which there are depth discontinuities (see Fig. 2) the interpolator will also try to model these cases and this will contribute considerable noise to the interpolating weights and to the depth estimates. One way of reducing this noise is to treat the discontinuity cases as outliers and to throw them out. Rather than making a hard decision about whether a case is an outlier, we make a soft decision by using a mixture model. For each training case, the network compares the locally extracted depth, d,, with the depth predicted from the nearby context, d,. It assumes that d, - d, is drawn from a zero-mean gaussian if it is a continuity case and from a uniform distribution if it is a discontinuity case, as shown in Figure 3. It can then estimate the probability of a continuity case: (3.1) where N is a gaussian, and kdiscont is a constant representing a uniform density.’ We can now optimize the average information d, and d, transmit about their common cause. We assume that no information is transmitted in discontinuity cases, so the average information depends on the probability of continuity and on the variance of d, + dc and d, - d, measured only in the continuity cases: (3.2) where Pcont = (pcont(dc - 4)). We tried several variations of this mixture approach. The network is quite good at rejecting the discontinuity cases, but this leads to only a modest improvement in the performance of the interpolator. In cases where there is a depth discontinuity between d, and db or between dd and d, the interpolator works moderately well because the weights on d, or d, are small. Because of the term Pcontin equation 3.2 there is pressure to include these cases as continuity cases, so they probably contribute noise to the interpolating weights. In the next section we show how to avoid making a forced choice between rejecting these cases or treating them just like all the other continuity cases. ‘We empiccally select a good (fixed)value of kdixont, and w,e choose a starting value of ccont(dc - d c ) (some proportion of the initial variance of d, -dJ, and gradually shrink it during learning. The learning algorithm’s performance is fairly robust with respect to variations in the choice of kdk& the main effect of changing this parameter is to sharpen or flatten the network‘s probabilistic decision function for labeling cases as continuous or discontinuous (equation 3.1). The choice of Vmnt(dc- &), on the other hand, turns out to affect the learning algorithm more critically; if this variance is too small, many cases will be treated as discontinuous, and the network may converge to very large weights which overfit only a small subset of the training cases. There is no pmblem, however, if this variance is too large initially; in this case, all patterns are h a t e d as continuous, and as the variance is shrunk during learning, some discontinuous cases are eventually detected.
272
Suzanna Becker and Geoffrey E. Hinton
Figure 2: (Top) A curved surface strip with a discontinuity created by fitting 2 cubic splines through randomly chosen control points, 25 pixels apart, separated by a depth discontinuity. Feature points are randomly scattered on each spline with an average of 0.22 features per pixel. (Bottom)A stereo pair of “intensity” images of the surface strip formed by taking two different projections of the feature points, filtering them through a gaussian, and sampling the filtered projections at evenly spaced sample points. The sample values in corresponding patches of the two images are used as the inputs to a module. The depth of the surface for a particular image region is directly related to the disparity between corresponding features in the left and right patch. Disparity ranges continuously from -1 to +1 image pixels. Each stereo image was 120 pixels wide and divided into 10 receptive fields 10 pixels wide and separated by 2 pixel gaps, as input for the networks shown in Figure 1. The receptive field of an interpolating unit spanned 58 image pixels, and discontinuities were randomly located a minimum of 40 pixels apart, so only rarely would more than one discontinuity lie within an interpolator’s receptive field.
3.2 Learning a Mixture of Interpolators. The presence of a depth discontinuity somewhere within a strip of five adjacent patches does not necessarily destroy the predictability of depth across these patches. It may just restrict the range over which a prediction can be made. So instead of throwing out cases that contain a discontinuity, the network could try to develop a number of different, specialized models of spatial coherence across several image patches. If, for example, there is a depth discontinuity between d, and d, in Figure lb, an extrapolator with weights of -1.0, $2.0, 0, 0 would be an appropriate predictor of d,. The network could also try to detect the locations of discontinuities, and use this information as the basis for deciding which model to apply on a given case. This information is useful not only in making clean decisions about which coherence model to apply, but it also provides valuable cues for interpreting the scene by indicating the locations of object boundaries in the image. Thus, we can use both the interpolated depth map, as well
Learning Mixture Models
273
A
continuity
d:
d:
Figure 3: The probability distribution of d,, P l ( d c ) , is modeled as a mixture of two distributions: a gaussian with mean = dr and small variance, and P 2 ( d c ) , a uniform distribution. Sample points for d, and d,, @ and dr are shown. In this case, $ and d t are far apart so $ is more likely to have been drawn from Pz. as the locations of depth discontinuities, in subsequent stages of scene interpretation. A network can learn to discover multiple coherence models using a set of competing interpolators. Each interpolator tries, as before, to achieve high agreement between its output and the depth extracted locally by a module. Additionally, each interpolator tries to account for as many cases as possible by maximizing the probability that its model holds. The objective function maximized by the network is the sum over models, i, of the agreement between the output of the ith model, &,, and the predicted depth, d,, weighted by the probability of the ith model:
(3.3) where the V’s represent variances given that the ith model holds. The probability that the ith model is applicable on each case a, pp, can be computed independently of how well the interpolators are doing;’ this can be done by adding extra “controller” units to the network, as shown in Figure 4, whose sole purpose is to compute the probability, pi, that each interpolator’s model holds. The weights of both the controllers and the interpolating experts can be learned simultaneously, so as to maximize *More precisely, this computed probability is conditionally independent of the interpolators’ performance on a particular case, with independence being conditioned on a fixed set of weights. As the reviewer has pointed out, when the weights change over the course of learning, there is an interdependence between the probabilities and interpolated quantities via the shared objective function.
Suzanna Becker and Geoffrey E. Hinton
274
controller 1 exoert 2
controller 2
controller 4 expert 5
controller 5
Figure 4 An architecturefor learning a mixture model of curved surfaces with discontinuities, consisting of a set of interpolators and discontinuity detectors. We actually used a larger modular network and equaIity constraints between the weights of correspondingunits in different modules, with 6 copies of the architectureshown here. Each copy received input from different but overlapping parts of the input. I”. By assigning a controller to each expert interpolator, each controller should learn to detect a discontinuity at a particular location (or the absence of a discontinuity in the case of the interpolator for pure continuity cases). And each interpolating unit should learn to capture the particular type of coherence that remains in the presence of a discontinuity at a particular location. The outputs of the controllers are normalized, so that they represent a probability distribution over the interpolating experts’ models. We can think of these normalized outputs as the probability with which the system selects a particular expert. Each controller’s output is a normalized exponential function of its squared total input, xi: (3.4)
Squaring the total input makes it possible for each unit to detect a depth edge at a particular location, independently of the direction of contrast change. We normalize the s uared total input in the exponential by an estimate of its variance, 6(x$ = k w,:. (This estimate of the variance of the total weighted input is exact if the unweighted individual inputs are independent, gaussian, and have equal variances of size k.) This discourages any one unit from trying to model all of the cases simply by having huge weights. The controllers get to see all five local depth estimates, da . . .d,. As before, each interpolating expert computes a h e a r function of four contextually extracted depths, d i c = Wbda +Wi& wddd wiede, in order to try to predict the centrally extracted depth d,.
cji
+
+
Learning Mixture Models
275
We first trained the network using the original continuous model, as described in Section 2, on a training set of 1000 images with discontinuities, until the lower layers of the network became well tuned to depth. So the interpolators were initially pretrained using the continuity model, and all the interpolators learned similar weights. We then froze the weights in the lower layers, added a small amount of noise to the interpolators’ weights (uniform in [-0.1,0.1]), and applied the mixture model to improve the interpolators and train the controller units. We ran the learning procedure for 10 runs, each run starting from different random initial weights and proceeding for 10 conjugate gradient learning iterations. The network learned similar solutions in each case. A typical set of weights on one run is shown in Figure 5. The graph on the right in this figure shows that four of the controller units are tuned to discontinuities at different locations. The weights for the first interpolator (shown in the top left) are nearly symmetrical, and the corresponding controller’s weights (shown immediately to the right) are very small; the graph on the right shows that this controller (shown as a solid line plot) mainly responds in cases when there is no discontinuity. The second interpolator (shown in the left column, second from the top) predominantly uses the leftmost three depths; the corresponding controller for this interpolator (immediatelyright of the top left interpolator’s weights) detects discontinuitiesbetween the rightmost two depths, d, and d d . Similarly, the remaining controllers detect discontinuities to the right or left of d,; each controller’s corresponding interpolator uses the depths on the opposite side of the discontinuity to predict d,. 4 Discussion
We have described two ways of modeling spatially coherent features in images of scenes with discontinuities. The first approach was to simply try to discriminate between patterns with and without discontinuities, and throw away the former. In theory, this approach is promising, as it provides a way of making the algorithm more robust against outlying data points. We then applied the idea of multiple models of coherence to a set of interpolating units, again using images of curved surfaces with discontinuities. The competing controllers in Figure 4 learned to explicitly represent which regularity applies in a particular region. The output of the controllers was used to compute a probability distribution over the various competing models of coherence. The representation learned by this network has a number of advantages. We now have a measure of the probability that there is a discontinuity that is independent of the prediction error of the interpolator. So we can tell how much to trust each interpolator’s estimate on each case. It should be possible to distinguish clear cases of discontinuities from cases that are simply noisy, by the entropy of the controllers’ outputs.
276
Suzanna Becker and Geoffrey E. Hinton
Figure 5 (a) Typical weights learned by the five competing interpolators and correspondingfive discontinuitydetectors. Positive weights are shown in white, and negative weights in black. (b) The mean probabilities computed by each discontinuity detector are plotted against the distance from the center of the units’ receptive field to the nearest discontinuity. The probabilistic outputs are averaged over an ensemble of lo00 test cases. If the nearest discontinuity is beyond f30 pixels, it is outside the units’ receptive field and the case is therefore a continuity example.
Furthermore, the controller outputs tell us not only that a discontinuity is present, but exactly where it lies. This information is important for segmenting scenes, and should be a useful representation for later stages of unsupervised learning. Like the raw depth estimates, the location of depth edges should exhibit coherence across space, at larger spatial scales. It should therefore be possible to apply the same algorithm recursively to the outputs of the controllers, to find object boundaries in two-dimensional stereo images. The approach presented here should be applicable to other domains that contain a mixture of alternative local regularities across space or time. For example, a rigid shape causes a linear constraint between the locations of its parts in an image, so if there are many possible shapes, there are many alternative local regularities (Zemel and Hinton 1991).
Learning Mixture Models
277
Acknowledgments
This research was funded by grants from NSERC and the Ontario Information Technology Research Centre. Hinton is Noranda fellow of the Canadian Institute for Advanced Research. Thanks to John Bridle and Steve Nowlan for helpful discussions.
References Becker, S., and Hinton, G. E. 1992a. Learning to make coherent predictions in domains with discontinuities. In Advances in Neural Information Processing Systems 4. Morgan Kaufmann, San Mateo, CA. Becker, S., and Hinton, G. E. 1992b. A self-organizing neural network that discovers surfaces in random-dot stereograms. Nature (London) 355, 161163. Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. 1991. Adaptive mixtures of local experts. Neural Cornp. 3(1), 79-87. McLachlan, G. J., and Basford, K. E. 1988. Mixture Models: Inference and Applications to Clustering. Marcel Dekker, New York. Shannon, C. E., and Weaver, W. 1964. The Mathematical Theory of Communication. The University of Illinois Press, Urbana, IL. Zemel, R. S., and Hinton, G. E. 1991. Discovering viewpoint-invariant relationships that characterize objects. In Advances in Neural Information Processing Systems 3, pp. 299-305. Morgan Kaufmann, San Mateo, CA.
Received 5 June 1992; accepted 26 June 1992.
This article has been cited by: 2. K. Lee, H. Buxton, J. Feng. 2005. Cue-Guided Search: A Computational Model of Selective Attention. IEEE Transactions on Neural Networks 16:4, 910-924. [CrossRef] 3. Ryotaro Kamimura, Taeko Kamimura, Thomas R. Shultz. 2001. Information Theoretic Competitive Learning and Linguistic Rule Acquisition. Transactions of the Japanese Society for Artificial Intelligence 16, 287-298. [CrossRef] 4. Suzanna Becker, Mark Plumbley. 1996. Unsupervised neural network learning procedures for feature extraction and classification. Applied Intelligence 6:3, 185-203. [CrossRef] 5. Suzanna Becker. 1996. Network: Computation in Neural Systems 7:1, 7-31. [CrossRef]
Communicated by Steve Suddarth
Hints and the VC Dimension Yaser S. Abu-Mostafa California lnstitute of Technology, Pasadena, CA 91225 USA
Learning from hints is a generalization of learning from examples that allows for a variety of information about the unknown function to be used in the learning process. In this paper, we use the VC dimension, an established tool for analyzing learning from examples, to analyze learning from hints. In particular, we show how the VC dimension is affected by the introduction of a hint. We also derive a new quantity that defines a VC dimension for the hint itself. This quantity is used to estimate the number of examples needed to "absorb" the hint. We carry out the analysis for two types of hints, invariances and catalysts. We also describe how the same method can be applied to other types of hints. 1 Introduction Learning from examples deals with an unknown function f that is represented by examples to the learning process. The process uses the examples to infer an approximate implementation off. Learning from hints (Abu-Mostafa 1990) generalizes the situation by allowing other information that we may know about f to be used in the learning process. Such information may include invariance properties, symmetries, correlated functions (Suddarth and Holden 19911, explicit rules (Winand Giles 1992),minimum-distance properties (Al-Mashouq and Reed 1991), or any other fact about f that narrows down the search. In many practical situations, we do have some prior information about f, and the proper use of this information (instead of just using "blind examples off) can make the difference between feasible and prohibitive learning. In this paper, we develop a theoretical analysis of learning from hints. The analysis is based on the VC dimension (Blumer et al. 19891, which is an established tool for analyzing learning from examples. Simply stated, the VC dimension VC(G) furnishes an upper bound for the number of examples needed by a learning process that starts with a set of hypotheses G about what f may be. The examples guide the search for a hypothesis g E G that is a good replica off. Since f is unknown to begin with, we start with a relatively big set of hypotheses G to maximize our chances of finding a good approximation off among them. However, the larger G is, the more examples off we Neural Computation 5,278-288 (1993) @ 1993 Massachusetts Institute of Technology
Hints and the VC Dimension
279
need to pinpoint the good hypothesis. This is reflected in a bigger value of V C ( G ) . How do we make G smaller without the risk of losing good approximations off? This is where the hints come in. Since a hint is a known property of f, we can use it as a litmus test to weed out bad gs thus shrinking G without losing good hypotheses. The main result of this paper is the application of the VC dimension to hints in two forms. 1. The VC dimension provides an estimate for the number of examples needed to learn f. When a hint H is given about f , the number of examples off can be reduced. This is reflected in a smaller “VC dimension given the hint’’ V C ( G I H). 2. If H itself is represented to the learning process by a set of examples,
we would like to estimate how many examples are needed to absorb the hint. This calls for a generalization of the VC dimension to cover examples of the hint as well as examples of the function, which is reflected in a “VC dimension for the hint” V C ( G ; H ) . We will study two types of hints in particular, invariances and catalysts. We will discuss how the same framework can be used to study other types of hints. A detailed account of the VC dimension can be found in (Blumer et al. (1989) and Vapnik and Chervonenkis (1971). We will provide a brief background here to make the paper self-contained. The setup for learning from examples consists of an environment X and an unknown function f : X ---t ( 0 , l ) that we wish to learn. The goal is to produce a hypothesis g : X -, (0,l) that approximates f . To do this, the learning process starts with a set of hypotheses G and tries to select a good g E G based on a number of examples [ x l , f ( x l ) ]; . . . ; [xN,f(xN)]off. To generate the examples, we assume that there is a probability distribution P ( x ) on the environment X. Each example is picked independently according to P ( x ) . The hypothesis g that results from the learning process is considered a good approximation off if the probability [wxt. P(x)l that g ( x ) # f ( x ) is small. The learning process should have a high probability of producing a good approximation off when a sufficient number of examples is provided. The VC dimension helps determine what is “sufficient.” Here is how it works. Let rg= Pr[g(x) = f ( x ) ] ,where Pr[.] denotes the probability of an event. We wish to pick a hypothesis g that has rgx 1. However, f is unknown and thus we do not know the values of these probabilities. Since f is represented by examples, we can compute the frequency of agreement between each g and f on the examples and base our choice of g on the frequencies instead of the actual probabilities. Let hypothesis g agree with f on a fraction vg of the examples. We pick a hypothesis that has vg M 1. The VC inequality asserts that the values of vgs will be close to rgs. Specifically, r
1
Yaser S. Abu-Mostafa
280
where “sup” denotes the supremum, and rn is the growth function of G. m(N)is the maximum number of different binary vectors g(x1) . - . g ( x ~ ) that can be generated by varying g over G while keeping XI, . . . ,X N E X fixed. Clearly, m(N)5 2N for all N. The VC dimension V C ( G )is defined as the smallest N for which m(N) < 2N. We assume that G has a finite VC dimension. If VC(G)= d, the growth function m(N)can be bounded bY
When this estimate is substituted in the VC inequality, the right-hand side of the inequality becomes arbitrarily small for sufficiently large N. This means that it is almost certain that each vgis approximately the same as the corresponding 7rg. This is the rationale for considering N examples sufficient to learn f . We can afford to base our choice of hypothesis on vg as calculated from the examples, because it is approximately the same as 7rg. How large N needs to be to achieve a certain degree of approximation is affected by the value of the VC dimension. In this paper, we assume that f E G. This means that G is powerful enough to implement f . We also assume that f strictly satisfies the hint H. This means that f will not be excluded as a result of taking H into consideration. Finally, we assume that everything that needs to be measurable will be measurable. 2 Invariance Hints
It is often the case that we know an invariance property of an otherwise unknown function. For example, speaker identification based on a speech waveform is invariant under time shift of the waveform. Properties such as shift invariance and scale invariance are commonplace in pattern recognition, and dozens of methods have been developed to take advantage of them (e.g., Hu 1969). Invariances have also been used in neural networks, for example, group invariance of functions (Minsky and Papert 1988) and the use of invariances in backpropagation (Abu-Mostafa 1990). An invariance hint H can be formalized by the partition
x=ux, (2
of the environment X into the invariance classes X,, where a is an index. Within each class X,, the value off is constant. In other words, x , x‘ E X, implies that f ( x ) = f (x‘) . Some invariance hints are “strong” and others are “weak,“ and this is reflected in the partition X = U, X,. The finer the partition, the weaker the hint. For instance, if each X, contains a single point, the hint is extremely weak (actually useless) since the information that x,x‘ E X,
Hints and the VC Dimension
281
implies that f ( x ) = f ( x ' ) tells us nothing new as x and x' are the same point in this case. On the other extreme, if there is a single X, that contains all the points (X, = X), the hint is extremely strong as it forces f to be constant over X (either f = 1 or f = 0). Practical hints, such as scale invariance and shift invariance, lie between these two extremes. In what follows, we will apply the VC dimension to an invariance hint H. We will start by assessing the impact of H on the original VC dimension. We will then focus on representing H by examples and address what an example of H is, how to define a VC dimension for H, and what it means to approximate H. Finally, we will discuss relations between different VC dimensions.
2.1 How the Hint Affects VC(G). The VC dimension is used to estimate the number of examples needed to learn an unknown function f . It is intuitive that, with the benefit of a hint about f, we should need fewer examples. To formalize this intuition, let the invariance hint H be given by the partition X = U, X,. Each hypothesis g E G either satisfies H or else does not satisfy it. Satisfying H means that whenever x, x' E X,, then g ( x ) = g(x'). The set of hypotheses that satisfy H is G G = {g E G I x,X' EX, + - g ( x ) = g ( x ' ) }
G is a set of hypotheses and, as such, has a VC dimension of its own.
This is the basis for defining the VC dimension of G given H VC(G 1 H ) = VC(G) Since G C_ G, it follows that VC(G I H) 5 VC(G).Nontrivial hints lead to a significant reduction from G to G, resulting in VC(G [ H) < VC(G). On the other hand, some hints may have VC(G I H) = VC(G). For instance, in the case of the weak hint we talked about, every g trivially satisfies the hint, hence G = G. VC(G 1 H ) replaces VC(G) following the "absorption" of the hint. Without the hint, VC(G)provides an estimate for the number of examples needed to learn f . With the hint, VC(G 1 H) provides a new estimate for the number of examples. This estimate is valid regardless of the mechanism for absorbing the hint, as long as it is completely absorbed. If, however, the hint is only partially absorbed (which means that some gs that do not strictly satisfy the invariance are still allowed), the effective VC dimension lies between VC(G)and VC(G I H). 2.2 Representing the Hint by Examples. What is an example of an invariance hint? If we take the hint specified by X = U, X,, an example would be " f ( x ) = f(x')," where x and x' belong to the same invariance class. In other words, an example is a pair ( x , x ' ) that belong to the same X,.
Yaser S. Abu-Mostafa
282
The motivation for representing a hint by examples is twofold. The hint needs to be incorporated in what is already a learning-from-examples process. The example f ( x ) = f(x') can be directly included in descent methods such as backpropagation along with examples of the function itself. To do this, the quantity [g(x) - g(x')I2 is minimized the same way [g(x) - f ( x ) l Zis minimized when we use an example o f f . In addition, we may represent a hint by examples if it cannot be easily expressed as a global mathematical constraint. For instance, invariance under elastic deformation of images does not readily yield an obvious constraint on the weights of a feedforward network. In contrast to the function f that is represented by a controlled number of examples and is otherwise unknown,a hint can be represented by as many examples as we wish, since it is a known property and hence can be used indefinitely to generate examples. Examples of the hint, like examples of the function, are generated according to a probability distribution. One way to generate ( x , x ' ) is to pick x from X according to the probability distribution P ( x ) , then pick x' from X, (the invariance class that contains x ) according to the conditional probability distribution P(x' 1 Xa).A sequence of N (pairs of) examples (XI, x:); (XZ, 2');.. .;( X N , YN)would be generated in the same way, independently from pair to pair. 2.3 A VC Dimension for the Hint. As we discussed in the introduction, the VC inequality is used to estimate how well f is learned. We wish to use the same inequality to estimate how well H is absorbed. To do this, we transform the situation from hints to functions. This calls for definitions of new X, P, G, and f. Let H be the invariance hint X = U,X,. The new environment is defined by
x=ux; a
(pairs of points coming from the same invariance class) with the probability distribution described above P ( x , 2)= P(x)P(x' 1 X,) where X, is the class that contains x (hence contains x'). The new set of hypotheses G, defined on the environment X, contains a hypothesis g for every hypothesis g E G such that
and the function to be "learned" is f(x,x') = 1
Hints and the VC Dimension
283
The VC dimension of the set of hypotheses G is the basis for defining a VC dimension for the hint. VC(G;H ) = VC(G) VC(G;H)depends on both G and H since G is based on G and the new environment X (which in turn depends on HI. 2.4 Approximation of the Hint. If the above learning process resulted in the hypothesis g = f (the constant 11, the corresponding g E G would obviously satisfy the hint. Learning from examples, however, results only in g that approximates f well (with high probability). The approximation is in terms of the distribution P(x,x') used to generate the examples. Thus, w.r.t. to P, Pr[g(x,x') # 11 0 as the number of examples N becomes large. Can we translate this statement into a similar one based only on the original distribution P(x)? To do this, we need to rid the statement of x'. Let --f
P r W # g(x')l = 7 By definition of g, Pr[g(x,x') # 11 is the same as Pr[g(x) # g(x')]. This implies that y 0 as N + 00. In words, if we pick x and x' at random according to P(x,x'), the probability that our hypothesis will have different values on these two points is small. To get rid of x' from this statement, we introduce hint-satisfying versions of the gs. For each g E G, let g be the best approximation of g that strictly satisfies the hint. This means that, within each invariance class X,, g(x) is constant and its value is the more probable of the two values of g(x) within X, (ties are broken either way). We will argue that P r W # g(4l I Y Since y + 0, this statement [which is solely based on P ( x ) ] implies that "g approximately satisfies the hint" in a more natural way. Here is the argument. Let q be the probability that g(x) # g(x). Given X,, let be the conditional probability that g(x) # g ( x ) , and let ya be the conditional probability that g(x) # g(x'). From the definition of g, q, must be I (otherwise, the value of g in X, should be flipped). Within each X,, since g is constant, g(x) # g(x') if, and only if, g agrees with g on either x or x' and disagrees on the other. This means that
+,
1
Ya = 2%(1 -%)
2 ?a (since 1 - q, 2 f). This is true for every class X,. Averaging over a, we get y 2 9, hence Prk(x) #Wl = 3 5 7 4
0
This establishes the more natural notion of approximating the hint.
Yaser S. Abu-Mostafa
284
2.5 A Bound on VC(G;H). As in the case of the set G and its growth function m ( N ) , the VC dimension V C ( G ; H )= V C ( G )is defined based on the growth function m(N)of the set G. m(N) is the maximum number of patterns of 1s and 0s that can be obtained by applying the g’s to (fixed but arbitrary) N examples (x,, x i ) ; (XZ, x i ) ; . . .;( X N , xh). VC(G;H ) is the smallest N for which m(N)< 2N. The value of V C ( G ; H )will differ from hint to hint. Consider our two extreme examples of weak and strong hints. The weak hint has VC(G;H ) as small as 1 since each g always agrees with each example of the hint [hence every g is the constant 1, and m(N)= 1 for all NI. The strong hint has V C ( G ; H )as large as it can be. How large is that? In Fyfe (19921, it is shown that for any invariance hint H ,
V C ( G ; H )< X V C ( G ) where X = 4.54. The argument goes as follows. For each pattern generated by the g’s on x1, x{i x2i x;i . . . X N , there is at most one distinct pattern generated by the g’s on 9
(XI 1
4
1; (x2,d);.. ‘ ;(XN,xk)
because g ( X n 1 xb) is uniquely determined by g ( x n ) and g(x‘,). Therefore,
m(N)5 m(2N) If V C ( G )= d, we can use Chernoff bounds (Feller 1968)to estimate m(2N) for N 2 d as follows
< - 2U(d/Wx2N where X ( 8 ) = -0 log, B - (1- 0 ) log,( 1- 6 ) is the binary entropy function. Therefore, once X(dI2N) 5 m(N) will be less than 2N and N must have reached, or exceeded, the VC dimension of G. This happens at N l d x 4.54. In many cases, the relationship between V C ( G I H ) and V C ( G ; H ) can be roughly stated as follows: the smaller one is, the bigger the other is. Strong hints generally result in a small value of VC(G I H) and a large value of V C ( G ; H ) ,while weak hints result in the opposite situation [the loose similarity with the average mutual information I(X;Y) and the conditional entropy H(X 1 Y) in information theory is the reason for choosing this notation for the various VC dimensions]. This relationship between V C ( G I H ) and V C ( G ; H )may suggest that we do not save when we use examples of a hint and, as a result, use fewer examples of the function. However, it should be noted that examples of the hint can be generated at will, while examples of the function may be limited in number or expensive to generate.
it
Hints and the VC Dimension
285
3 Catalyst Hints
Catalyst hints (Suddarth and Holden 1991) were introduced as a means of improving the learning behavior of feedfonvard networks. The idea is illustrated in Figure 1. A network attempting to learn the function g = f is augmented by a catalyst neuron out of the last hidden layer. This neuron is trained to learn a related function g' = f'. In doing so, the hidden layers of the network are influenced in a way that helps the main learning task g = f . After the learning phase is completed, the catalyst neuron is removed. The catalyst function f' is typically a "well-behaved version" of f that can be learned more easily and more quickly. When f' is learned, the internal representations in the hidden layers of the network will be suited for the implementation of the main function f. As a hint, namely a piece of information about f l the catalyst is the assertion that there is a way to set the weights of the network that simultaneously implements g = f and g' = f'. Unlike invariances, catalysts are very particular to the network we use. To formalize the catalyst hint, let 0 be the set of pairs of hypotheses (g,g') that can be simultaneously implemented by the network (when the catalyst neuron is present). The values of the weights in the different
Figure 1: A network that uses a catalyst hint.
Yaser S. Abu-Mostafa
286
layers of the network determine (g,g‘). A particular g may appear in different pairs (8, g‘) and, similarly, a particular g‘ may appear in different pairs (g,g‘). Since the catalyst hint puts a condition on g‘, its impact on g is indirect through these pairings of (g,g‘). This suggests the following notation: (g,8’) denotes the hypothesis g when the catalyst hypothesis is g‘ and (g,g’) denotes the hypothesis g‘ when the main hypothesis is g. Applied to a point x E X, we use the convention (g,gO(x) = g(x) (g,g’)(x) = g’(4
Thus (g,g‘) and (g,8’) provide an inflated notation for the hypotheses g and g‘, respectively. In these terms, the set of hypotheses G is defined by G = {(g,g’) I (g,g’) E 8)
To apply the VC dimension to catalyst hints, we will follow the same steps we used for invariance hints. The catalyst hint H is given by the constraint g‘ = f’. When H is absorbed, G is reduced to G
Obviously, G E G. The VC dimension of G given H is VC(G I H ) = VC(G)
Again, VC(G I H) 5 VC(G). How small VC(G I H) will be depends on the catalyst function f’. For instance, the degenerate case of a constant f’ results in VC(G I H) = VC(G) since the constant can be implemented by the catalyst neuron alone and would not impose any constraint on the weights of the original network. On the other hand, a complex f’ will take specific combinations of weights to implement, thus significantly restricting the network and resulting in VC(G I H) VC(G). If the hint is only partially absorbed, the effective VC dimension lies between VC(G) and VC(G I H). One situation that leads to partial absorption is when the hint is represented by examples. An example of the hint H: g‘ = f’ takes the form g’(x) = f’(x). In this case, examples of H are of the same nature as examples off; x is picked from X according to P ( x ) and f ’ ( x ) is evaluated. The definition of examples of H leads to the definition of G, the set of agreement/disagreement patterns between the hypotheses and the hint. For each hypothesis (g,g’) E G, there is a hypothesis g E G such that
Hints and the VC Dimension
287
The VC dimension of G is the basis for defining VC(G;H), the VC dimension that will indicate how many examples [x,f’(x)] are needed to absorb H. It is given by
VC(G;H)= V C ( G ) Unlike an invariance hint, the particular choice of a catalyst hint (the function f’) does not affect the value of VC(G;H). The VC inequality asserts that a sufficient number of examples will lead to a hypothesis (g,g‘) that satisfies
W(g, g W ) # fW1
+
0
where the probability is taken wxt. P ( x ) . Therefore, we will get a hypothesis g that pairs up with a good approximation off’. This establishes a natural notion of approximating the hint. 4 Conclusion
We have analyzed two different types of hints, invariances and catalysts. The highlight of the analysis is the definition of VC(G I H ) and VC(G;H). These two quantities extend the VC inequality to cover learning f given the hint, and learning the hint itself. Other types of hints can be quite different from invariances and catalysts, and will require new analysis. However, the common method for dealing with any type of hint in this framework is as follows. 1. The definition of the hint should determine for each hypothesis in G whether or not it satisfies the hint. The set G contains those hypotheses which do satisfy the hint. V C ( G 1 H) is defined as VC(G). 2. A scheme for representing the hint by examples should be selected.
Each example is generated according to a probability distribution P that depends on the original distribution P. Different examples are generated independently according to the same distribution. 3. For every hypothesis and every example of the hint, we should be able to determine whether or not the hypothesis agrees with the example. The agreement/disagreement patterns define the set of hypotheses G,and V C ( G ) defines VC(G;H). A hypothesis will agree with every possible example if, and only if, it satisfies the hint. 4.
How well a hypothesis approximates the hint is measured by the probability (w.r.t. P) that it will agree with a new example. An approximation in this sense should imply a partial absorption of the hint.
288
Yaser S. Abu-Mostafa
5. How the hint is represented by examples may not be unique. The choice of representation affects the definition of VC(G;H) and also affects what partial absorption means. A minimum consistency requirement is that no hypothesis that strictly satisfies the hint should be excluded as a result of the partial absorption process. A good process will exclude as many hypotheses as possible without violating this requirement.
Our analysis here dealt with the situation where the unknown function f strictly satisfies the hint, and strictly belongs to G. Relaxing these conditions is worth further investigation. It is also worthwhile to extend this work to cover real-valued functions, as well as average-case measures instead of the worst-case VC dimension. Finally, schedules for mixing examples off with examples of the hint in learning protocols are worth exploring.
Acknowledgment This work was supported by AFOSR Grant 92-J-0398 and the FeynmanHughes fellowship. The author wishes to thank Dr.Demetri Psaltis for a number of useful comments.
References Abu-Mostafa, Y. 1990. Learning from hints in neural networks. 1. Complex. 6, 192-1 98. Al-Mashouq, K.,and Reed, I. 1991. Including hints in training neural networks. Neural Comp. 3, 418427. Blumer, A., Ehrenfeucht, A., Haussler, D., and Warmuth, M. 1989. Learnability and the Vapnik-Chervonenkisdimension. 1.ACM 36,929-965. Feller, W. 1968. An Introduction to Probability Theory and Its Applications, Vol. 2. Wiley, New York. Fyfe, A. 1992. Invariance hints and the VC dimension. Ph.D. Thesis, Caltech. Hu, M. 1962. Visual pattern recognition by moment invariants. IRE Trans. Inform. Theory IT-8, 179-187. Minsky, M.,and Papert, S. 1988. Perceptrons, expanded edition. MIT Press, Cambridge, MA. Omlin, C., and Giles, C. 1992. Training second-order recurrent neural networks using hints. In Machine Learning: Proceedings of the Ninth lnternational Conference (ML-92), D. Sleeman and P. Edwards, eds. Morgan Kaufmann, San Mateo, CA. Suddarth, S., and Holden, A. 1991. Symbolic neural systems and the use of hints for developing complex systems. Intl. 1.Machine Stud. 35,291. Vapnik, V., and Chervonenkis, A. 1971.On the uniform convergence of relative frequencies of events to their probabilities. Theory Probabil.Awl. 16,264-280. Received 21 April 1992; accepted 15 July 1992.
This article has been cited by: 2. Eshel Faraggi, Bin Xue, Yaoqi Zhou. 2009. Improving the prediction accuracy of residue solvent accessibility and real-value backbone torsion angles of proteins by guided-learning through a two-layer neural network. Proteins: Structure, Function, and Bioinformatics 74:4, 847-856. [CrossRef] 3. J. Ghosn, Yoshua Bengio. 2003. Bias learning, knowledge sharing. IEEE Transactions on Neural Networks 14:4, 748-765. [CrossRef] 4. E.A. Rietman, S.A. Whitlock, M. Beachy, A. Roy, T.L. Willingham. 2001. A system model for feedback control and analysis of yield: A multistep process model of effective gate length, poly line width, and IV parameters. IEEE Transactions on Semiconductor Manufacturing 14:1, 32-47. [CrossRef] 5. P. Niyogi, F. Girosi, T. Poggio. 1998. Incorporating prior information in machine learning by creating virtual examples. Proceedings of the IEEE 86:11, 2196-2209. [CrossRef] 6. E.A. Rietman, D.J. Friedman, E.R. Lory. 1997. Pre-production results demonstrating multiple-system models for yield analysis. IEEE Transactions on Semiconductor Manufacturing 10:4, 469-481. [CrossRef] 7. Yaser S. Abu-Mostafa . 1995. HintsHints. Neural Computation 7:4, 639-671. [Abstract] [PDF] [PDF Plus] 8. Vladimir Vapnik , Esther Levin , Yann Le Cun . 1994. Measuring the VC-Dimension of a Learning MachineMeasuring the VC-Dimension of a Learning Machine. Neural Computation 6:5, 851-876. [Abstract] [PDF] [PDF Plus]
Communicated by Ralph Linsker
Redundancy Reduction as a Strategy for Unsupervised Learning A. Norman Redlich The Rockefeller University,2230 York Ave., New York, NY 10021 USA A redundancy reduction strategy, which can be applied in stages, is proposed as a way to learn as efficiently as possible the statistical properties of an ensemble of sensory messages. The method works best for inputs consisting of strongly correlated groups, that is features, with weaker statistical dependence between different features. This is the case for localized objects in an image or for words in a text. A local feature measure determining how much a single feature reduces the total redundancy is derived which turns out to depend only on the probability of the feature and of its components, but not on the statistical properties of any other features. The locality of this measure makes it ideal as the basis for a ”neural” implementation of redundancy reduction, and an example of a very simple non-Hebbian algorithm is given. The effect of noise on learning redundancy is also discussed. 1 Introduction Given sensory messages, for example, the visual images available at the photoreceptors, animals must idenhfy those objects or scenes that have some value to them. This problem, however, can be very tricky since the image data (e.g., photoreceptor signals) may underdetermine the scene data (e.g., surface reflectances) needed to find and idenhfy objects (Kersten 1990). In the case of very primitive organisms crude special purpose filters may suffice, such as the “fly detector” in frogs. But for more general object detection and for the reconstruction of physical scenes from noisy image data, some additional clues or constraints are needed. One type of clue is knowledge of the statistical properties of scenes and images (Attneave 1954, Barlow 1961, 1989). Such information can be used to recover physical scene data from noisy image data, as shown for example by Geman and Geman (1984). Barlow (1989) has also argued that such information is necessary for object recognition, since it allows objects to be discriminated from irrelevant background data. Also, since objects are encoded redundantly in sensory messages, knowing this redundancy can aid in their recognition. But how can an organism go about learning the statistical properties of sensory messages? And second, what is the most efficient way of sforNeural Computation 5,289304 (1993) @ 1993 Massachusetts Institute of Technology
290
A. Norman Redlich
ing this statistical knowledge? The enormity of these problems becomes obvious when one considers just how many numbers in principle must be learned and stored. In vision this amounts to storing the probability of every possible set of pixel values in both space and time. For a conservative estimate of this number for humans, assume one million cones sampled in space only-temporal sampling would add considerably to this. Then assume a grey scale of roughly 100, which is less than the number of contrast units that can be discriminated in bright light-and ignores luminance data. This gives 100’@“)~ooo v b l e images whose probabilities could not possibly be stored as 100 wo*wo numbers in the brain, which has no more than 1OI6 synapses. However, there are two very important properties of images which allow this number to be decreased enormously. The first and most obvious is noise: most images differ from each other only by noise or by symmetries, so there is no need to learn and store their individual probabilities. The second s i m p w n g property is that sensory message probabilities can often be derived from a far smaller set of numbers. This is the case when the set of probabilities P(Z) for the images I = ( 1 1 , h,l 3 , . . .l,}, with pixel values I,, can be factorized into a far smaller set of statistically independent probabilities for the subimages ( 1 1 , Izr4,. . .I,} as P ( I ) = P(Il)P(12)P(13). . .P(Zm). Thus, as Barlow (1989) has emphasized, the most efficient way to store the probabilities P(Z) would be to find a transformation, a factorial code, from the pixel representation I, to the statistically independent representation I, with smallest m. It can be demonstrated (Atick and Redlich 1990a) that this explains one purpose of the retinal transfer function, which approximately removes (second-order) statistical dependence from the optic nerve outputs { I , , Zzr1 3 1 . . .Im}. Finding a transformation to a factorial representation is an unsupervised learning problem that typically requires many learning stages. At each stage I assume that only the local probabilities P(I,) are measured, but as statistical independence is “increased, products of these give better approximations to the joint probabilities P ( I ) . To quantdy just how statistically independent a representation is at each stage, it is necessary to define a global learning measure L,which should be a function only of the local probabilities P(l,) (global denotes a property of the entire representation at each stage). Such a measure is defined here based on the redundancy,’ a quantity that is minimal only when the code is factorial. Learning a redundancy reducing transformation at each stage can be very difficult and may depend on the nature of the redundancy at that stage. In the retina, the greatest source of redundancy is due to ‘I use “redundancy reduction” to refer to statistical dependence between pixels.
This is not strictly speaking the only source of redundancy, which also can come from the uneven probability distribution of grayscale values. Nevertheless, I use the term “redundancy reduction” because it im lies an information preserving transformation (unlike, e.g., “entropy reduction”) ancfalso because the word “redundancy” has an intuitive appeal. The precise meaning of redundancy reduction here is a transformation which increases the learning measure L,to be defined in Section 2.
Redundancy Reduction Strategy
291
multiscale second-order correlations between pixels-corresponding to scenes being very smooth over large spatial and temporal regions-and this redundancy can be removed easily through a linear filter (Atick and Redlich 1991). But this is the exception, since in general quite complicated nonlinear coding is required (for some progress see, e.g., Barlow and Foldiak 1989; Redlich 1992). However, there is one common type of nonlinear redundancy reduction that is relatively straightforward to learn. This is the redundancy in images coming from strong correlations within sharply delineated features which are in turn weakly correlated with each other (the features can be spatially extended as long as they decouple from other parts of the image). The procedure for factorizing in this case is to look first for the subfeatures that are most tightly bound, and therefore are responsible for the most redundancy. These may then be pieced together in stages, until eventually a statistically independent set is found. What makes this much simpler than I expected is the existence of a completely local measure of how much an individual feature (or subfeature) contibutes to the global redundancy. By local I mean that this measure is only a function of the probabilities of the feature and its components, but not of the probabilities of any other feature or components. The locality of this feature measure also allows simple implementation of redundancy reduction through unsupervised ”neural” learning algorithms. One such non-Hebbian algorithm will be discussed here, and compared to some other unsupervised algorithms (von der Malsburg 1973; Bienenstock et al. 1982; Hinton and Pearlmutter 1986). The closest connection is with Hinton and Pearlmutter’s algorithm because their single-unit feature measure is mathematically related to the one here, though this is manifest only in a particular approximation. This connection is not surprising since (see,e.g., Hinton and Sejnowski 1983) their aim was also to learn statistical regularities. Some of the major distinctions between this work and theirs are the focus here on efficiency of storage and learning (on statistical independence) and also the insistence here on transformations which preserve information (see Section 6). To demonstrate the power of the present approach, I apply it to strip the redundancy from English text-to learn the text’s statistical properties. This example is used because we all know a fairly good solution to the problem: transform from the letter representation to the word representation. Of course to make the problem sufficiently difficult all clues to the solution, such as spaces between words, punctuation, and capitalization, are first eliminated from the text. The algorithm eventually segments the text just as desired into words and tightly bound groups of words. Although it is not the main purpose of this paper, I shall also indicate how useful the algorithm can be for recovering messages from noisy signals. This works best when the useful information is coded redundantly while the noise is random. It then turns out that the algorithm used here
A. Norman Redlich
292
finds only the useful portion of the input, and this will be demonstrated using noisy text. Finally, I should emphasize that my aim is not to find redundancy in language or to claim that words are learned or stored in the brain as found here. Instead, my ultimate motivation is to find an environmentally driven, self-organizing principle for the processing of visual images (and other sensory signals) to facilitate object or pattern identification (see Redlich 1992). So by "words" here I always wish to imply visual features, with letter positions in a text corresponding to pixel locations in an image and particular letters corresponding to image grayscale (or color) values. The next step of applying the algorithms derived here to visual images will appear in future papers.
2 Global Learning Measure C
Taking the English text example, the input consists of an undifferentiated stream of "pixel" values L = {a, b, c, . . .} (as in Fig. la) and the goal is to learn the probability functions P(Z) = P(ll ,l p , /J, . . .l,), with the subscript n denoting the position of letter 1, in the text. In practice, the aim is to learn P ( I ) for string length n roughly equal to the correlation length of the system. But even for n as small as 12 this in principle requires storing and updating as many as 2612 numbers. To find P(Z) more efficiently, at each stage letters will be grouped together into "words," which at first will be only pieces of real English words. Then at successive stages the new set of words W = {w~, w2,w3,. . .wm} will be built by combining the previous "words" into larger ones, among which will be real English words and also tightly correlated groups of real words (from now on quotes around words are dropped). At the very first stage P ( I ) = P(ll,l 2 , l 3 , . . . In) is very poorly approximated by the product of letter probabilities P ( l 1 ) P ( l ~ ) P ( l 3. .) .P(ln), but as the redundancy is reduced, products of word probabilities P ( w I ) P ( w $ ( w ~ ) . . . P ( W m ) give better and better approximations to P(1). To quantitatively measure how well P(I) is known at each stage, we can use information theory (see also Barlow 1961) to define a global learning measure (2.1) where HLis the entropy in the original letter code,
HL = -
cPI log(P1) IEL
(2.2)
Redundancy Reduction Strategy
293
and Hw/S is the word entropy per letter at a particular stage in learning:
Hw
= -
c
Pwlog(Pw)
WEW
s
=
CP,SW
(2.3)
with 3 the average of the word lengths sw.
a~iccwasheginningtogetverytiredofsittinghyhersisteron thehankandofhavingnothingtodoonceortw iceshehadpeep edintothehookhersisterwasreadingbutithadnopi~tures~r conversationsinitandwhatistheuseofahookthoughtalicew ithoutpicturesorconversationssoshewasconsider inginhe r o w n m i n d a s w e l l a s s h e c o u l d f o r t h a h o t d a y m ad e h e r f e e l v e r y sleepyandstupidwhetherthepleasureofmakingada isychai nwouldheworthlhetrouhleofgettingupandpickinglhedaini eswhensuddenlyawhiterahhitwithpinkeyesranclosehyher t h e r e w a s n o t h i n g s o v e r y r e m a r k a b l e i n t h a t n o r d i d a l i c e thi n k i Is o v e r y m u c h o u t o f I h e w a y t o h e a r t h e ra h h i t s a y to i t s e If o hdearohdear (a)
alice was b e g in n ing t o g e t very I i r e d o f s i t t ing b y her s i s t e r 0 n the h a n k and o f h a v ing n o th ing t o d o o n c e o r t w i c e she h a d p e e p e d in t o the b o o k her s is I e r was r e a d ing hut i th a d n o p i c t u r e s o r c o n ver s a t i o n s in i t and w h a t i s the u s e o f a h o o k th ough t alice with ou t p i c t u r e s fl r c o n vet s a t i o n s s o she was c o n s i d e ring in hero w n m in d a s w e I I a s she c ould for the h o I d a y m a d e her f e e I very s I e e p y and s t u p i d w he the r the p 1 e a s u r e o f m a k ing a d a i s y c h a in w ould h e w o r th the t r ou h I e o f g e t t ing u p and p i c k ins the d a i s i e s w h e n s u d d e n 1 y a w h i t e r a h h i t with p in k e y e s r a n c I o s e h y her the r e was n o th ing s o very r e m a r k a h I e in that n o r d i dalic e th ink i t s o very m u c h ou t o fthe w a y t o he a r the r a h h i t s a y t o i t s e lfohdearohdear
(h) alice was beg in n ing to g e t very t i r ed of s it 1 ing h y her s i s t e r on the b a n k and of ha v ing n o thins to d o on c e o r I w i c e shehad p e e p ed in to the h ook her s i s t e r was r e a d ing hut i th a d n o p i c t u r e s o r c o n ver s a t i on s in it and what i s the u s e of a h ook thought alice with ou t p i c I u r e s o r c on ver s a t i on s s o she was c on s i d e r ing in her o w n m in d a s w e 1 I a s shecould for the h o t d a y m a d e her f e e I very s I e e p y and s t u p i d w he thu the p I e a s u r e of m a k ing a d a i s y ch a in would he w o r th the t r ou b 1 e of g e t t ing u p and p i c k ing the d a i s i e I when s u d d e n ly a whi t e r a h h it with p i n k e y e s r a n c I0 s e h y her her e was n o thing s o very r e m a r k a h I e in that n o r d'i dalic e think it s o very m uch ou t ofthe way to he a r ther a h h its a y to it s elf o h d e a r o h d e a r (C)
Figure 1: A small sample of the text with all clues to the redundancy removed. In (a) single letters are treated as words, indicated by the spaces between them, and the entropy is HL = 4.16 bits. As the redundancy is reduced letters are combined into words, indicated by removing spaces between them. Only some of the redundancy reduction stages are shown in (b)-(fl, with the entropy per letter reduced to Hw/S = 3.75, 3.46, 2.84, 2.51, and 2.35 bits, respectively (the real word entropy per letter is Hw/S = 2.17 bits). Continued.
A. Norman Redlich
294
alice was heg in n ing toge I very t ir ed of s it t ing hy her s is ter on the h an k and of having nothing to do on ce on w ice shehad p e e p ed intothe h ook her s is ter was read ing hut i thad no p i c tur e s or conversation s in it and what is the us e of a h ook thoughtalice without p i c tur e s or conversation s s o shewas con side r ing in her own m ind as well asshecould for the h o t day made her fee I very s I e e p y and s t up id w he ther the pleas ure of mak ing ad a is y ch a in would he wor th the t r ou hle of getting up and p i ck ing the d a is i e s when suddenly a whiterahhit with p in key e s r an close by her therewas nothing s o very re mark a hle in that no r d i dalic e think it Y o very much outofthe way to hear therahhit say to it self ohdear ohdear
(d) alice was heginn ing toget verytiredof s it I ing hy her s is tcr on the hank and of having nothing to do on ce ortwice shehad peeped intothe hook her s is ter was read ing hut ithad no picture s or conversation s in it and what is the us e of a hook thoughtalice without picture s or conversation s so shewas consider ing in her own mind aswell asshecould for the hot day made her fee I very sleep y and stupid whether the pleas ure of making ad a is y ch a in wouldhe wor th the t rouhle of getting up and p ick ing the d a is i e s when suddenly a whiterahhit with p in key e s r an close hy her therewas nothing so very remark a hle in that no r d i dalic e think it so very much outofthe way to hear therahhit say to it self ohdear ohdear (a)
alice was heginning toget verytiredof sitting hy her s is ter onthehank and of having nothingtodo onceortwice shehad peeped intothe hook her s is ter was read ing hut ithad no picturesor conversation s in it and what is the us e of a hook thoughtalice without picturesor conversation s so shewas consider ing in her own mind aswcll asshecould for the hot Jay made her feelvery sleepyand stupid whether the pleas ure of making ad a is y ch a in wouldhe wor th the t rouhle of getting up and p ick ing the d a is i e s when suddenly a whiterahhit with p in key e s r an close hy her therewas nothing so very remark able in that no r d i dalic e think it so very much outofthe way to hear therahhit say toitself ohdear ohdear
(f )
Figure 1: Continued.
Initially, the set of words W is also the set of letters L so Hw/S = HLand C = 0, indicating that no learning has occurred. At the other extreme the word code is factorial for which2Hw/S = H, where H is the total entropy per letter of the text: (2.4)
[It is well known (Shannon and Weaver 1949) that H w / s 5 H with equality only when the words w in W become completely independent.] So 2This is true because the word code is reversible, .so H is invariant; for another type of reversible code see Redlich (1992).
Redundancy Reduction Strategy
295
the learning measure C starts out equal to zero and grows as redundancy is reduced until it approaches its maximum value
where Rc is the total redundancy in the text due to correlations between letters. If there are no correlations between letters then H = H L and Rc = 0. It is important to note that although C is bounded from above by Rc (Hw/S is bounded from below by H ) , it can go negative, so the system can in effect unlearn or increase redundancy. This happens when words (or letters) at one stage which are already independent of each other are mistakenly combined into new words. 3 Local Feature Measure F
Now that we have a global learning measure C, how do we go about finding the word/letter combinations W -+ W’, which increase C? For this purpose it is useful to have a local measure of how much an individual new word or feature increases L. Such a local feature measure F can be derived directly from C by calculating the change in C caused by including in W + W’a single new feature. Actually, since increasing C corresponds to decreasing Hw/S, we need to calculate the change in Hw/S. For extra clarity, let us first calculate the change in Hw/S when only two words in Ware combined to form a new word. Assume for simplicity that the current word set W still contains many single letters, including the letters 9’’ and ”n.” Let us see, as an example, how combining these letters into the particular word w = “in” changes Hw/S in 2.3. Following
w+
W’
where Pi,,, Pi, and P , denote the probabilities of the example word “in,” and of the letters ’?“ and “n.“ Also the “in“ terms have been separated out, so the sum C still runs over the old set W (assuming “i” and/or “n” still exist as independent elements in the set W’). To calculate the change HL/S’ - Hw/S, the new probabilities P‘ must be expressed in terms of the old probabilities P. This is easily accomplished using Pw = N w / N , where Nw = number of times word w appears, and N = total word count for the text (later N can be taken to infinity). After combining ”i” and “n” into “in,” the number N -+ N’= N - Nin, since every time the word “in” occurs it is counted as one word in W’, but was
A. Norman Redlich
296
counted as two words in W. Likewise, Ni Therefore,
-+
Ni - Nin,and N ,
-+
N , - Ni,.
Substituting these P’ into 3.1 gives
=
s/(1 -Pin)
(3.3)
so that
which defines the feature measure 3
The original average word length 3 has not been included in the definition of 3 because it is the same for all new features built out of W. As promised, this feature measure depends only on the local data Pi,, Pi, and P,. The local measure 3 can also be derived in general for new words of any length. We need to take into account the number m of old words making up the new word, as well as the number of times, m,, each old word w appears. For example if the new word “trees” is built out of the old words ”tr,” “e,” and “s” in W,then m = 4, while mt,= 1, m, = 2, m, = 1 giving Em, = m. The new probabilities P can then be derived from Pw and m, using counting arguments only slightly more complicated than before. Thus, denoting the new word by f, for feature,
Redundancy Reduction Strategy
297
N -t N‘ = N - (m - l)Nf, while N, + N , - m,Nf for w in the set W, of old words in f , and N , + N , otherwise. With these adjustments, the general feature measure defined by 3.4 is
F
=
F(P,,P, w E W,)
I
-[1 - (m- l)P,] log[l - (m - l)Pf]
(3.6)
This reduces to 3.5 in the special case f = “in,” W,= {”i ” ”n”) , m = 2 , mi = 1, m, = 1. To gain some intuition into just what statistical properties are measured by F it is useful to approximate 3 for the case Pf << P,, for all w E W,.In the case of English text, this is a good approximation in the first stages of redundancy reduction. The approximate F is I
Fppl(log(
)
n,, pf w, Ew -1)
(3.7)
using log base e. In this form, F bears its closest resemblance to the single-unit G-maximization measure of Hinton and Pearlmutter (1986). This is because the first term in 3.7, that is, neglecting -Pf,is like a single term in the Kullback information when the original probability estimate is statistical independence of inputs. The actual Kullback information requires summing such terms over all features f . For the feature f to significantly reduce redundancy, F needs to be strongly positive and for this, we see from 3.7 that the feature must have two statistical properties: First, the term in parentheses must be large and positive, which requires Pf >> n P w . This term plus one is the mutual information (Shannon and Weaver 19491, which measures how strongly the components that make up the feature? are correlated. A good example in English is “qu” which has high mutual information because “q” always predicts “u” so the “u’’ following “q” is redundant. The second requirement is that the feature be relatively frequent since Pf multiplies the mutual information. Otherwise, the feature could be highly self-correlated, but not common enough to significantly reduce the global redundancy. This is very important, since the mutual infor31n physics language, the feature is analogous to a bound state like an atom built out of protons and electrons. The mutual information is then proportional to the difference between the bound state (feature)energy and the sum of the energies of its components. This is the amount of energy that is gained by building the bound state (atom).
298
A. Norman Redlich
mation alone tends to favor very rare features composed of very rare elements. On the other hand, large P, alone is a dangerous criterion since there are many common features with small or even negative mutual information. Including these in the new set W actually increases the redundancy, since it effectively creates a correlated structure out of already statistically independent elements. In English text an example of a redundancy increasing feature is ”tte,” built out of “t” and “e.” 4 Experimental Results
F can be applied to devise a redundancy reduction algorithm for English text. One simple strategy would be to find at each stage the single new word which has the largest F,and thus find W from W. However, in practice it turns out that a far more time efficient approach is to find the set of new words with largest F,say 10 to 100 new words at each step. Another computational efficiency is gained by limiting the number m of component words to some small number, such as three or four. It turns out that for English text-likely also for many other ensembles-using only up to third-order correlations at each stage (m = 2,3)is sufficient, since larger words are most often composed of redundant subwords. To experimentally test how well this works I applied it to learning about 25 pages of a well known children’s book (Carroll 1865), chosen for its moderately sized vocabulary of roughly 1700 (real English) words. After eliminating all punctuation, capitals, and word spaces, the excerpt contained approximately48,000 characters. The letter entropy was found to be HL= 4.16 bits, while the entropy per letter for real English words is 2.17 bits. Figure l a shows a small piece of the text after it was stripped of any redundancy clues. Spaces are used between letters to indicate that they are being treated here as separate “words.” Figure lb-f then show the text sample in various stages of redundancy reduction. At each stage, when new words are built the spaces between their component words are eliminated. Figure 1 shows the results of using only second- and third-order joint probabilities at each step to find roughly 10 to 100 new words per stage. About 20 such stages were required to get the redundancy down to the Hw = 2.35 bits of Figure l e (only 5 of the 20 stages are actually shown in the figure) which is close to the real word entropy. Even computing all second- and third-order joint probabilities, these results represent only a few hours computation on a Macintosh computer. But the computation time and array storage needed can be reduced even further by calculating the joint probabilities only for a sample of possible new words, as will be discussed in the next section. Figure 2 shows the improvement possible using up to fourth-order probabilities; only the last stage is shown in the figure. Since there is only a small improvement over the third-order result, this demonstrates that fourth-order is not absolutely necessary.
Redundancy Reduction Strategy
299
alice was heginning toget verytiredof sitting hy hersister onthehan k and of having nothingtodo onceortwice shehad peeped intothe hook hersister was read ing hut ithad no pictures or conversation s in it and what is theuseof a hook thoughtalice without pictures or conversation s so shewas consider ing in her ow n mind aswell asshecould for the hot day made her feel very sleepy and stupid whether the plea sure of making ad a is y ch a in wouldhe wor th the trouhle of gettinypand p i c king the d a is ies when suddenly a whiterahhit with p in k eyes r an close hy her therewas nothing so very remark ahle in that n or did alice think it so very much outoftheway to hear therahhit saytoitself ohdear ohdear
Figure 2: The same sample of text, but using up to fourth-order correlation per stage instead of the third-order limit in Figure 1. Only the last stage is shown. It has Hw/S = 2.28 bits. Reviewing the results in Figure 1, one may note that some real English words, such as “daisies,” are not found, but this is due to the relatively small sample of English text used. In fact, the word “daisies” appears in the text only once so it would have been an error for it to quahfy as a redundant feature. However, the algorithm is superbly sensitive to redundant words which appear in the text as few as two or three times. Another thing to observe is that many groups of real words are combined into single features. Some of this reflects actual redundancy in English, for example “ofthe” is likely a truly redundant combination in English, but many of these, such as “whiterabbit” are only redundant for this sample text. Such real word groupings would have far lower redundancy (lower 37 in a much larger sample text which includes many different subjects and writing styles. The most significant success of the redundancy reduction algorithm is the segmentation of the text, which is almost always broken at the boundary between real words. This efficient segmentation corresponds to finding a cover W (Fig. le) of the entire sample with a small number of words-less than the number of real words. This is close to the smallest number of (approximately) statistically independent words. Such efficient segmentation would not have been found using an algorithm that chooses only high probability words.
5 Neural Implementation
In a ‘,neural” implementation of the algorithm used in Section 4, neurons, or dendrites, calculate the local data Pf and P,. Actually, only Pf needs to be calculated since the P, are computed by the previous stage and may be encoded in the neural output strengths. Finding F(Pf,Pw)still requires some computation, but (especiallyin 3.7) this reduces essentially
300
A. Norman Redlich
to computing logarithms! The real problem is not how to calculate Pf or F(Py), but how to search the space of possible features for those with largest 3. One option is to convert this search to one over a set of (continuous) synaptic weights and then apply gradient descent to maximize 3. This is the technique used by Hinton and Pearlmutter (1986) to maximize the Kullback information. Though its application to 3is somewhat different, I believe it might work, although I have not attempted it. Instead, I wish to explore here a more direct approach which avoids the convergence problems often associated with gradient descent. The simplest and most direct approach would be to exhaustively calculate Pr for all features of size 5 m. Of course m small enough to make this computationally feasible might be too small to discover the redundancy. But, there really is no need for an exhaustive search, since a prerequisite for large 3is large P f , and a more limited sampling will usually find these common features. Then only those common features with sufficiently large 3 need be kept. I now use this to develop a temporal search algorithm. Suppose first that there are a fixed number (smaller than needed for an exhaustive search) of feature neurons at each learning stage, which can be in one of two states, occupied or pee. Occupied neurons respond to one feature, and their job is to quickly calculate a good approximation for F. As soon as the occupied neuron discovers that 3 is below some constant threshold 3*, it becomes free and is available to test another feature. The neurons are mutually inhibiting so no two neurons can be occupied by the same feature. Also there is some ordering to decide which free neuron takes the next possible feature. To approximate 3,a neuron only needs an approximation for Pfsince the P, were calculated by the previous stage. How big Pf needs to be for F(Pf,P,) > F* depends on the probabilities of the input elements P, that make up the feature. In effect, the feature neuron uses a featuredependent threshold A(P,) for P f . (If the criterion were simple frequency of the feature, on the other hand, one would use a fixed threshold A for Pf.) Features that are built out of infrequent inputs w have lower threshold for P f , as can be seen most easily in 3.7. The final ingredient is an approximation for P f ( t ) at time t, where t = 0 is the time when the neuron first picks up its feature. For this, I make a very simple choice: If the feature has occurred only once at time t = 0, then for t > 0 approximate P f ( t )= l/t; if the feature occurs a second time at t = TI use for t > TI, P f ( t ) = 2 / t ; and if the feature has 41t should be noted that - log(P) has a very nice interpretation as the infomation in or improbability of the signal. If neurons have output strengths proportional to the information they carry, then the mutual information, one of the ingredients needed for 3,can be calculated through simple addition of neuronal outputs. This was suggested by Uttley (1979) as one of the attractions of using the mutual information to build a conditional probability computer (Singh 1966). Also, the idea that neurons signal improbability has been proposed by Barlow (19891, and there is evidence for this in the retina.
Redundancy Reduction Strategy
301
+
occurred n 1 times use P f ( t ) = n/t, which eventually approaches the true Pf for large n. If at any time P f ( t ) drops below the threshold A(P,), that is, F ( t ) drops below 7, then the occupied neuron is freed to search for other features. Of course, since for small t P f ( t ) may be a poor approximation, good features will be dropped occasionally, but these are likely to be picked back up again since they must be relatively frequent. On the other hand, the longer a neuron is occupied by a feature, the better the approximate Pf(f)becomes and the less susceptible to such errors. In fact, I have simulated this algorithm for the beginning stages of learning for the sample text used in Section 4,and it finds exactly the same set of features as does an exhaustive search, but it requires far less memory. One may also ask how this learning algorithm compares with other unsupervised “feature” detection algorithms. First, as has been discussed, this approach is related to Hinton and Pearlmutter’s: both favor features with large Pf and with Pf >> n P W ,although theirs is not guaranteed to find a factorial code. The greater distinction is between algorithms that use these criteria and algorithms of the type proposed by von der Malsberg (1973) and by Bienenstock et al. (1982). Those also favor features with large Pf,but they prefer features composed of elements with large P,. This may lead to features with small mutual information, and thus may include false background elements. For words in text this leads to poor segmentation, since many very tightly bound words are composed of relatively rare subwords. 6 Noise and Generalization
As mentioned in the introduction, desirable input information is often encoded redundantly (e.g., words in text) so redundancy can be used to distinguish true signal from noise. This is the case for example when the noise is not correlated with the true signal or with itself. Then the feature detection algorithm still finds the true signal redundancy-the true signal statistics-even though the total signal is noisy. To show this, consider an English text with random noise, that is, a certain fraction of letters, chosen randomly, are incorrect. Taking the same sample text used in Section 4, but with 1/13 letters randomly incorrect, I applied the same algorithm as before. The result, shown in Figure 3, is that only real words and word combinations are chosen by the algorithm, while noisy letters are ignored. So noise does not confuse the feature detection. Once the features have been found, the text can be restored by using the probabilities of the redundant words to predict the noiseincorrect letters, that is, to build Bayesian filters. It should be noted that in order to reconstruct the true text, one needs to know more than just the statistical properties of the noisy input messages. In the above example, one additionally needs to know that the
A. Norman Redlich
302
a licewashegiuningtogetverytiredofsittinghyhersisteron thehankandofharingnothitgtodoonceortwzkeshehadpeep e d i k t o z h e h o o k h e r s i s t e r w a s r e a d in g h u t i Ih a d n o p i I t u r e s hr chnversationsinitandwhvtistheuseofahookthoughtalicew ithoutpiqturesosconv~rsationssrshewasconsideringfnhe rownmiodasweclasshecouldforthehotdaymadeherfeelvery sleepyondstupidwhethfrthL?pleasureofmaiingadahsuchai gwouldheworththetrouhljofgsttingupanzpickingthedaqs~ eswhensuddenlyawhinerahhitwithpinkeyesranclosehahe rtherewasnothingsoderyremarkahleinthatnordidalicethi n e i t s o v e r j m u a h o u t o f t h e w a y t o h e a r t h e r c h h i t s a y toitselfo hdearohdeas (a) alice was beg i u ning toget verytiredof sitting by hersister onthe hank and of ha r ing no th it g todo onceortw z k e shehad peeped i k to z he hook hersister was read ing hut ithad no pi1 tures hr c h n ver sation s in it and w h v t i s thcuseof a hook thoughtalice without p i q tures o s conversation s s r shcwas considering f n her own m i o d a swec las shecould for the hot day madeher feelvery sleepyand stupi d whe th f r the plea sure of m a i ing a d a h sucha i g wouldhe worth theu ou h I j of gening up an z pick ing the d a q s i e s whensudden ly a w h in e rahhit with p in key e s r anc 10s e b a her therewas nothing s o der y remark able in that nor d idalice thine its over j m u a h outoftheway tohear ther c hhit say toitself ohdear oh d e a s (b)
Figure 3: Again, the same sample of text as in Figure 1, but with one out of 13 letters randomly incorrect. The noisy text before any redundancy reduction is shown in (a); it has HL = 4.26 bits, which is slightly higher than the original text because it is less correlated. One of the later stages in redundancy reduction is shown in (b);it has entropy per letter Hw/3 = 2.99 bits. Note that the noise does not confuse the algorithm into finding false words or word combinations. noise is random. In other words, one needs at least some outside knowledge or supervision. For example, mean squared filtering that uses the autocorrelator of an ensemble to filter out noise, can be implemented through a supervised perceptron-type algorithm (see Atick and Redlich 1990b). This leads to an important point: purely unsupervised learning based strictly on statistics, does not lead to conceptualization. This is due to the implicit assumption that every distinguishable input state potentially carries a different message. In conceptualizing, on the other hand, different input states which carry the same useful message are grouped together. This grouping requires some further knowledge that distinguishes signal from noise, or provides a measure of closeness on the signal space (Kohonen 19841, or provides active supervision as in perceptron learning. Also, the information that distinguishes between different members of a concept can be thrown away, as in noise filtering. Since this information reduction effectively lowers the number of input states, it also simplifies the problem of learning and storing statistics. So one challenge is to in-
Redundancy Reduction Strategy
303
corporate in the present redundancy reduction strategy a controlled or supervised information reduction. Some first steps in this direction have been taken by Linsker (1989) and by Atick and Redlich (1990a), both using the mutual information between the desired scene data and the noisy image signal (for a different application of redundancy reduction to supervised learning, see Redlich 1992).
Acknowledgments I thank J. Atick for his very perceptive comments on the manuscript. Also, this work was supported in part by a grant from the Seaver Institution and in part by the DOE DE-FG02-90ER40.542.
References Atick, J. J., and Redlich, A. N. 1990a. Towards a theory of early visual processing. Neural Comp. 2,308-320. Atick, J.J., and Redlich, A. N. 199Ob. Predicting ganglion and simple cell receptive field organizations. Int. J. Neural Syst. 1,305. Atick, J.J., and Redlich, A. N. 1991. Convergent algorithm for sensory receptive field development. Neural Comp. In press. Atick, J. J., and Redlich, A. N. 1992. What does the retina know about natural scenes? Neural Comp. 4,196-210. Attneave, F. 1954. Some informational aspects of visual perception. Psychol. Rev. 61,183-193. Barlow, H. B. 1961. Possible principles underlying the transformation of sensory messages. In Sensory Communication, W. A. Rosenblith, ed. MIT Press, Cambridge, MA. Barlow, H. B. 1989. Unsupervised learning. Neural Comp. 1,295-311. Barlow, H.B., and Foldiak, F! 1989. In The Computing Neuron. Addison-Wesley, New York. Bienenstock, E. L., Cooper, L. N., and M u m , F! W. 1982. Theory for the development of neuron selectivity: Orientation specificity and binocular interaction in visual cortex. I. Neurosci. 2,3243. Carroll, L. 1865. Alice in Wonderland. Castle, Secaucus. Eriksson, K., Lindgren, K., and Mansson, B. A. 1987. Structure, Context, Complexity, Organization, Chap. 4. World Scientific, Singapore. Geman, S., and Geman, D. 1984. Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. Trans. Patt. Anal. Machine Intell. PAMI-6, 72141. Hinton, G. E., and Pearlmutter, B. A. 1986. G-maximization: an unsupervised learning procedure for discovering regularities. In Neural Nehvorks for Computing, A l p Conference Proceedings, Snowbird, UT,J. S. Denker, ed. AIP Press, New York. Hinton, G. E.,and Sejnowski, T. J. 1983. Optimal perceptual inference. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, 448453.
304
A. Norman Redlich
Kersten, D., 1990. Statistical limits to image understanding. In Vision: Coding and Eficiency, C. Blakemore, ed. Cambridge University Press, Cambridge. Kohonen, T., 1984. Self Organization and Associative Memory, Springer-Verlag, Berlin. Kullback, S., 1959. Information Theory and Statistics. Wiley, New York. Linsker, R. 1989. An application of the principle of maximum information preservation to linear systems. In Advances in Neural Information Processing Systems, D. S. Touretzky, ed., Vol. 1, pp. 186-194. Morgan Kaufmann, San Mateo, CA. Redlich, A. N. 1992. Supervised factorial learning. Preprint. Shannon, C. E., and Weaver, W. 1949. The Mathematical Theory of Communication. The University of Illinois Press, Urbana. Singh, J., 1966. Great Ideas in lnformation Theory, Languageand Cybernetics, Chap. 16, Dover, New York. Uttley, A. M., 1979. Information Transmission in the Nervous System. Academic Press, London. von der Malsburg, C. 1973. Self-organization of orientation sensitive cells in the striate cortex. Kybernetik 14, 85-100. Received 12 December 1991; accepted 29 September 1992.
This article has been cited by: 2. Jim W. Kay, W. A. Phillips. 2010. Coherent Infomax as a Computational Goal for Neural Systems. Bulletin of Mathematical Biology . [CrossRef] 3. S. M. Boker, J. F. Cohn, B.-J. Theobald, I. Matthews, T. R. Brick, J. R. Spies. 2009. Effects of damping head movement and facial expression in dyadic conversation using real-time facial expression tracking and synthesized avatars. Philosophical Transactions of the Royal Society B: Biological Sciences 364:1535, 3485-3495. [CrossRef] 4. M.A. Sanchez-Montanes, F.J. Corbacho. 2004. A New Information Processing Measure for Adaptive Complex Systems. IEEE Transactions on Neural Networks 15:4, 917-927. [CrossRef] 5. J. Michael Herrmann. 2001. Dynamical systems for predictive control of autonomous robots. Theory in Biosciences 120:3-4, 241-252. [CrossRef] 6. Sepp Hochreiter , Jürgen Schmidhuber . 1999. Feature Extraction Through LOCOCODEFeature Extraction Through LOCOCODE. Neural Computation 11:3, 679-714. [Abstract] [PDF] [PDF Plus] 7. D. Obradovic , G. Deco . 1998. Information Maximization and Independent Component Analysis: Is There a Difference?Information Maximization and Independent Component Analysis: Is There a Difference?. Neural Computation 10:8, 2085-2101. [Abstract] [PDF] [PDF Plus] 8. Jean-Pierre Nadal , Nestor Parga . 1997. Redundancy Reduction and Independent Component Analysis: Conditions on Cumulants and Adaptive ApproachesRedundancy Reduction and Independent Component Analysis: Conditions on Cumulants and Adaptive Approaches. Neural Computation 9:7, 1421-1456. [Abstract] [PDF] [PDF Plus] 9. J. Gerard Wolff. 1995. Computing as compression: An overview of the SP theory and system. New Generation Computing 13:2, 187-214. [CrossRef] 10. Gustavo Deco, Bernd Schürmann. 1995. Learning time series evolution by unsupervised extraction of correlations. Physical Review E 51:3, 1780-1790. [CrossRef] 11. G. Deco , W. Finnoff , H. G. Zimmermann . 1995. Unsupervised Mutual Information Criterion for Elimination of Overtraining in Supervised Multilayer NetworksUnsupervised Mutual Information Criterion for Elimination of Overtraining in Supervised Multilayer Networks. Neural Computation 7:1, 86-107. [Abstract] [PDF] [PDF Plus] 12. A. Norman Redlich . 1993. Supervised Factorial LearningSupervised Factorial Learning. Neural Computation 5:5, 750-766. [Abstract] [PDF] [PDF Plus]
Communicated by Halbert White
Approximation and Radial-Basis-Function Networks Jooyoung Park Irwin W.Sandberg Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX 78712 USA
This paper concerns conditions for the approximation of functions in certain general spaces using radial-basis-function networks. It has been shown in recent papers that certain classes of radial-basis-function networks are broad enough for universal approximation. In this paper these results are considerably extended and sharpened.
1 Introduction
This paper concerns the approximation capabilities of radial-basis-function (RBF) networks. It has been shown in recent papers that certain classes of RBF networks are broad enough for universal approximation (Park and Sandberg 1991; Cybenko 1989). In this paper these results are considerably extended and sharpened. Throughout this paper, we use the following definitions and notation, in which N and R denote the natural numbers and the set of real numbers, respectively, and, for any positive integer Y, R' denotes the normed linear space of real r-vectors with norm 11 11. (., .) denotes the standard inner product in R'. LP(%'), Lm(%'),and C,(Rr), respectively, denote the usual spaces of R-valued maps f defined on 92' such that f is pth power integrable, essentially bounded, and continuous with compact support. With W c R', C(W) denotes the space of continuous %-valued maps d e fined on W.The usual LP and uniform norms are denoted by 11 . (Ip and 11. l m, respectively. The characteristic function of a Lebesgue measurable subset A of R' is denoted by 1 ~ The . convolution operation is denoted by "*," and the Fourier transform (Stein and Weiss 1971) of a Fouriertransformable function f is written as By a cone in X' we mean a set C C R' such that x E C implies that a x E C for all a 1 0. By a proper cone we mean a cone that is neither empty nor the singleton (0). The block diagram of a typical RBF network with one hidden layer is shown in Figure 1. Each unit in the hidden layer of this RBF network has its own centroid, and for each input x = (xl, x2, . . . ,xr), it computes the distance between x and its centroid. Its output (the output signal at one
i.
Neural Computation 5,305-316 (1993) @ 1993 Massachusetts Institute of Technology
306
JooyoungPark and Irwin W.Sandberg
Figure 1: A radial-basis-functionnetwork.
of the kernel nodes) is some nonlinear function of that distance. Thus, each kernel node in the RBF network computes an output that depends on a radially symmetric function, and usually the strongest output is obtained when the input is at the centroid of the node. Each output node gives a weighted summation of the outputs of kernel nodes. We first consider RBF networks represented by functions q : '8' 4 B of the form
where M E N is the number of kernel nodes in the hidden layer, Wi E W is the weight from the ith kernel node to the output node, x is an '1,and K is the common radially symmetric input vector (an element of 8 kernel function of the units in the hidden layer. Here zi E 92' and CY > 0 are the centroid and smoothing factor (or width) of the ith kernel node, respectively. We call this family S o ( K ) . Note that the networks in this family have the same positive smoothing factor in each kernel node. Families with a translation-invariant vector space structure are also often important. For example, networks are widely used in which the smoothing factors are positive real numbers as in So(K), but can have different values across kernel nodes. This family is the smallest vector space among those containing So(K) as a subset. We call this vector space
Approximation and Radial-Basis-Function Networks
S,(K). Its general element q : R'
-+
307
?J? is represented by
where M E N, oi > 0, wi E R, and zi E R' for i = 1,2,. . . ,M. For the sake of clarity and convenience, we consider only a onedimensional output space instead of outputs represented by multiple nodes as in Figure 1. The extension of our results to multidimensional output spaces is trivial. Notice that the kernel function K characterizes the families So(K) and S l ( K ) , and that each kernel node has its output derived from K indexed by two parameters (the centroid and smoothing factor), one for position and the other for scale. Ordinarily K is radially symmetric with respect to the norm 11 11 in the sense that llxll = llyll implies K ( x ) = K(y). However, as we shall see in the next section, radial symmetry of the kernel function K : R' R is needed in the development of only one of the approximation results in this study, Except where indicated to the contrary, radial symmetry of the kernel function K is not assumed. In Park and Sandberg (1991)it is shown that S o ( K ) is dense in LP(R'), p E [l,00) if K is an integrable bounded function such that K is continuous almost everywhere and Jsl,K ( x ) dx # 0. In Cybenko (1989)it is pointed out that a consequence of a generalization of a theorem due to Wiener is that the elements of a vector space related to & ( K ) are capable of approximating functions in L1 (8').The purpose of this paper is to report on a substantial sharpening of the results in Park and Sandberg (1991) and Cybenko (1989). -+
2 Approximation Results
As mentioned above, in Park and Sandberg (1991)it is shown that So(K) is dense in Lp(%'), p E [l,00) if K is an integrable bounded function such that K is continuous almost everywhere and Jslr K ( x ) dx # 0. Our first theorem concerns the p = 1 case; a necessary and sufficient condition is
given for approximation with So(K).
Theorem 1. Assuming that K : 3' -+ R is integrable, S,-,(K) is dense in L'(R') ifand only ifJR,K ( x )dx # 0. Proof. Suppose first that JR, K ( x ) dx # 0, and define ] = I Jslr K ( x )dxl. Let f E L' (3') and E > 0 be given. Since Cc(3') is dense in L' (R') (Rudin 19871, we can choose a nonzero fc E Cc(Rr)such that
Ilf - f c l l 1
Since fc has a compact support, there exists a positive T such that supp fc c [-T,TI'
(1)
Jooyoung Park and Irwin W. Sandberg
308
Choose a function Kc E Cc(Rr)such that
and
By Lemma 1 (in the appendix), we have llfc
- $u
*fClll
+
0
as
-+
0
Choose 0 > 0 such that llfc
- 4u
(3)
*fClll < 4 4
Note that &(a - .)fc(.) is Riemann integrable on [-T, TI', since $, and fc are each continuous and bounded. Define vn : 8' + R by
where the set of the form -T
{ai
+ -,2ilT
E R' : i = 1,2,. . . ,nr} consists of all points in [-T, TI'
. . . , -T
+
2irT1
,
, . . . ,ir = 1,2, . . . ,n
il i2,
Note that v,(a) is a Riemann s u m for &T,Tl, &(a - x)fc(x) dx, and that
Approximation and Radial-Basis-Function Networks
309
Thus, for any a E Sr,
vn(a) ( h* f c ) ( a ) as IZ 00 Since (&*fc) and the v, are dominated by an integrable bounded function with compact support, by the dominated convergence theorem +
Ar
+
as
1(4o *fc>(a) -vn(a>lda + 0
n
+
00
Thus, there is an N E N for which
Since
ijN
: %r + S defined by
So(K) is dense in L1(Sr). To show the "only if" part, we prove the contrapositive: Assume that JR,K(x)dx = 0. Then for anyf E L1(Sr) such that JR,f(x)dx A I > 0, there is no g E So(K) satisfymg [ I f - gill < J / 2 ,because
I I -glli ~
2
J , , v c ~-g(x)ldx= )
J f(x)dx=~ 821
for g E S o ( K ) . Thus So(K) is now not dense in L1(%'), which completes the proof. 0 Since the family So(K) is a proper subset of S1(K), the ',if" part of this theorem holds also with So(K) replaced by S I ( K ) . A family similar to S1(K) is considered in Cybenko (1989) with regard to the approximation of functions in L1(Rr); it is noted there that if K E L1(Sr) and JRr K ( x ) d x #
Jooyoung Park and Irwin W. Sandberg
310
0, then the family S 2 ( K ) consisting of functions q : R' form is dense in L'(R'): M
q(X) =
C Wi
*
K(tiX
+
--t
R of the following
yi)
i=l
where M E N , t; E R, and yi E R' for i = 1,.. .,M. The proof of this follows immediately from a generalization (Rudin 1973, Theorem 9.4) of a theorem due to Wiener. For the readers' convenience we state the theorem in the appendix. Here we make some pertinent observations: 1. Sl(K) defined above is a proper subset of S2(K), and even SI(K) is dense in L'(R') under the condition & , K ( x ) d x # 0. This can be easily shown: Assume to get a contradiction that there is an SO E R' such that SO) = 0 for all f E SI(K). Then K(as0) = 0 for all a > 0, since
Since K is continuous, the above implies that K(0) =
1 K(x) R'
dx = 0
which contradicts the nonzero-integral condition. The main difference between S1(K) and Sz(K) is the set from which the smoothing factors are drawn. In this connection, it is easy to see that our conclusion here, and also in Theorem 1, can be strengthened in that they hold if our ai > 0 and a > 0 conditions are replaced by the conditions that a; E S and a E S, where S is any subset of (0,oo) such that zero is a cluster point of S. Also, note that the denseness of SI(K) in L1(Rr) is a corollary of Theorem 1. 2. When K : R' -+ R is integrable, S I ( K ) is dense in L'(R') only if JslrK ( x ) dx # 0. The "only if" part of the proof of Theorem 1 shows this. The above observations give the following theorem:
Theorem 2. Assuming that K : R' if and only if JR, K ( x ) dx # 0.
+R
is integrable, SI(K) is dense in L1(Rr)
Up to this point our results concern the approximation of functions in L'(8') under the condition that JRr K ( x ) dx # 0. As shown above, this condition is necessary for approximation with So(K) or SI(K). A natural question that arises is whether the nonzero-integral condition is necessary for approximation in Lp(Rr), p E (1,m). We will see below that it is not necessary for p = 2.
Approximation and Radial-Basis-Function Networks
311
In the following theorem, attention is focused on kernel functions K : 8' .+ R with the property that for all M c R' with positive measure there is a ~7 > 0 such that K(a.)# 0 almost everywhere on some positive measure subset of M . We call such K pointable. We shall use the fact that the negation of this condition on K is that for some M of positive measure, K(P) = 0 almost everywhere on M for all (T > 0.
Theorem 3. Assuming that K : R' + R is a square integrablefunction, SI(K) is dense in L2(Rr) ifand only if K is pointable.
Proof. We make use of the following characterization of closed translationinvariant subspaces of L2(Rr), which is an easy modification of (Rudin, 1987, Theorem 9.17). 0 Lemma 2. Associate to each measurable set E c R' the linear space MEof all f E L2(Rr) such that = 0 almost everywhere on E . Then each M E is a closed translation-invariant subspace of L2(Rr),and every closed translation-invariant subspace of L2(Rr) is M Efor some E . Consider any K satisfying the indicated conditions, and suppose that the closure of S I ( K ) is not L2(Rr). Then, since this closure is translationinvariant, by Lemma 2 there is a measurable subset E of R' having positive measure such that
f =0
almost everywhere on E
for any f in the closure of
S1 (K).
a' exp(-2742, .))K(a.)= 0
In particular, almost everywhere on E
for any z E $2' and ~7 > 0. Thus, K(a.)= 0 almost everywhere on E for all a > 0, which contradicts our supposition. To show the "only if" part, we prove the contrapositive: Assume that there is a measurable set M c R' with positive measure such that
K(P) =0
almost everywhere on M
for all a > 0. Then for any f E L2(Rr) with ] g E S, ( K ) satisfying [If - g((2< ]/2, because
2 This completes the proof.
IlflMll2 =
l l f l ~ 1 1 2>
0,' there is no
I 0
'Here we use (1 . 112 to denote also the usual norm on the space of complex-valued square-integrable functionals.
JooyoungPark and Irwin W. Sandberg
312
A large class of kernel functions satisfies the conditions of pointability. For example, kernel functions K such that K # 0 almost everywhere on some ball centered at the origin are pointable. Note that this class includes functions K with JW, K ( x ) dx = 0. A result for the general LP(Rr) case along the lines of the "if part" of Theorem 2 is:
Proposition 1. With p that
E
(1,oo),let K : Rr + R be an integrublefunction such
and
Then SI(K) is dense in LP(gr). Proof. Suppose that S I ( K ) is not dense in LP(Rr). Then by the HahnBanach theorem (Rudin 19871, there exists a bounded linear functional A on Lp(Rr) such that A[the closure of S , ( K ) ] = (0)
(6)
but
w p ( n r )#)(01 By the Riesz representation theorem (Rudin 1987), A : L p ( a r )+ R can be represented by
for some function 8,f in Lq(Rr)? where q is the conjugate exponent of p defined by 1/p + l / q = 1. In particular, from equation 6
for any z E Rr and > 0. Define K : Rr -+ R and KO : Rr + R for u > 0 by
and
(-)
1- x K b ( x ) = 7K D
O
*The strategy of using the Hahn-Banach theorem together with representations of linear functionals was first used in the neural-networks literature in Cybenko (1989).
Approximation and Radial-Basis-Function Networks
313
Note that for any u > 0 and z in IR',
Since K E L1(IRr) and
JR,
K ( x )dx = 1, by Lemma 1 (in the appendix),
II&* g n -gAllq -, 0
as
(r
-,0
(8)
By 7 and 8, we conclude that gA is zero almost everywhere. This implies that A is the zero functional, which contradicts our supposition. Our focus has been on LP approximation. We next give a theorem concerning the uniform approximation of continuous functions on compact subsets of W. Theorem 4. Let K : IR' -,IR bean integrablefunction such that K is continuous Then &(K) is dense in C(W )with and such that K - I ( O ) includes no proper respect to t h e n o m 11 . Ilmforany compact subset WofIR".
Proof. Consider any compact subset W of IR'. Suppose that & ( K ) is not dense in C(W). Then proceeding as in the proof of Proposition 1, we see that there is a nonzero finite signed measure p that is concentrated on W and that satisfies
for any z E IR' and u > 0. With z E R', u > 0, and any function h E L1(Sr)n Lw(Rr) whose Fourier transform has no zeros4 (e.g., the gaussian function exp(-all. I[$) with a > 0), consider the integral
Note that
where lpl is the total variation of p. By equation 9 and Fubini's theorem (see, e.g., Rudin 19621,we have
- (' -
O
= Jr[k1K[
u
] d p ( y )] h(x)dx
3Since k(-w) equals the conjugate of K(w) for any w in R', this condition can be stated in terms of subspaces instead of cones. "ere we use a strategy along the lines of Hornick (1991, proof of Theorem 5).
JooyoungPark and Irwin W. Sandberg
314
By the change of variable x+y equation 10 is equivalent to
--+
x and Theorem 3:4.5 of Petersen (1983),
Note that h * p is integrable (by Theorem 1:4.5 of Petersen 1983). It is also essentially bounded, because Kh * P)(X)l
J,, lh(x - Y)I dlPl(Y)
I
5 Ilhllml/4(Rr) for almost all x E V. Consider the closed translation-invariant subspace I of L'( !I? defined ) as the L'-closure of S l ( K ) . By equation 11 and the essential boundedness of h * p, it easily follows that
for any f in I. Following the notation in Rudin (19621, define the zero set Z ( I ) of I to be the set of w where the Fourier transforms of all functions in I vanish. We claim that a nonzero element in Rr cannot be a member of Z ( I ) when K-'(0) includes no proper cone. Assume to get a contradiction that w # 0 and w E Z ( I ) . Then, using the definition of Z ( I ) ,
K
(5)(w)
= urexp(-2ai(z,w))K(aw) = o
for any z E R' and c > 0. This implies that &ow) = o
for all u > o
Since K is continuous, this means that k-'(O) includes the cone {uw E Rr : u 2 0}, which contradicts the cone condition. Thus, Z(1) is either the empty set or (0). In either case, by Theorems 7.1.2 and 7.2.4 of Rudin (19621, any integrable function from Rr to R with zero integral is a member of I. Thus, equation 12 gives
for any f in L'(Rr) wth JR,f(x)dx = 0.
Approximation and Radial-Basis-Function Networks
315
Note that the property 13 can hold only for h * p in the class of almost everywhere constant functions. But since h * p E L'(R) and zero is the only constant function in L1(Rr),we have
h*p =0
almost everywhere.
(14)
Since h has no zeros, by Theorem 2:2.2 of Petersen (1983) and Theorem 1.3.6 of Rudin (19621, equation 14 implies p = 0. This contradicts our supposition, and thus proves the theorem. A corollary of this theorem is that S1(K)is dense in C( W) for any compact subset W of $2' when the kernel K : 8' + R is integrable, continuous and satisfies Jsl,K ( x ) dx # 0. Finally, when K : R' -+ R is integrable and radially symmetric with respect to the Euclidean norm, K is also radially symmetric with respect to the Euclidean norm (Bochner and Chandrasekharan 1949, p. 69). In this setting, every K not equivalent to the zero element of L'(%') satisfies the cone condition of Theorem 4. This observation gives the following: Theorem 5. Let K : 92' + R be a nonzero integrable function such that K is continuous and radially symmetric with respect to the Euclidean norm. Then S1( K ) is dense in C( W)with respect to the norm (1 . lloofor any compact subset W of Rr.
3 Concluding Remarks
The results in this paper significantly improve previous results. In particular, we have given sharp conditions on the kernel function under which radial-basis-function networks having one hidden layer are capable of universal approximation on R' or on compact subsets of Rr. A related result concerning uniform approximation using the elements of So(K) with integrable K is given in Park and Sandberg (1991, p. 254). The results in Section 2 concern the approximation of real-valued functions. Approximations for complex-valued functions are also of interest. In this connection, it is a straightforward exercise to verify that Theorems 1-5 and Proposition 1 remain true if " K : Rr R" is replaced with the condition that K maps R' into the set C of complex numbers, LP(Rr) denotes instead the corresponding space of C-valued functions, the elements of C( W)are taken to be C-valued, and So(K) and S1(K)refer instead to the corresponding sets in which the weights wi are drawn from C. An important problem we have not addressed is that of determining the network parameters so that a prescribed degree of approximation is achieved. -+
Jooyoung Park and Irwin W. Sandberg
316
Appendix
L e m m a 1. Letf function such that
E
Lp(Rr), p
E
[I,oo),and let q ? ~ : Rr + R be an integrable
s,, $44
dx = 1
Define q?Jc : Rr + 2 b y
q?JN = (l/Er)q?J(x/E) for 6 > 0. Then
*f -flip -, Oas E
.+0.
Theorem 9.4 of Rudin (1973). If Y is a closed translation-invariant subspace of L' (W), and if
Z(Y)= nfEv{sE R' :?(s)
= 01
is empty, then Y = I-.'(%'). References Bochner, S., and Chandrasekharan, K. 1949. Fourier Transforms. Princeton University Press, Princeton. Cybenko, G. 1989. Approximation by superpositions of a sigmoidal function. Math. Control, Signals, Syst. 2, 303-314. Hornik, K. 1991. Approximation capabilities of multilayer feedforward networks. Neural Networks 4, 251-257. Park, J., and Sandberg, I. W. 1991. Universal approximation using radial-basisfunction networks. Neural Comp. 3, 246-257. Petersen, B. E. 1983. Introduction to the Fourier Transform and Pseudo-Differential Operators. Pitman, Marshfield, M A . Rudin, W. 1962. Fourier Analysis on Groups. Interscience Publishers, New York. Rudin, W. 1973. Functional Analysis. McGraw-Hill, New York. Rudin, W. 1987. Real and Complex Analysis, 3rd ed. McGraw-Hill, New York. Stein, E. M., and Weiss, G. 197l. Introducfion to Fourier Analysis on Euclidean Spaces. Princeton University Press, Princeton. Received 13 April 1992;accepted 3 September 1992.
'This lemma is used in Park and Sandberg (1991)where it is observed to be a slight modification of a theorem in Bochner and Chandrasekharan (1949). We have since found earlier proofs of the lemma (e.g., Petersen 1983, p. 72).
This article has been cited by: 2. Hong-Jiang Wang, Chi-Sing Leung, Pui-Fai Sum, Gang Wei. 2010. Kernel Width Optimization for Faulty RBF Neural Networks with Multi-node Open Fault. Neural Processing Letters 32:1, 97-107. [CrossRef] 3. María D. Perez-Godoy, Antonio J. Rivera, Francisco J. Berlanga, María José Del Jesus. 2010. CO2RBFN: an evolutionary cooperative–competitive RBFN design algorithm for classification problems. Soft Computing 14:9, 953-971. [CrossRef] 4. Arta A. Jamshidi, Michael J. Kirby. 2010. Skew-Radial Basis Function Expansions for Empirical Modeling. SIAM Journal on Scientific Computing 31:6, 4715. [CrossRef] 5. He Huang, Ji-cheng Bai, Ze-sheng Lu, Yong-feng Guo. 2009. Electrode wear prediction in milling electrical discharge machining based on radial basis function neural network. Journal of Shanghai Jiaotong University (Science) 14:6, 736-741. [CrossRef] 6. Masoud Mirmomeni, Caro Lucas, Masoud Shafiee, Babak N. Araabi, Elaheh Kamaliha. 2009. Fuzzy descriptor systems and spectral analysis for chaotic time series prediction. Neural Computing and Applications 18:8, 991-1004. [CrossRef] 7. Hui Li, Yuping Zhang, Haiqi Zheng. 2009. Gear fault detection and diagnosis under speed-up condition based on order cepstrum and radial basis function neural network. Journal of Mechanical Science and Technology 23:10, 2780-2789. [CrossRef] 8. Dongchuan Yu, Stefano Boccaletti. 2009. Real-time estimation of interaction delays. Physical Review E 80:3. . [CrossRef] 9. Yan Hua, Wei Ping, Xiao Xian-Ci. 2009. A method to improve the precision of chaotic time series prediction by using a non-trajectory. Chinese Physics B 18:8, 3287-3294. [CrossRef] 10. Antonio Sánchez-García, Patricio Muñoz-Esparza, José Luis Sancho-Gomez. 2009. A novel image-processing based method for the automatic detection, extraction and characterization of marine mammal tonal calls. Journal of the Marine Biological Association of the United Kingdom 1. [CrossRef] 11. M. Islam, A. Sattar, F. Amin, Xin Yao, K. Murase. 2009. A New Adaptive Merging and Growing Algorithm for Designing Artificial Neural Networks. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 39:3, 705-722. [CrossRef] 12. D. Achela K. Fernando, Asaad Y. Shamseldin. 2009. Investigation of Internal Functioning of the Radial-Basis-Function Neural Network River Flow Forecasting Models. Journal of Hydrologic Engineering 14:3, 286. [CrossRef] 13. Bor-Shyh Lin, Bor-Shing Lin, Fok-Ching Chong, Feipei Lai. 2009. Higher Order Statistics-Based Radial Basis Function Network for Evoked Potentials. IEEE Transactions on Biomedical Engineering 56:1, 93-100. [CrossRef]
14. Dong Nan, Wei Wu, Jin Ling Long, Yu Mei Ma, Lin Jun Sun. 2008. L p approximation capability of RBF neural networks. Acta Mathematica Sinica, English Series 24:9, 1533-1540. [CrossRef] 15. Alberto Guillén, Ignacio Rojas, Jesús González, Héctor Pomares, L. J. Herrera, O. Valenzuela, F. Rojas. 2007. Output value-based initialization for radial basis function neural networks. Neural Processing Letters 25:3, 209-225. [CrossRef] 16. Ali Gholipour, Caro Lucas, Babak N. Araabi, Masoud Mirmomeni, Masoud Shafiee. 2007. Extracting the main patterns of natural time series for long-term neurofuzzy prediction. Neural Computing and Applications 16:4-5, 383-393. [CrossRef] 17. Enrique Romero, Daniel Toppo. 2007. Comparing Support Vector Machines and Feedforward Neural Networks With Similar Hidden-Layer Weights. IEEE Transactions on Neural Networks 18:3, 959-963. [CrossRef] 18. Bor-Shyh Lin, Bor-Shing Lin, Fok-Ching Chong, Feipei Lai. 2007. Higher-Order-Statistics-Based Radial Basis Function Networks for Signal Enhancement. IEEE Transactions on Neural Networks 18:3, 823-832. [CrossRef] 19. Koldo Basterretxea, Jos Manuel Tarela, Ins del Campo, Guillermo Bosque. 2007. An Experimental Study on Nonlinear Function Computation for Neural/Fuzzy Hardware Design. IEEE Transactions on Neural Networks 18:1, 266-283. [CrossRef] 20. Alexey Kononov, Dries Gisolf, Eric Verschuur. 2007. Application of neural networks to traveltime computation. SEG Technical Program Expanded Abstracts 26:1, 1785. [CrossRef] 21. Masoud Mirmomeni, Caro Lucas, Babak Nadjar Araabi, Masoud Shafiee. 2007. Forecasting sunspot numbers with the aid of fuzzy descriptor models. Space Weather 5:8. . [CrossRef] 22. S. Padma, R. Bhuvaneswari, S. Subramanian. 2007. Application of soft computing techniques to induction motor design. COMPEL: The International Journal for Computation and Mathematics in Electrical and Electronic Engineering 26:5, 1324-1345. [CrossRef] 23. K. Schwab, M. Eiselt, P. Putsche, M. Helbig, H. Witte. 2006. Time-variant parametric estimation of transient quadratic phase couplings between heart rate components in healthy neonates. Medical & Biological Engineering & Computing 44:12, 1077-1083. [CrossRef] 24. Ali Gholipour, Babak N. Araabi, Caro Lucas. 2006. Predicting Chaotic Time Series Using Neural and Neurofuzzy Models: A Comparative Study. Neural Processing Letters 24:3, 217-239. [CrossRef] 25. D. Wedge, D. Ingram, D. Mclean, C. Mingham, Z. Bandar. 2006. On Global–Local Artificial Neural Networks for Function Approximation. IEEE Transactions on Neural Networks 17:4, 942-952. [CrossRef] 26. Nam Mai-Duy, Thanh Tran-Cong. 2005. An efficient indirect RBFN-based method for numerical solution of PDEs. Numerical Methods for Partial Differential Equations 21:4, 770-790. [CrossRef]
27. A. Krzyzak, D. Schafer. 2005. Nonparametric Regression Estimation by Normalized Radial Basis Function Networks. IEEE Transactions on Information Theory 51:3, 1003-1010. [CrossRef] 28. G. H. Schmitz, H. Puhlmann, W. Droge, F. Lennartz. 2005. Artificial neural networks for estimating soil hydraulic parameters from dynamic flow experiments. European Journal of Soil Science 56:1, 19-30. [CrossRef] 29. M. Arif, T. Ishihara, H. Inooka. 2004. Intelligent Learning Controllers for Nonlinear Systems using Radial Basis Neural Networks. Control and Intelligent Systems 32:2. . [CrossRef] 30. S.-H. Huh, J.-H. Park, I. Choy, G.-T. Park. 2004. Nonlinear uncertainty observer for AC motor control using the radial basis function networks. IEE Proceedings - Control Theory and Applications 151:3, 369. [CrossRef] 31. J. Gonzalez, I. Rojas, J. Ortega, H. Pomares, J. Fernandez, A. Diaz. 2003. Multiobjective evolutionary optimization of the size, shape, and position parameters of radial basis function networks for function approximation. IEEE Transactions on Neural Networks 14:6, 1478-1495. [CrossRef] 32. Ivan Tyukin , Cees van Leeuwen , Danil Prokhorov . 2003. Parameter Estimation of Sigmoid Superpositions: Dynamical System ApproachParameter Estimation of Sigmoid Superpositions: Dynamical System Approach. Neural Computation 15:10, 2419-2455. [Abstract] [PDF] [PDF Plus] 33. Yoshifusa Ito . 2003. Activation Functions Defined on Higher-Dimensional Spaces for Approximation on Compact Sets with and without ScalingActivation Functions Defined on Higher-Dimensional Spaces for Approximation on Compact Sets with and without Scaling. Neural Computation 15:9, 2199-2226. [Abstract] [PDF] [PDF Plus] 34. Irwin W. Sandberg. 2003. Gaussian radial basis functions and the approximation of input-output maps. International Journal of Circuit Theory and Applications 31:5, 443-452. [CrossRef] 35. E. Lavretsky, N. Hovakimyan, A.J. Calise. 2003. Upper bounds for approximation of continuous-time dynamics using delayed outputs and feedforward neural networks. IEEE Transactions on Automatic Control 48:9, 1606-1610. [CrossRef] 36. Haojian Xu, P.A. Ioannou. 2003. Robust adaptive control for a class of mimo nonlinear systems with guaranteed error bounds. IEEE Transactions on Automatic Control 48:5, 728-742. [CrossRef] 37. Miroslav K�rn�, Josef B�hm, Tatiana V. Guy, Petr Nedoma. 2003. Mixture-based adaptive probabilistic control. International Journal of Adaptive Control and Signal Processing 17:2, 119-132. [CrossRef] 38. Irwin W. Sandberg . 2003. Indexed Families of Functionals and Gaussian Radial Basis FunctionsIndexed Families of Functionals and Gaussian Radial Basis Functions. Neural Computation 15:2, 455-468. [Abstract] [PDF] [PDF Plus]
39. Michael Schmitt . 2002. Descartes' Rule of Signs for Radial Basis Function Neural NetworksDescartes' Rule of Signs for Radial Basis Function Neural Networks. Neural Computation 14:12, 2997-3011. [Abstract] [PDF] [PDF Plus] 40. Michael Schmitt . 2002. Neural Networks with Local Receptive Fields and Superlinear VC DimensionNeural Networks with Local Receptive Fields and Superlinear VC Dimension. Neural Computation 14:4, 919-956. [Abstract] [PDF] [PDF Plus] 41. Ming Zhang, Shuxiang Xu, J. Fulcher. 2002. Neuron-adaptive higher order neural-network models for automated financial data modeling. IEEE Transactions on Neural Networks 13:1, 188-204. [CrossRef] 42. J. Gonzalez, H. Rojas, J. Ortega, A. Prieto. 2002. A new clustering technique for function approximation. IEEE Transactions on Neural Networks 13:1, 132-142. [CrossRef] 43. Abelardo Errejon, E. David Crawford, Judith Dayhoff, Colin O'Donnell, Ashutosh Tewari, James Finkelstein, Eduard J. Gamito. 2001. Use Of Artificial Neural Networks In Prostate Cancer. Molecular Urology 5:4, 153-158. [CrossRef] 44. Irwin W. Sandberg. 2001. Gaussian radial basis functions and inner product spaces. Circuits Systems and Signal Processing 20:6, 635-642. [CrossRef] 45. Judith E. Dayhoff, James M. DeLeo. 2001. Artificial neural networks. Cancer 91:S8, 1615-1635. [CrossRef] 46. Irwin W. Sandberg. 2000. Constructive approximation of non-linear discrete-time systems. International Journal of Circuit Theory and Applications 28:2, 109-120. [CrossRef] 47. I.W. Sandberg. 2000. Time-delay polynomial networks and quality of approximation. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications 47:1, 40-45. [CrossRef] 48. I.W. Sandberg. 1999. Corrections to "separation conditions and approximation of continuous-time approximately finite memo. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications 46:10, 1305. [CrossRef] 49. I.W. Sandberg. 1999. Separation conditions and approximation of continuous-time approximately finite memory systems. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications 46:7, 820-826. [CrossRef] 50. N.B. Karayiannis. 1999. Reformulated radial basis neural networks trained by gradient descent. IEEE Transactions on Neural Networks 10:3, 657-671. [CrossRef] 51. T. Chau, A.K.C. Wong. 1999. Pattern discovery by residual analysis and recursive partitioning. IEEE Transactions on Knowledge and Data Engineering 11:6, 833-852. [CrossRef] 52. M.M. Polycarpou, J.Y. Conway. 1998. Indirect adaptive nonlinear control of drug delivery systems. IEEE Transactions on Automatic Control 43:6, 849-856. [CrossRef]
53. Irwin W. Sandberg. 1998. Separation conditions and approximation of discrete-time and discrete-space systems. Circuits Systems and Signal Processing 17:3, 305-320. [CrossRef] 54. Zi-Jiang Yang, Koichi Hirata, Teruo Tsuji. 1997. Servo motor adaptive speed control under angle-dependent disturbances using an RBF network. Electrical Engineering in Japan 119:4, 77-86. [CrossRef] 55. I.W. Sandberg, L. Xu. 1997. Uniform approximation of multidimensional myopic maps. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications 44:6, 477-500. [CrossRef] 56. Tin-Yau Kwok, Dit-Yan Yeung. 1997. Constructive algorithms for structure learning in feedforward neural networks for regression problems. IEEE Transactions on Neural Networks 8:3, 630-645. [CrossRef] 57. Irwin W. Sandberg, Lilian Xu. 1997. Uniform Approximation of Discrete-Space Multidimensional Myopic Maps. Circuits Systems and Signal Processing 16:3, 387-403. [CrossRef] 58. N.B. Karayiannis, G.W. Mi. 1997. Growing radial basis neural networks: merging supervised and unsupervised learning with network growth techniques. IEEE Transactions on Neural Networks 8:6, 1492-1506. [CrossRef] 59. A. Roy, S. Govil, R. Miranda. 1997. A neural-network learning theory and a polynomial time RBF algorithm. IEEE Transactions on Neural Networks 8:6, 1301-1313. [CrossRef] 60. Tin-Yan Kwok, Dit-Yan Yeung. 1997. Objective functions for training new hidden units in constructive neural networks. IEEE Transactions on Neural Networks 8:5, 1131-1148. [CrossRef] 61. Irwin W. Sandberg, Lilian Xu. 1996. Network approximation of input-output maps and functionals. Circuits Systems and Signal Processing 15:6, 711-725. [CrossRef] 62. A. Krzyzak, T. Linder, C. Lugosi. 1996. Nonparametric estimation and classification using radial basis function nets and empirical risk minimization. IEEE Transactions on Neural Networks 7:2, 475-487. [CrossRef] 63. Maxwell B. Stinchcombe . 1995. Precision and Approximate Flatness in Artificial Neural NetworksPrecision and Approximate Flatness in Artificial Neural Networks. Neural Computation 7:5, 1021-1039. [Abstract] [PDF] [PDF Plus] 64. Bjørn Lillekjendlie, Dimitris Kugiumtzis, Nils Christophersen. 1994. Chaotic time series. Part II. System Identification and Prediction. Modeling, Identification and Control: A Norwegian Research Bulletin 15:4, 225-245. [CrossRef] 65. Lipo Wang, Kiuju FuArtificial Neural Networks . [CrossRef]
Communicated by Richard Lippmann
A Polynomial Time Algorithm for Generating Neural Networks for Pattern Classification: Its Stability Properties and Some Test Results Somnath Mukhopadhyay Asim Roy Lark Sang Kim Sandeep Govil Department of Decision and Information Systems, Arizona State University, Tempe, AZ 85287 USA Polynomial time training and network design are two major issues for the neural network community. A new algorithm has been developed that can learn in polynomial time and also design an appropriate network. The algorithm is for classification problems and uses linear programing models to design and train the network. This paper summarizes the new algorithm, proves its stability properties, and provides some computational results to demonstrate its potential.
1 Introduction One of the critical issues in the field of neural networks is the develop ment of polynomial time algorithms for neural network training. With the advent of polynomial time methods (Karmarkar 1984; Khachian 1979 and others), linear programming has drawn increased attention for its potential for training neural networks in polynomial time (Glover 1990; Mangasarian et al. 1990; Bennett et al. 1992; Roy and Mukhopadhyay 1991; Roy et al. 1992). This paper presents the method of Roy and Mukhopadhyay (1991)and Roy et al. (1992) in summary form and proves its stability properties under translation and rotation of data points. Application of the method to some well-known learning problems is also shown. 2 A Linear Programming Method for Neural Network Generation -
The following notation is used henceforth. An input pattern is represented by the N-dimensional vector x , x = (XI,X2,.. . ,XN).The pattern space, which is the set of all possible values that x may assume, is represented by 0,. K denotes the total number of classes. The method is for supervised learning where the training set xl, x2, . . . ,x, is a set of sample patterns with known classification. Neural Computation 5,317-330 (1993) @ 1993 Massachusetts Institute of Technology
S. Mukhopadhyay et al.
318
The basic idea of this method is similar to the hypersphere method of Reilly et al. (1982), where a class region is "covered" by a set of hyperspheres of varying size. This method, however, generates the "covers" in a completely different way so as to obtain a polynomial time algorithm. Any complex nonconvex region can be covered by a set of elementary convex forms of varying size, such as hyperspheres and hyperellipsoids. Nonconvex covers can also be used when there is no problem in doing so. Let p elementary covers or masks (henceforth generally referred to as masks) be required to cover the region of a certain class P. To classify an input pattern as being in class P, it is necessary to determine if it falls within the area covered by one of the p masks. If the pattern space is two-dimensional and one of the p masks is a circle centered at ( a , b) with a radius r, a simple masking functionf (XI,X 2 ) = t.2 - [(XI-a)2 (X,can determine if an input pattern is a member of this mask. If (Xi, X;) is an input pattern, and
+
if f(Xi, Xi)2 0, then (Xi,Xi) is inside this mask and (Xi, Xi) belongs to class P; if f ( X { ,Xi) < 0, then (Xi Xi) is not inside this mask. In the learning phase, the procedure actually requires f ( x ) to be at least slightly positive [ f ( x ) 2 E] for membership in the mask and at least slightly negative v ( x ) 5 -61 for nonmembership. The membership criteria can be relaxed to f ( x ) 2 -6 in the testing phase, if warranted by numerical accuracy considerations. So, in general, let p k be the number of masks required to cover a class k, k = 1,. . . K. Let f f ( x ) , . . . , f L ( x ) denote these masking functions for class k. Then an input pattern x' will belong to class j if and only if one or more of its masks is at least slightly positive, and the masks for all other classes are at least slightly negative. Here, each mask will have its own threshold value E as determined during its construction. Expressed in mathematical notation, an input pattern x' is in class j , if and only if
' x ' ) >-. j
fi(
for at least one mask i,i = 1,.. . , p i
and
fik(x') < -6;
for all k # j and i = 1 . . . p k
(1)
If all masks are at least slightly negative (i.e., below their individual - E thresholds), the input pattern cannot be classified. If masks from two or more classes are at least slightly positive, then also the input pattern cannot be classified, though an indication can be given about the possible contenders. Unlike the hypersphere method of Reilly et al. (19821, the standard masking function (if there is to be one) of this method can possibly be any
Polynomial Time Algorithm
319
function that is linear in terms of the parameters to learn. For example, the following functions of the input vector x:
i=l
i=l
i=l j=i+l
and
i=l
i=l i=i+l
are acceptable masking functions since they are linear functions of the parameters to learn-the UP, bjs, cijs, and d. In experimental studies of this procedure, a quadratic function, as shown in 2, has been used as the standard mask. A quadratic function is able to generate many shapes, such as hyperspheres and hyperellipsoids. It can also generate nonconvex shapes, which is acceptable as long as they help to cover a class region properly.
2.1 Generating Masks Via Linear Programming. The procedure is outlined on the basis of using a standard mask. But that need not be the case. A priori, it is not known how many masks will suffice for any of the class regions. Thus, the first attempt is to define a single mask that will cover the whole class region. If that fails, the sample patterns in that class are generally split into two or more clusters by using a clustering procedure that produces a predetermined number of clusters and then attempts are made to define separate masks for each of these clusters. If that should fail, or if only some are masked, then the unmasked clusters are further split for separate masking until masks are provided for each ultimate cluster. The general idea is to define as large a mask as possible to include as many of the sample patterns within a class in a given mask as is feasibly possible, thereby minimizing the total number of masks required to cover a given region and obtaining the best generalization. When it is not feasible to cover with a certain number of masks, the region is subdivided into smaller pieces and masking is attempted for each piece. That is, the unmasked sample patterns are successively subdivided into smaller clusters for masking. At any stage of this iterative procedure, there will be a number of clusters to be masked. It might be feasible to mask some of them, thereby necessitating the breakup only of the remaining unmasked clusters. This “divide and conquer” procedure is heuristic. One can explore many variations of it.
S. Mukhopadhyay et al.
320
X"
x,'
Figure 1: Masking functions generate a multilayer perceptron. The feasibility of covering a set of sample patterns S, of class i with a mask f(x) is determined by solving the following linear program (LP): Minimize E s.t. f(xp) 2
f(xp) 5 E
2
E
-6
(4)
for all pattern vectors xp in the given set Si to be masked (xP E Si) for all pattern vectors xp in classes other than class i a small positive constant
If the LP solution is feasible and optimal, masking of set Si is complete and the LP solution to the parameters of f(x) defines the mask. If the LP is infeasible, masking of pattern set Si with mask f(x) is not feasible and it must be broken up for feasible masking with maskf(x). The infeasibility of an LP can be determined in polynomial time. 2.2 Constructing a Multilayer Perceptron from the Masks. The masking procedure actually generates a multilayer perceptron. Figure 1 shows how a multilayer perceptron is constructed from the masking functions when quadratics are used as masks. Suppose class A has k masks and
Polynomial Time Algorithm
321
class B has p. Each mask is evaluated in parallel at nodes Al through Ak and B1 through B,. For a given input pattern, the output of a node is 1 if the corresponding mask is at least slightly positive (2 and zero otherwise. A hard limiting nonlinearity (linear threshold unit) is used at these nodes. Class A hidden nodes A, through Ak are connected to the final output node A for the class and likewise for class B. The output of node A is 1 if at least one of the inputs is 1, it is zero otherwise, and likewise for node B. Again, hard limiting nonlinearities are used at these output nodes. An input pattern is in class A if the output of node A is 1 and that of node B is zero and vice versa. The masking function coefficients correspond to the connection weights and are placed on the connections between the input nodes and hidden layer nodes, as shown in the figure. The higher order product and power terms have been shown as direct inputs to the network. Actually, one more layer is needed at the input end to compute these higher order terms. In this constructive procedure, each hidden node is generated in an incremental fashion-there is no predesigned, fixed net within which learning takes place. Unlike classical connectionist learning, learning here is based on complete access to available information and on the ability to explicitly compare cases (by means of constraints). The net generated is shallow, is allowed to grow laterally only, and learning takes place in a single layer. As can be seen, the masking procedure constructs a restricted high-order polynomial network (Giles and Maxwell 1987) that is allowed to grow laterally in the hidden layer. This incremental growth of the net is similar in spirit to the GMDH method (Farlow 1984) that adds nodes when necessary.
4)
2.3 Outliers in Classification Problems. Many classification problems, by their very nature, generate patterns that fall outside their core class regions. All classification systems attempt to extract the core regions of each class. Because outliers are not identified a priori in the training set, their weeding out has to be performed either before or in conjunction with drawing of the boundaries. In this procedure, some weeding out is performed prior to and some during masking. Weeding out some outliers prior to masking is done in the following way. A clustering procedure that can produce exactly k clusters, such as K-means and hierarchical clustering, is used to divide the training set into k small groups (clusters) and all minority class members in each group are discarded as outliers. This procedure, in essence, assigns a small neighborhood, as represented by one of the k clusters, to the class of its majority members. The process of breaking up core class regions into smaller pieces, cleansing them of outliers, and then masking them is, in effect, an attempt to minimize classification error. The weeding out is actually performed in three steps with the average cluster size
S. Mukhopadhyay et al.
322
gradually decreasing to settle the allocation of unresolved neighborhoods. Sensitivity analysis is also performed in the first step. For example, if the training set has 700 samples, a clustering procedure can be used to divide the set into 100 clusters of average size 7. Suppose one such cluster has 5 class A and 2 class B patterns. The 2 class B patterns are in a minority in the cluster and are put in a candidate list of outliers. Suppose another cluster has 4 class A and 4 class B patterns. Since neither class commands a majority, the cluster cannot be assigned to any class. Such clusters are collected and broken up into smaller clusters in the next step. In the first step, the number of breakup clusters k is varied to test the sensitivity of the outlier diagnosis. Thus the training set is broken up twice into a different number of clusters and the consistency of each outlier diagnosis verified. For example, the training set of 700 samples may first be broken up into 80 clusters and an outlier candidate list developed. The same set may then be broken up into 120 clusters and a second outlier candidate list developed. Only those patterns that appear on both lists are discarded as outliers. In the second step, the unassigned clusters of step 1 are split into smaller clusters and a relaxed majority rule (> 50% membership defines majority class) used for territory allocation. So, for example, after the first step, out of the original 700 patterns, there might be 50 patterns unassigned, 70 thrown out as outliers and the remaining 580 retained for masking. In step 2, the 50 unassigned patterns may be split into 10 clusters of a reduced average size of 5. Suppose one such cluster has 3 class A and 2 class B patterns. The 2 class B patterns define a minority class and are discarded as outliers. The 3 class A patterns are retained for masking. In step 3, clusters can be arbitrarily assigned to a class if there is a membership tie-nothing remains unassigned. The LP algorithm is summarized below. Further details are available in Roy and Mukhopadhyay (1991) and Roy et al. (1992), including a proof of its polynomial time convergence. 2.4 The Algorithm. Phase I: Weed Out Outliers
step 1
1. Break up the training set into two different cluster sets A and B, each consisting of a different number of clusters, using a clustering procedure that can produce a predetermined number of clusters, such as K-means and hierarchical clustering. (The number of breakup clusters is determined, in turn,by the average cluster size chosen. We generally vary it between 6 to 8 to check sensitivity.) 2. In each cluster set, classify a pattern as an "outlier" if its class has less than one-third of the members, as a "core pattern" if its class has at least a two-thirds majority, and as "unassigned" otherwise.
Polynomial Time Algorithm
323
3. Compare these classifications across the cluster sets A and B. If a pattern is classified as a “core pattern” in one set and as an “outlier” in another, reclassify it as a “core pattern.” All other inconsistently classified patterns are reclassified as “unassigned.” 4. “Unassigned” patterns are carried over to step 2, the ”outlier” pat-
terns discarded, and the ”core patterns” retained for masking in phase 11. step 2
1. Break up the remaining “unassigned” patterns into smaller sized clusters using a clustering procedure that can produce k clusters and classify patterns as ”outlier,” “core pattern,” or “unassigned using a relaxed majority rule (over 50% only) for the ”core pattern” classification. A pattern is an “outlier” when its class possesses less than 50% of the members and is “unassigned” otherwise. 2. “Unassigned” patterns are carried over to step 3, the “outliers”
discarded and the “core patterns” retained for masking in phase 11.
Step 3 Repeat step 2 with the remaining “unassigned” patterns, splitting them into smaller sized clusters. In this step, the ”simple majority” rule is used to classify “core patterns,” classification ties are resolved arbitrarily, and no patterns remain “unassigned.” Phase ZI: Construct Masking Functions 0. For each class i, i = 1,...K, perform the following two steps: 1. For each unmasked cluster of class i, set up the LP in (4). Initially, a class has only one unmasked cluster consisting of all the “core patterns” from phase I, unless that is split up to start with. Solve the LPs. If all LP solutions are feasible and optimal, masking of class i is complete; go to step 0 for next class. Otherwise, when some or all LPs are infeasible, save all feasible LP solutions (masks), if any, and go to step 2. 2. Collect all unmasked clusters of class i and split them into smaller
clusters using a clustering procedure. Discard all resulting clusters that are very small in size (e.g., has less than 2% of the total sample patterns) as “outliers.” Return to step 1. If the problem has no noise, phase I can be skipped. Outliers remaining after phase I can cause some mask break up. Phase I1 can be rerun as a cleanup pass to obtain bigger masks producing better generalization. In this procedure, the basic purpose of clustering is to dissect the data and not to uncover “real” clusters.
S. Mukhopadhyay et al.
324
2.5 Stability Properties. To show the translation and rotation invariance properties of this method, the following have to be demonstrated: (1)stability of the steps of the algorithm that use clustering, and (2) stability of the linear programming solutions. Consider the following related pair of problems: Problem I .
set.
Minimize c f ( x p ) 2 p E GI f (xp) I - 6 , p E GZ E 2 a small positive constant €7
(5)
Problem Il. s.t.
Minimize E f ( x i R t ) 2 E , p E G1 f($R t ) I - E , P E Gz E 2 a small positive constant
+ +
(6)
where R is a rotation matrix, t a translation vector, GI the set of pattern vectors to be masked, GZ the set of pattern vectors belonging to classes other than that of GI, and f ( x p ) a linear mask. It is assumed that R is nonsingular and that the transpose of a rotation matrix is also its inverse. The following stability results are proven for a quadratic mask, f ( x ) = xTAx bTx c, where A is an N x N matrix, b a vector of size N, and c a scalar. Similar stability results can be shown for other linear masks.
+ +
Stability Theorem 1. The optimum objective function values for problems I and 11, when they are feasible, are the same and the solutions to the quadratic masks are equivalent. Proof. Let
A2 = RA1R-I b: = bfR-' - 2tTA2 cz = PA2t - bfR-'t + CI €2
=
€1
(7)
(8) (9) (10)
It is shown that if the solution (Al,b l , c l , q ) is optimal for problem I, then (Az,bz, CZ, Q), as defined in 7-10, is optimal for problem 11, and if (A2,bzrc2,c2) is optimal for 11, then ( A l ,bl,cl, q ) is optimal for I. Given the assumed optimality of the solutions ( A I b, l , c1, €1) and (Az,b2, c2, €2) for the respective problems (I and 111, they must also be feasible for these two problems. By substituting the solution 7-10 in problem 11, one obtains €1 2 €2, and by similar substitution in problem I, one obtains q 5 FZ. 0 Consequently, = €2 and the stated conclusions follow at once.
Corollary: Stability of Infeasible LPs. When problem I is infeasible, so is problem I1 and vice versa.
Polynomial Time Algorithm
325
Proof. Follows directly from Stability Theorem 1 by means of contradiction. If one problem is feasible, then so is the other. CI Rotation and/or translation of data points do not affect the relative distances between them. If the distance matrix remains unchanged, any hierarchical clustering method used to produce k clusters will produce the same set of clusters (Everitt 1980; Hartigan 1975), independent of rotation and/or translation of data points. Hence, if the algorithm generates the same set of pattern vectors for the clustering step at every stage of phases I and 11, and if they are split into the same number of clusters, the resulting clusters at each stage will be identical. Stability Theorem 2. The masking and clustering outcomes, when a hierarchical clustering method is used to produce k clusters, are unaffected by any rotation andlor translation of the training set, when all other conditions of the algorithm remain unchanged. Proof. Follows directly from theorem 1, its corollary, and from the obser0 vations on clustering outcomes noted above. 3 Computational Results
All results have been obtained from an implementation of this algorithm on the SAS system (SAS Manual 1988). The problems were solved on an IBM 3090 operating under the MVS operating system. For clustering, the average linkage method of hierarchical clustering (Sokal and Michener 1958) was used. 3.1 The Parity Problem. Rumelhart et al. (1986) tried the parity problem ranging in size from two to eight. They used a single hidden layer architecture which requires at least N hidden units to solve parity with N inputs. Muhlenbein (1990) reports that the bp algorithm never converged for N 2 6 with N hidden units, but converged when provided with 2N hidden units. Tesauro and Janssens (1988) also used 2N hidden units to overcome the local minimum problem. Table 1 shows the clustering and LP solution times for this procedure. Since it is a two-class problem, only one class was masked. Phase I was not used since there is no noise in this problem. The table shows that the algorithm is extremely fast. 3.2 The Symmetry Problem. Rumelhart et al. (1986) discovered that it can always be solved with only two hidden units in a single layer of hidden units. Table 2 shows the results of this procedure on the symmetry problem. Since it is a two-class problem, only one class was masked. It is solved in all cases with a single hidden unit (mask) and the LP solution times are close to zero. A masking function with linear and square terms only was used.
S. Mukhopadhyay et al.
326
Table 1: Solution Times for the Parity Problem
N, No. of bits
Clustering time (sec)
2 3 4 5 6 7 8
0 0.05 0.05 0.16 0.65 1.95 5.85
No. of masking functions
LP time (sec) 0 0 0 0 0 13 40
1 2 2 3 7 14 28
Table 2: Solution Times for the Symmetry Problem N, Clustering LP No. of No. of bits time (sec) time (sec) masking functions
2 3 4 5 6 7 8
-
0 0
0 0 0 0
0
1 1 1 1 1 1 1
3.3 Overlapping Gaussian Distributions. An obvious question about this method is, how well would the outlier detection heuristic work on classes with dense overlap? To test the heuristic under those circumstances, the following two problems were set up.
Problem I: The I-I Problem. A simple two-class problem where both classes are described by gaussian distributions with different means, and with covariance matrices equal to identity I. A four-dimensional problem with mean vectors [OOOO]and [1111] was tried. The Bayes error is about 15.2% in this case. Problem ZI: The I 4 Problem. A two-class problem where both classes are described by gaussian distributions with zero mean vectors and covariance matrices equal to I and 41. The optimal classifier here is quadratic, and Bayes error for a four-dimensional problem is 17.64% and for an eight-dimensional problem is 9%. Both problems were tried with randomly generated training sets of different sizes. Tables 3 and 4 show the results. Since both are two-class
12
10 13
16 16 16
23 27 27
M = ml,m2,m3
M = 6,.5,3
M = 7,5,3 M = 8,S,3
M=6,S M = 7,s M=8,5
M =6,s M =7 3
M = 8,s
phase I
No. of outliers found in phase I
Cluster sizes
Clustering + LP solution time (sec)
10.1 10.4 10.1 11.98 11.98 11.98
20
20
0 0
76.76
23
Number of Training Patterns (n = 180)
0 0 0
Number of Training Patterns (n = 120)
0 0 0
1 1 1
1 1 1
1 1 1
hCti0nS
No. of Masking
Number of Training Patterns (n = 60)
found in phase 11
No. of outliers
Table 3 LP Solution Ties and Results for Problem I (Bayes Error-15.2%)
21 17.1 17.1
18.6 18.6 18.6
23.6 20 2.5
Error (%)
17.8
17
17
Error (%) of SAS linear disaiminant function
M=7,5 M = 8,5
M=6,5
M = 7.5 M = 8,5
M = 6,5
Cluster sizes M = ml,m2,m3 phase I No. of outliers found in phase I1
Clustering + LP solution time (sec) No. of Masking functions
0 0
0 18.44
18.4 18.52
38 49 63 15 20 3
837.8 379.5 484.6
1 1 1
Number of Training Patterns (n = 4001,eight dimensions (Bayes error, 9%)
29 29 29
10.9 12.9 12
19 19 19
Error (%)
Number of Training Patterns ( n = 180),four dimensions (Bayes error, 17.64%)
No. of outliers found in phase I
Table 4 LP Solution limes and Results for Problem II
12.5
22.08
Error (%) of SAS linear discriminant function
Polynomial Time Algorithm
329
problems, only one of the classes need be masked. As shown, a single hidden node (mask) suffices for all cases. For both problems, the error rate generally decreases as the training set size is increased and tends to the theoretical minimum Bayes error rate. A randomly generated test set of 400 examples was used for each problem. The table entries M = 7, 5, 3, etc. show the average cluster size that was used in each step of the three-step phase I procedure. Tables 3 and 4 also show the error rates of the SAS linear discriminant function on these same problems. For problem 11, the SAS linear discriminant function was provided with squares of the input values as additional inputs. 4 Conclusions
This paper describes a new algorithm that uses linear programming to find the weights of a network. Its advantages are that it can both design and train a net in polynomial time. The network design issue has often been overlooked by the neural network community. But a true neural network algorithm should be able to both design and train a network in polynomial time. This classifier, however, does not provide estimates of Bayesian probabilities and is currently implemented using SAS.
Acknowledgment This research was supported by the National Science Foundation Grant IRI-9113370.
References Bennett, K. P., and Mangasarian, 0. L. 1992. Neural network training via linear programming. In Advances in Optimization and Parallel Computing, P. M. Pardalos, ed. North Holland, Amsterdam. Everitt, B. S. 1980. Cluster Analysis, 2nd ed. Heinemann Educational Books, London. Farlow, S. 1984. Self-organizing Methods in Modeling. Marcel Dekker, New York. Giles, C. L., and Maxwell, T. 1987. Learning, invariance, and generalization in high-order networks. Appl. Optics 26(4),972-974,978. Glover, F. 1990. Improved linear programming models for discriminant analysis. Decision Sci. 21(4), 771-785. Hartigan, J. A. 1975. Clustering Algorithms. John Wiley, New York. Karmarkar, N. 1984. A new polynomial time algorithm for linear programming. Combinatorica 4,373-395. Khachian, L. G. 1979. A polynomial algorithm in linear programming. Dokl. Akad. Nauk SSR 244(5), 1093-1096; Soviet Math. Dokl. 20, 191-194.
330
S. Mukhopadhyay et al.
Mangasarian, 0. L., Setiono, R., and Wolberg, W. H. 1990. Pattern recognition via linear programming: Theory and application to medical diagnosis. In Large-Scale Numerical Optimization, T. F. Coleman and Y. Li, eds., pp. 22-30. SIAM, Philadelphia. Miihlenbein, H. 1990. Limitations of multilayer perceptrons-Steps towards genetic neural networks. Parallel Comput. 14(3), 249-260. Reilly, D. L., Cooper, L. N., and Elbaum, C. 1982. A neural model for category learning. Biol. Cybern. 45, 35-41. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing: Explorations in Microstructure of Cognition, Vol. 1: Foundations, D. E . Rumelhart and J. L. Mdlelland, eds., pp. 318-362. MIT Press, Cambridge, MA. Roy, A., and Mukhopadhyay, S. 1991. Pattern classification using linear programming. ORSA J. Comput. 3(1), 66-80. Roy, A,, Kim, L. S.,and Mukhopadhyay, S. 1992. A polynomial time algorithm for the construction and training of a class of multilayer pemeptrons. Neural Nefwrks, in press. SAS Institute Inc. 1988. SAS Manual, Cary, NC. Sokal, R. R., and Michener, C. D. 1958. A statistical method for evaluating systematic relationships. Univ. Kansas Sci. Bull. 38, 1409-1438. Tesauro, G., and Janssens, R. 1988. Scaling relationships in backpropagation learning. Complex Syst. 2, 39-44. Received 1 November 1991; accepted 1 September 1992.
This article has been cited by: 2. Y. Takahashi. 2000. A mathematical solution to a network construction problem. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications 47:2, 166-184. [CrossRef] 3. Ling Zhang, Bo Zhang. 1999. A geometrical representation of McCulloch-Pitts neural model and its applications. IEEE Transactions on Neural Networks 10:4, 925-929. [CrossRef] 4. C. Citterio, A. Pelagotti, V. Piuri, L. Rocca. 1999. Function approximation-fast-convergence neural approach based on spectral analysis. IEEE Transactions on Neural Networks 10:4, 725-740. [CrossRef] 5. J. Manuel Torres Moreno, Mirta B. Gordon. 1998. Efficient Adaptive Learning for Classification Tasks with Binary UnitsEfficient Adaptive Learning for Classification Tasks with Binary Units. Neural Computation 10:4, 1007-1030. [Abstract] [PDF] [PDF Plus]
Communicated by Fernando Pineda
Neural Networks for Optimization Problems with Inequality Constraints: The Knapsack Problem Mattias Ohlsson Carsten Peterson Bo Soderberg Department of Theoretical Physics, University of Lund, Shegatan 14A, $22362 Lund, Sweden
A strategy for finding approximate solutions to discrete optimization
problems with inequality constraints using mean field neural networks is presented. The constraints x I 0 are encoded by xO(x) terms in the energy function. A careful treatment of the mean field approximation for the self-coupling parts of the energy is crucial, and results in an essentially parameterfree algorithm. This methodology is extensively tested on the knapsack problem of size up to lo3 items. The algorithm scales like NM for problems with N items and M constraints. Comparisons are made with an exact branch and bound algorithm when this is computationally possible (N I 30). The quality of the neural network solutions consistently lies above 95% of the optimal ones at a significantly lower CPU expense. For the larger problem sizes the algorithm is compared with simulated annealing and a modified linear programming approach. For “nonhomogeneous“ problems these produce good solutions, whereas for the more difficult “homogeneous” problems the neural approach is a winner with respect to solution quality and/or CPU time consumption. The approach is of course also applicable to other problems of similar structure, like set covering. 1 Background
Feedback artificial neural networks (ANN)have turned out to be powerful in finding good approximate solutions to difficult combinatorial optimization problems (Hopfield and Tank 1985; Peterson and Soderberg 1989; Peterson 1990; Gislkn et al. 1989, 1991). The basic procedure is to map the problems onto neural networks of binary (Ising spin) or K-state (Potts spin) neurons with appropriate choice of energy functions, and then to find approximate minima of the energy using mean field theory (MFT) techniques. In this way essentially “black box” procedures emerge. The application areas dealt with in Hopfield and Tank (1985) and Peterson and Siiderberg (1989), Gislkn et al. (1989,1991) (traveling salesman, Neural Computation 5,331-339 (1993) @ 1993 Massachusetts Institute of Technology
M. Ohlsson, C. Peterson, and B. Soderberg
332
graph partition and scheduling) are characterized by global equality constraints, which can be implemented as quadratic penalty terms. These contain self-interaction parts (diagonal terms), which can be balanced by counterterms to assure reliable MFT dynamics. However, in many real-world optimization problems, in particular those of resource allocation type, one has to deal with inequalities. The objective of this work is to develop a mapping and MFT method to deal with this kind of problem. As a typical resource allocation problem we choose the knapsack problem for our studies. Although artificial, we feel it is a realistic enough test bed. A crucial ingredient in our approach is to avoid self-couplings by a proper MFT implementation of the constraint terms. 2
T h e Knapsack Problem
In the knapsack problem one has a set of N items i with associated utilities ci and loads ski. The goal is to fill a "knapsack" with a subset of the items such that their total utility, N
u = ccisj
(1)
i=l
is maximized, subject to a set of M load constraints, N
defined by load capacities bk. In equations 1 and 2 si are binary ( 0 , l ) decision variables, representing whether or not item i goes into the knapsack. The variables (ci, ski, and bk) that define the problem are all real numbers. We will consider a class of problems, where akj and ci are independent uniform random numbers on the unit interval, while bk are fixed to a common value b. With b = N / 2 , the problem becomes trivial-the solution will have almost all si = l. Conversely, with b << N/4, the number of allowed configurations will be small and an exact solution can easily be found. We pick the most difficult case, defined by b = N/4. The expected number of used items in an optimal solution will then be about N / 2 , and an exact solution becomes inaccessible for large N. In the optimal solution to such a problem, there will be a strong correlation between the value of c; and the probability for Sj to be 1. With a simple heuristic based on this observation, one can often obtain near-optimal solutions very fast. We will therefore also consider a class of more difficult problems with more homogeneous ci distributions-the extreme case is when ci are constant, and the utility proportional to the number of items used. We note in passing that the set covering problem is a special case of the general problem, with random aki E (0, l}, and bk = 1. This defines a
Optimization Problems with Inequality Constraints
333
Figure 1: (a) The sigmoid g(x; T) of equation 4. (b) The penalty function x 0 ( x ) of equation 5. comparatively simple problem class, according to the above discussion, and we will stick to the knapsack problem in what follows. 3 Neural Network Formulation and Solution Strategy
3.1 Neural Mapping. We start by mapping the problem defined in equations 1 and 2 onto a generic neural network energy function E,
where ip is a penalty function to ensure that the constraint in equation 2 is fulfilled. The coefficient Q governs the relative strength between the utility and constraint terms. For equality constraints an appropriate choice of @ ( x ) would be @ ( x ) = x2. Having inequalities we need a @ ( x ) that penalizes only configurations for which x 2 0. One possibility is to use a sigmoid, @ ( x ) = g(x; 7') (see Fig. la), I
g(x; T ) = z[l+ tanh(x/T)]
(4)
This option has the potential disadvantage that the penalty is the same (= 1) no matter how badly the constraints are violated. An alternative that gives penalty in proportion to the degree of violation is @(x) =xQ(x)
(5)
This function (see Fig. lb) has the additional advantage that no extra parameter like the temperature T in the sigmoid is needed. The slope of ip is implicitly given by Q in equation 3. The xO(x) alternative consistently gives better performance and is used throughout this paper.
M. Ohlsson, C. Peterson, and B. Soderberg
334
3.2 Mean Field Dynamics. We want to minimize equation 3 with the mean field approximation (MET), which has turned out to be very powerful for other optimization problems (Peterson and Soderberg 1989; Peterson 1990; GislCn et al. 1989, 1991). Due to the nonpolynomial form of the constraint terms (equations 4 and 5 ) special care is needed when implementing the MFT approximation. Recall that the MFT approximation consists of replacing the binary variables si with mean field variables at temperature T, vi = ( s i ) ~and solving the MFT equations,
by iteration. In problems with equality constraints implemented by quadratic penalty terms the diagonal pieces are compensated for by adding appropriate self-coupling terms. Such a procedure is not trivial in this case of strongly nonlinear constraint penalties. Instead, we avoid self-couplings altogether, by replacing h’E/dvi with the difference in E computed at vi = 1 and vi = 0, respectively. One obtains
Equations 6 and 7 are solved iteratively by annealing in T. To avoid small final constraint violations, we employ a progressive constraint term, (Y c( 1/T. This means that the slope of xO(x) increases during convergence. We will present a standardized scheme below when testing the algorithm numerically. The number of computational steps for solving equations 6 and 7 scales like NMn7, where n, is the number of iterations needed for convergence, which turns out to be fairly problem size independent [as was observed in other MET approaches to optimization problems (Peterson and Soderberg 1989; GislCn et al. 1989, 199111. A factor N has been gained by “recycling” the sums appearing in the argument of @these are saved and need not be completely recomputed for each update of vi. 3.3 High-T Fixpoints and Critical Temperature. At a high temperature T, the system will approach a fixpoint with all vi close to 1/2 (see Fig. 2). With random aki and ci on [0,1],and fixed bk = b, two distinct types of high-T behavior emerge.
With b well above b d t E N/4, all constraints are safe at high T, and the system is stuck at a trivial fixpoint, Vj = g(Ci; T ) . 0
With b instead well below befit, all constraints are violated at high T, and the trivial fixpoint is instead Vj = g(Cj - a! Ck ah; T ) .
Optimization Problems with Inequality Constraints
Figure 2 Evolution of rand[O.45,0.55].
{ v i } for an
N
335
= M = 40 knapsack problem with c, =
In both cases a statistical analysis shows that vi remain close to 1/2 for
Thus, in the case at hand of b
=
bht, a suitable starting point for the
annealing will be T x 10. 4 Other Approaches
To see how well our MlT algorithm works we need to compare it with other approaches. For reasonably small problem sizes it is feasible to use an exact algorithm, branch and bound, for comparison. For larger problem sizes, one is confined to other approximate methods, simulated annealing (Kirkpatrick et al. 19831, greedy heuristics, and linear programming based on the simplex method (Press et al. 1986). Branch and Bound (BB): The knapsack problem is NP-complete, and the effort to find the optimal solution by brute force scales like 2N. Using a branch and bound tree search technique one can reduce the number of computational steps. This method consists in going down a search tree,
M. Ohlsson, C. Peterson, and 8. Soderberg
336
checking bounds on constraints or utility for subtrees, thereby avoiding unnecessary searching. In particular for nonhomogeneous problems, this method is accelerated by ordering the cis according to magnitude: c1
> c2 >
"'
> CN
(9)
For problems where the constraints are "narrow" (b not too large) this method can require substantially lower computation needs. However, it is still based on exploration and it is only feasible for problem sizes less than M = N FZ 30 - 40. Greedy Heuristics (GH): This is a simple and fast approximate method for a nonhomogeneous problem. Proceeding from larger to smaller ci (cf. equation 91, collect every item that does not violate any constraint. This method scales like NM. Simulated Annealing (SA): Simulated annealing (Kirkpatrick et al. 1983) is easily implemented in terms of attempted single-spin flips, subject to the constraints. Suitable annealing rates and other parameters are given below. This method also scales like NM times the number of iterations needed for thermaliation. Linear Programming with Greedy Heuristics (LP):Linear programming based on the simplex method (Press et al. 1986) is not designed to solve discrete problems like the knapsack one. It does apply, however, to a modified problem with si E [0,l].For the ordered (equation 9) nonhomogeneous knapsack problem this gives solutions with a set of leading Is and a set of trailing Os, with a window in between containing real numbers. Augmented by greedy heuristics for the elements in this window, fairly good solutions emerge. The simplex method scales like
N2M. 5 Numerical Comparisons
Neural Network ("1: Convenient measures for monitoring the decision process are the saturation C = (4/N) Ci(v;- 0.5)2and the evolution rate A = (1/N) Ci(Avi)2,where Avi = vi(t At) - vi(t). The saturation starts off around 0 at high temperature T, and increases to 1 in the T -, 0 limit. We have chosen an annealing schedule where TO= 10, T , = kT,-I, where k = 0.985 if 0.1 < C < (N - l ) / N and 0.95 otherwise. At each temperature every neuron is updated once. We employ a progressive constraint coefficient, a = O.l/T, to avoid small final constraint violations. The algorithm is terminated when C > 0.999 and A < 0.00001. Should the final solution violate any constraint (which is very rare), the annealing is redone with a higher a. In Figure 2 we show a typical evolution of {v;} for an N = M = 40 problem. Simulated Annealing (SA): The performance of this method depends on the annealing schedule. To compare the performance of this method with that of the neural network approach we have chosen the parameters
+
Optimization Problems with Inequality Constraints
337
Table 1: Comparison of Performance and CPU Time Consumption for the Different Algorithms on an N = M = 30 problem! c; = rand[O,11
Algorithm
Perf.
BB
1
NN SA
0.98 0.98 0.98 0.97
LP GH
CPU time 16 0.80 0.80 0.10
0.02
c; = rand[0.45,0.55] Perf. CPU time 1 1500 0.95 0.70 0.95 0.80 0.93 0.25 0.88 0.02
C; = 0.5
Perf. 1 0.97 0.96 0.93 0.85
CPU time 1500 0.75
0.80 0.30 0.02
'The CPU consumption refers to seconds on a DEC3100 workstation.
such that the time consumption of the two methods is the same. This is accomplished with TO = 15, Tfinal= 0.01 and annealing factor k = 0.995. First we compare the NN, SA, and LP approaches with the exact BB for an N = M = 30 problem. This is done both for nonhomogeneous and homogeneous problems. The results are shown in Table 1. As expected LP and in particular GH benefit from nonhomogeneity both quality- and CPU-wise, while for homogeneous problems the NN algorithm is the winner. For larger problem sizes it is not feasible to use the exact BB algorithm. The best we can do is to compare the different approximate approaches, NN, SA, and LP. The conclusions from problem sizes ranging from 50 to 500 are the same as above. The real strength in the NN approach is best exploited for more homogeneous problems. In Figures 3 and 4 we show the performance and CPU consumption for N E [50,500] with M = N. 6 Summary
We have developed a neural mapping and MFT solution method for finding good solutions to combinatorial optimization problems containing inequalities. The approach has been successfully applied to difficult knapsack problems, where it scales like NM. For the difficult homogeneous problems the MFT approach is very competitive as compared to other approximate methods, both with respect to solution quality and time consumption. It also compares very well with exact solutions for problem sizes where these are accessible. In addition, the MFT approach of course has the advantage of being highly parallelizable. This feature was not explored in this work. In Vinod et al. (1990) an ANN approach different from ours was applied to the knapsack problem. The idea in Vinod et al. (1990) is to make orthogonal projections onto convex sets. Since the difficult parameter region was not explored there, a numerical comparison would not be meaningful.
M. Ohlsson, C. Peterson, and B. Werberg
338
I
U
Figure 3: Performance of the neural network (NN)and linear programming approaches (LP)normalized to simulated annealing (SA) for problem sizes ranging from 50 to 500 with M = N. (a) Ci = rand[0.45,0.55]and (b) ci = 0.5.
Figure 4: CPU consumption of the neural network (NN) and linear programming approaches(LP) normalized to simulated annealing (SA) for problem sizes ranging from 50 to 500 with M = N. (a) ci = rand[O.45,0.55] and 6)Ci = 0.5. The numbers refer to DEC3100 workstations.
Optimization Problems with Inequality Constraints
339
Note: We recently became aware of a similar neural approach to the knapsack problem (Hellstrom and Kanal 1992), where the authors also use a discretized form of the derivative. Their treatment is confined to integer problems with a single constraint, whereas ours treat the more general case. Another difference is that the problems probed in Hellstrom and Kanal (1992) are in a nondifficult region (b << N/4) and of fairly small sizes (N= 20). References Gislkn, L., Peterson, C., and Soderberg, B. 1989. Teachers and classes with neural networks. Znt. J . Neural Syst. 1, 167. Gislkn, L., Peterson, C., and Soderberg, B. 1992. Scheduling high schools with neural networks. Neural Comp. 4,805. Hellstrom, B. J., and Kanal, L. N. 1992. Knapsack packing networks. I€€€ Trans. Neural Networks 3, 202. Hopfield, J. J., and Tank, D. W. 1985. Neural computation of decisions in optimization problems. Biol. Cybern. 52, 141. Kirkpatrick, S., Gelatt, C. D., and Vecchi, M. P. 1983. Optimization by simulated annealing. Science 220, 671. Peterson, C. 1990. Parallel distributed approaches to combinatorial optimization. Neural Comp. 2, 261. Peterson, C., and Sijderberg, 8.1989. A new method for mapping optimization problems onto neural networks. Int. J. Neural Syst. 1, 3. Press, W. P., Flannery, B. P., Teukolsky, S. A., and Vettering, W. T. 1986. Numerical Recipes, The Art of Scientific Computing. Cambridge University Press, Cambridge. Vinod, V. V., Ghose, S., and Chakrabarti, P. P. 1990. Resultant projection neural networks for optimization under inequality constraints. Kharagpur Department of Computer Science Preprint.
Received 9 April 1992; accepted 2 September 1992.
This article has been cited by: 2. Arnaud Fréville, SaÏd Hanafi. 2005. The Multidimensional 0-1 Knapsack Problem—Bounds and Computational Aspects. Annals of Operations Research 139:1, 195-227. [CrossRef] 3. Youshen Xia . 2004. An Extended Projection Neural Network for Constrained OptimizationAn Extended Projection Neural Network for Constrained Optimization. Neural Computation 16:4, 863-883. [Abstract] [PDF] [PDF Plus] 4. Manfred Opper, Ole Winther. 2001. Adaptive and self-averaging Thouless-Anderson-Palmer mean-field theory for probabilistic modeling. Physical Review E 64:5. . [CrossRef] 5. Henrik Jönsson , Bo Söderberg . 2001. An Information-Based Neural Approach to Constraint SatisfactionAn Information-Based Neural Approach to Constraint Satisfaction. Neural Computation 13:8, 1827-1838. [Abstract] [PDF] [PDF Plus] 6. P. Persson, S. Nordebo, I. Claesson. 2001. Hardware efficient digital filter design by multimode mean field annealing. IEEE Signal Processing Letters 8:7, 193-195. [CrossRef] 7. P. Persson, S. Nordebo, I. Claesson. 2001. Multimode mean field annealing technique to design recursive digital filters. IEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing 48:12, 1151-1154. [CrossRef] 8. M. Pelillo, K. Siddiqi, S.W. Zucker. 1999. Matching hierarchical structures using association graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence 21:11, 1105-1120. [CrossRef] 9. A. Le Gall, V. Zissimopoulos. 1999. Extended Hopfield models for combinatorial optimization. IEEE Transactions on Neural Networks 10:1, 72-80. [CrossRef] 10. Jari Häkkinen , Martin Lagerholm , Carsten Peterson , Bo Söderberg . 1998. A Potts Neuron Approach to Communication RoutingA Potts Neuron Approach to Communication Routing. Neural Computation 10:6, 1587-1599. [Abstract] [PDF] [PDF Plus] 11. Xiaopeng Chen, K.M. Chugg, M.A. Neifeld. 1998. Near-optimal parallel distributed data detection for page-oriented optical memories. IEEE Journal of Selected Topics in Quantum Electronics 4:5, 866-879. [CrossRef] 12. B López, W Kinzel. 1997. Journal of Physics A: Mathematical and General 30:22, 7753-7764. [CrossRef] 13. Martin Lagerholm , Carsten Peterson , Bo Söderberg . 1997. Airline Crew Scheduling with Potts NeuronsAirline Crew Scheduling with Potts Neurons. Neural Computation 9:7, 1589-1599. [Abstract] [PDF] [PDF Plus] 14. Zhou Qingshan, Zou Yong, Hu Jiandong. 1997. A neural network approach to gate matrix layout. Journal of Electronics (China) 14:3, 209-214. [CrossRef] 15. Ibrahim H. Osman, Gilbert Laporte. 1996. Metaheuristics: A bibliography. Annals of Operations Research 63:5, 511-623. [CrossRef]
16. M. Aourid, B. Kaminska. 1996. Minimization of the 0-1 linear programming problem under linear constraints by using neural networks: synthesis and analysis. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications 43:5, 421. [CrossRef] 17. Carsten Peterson, Ola Sommelius, Bo Söderberg. 1996. Variational approach for minimizing Lennard-Jones energies. Physical Review E 53:2, 1725-1731. [CrossRef] 18. K. Smith. 1996. An argument for abandoning the travelling salesman problem as a neural-network benchmark. IEEE Transactions on Neural Networks 7:6, 1542-1544. [CrossRef] 19. J F Fontanari. 1995. Journal of Physics A: Mathematical and General 28:17, 4751-4759. [CrossRef] 20. Roberto Battiti, Giampietro Tecchiolli. 1995. Local search with memory: benchmarking RTS. OR Spektrum 17:2-3, 67-86. [CrossRef] 21. E Korutcheva, M Opper, B Lopez. 1994. Journal of Physics A: Mathematical and General 27:18, L645-L650. [CrossRef] 22. Laurene V. FausettBoltzmann Machines . [CrossRef]
ARTICLE
Communicated by Richard Durbin
A Model for Motor Endplate Morphogenesis: Diffusible Morphogens, Transmembrane Signaling, and Compartmentalized Gene Expression Michel Kerszberg; Neurobiologie Cellulaire, Institut Pasteur, 25, rue du Docteur Roux, F-75724 Paris Cedex 15, France
Jean-Pierre Changeux Neurobiologie Moliculaire, CNRS VA D l 284, lnstitut Pasteur, 25, rue du Docteur Roux, F-75724 Paris Cedex 15, France
A mathematical model for the formation and maintenance of synaptic contacts at the motor endplate is proposed. It is based on diffusion between sarcoplasmic nuclei of limiting amounts of a morphogen substance. The morphogen is postulated to act on genetic switch-like intranuclear units and to regulate positively both the transcription of its o w n gene and that of acetylcholine receptor (AChR) subunit genes. The efficacy of autoregulation is assumed to be depressed by electrical activity; while AChR genes transcription is enhanced by anterograde neural factors. Thus the model involves Turing's classical ingredients: autocatalysis and short range activation by the morphogen, and long range inhibition by electrical activity. Our predictions include: the stabilization of a single, transcriptionally active nucleus located in the central region of the developing muscle fiber (or myotube); the frequent occurrence of transcriptional activity in nuclei at the tendinous ends; and the onset, upon denervation of adult muscle, of transcription waves, starting from both the central site and the tendinous nuclei. In noninnervated fibers, the calculations show that spontaneous, irregular electrical activity leads to a variety of near-periodic spatial patterns of transcription; these are also predicted in innervated fibers when the depressing effect of electrical activity is weak, giving rise to the stabilization of multiple endplates as occurs in muscles with distributed innervation.
'Present address: Neurobiologie Moleculaire, CNRS VA D1284, Institut Pasteur, 25, rue d u Docteur Roux, F-75724 Paris Cedex 15, France.
Neural Computation 5,341-358 (1993) @ 1993 Massachusetts Institute of Technology
342
Michel Kerszberg and Jean-PierreChangeux
1 Introduction
The molecular mechanisms by which synaptic contacts become established and stabilized in the nervous system remain a major issue of developmental biology. Connections must be adapted to their function and must remain so in spite of perturbations and molecular turnover: the latter suggesting that genetic control mechanisms are most certainly involved. The recent application to this question of recombinant DNA technology has opened the possibility to directly analyze the relevant processes in terms of the regulation of gene expression. In the present paper, we should like to propose a model for the operation of such mechanisms in the particular case of one of the best studied synaptic systems, the motor endplate that forms at the neuromuscular junction. At the adult motor endplate, the receptor for the neurotransmitter, acetylcholine (ACh), is densely accumulated under the motor nerve ending, while a few micrometers away from the synapse, its surface density falls rapidly. Such a privileged distribution of AChR molecules develops from an initial state of the embryonic muscle fiber where AChR molecules are dispersed over the entire surface of the muscle membrane (Salpeter and Loring 1985; Laufer and Changeux 1989; Changeux 1991). With the onset of motor innervation, AChR starts to cluster beneath the exploratory nerve ending while the density of extrajunctional AChR progressively decreases. In the majority of higher vertebrates’ skeletal muscles, a single or “focal” aggregate forms, while in a few others [such as anterior latissimus dorsi (ALD) in the chick], multiple, regularly spaced postsynaptic AChR clusters persist in the adult (Toutant et al. 1980). Among the factors that may contribute to this subcellular morphogenesis, prominent is the presence of multiple nuclei in the developing and adult muscle fiber, and the possible relationship between a differential distribution of AChR molecules and the topology of muscle nuclei actively transcribing AChR genes. In the developing myotube, all nuclei from the muscle syncytium do transcribe AChR genes and this expression becomes progressively restricted to the junctional or “fundamental” nuclei in the absence of intercellular cleavage. A compartmentalized expression of AChR genes thus takes place (Changeux 1991; Changeux et al. 1990) that has been postulated to operate under the control of neural factors [such as calcitonin gene-related peptide (CGRP) (Fontaine et al. 1986) or the ARIA (“AChR-inducing activity”) 42-kDa polypeptide (Harris et al. 199111 as well as electrical activity, via distinct intracellular pathways of second messengers and transcription factors (Fontaine et al. 1987). The aim of the present model is to account for endplate morphogenesis by linking into a coherent, minimal theoretical construct: gene transcription regulation, nuclear topology, and diffusible substances involved in transmembrane as well as internuclear signaling (i.e., morphogens).
Model for Motor Endplate Morphogenesis
343
2 Presentation of the Model
The model is designed to encompass three main components: (1) the sarcoplasmic nuclei that potentially express AChR genes, (2) transmembrane signals, both localized at the interface between functional innervation and muscle sarcoplasm, as well as propagating along the fiber length (action potentials), and (3) the cytoplasm, as a medium through which internuclear signals diffuse. Concerning these subsystems, the salient experimental findings, which we shall use as our biological premises, are as follows. At the nucleus level, in situ hybridization with mixed exonic-intronic and strictly intronic probes reveals a discrete distribution of AChR a-#P-, 6-, or €-subunit mRNA at junctional nuclei (Fontaine et al. 1988; Goldman and Staple 1989; Fontaine and Changeux 1989); also, in noninnervated cultures (Fontaine and Changeux 1989; Harris et al. 1989; Bursztajn et al. 1989; Horovitz et al. 1989; Berman et al. 19901, or in denervated muscle fibers, nuclei may coexist with contrasted levels of AChR a-subunit messenger, suggesting that all-or-none switch mechanisms may control transcription (Fontaine and Changeux 1989). Similar switches have been postulated (Monod and Jacob 1962) or reported in other developing systems as well (Kaufmann 1986; Thomas and IYAri 1990). A simple biological mechanism for such a switch consists of a single gene a (and its product A ) whose activity is being controlled through a positive feedback loop by the transcription factor it codes for. Here, A may for instance turn on, in addition, a cascade (Britten and Davidson 1969) leading to transcription of the receptor subunit genes at the nucleus. Another, stronger mechanism, would involve two genes a and i coding for transcription factors A and I, each factor enhancing its own synthesis while repressing that of the other. This simple genetic regulation network may evolve into a "flip-flop" with two highly stable states, one with high A, the other with high I. Product A might stand for one or several myogenic factors such as proteins belonging to the MyoD family (including myogenin), which induce myoblast differentiation (Davis et al. 1987) and are positively autoregulated (Thayer et al. 1989); while I could be identified with some of their putative antagonists, for example, the c-Jun or Id proteins. The identification for I, however, is less clear than that of A, and we shall thus focus our attention on the latter. The membrane of the muscle fiber transmits two types of signal fast, propagating electrical impulses, and responses to "anterograde" factors released by motor nerve endings such as CGRP or ARIA. Electrical activity is known to repress the transcription of AChR subunit genes both in vivo and in vitro (Changeux 1991). Propagation of the electrical action potential along the fiber membrane causes the entry of Ca2+ ions that could possibly serve as an intracellular second messenger for AChR gene repression. Among the candidate regulatory molecules taking part directly or indirectly in this repression is the Ca2+ lipid-dependent pro-
344
Michel Kerszberg and Jean-PierreChangeux
tein kinase C, whose activation reduces AChR a-subunit mRNA levels (Klarsfeld et al. 1989). The model thus assumes that electrical activity, through one or several messenger systems of this type, reduces AChR gene transcription, and it does so through a reduction in activity of the factor A as defined above: as a consequence the autocatalytic effect of A is lowered, and the probability of going or staying in the high-A state is diminished. The propagation of this effect is taken as infinitely fast, and Ca2+ influx is viewed as occurring simultaneously at all points of the fiber. In contrast, the effect of trans-synaptic, anterograde factors such as CGFW is localized to the immediate vicinity of the neural release site. Another important difference is that while the effects of electrical activity on mRNA levels are sensitive to translation inhibitors-making it likely that they involve continuous synthesis of metabolically short-lived transcription factors, such is not the case for the action of CGRP, a finding consistent with the notion that these two phenomena involve distinct intracellular pathways and protein regulatory units (Duclert et al. 1990; Fontaine et al. 1987). At the cytoplasm level, the model requires some means for nuclei to communicate their state of activity to each other. We posit that this function is carried out by a diffusible substance or morphogen (Wolpert 1969). There is no definite experimental proof as yet for the existence of biological morphogens, that is, substances whose concentration gradients would be responsible for the differentiation of spatial regions during development. However, plausible candidates have been suggested (Eichele and Thaller 1987; Driever and Nusslein-Vollhard 1988a,b). In the present instance, myogenic factors of the MyoD family (Davis et al. 1987)might satisfy our theoretical requirements for a morphogen, since their metabolic lifetime is quite short, and since both themselves and their mRNAs are diffusible substances that can ensure information transfer between nuclei. As the morphogen concentration must be both representative of the transcription state of, say, the AChR a-subunit gene (information transmission), and capable of modulating AChR a-subunit mRNA transcription, it appears legitimate to identify the morphogen with substance A (see above). Nuclei would thus establish communication (Blau etal. 1983) by transcribing a morphogen mRNA, leading to the synthesis of limiting amounts of morphogen in the cytoplasm. Cytoplasmic morphogen, in turn, would penetrate muscle nuclei through their nuclear membrane and bind to complementary DNA promoter elements, thereby becoming trapped while activating its own transcription. The elements of the model are summarized in Figure 1. 3 Formal Definition of the Model
We shall formalize the postulated genetic switches as a set of simple ONOFF Boolean variables Si with values 0 or 1 attached to each nucleus i.
Model for Motor Endplate Morphogenesis
9
................................................
345
CGRP
\ ,______
I
k I
i
........
i i
! P -/
............................................
Figure 1: The model. Three nuclei (dotted boxes) are depicted; from left to right: -Subsynaptic nucleus actively synthesizing AChR. The activator gene (a), coding for the diffusing morphogen (A), is active, enhancing its own transcription and that of AChR, while repressing that of the inhibitor. Neural factors (”CGRP”) arrive from the presynaptic terminal as well, further enhancing AChR transcription through second messenger system(s). - Non-subsynaptic nucleus. The inhibitor (I) is synthesized, turning off transcription of the activator, and that of AChR. Amount of activator diffusing from other nuclei is not sufficient to overcome this effect. - Silent subsynaptic nucleus. Neural factors alone are not sufficient to start AChR expression. Along the whole fiber, the morphogen A autocatalytic action is reduced by second messengers activated through electrical signaling.
By convention, Si = 1denotes a high rate of transcription for some AChR subunit gene (e.g., the a-subunit), and Si = 0 a low rate. This type of mechanism may easily be generalized to other situations, for example, involving more genes and thus more variables per nucleus i linked by local enhancement-repression type of interactions (Thomas and DAri 1990). The switches interact with one another through competition for limiting amounts of morphogen diffusing (Blau et al. 1983) between them. As discussed above, the identification of a positive regulator (e.g., as some member of the Myo-D family) is relatively secure, that of a negative regulator less so. Our computations being designed so as not to depend on details of the switch implementation or the exact chemical nature of the factors involved, we shall concentrate on the more essential positive regulator A. Two sets of equations must be written, one for the states Si
Michel Kerszberg and Jean-PierreChangeux
346
of the switches i, as controlled by the trapping of morphogen molecules in the nuclear region and at the promoter site(& the other for diffusion of the morphogen A itself. We assume that A has concentration-dependent probabilities for occupying the a promoter site(& and that whenever a promoter is occupied by A, transcription takes place. The transition probabilities per time step for the switches i are thus Si(t) = 1
+ 1) = 1 with probability p + af(Aj) + I ) = 0 with probability 1 - p - af(Ai) + Sj(t + 1)= 1 with probability 1 - u + Pf(Ai) S i ( t + 1)= 0 with probability v - Pf(Ai) +
Sj(t
Sj(t
Si(t) =0
(3.1)
where f ( a ) denotes a threshold function, that is,
A < T + f ( A )= O A > T =+ f ( A ) = 1
(3.2)
while p, a, u, and P are parameters obeying such restrictions that make the probabilistic interpretation of equations 3.1 possible. The probabilities of staying or going “ O N are both increased when Ai 2 T. It is interesting to note that while we shall be interested exclusively in regimes leading to smooth, uniform behavior of Si(t), values of p, u, a, and P are not precluded, which would yield oscillatory behavior of the activities, a situation that may be of interest in other experimental systems (Bargiello et al. 1984). Morphogen diffusion and synthesis are postulated to be described by the following equations for Ai:
Aj(t + 1) =
cb
+ Aj(t){
~i T,E(t) - %[I - b S j ( f ) ] }
+ k {[I- 0 S i - 1 ( t ) ] A i -(~t ) } + k {[I- ~ S i + l ( t ) ] A i + ~ ( t ) } .
(3.3)
Equations 3.1,3.2, and 3.3 are the central expressions defining our model. In 3.3, c b denotes the basal level of synthesis; k is the diffusion coefficient, or internuclear “hopping” probability per unit time for A molecules. We neglect, in this simple model, spatial fluctuations in k: the nuclei are indifferently spaced and the diffusion rates between them (assumed to be controlled mostly by the nuclear membrane barrier rather than by the internuclear distance, which may vary in the course of contraction) are the same for any two neighbors. Note that the equations must be modified for those nuclei closest to the tendinous ends of the fiber. It is important to realize that when the nuclear switch is turned “ON,” only a fraction 1 - 0 of the morphogen is available for diffusion: the rest is trapped in the nucleus where it directs enhanced mRNA transcription (see below). This interpretation of 3.3 requires that the involved transcription
Model for Motor Endplate Morphogenesis
347
factors be rather scarce, which seems indeed to be the case for, say, the MyoD molecule. The A-enhanced transcription of the a gene yields fresh A product at a rate described by q. The latter embodies the net effect of turnover (degradation) and autocatalytic biosynthesis. It depends on whether a nerve ending is located in the immediate vicinity of nucleus i and on the state of activity Si of this nucleus: isolated nucleus ri = under a synaptic terminal Ti =
70 70
+ TnSi + 7s + TnSi
(3.4)
When the switch is "ON," synthesis proceeds at a faster pace, as described by 7,. Transcription is also boosted further by anterograde factors such as CGRP present under a terminal ( T ~ )and acting there as "synaptotrophic" effectors. The latter are independent of the autocatalytic synthesis state Si, and their efficacy need not be very strong (i.e., Ts
<< 'Tn,'To).'
On the other hand, we postulate that the depressing effect of electrical activity E results from a reduction of the net autocatalytic synthesis of A. Thus in the absence of electrical activity E ( t ) = 0 ongoing electrical activity E ( t ) = 1 - gn(')
(3.5)
E is a measure of the extent of second-messenger activation by channel opening due to impulse arrival, and n ( t ) denotes the number of subsynaptic nuclei actively transcribing the a gene at time t. We assume n is indicative of the number of functional synaptic boutons; as one bouton is already sufficient to generate action potentials, equation 3.5 must rise sharply with n and saturate quickly at 1 (this will occur provided g is small). 4 Analytical Results for the Adult Endplate We calculate now an approximate analytical solution of great interest, namely one where only a single focal nucleus per muscle fiber is actively expressing AChR genes. While the full problem is nonlinear, equations 3.3 are linear in Ai. Therefore, one can conveniently assume that diffusion of A is slow with 'This can be viewed mathematically as the application of a so-called "symmetrybreaking field." Assuming all nuclei are equivalent in an infinite muscle fiber, a situation involving synthesis in only a subset of nuclei necessarily entails a breaking of the original equivalence. While the existence of symmetry-breaking solutions is an intrinsic property of the system, the choice of how exactly the equivalence is broken may be the result of an extremely weak '%biasing effect" such as that exerted by our synaptotrophic field.
Michel Kerszberg and Jean-Pierre Changeux
348
respect to the switch dynamics? and that the switch is therefore at all times in a stationary situation constrained by fixed A? Accordingly, the focal solution, obtained by speclfying that one given nucleus only is active (the question of its location will be discussed later) is now determined by solving first the master equation corresponding to equation 3.1; in order to extract the probabilities for finding the switch i in the "ON position when Ai is below equation 4.3 or above equation 4.4threshold. We obtain s- = I - u Ai < T (4.3) 2-p-u1
s+
=
1-u+p 2 - p - v - cr + p '
Ai 1 T
(4.4)
Substituting this in 3.3, we get a slow-time effective equation for Ai, enabling us to compute quantities of interest. Chief among those is the diffusive decay length that describes, once the system has reached a stable state, the spatial variation of morphogen concentration around the nucleus where AChR gene expression takes place. This stable, time-independent concentration of A will be given by solving 3.3 with 4.4 substituted in the equation for A0 (subscript 0 denoting the active site), and 4.3 in the others (Ail i # 0). Writing A(t 1) = A(t) (time invariance), we find that the stabilized ratio of Ai to Aj+l, if0is
+
(4.5) which defines the decay length A, where ~ ( 1is) the net synthesis rate at the nonactive sites and A, is the concentration far away from the active nucleus, that is, A, = cb/[l - 7(1)]. The single-active-nucleus solution can be further specified, by computing the ratio Ao/AI, not covered by
- q,
2The diffusional time scale td where kff= k(1- a). 3While useful for analytical work, the biological validity of this so-called "quasistationary" assumption is far from proven. It is certainly correct as far as the time independent solutions to be discussed below; as to dynamic phenomena, we have seen that similar results are obtained by computer simulation, whether the assumption is used or not. 4These equations are
q+1 Pf'l
- 11, = l-v+p+Pf[p+v-l+a-P],
= 1- v+PI [ p + v
Cj < T Cj>T
(4.1) (4.2)
where Pi'stands for the probability that Sj = 1 at time t . It is easy to ensure that either of these equations relaxes to equilibrium in a time much less than td (see note 3).
Model for Motor Endplate Morphogenesis
349
the previous formulaa5This defines the one-active-nucleus solution completely. However, for the solution to be consistent, it is imperative that A0 be higher than the threshold T, and A1 lower. Furthermore, T must be higher than A,. With parameter values as described in the legend of Figure 2, we find a decay wavelength X = 7.09, while A0 = 2.97 and A, = 0.40 (all these under electrical stimulation). The wavelength is such that around the isolated nucleus, on the order of 7 nuclei will be inactivated on each side. The precise value depends on the threshold T above which the morphogen becomes effective. Why are the nuclei inactivated? The morphogen is trapped by the promoter sites in the nucleus that it activates. Its mobility is thereby reduced, as well as its concentrationin the vicinity: hence, the low value of A,. The concentration far away from the single active nucleus is A, = 0.52. Thus, by chosing a threshold T comprised between A, and Ao, one ensures that only one active nucleus may stably exist in the fiber. Direct dynamic simulations indeed confirm this expectation (see below), yet the exact position of the active nucleus appears difficult to assess analytically at this stage. In general, the value of A0 when the active nucleus is at the tendon6 will be lower than otherwise: for the case at hand, A0 becomes 2.38. This means the buildup of morphogen concentration required to stabilize the active nucleus occurs more readily at the tendinous end-where it has only one escape direction open. It is thus clear that the average transcription level will be rather high there. It must be stressed that X has a significance that goes well beyond the strict framework of the single-active-site solution. It will more generally be related to the approximate wavelength or spatial periodicity of any stable structure evolving from the system dynamics. Indeed, nothing in what we have said until now excludes the appearance, farther than a few X away from the active nucleus, of some additional activity pattern, in the form, for example, of another isolated active site. In general, periodic solutions may be expected with M active, N inactive alternating nuclei. Relative stability of those putative solutions is an extremely arduous problem to tackle analytically, so we shall presently turn to computer simulations. 5Morphogenconcentrations at and in the immediate vicinity of the active site are given as the solutions of the following set of linear equations: (4.6) (4.7)
In the first of these equations, L equals 2 when the active nucleus is in the midst of the fiber itself, or 1 when it is located at a tendinous end. 6That is, L = 1 in equations 4.6 and 4.7.
350
Michel Kerszberg and Jean-Pierre Changeux
Figure 2: (a) Dynamics of transcription states in a focally innervated muscle fiber. States are sampled at intervals of lo00 computation steps (successive lines, from top to bottom). Initial innervation is random. Where and when transcription takes place a box is drawn, full for subsynaptic nuclei, empty for the others. Before the onset of electrical activity, nuclei are mostly active; in the presence of electrical inputs (arrow), transcription is generally repressed but persists in two nuclei, one subsynaptic and located near the center of the fiber, the other near a tendinous end. Note that this latter feature is not always present, although finer time-dependence analysis always shows repression to proceed slowest near the tendons and at the center. (Fiber is innervated by 10 randomly distributed terminals and contains 30 nuclei with genetic switches having a = 0.88, B = 0.80, p = 0.10 and v = 0.95, u = 0.9. Parameter g of equation 3.5 is 0.2, and morphogen diffusion is controlled by k = 0.1. Net synthesis rates (refer to Eqs.3.3and 3.4)are T~ = 0.002, TO = 0.9997,T,, = 0.00025 and q = 0.00004.)(b) The effects of denervation. This is simulated here by switching off electrical input as well as anterograde signaling from the afferent motor endings at all times below the lower arrow. One observes "waves" of renewed transcriptional activity, spreading from the "central" nucleus, as well as from the tendinous ends (whether the nuclei there were initially active or not). Such a spread is typical of diffusion and this figure thus summarizes a strong prediction of the model as a modified reaction-diffusion system.
Model for Motor Endplate Morphogenesis
351
5 Computer Simulations of Developmental Dynamics Computer simulations of the system are consistent with analytical figures wherever the latter are available, that is, in the case of stabilized focal innervation. Moreover they open new perspectives into the behavior of the full nonlinear set of equations, which is out of the reach of analytical methods. Numerical calculations do not depend on the approximation used above, namely that the genetic switch adjustment is fast with respect to morphogen dynamics; yet they are compatible with our previous findings. They cover three cases, those of focal or multiple innervation, and that of noninnervated systems. We assume that 30 nuclei are present along the fiber, and that only a fraction of them are located under newly formed presynaptic terminals. Initially, 10 of those are distributed at random, each above one nucleus. This is a plausible description of the initial state in early endplate morphogenesis.
5.1 Focal Innervation. The upper part (a) of Figure 2 displays successive "snapshot" pictures of the state of activity of the 30 nuclei, at progressively later times from top to bottom. It can be seen how, with sustained electrical activity (which starts below the horizontal line), transcription in most of the nuclei is repressed. Analysis on a finer time scale shows transcription to diminish differently depending on position, with transcription around the fiber center and in nuclei situated near the tendons resisting longest. The final configuration usually comprises one active, subsynaptic nucleus located rather near the middle of the fiber (Salpeter and Loring 1985; Laufer and Changeux 1989; Changeux 1991). Persistent activity near the tendinous ends, while surprising at first, actually confirms the analytical calculations of the previous paragraph. When we take into account late growth of the myotube through myoblast fusion at its extremities (see Fig. 31, we find that with such growth, transcription may remain high in the nuclei incorporated latest in the syncytium, that is, those at the tendinous ends. Experimental support for this finding has been reported recently (Fontaine and Changeux 1989; Klarsfeld et al. 1991). Not pictured here, but observed in some cases, are anomalous patterns such as persistence of two nuclei actively engaged in transcription, or extinction of sustained transcription altogether. 5.2 Denervation. A situation of much interest concerns denervation. Experimental denervation usually leads to a reactivation of AChR gene transcription by many extrajunctional nuclei (Salpeter and Loring 1985; Laufer and Changeux 1989; Changeux 1991; Goldman and Staple 1989). We simulate denervation here by assuming that it simply causes the E variable (see equation 3.5) and the T~ parameter to decrease to zero values.
352
Michel Kerszberg and Jean-PierreChangeux
Figure 3: Myotube growth through myoblast fusion at the extremities is simulated on this figure. Growth starts after an initial innervation and stabilization phase. The myoblasts fusing with the fiber are assumed to express the AChR a-subunit gene at a high rate. It is seen how, through diffusion of the activator, this state is lost more or less quickly (compare the two sides of the figure!). On one side, strong transcription-related labeling would be expected, as observed experimentally(see text).
Figure 2b displays the results of a computer run where "denervation" occurs after a period of electrical stimulation (i.e., at the second horizontal line from the top). It shows waves of transcription onset starting from "seed" nuclei at which concentration of A was initially high, and spreading progressively to the whole fiber. Such seed nuclei include the initially active (subsynaptic) ones, as well as the near-tendon units. Indeed, denervation experiments performed on adult muscle reveal a nonuniform reappearance of the receptor protein and mRNAs, which are first re-expressed in the neighborhood of the endplate (Goldman and Staple 1989; Salpeter and Loring 1985; Neville et al. 1991).
Model for Motor Endplate Morphogenesis
353
a
b
Figure 4: (a) Noninnervated muscle cultures. Spontaneous electrical activity is present in this system, and leads to multiple transcriptional foci. Note that strong lateral labeling by transcription markers is predicted. For T~ = 0, T~ = 0.0003, T = 2.50 (all other parameters unchanged) we find that clusters of transcribing nuclei appear in the form of doublets or transient triplets. The groups themselves are more or less regularly spaced. When one takes T~ = 0.0006, four isolated, "regularly" spaced nuclei are found (not shown). (b) Dynamics of AChR gene expression in the multiply innervated muscle fiber. Here again, we assumed a reduced efficacy ( T ~= 0.00035) in the repressing effect of electrical activity, due, e.g., to spike "bunching." Under these conditions, more or less regularly spaced transcription sites persist, as indeed observed, e.g., in the chick anterior l a h i m u s dorsi muscle. Notice that nuclei at the tendinous ends are invariably the site of intense transcription in this case. From parts (a) and (b) of the figure it is apparent that the final innervation pattern is determined to a large extent by the underlying sarcoplasmic morphogenesis (a) interacting with a set of available exploratory synaptic boutons.
5.3 Cultured Fibers without Innervation. Figure 4a pertains to the situation where electrical activity occurs spontaneously, as is the case in primary cultures of chick embryonic muscle fibers (Fontaine and Chang-
354
Michel Kerszberg and Jean-PierreChangeux
eux 1989; Harris et al. 1989; Bursztajn et al. 1989; Horovitz et al. 1989; Berman et al. 1990). This corresponds formally to T~ = 0. The resulting structure may be called “imperfectly periodic”: there is obviously a preferred spacing between active units, but this is not a strict constraint. Strings of “doublets” or even the transient existence of “triplets” of active nuclei are predicted. For different numerical settings (not shown), strings of isolated, or “singlet” active nuclei become apparent. More complex but less stable configurations than those described here might occur if a larger range of parameter values were to be explored. Interestingly, in situ hybridization experiments with cultured chick myotubes reveal, within the same fiber, silent nuclei alternating with others expressing AChR subunit mRNA, following a pattern of rather poor regularity (Fontaineand Changeux 1989; Harris et al. 1989; Bursztajn et al. 1989; Horovitz et al. 1989; Berman et al. 1990). 5.4 Multiple Innervation (ALD). On Figure 4b, we have selected, for the simulation, numerical parameters which yield a final pattern reminiscent of ALD multiple-focus innervation. One notices rather short-lived clusters of active, mostly sub- or near synaptic nuclei. They are obviously reminiscent to the doublets seen in Figure 4a, and display again a “near-periodicity.” It is quite apparent again how transcription is actively proceeding at the tendons, which seem to “anchor” naturally the nearperiodic pattern of active nuclei. These results suggest very strongly that, in our model, the underlying pattern of gene activation is the dominant factor controlling synapse stabilization. We see here an illustration of how a reduction in the average depressing effect of electrical activity may cause a change in the spatial pattern of genetic expression. On the time scales considered, which are slow compared to nerve action potentials (or even interspike intervals) the reduction in T~ leading to this change may be seen as originating, for instance, from a time patterning of electrical stimulation. Yet, a detailed implementation of such time patterns in electrical stimulation, and their effects on synapse morphogenesis, has not been attempted at this stage.
6 Outlook
Many models, reviewed recently (Van Essen 19821, have been introduced in order to explain the inhomogeneous distribution of AChR along the muscle fiber. None of these previous attempts, however, took account of the inhomogeneity already present at the level of transcription. Here we present a theoretical hypothesis whereby differential AChR gene expression, as controlled by scarce transcription factors for which nuclei enter into competition, may be one of the critical determinative events in motor endplate morphogenesis.
Model for Motor Endplate Morphogenesis
355
The model, which emphasizes the autonomous evolution of the muscle nuclei, leads to satisfactory agreement with a set of known experimental facts concerning the development of the motor endplate upon innervation. It may, however, be of broader import and apply, with suitable modifications, to a variety of situations involving morphogenesis (Hafen et al. 1984; Izpis6a-Belmonte et al. 1991). Crucial for the relevance to cellularized embryonic structures may well be the recent report (Joliot et ul. 1991) that homeotic regulation factors might be able to cross membrane boundaries. The mechanisms proposed here apply to the case of syncytia; when individual, mononucleated cells (such as neurons) are present, one has to include in the model intercellular "morphogens," the mechanisms of their secretion (Torre and Steward 19921, of their recognition by membrane receptors, and of signal transduction. The model could, with suitable modifications, be generalized to synaptogenesis elsewhere in the peripheral and central nervous systems. Synaptogenesis through selective stabilization is of course taking place during development ( k w e s and Lichtman 1980) and has been reported in the cerebellum (Crepe1 et al. 1976; Mariani and Changeux 1980) and several areas of neocortex (Shatz et al. 19901, but has been observed directly in the living adult parasympathetic system as well (Purves and Voyvodic 1987). Recently, transient overproduction of neurotransmitter receptors has also been reported in diverse regions of the primate cerebral cortex (Lidow et al. 1991), and might lead to subsequent morphogenetic phenomena as modeled here if one considers a cortical column as a cellularized analog of the developing myotube. One of our findings that may be of significance for morphogenesis in general is the transcription onset waves we have observed on denervation. The latter are characteristic of a diffusion process and may help substantiate rather simply the morphogen hypothesis in a variety of situations. In mathematical terms, the formulation introduced here represents both a simplification of the original morphogenesis formalization (Turing 1952; Meinhardt 1986) and its generalization. It is a mathematical simplification because the nonlinear aspects are limited to a set of points (the nuclei) rather than being spread over the whole space, thus rendering the model highly tractable analytically; conversely, it represents a generalization since it includes an interaction of the morphogen with the genetic machinery involved in morphogen synthesis itself. The basic ingredients are an autocatalytic loop through the enhancement of activator (morphogen) transcription by its own gene product, long-range inhibition through trapping of the activator by DNA elements in nuclei, thus reducing its availability; and even longer (infinite) range inhibition by electrical activity. The model clearly produces a number of predictions at the level of the molecular mechanisms of gene expression that can be experimentally tested.
356
Michel Kerszberg and Jean-Pierre Changeux
References Bargiello, R. A., Jackson, F, R., and Young, B. W. 1984. Restoration of circadian behavioral rhythms by gene transfer in Drosophila. Nature (London) 312,752754. Berman, S. A., Bursztajn, S., Bowen, B., Gilbert, W. 1990. Localization of acetylcholine receptor intron to the nuclear membrane. Science 247,212-214. Blau, H., Chiu, C.-P., Webster, C. 1983. Cytoplasmic activation of human nuclear genes in stable heterocaryons. Cell 32, 1171-1180. Britten, R. J., and Davidson, E. H. 1969. Gene regulation for higher cells: A theory. Science 165,349-356. Busztajn, S., Berman, S. A., and Gilbert, W. 1989. Differential expression of acetylcholine receptor mRNA in nuclei of cultured muscle cells. Proc. Natl. Acad. Sci. U.S.A. 06, 2928-2932. Changeux, J. P. 1991. Compartmentalized transcription of acetylcholine receptor genes during motor endplate epigenesis. New Biologist 3,413-429. Changeux, J. P., Babinet, C., Bessereau, J. L., Bessis, A., Cartaud, A., Cartaud, J., Daubas, P., Devillers-Thibry,A., Duclert, A., Hill, J. A., Jasmin, B., Klarsfeld, A., Laufer, R., Nghiem, H. O., Piette, J., Roa, M., and Salmon, A. M. 1990. Compartmentalizationof acetylcholine receptor gene expression during development of the neuromuscular junction. Cold Spring Harbor Symp. Quant. Biol. LV,381-396. Crbpel, F., Mariani, J., and Delhaye-Bouchaud, N. 1976. Evidence for a multiple innervation of Purkinje cells by climbing fibers in immature rat cerebellum. J. Neurobiol. 7, 567-578. Davis, R. L., Weintraub, H., Lassar, A. B. 1987.Expression of a single transfected cDNA converts fibroblasts to myoblasts. Cell 51,987-1000. Driever, W., and Niisslein-Vollhard, C. 1988a. A gradient of bicoid protein in Drosophila embryos. Cell 54, 83-93. Driever, W., and Nusslein-Vollhard, C. 1988b. The bicoid protein determines position in the Drosophila embryo. Cell 54, 95-104. Duclert, A., Piette, J., and Changeux, J. P. 1990. Induction of acetylcholine receptor a-subunit gene expression in chicken myotubes by electrical activity blockade requires ongoing protein synthesis. Proc. Natl. Acad. Sci. U.S.A. 07, 1391-1395. Eichele, G., and Thaller, C. 1987. Characterization of concentration gradients of a morphogenetically active retinoic acid in the chick limb bud. J. Cell. Biol. 105,1917-1923. Fontaine, B., Klarsfeld, A., Hokfelt, T., and Changeux, J. P. 1986. Calcitonin-gene related peptide, a peptide present in spinal cord motoneurons, increases the number of acetylcholine receptors in primary cultures of chick embryo myotubes. Neurosci. Lett. 71,59-65. Fontaine, B., Klarsfeld, A., and Changeux, J. P. 1987. Calcitonin-gene related peptide and muscle activity regulate acetylcholine receptor a-subunit mRNA levels by distinct intracellular pathways. J. Cell Biol. 105, 1337-1342. Fontaine, B., Sassoon, D., Buckingham, M., and Changeux, J. P. 1988. Detection of the nicotinic acetylcholine receptor a-subunit mRNA by in situ hybridiza-
Model for Motor Endplate Morphogenesis
357
tion at neuromuscular junctions of 15-day-old chick striated muscle. EMBO
J. 10, 625-632. Fontaine, B., and Changeux, J. P. 1989. Localization of nicotinic acetylcholine receptor a-subunit transcripts during myogenesis and motor endplate development in the chick. J. Cell Biol. 108, 1025-1037. Goldman, D.,and Staple, J. 1989. Spatial and temporal expression of acetylcholine receptor RNAs in innervated and denervated rat soleus muscle. Neuron 3, 219-228. Hafen, E., Kuroiwa, A., and Gehring, W. J. 1984. Spatial distribution of transcripts from the segmentation gene fushi tarazu during Drosophila embryonic development. Cell 37, 833-841. Hams, D. A., Falls, D. L., and Fischbach, G. D. 1989. Differential activation of myotube nuclei following exposure to an acetylcholine receptor-inducing factor. Nature (London) 337, 173. Harris, D. A., Falls, D. L., Johnson, F. A., and Fischbach, G. D. 1991. A prionlike protein from chicken brain copurifies with an acetylcholine receptorinducing activity. Proc. Natl. Acad. Sci. U.S.A. 88, 7664-7668. Horovitz, O.,Spitsberg,V., and Salpeter, M. M. 1989. Regulation of acetylcholine receptor synthesis at the level of translation in rat primary muscle cells. J. Cell Biol. 108, 1817. Izpisba-Belmonte, J. C., Tickle, C., Dollb, P., Wolpert, L., and Duboule, D. 1991. Expression of the homeobox Hox-4 genes and the specification of position in chick wing development. Nature (London) 350,585-589. Joliot, A. H., Triller, A., Volovitch, M., Pernelle, C., and Pmchiantz, A. 1991. a-2,8-polysialic acid is the neuronal surface receptor of antennapedia homeobox peptide. Nao Biologist 3, 1121-1134. Kaufmann, S. A. 1986. Boolean systems, adaptive automata, evolution. In Disordered Systems and Biological Organization, E. Bienenstock, F. FogelmanSoulib, and G. Weisbuch, eds., pp. 339-360. Plenum Press. Klarsfeld, A., Laufer, R., Fontaine, B., Devillers-Thiery, A., Dubreuil, C., and Changeux, J. P. 1989. Regulation of muscle AChR a-subunit expression by electrical activity: Involvement of protein kinase C and Ca++. Neuron 2, 1229-1236. Klarsfeld, A., Bessereau, J. L., Salmon, A. M., Triller, A., Babinet, C., and Changeux, J. I? 1991. An acetylcholine receptor a-subunit promoter conferring preferential synaptic expression in muscle of transgenic mice. EMBO J. 10, 625-632. Laufer, R., and Changeux, J. l? 1989. Activity-dependent regulation of gene expression in muscle and neuronal cells. Mol. Neurobiol. 3, 1-35. Lidow, M. S., Goldman-Rakic, P. S., and Rakic, P. 1991. Synchronized overproduction of neurotransmitter receptors in diverse regions of the primate cerebral cortex. Proc. Natl. Acad. Sci. U.S.A. 88, 10218-10221. Mariani, J., and Changeux, J. P. 1980. Multiple innervation of Purkinje cells by climbing fibers in the cerebellum of the adult staggerer mutant mouse. J. Neurobiol. 11,41-50. Meinhanit, H. 1986. Hierarchical inductions of cell states: A model for segmentation in Drosophila. J. Cell Sci. Suppl. 4, 357-381.
Michel Kerszberg and Jean-Pierre Changeux
358
Monod, J., and Jacob, F, 1962. General conclusions: Teleonomic mechanisms in cellular metabolism. Cold Spring Harbor Symp. Quant. B i d . XXVI,389. Neville, C., Schmidt, M., and Schmidt, J. 1991. Kinetics of expression of Ach receptor a-subunit mRNA in denervated and stimulated muscle. NeuroReport 2, 655-657. New, H. V., and Mudge, A. W. 1986. Calcitonin-gene related peptide regulates muscle acetylcholine receptor synthesis. Nature (London) 323, 809-811. Purves, D., and Lichtman, J. 1980. Elimination of synapses in the developing nervous system. Science 210,153-157. Purves, D., and Voyvodic, J. 1987. Imaging mammalian nerve cells and their connections over time in living animals. Trends Neurosci. 10, 398-404. Salpeter, M., and Loring, R. H. 1985. Nicotinic acetylcholine receptors in vertebrate muscle: Properties, distribution and neural control. Prog. Neurobiol. 25, 297-325. Shatz, C. J., Gosh, A., McConnell, S. K., Allendoerfer, K. L., Friauf, E., and Antonioni, A. 1990. Pioneer neurons and target selection in cerebral cortical development. Cold Spring Harbor Symp. Quant. B i d . LV,469480. Thayer, M. J., Tapscott, S. J., Davis, R. L., Wright, W. E., Lasser, A. B., and Weintraub, H. 1989. Positive autoregulation of the myogenic determination gene MyoDl. Cell 58,241-248. Thomas, R., and D’Ari, 1990. Biological Feedback. CRC Press, Boca Raton, FL, and references therein. Torre, E. R., and Steward, 0. 1992. Demonstration of local protein synthesis within dendrites using a new cell culture system that permits the isolation of living axons and dendrites from their cell bodies. 1.Neurosci. 12,762-772. Toutant, M., Bourgeois, J. P., Toutant, J. P., Renaud, D., Le Douarin, G. H., and Changeux, J. P. 1980. Chronic stimulation of the spinal cord in developing chick embryo causes the differentiation of multiple clusters of acetylcholine receptor in the posterior latissimus dorsi muscle. Dev. B i d . 76, 384-395. Turing, A. M. 1952. The chemical basis of morphogenesis. Phil. Trans. R. SOC. (London) B 237, 37-72. Van Essen, D. C. 1982. Neuromuscular synapse elimination: Review. In Neuronal Development, N. C . Spitzer, ed. Plenum Press, New York. Wolpert, L. 1969. Positional information and the spatial pattern of cellular differentiation. 1. Theor. Biol. 25, 147. --
~~
Received 13 May 1992; accepted 24 September 1992.
This article has been cited by: 1. Michel Kerszberg, Jean-Pierre Changeux. 1998. A simple molecular model of neurulation. BioEssays 20:9, 758-770. [CrossRef]
NOTE
Communicated by John Platt
Universal Approximation by Phase Series and Fixed-Weight Networks Neil E. Cotter Peter R. Conwell Electrical Engineering Department, University of Utah, Salt Lake City, UT84112 USA
In this note we show that weak (specified energy bound) universal approximation by neural networks is possible if variable synaptic weights are brought in as network inputs rather than being embedded in a network. We illustrate this idea with a Fourier series network that we transform into what we call a phase series network. The transformation only increases the number of neurons by a factor of two. 1 Technical Preliminaries
Let g(x) be the bounded measurable real-valued function that we wish to approximate. We take the domain of g to be the unit hyper cube D = [-1/2, 1/2INwhere N is the number of entries in x = (XI,. . . , x N ) . To generalize our results below to an interval [ - X / 2 , X / 2 ] we would divide frequencies by X in our final phase series. By Lusin's theorem (Royden 19681, for any 6 > 0 there exists a continuous function, f, such that the measure of the set where f is not equal to g is less 6. Thus, by successfully approximating f we can restrict errors to an arbitrarily small percentage of inputs. We can represent f in turn with a Fourier series. We will assume that If1 is bounded by M / 2 & where M is a value we must specify before constructing our network. This limits us to what we call "weak universal approximation." Since we may take M as large as desired, however, this is a mild limitation. For the derivation we assume M = 1. This amplitude bound translates into a total energy bound of 1/8 on domain D. 2 Phase Series Derivation
Since the energy of a sinusoid on D is one-half its squared amplitude, an energy bound of 1/8 translates into a bound of f 1 / 2 for the coefficients of a Fourier series for f(x): f(x) =
C C abncos(2m o x + 6n/2)
(2.1)
~ = o , In
Neural Computation 5, 359-362 (1993) @ 1993 Massachusetts Institute of Technology
Neil E. Cotter and Peter R. Conwell
360
where n = (nl,. . . ,nN) is a frequency vector with entries ranging over all possible combinations of positive and negative integer values and 6 is a binary variable used to obtain sine terms by phase-shifting cosine terms. We observe that the left-hand side of the following trigonometric identity has the same form as the summand in equation 2.1: 2 COSAcos B = COS(B- A) + COS(B+ A)
(2.2)
We make the identification a6n = 2 cosA
(2.3)
Solving for A and substituting into equation 2.1 yields the phase series representation
f(x) =
CC C
cos(2m o x
+ 6a/2 + r]cos-'[uan/2])
(2.4)
6=0,1 n q=-i,i
The phase series has several features we wish to highlight 1. The phase and Fourier series have exactly the same value everywhere on D. 2. The coefficients of the cosine terms are unity regardless of f ( x ) . 3. The coefficients, c0s-~[a6,,/2],specifying f(x) are processed in the same way as input data-both the coefficients and the data are part of a weighted sum. 4. The multipliers, n and
r],
of the weighted s u m are independent of
f (XI. 3 Phase Series Neural Networks
We can implement both the original Fourier series in equation 2.1 and the phase series in equation 2.4 as neural networks having a single hidden layer and a linear output neuron. In the hidden layer, we could approximate the cosine term by summing familiar sigmoids such as the logistic squasher (Rumelhart and McClelland 1986) or the hyperbolic tangent (Hopfield 1984). For an exact representation, we sum copies of a "cosig" squasher (see Gallant and White 1988): 00
cosx = -1
+C C
+
cosig(xx - x2nk n/2)
(3.1)
k=-00 x=-I,l
where x 5 -R/2 - n/2) -7r/2 < x < 7r/2 x 2 R/2
(3.2)
Universal Approximation
361
Figure 1: Phase series network based on “cosig” sigmoid embedding one halfcycle of a cosine. Stacked boxes indicate a structure that is repeated for all values taken on by indices in the lower right-hand comer. The output sum thus has many inputs.
Substituting 3.1 into 2.4 yields the formula for a phase series neural network
c c c [-I + c c cosig(X2m oo
f(x)
=
6=o,1 n q=-i,i
ox
+ xb.rr/~
k=-oo x=-l,l
+ xq cos-’[&n/2] - X2Tk + */2)]
(3.3)
Figure 1 illustrates the phase series network. Note that the Fourier coefficients defining f(x) are embedded in network inputs. Thus, we have derived a neural network, capable of universal approximation, in which internal weights are fixed. Because the identity in equation 2.2 substitutes two cosine terms for one cosine term and a coefficient, this new network has only twice as many neurons as a Fourier network. 4 Discussion and Conclusion
There are possible advantages to using phase series in VLSI or optical circuits. First, we eliminate circuitry for accessing internal parameters. Second, more technologies are suitable for implementing the fixed synap tic weights inside the phase series network than are suitable for implementing varying synaptic weights inside a conventional neural network.
362
Neil E. Cotter and Peter R. Conwell
Third, constant parameters are less expensive and occupy less room than variable parameters in a circuit. A fixed resistance in a VLSI circuit or a fixed opacity in an optical circuit is relatively easy to manufacture.
Acknowledgments The authors are greatly indebted to an anonymous reviewer who pointed out that the identity in equation 2.2 is simpler than the one we originally
used.
References Gallant, A. R., and White, H. 1988. There exists a neural network that does not make avoidable mistakes. Proc. Int. Joint Conf. Neural Networks (IJCNN),Sun Diego, C A I, 657-664. Hopfield, J. J. 1984. Neurons with graded response have collective computational properties like those of two-state neurons. Proc. Natl. Acad. Sci. U.S.A. 81,3088-3092. Royden, H. L. 1968. Real Analysis, 2nd ed. Macmillan, New York. Rumelhart, D. E., and McClelland, J. L., (eds.) 1986. Parallel Distributed Processing: Exploratioiis in the Microstructures of Cognition. Vol. 1: Foundations. MIT Press, Cambridge, MA. Received 18 May 1992; accepted 22 September 1992.
This article has been cited by: 1. Judith E. Dayhoff. 2007. Computational Properties of Networks of Synchronous Groups of Spiking NeuronsComputational Properties of Networks of Synchronous Groups of Spiking Neurons. Neural Computation 19:9, 2433-2467. [Abstract] [PDF] [PDF Plus] 2. A.S. Younger, P.R. Conwell, N.E. Cotter. 1999. Fixed-weight on-line learning. IEEE Transactions on Neural Networks 10:2, 272-283. [CrossRef]
Communicated by Halbert White
NOTE
Backpropagation with Homotopy Liping Yang Wanzhen Yu Department of Applied Mathematics, Tsinghua University, 100084, Beijing, China
When training a feedforward neural network with backpropagation (Rumelhart et al. 19861, local minima are always a problem because of the nonlinearity of the system. There have been several ways to attack this problem: for example, to restart the training by selecting a new initial point, to perform the preprocessing of the input data or the neural network. Here, we propose a method which is efficient in computation to avoid some local minima. For a neural network, the output of every node is characterized by a nonlinear function (for example, the sigmoid function) that is the origin of local minima. Consider the following homotopy function fx(x) = Ax
+ (1 - A)s(x)
(1)
where s(x) is a sigmoid function and A E [0,1]. fx(x) forms a homotopy between linear and sigmoid function. Denote Nx to be the neural network characterized by f ~ We . start the training with A. = 1, that is, every node , choose &+I < Ak and is linear. After achieving a minimum of N A ~we continue the backpropagation procedures until Xks = 0 with which f x ( x ) is just the original sigmoid function. The learning of a feedforward network is to solve a nonlinear leastsquares problem,
I "
minF(w) = - c[gi(w) - siI2 2 i=l
(2)
where n is the number of training samples and gi is the output of network whose weight is w. This problem can also be treated by solving
VF(w) = 0
and
V2F(w)2 0
(3)
The homotopy method for solving nonlinear equations has been studied since 1978 (Chow et al. 1978; Li 1987). This method begins the homotopy process with an easily solvable equation (at X = 1) and tracing the zero point of homotopy function until getting the solution of original nonlinear equations (at A = 0). Neural Computation 5, 363-366 (1993) @ 1993 Massachusetts Institute of Technology
Liping Yang and Wanzhen Yu
364
As for the training of neural network, it is known that F(w) has a lot of local minima. For ordinary minimization methods, the algorithm often stops at local minima with large objective value and it is not easy to get a satisfactory solution. The homotopy method can overcome this difficulty to some extent. We can observe that the objective function F(w, A) is a polynomial of w when X = 1 and there exist very few minimum points. As X decreases to 0, the nonlinearity of objective function increases and more new minimum points appear. Because we have achieved a minimum point wk of F(w, Xk), which can provide a relatively better initial point for minimizing F(w, &+I), many unwanted local minima of F(w, are avoided. When computing with conventional BP method, it may occur that some components of w become so large that numerical instability will arise. This is because a very large change of x can only cause a very small change of s ( x ) , especially when 1x1 itself is large, that is, s'(x) -, 0 as 1x1 + 00. For this case, the usual treatment is to perform the scaling of w. The homotopy approach can avoid the infinity growth of w, because f { ( x ) > A. Our computational experiments show that the behavior of the homotopy method is good, although it does not guarantee the global minimum. In what follows, we shall study the process for decreasing A. Assume that we have solved minF(w, A) = F(w0, A) for a fixed X E (0,I]. Taking AX < 0, we need to solve minF(w, X + AX), which implies
vwF(w,
+ AX)lwo+Aw = 0
(4)
When w = WO, it gives
0
=
VWF(wo+ Aw, X + AX) - VWF(wo,A)
Whenever ViF(w0, A) is nonsingular, we can compute Aw by
a
AW = -V;F(wo, X ) - ' ~ V w F ( ~ X)AX o,
(7)
In practical computation, w o+ Aw can serve as a prediction of minimum point of F(w, X + AX). Table 1 illustrates an example of a separation problem. This problem contains 21 input points with corresponding outputs equal to +1 or -1. We use a neural network that has one hidden layer with 12 nodes. The sigmoid function s ( x ) is taken as 2/(1 + e-*) - 1. For this problem, the conventional BP algorithm fails to arrive at a totally correct solution, although many initial points have been tested. The homotopy method achieves a minimum point after 5172 iterations, which separate the input points correctly. The result is shown in Figure 1, where the region in white indicates the positive output of the neural network and the region in black indicates the negative output.
Backpropagation with Homotopy
365
Table 1: Separation Problem Input (0.05,0.05) (0.95,0.95) (0.05,0.95) (0.95,0.05)
Output
$1 +1 +1 +1 (0.50,0.50) -1 (0.95,0.50) +l (0.05,0.50) +1
Input
Output
Input
Output
(0.50,0.05) (0.50,0.95) (0.95,0.30) (0.30,0.05) (0.70,0.05) (0.70,0.95) (0.30,0.95)
+1 +1 -1 -1 -1 -1 -1
(0.05,0.30) (0.05,0.70) (0.40,0.50) (0.60,0.50) (0.50,0.40) (0.50,O.a) (0.95,0.70)
-1 -1 +1 +1 +1 +1 -1
Figure 1: Result of the homotopy method.
366
Liping Yang and Wanzhen Yu
Our computational experiments are made on about 20 separation problems whose sizes are similar to the illustrated one. The initial weights of neural networks are fixed for all problems. The conventional BP algorithm fails to get correct separation for one-third of the tested problems, especially for the problems for which the input data are irregular. The homotopy method can arrive at correct separation for all problems except one. It should be noted that the homotopy method usually takes more iterations compared to the conventional BP algorithm, because the former solves many minimization problems step by step as Xk decreases. This is a problem that needs further study. References Chow, S. N., Mallet-Paret, J., and Yorke, J. 1978. Finding zeros of maps: Homotopy methods that are constructive with probability one. Math. Cornp. 32, 887-889. Li, T. Y. 1987. Solving polynomial systems. The Math. Intelligencer 9, 33-39. Orfanidis, S. J. 1990. Gram-Schmidt neural nets. Neural Cornp. 2, 116-126. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. 1986. Learning internal representationsby error propagation. In Parallel DistributedProcessing, Vol. 1, D. E. Rumelhart and J. L. McClelland, eds. MIT Press, Cambridge, MA. Weymaere, N., and Martens, J. P. 1991. A fast and robust learning algorithm for feedforward neural networks. Neural Networks 4, 361-369.
Received 26 June 1992; accepted 29 September 1992.
This article has been cited by: 1. Nandakishore Kambhatla , Todd K. Leen . 1997. Dimension Reduction by Local Principal Component AnalysisDimension Reduction by Local Principal Component Analysis. Neural Computation 9:7, 1493-1516. [Abstract] [PDF] [PDF Plus] 2. B. Verma. 1997. Fast training of multilayer perceptrons. IEEE Transactions on Neural Networks 8:6, 1314-1320. [CrossRef] 3. F.M. Coetzee, V.L. Stonick. 1996. On a natural homotopy between linear and nonlinear single-layer networks. IEEE Transactions on Neural Networks 7:2, 307-317. [CrossRef] 4. Frans M. Coetzee , Virginia L. Stonick . 1995. Topology and Geometry of Single Hidden Layer Network, Least Squares Weight SolutionsTopology and Geometry of Single Hidden Layer Network, Least Squares Weight Solutions. Neural Computation 7:4, 672-705. [Abstract] [PDF] [PDF Plus]
NOTE
Communicated by Halbert White
Improving Rejection Performance on Handwritten Digits by Training with ”Rubbish” Jane Bromley John S. Denker A T b T Bell Laboratories, Holmdel, NJ 07733, U S A
Introduction Very good performance for the classification of handwritten digits has been achieved using feedforward backpropagation networks (LeCun et al. 1990; Martin and Pittman 1990). These initial networks were trained and tested on clean, well-segmented images. In the real world, however, images are rarely perfect, which causes problems. For example, at one time one of our best performing digit classifiers interpreted a horizontal bar as a 2; in this example the most useful response would be to reject the image as unclassifiable. The aim of the work reported here was to train a network to reject the type of unclassifiable images (“rubbish”)typically produced by an automatic segmenter for strings of digits (e.g., zip codes), while maintaining its performance level at classifying digits, by adding images of rubbish to the training set.
Solution to the Problem Our data consisted of 39,740 handwritten characters, obtained from automatically segmented zip codes. The segmentation process used a number of heuristic algorithms that selected the best vertical cuts through the zip codes.’ Since the cuts were vertical and the heuristics imperfect, many of the zip codes were poorly segmented resulting in 94% images of single digits and 6% images of rubbish. An example segmentation is shown in Figure 1. There are, of course, innumerable other patterns that are not good digits, but the rubbish created in this way was particularly relevant to our task. This data was used to train two networks-GOOD
‘An improved zip code reader has since been developed,which relies less on heuristics and for which the training and segmenting are even more strongly coupled (Burges et al. 1992).
Neural Computation 5,367-370 (1993) @ 1993 Massachusetts Institute of Technology
368
Jane Bromley and John S. Denker
Figure 1: A typical zip code showing, with dotted lines, one possible, but incorrect, segmentation. We designated as "rubbish" any subimage that could not be identified as a single digit because (1) there was more than one digit present, (2) there was only a small part of a digit present, or (3) the image was not of a digit. All images were labeled by hand. During training, the desired output vector for rubbish images was chosen to have all neurons low, while for a digit the corresponding neuron was high, the rest low.
was trained on 27,359 images of good digits and GOOD+RUBBISH was trained on these plus an extra 1642 images of rubbish. The architecture and training using backpropagation of these networks are described by LeCun et al. (1990). In Table 1 we see that training on rubbish caused no degradation when the nets were tested on good digits only, while it distinctly improved the ability of the network to reject rubbish-it went from rejecting 29.4 to 20.9% of the test patterns for 1% error on the test set. The rejection criterion was based on the difference between the two highest network outputs with highest confidence being assigned to classifications with the largest difference between these two outputs. An experimental investigation of rejection criteria for this network architecture has been made and this criterion came out as about the best (private communication from Yann LeCun). (No improvement is to be expected in the raw error rate, since the rubbish digits were scored as errors for both networks in this case.) The MSE showed no significant variability.
Improving Performance on Handwritten Digits
369
Table 1: Comparison of the Performance of the Two Networks after 20 Passes Through Their Respective Training Sets# Tested on: Trained on:
GOOD
MSE Error rate Reject
.026 4.3 8.5
Good GOOD+RUBBISH
Good+Rubbish GOOD GOOD+RUBBISH
4.4
.031 10.3
8.3
29.4
.027
.029 10.4 20.9
‘Each network was tested on two different testing sets: one consisting of only good digits and the other containing 6% of rubbish. MSE is the analog mean square error between desired and actual outputs. Reject is the number of patterns that had to be rejected (by the network) to achieve a 1% error rate on the remaining test digits.
Conclusions 0
0
0
0
0
Neural networks only do what they are trained to do. Contrary to other findings (Lee 19911, neural networks can generate an effective confidence judgment for rejecting ambiguous inputs. Rubbish subimages are ambiguous digits, multiple digits, partial digits, and noise. These are common in real-world images of handwritten digits. Our results show that including rubbish images in the training set improves the performance of a neural network digit classifier at rejecting such patterns. Performance on well-segmented digits is unaffected by this extra training. Accurate rejection is crucial in a system that automatically segments multidigit images and relies on its classifier to accept or reject possible segmentations of the image (Matan et al. 1992). Using this classifier in such a system led to an improvement in the recognition rate per zip code from 69 to 78%. Even more importantly, there was a vast improvement in the rejection performance of the whole system. Zip codes were sorted according to the neural network confidence measure of their being correct. Prior to training on rubbish, when the first 60%correctly classified zip codes are accepted, 10%erroneously classified zip codes were also accepted, while after training only 3% erroneously classified zip codes were included.
Acknowledgments Support of this work by the Technology Resource Department of the U.S.Postal Service under contract number 104230-90-C-2456is gratefully acknowledged.
370
Jane Bromley and John S. Denker
References Burges, C. J. C., Matan, O., LeCun, Y., Denker, J. S., Jackel, L. D., Stenard, C. E., Nohl, C. R., and Ben, J. I. 1992. Shortest path segmentation: A method for training a neural network to recognize character strings. Proc. Intl. Joint Conf. Neural Networks, IEEE, 3, 165-172. LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D. 1990. Handwritten digit recognition with a backpropagation network. In Neural Information Processing Systems,D. s. Touretzky, ed., pp. 396-494. Morgan Kaufmann, San Mateo, CA. Lee, Y. 1991. Handwritten digit recognition using K nearest-neighbor, radialbasis function, and backpropagation neural networks. Neural Comp. 3,440449. Matan, O.,Bromley, J., Burges, C. J. C., Denker, J. S., Jackel, L. D., LeCun, Y., Pednault, E. P. E., Satterfield, W. D., Stenard, C. E., and Thompson, T. J. 1992. Reading handwritten digits: A zip code recognition system. IEEE Computer 25,5943. Martin, G. L., and Pittman, J. A. 1990. Recognizing hand-printed letters and digits. In Neural Information Processing Systems, D. S. Touretzky, ed., pp. 405414. Morgan Kaufmann, San Mateo, CA. Received 27 July 1992; accepted 20 October 1992.
This article has been cited by: 1. Cheng-Lin Liu, H. Sako, H. Fujisawa. 2004. Effects of classifier structures and training regimes on integrated segmentation and recognition of handwritten numeral strings. IEEE Transactions on Pattern Analysis and Machine Intelligence 26:11, 1395-1407. [CrossRef] 2. C.-L. Liu, H. Sako, H. Fujisawa. 2004. Discriminative Learning Quadratic Discriminant Function for Handwriting Recognition. IEEE Transactions on Neural Networks 15:2, 430-444. [CrossRef] 3. D.R. Lovell, T. Downs, Ah Chung Tsoi. 1997. An evaluation of the neocognitron. IEEE Transactions on Neural Networks 8:5, 1090-1105. [CrossRef] 4. M. Revow, C.K.I. Williams, G.E. Hinton. 1996. Using generative models for handwritten digit recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 18:6, 592-606. [CrossRef] 5. G. C. Vasconcelos, M. C. Fairhurst, D. L. Bisset. 1995. Efficient detection of spurious inputs for improving the robustness of MLP networks in practical applications. Neural Computing & Applications 3:4, 202-212. [CrossRef]
NOTE
Communicated by Eric Baum
Vapnik-Chervonenkis Dimension Bounds for Two- and Three-Layer Networks Peter L. Bartlett' Department of Electrical and Computer Engineering, University of Queensland, Qld, 4072, Australia We show that the Vapnik-Chervonenkis dimension of the class of functions that can be computed by arbitrary two-layer or some completely connected three-layer threshold networks with real inputs is at least linear in the number of weights in the network. In Valiant's "probably approximately correct" learning framework, this implies that the number of random training examples necessary for learning in these networks is at least linear in the number of weights. This note addresses the question, How many training examples are necessary for satisfactory learning performance in a multilayer feedforward neural network used for classification? To define "satisfactory learning performance," we assume that the examples are generated randomly, and say that the trained network is approximately correct if it correctly classifies a random example with high probability. We require that the trained network will almost always be approximately correct, for any desired target function and any probability distribution of examples. This is known as "probably approximately correct" (or pac) learning (Valiant 1984). Blumer et al. (1989) show that the number of labelled examples necessary and sufficient for pac learning depends linearly on the VapnikCheruonenkis dimension (VC-dimension) of the set of functions that the learner can choose from. Definition 1. A class F of (0,1)-valued functions defined on a set X is said to shatter a finite set S C_ X if, for each of the 21sl classifications of the points in S, there is a function in F that computes the classification. The VC-dimension of F [written VCdim(F)l is the size of the largest subset of X that F shatters. We consider networks of processing units in layered, feedforward architectures with real-valued inputs and a single binary output. A feedfonvard architecture is a set of units (input units and processing units) arranged in a number of layers, and a set of connections, each of which joins 'Current address: Department of Systems Engineering, RSPhysSE, Australia National University, 0200, Australia.
Neural Computation 5,371-373 (1993) @ 1993 Massachusetts Institute of Technology
Peter L. Bartlett
372
one unit to another unit in a later layer. An L-layer network contains L layers of processing units. A feedfonoard threshold network consists of a feedforward architecture that has a particular real-valued weight and threshold associated with each connection and processing unit, respectively. Each processing unit in the network computes a linear threshold function, f ( x ) = 3-1 (Cixiwi - 0), where xi, wi, and O are the real-valued inputs to the unit, weights, and threshold, respectively, and %(a) is 1 if a 2 0 and 0 otherwise. Notice that a network consists of an architecture together with the weights and thresholds, so it computes a particular (0, 1)-valued function of its inputs. We refer to the VC-dimension of the class of functions that can be computed by threshold networks with a particular feedforward architecture A as the VC-dimension of that architecture, and write VCdim(A). The VC-dimension of an arbitrary feedforward architecture is not known precisely. Baum and Haussler (1989) show that the VC-dimension of a feedforward architecture with N processing units and W weights is no more than 2Wlog, eN (where e is the base of the natural logarithm), and that the VC-dimension of a completely connected two-layer architecture with ko input units and kl first-layer units is at least 2[k1/2]ko. (A completely connected multilayer network has connections between all pairs of units in adjacent layers.) In this note, we give lower bounds on the VC-dimension of arbitrary two-layer architectures and some completely connected three-layer architectures. By the results of Blumer et al. (1989) the bounds indicate in all cases that the sample size necessary for pac learning is at least proportional to W, the number of weights in the network. Proofs of the results are given in the full version of this note (Bartlett 1992). To show that the VC-dimension of an architecture is at least d, we can construct a shattered set of size d . The problem of constructing such a set can be decomposed by separately constructing defining sets for units in a network.
Definition 2. A set S = { x , , x2, . . . ,x,} c U& is a defining set for a unit u in a feedforward threshold network with ko real-valued inputs if 1. We can classify the points in S in each of the 21sI distinct ways by slightly perturbing the weights and threshold of unit u.
2. Slightly perturbing the weights and threshold of units other than u will not affect the classification of the points in S. A point x E I& is an oblivious point for this network if the classification of x is unaffected by sufficiently small perturbations of the network weights.
Theorem 3. Let A be a feedforward architecture. Consider a set of processing units U in this architecture and a threshold network N with architecture A that has an oblivious point. If there is a finite defining set S , for each unit u in U , then VCdim(A) 2 CuEU IS,[ 1.
+
Vapnik-Chervonenkis Dimension
373
By finding appropriate defining sets, we can use Theorem 3 to give the following lower bounds.
Theorem 4. Let A be an arbitrary two-layer feedforward architecture. If A has I connectionsfrom the input units to other units, then VCdim(A) 2 I + 1. Theorem 5. Let A be the three-layer, completely connected architecture with ko > 0 input units, kl > 0 first-layer units, k2 > 0 second-layer units, and a single output unit.
+
(a) If ko 2 k l , and kz 5 2&1/(g/2 k1/2 + l), then VCdim(A) 2 kokl ki(kz - 1) 1. (b) If 1 < ko < k1 2 k2, then VCdim(A) 2 kok1 kl(k2 - 1)/2 1.
+
+
+
+
These results imply that for learning two-layer networks or completely connected three-layer networks with kz not too large, the sample size must increase at least linearly with the number of weights. The results also give lower bounds for learning in networks of processing units with a sigmoid transfer function, since a sigmoid network can compute any function on a finite set that a threshold network can compute.
Acknowledgments This work was supported by the Australian Telecommunications and Electronics Research Board. Thanks to T. Downs, R. Lister, D. Lovell, R. Williamson, and S. Young for comments on a draft.
References Bartlett, P. L. 1992. Lower bounds on the Vupnik-Chervonenkisdimension of multi-layer threshold networks. Intelligent Machines Laboratory, Department of Electrical and Computer Engineering, University of Queensland, Brisbane, Australia, Tech. Rep. IML92/3. October 1992. Baum, E. B., and Haussler, D. 1989. What size net gives valid generalization? Neural Cornp. 1,151-160. Blumer, A., Ehrenfeucht, A., Haussler, D., and Warmuth, M. K. 1989. Learnability and the Vapnik-Chervonenkisdimension. J. Assoc. Computing Machin. 36(4), 929-965. Valiant, L. G . 1984. A theory of the learnable. Commun. ACM 27(11), 1134-1143. Received 24 September 1992;accepted 29 September 1992.
This article has been cited by: 1. Hiroki Suyari, Ikuo Matsuba. 1999. Information theoretical approach to the storage capacity of neural networks with binary weights. Physical Review E 60:4, 4576-4579. [CrossRef] 2. Ansgar H L West, David Saad. 1998. Journal of Physics A: Mathematical and General 31:45, 8977-9021. [CrossRef]
Communicated by Richard Andersen
A Neural Network for the Processing of Optic Flow from Ego-Motion in Man and Higher Mammals Markus Lappe Josef P. Rauschecker Laboratory of Neurophysiology, N M H , Poolesville, M D 20837, USA and Max Planck Institute for Biological Cybernetics, Tubingen, Germany
Interest in the processing of optic flow has increased recently in both the neurophysiological and the psychophysical communities. We have designed a neural network model of the visual motion pathway in higher mammals that detects the direction of heading from optic flow. The model is a neural implementation of the subspace algorithm introduced by Heeger and Jepson (1990). We have tested the network in simulations that are closely related to psychophysical and neurophysiological experiments and show that our results are consistent with recent data from both fields. The network reproduces some key properties of human ego-motion perception. At the same time, it produces neurons that are selective for different components of ego-motion flow fields, such as expansions and rotations. These properties are reminiscent of a subclass of neurons in cortical area MSTd, the triple-component neurons. We propose that the output of such neurons could be used to generate a computational map of heading directions in or beyond MST.
1 Introduction
The concept that optic flow is important for visual navigation dates from the work of Gibson in the 1950s. Gibson (1950)showed that the optic flow pattern experienced by an observer moving along a straight line through a static environment contains a singularity that he termed the focus of expansion. He hypothesized that the visual system might use the global pattern of radial outflow originating from this singularity to determine the translational heading of the observer. A host of studies in human psychophysics have followed up Gibson’s ideas (Regan and Beverly 1982;Rieger and Toet 1985;Warren et al. 1988; Warren and Hannon 1988, 1990). Regan and Beverly (1982)rejected his hypothesis on the basis that the optic flow pattern that arrives on the Neural Computation 5,374-391 (1993) @ 1993 Massachusetts Institute of Technology
Neural Network for Processing Optic Flow
375
retina is radically altered by eye movements of the observer. Then the flow field becomes a superposition of the radial outflow pattern with a circular flow field that is obtained when the eyes move in the orbita. Generally the resulting vector field may also have a singular point similar to a focus of expansion, but this point does not necessarily coincide with the heading direction. If, for instance, the eye rotation results from the fixation of a point in the environment, the singularity will be at the fixation point instead of the destination point. Nevertheless, Warren and Hannon (1990) found humans capable of judging their heading with great accuracy from optic flow patterns that simulated translation plus eye rotation. Their subjects were able to perceive their heading with a mean error between one and two degrees solely from the optic flow. No nonvisual information such as oculomotor signals was necessary. This ability persisted over a natural range of speeds and over a variation of the number of visible moving points between 10 and several hundred. The performance of the subjects was at chance, however, when no depth information in the form of motion parallax was available. In the visual system there are at least two (maybe three) main streams of information flow (Mishkin etal. 1983; Livingstone and Hubel 1988; Zeki and Shipp 1988). In the simplest depiction, there is an inferotemporal system that is mainly responsible for the processing of form, and a parietal system that processes motion (Ungerleider and Mishkin 1982). Within the cortical motion system, one of the prominent and most investigated areas in primates is the middle temporal area or area MT (Allman and Kaas 1971). In cats the probable homologue for MT is the Clare-Bishop area (Clare and Bishop 1954), also called area PMLS (Palmer et al. 1978). Evidence from both areas suggests that they participate in the processing of flow field information. Both areas contain neurons that are highly direction selective and respond well to moving stimuli. It has first been found in cat area PMLS that a majority of neurons prefer movement away from the area centralis, that is, centrifugal motion (Rauschecker et al. 1987a,b; Brenner and Rauschecker 1990). The same has been found in monkey area MT (Albright 1989), thus strengthening the likelihood of a homology between these two areas. Other studies have revealed single neurons in PMLS that respond well to approaching or receding objects (Toyama et al. 1990). More recently, a number of studies have described neurons in the dorsal part of monkey area MST (MSTd) that respond best to large expanding/contracting, rotating, or shifting patterns (Tanaka and Saito 1989a,b; Andersen et al. 1990; Duffy and Wurtz 1991a,b). The response of these neurons often shows a substantial invariance to the position of the stimulus. Duffyand Wurtz (1991a,b) found that a majority of the neurons in MSTd responded not only to one component of motion of the stimulus pattern (e.g., expansion or contraction), but rather to two or all three of them separately. About one-third of MSTd cells displayed selectivity to
376
Markus Lappe and Josef I? Rauschecker
expansions or contractions and clockwise or counterclockwise rotations and showed broad directional tuning for shifting dot patterns when tested with these stimuli one after another. It is these "triple component cells" that our model is mainly concerned with. Furthermore, cells in MSTd are unselective for the overall speed of a stimulus and for the amount of depth information available in the stimulus. There have been a number of computational approaches to extract navigational information from optic flow focusing on different mathematical properties of the flow field. The difficulty of the task is that in the mapping of three-dimensional movements onto a two-dimensional retina some information is lost that cannot be fully recovered. Models that use differential invariants (Koenderink and van Doorn 1981; LonguetHiggins and Prazdny 1980; Waxman and Ullman 1985) require dense optic flow to compute derivatives. By contrast, humans are quite successful with sparse fields (Warren and Hannon 1990). Models based on algorithms that solve a set of equations for only a small number of vectors (Prazdny 1980; Tsai and Huang 1984), on the other hand, require precise measurements and are very sensitive to noise. Methods that rely on motion parallax or local differential motion (Longuet-Higgins and Prazdny 1980; Rieger and Lawton 1985) are in agreement with the psychophysical data in that they fail in the absence of depth in the environment. However, they require accurate measurements at points that are close to each other in the image but are separated in depth, which is an especially difficult task to accomplish. Furthermore, recent psychophysical studies (Stone and Perrone 1991) have shown that local depth variations are not necessary. Least-square minimization algorithms (Bruss and Horn 1983; Heeger and Jepson 1990) that use redundant information from as many flow vectors as are available are robust and comparatively insensitive to noise. None of the above-mentioned algorithms is clearly specified in terms of a neural model. Given the current advances in visual neurophysiology, it seems desirable to construct a neural network for ego-motion perception that is consistent with the neurophysiological and psychophysical data. Recently a network model of heading perception in the simpler case without eye movements has been described (Hatsopoulos and Warren 1991), which accounts for some psychophysical findings. A neural model we presented in brief form earlier together with first results from the model described in this paper (Lappe and Rauschecker 1991) is also concerned with pure translations. It uses a centrifugal bias similar to the one found in PMLS and MT to achieve precise heading judgments with neuronal elements that are as broadly directionally tuned as the cells found in these areas. In this article we present a new neural network that succeeds when the radial flow pattern is disturbed by eye movements. The network is capable of reproducing many of the psychophysical findings, and the
Neural Network for Processing Optic Flow
377
single units exhibit great similarity to the triple component cells of Duffy and Wurtz (1991a,b) in area MSTd. 2 The Model
Our network is built in two layers. The first layer is designed after monkey area MT and represents the input to the network. The second layer is constructed to yield a representation of the heading direction as the output of the net and thus could form a model of MSTd. In each network layer we employ a population encoding of the relevant variables, namely the speed and direction of local movements in layer one and the heading direction of the individual in layer two. The computation of the direction of translation is based on the subspace algorithm by Heeger and Jepson (1990). Its main course of action is to eliminate the dependencies on depth and rotation first and thereby gain an equation that depends only on the translational velocity. Therefore it bears some similarity to Gibson's original claim that the visual system can decompose the optic flow into its translational and rotational components. We will restrict the scope of our model to such eye movements as occur when the observer keeps his eyes fixed on a point in the environment while he is moving. This is a natural and frequently occurring behavior, and we believe that using assumptions that are a reflection of the behavior of an animal or a human being makes it more likely to gain results that can be compared with experimental data. Although it is mathematically possible to include any type of eye movements, it is not very likely that the eyes would rotate around their long axis to a significant amount during locomotion. Note that our assumption includes the case of no eye movements at all, since it can be described as gazing at a point infinitely far away. 2.1 Optic Flow and the Subspace Algorithm. Optic flow is the projection of the motion of objects in the three-dimensional world onto a two-dimensional image plane. In three dimensions, every moving point has six degrees of freedom: The translational velocity T = (Tx,T,,TJ and the rotation 52 = (ax, R,, R#. When an observer moves through a static environment all points in space share the same six motion parameters. The motion of a point R = (X, Y, Z)' in a viewer-centered coordinate system is V = -(a x R T). This motion is projected onto an image plane. Writing two-dimensional image vectors in small letters, the perspective projection of a point is r = ( x , y)' = f (X/Z, Y/Z)', where f denotes the focal length. Following Heeger and Jepson (1990) the image velocity can be written as the s u m of a translational and a rotational component:
+
Markus Lappe and Josef l? Rauschecker
378
where p ( x ,y) = 1/Z is the inverse depth, and
A(x,y)=
( -f
")
-f Y
0
The unknown depth and translational velocity are multiplied together and can thus only be determined up to a scale factor. Regarding therefore the translation T as a unit vector, one is left with six unknowns: p, the two remaining components of T, and the three components of 51, but only two known quantities, Ox, Oy. The subspace algorithm uses flow vectors at five distinct image points to yield an overdetermined system of equations that is solved with a minimization method in the following way: The five separate equations are combined into one matrix equation 0 = C(T)q where 0 = (el,.. . ,&)' is now a 10-dimensional vector consisting of the components of the five image velocities, q = [P(x~,yl), . . . ,p(x5,ys),R,, R,, R,]' an eight-dimensional vector, and C(T) a 10 x 8 matrix composed of the A(xi,yi)Tand B(xi,yi)matrices:
C(T) =
(
A h y1)T 1
d
B(x1,y1)
A(x5,y5)T B("5, y5)
Heeger and Jepson (1990) then show that the heading direction can be recovered by minimizing the residual function
R(T) = IIQ'C'(T)112, where C'-(T)is a matrix that spans the two-dimensional orthogonal complement of C(T). 2.2 Restrictionto Fixations during Locomotion. We now restrict ourselves to only those eye movements that arise through the fixation of a ' the center of the visual field while the observer point F = (O,O,l / p ~ ) in moves along a straight line. The rotation that is necessary to h a t e this point can be derived from the condition that the flow at this point has to be zero:
(:)=p.(
-f
0 0 O - f O f) 0 -f O)T+(f 0 0)
Choosing R, = 0 we find
f) = PF(T,,-Tx,O)*. The
optic flow then is:
Neural Network for Processing Optic Flow
379
The case of a straight translation without any eye movements can easily be described within this framework by considering a fixation point that is infinitely far away. Then PF and the rotational velocity s1 are zero, resulting in a purely translational flow. The optic flow equation above has only four unknowns: p ( x , y ) , PF, T,, and Ty.Combining the equations for two different flow vectors into one matrix equation in the same way as before yields 0 = C (T) [ P ( x I , ~ ~ ) , P ( x ~ , Y ~ ) ,where ~ F ] ' , C(T) is now only a 4 x 3 matrix, the orthogonal complement of which is a line given by the vector C'(T). The residual function becomes the scalar product between this vector and the observed flow:
R(T) = IQtC'(T)I2
(2.2)
Since the optic flow is a linear function of the translational direction, R(T) does not have a single minimum but is equal to zero along a line in the (T,, Ty ) .plane. Therefore one such minimization alone cannot give the translational velocity, rather several pairs of flow vectors with different R(T) functions have to be used in conjunction. 2.3 The Network. In the first layer of the network, which constitutes the flow field input, 300 random locations within 50" of eccentricity are represented. We assume a population encoding of the optical flow vectors at each location by small sets of neurons that share the same receptive field position but are tuned to different directions of motion. Each such group consists of n' neurons with preferred directions ek, k = 1, . . . ,n'. The flow vector 8 is represented by the sum over the neuronal activities sk in the following way: n'
8 = Cskek
(2.3)
k=l
We do not concern ourselves with how the optic flow is derived from the luminance changes in the retina or how the aperture problem is solved. Neural algorithms that deal with these questions have already been developed (Bulthoff etal. 1989; Hildreth 1984; Yuille and Grzywacz 1988). A physiologically plausible network model that yields as its output a population encoding like the one we use here has been proposed by Wang et al. (1989). It can be thought of as a preprocessing stage to our network, modeling the pathway from the retina to area MT or PMLS. Since we start out with a layer in which the optic flow is already present, we have to guarantee that the tuning curves of the neurons and the distributions of the preferred directions match the requirement of equation 2.3. As the simplest choice for our model, we use a rectified cosine function with n' = 4. It preserves the most prominent feature of the observed directional tuning curves in MT/PMLS,namely broad
Markus Lappe and Josef P. Rauschecker
380
unidirectional tuning with no response in the null direction. The preferred directions are equally spaced, e k = [cos(nk/2),sin(xk/2)], and for the unit's response to a movement with speed 60 and direction $, the tuning curve is
6, cos(4 - nk/2) if cos($ - nk/2) > 0 s k = {
0
otherwise
The second layer represents a population encoding of the translational direction of the movement of the observer, which is represented by the intersection point of the 3D-movement vector T with the image plane. There are populations of n neurons at possible intersection points whose combined activities U I give the perceived direction. But here the sum of the activities U = Cbl U Iat each position yields a measure of how likely this position is to be the correct direction of movement. The perceived direction is chosen to be the one that has the highest total activity. The output of a second layer neuron is a sigmoid function g(x) of the sum of the activities of its m input neurons weighted by synaptic strengths Ijkl and compared to a threshold p:
Here IikI denotes the strength of the connection between the Ith output neuron and the kth input neuron in the population that represents image location i. The sigmoid function is symmetric such that g ( - x ) = 1-g(x). The connections and their strengths are set once before the network is presented with any stimuli, and are fixed afterward. First a number of image locations are randomly assigned to a second layer neuron. Then, values for the synaptic strengths are calculated so that the population of neurons encoding a specific T is maximally excited when R(T) equals zero. Although the neuron may receive input from a large number of image locations we start the calculation of the connections with only two in order to keep it simple. We want the sum in equation 2.4 to equal the scalar product on the right side of equation 2.2:
For every single image location i we have
Substituting equation 2.3 we find
Neural Network for Processing Optic Flow
381
Therefore we set the synaptic strengths to
If the neuron is connected to more than two image locations the input connections are divided into pairs and the connections are calculated separately for each pair. Now the question of when R(T) is minimal comes down to the question of when all the neurons’ inputs balance each other to give a net input of zero. Consider two output neurons U Iand UI’receiving input from the same set of first layer neurons but with inverse connections such that lik,’ = -Jiu. Then, if the threshold ,LA equals zero, the sum of both neurons’ activities is equal to 1 regardless of their inputs, since the sigmoid input/output function is symmetric. If, however, ,LA has a slightly negative value, both sigmoid functions will overlap and the sum will have a single peak at an input value of zero. Such a matched pair of neurons generates its maximal activity when R(T)= 0. MSTd neurons have very large receptive fields and do certainly receive input from more than 2 image locations. Also MSTd neurons show the same response in the case of as little as 25 visible moving dots as they do in the case of 300 (Duffyand Wurtz 1991a). We chose each of our model neurons to receive input from 30 image locations. We restrict the space for the encoded heading directions to the innermost 20 x 20” of the visual field, since this approximates the range over which the psychophysical experiments have been carried out. Nevertheless, each layer-two neuron may receive input from a much larger part of the visual field. The layer-two neurons form a three-dimensional grid with 20 x 20 populations encoding one degree of translation-space each, and 20 pairs of neurons in each population. 3 Results 3.1 Comparison of the Network’s Performance with Human Psychophysical Data. The network was tested with simulated flow fields with different motion parameters. We used a cloud-like pattern that consisted of a number of dots, the depths of which were randomly distributed within a given range. To test the behavior without eye movements a translational direction was randomly chosen within the innermost 20 x 20” and the rotation was set to zero. To test cases with eye rotation the translational direction was again chosen randomly and the fixation point was set in the center of the image plane and assigned a specific depth. The rotational component was then calculated from the condition that the flow at the fixation point must be zero. Each simulation run consisted of 100 presentations of different flow fields, after which we calculated the mean error as the mean angular difference between the network’s computed direction and the correct direction.
Markus Lappe and Josef P. Rauschecker
382
12
1-
-
10 -
1 e
4 -
.
2-
". ........................... 04
0
2
4
6
8
10
12
14
16
18
20
Number of flow vectors
Figure 1: Performance with sparse flow fields. The heading error becomes small with as little as 10 vectors. The number of dots necessary is about the same with or without eye movements.
We found the network's performance to be well within the range of human performers (Warren et al. 1988). For pure translation as well as with eye movements the mean error settled between 0.5 and 1.5", showing that the network always has its activity maximum at a position close to the translational direction. Consistent with the experiments of Warren et al. (1988) we found very little influence of speed on the performance of the network. Humans are able to detect their heading with very sparse flow fields consisting of only ten dots (Warren and Hannon 1988,1990). In order to test how many flow vectors are needed in our model under otherwise optimal conditions we made an additional assumption: We assumed that a given pair of vectors in the flow field serves as input to at least one pair of neurons in each population of the output layer. If this were not the case, some populations would receive more information than others and the number of dots neccessary for correct heading estimation would depend on the heading direction. Our assumption ensures that all heading directions are represented equally. Considering the large number of cortical neurons this assumption is biologically reasonable since it would be approximately fulfilled if the number of neurons in the output layer were large. For the simulations, we distributed the connections between input and output neurons in such a way as to fulfill the assumption. The results of the simulations are shown in Figure 1. The cloud of dots extended in depth from 11 to 31 m with a fixation point at 21 m. The translational
Neural Network for Processing Optic Flow
383
speed was 2 m/sec. In both the pure translation and in the eye rotation case the network started to detect the heading with the desired accuracy at approximately 10 points, although with eye rotation the error did not quite reach the optimum and continued to decrease as more flow vectors were provided. Mathematically two vectors are sufficient to compute the heading of a purely translational movement (Prazdny 19801, but humans fail to detect their heading with only two visible dots (Warren et al. 1988). Our network does not know a priori if the flow field is generated by a translation alone. It therefore has to rely on the flow pattern and needs about the same number of vectors as with the eye movements. Humans also fail when eye rotations are paired with a perpendicular approach to a solid wall, where all points are at the same depth (Rieger and Toet 1985; Warren and Hannon 1990). In this case the subjects' performances are at chance and they often report themselves as heading toward the fixation point. Because of a well-known ambiguity in planar flow fields (Tsai and Huang 19841, we were not able to test the depth dependence of the network with approaches to a plane at different angles. We therefore varied the depth range of the cloud. Doing this revealed that with decreasing depth the peak in the second layer grows broader and covers the fixation point as well as the heading direction. This can be seen in Figure 2 where the summed population activities in the output layer are shown on a grayscale map, together with reduced pictures of the input flow fields. Input and output are compared for situations that differ in the amount of depth in the image. In Figure 2a a flow field is shown in which the depth range of the cloud of dots is large, extending from 7 to 30 m. The observer moves toward the cross while he is keeping his eyes fixed on an object ( x ) in the center. There is no apparent focus of expansion. The network output (Fig. 2b) shows an easily localizable brightness peak in the upper left that corresponds to the correct heading direction as indicated by the cross. Figure 2c shows the same movement as Figure 2a, but here the depth range of the cloud is much smaller, ranging from 19 to 21 m. In this case the flow field looks very much like an expansion centered at the fixation point. In the corresponding network output (Fig. 2d), the peak is very broad and includes the fixation point in the center. A maximum nevertheless still exists, although much less pronounced, and in the simulations the network was still able to compute the right heading. However, the solution is unstable and very sensitive to noise. To illustrate this, we randomly varied the amplitudes of the flow vectors by stretching them by a factor distributed uniformly between 0.9 and 1.1, thus adding 10% noise. The results for all conditions are shown in Figure 3 for different depth ranges. This small amount of noise increases the error for the rotational movement to around 7", whereas in the purely translational case the network performance is unaffected. With growing depth differences this separation becomes less pronounced and the error values for the rotational case decrease.
384
Markus Lappe and Josef P. Rauschecker
Figure 2 Influence of image depth on the heading judgment of the network. (a) Depth-rich flow field. Movement is toward the cross (+) while the x in the center is hated. (b) Output of the network. The response peak gives the correct heading. (c) Same movement with only little depth differences. (d) Brightness maximum in the output of the network is very broad and includes the fixation point. 3.2 Comparison with Single Cell Properties in MSTd. The output layer cells of our model network exhibit a remarkable resemblance to some triple component neurons in MSTd. Figure 4 shows the response of one output layer cell to presentations of each of the components (e.g., expansions, rotations) at different places in the visual field. The neuron
Neural Network for Processing Optic Flow
385
-
.......... ..........
2-
Eye8 lixed on wga
. no nOiM
No aye movaments . no noiw E u a lixed on w0.1 - 10% noiw
.....
I....
'i.,
............................................................
* ..............................
Figure 3 Heading error versus depth. In the noise-free condition, heading calculation is accurate despite the broad peak in the network output depicted in Figure 2d. Adding a small amount of noise, however, shows that the solution in the eye movement case is unstable and gives rise to a large error. receives input from 30 positions distributed inside a 60 x 60" receptive field centered in the lower right quadrant of the visual field and extending u p to 10" into each of the neighboring quadrants, thus including the vertical and horizontal meridians and the fovea or area centralis (Fig. 4a). This receptive field characteristic is common for MSTd neurons ( h f f y and Wurtz 1991b). The neuron in our example is a member of the population that represents a heading direction in the upper right quadrant at an eccentricity of 11". Figure 4b shows the cell's broad unidirectional tuning and little selectivity for stimulus speed. The plots c-f in Figure 4 illustrate the responses of the neuron to expansions, contractions, clockwise rotations, and counterclockwise rotations, respectively. The ( x , y)-plane represents a visual field of 100 x loo", the height is the response of the neuron to a stimulus centered at (x,y). The size of the stimulus was always large enough to cover the whole receptive field of the cell. For a stimulus in the center of the visual field the cell responds favorably to counterclockwise rotations and expansions, although there also is a smaller response to contractions. There are very large areas of position invariance covering almost half of the visual field for a given stimulus movement. The response to counterclockwise rotations, for instance, is constant in most of the upper two quadrants. The cell also shows the reversals in selectivity observed in 40% of triple-component neurons in MSTd (Duffyand Wurtz 1991b). In our example, moving the center of the stimuli to the right causes the response
386
Markus Lappe and Josef P. Rauschecker
Figure 4 Responses of one output layer cell. (a) Receptive field of the cell as defined by its input connections. (b) Broad unidirectional response to global shifts of a dot pattern. No tuning to a particular stimulus speed. (c-f) Responses to expanding, contracting, and rotating patterns centered at different positions within the visual field reveal large areas of position invariance and sudden reversals of selectivity.
Neural Network for Processing Optic Flow
387
to contractions to disappear. Moving the center of the stimuli to the lower left causes the cell's selectivity to change to favor contractions and clockwise rotations. There are intermediate positions where the cell responds to both modes of one component. For example, in plots b and c, there is a vertical strip in the center where the cell responds to expansions as well as to contractions. The response reversals take place along edges running across the visual field, which is similar to the findings of h f f y and Wurtz (1991b). The reason for this is that the residual function, which is computed by the neuron, equals zero along a line in the (Tx,Ty)space, as mentioned before. The edge of the surface that marks the neuron's response to expansions follows this line. The neuron signals only that the heading direction lies somewhere along the edge. The edges of all neurons in one population overlap at the point that corresponds to the heading represented by that population, When the network is presented with a flow field, the population encoding the correct heading is maximally excited since all of their neurons will respond. In populations representing other directions, only part of the neurons will be active, so that the total activity will be smaller. It is worth noting that the edges of reversal do not necessarily cross the receptive field of the cell. In the example of Figure 4, the reversal from selectivity for expansion to selectivity for contraction takes place in the left half of the visual field outside the cell's receptive field, which occupies the lower right quadrant. Likewise, it sometimesoccurred in the simulations that the reversal for rotation was not even contained within the 100 x 100" visual field. Another interesting observation is that the edges for rotation and expansion/contraction often cross each other approximately orthogonally. The position of the intersectionpoint, on the other hand, can vary widely between cells.
4 Discussion
We have designed a neural network that detects the direction of egomotion from optic flow and is consistent with recent neurophysiological and psychophysical data. It solves the traditional problem of eye movements distorting the radial flow field by means of a biologically reasonable mechanism. The model reproduces some key properties of human ego-motion perception, namely, the ability to function consistently over a range of speeds, the ability to work with sparse flow fields, and the difficulties in judging the heading when approaching a wall while moving the eyes. The network also generates interesting neuronal properties in its output layer. Simple intuitive models for heading perception might expect a
388
Markus Lappe and Josef P.Rauschecker
single neuron to show a peak of activity for an expansion at a certain preferred heading direction. Instead, our model uses a population encoding in which single cells do not carry all the information about the perceived heading, but rather the combined activity of a number of cells gives that information. At the level of a single neuron, the position information is contained in the edges of reversal of the cell's preferred direction of stimulus motion. The resulting characteristics of the output neurons in our network show great similarity to the response properties of a particular cell class recently described in MSTd, the triplecomponent neurons (Duffy and Wurtz 1991a,b). These cells, which comprise about one-third of all neurons in MSTd, display selectivity not only for expansion or contraction, but also for one type of rotation and one direction of shifting patterns. Most of the neuronal outputs produced by our network have similar properties. It appears tempting to postulate, therefore, that the output of triple-component cells could be used to compute directional heading, either within MST or in another area. A potential problem for using the output of MSTd neurons to compute heading direction concerns their apparent position invariance. In a neural network that is supposed to signal the directional heading, the response of the output layer cells has to depend on the position of the stimulus in some way. Most neurons in MSTd seem to be insensitive against changes of stimulus position, although the proportions of position invariant cells reported in different studies vary and obviously depend on the exact stimulus paradigm (Andersen etal. 1990; Duffy and Wurtz 1991b; Orban et al. 1992). In our network model many output neurons would appear position invariant when tested over a limited, wide range of stimulus positions. Interestingly, the proportion of position dependent responses seems to be highest among triple-component neurons (Duffy and Wurtz, 1991b): In about 40% of these cells component selectivity for a flow field stimulus is reversed along oriented edges, which conforms exactly with the behavior of our model neurons. It is conceivable, therefore, that it is this subtype of triple-component neurons that is involved in the computation of heading direction. More neurons of this type might be encountered in MSTd if one specifically looks for them. Their frequency of occurrence may depend on laminar position, or they might be found even more frequently at another processing stage. A closer look at the experimental data reveals that the number of triple component cells in MSTd may indeed have been underestimated. The different cell types in MSTd do not fall in strictly separate classes but rather form a continuum changing smoothly from triple to single component cells (Duffy and Wurtz 1991a). Therefore, double and single component cells might be regarded as possessing some, albeit weak, responses to the other components. It is equally possible, however, that single and double component cells simply do not participate in the detection of heading direction, but serve some other purpose. Single com-
Neural Network for Processing Optic Flow
389
ponent cells, for example, could be involved in the analysis of object motion. The network can also generate cells that are selective to fewer components when the restriction is removed that rotations are due to the fixation of an object. Allowing arbitrary rotations, including ones around a sagittal axis through the eye, results in neurons that are unselective for rotations and respond only to translations and expansions/contractions. Under the different assumption that only frontoparallel rotations, including for instance pursuit eye movements, will occur, the neurons show strong, fully position invariant responses to rotational stimuli, which dominate over the selectivity for translation and expansion/contraction (Lappe and Rauschecker 1993). We would like to emphasize that the neurons in our model do not decompose the flow field directly. At no point is the translational part of the optic flow actually computed. The neurons rather test the consistency of a measured optic flow with a certain heading direction. In this way, a response selectivity for rotations, for example, does not mean that the neuron is actually tuned to the detection of a rotation in the visual field, but this property rather has to be regarded as the result of a more complex selectivity. The cells in the output layer of our model form a computational map of all possible heading directions. However, it would not be easy to find this map in an area of visual cortex, since the topography reveals itself only in the properties of cell populations. Simultaneous recording from an array of electrodes would perhaps be the only way to demonstrate this computational map experimentally. Our model suggests that one has to focus on the mapping of selectivity reversals and explore these more thoroughly, especially in triple component cells: Neurons in neighboring columns should show smooth shifts of their preferences. The concurrent activity of such cells in a hypercolumn would signal one particular heading direction in space, which is given by the intersection point of their reversal edges for expansion and contraction.
References Albright, T. D. 1989. Centrifugal directionality bias in the middle temporal visual area (MT) of the macaque. Visual Neurosci. 2, 177-188. Allman, J. M., and Kaas, J. H. 1971. A representation of the visual field in the caudal third of the middle temporal gyrus of the owl monkey (Aotus trivirgatus). Brain Res. 31, 85-105. Andersen, R., Graziano, M., and Snowden, R. 1990. Translational invariance and attentional modulation of MST cells. SOC.Neurosci. Abstr. 16, 7 . Brenner, E., and Rauschecker, J. I? 1990. Centrifugal motion bias in the cat’s lateral suprasylvianvisual cortex is independent of early flow field exposure. J. Physwl. 423,641460.
390
Markus Lappe and Josef P. Rauschecker
Bruss, A. R., and Horn, B. K. I? 1983. Passive navigation. Computer Vision, Grahics, Image Process. 21, 3-20, Bulthoff, H., Little, J., and Poggio, T. 1989. A parallel algorithm for real-time computation of optical flow. Nature (London) 337,549-553. Clare, M. H., and Bishop, G. H. 1954. Responses from an association area secondarily activated from optic cortex. 1.Neurophysiol. 17,271-277. Duffy, C. J., and Wurtz, R. H. 1991a. Sensitivity of MST neurons to optic flow stimuli. I. A continuum of response selectivity to large-field stimuli. J. Neurophysiol. 65(6), 1329-1345. Duffy, C. J., and Wurtz, R. H. 1991b. Sensitivity of MST neurons to optic flow stimuli. 11. Mechanisms of response selectivity revealed by small-field stimuli. 1.Neurophysiol. 65(6), 1346-1359. Gibson, J. J. 1950. The Perception of the Visual World. Houghton Mifflin, Boston. Hatsopoulos, N. G., and Warren, W. H., Jr. 1991. Visual navigation with a neural network. Neural Networks 4(3),303-318. Heeger, D.J., and Jepson, A. 1990. Visual perception of three-dimensional motion. Neural Comp. 2, 129-137. Hildreth, E. C. 1984. The Measurement of Visual Motion. MIT, Cambridge, MA. Koenderink, J. J., and van Doom, A. J. 1981. Exterospecific component of the motion parallax field. 1.Opt. SOC.Am. 71(8), 953-957. Lappe, M., and Rauschecker, J. P. 1991. A neural network for flow-field processing in the visual motion pathway of higher mammals. SOC.Neurosci. Abstr. 17,441. Lappe, M., and Rauschecker, J. P. 1993. Computation of heading direction from optic flow in visual cortex. In Advances in Neural Information Processing Systems, Vol. 5, C. L. Giles, S. J. Hanson, and J. D.Cowan, eds. (in press). Morgan Kaufmann, San Mateo, CA. Livingstone, M., and Hubel, D. 1988. Segregation of form, color, movement, and depth: Anatomy, physiology, and perception. Science 240, 740-749. Longuet-Higgins, H. C., and Prazdny, K. 1980. The interpretation of a moving retinal image. Proc. R. SOC. London B 208,385-397. Mishkin, M.,Ungerleider, L. G., and Macko, K. A. 1983. Object vision and spatial vision: Two cortical pathways. Trends Neurosci. 6,414-417. Orban, G. A., Lagae, L., Verri, A., Raiguel, S., Xiao, D.,Maes, H., and Torre, V. 1992. First-order analysis of optical flow in monkey brain. Proc. Nutl. Acad. Sci. U.S.A. 89, 2595-2599. Palmer, L. A., Rosenquist, A. C., and Tusa, R. J. 1978. The retinotopic organization of lateral suprasylvian visual areas in the cat. 1. Comp. Neurol. 177, 237-256. Prazdny, K. 1980. Egomotion and relative depth map from optical flow. Biol. Cybern. 36,87-102. Rauschecker, J. P., von Griinau, M. W., and Poulin, C. 1987a. Centrifugal organization of direction preferences in the cat’s lateral suprasylvian visual cortex and its relation to flow field processing. J . Neurosci. 7(4),943-958. Rauschecker, J. P., von Griinau, M. W., and Poulin, C. 1987b. Thalamocortical connections and their correlation with receptive field properties in the cat’s lateral suprasylvian visual cortex. Exp. Brain Res. 67, 100-112.
Neural Network for Processing Optic Flow
391
Regan, D., and Beverly, K. I. 1982. How do we avoid confounding the direction we are looking and the direction we are moving? Science 215, 194-196. Rieger, J. H., and Lawton, D. T. 1985. Processing differential image motion. J. Opt. SOC.Am. A 2, 354-360. Rieger, J. H., and Toet, L. 1985. Human visual navigation in the presence of 3-D rotations. Biol. Cybern. 52, 377-381. Stone, L. S., and Perrone, J. A. 1991. Human heading perception during combined translational and rotational self-motion. In SOC. Neurosci. Abstr. 17, 857. Tanaka, K., and Saito, H.-A. 1989a. Analysis of motion of the visual field by direction, expansion/contraction, and rotation cells clustered in the dorsal part of the medial superior temporal area of the macaque monkey. J. Neuropkysiol. 62(3), 626-641. Tanaka, K., and Saito, H.-A. 1989b. Underlying mechanisms of the response specificity of expansion/contraction and rotation cells in the dorsal part of the medial superior temporal area of the macaque monkey. J. Neuropkysiol. 62(3), 642-656. Toyama, K., Fujii, K., and Umetani, K. 1990. Functional differentiation between the anterior and posterior Clare-Bishop cortex of the cat. Exp. Brain Res. 81, 221-233. Tsai, R. Y., and Huang, T. S. 1984. Uniqueness and estimation of three-dimensional motion parameters of rigid objects with curved surfaces. IEEE Trans. Pattern Anal. Machine lntelligence 6, 13-27. Ungerleider, L. G., and Mishkin, M. 1982. Two cortical visual systems. In Analysis of Visual Behavior, D. J. Ingle, M. A. Goodale, and R. J. W. Mansfield, eds., pp. 549-586. MIT Press, Cambridge, MA. Wang, H. T., Mathur, 8. P., and Koch, C. 1989. Computing optical flow in the primate visual system. Neural Comp. 1(1), 92-103. Warren, W. H., Jr., and Hannon, D. J. 1988. Direction of self-motion is perceived from optical flow. Nature (London) 336, 162-163. Warren, W. H., Jr., and Hannon, D. J. 1990. Eye movements and optical flow. J. Opt. SOC.Am. A 7(1), 160-169. Warren, W. H., Jr., Morris, M. W., and Kalish, M. 1988. Perception of translational heading from optical flow. J. Exp. Psyckol.: Human Percept. Perform. 14(4), 646460. Waxman, A. M., and Ulllgan, S. 1985. Surface structure and three-dimensional motion from image flow: A kinematic analysis. lnt. J . Robotics Res. 4, 72-94. Yuille, A. L., and Grzywacz, N. M. 1988. A computational theory for the perception of coherent visual motion. Nature (London) 335, 71-74. Zeki, S., and Shipp, S. 1988. The functional logic of cortical connections. Nature (London) 335,311-317. Received 7 January 1992; accepted 2 October 1992.
This article has been cited by: 2. Frank Bremmer, Michael Kubischik, Martin Pekel, Klaus-Peter Hoffmann, Markus Lappe. 2010. Visual selectivity for heading in monkey area MST. Experimental Brain Research 200:1, 51-60. [CrossRef] 3. Douglas A. Hanes, Julia Keller, Gin McCollum. 2008. Motion parallax contribution to perception of self-motion and depth. Biological Cybernetics 98:4, 273-293. [CrossRef] 4. Jason T. Richards, Ajitkumar P. Mulavara, Jacob J. Bloomberg. 2004. Postural Stability During Treadmill Locomotion as a Function of the Visual Polarity and Rotation of a Three-Dimensional Virtual EnvironmentPostural Stability During Treadmill Locomotion as a Function of the Visual Polarity and Rotation of a Three-Dimensional Virtual Environment. Presence: Teleoperators and Virtual Environments 13:3, 371-384. [Abstract] [PDF] [PDF Plus] 5. Markus Lappe, Charles J. Duffy. 1999. Optic flow illusion and single neuron behaviour reconciled by a population model. European Journal of Neuroscience 11:7, 2323-2331. [CrossRef] 6. Seth Cameron , Stephen Grossberg , Frank H. Guenther . 1998. A Self-Organizing Neural Network Architecture for Navigation Using Optic FlowA Self-Organizing Neural Network Architecture for Navigation Using Optic Flow. Neural Computation 10:2, 313-352. [Abstract] [PDF] [PDF Plus] 7. Richard A. Andersen, Lawrence H. Snyder, David C. Bradley, Jing Xing. 1997. MULTIMODAL REPRESENTATION OF SPACE IN THE POSTERIOR PARIETAL CORTEX AND ITS USE IN PLANNING MOVEMENTS. Annual Review of Neuroscience 20:1, 303-330. [CrossRef] 8. A V. van den Berg, J A. Beintema. 1997. Motion templates with eye velocity gain fields for transformation of retinal to head centric flow. NeuroReport 8:4, 835-840. [CrossRef] 9. J-C Hsu, R-H Lin, E C Yeh. 1997. Vision-based motion measurement by directly extracting image features for vehicular steering control. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering 211:4, 277-289. [CrossRef] 10. Markus Lappe. 1996. Functional Consequences of an Integration of Motion and Stereopsis in Area MT of Monkey Extrastriate Visual CortexFunctional Consequences of an Integration of Motion and Stereopsis in Area MT of Monkey Extrastriate Visual Cortex. Neural Computation 8:7, 1449-1461. [Abstract] [PDF] [PDF Plus] 11. Ruye Wang . 1995. A Simple Competitive Account of Some Response Properties of Visual Neurons in Area MSTdA Simple Competitive Account of Some Response Properties of Visual Neurons in Area MSTd. Neural Computation 7:2, 290-306. [Abstract] [PDF] [PDF Plus]
12. Markus Lappe, Josef P. Rauschecker. 1995. Motion anisotropies and heading detection. Biological Cybernetics 72:3, 261-277. [CrossRef]
Communicated by Richard Durbin and Graeme Mitchison
Arbitrary Elastic Topologies and Ocular Dominance Peter Dayan Computational Neurobiology Laboratory, The Salk Institute,
PO.Box 85800, San Diego, CA 92186-5800 USA The elastic net, which has been used to produce accounts of the formation of topology-preserving maps and ocular dominance stripes (OD), embodies a nearest neighbor topology. A Hebbian account of OD is not so restricted-and indeed makes the prediction that the width of the stripes depends on the nature of the (more general) neighborhood relations. Elastic and Hebbian accounts have recently been unified, raising a question mark about their different determiners of stripe widths. This paper considers this issue, and demonstrates theoretically that it is possible to use more general topologies in the elastic net, including those effectively adopted in the Hebbian model. 1 Introduction
Durbin and Willshaw’s (1987) elastic net algorithm for solving the traveling salesperson problem (TSP) is based on a method for developing topology-preserving maps between the eye and brain (or lateral geniculate nucleus and cortex) due to von der Malsburg and Willshaw (1977) and Willshaw and von der Malsburg (1979). The elastic algorithm inspired a host of similar ones aimed at different optimization tasks, one of which is this topology problem, augmented by two associates-forming ocular dominance stripes and orientation selective cells (Goodhill and Willshaw GW 1990; Durbin and Mitchison 1990). SimiL. (1990, 1991) and Yuille (1990) looked at the relationship between elastic algorithms and Hebbian inspired ones (Hopfield and Tank 19851, showing that both mechanisms could be viewed as optimizing the same functions, albeit implementing differently the constraints (for the TSP, that each city should be visited exactly once). More recently, Yuille, Kolodny, and Lee (YKL 1991) repeated the feat and aligned elastic and Hebbian (Miller, Keller, and Stryker, MKS 1989) accounts of ocular dominance. The elastic net for the TSP consists of a set of points on a computational rubber band, pulled by forces toward the cities that have to be visited and by tension. The energy in a stretched rubber band is proportional to the square of its extension, which is incorrect for modeling the length of a tour (proportional just to the extension, in this model), Neural Computation 5, 392-401 (1993) @ 1993 Massachusetts Institute of Technology
Arbitrary Elastic Topologies
393
but Durbin (cited as a personal communication in Yuille 1990) suggests that changing the elastic net to use the absolute distance rather than its square is infelicitous. Hopfield and Tank’s (1985) model does in fact use the actual distances, and so, as they lucidly discuss, SimiC‘s and Yuille’s match between the elastic and Hebbian algorithms is not perfect. The nature of the topologies is even more mysterious in the match between Hebbian and elastic algorithms for ocular dominance. Topology enters MKS’ model through a cortical interaction function, which involves more than just the nearest neighbors. Conversely, these are the natural topology for the elastic version. This is one factor leading to an apparent difference between the predictions of MKS and GW. MKS suggested that the width of ocular dominance stripes is governed by the width of the cortical interaction function, whereas GW predicted that it is dependent on the relative correlations within and between the two eyes. This paper considers the issue by examining the two models of ocular dominance. The next section reviews YKLs analysis, and Section 3 looks at generalizing the nearest neighbor topology, testing the generalization in a onedimensional version of ocular dominance. 2 Yuille, Kolodny, and Lee’s Analysis
YKL un* the two models through the medium of a single cost function, which defines a generalized deformable model:’ E [VL,V R ,Y]
where V$ and V: are the variables matching the ith unit in the left and right eyes (more correctly lateral geniculate nucleus layers), respectively, to the ath unit in the cortex, xf and x; are the retinal “positions” of the ith unit in the left and right eyes, ya is the “position” of the ath unit in the cortex, and Y I {yo}. As GW and YKL say, these “positions” are defined somewhat abstractly; however, they are intended to capture something like the correlations between the firings of the retinal and cortical units. u is a constant that governs the relative weighting of the first term, which measures how close, correlationally, matching cells are, and the second term, which measures how close neighboring cortical cells are. This cost function owes its power of unification to having both matching V and continuous Y variables. The constraint on both retinal and cortical fields on a solution-that each cell should have a unique partner-is effectively duplicated in these ‘For convenience, this paper will look at the onedimensional versions of the various tasks. Extensions to the second dimension are straightforward, but messy, Also, YKL separate out the retinotopy dimension-whereas it is incorporated here into the continuous variables x and Y. MKS arbor functions are also neglected.
Peter Dayan
394
two sets of variables.* Minimizing E subject to these constraints leads to the optimal map. Hebbian and elastic methods are effectively different ways of minimizing this function, imposing different constraints in different manners on the way to deriving a solution. Both use Hopfield and Tank's key insight for the TSP that the constraints need not all hold throughout the optimization process, so long as they are guaranteed to be satisfied by the time the algorithm terminates. The reduction to an elastic net comes from eliminating the V Land V R variables using a Gibbs trick. The probability of a particular assignment of V and Y is declared to be proportional to e-flEIVL~VR~~, and these terms are summed over the set of V Rand V Lthat satisfy the partial constraint that each cell in the cortex maps to a unit in either the left or the right eye, but not both. The resulting elastic energy function is3
v
+5
c IYrl - Ya+*12
(2.2)
a
Note that the topology term survives this reduction intact, since it does not depend on the V . The alternative to eliminating the V variables is to eliminate the Y. YKL do this by regarding E[VL,VR,Y] as a quadratic form in Y, which has a minimum at Y,[VL,VR].Imposing the normalization constraint (see MKS) that each cortical cell receives a constant weight from the retina:
gives
where X L = {#} and X R = {xf}, and
21n terms of these variables: for each a, one of the collection over i of { V i ,V j } should be 1 and all the vt 0, and for each i the same should be true of the collection,overa. Also, for each a, ya should be the same as one # or .;",and for each i, there should be different 111 and a2 such that yal = # and y, = .;". 3Here and throughout, boundary conditions are avoided by assuming toroids and using modulo arithmetic.
Arbitrary Elastic Topologies
395
embodies the toroidal nearest neighbor topology. Therefore, at the minimum
1
+
YT = (2 q - 1 (ULXLT + URXR’ where the inverse exists for v > 0. Substituting back into equation 2.1, imposing the constraints in equation 2.3,and ignoring terms that do not depend on the U,gives
MKS’ Hebbian system regards the output oa of cortical cell a as coming partly from the input from the two eyes iL and iR through the connection matrix, [ULiL URiR],,and partly from the other cortical cells [Dola(MKS call C = (Z- D)-’the cortical interaction function):
+
o = VLiL+uRiR +DO
(W+ uRiR)
= (1 - D)-’
(2.5)
Hebbian learning changes U$ proportional to (oaik), where the angle brackets represent an averaging process. Defining Djk LL - ( i4j.i*L k), = (iFi;), and DF = (iii,”)
qt
YKL show that the U are moving down the gradient of
+
1
R RDRR
- D), vai’bj
ij
L RDLR + 2(Z-D)ab 1vaivbj ij }
(2.6) Compare equations 2.4 and 2.6. YKL argue that for the intent of comparing minima, one can identify K - lxf - x ~ I ’ with D f and similarly for and Df“, for some constant K. Therefore, if -23 = v 7 , these two expressions will have the same interesting minima-so, provided that the constraints are properly satisfied during learning, they should lead to the same ultimate solution: Note that this can make the effective correlations negative at some distance, which, as MKS discuss, allows correlation width to determine stripe width in their model. The cortical interaction function calculated from 7 (using v = 3/41 is shown in Figure 1. Although YKL show that this is enough to pro4YKL actually derive a different condition for matching-that (‘T = (Z- D)-I for some constant <. The truth of this would appear to depend on Caij ViV:lx; - xfl’, and the similar expressions for V R V Rand VRVL,being constant over the V that satisfy the partial constraints.
Peter Dayan
396
a -0.1
L
Distance
I b
Figure 1: Cortical interaction functions. (a) Elastic topologies. (b) MKS topologies.
duce interesting ocular dominance stripes, it is clearly not the same as
MKS cortical interaction function, which' is shown in the same figure. One reason why elastic and Hebbian models make different predictions about the factors determining stripe widths is also obvious-the cortical
Arbitrary Elastic Topologies
397
interaction function corresponding to the elastic topology is immutable. The next section considers alternatives. 3 Generalizing Elastic Topologies
The shape of the interaction function comes from the term Calya - ya+1l2 in equation 2.1. A more general quadratic form for this is
then this reduces to exactly the same expression For instance, if S = 7, as in equation 2.1. Note also that this formulation is sufficiently general as to model the case of two-dimensional retinas and cortex, although it does not extend to nonquadratic cases such as the length rather than the square of the length for the TSP. Such a change has little effect on the elastic energy function from equation 2.2, which becomes
However, differentiating E as a quadratic form to eliminate the Y leads to
+
+
(2 .S)Y,T = VLXLT PXRT
assuming that S is symmetric. If S also has a similarity property such that x b ( 2 + ~43);' does not depend on :u then substituting back in gives the energy function
As above, setting D = -US to unify the elastic and Hebbian energy functions allows Hebbian modeling of arbitrary elastic topologies and vice versa. One way of generating elastic topologies is to consider them in terms of an estimation problem. Say that X b E a b Y b is an estimate of ya. Then, the total square estimate error is 2
c Iya c 6 y b l c[(I- E)T(z- E)]abyTyb =
-
b
ab
5This holds if the topology is the same over the whole cortex.
Peter Dayan
398
Comparing this with equation 3.1 shows that S = (2-&)T(I-&). 7can be generated this way, by making &,,(,,+I) = 1 and the remaining components 0. Another example comes from estimating y,, as the average of both its neighbors, that is, setting 1 b=a+l,b=a-l 0 otherwise 6 b=a -4 b = a + l , b = a - l ; IS,,b = 4 1 b=a+2, b = a - 2 l l 0 otherwise 1
=
2
{
(3.2)
whose associated cortical interaction function more closely resembles the MKS Mexican hats (see Fig. 1). Note that although for any & there is a unique associated S and therefore C , the same is not true the other way around. Symmetric S will only have a square root if all its eigenvalues are positive, and it is easy to generate seemingly plausible C for which this is not the case. MKS generated their C as (3.3)
where K = 1/7.5and D = 7 was the width of their arbor function (the number of cortical cells to which a retinal cell would connect). Changing D changes the length scale of the cortical interaction, and so changes the optimal stripe width. Figure 1 shows graphs of the elastic net cortical interaction function, the Mexican hat one from equation 3.2, and two generated from M f f i n e with D = 7 and one with D = 14. One way to test the generalized topology terms is to use them in the cost function of equation 2.1 and to consider the optimal stripe width for the ocular dominance maps this defines? It is convenient to study the one-dimensional case, since the interesting optima are just “ Z folds, as in the left-hand side of Figure 2 (after GW). Maps inspired by the sideways ” U shape on the right-hand side of the figure will, in many cases, have lower costs than these-however, they are ruled out as the cortex does not traverse the retinas appropriately. Given particular spatial locations of the retinal cells, it is straightforward to calculate the cost per unit length of Z-folds of varying widths-the width that minimizes this is the one both Hebbian and elastic algorithms should find. GW show that the optimal width of a stripe for the basic elastic topology is 21/d, where I is the separation of the two retinas and d is the distance between two cells within a retina. simulations verified this, using the elastic net topology 7.Note that increasing I increases: . 1 - xfI2 in the third term of equation 2.4, leaving the other distance terms unaltered. 61f the appropriate constraints are satisfied, one of the equivalent equations such as 2.4 can also be used.
Arbitrary Elastic Topologies
Left retina
399
d
Left retina
Cortex
Figure 2: One-dimensional ocular dominance maps. (a) "Z-fold stripes. (b) Optimal map. For the MKS topology, the optimal width should increase with D, the length scale of the cortical interaction function. The analysis above suggests that it should also increase with I, given the common energy function. Both of these are demonstrated in Figure 3, which shows how the optimal Z-fold stripe width w varies with both I and D. With the exception of the patch for low I and high D, this is monotonic in both variables. Note that as D increases, the matrix C becomes increasingly singular, which forces implausible constraints on the cortical connectivity. Also small stripes are favored for large D and small I, since they benefit more from the negative contributions from the influences of widely separated cells than they lose through the cost of switching between the retinas. In fact the cost function becomes at least trimodal in the width of the stripes in this regime; one optimum is at the minimum stipe width, another is at the sideways " U of Figure 2, and the third is at the width that would preserve monotonicity in Figure 3. 4 Discussion
It is natural to wish to incorporate more extensive topologies into the elastic net than the nearest neighbor one with which it is presently endowed. One particular motivation for this comes from the apparent conflict between the predictions of stripe width from the elastic and Hebbian theories of the development of ocular dominance. However it is important in other cases such as graph matching in von der Malsburg's (1981) correlation theory of brain function. In this, fine scale temporal correlations in the firing of cells in a field are determined by the topology of the object being represented on that field, and inference consists of matching
400
Peter Dayan
0.5 -1- 10 7 -D32 2 -w16
Figure 3: Optimal Z-fold stripe width w versus length scale D of the cortical interaction function and distance I between the retinas. The cost function values E were calculated from equation 2.4, replacing (Z+u7)-' with C generated using equation 3.3.
this graph with an isomorphic one on another field. If the fine scale temporal correlations embody more than nearest neighbor correlations (and inference will be faster if they do), then describing this process in elastic terms will require a more general topology too. This paper has used the formalism of generalized deformable models to consider how general topologies fit into an elastic net framework. It demonstrates that this is effective by showing how the optimal stripe widths theoretically change with changing cortical length scales. However, it does remain to be seen which of the alternatives lead to stable elastic algorithms. Designer topologies are as simple to specify as designer error functions, and it will be interesting to see if there is an equivalent wealth of well-motivated examples.
Arbitrary Elastic Topologies
401
Acknowledgments
I am very grateful to Geoff Goodhill for introducing me to the problems of ocular dominance and for providing constant encouragement and assistance throughout this study, I also thank David MacKay, Ken Miller, Terry Sejnowski, Martin Simmen, David Willshaw, and Alan Yuille for their help. Support was from the SERC. References Durbin, R., and Mitchison, G. 1990. A dimension reduction framework for cortical maps. Nature (London) 343, 644-647. Durbin, R., Szeliski, R., and Yuille, A. L. 1989. An analysis of the elastic net approach to the traveling salesman problem. Neural Comp. 1,348-358. Durbin, R., and Willshaw, D. J. 1987. An analogue approach to the traveling salesman problem using an elastic net method. Nature (London) 326,689-691. Goodhill, G. J.,and Willshaw, D. J. 1990. Application of the elastic net algorithm to the formation of ocular dominance stripes. Network 1,41-61. Hopfield, J. J.,and Tank, D. W.1985. Neural computation of decisions in optimization problems. Biol. Cybern. 52, 141-152. Miller, K. D.,Keller, J. B., and Stryker, M. P. 1989. Ocular dominance column development: Analysis and simulation. Science 245,605-615. Simit, P. D. 1990. Statistical mechanics as the underlying theory of ”neural” and “elastic” optimizations. Network 1,89-104. Simit, P. D.1991. Constrained nets for graph matching. Neural Comp. 3, 268281. von der Malsburg, C. 1981. The Correlation Theory of Bruin Function. Internal report 81-2,Max-Planck Institute for Biophysical Chemistry, Gottingen, Germany. von der Malsburg, C., and Willshaw, D. J. 1977. How to label nerve cells so that they interconnect in an ordered fashion. Proc. Natl. Acad. Sci. U.S.A. 74, 5176-5178. Willshaw, D.J.,and von der Malsburg, C. 1979. A marker induction mechanism for the establishment of ordered neural mappings: Its application to the retinotectal problem. Phil. Transact. R. SOC.B 287,203-243. Yuille, A. L. 1990. Generalized deformable templates, statistical physics and matching problems. Neural Comp. 2, 1-24. Yuille, A. L., Kolodny, J. A., and Lee, C. W. 1991. Dimension reduction, generalized deformable models and the development of ocularity and orientation. Proceedings of the International Joint Conference on Neural Networks, Seattle, WA. Received 21 January 1992; accepted 24 September 1992.
This article has been cited by: 2. O. Scherf, K. Pawelzik, F. Wolf, T. Geisel. 1999. Theory of ocular dominance pattern formation. Physical Review E 59:6, 6977-6993. [CrossRef] 3. Richard S. Zemel, Geoffrey E. Hinton. 1995. Learning Population Codes by Minimizing Description LengthLearning Population Codes by Minimizing Description Length. Neural Computation 7:3, 549-564. [Abstract] [PDF] [PDF Plus] 4. Geoffrey J. Goodhill. 1993. Topography and ocular dominance: a model exploring positive correlations. Biological Cybernetics 69:2, 109-118. [CrossRef]
Communicated by Richard Lippmann
Neural Networks for Fingerprint Recognition Pierre Baldi Jet Propulsion Laboratory and Division of Biology, California Institute of Technology, Pasadena, C A 92209 U S A
Yves Chauvin Net-ID, Inc. and Department of Psychology, Stanford University, Stanford, C A 94305 USA
After collecting a data base of fingerprint images, we design a neural network algorithm for fingerprint recognition. When presented with a pair of fingerprint images, the algorithm outputs an estimate of the probability that the two images originate from the same finger. In one experiment, the neural network is trained using a few hundred pairs of images and its performance is subsequently tested using several thousand pairs of images originated from a subset of the data base corresponding to 20 individuals. The error rate currently achieved is less than 0.5%. Additional results, extensions, and possible applications are also briefly discussed. 1 Introduction
The fast, reliable, and computerized classification and matching of fingerprint images is a remarkable problem in pattern recognition that has not, to this date, received a complete solution. Automated fingerprint recognition systems could in principle have an extremely wide range of applications, well beyond the traditional domains of criminal justice and, for instance, render the use of locks and identification cards obsolete. Our purpose here is to give a brief account of our preliminary results on the application of neural network ideas to the problem of fingerprint matching. In particular, we shall describe the architecture, training, and testing of a neural network algorithm that, when presented with two fingerprint images, outputs a probability p that the two images originate from the same finger. There are several reasons to suspect that neural network approaches may be remarkably well suited for fingerprint problems. First, fingerprints form a very specific class of patterns with very peculiar flavor and statistical characteristics. Thus the corresponding pattern recognition problems seem well confined and constrained, perhaps even more so than in other pattern recognition problems, such as the recognition Neural Computation
5,402-418 (1993) @ 1993 Massachusetts Institute of Technology
Neural Networks for Fingerprint Recognition
403
of handwritten characters, where neural networks have already been applied with reasonable success (see, for instance, Le Cun et al. 1990). Second, neural networks could avoid some of the pitfalls inherent to other more conventional approaches. It has been known for over a century (see Moenssens 1971 for an interesting summary) that pairs of fingerprint images can be matched by human operators on the basis of minutia and/or ridge orientations. Minutia are particular types of discontinuities in the ridge patterns, such as bifurcations, islands, and endings. There is typically of the order of 50 to 150 minutia (Fig. 2a) on a complete fingerprint image. Ten matching minutia or so are usually estimated as sufficient to reliably establish identity. Indeed, it is this strategy based on minutia detection and matching that has been adopted in most of the previous attempts to find automated solutions. The minutia-based approach has two obvious weaknesses: it is sensitive to noise (especially with inked fingerprints, small perturbations can create artificial minutia or disguise existing ones) and computationally expensive since it is essentially a graph matching problem. Third, neural networks are robust, adaptive, and trainable from examples. This is particularly important since fingerprint images can include several different sources of deformation and noise ranging from the fingers and their positioning on the collection device (translation, roll, rotation, pressure, skin condition) to the collection device itself (ink/optical). Furthermore, it is important to observe that the requirements in terms of speed, computing power, probability of false acceptance and false rejection, memory and data base size can vary considerably depending on the application considered. To access a private residence or private car, one needs a small economic system with a small modifiable data base of a few people and a response time of at most a few seconds. On the other hand, forensic applications can require rapid searches through very large data bases of millions of records using large computers and a response time that can be longer. Neural networks can be tailored and trained differently to fit the particular requirement of specific applications. From a technical standpoint, there are two different problems in the area of fingerprint analysis: classification and matching. The classification of fingerprints into subclasses can be useful to speed up searches through large data bases. It is of interest to ask whether neural networks can be used to implement some of the conventional classification schemes, such as the partition of fingerprints patterns into whorls, arches, and loops (“pattern level classification”), or to create new classifications boundaries. Classification problems, however, will be discussed elsewhere. Here, we shall exclusively concentrate on the matching problem. Indeed, at the core of any automated fingerprint system, whether for identification or verification purposes and whether for large or small data base environments, there should be a structure that, when presented with two fingerprint images, decides whether or not they originate from
404
Pierre Baldi and Yves Chauvin
the same finger. Accordingly, our goal is going to be the design and testing of such a neural algorithm. Because neural networks are essentially adaptive and need to be trained from examples, we next describe our data base of training and testing examples and how it was constructed. We then consider the matching algorithm that consists of two stages: a preprocessing stage and a decision stage. The preprocessing stage basically aligns the two images and extracts, from each one of them, a central region. The two central regions are fed to the decision stage, which is the proper neural network part of the algorithm and subject to training from examples. Whereas the preprocessing stage is fairly standard, the decision stage is novel and based on a neural network that implements a probabilistic Bayesian approach to the estimate of the probability p of a match. In the main experiment, the network is trained by gradient descent using a training set of 300 pairs of images coming from 5 fingers, 5 images per finger. Its performance is then validated using an additional set of 4650 pairs coming from 15 additional fingers, 5 images per finger also. After training, the network achieves an overall error rate of 0.5%. Additional results and possible extensions are discussed at the end. 2 Data Base
Although there exist worldwide many fingerprint data bases, these are generally not available for public use. In addition, and this is a crucial issue for connectionist approaches, most data bases contain only one image or template per finger whereas training a neural network to recognize fingerprint images requires that several noisy versions of the same record be available for training. Therefore, to train and test a neural network, one must first construct a data base of digitized fingerprint images. Such images can be obtained in a variety of ways, for instance by digital scanner with inked fingerprints or by more sophisticated holographic techniques (Igaki et al. 1990). We decided to build our own collection device, using a simple principle. The device basically consists of a prism placed in front of a CCD (charge coupled device) camera connected to a frame grabber board installed on a personal computer (Fig. 1). When a finger is positioned on the diagonal face of the prism, incoming rays of light from one of the square sides of the prism are refracted differently depending on whether they encounter a point of contact of the finger with the prism (corresponding to a ridge) or a point of noncontact. This creates a pattern of bright and dark ridges in the refracted beam that can easily be focused, with a lens, on the CCD camera and then digitized and stored in the computer. Our resulting images are 512 x 464 pixels in size, with 8 bits gray scale per pixel. On the corresponding scale, the thickness of a ridge corresponds to 6 pixels or so. This is of course not a very economical format for the storage of fingerprint images that contain
Neural Networks for Fingerprint Recognition
405
Figure 1: Collection device: diffuse light entering the prism is not reflected where the ridges are in contact with the prism. The corresponding pattern of light and dark ridges is focused on a CCD camera, digitized on a personal computer, and sent to a workstation for further processing. a much smaller amount of useful information. Yet, this format is necessary at least in the developing phase, in particular in order to fine tune the preprocessing. In this way, we have assembled a data base of over 200 fingerprint images using various fingers from 30 different persons. To solve the matching problem, it is imperative that the data base contains several different images of the same finger taken at different times. Thus, for what follows, the most important part of the data base consists of a subset of 100 images. These are exclusively index finger images from 20 different persons, 5 different images being taken for each index finger at different times. At each collection time, we did not give any particular instruction to the person regarding the positioning of the finger on the prism other than to do so "in a natural way." In general, we made a deliberate attempt not to try to reduce the noise and variability that would be present in a realistic environment. For instance, we did not clean the surface of the prism after each collection. Indeed, we do observe significant differences among images originating from the same finger. This variability results from multiple sources, mostly at the level of the finger (positioning, pressure, skin condition) and the collection device (brightness, focus, condition of prism surface). We have conducted several learning experiments using this data base, training the networks with image pairs originated from up to 7 persons
Pierre Baldi and Yves Chauvin
406
and testing the algorithm on the remaining pairs. Here, we shall mostly report the typical results of one of our largest ex eriments where, out of the = 4950 image pairs in this data base, = 300 image pairs originated from 5 different persons are used to train the network by gradient descent. The remaining 4650 pairs of images are used to test the generalization abilities of the algorithm. Given two fingerprint images A and B, the proposition that they match (or do not match) will be denoted by M(A,B) [or M(A,B ) ] . The purpose then is to design a neural network algorithm that when presented with a pair (A,B) of fingerprint images outputs a number p = p(M) = p[M(A,B ) ] between 0 and 1 corresponding to a degree of confidence (Cox 1946) or probability that the two fingerprints match. Here, as in the rest of the paper, we shall tend to omit in our notation the explicit dependence on the pair ( A ,B) except the first time a new symbol is introduced.
rr)
E)
3 Preprocessing Stage
Any algorithm for fingerprint recognition may start with a few stages of standard preprocessing where the raw images may be rotated, translated, scaled, contrast enhanced, segmented, compressed, or convolved with some suitable filter. In our application, the purpose of the preprocessing stage is to extract from each one of the two input images a central patch called the central region and to align the two central regions. Only the two aligned central regions are in turn fed to the decision stage. The preprocessing itself consists of several steps, first to filter out high-frequency noise, then to compensate for translation effects present in the images and to segment them and finally to align and compress the central regions. For ease of description, one of the input images will be called the reference image and the other one the test image, although there is no intinsic difference between the two. 3.1 Low Pass Filtering. To get rid of the numerous high-frequency spikes that seem to be present in the original images, we replace every pixel that significantly deviates from the values of its four neighbors by the corresponding average.
3.2 Segmentation. For each image, we first draw a tight rectangular box around each fingerprint using an edge detection algorithm and determine the geometric center of the box. The central region of the reference image is then defined to be the 65 x 65 central square patch that occupies the region immediately below the previously described center. For the test image, instead we select a similar but larger patch of size 105 x 105 (extending the previous patch by 20 pixels in each direction). This larger patch is termed the window.
Neural Networks for Fingerprint Recognition
407
3.3 Alignment. We slide, pixel by pixel, the central region of the reference image across the window of the test image (by 20 pixels up, down left and right) and compute at each step the corresponding correlation, until we find the position where the correlation is maximal. This, aside from the training period, is the most computationally expensive part of the entire algorithm. The central region of the test image is then determined by selecting the central 65 x 65 patch corresponding to the position of maximal correlation (Fig. 2b). 3.4 Compression and Normalization. Finally, each one of the two 65x 65 central regions is reduced to a 32 x 32 array by discrete convolution with a truncated gaussian of size 5 x 5. This 32 x 32 compressed central region contains a low-resolution image, which corresponds roughly to 10 ridges in the original image. The resulting pixel values are conveniently normalized between 0 and 1. In our implementation, all the parameters and in particular the size of the various rectangular boxes are adjustable. The values given here are the ones used in the following simulations and empirically seem to yield good results. To avoid border effects, a 2-pixel-wide frame is usually added around the various rectangular boxes, which explains some of the odd sizes. It is also natural to wonder at this stage whether a decision regarding the matching of the two inputs could not already be taken based solely on the value of the maximal correlation found during the alignment step (3.3) by thresholding it. It is a key empirical observation that this maximal correlation, due in part to noise effects, is not sufficient. In particular, the correlation of both matching and nonmatching fingerprint images is often very high (above 0.9) and we commonly observe cases where the correlation of nonmatching pairs is higher than the correlation of matching pairs. It is therefore essential to have a nonlinear decision stage following the preprocessing. Finally, it should be noticed that during training as well as testing, the preprocessing needs to be applied only once to each pair of images. In particular, only the central regions need to be cycled through the neural network during the training phase. Although the preprocessing is not subject to training, it can be implemented, for most of its part, in a parallel fashion compatible with a global neural architecture for the entire algorithm. 4 Neural Network Decision Stage
The decision stage is the proper neural network part of the algorithm. As in other related applications (see, for instance, Le Cun et al. 19901, the network has a pyramidal architecture, with the two aligned and compressed central regions as inputs and with a single output p. The bottom level of the pyramid corresponds to a convolution of the central
408
Pierre Baldi and Yves Chauvin
Figure 2: (a) A typical fingerprint image: the surrounding box is determined using an edge detection algorithm. Notice the numerous minutia and the noise present in the image, for instance in the form of ridge traces left on the prism by previous image collections.
Figure 2 (b) Preprocessing of two images of the same finger: the left image is the reference image, the right image is the test image (same as a). The 65 x 65 central region of the reference image is shown in black right under the geometrical center (white dot). The 105 x 105 window of the test image is shown in black. The white square is the central region of the test image and corresponds to the 65 x 65 patch, within the window, which has a maximal correlation with the central region of the reference image.
Pierre Baldi and Yves Chauvin
410
regions with several filters or feature detectors. The subsequent layers are novel. The final decision they implement results from a probabilistic Bayesian model for the estimation of p, based on the output of the convolution filters. Both the filtering and decision part of the network are adaptable and trained simultaneously. 4.1 Convolution. The two central regions are first convolved with a set of adjustable filters. In this implementation only two different filter types are used, but the extension to a larger number of them is straightforward. Here, each filter has a 7 x 7 receptive field and the receptive fields of two neighboring filters of the same type have an overlap of 2 pixels to approximate a continuous convolution operation. The output of all the filters of a given type form a 6 x 6 array. Thus each 32 x 32 core is transformed into several 6 x 6 arrays, one for each filter type. The output of filter type j at position (x, y) in one of these arrays is given (for instance for A) by
(4.1)
where lr,s(A)is the pixel intensity in the compressed central region of image A at the (Y,s) location, f is one of the usual sigmoids [f(x) = (1 e-')-*], w ~ , ~is ,the ~ ,weight ~ of the connection from the (Y, s) location in the compressed central region to the (x, y) location in the array of filter outputs, and ti is a bias. The sum in 4.1 is taken over the 7 x 7 patch corresponding to the receptive field of the filter at the (x,y) location. The threshold and the 7 x 7 pattern of weights are characteristic of the filter type (so-called "weight sharing"), so that they can also be viewed as the parameters of a translation invariant convolution kernel. They are shared within an image but also across the images A and B. Thus dx,y,r,s = zd(x - r , y - s) and, in this implementation, each filter type is characterized by 7 x 7 + 1 = 50 learnable parameters. In what follows, we simplrfy the notation for the location of the outputs of the filters by letting (x,y) = i. For each filter type j, we can now form an array Azj(A, B) consisting of all the squared differences
+
A{(A,B) = [&A) - Zj(B)]'
(4.2)
and let Az = Az(A, B) denote the array of all A&A, B) for all positions i and filter types j (Fig. 3). 4.2 Decision. The purpose of the decision part of the network is to estimate the probability p = p[M(A, B)/Az(A, B)] = p(M/Az) of a match between A and B, given the evidence Az provided by the convolution filters. The decision part can be viewed as a binary Bayesian classifier. There are four key ingredients to the decision network we are proposing.
Neural Networks for Fingerprint Recognition
411
Figure 3 Network architecture: at the bottom, reference and test images A and B are presented to the network as two 32 x 32 arrays. The network extracts features from the images by convolution with several different 7 x 7 filter types. 1. Because the output of the network is to be interpreted as a probability, the usual least mean square error used in most backpropagation networks does not seem to be an appropriate measure of network performance. For probability distributions, the cross entropy between the estimated probability output p and the true probability P, summed over
Pierre Baldi and Yves Chauvin
412
all patterns, is a well-known information theoretic measure of discrepancy (see, for instance, Blahut 1987)
where, for each image pair, Q = 1 - P and q = 1- p. H is also known as the discrimination function and can be viewed as the expected value of the log-likelihood ratio of the two distributions. The discrimination is nonnegative, convex in each of its arguments, and equal to zero if and only if its arguments are equal. 2. Using Bayes inversion formula and omitting, for simplicity, the dependence on the pair (A, B) (4.4)
The effect of the priors p(M) and p(M) should be irrelevant in the case of a large set of examples. Our data base is large enough for the decision to be driven only by the data, as confirmed by the simulations (see also F i g 4 ) . In simulations, the values chosen are typically p ( M ) = 0.1 and p(M) = 0.9 [the observed value of p(M) is roughly 16% in the training set and 4% in the entire data base]. 3. We make the simplifying independence assumption that p(Az/M) = n p ( A d / M )
(4.5)
i,j
P ( A m
=
IIP(A4/m
(4.6)
i,j
Strictly speaking, this is not true, especially for neighboring locations. However, in the center of a fingerprint where there is more variability than in the periphery, it is a reasonable approximation, which leads to simple network architectures and, with hindsight, yields excellent results. 4. To completely define our network, we still need to choose a model for the conditional distributions p(Ad/M) and p(Ad/M). In the case of a match, the probability p(Ad/M) should be a decreasing function of A4. It is therefore natural to propose an exponential model of the form
where 0 < Sij < 1 and, for proper normalization, the constant Cq must take the value C i j = logsv/sij- 1. In what follows, however, we use a less general but slightly simpler binomial model of the form (4.7)
Neural Networks for Fingerprint Recognition
413
~
Figure 4: Neural decision network. This is just a neural network implementation of equations 4.4-4.8.Except for the output unit, each unit computes its output by applying its transfer function to the weighted sum of its inputs. In this network, different units have different transfer functions including: id(x) = x, a(x) = (1 + e-X)-l, logx, and exp(x) = 8.The output unit is a normalization unit that calculates p(M/Az) in the form of a quotient x/x y. The coefficients p, and q, of 4.7 and 4.8 are implemented in the form p, = a(nj)and 1-qj = g(6j). T, and 6 , are the only adjustable weights of this part of the network and they are shared with the connections originating from the convolution filter outputs. All other connection weights are fixed to 1 except for the connections running from logp, to the exponential unit, which have a weight equal to 36, the size of the receptive field of the convolution filters. The exponential F i t on the left, for instance, computes p(Az/M)p(M)= p ( M )f l i i p , [ ( l - pj)/pjIA’: using the fact that (1 - p j ) / p , = -3.Notice that the priors play the role of a bias for the exponential units, which, after training, ends up having little influence on the output.
+
414
Pierre Baldi and Yves Chauvin
with 0.5 5 pj, 9, 5 1. This is again only an approximation used for its simplicity and for the fact that the feature differences Ad are usually close to 0 or 1. In this implementation and for economy of parameters, the adjustable parameters pj and 9j depend only on the filter type, another example of weight sharing. In a more general setting, they could also depend on location. In summary, the adjustable parameters of the neural network are zdx,y,r,s, tJ,pi, and q j . In this implementation, their total number is (7 x 7 + 1) x 2 + 2 x 2 = 104. At first sight, this may seem too large in light of the fact that the network is trained using only 300 pairs of images. In reality, each one of the 50 shared parameters corresponding to the weights and bias of each one of the convolution filters is trained on a much larger set of examples since, for each pair of images, the same filter is exposed to 72 different subregions. The parameters are initialized randomly (for instance the dx,y,r,s are all drawn independently from a gaussian distribution with 0 mean and standard deviation 0.5).They are then iteratively adjusted, after each example pair presentation, by gradient descent on the cross entropy error measure H. The specific formula for adjusting each one of them can readily be derived from 4.3-4.8and will not be given here for brevity. It should also be noticed that the adaptable pattern classifier defined by 4.3-4.8is not a neural network in the most restrictive sense of a layered system of sigmoid units. It is rather a nonlinear model with adjustable parameters which can be drawn (Fig. 41,and in several different ways, in a neural network fashion. The number of units and their types, however, depends on how one decides to decompose the algebraic steps involved in the computation of the final output p .
5 Results and Conclusions
We have trained the network described in the previous sections using 300 pairs of images from our data base and only two different filter types (Fig. 5). The network performance is then tested on 4650 new pairs. The network usually learns the training data base perfectly. This is not the case, however, when only one filter type is used. The separation obtained in the output unit between matching and nonmatching pairs over the entire population is good since 99% of the matching (or nonmatching) pairs yield an output above 0.8 (or below 0.2). The error rate on the generalization set is typically 0.5% with roughly half the errors due to false rejections and half to false acceptances. In many applications, these two types of error do not play symmetric roles. It is often the case, for instance, that a false acceptance is more catastrophic than a false rejection. If, by changing our decision threshold on p , we enforce a 0% rate of false acceptances, the rate of false rejections increases to 4%. This error rate
Neural Networks for Fingerprint Recognition
415
Figure 5: Unit activation throughout the network when two matching fingerprint images are presented as inputs. Flow of activation is left to right. Images A and B are presented to the network as 32 x 32 arrays. The network has two filters 1 and 2, each represented by the corresponding 7 x 7 pattern of weights. Each filter is convolved with each array and generates the set of feature arrays lA, 1B, 2A, and 2B. The next layer computes the squared feature differences for each filter. Finally, the similarity S = p ( M / A z )is computed. In this example the similarity is close to 1 (representedby a black vertical bar), close to the target value T = 1. Notice the essentially binary values assumed by the features and the compact representation of the input images with 72 bits each. needs of course to be reduced, but even so it could be almost acceptable for certain applications. As in other related applications, the interpretation of the filter types discovered by the network during learning is not always straightforward (Figs. 5 and 6). We have conducted various experiments with up to four filter types but on smaller data bases. Usually, at least one of the filter types always appears to be an edge or a ridge orientation detector. Some of the other filter types found in the course of various experiments may be interpretable in terms of minutia detectors, although this is probably more debatable. On the completion of the training phase, the outputs of the filters in the decision stage are close to being binary 0 or 1. Since the final decision of the network is entirely based on these outputs, these provide a very compressed &presentation of all the relevant matching information originally contained in the 512 x 464 x 8 bits images. Thus, in this im-
416
Pierre Baldi and Yves Chauvin
Figure 6 Examples of filters learned in different experiments: one filter is represented by the corresponding 7 x 7 patterns of weights. The size of each square represents the value of the corresponding weight. Black and white squares correspond to positive and negative weights, respectively. Whereas the first filter seems to be an edge or ridge orientation detector, the other two are more difficult to interpret. It may be tempting to describe them in terms of minutia detectors, such as an ending and a bifurcation detector, but this may not necessarily be the case. plementation, each image is roughly reduced to 36 x 2 = 72 bits, which is within a factor of two from a rough estimate of the theoretical lower bound (the number of human beings, past and present, is approximately 233 M 8.5 x 109). In applications, the algorithm would not be used in the same way as it has been during the training phase. In particular, only the central regions of the reference images need to be stored in the data base. Since the forward propagation through the decision stage of the algorithm is
Neural Networks for Fingerprint Recognition
417
very fast, one can in fact envision a variation on the algorithm where the alignment step in the preprocessing is modified and where the reference image is stored in the data base only in the most compressed form given by the corresponding outputs of the filters in the decision stage. In this variation, all the possible 65 x 65 square patches contained in the 105x 105 window of the test image would be sent, after the usual compression and normalization preprocessing steps, through the convolution filter arrays and then matched through the neural network with the corresponding outputs for the reference image. The final decision would then be based on an examination of the resulting surface of p values. Whether this algorithm leads also to better decisions needs further investigation. To reduce the error rate, several things can be tried. One possibility is to use more general exponential models in 4.7 and 4.8 rather than binomial distributions. Alternatively, the number of filter types or the number of free parameters could be increased (for instance by letting pi and qj depend also on location) as well as the size of the training and validation sets. Another possibility is to use in the comparison larger windows or, for instance, two small windows rather than one, the second window being automatically aligned by the alignment of the first one. A significant fraction of the residual false rejections we have seems to be due to rotation effects, that is, to the fact that fingers are sometimes positioned at different angles on the collection device. The network we have described seems to be able to deal with rotations up to a lo" angle. Larger rotations could easily be dealt with in the preprocessing stage. It is also possible to incorporate a guiding device into the collection system so as to entirely avoid rotation problems. In this study, we have attempted to find a general purpose neural network matcher that could be able, in principle, to solve the matching problem over the entire population. In this regard, it is remarkable that the network, having been trained with image pairs associated with only five different persons, generalizes well to a larger data base associated with 20 persons. Obviously, a general purpose network needs also to be tested on a larger sample of the population. Unlike the classification problem, however, the matching problem should be much less sensitive to the size of the training and testing sets. In classifymg whorls for instance, it is essential to expose the network to a large sample representative of whorl patterns across the entire population with all their subtle statistical variations. In our matcher, we are basically substracting one image from the other and therefore only the variability of the difference really matters. Specific applications, especially those involving a small data base, have particular characteristics that may be advantageously exploited both in the architecture and the training of networks and raise also some particular issues. For a car lock application, for instance, positive matches occur only with a pair of fingerprints associated with a small data base of only a few persons. Positive matches corresponding to fingerprints
418
Pierre Baldi and Yves Chauvin
associated with persons outside the data base are irrelevant. They may not be needed for training. Conceivably, in small data base applications, a different network could be trained to recognize each record in the data base separately against possible imposters. Finally, the approach we have described and especially the Bayesian decision stage with its probabilistic interpretation is not particular to the problem of fingerprint recognition. It is rather a general framework that could be applied to other pattern matching problems where identity or homology needs to be established. The corresponding neural networks can easily be embodied in hardware, especially once the learning has been done off-line. As already pointed out, most of the steps in the preprocessing and the decision stages are in fact convolutions and are amenable to parallel implementation. On a workstation, it currently takes on the order of 10 sec to completely process a pair of images. This time could be reduced by a few orders of magnitude with specially dedicated hardware. References Blahut, R. E. 1987. Principles and Practice of Information Theory. Addison-Wesley, Reading, MA. Cox, R. T. 1946. Probability, frequency and reasonable expectation. Am. J. Phys. 14(1), 1-13. Igaki, S., Eguchi, S., and Shinzaki, T. 1990. Holographic fingerprint sensor. Fujitsu Tech.J. 25(4), 287-296. L.e Cun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D. 1990. Handwritten digit recognition with a backpropagation network. In Neural Information Processing Systems, Vol. 2, pp. 396-404. Morgan Kaufmann, San Mateo, CA. Moenssens, A. A. 1971. Fingerprint Techniques. Chilton Book Company, Radnor, PA. Received 28 April 1992; accepted 6 October 1992.
This article has been cited by: 2. U. Halici, G. Ongun. 1996. Fingerprint classification through self-organizing feature maps modified to treat uncertainties. Proceedings of the IEEE 84:10, 1497-1512. [CrossRef]
Communicated by Yann Le Cun
Centered-Object Integrated Segmentation and Recognition of Overlapping Handprinted Characters Gale L. Martin MCC,Austin,TX 78759 USA Visual object recognition is often conceived of as a final step in a visual processing system, First, physical information in the raw image is used to isolate and enhance to-be-recognized clumps and then each of the resulting preprocessed representations is fed into the recognizer. This general conception fails when there are no reliable physical cues for isolating the objects, such as when objects overlap. This paper describes an approach, called centered object integrated segmentation and recognition (COISR), for integrating object segmentation and recognition within a single neural network. The application is handprinted character recognition. The approach uses a backpropagation network that scans a field of characters and is trained to recognize whether it is centered over a single character or between characters. When it is centered over a character, the net classifies the character. T h e approach is tested on a dataset of handprinted digits and high accuracy rates are reported. 1 Introduction Most character recognition systems fail when characters touch each other or when an individual character is broken up by intervening white space. The systems use intervening white space to segment a character string into individual characters, so that classificationcan be done one character at a time. As a preprocessing step prior to recognition, segmentation simplifies the recognition task. It restricts the number of input features to those associated with a single character and makes it possible to filter out irrelevant variations by normalizing for factors such as size and skew, on a character by character basis. The problem with this separation of segmentation and recognition is that it fails in the instances noted above, where human vision does not. For people, segmentation and recognition seem to be integrated, interdependent processes. This paper describes test results of an approach, called centered object integrated segmentation and recognition (COISR), for integrating character segmentation and recognition within one neural network (Martin 1990). The approach builds on previous work in presegmented character recognition (Le Cun et al. 1990; Martin and Pittman 1990), and on the sliding Neural Computation 5,419-429 (1993) @ 1993 Massachusetts Institute of Technology
420
Gale L. Martin
Figure 1: The COISR approach uses a multilayered net trained through backpropagation to classify what is centered in its input window. The input window scans over a field of characters that has been size normalized with respect to the vertical height of the field. The output layer consists of one node for each of the characters (0-9) and one node for a no-centered-character state. The latter state occurs whenever the scanning window is centered between characters or between a character and the edge of the field. To identify the characters in the field the output node activations are evaluated over time. When the NOTCENTERED node value is less than a threshold, the values for each character node are summed over time until the NOT-CENTERED node value exceeds the threshold. The maximum of these sums determines the classificationjudgment for that character. window conception used in neural network speech applications, such as NETtalk (Sejnowski and Rosenberg 1986) and Time Delay Neural Networks (Waibel el al. 1988). It is assumed that segmenting a character, in the sense of separating it from its background, is not a necessary precursor to recognizing the character, or to learning to recognize the character. The system is trained to recognize what is centered in its input window, and the window size is chosen to be large enough to maintain the natural contexts of the character-the surrounding characters, which may touch or overlap with the centered character. As shown in Figure 1, the net is trained on an input window, and a target vector representing what is in the center of the window. The window scans along the field, so that sometimes the middle of the window is centered on a character, and sometimes it is centered on a point
COISR of Overlapping Handprinted Characters
421
between two characters. The target output vector consists of one node per category and one node corresponding to a NOT-CENTERED condition. This latter node has a high target activation value when the input window is not centered over any character. There is no need to explicitly segment characters or to train the net on the exact physical extent of the characters, because recognition is defined as identifying what is in the center of the scanning window. That is, there is no need to pretrain the network on individual presegmented characters removed from their natural context of surrounding letters. The net learns to extract regularities in the shapes of individual characters even when those regularities occur in the context of overlapping and broken characters at both training and testing stages. The final stage of processing includes parsing the temporal stream produced by the computed output vectors of the net as it scans the field to identify the highest peaks of activation, which correspond to the characters in the field. The COISR approach was tested using the NIST database of handprinted digit fields. 2 Test of the Approach
2.1 Image Database. The NIST (National Institute of Standards and Technology)database contains 273,000 samples of handwritten numerals collected from Bureau of Census field staff. Each of 2100 census workers filled in a form with 33 fields, 28 fields of which contain only handwritten digits. The present study used fields 6-30 of the form, which correspond to five different fields of length 2, 3, 4, 5, or 6 digits each. The training data included roughly 80,000 digits (800 forms, 20,000 fields), and came from forms labeled fO000-fO499 and f1500-fl799 in the dataset. The test data consisted of roughly 20,000 digits (200 forms, 5000 fields) and came from forms labeled f1800-fl899 and f2000-f2099 in the dataset. The samples were scanned at a 300 pixel/inch resolution. 2.2 Image Preprocessing. Each original field image depicts a region that contains a small amount of white space surrounding a box, in which the digit field was written. The relative heights of the digits with respect to the box differ across fields. To minimize input size, each field image is preprocessed to eliminate the white space surrounding the box, the box itself, and the white space surrounding the digit field. Each field is then size normalized with respect to the vertical height of the digit field. Experimentation with different heights revealed that 20 pixels is sufficient to enable resolution of the digits by a human observer. Since the input is size normalized to the vertical height of the field of characters, the actual number of characters in the constant-width input window varies depending on the relative height-to-width ratio for each character. The visual context provided by the surrounding characters may help in disambiguating confusions in cases where there is sufficient redundancy
422
Gale L. Martin
in the sequence of characters (Martin 1990). An input pattern generator is then passed over the field, creating input windows 36 pixels wide and 20 pixels high, at 3-pixel increments across the field. Thus, the number of input patterns created for a field is greater than the number of digits in the field. 2.3 Position Labeling. A key design principle of the present approach is that highly accurate integrated segmentation and recognition requires training on both the shapes of characters and their positions within the input window. The field images used for training were labeled with the horizontal center positions of each character in the field. The human labeler simply pointed at the horizontal center of each digit in sequence with a mouse cursor and clicked on a mouse button. The horizontal position of each character was then paired with its category label (0-9) in a datafile. This labeling process is quite efficient and considerably less time consuming than the standard segmentation method of drawing a box around each character. The process is not unlike a human reading teacher using a pointer to indicate the position of each character as he or she reads aloud the sequence of characters making up the word or sentence. During testing this position information is not used. 2.4 Target Outputs. The position information about the locations of character centers is used to generate target output values during the generation of input patterns for training. When the center position of a window is within plus or minus 2 pixels of the center of a character, the target value of that character's output node is set at the maximum (0.8 during initial training and 1.0 during subsequent training), with the target value of the NOT-CENTERED node set at the minimum (0.2 during initial training and 0.0 during subsequent training). The activation values of all other characters' output nodes are set at the minimum. When the center position of a window is within plus or minus 2 pixels of the halfway point between two character centers, the reverse situation holds. The target values of all character output nodes are set to the minimum and the target value of the NOT-CENTERED node is set to a maximum. Between these two extremes, the target values vary linearly with distance, creating a trapezoidal function. 2.5 Network Architecture. The neural network is a 2-hidden-layer backpropagation network, with local, shared connections in the first hidden layer, and local connections in the second hidden layer (see Fig. 2). This architecture was chosen to minimize memory requirements and the time required to process each pattern, while retaining sufficient capacity for the net to sufficiently learn the training patterns. Several attempts to train nets with shared weights in both layers, and thereby significantly reduce memory requirements, yielded poor training performance. Pre-
COISR of Overlapping Handprinted Characters
-To
423
2nd Hidden Layer
Local connections (6JdXl&OffXt 2)
1st Hidden Layer
Local, shared connection
Figure 2: The architecture for the centered-object integrated segmentation and recognition net has two hidden layers. sumably, this is because position information is necessary to classify what is centered in the input window. The input window to the network is 36 pixels wide by 20 pixels high, with pixels taking on grayscale values ranging from 0 to 1. The first hidden layer consists of 2016 nodes, or more specifically 18 independent groups of 112 (16 x 7) nodes, with each group having local, shared connections to the input layer. It can be visualized as a 16 x 7 x 18 cube of nodes. The local, shared connections within a group ensure that the same feature map develops for all nodes in that group. Node bias values within a group are shared as well. Each node receives input from a local 6 x 8-pixel region. These local, overlap
424
Gale L. Martin
ping regions are offset by 2 pixels, such that the regions covered by each group of nodes span the input layer. The second hidden layer consists of 180 nodes, having local, but not shared connections. This layer can be visualized as a 6 x 3 x 10 cube. Each node in this layer receives input from a local 6 x 3 x 18-node region on the first hidden layer. These local, overlapping regions are offset by 2 pixels. The output layer consists of 11 nodes, with each of these nodes connected to all of the nodes in the second hidden layer. The net has a total of 2927 nodes (includes input and output nodes) and 157,068 connections. In a feedforward (nonlearning) mode on a DEC 5000 workstation, in which the net is scanning a field of digits, the system processes about two digits per second. This figure includes the image preprocessing, as well as the number of necessary feedforward passes to recognize 2 digits. The actual number of forward passes required to recognize a given field of characters depends on the width of the characters, the distance between characters, and the scan increment. As an example, at the present scan increment of 3 pixels, a typical field of 6 digits required 33 forward passes of the net, or between 5 and 6 forward passes per recognized character. 2.6 Tkaining Parameters. During initial training, the target output values were set at 0.8 (on) and 0.2 (off), with these changing to 1.0 and 0, respectively, later on. The learning rate was set at 0.05 during initial training, with this value changing to 0.01 when a plateau in training performance was reached. The momentum term was set at 0.9 throughout training. Training and testing alternated, with the highest testing performance reported. The training was done in a randomly permuted order, such that the order of training images is quite different than the sequential scans used during testing.
2.7 Output Parsing. As the net scans horizontally, the activation values of the 11output nodes create a trace as shown in Figure 1. To convert this to an ASCII string corresponding to the digits in the field, the trace is parsed as follows. The state of the NOT-CENTERED node is monitored continuously. When its activation value falls below a threshold (0.41, a summing process begins for each of the other nodes, which ends when the activation value of the NOT-CENTERED node exceeds the threshold. The activation values of each of the other nodes are multiplied by 1 minus the activation value of the NOT-CENTERED node, and added to a running total that is accumulated for each node. When the activation value of the NOT-CENTERED node then exceeds the threshold, this is interpreted as the input window having moved off of a character, and the system classifies the character, on the basis of which output node has the highest running total, or peak of activation, at this place in the field. Across the characters in the field thus classified, the system also orders the corresponding peaks of activation. In applications involving
COISR of Overlapping Handprinted Characters
'0
10
20
x)
4
50
425
m
70
m
w i m
90 REJECTIONS
Figure 3: Test error and reject rates for fields of lengths 2 through 6 digits. The test set consists of about 20,000 characters and 5000 fields. The digits within the lines correspond to the number of characters in the field. digit fields, it is often the case that there is a priori knowledge of how many digits are in the field. When the parsing process indicates that too many digits have been identified (i.e., an insertion error has occurred), the system takes the n highest peaks, where n is the number of expected digits in the field. When the parsing process indicates that too few characters have been identified, the field is automatically rejected. The results reported below were obtained using this method of reducing the number of insertion and deletion errors. 3 Results
As shown in Figure 3, the COISR technique achieves low field-based error rates. The error rates are field based in the sense that if the network
426
Gale L. Martin
Figure 4: Examples from the test set of handwritten digit fields containing touching, overlapping, and broken characters that the COISR net correctly recognizes. misclassifies one character in the field, the entire field is considered as misclassified. Rejections are based on placing a threshold for the acceptable distance between the highest and the next highest running activation total. In this way, by varying the threshold, the error rate can be traded off against the percentage of rejections. Since the reported data apply to fields, the threshold applies to the smallest distance value found across all of the characters in the field. Figure 4 provides examples, from the test set, of fields that the COISR network correctly classifies. The examples illustrate several common problems that plague conventional character segmentation and recognition systems and yet are dealt with successfully by the present system. These include fields containing multiple, contiguous touching characters (e.g., 95304,03283,8201), pairs of characters that touch at multiple points (e.g., 05, 556, 399, 1509, 091, image noise that places lines through multiple characters in a field (e.g., 500), and broken characters (e.g., 6354, 157, 641, 7963). Broken characters can be troublesome to conventional character segmentation and recognition systems for at least two reasons. Characters that are randomly broken (e.g., 79631,where this is primarily caused by image degradation or poor capture, can be difficult to recognize by systems that use a separate feature extraction stage prior to classification. Some of the features searched for are defined by small dot or stroke-like segments surrounded by white space, and hence can be mistakenly classified with broken characters. The other problem is caused by the previously mentioned, separate segmentation stage. One
COISR of Overlapping Handprinted Characters
427
type of preprocessor looks for columns of white space, or low points in the density histogram, to use as the basis for separating characters. Using this strategy, it seems quite difficult to come up with a reliable accurate segmenter that works for both broken characters (e.g., 6354,157,641) and nonbroken close or touching characters (e.g., 92,43862,389). The COISR technique works around these problems because the net is trained on image input and does not require the separate, preceding segmentation stage. 4 Discussion
The COISR technique makes it possible to do something that conventional character recognition systems cannot do: robustly recognize character fields containing touching, overlapping, and broken characters. Conventional character recognition systems require separate segmentation that precedes classification in both training and testing. Since such segmentation is based only on physical indicators (e.g., density histograms), and usually is independent of the recognition stage, it fails in cases where the appropriate segmentation depends on the classification judgment. Conventional systems can be altered to achieve integrated segmentation and recognition in limited cases that involve handcrafting and a significant amount of iterative processing (Fenrich 1991; Shridhar and Badreldin 1986,1987). One approach is to use multiple schemes for segmentation during training and testing, thereby creating multiple possible segmentations. These include histograming techniques, searching for vertical concavities that might signal a valid breakpoint between characters, and using rules that can effectively reconstruct commonly broken characters. Classification is performed for each such possible segmentation. This is integrated segmentation and recognition in the sense that final segmentation and classification judgments are interdependent. However, the approach presumably breaks down as the number of possible segmentations increases, as would occur for example if individual characters are broken or touching in multiple places, multiple letters in a sequence are connected, and the size of the vocabulary and variability of the writing styles increase. The COISR approach does not have this weakness because the net learns to classify images of characters in their natural states, be that broken, connected or unconnected, and because the NOT-CENTERED output node receives its input from the same hidden nodes that develop to classify characters. The COISR approach has similarities and differences with respect to another backpropagation-based integrated segmentation and recognition approach initially proposed by Rumelhart (1989) and then developed jointly at MCC by Keeler and Rumelhart (Keeler et al. 1991; Keeler and Rumelhart 1992). Keeler et al. refer to their approach as se2f-organizing
428
Gale L. Martin
integrated Segmentation and recognition (SOISR). Both COISR and SOISR techniques do not require any separate segmentation stage during testing, and both achieve integrated segmentation and recognition by first convolving a simple variant of a backpropagation network over a character field, and then parsing the resulting activation peaks to interpret the output of the convolutions. One implementation-level difference between the two is that SOISR performs the convolution in parallel, rather than over time as is the case with COISR. The primary difference between the two lies in what information about position or physical extent is required for training. To achieve their reported accuracy rates, Keeler et al. pretrain on characters that have first been presegmented; a person essentially draws a box around each character and the net learns to recognize these isolated characters. In subsequent training, a net with these pretrained weights replicated across the extent of a fixed-width input field is further trained on examples of fields that can contain connecting or broken characters. No position information, in the sense of the character-center positions used by COISR, is required with the SOISR approach. The COISR approach, on the other hand, does not require pretraining on isolated characters to achieve the reported accuracy rates. An approach very similar to the SOISR system has recently been developed by Matan and his colleagues (Matan et al. 1992). A weakness of the COISR approach is that it performs essentially an exhaustive scan over the to-be-classified input field. This means that the components needed to recognize a character in a given location must be replicated across the length (and height, if two dimensions are used) of the to-be-classified input field, at the degree of resolution necessary to recognize the smallest and closest characters. Since completion of the present work, we have developed a similar system, modeled loosely after human eye movements, that is trained to jump over blank areas within a field (Martin and Rashid 1992; Martin et al. 1993). This reduces the number of forward passes required per character to about 1.5. Acknowledgments I thank Lori Barski, John Canfield, David Chapman, Roger Gaborski, Jay Pittman, Mosfeq Rashid, and Dave Rumelhart for helpful discussions and/or development of supporting image handling and network software. I also thank Jonathan Martin for help with the position labeling. References Fenrich, R. 1991. Segmentation of automaticallylocated handwrittenwords. Paper presented at the International Workshop on Frontiers in Handwriting Recognition, Chateau de Bonas, France, 23-27 September.
COISR of Overlapping Handprinted Characters
429
Keeler, J., and Rumelhart, D. E. 1992. A self-organizing integrated segmentation and recognition neural network. In Advances in Neural Information Processing Systems, Vol. 4, J. E. Moody, S. J. Hanson, and R. P. Lippmann, eds. Morgan Kaufmann, San Mateo, CA. Keeler, J. D., Rumelhart, D. E., and Leow, Wee-Kheng. 1991. Integrated segmentation and recognition of handprinted numerals. In Advances in Neural Informution Processing Systems, Vol. 3, J. E. Moody and D. S. Touretzky, eds., pp. 557-563. Morgan Kaufmann, San Mateo, CA. Le Cun, Y., Boser, B., Denker, J., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D. 1990. Handwritten digit recognition with a backpropagation network. In Advances in Neural Information Processing Systems, Vol. 2, D. S. Touretzky, ed. Morgan Kaufmann, San Mateo, CA. Martin, G. L. 1990. Integrating segmentation and recognition stages for overlapping handprinted characters. MCC Tech. Rep. ACT-"-320-90. Martin, G. L., and Rashid, M. 1992. Recognizing overlapping hand-printed characters by centered-object integrated segmentation and recognition. In Advances in Neural Information Processing Systems, Vol. 4, J. E. Moody, S. J. Hanson, and R. P. Lippmann, eds. Morgan Kaufmann, San Mateo, CA. Martin, G. L., Rashid, M., and Pittman, J. 1993. Integrated segmentation and recognition through exhaustive scans or learned saccadic jumps. International Journal of Pattern Recognition and Artificial Intelligence (in press). Martin, G. L., and Pittman, J. A. 1990. Recognizing hand-printed letters and digits. In Advances in Neural Information Processing Systems, Vol. 2, D. S. Touretzky, ed. Morgan Kaufmann, San Mateo, CA. Matan, O., Burges, C. J. C., Le Cun, Y., and Denker, J. S. 1992. Multi-digit recognition using a space displacement neural network. In Advances in Neural Information Processing Systems, Vol. 4, J. E. Moody, S. J. Hanson, and R. P. Lippmann, eds. Morgan Kaufmann, San Mateo, CA. Rumelhart, D. 1989. Learning and generalization in multilayer networks. Presentation given at the NATO Advanced Research Workshop on Neuro Computing Algorithms, Architectures and Applications, Les Arcs, France, February. Sejnowski, T. J., and Rosenberg, C. R. 1986. NETtalk: A parallel network that learns to read aloud. JohnsHopkins University Electrical Engineering and Computer Science Tech. Rep. JHU/EEC!3-86/01. Shridhar, M., and Badreldin, A. 1986. Recognition of isolated and simply connected handwritten numerals. Pattern Recog. 19, 1-12. Shridhar, M., and Badreldin, A. 1987. Context-directed segmentation algorithm for handwritten numeral strings. Image Vision Comput. 5,3-9. Waibel, A., Sawai, H., and Shikano, K. 1988. Modularity and scaling in large phonemic neural networks. ATR Interpreting Telephony Research Laboratories Tech. Rep. TR-1-0034. Received 7 November 1991; accepted 19 October 1992.
This article has been cited by: 2. Cheng-Lin Liu, H. Sako, H. Fujisawa. 2004. Effects of classifier structures and training regimes on integrated segmentation and recognition of handwritten numeral strings. IEEE Transactions on Pattern Analysis and Machine Intelligence 26:11, 1395-1407. [CrossRef] 3. C. Garcia, M. Delakis. 2004. Convolutional face finder: a neural architecture for fast and robust face detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 26:11, 1408-1423. [CrossRef] 4. Gale L. Martin. 2004. Encoder: A Connectionist Model of How Learning to Visually Encode Fixated Text Images Improves Reading Fluency. Psychological Review 111:3, 617-639. [CrossRef] 5. Rafael Palacios, Amar Gupta, Patrick S. P. Wang. 2003. Feedback-based architecture for reading courtesy amounts on checks. Journal of Electronic Imaging 12:1, 194. [CrossRef] 6. David Horn , Irit Opher . 1996. Temporal Segmentation in a Neural Dynamic SystemTemporal Segmentation in a Neural Dynamic System. Neural Computation 8:2, 373-389. [Abstract] [PDF] [PDF Plus]
Communicated by Steven Zucker
Surface Interpolation Networks Alex P. Pentland Perceptual Computing Group, The Media Laboratory, Massachusetts Institute of Technology, Room E15-387,20 Ames Street, Cambridge M A 02139 USA
Orthogonal wavelets can be used as models for receptive fields in the human visual system. They may also be used to solve spatial interpolation problems formulated either as regularization or 2-DKalman filtering. The solutions take the form of simple feedback networks, and only a few iterations are required for convergence. 1 Introduction There exist families of interpolation problems in vision. Most familiar are the interpolation problems of stereopsis and contrast. Stereopsis provides sparse measurements of disparity which are filled in to produce the percept of a continuous surface. Similarly, the visual system’s initial filters respond only to constrast changes at moving edges; these contrast changes are filled in to produce the perceived pattern of lightness. Perhaps the most well-known interpolation theory in computational vision is regularization (Poggio ef al. 1985; Terzopoulos 1988). Using this approach optimal RMS estimates of the surface can be obtained under the assumption that the surface can be characterized as a stationary Markov process. However, this theory has the drawback that the interpolation network requires hundreds or even thousands of iterations to produce a smoothly interpolated surface. Thus there is a need for a more efficient theory of interpolation. In addition, there is a desire for more biologically plausible ones. Interpolation methods such as regularization are applicable only to stationary signals, that is, single images or static environments. To inte grate information across multiple views in a nonstationary environment requires the techniques of optimal estimation using dynamic models. The Kalman filter, perhaps the most common optimal estimation technique, produces optimal RMS estimates of the surface for both stationary processes and nonstationary processes with linear dynamics. In computer vision Kalman filter systems have been built (Matthies et al. 19891,and have been shown to be more accurate and robust than singleimage techniques such as regularization. However, these algorithms have been complex, computationally expensive, and not plausible as biological models. Neural Computation 5,430442 (1993) @ 1993 Massachusetts Institute of Technology
Surface Interpolation Networks
431
In this paper I will show how efficient, biologically plausible solutions to these interpolation problems can be obtained by using networks with orthogonal wavelet receptive fields. Numerical examples using natural imagery will be shown. An implementation in C code is available by anonymous FTP from whitechapel.media.mit.eduin the file /u/ftp/misc/wavelet.reg.tar.Z.
1.1 Background: Regularization. Surface interpolation typically involves constructing a piecewisesmooth surface given a sparse set of noisy contrast or disparity measurements. Because the measurements are sparse, the problem is ill-posed and requires adding a smoothing or regularizing term to obtain a solution in areas away from the measured disparities or contrasts. Mathematically, the static interpolation problem is formulated as finding a suitable function U that minimizes a smooth energy functional E(U) given boundary conditions 2). By taking the variational derivative 6, of the energy functional and discretizing over a lattice of n nodes, the following matrix equation is obtained (Poggio et al. 1985; Terzopoulos 1989): XKU + SU - D = 0
(1.1)
In this equation U is an n x 1 vector of unknown displacements for each of the n nodes, K is an n x n matrix called the regularizing or smoothness matrix, D is an n x 1 vector whose nonzero entries are the measured sensor data, S is a diagonal "selection matrix" with ones at nodes with sensor measurements, and X is a scalar constant that balances the relative influence of the data and regularization terms. An interpolated surface U that solves equation 1.1 can be obtained by iterating a two-layer network with center-surround receptive fields (Terzopoulos 1989). Unfortunately, several thousand iterations are typically required to obtain an interpolated surface; even if complex multiresolution techniques are employed, several hundred iterations are still required. 2 Orthogonal Wavelets
The amount of computation required for surface interpolation is proportional to both the bandwidth and condition number of K. Both can be greatly reduced by transforming the problem to another basis or coordinate system. In neural systems, such a transformation can be accomplished by passing incoming disparity or contrast measurements through a set of receptive fields; the shapes of the receptive fields are the new basis vectors, and the resulting neural activities are the coordinates of the measurement data in the coordinate system defined by these basis vectors. If the receptive fields are orthonormal, then we can convert back
432
Alex P. Pentland
Figure 1: Left column: The orthogonal wavelet filter family used in this paper (filters have been arbitrarily scaled for display). Right column: The power spectra of these filters on a linear scale.
to the original coordinate system by adding up the same receptive fields in amounts proportional to the associated neuron’s activity. A class of bases that greatly improve the bandwidth ahd condition number of K are generated by functions known as orthogonal wavelets. A family of orthogonal wavelets ha,b is constructed from a single function
Surface Interpolation Networks
433
h by dilation of a and translation of b (Daubechies 1988; Mallat 1989; Simoncelli and Adelson 1990))
ha,$= (al-”2h
(?)
,
a
#0
(2.1)
Typically u = 2j and b = 1,.. . ,n = 2j for j = 1 , 2 , 3 .. . . In an orthogonal wavelet family all of the wavelets-including all translations, sizes, and orientations-are orthogonal to one another, and thus form a multiscale orthonormal basis. I will call such a basis awl where the columns of the n x n matrix iDw are the basis vectors. Because awforms an orthonormal basis, it (like the Fourier transform) is self-inverting, for example, *Law= 4Dw@; = I. Consequently the computational properties of orthogonal wavelets are very different from, for example, Gabor or derivative-of-gaussian wavelets, even though they may appear visually similar. The wavelet basis awused in this paper was ”learned” using a gradient-descent procedure (Simoncelli and Adelson 1990) starting with difference-of-gaussians as the initial “guess” at an orthogonal basis. The lefthand column of Figure 1 shows a subset of aw; from top to bottom are the basis vectors corresponding to u = 1,2,4,8,16and b = n/2, together with their Fourier power spectrum. As can be seen, these wavelets are similar to those known to exist in human vision; for instance, there is only a 7.5% MSE difference between these wavelets and those of the Wilson-Gelb model of human spatial vision (Pentland 1991; Wilson and Gelb 1984). On digital computers transformation to the wavelet coordinate system is normally computed recursively using separable filters (Simoncelli and Adelson 1990). Because the process is recursive it requires only O(n) operations, and is thus among the most efficient of all orthogonal transforms. The wavelet transform of a 128 x 128 pixel image requires only 2.0 sec on a Sun 4/330. 3 Surface Interpolation Using Wavelet Bases
It has been proven that by using orthogonal wavelet bases smooth linear operators can be represented extremely compactly (Albert et al. 1990). This suggests that awis an effective preconditioning transform, and thus may be used to obtain very fast approximate solutions. The simplest method is to transform a previously defined K to the wavelet basis, discard off-diagonal elements,
i-2;
= diag
(@:Kaw)
(3.1)
and then solve. Note that for each choice of K the diagonal matrix i-2; is calculated only once and then stored; further, this calculation requires only O(n) operations. In ‘numericalexperiments I have found that for a
Alex P. Pentland
434
typical K the summed magnitude of the off-diagonals of K is approximately 5% of the diagonal's magnitude, so that we expect to incur only small errors by discarding off -diagonals. Case 1. The simplest case of surface interpolation is when sensor measurements exist for every node so that the sampling matrix S = I. Substituting @,U = U and premultiplying by @; converts equation 1.1 to X@;K@,U
+ @;@,U= @;D
(3.2)
By employing equation 3.1, we then obtain
(An: + 1)U = @;D
(3.3)
The approximate interpolation solution U is therefore U = @,(Xs2:
+ I)-'@;D
(3.4)
Note that this computation is accomplished by simply transforming the input data D to the wavelet basis, multipling the filter output appropriately at each level of recursion, and then transforming back to the original coordinate system. To obtain an approximate regularized solution for an fi x fi image using a wavelet of width w therefore requires approximately 8wn + n add and multiply operations. Case 2. In the more usual case where not all nodes have sensor measurements, the interpolation solution may require iteration. In this case the sampling matrix S is diagonal with ones for nodes that have sensor measurements, and zeros elsewhere. Again substituting aWU =U and premultiplying by @; converts equation 1.1 to X*;K@,U
+ @P1;S@,U = @;D
(3.5)
The matrix @;S@, is diagonally dominant so that the interpolation solution U may be obtained by iterating
U"' = @(,
+ S)-'@iD' + Uf
(3.6)
where S = diag(@;SaW) and Dt = D - (K + S)U' is the residual at iteration t. Normally no more than three to five iterations of equation 3.6 are required to obtain an accurate estimate of the interpolated surface; often a single iteration will suffice. Note that for this procedure to succeed, the largest gaps in the data sampling must be significantly smaller than the largest filters in the wavelet transform. Further, when X is small and the data sampling is sparse and irregular, it can happen that the off-diagonal terms of @;S*, introduce significant error. Therefore when using small X it is best to perform one initial iteration with a large A, and then reduce X to the desired value in further iterations. Discontinuities. The matrix K describes the connectivity between adjacent points on a continuous surface; thus whenever a discontinuity
Surface Interpolation Networks
435
occurs K must be altered. Following Terzopoulos (1988), we can accomplish this by disabling receptive fields that cross discontinuities. In a computer implementation, the simplest method is to locally halt the recursive construction of the wavelet transform whenever one of the resulting bases would cross a discontinuity. An Example. Figure 2a shows synthetic disparity measurements input to a 64 x 64 node interpolation problem (zero-valued nodes have no data); the vertical axis is disparity. These data were generated using a sparse (10% random sampling of the function z = lOO[sin(kxj+ sin(ky)]. Figure 2b shows the resulting interpolated surface. In this example equation 3.6 converged to within l%of its true equilibrium state with a single iteration. Execution time was approximately 1 sec on a Sun 4/330. 4 Surface Interpolation in Nonstationary Environments
The technique of regularization is applicable only to stationary signals, that is, single images or static environments. To integrate information across multiple views in a non-stationary environment, the techniques of optimal estimation and dynamic systems must be used to smooth the interpolated surface over time as well as space (Friedland 1986; Matthies et al. 1989). The simplest such technique is known as the Kalmun filter; for problems with linear dynamics it produces an optimal RMS error estimate across time as well as space. The key idea of the Kalman filter is that to obtain optimal estimates we must take into account the environment’s dynamics. We will model these dynamics as a linear differential equation,
d
-X = AX at
+ BU
(4.1)
where X is a vector of unknowns, A and B are matrices, and a is a white noise process. The sensor observations will be modeled as linear function of the unknowns and a second white noise process n, Y=CX+n
(4.2)
Then the optimal RMS estimate X of X is given by the following continuous Kalman filter
d -x dt
= AX
+ K(Y - cic)
(4.3)
where the Kalman gain matrix K depends on the matrices A, B, and C, and on the noises a and n. Equation 4.3 provides a robust, optimal method of estimating the unknown state variables of a nonstationary linear process. However interpolating disparity or contrast in this manner is typically very expensive,
F i p 2 A typical disparity interpolation problem. (a) Disparity data input to a 64 x 64 node interpolation problem, vertical axis is disparity. (b) Interpolated surface after one iteration (approximately 1 sec on a Sun 4/330).
Surface Interpolation Networks
437
Figure 3 (a) A scene, (b) inverse distances (disparities) for this scene, (c) the autocorrelation function for this scene, (d) the autocorrelation function after transformation to the wavelet coordinate system. because correlations in the data cause the matrices N, A, and K to have many nonzero bands. Thus it is desirable to transform the problem to a basis in which the input data is decorrelated. In such a basis the matrices will be nearly diagonal, so that the interpolated surface can be obtained efficiently. Figure 3 illustrates the performance of the wavelet transform at decorrelating image structure. Figure 3a shows a simple scene, and Figure 3b shows the inverse distances (disparities) for the same scene, obtained as described in Pentland (1987). Figure 3c shows one row from the autocorrelation matrix computed from these disparities. This autocorrelation is similar to that of a second-order Gauss-Markov process; note that significant correlations exist across more than 64 pixels distance. Figure 3d shows the same autocorrelation function after transformation to the wavelet coordinate system. As can be seen, in the wavelet coordinate system the autocorrelation matrix is very nearly diagonal, reducing the bandwidth of the matrices N, A, and K. 4.1 A Simple Example. As an example I will show how to construct a simple Kalman filter for disparity U and its time derivative U at each
Alex P. Pentland
438
point in an fi x fi image. In this simple example we will assume that we have noisy disparity measurements available at every image point; the sparse-data case is similar. In state-space notation our system of equations is d
[ u ] [ x 4[ 4
;II
U
=
(4.4)
+ [ ; ] a
where U and U are n x 1 vectors, I is the n x n identity matrix, and a is an n x 1 noise vector due to the unknown accelerations of the observed points. The observed variable will be estimated point-by-point disparities U,:
.[
U,=U+n
(4.5)
where n is an n x 1 vector of observation noise. The Kalman filter is therefore
d
u] u
=[:I+[ I;:
(uo-II
01[
q)
(4.6)
where KIand KZ are the n x n Kalman gain matrices for velocity and acceleration, respectively. We will assume that n and a originate from independent second-order Gauss-Markov noise processes. As illustrated in Figure 3, such processes are nearly completely decorrelated in a wavelet-defined coordinate system, so that their autocorrelation matrices become nearly diagonal. Thus we have that
where N and A are diagonal matrices. Given this approximation to N and A we may then determine the Kalman gain matrices in the standard way (Friedland 19861, which are
Substituting this result into equation 4.6 we obtain (4.9)
Surface Interpolation Networks
439
Letting U = @iU, and premultiplying by
Po
(AN-’)
@it we obtain (4.10)
- U)
as a;@, = I. As can be seen, the unknowns U are functions only of the measurements Uoand the diagonal matrices A and N. Consequently, in the wavelet coordinate system the Kalman filter equations are decoupled into n independent two-variable Kalman filters. The major consequence of this decoupling is that only O(n) computations and O(n) storage locations are required. Even in the variable-noise case (not discussed here due to space limitations) only O(n)computations are required, as even space-varying Markov N are approximately diagonal in the wavelet coordinate system. Results. Figure 4a shows the sixth frame from a synthetic image sequence of Yosemite Valley, as seen from the vantage point of a small plane flying down the center of the valley floor. A sequence of corresponding disparity images, the sixth of which is shown in Figure 4b, was generated by reprojecting a digital terrain map of the area from the same set of viewpoints. These 128 x 128 disparity images were then corrupted by the addition of uniformly distributed correlated noise (Power(w) = w-O.’ where w is spatial frequency) resulting in a sequence of disparity images with a signal-to-noise ratio of 1:1, as is illustrated by Figure 4c. Accurate estimates of disparity cannot be obtained by averaging successive frames of the noisy range images, because of the curved camera path and perspective distortion. Similarly, regularization of individual frames does not produce accurate estimates due to the large amount of correlated noise. These corrupted disparity estimates were fed into the Kalman filter described above. The computation required approximately 5 sec per frame on a Sun 4/330. The disparity estimates at frame six are shown in Figure 4d. Comparing these estimates to the true disparities, shown in Figure 4b, it can be seen that a good interpolation was obtained. At frame six the mean per-pixel error in the disparity estimate was 8.5% of the initial error (an improvement of 21 db) demonstrating the stability and efficiency of this formulation. 5 Possible Biological Implementations
Figure 5a illustrates the process of surface interpolation using wavelet receptive fields. The input data D are passed through a layer of neurons with receptive fields such as are shown in Figufe 1 at each location. This computes @iD,the wavelet transform of D. The activity of each neuron
Alex I? Pentland
440
Figure 4 (a) The sixth frame of a fly-through of Yosemite valley, (b) true disparities for this image, (c) true disparity plus additive correlated noise with a signal-to-noise ratio of 1:1, (d) Kalman filter estimates of disparity.
is then scaled by a factor dependent upon its central frequency, thus computing (An: I)-'*LD. Finally, each neuron's output is summed with a spatial distribution equal to its receptive field, thus computing the inverse wavelet transform and obtaining the interpolated surface U = iP,(AQ~ +I)-'iPfD. Note that this figure illustrates the case where S = I, more generally a second such mechanism is required in order to also compute S = i ~ $ + ~ . Figure 5b illustrates one way this computation can be mapped onto neurons. In this figure the input layer arborizes very locally, with the pyramidal cell's basal dendrites producing receptive fields shaped as in Figure 1. This transforms the input to the wavelet basis. The sensitivity of these neurons is presumed inversely proportional to their central frequency. The output axons then arborize with the spatial distribution
+
Surface Interpolation Networks
441
Layer 1 Layer 2.3
sparse, noisy, inpul dals
Layer 4
14 Inlerallons n n d d lor sparse dala
Figure 5: (a) Interpolation via regularization, (b) one possible neural implementation. (c) Interpolation via Kalman filtering, (d) one possible neural implementation.
of Figure 1 among the apical dendrites of a second layer of neurons, producing receptive fields similar to those of the basal dendrites. This produces the inverse wavelet transform, so that the output of the second layer of neurons is an interpolated surface. It is important to note that the required orthogonal receptive field structure can be learned from an initial, unspecific center-surround r e ceptive field. In fact, the wavelets shown in Figure 1 were derived in this manner, using a technique similar to that of Kohonen (1982) or Linsker (1986). The Kalman filter interpolation network is nearly identical to the regularization network. The only difference is that a simple neuron-by-neuron temporal feedback loop is required to implement the computation of equation 4.10, as illustrated in Figure 5c. This feedback loop combines the current excitatory input to each neuron with its activity at the previous instant in time, thus smoothing the neuron's output over time. One way of implementing this feedback loop is with recurrent axons, as shown in Figure 5d. Note that the recurrent axons must also arborize with the spatial distribution of Figure 1.
442
Alex P. Pentland
6 Summary
I have described two interpolation methods that use orthogonal wavelets to efficiently obtain good surface interpolations. These methods have a simple biological implementation, and use wavelets that are similar to those found in human spatial vision. The first network performed surface interpolation via regularization, and is applicable to single images or static environments. For changing environments, it is neccessary to employ optimal estimation techniques, for example, Kalman filtering, to integrate information across time. A network that accomplishes surface interpolation via Kalman filters was also described; it is the same as the regularization network except for the addition of a feedback loop. References Albert, B., Beylkin, G., Coifman, R., and Rokhlin, V. 1990. Wavelets for the Fast Solution of Second-Kind Integral Equations. Yale Research Report DCS.RR-837, December. Daubechies, I. 1988. Orthonormal bases of compactly supported wavelets. Commun. Pure Appl. Math. XLI, 909-996. Friedland, B. 1986. Control System Design. McGraw-Hill, New York. Kohonen, T. 1982. Self-organized formation of topologically correct feature maps. B i d . Cyber. 43, 59-69. Linsker, R. 1986. From basic network principles to neural architecture. Proc. Natl. Acud. Sci. U.S.A. 83, 7508-7512, 8390-8394, 877944783. Mallat, S. G. 1989. A theoy for multiresolution signal decomposition: The wavelet representation. IEEE Trans. PAMI 11(7), 674-693. Matthies, L., Kanade, T., and Szeliski, R. 1989. Kalman-filter based algorithms for estimatingdepth from image sequences. 1nt.J.Computer Vision 3,209-236. Pentland, A. P. 1987. A new sense for depth of field. IEEE Trans. PAMl 9(4), 523-531. Pentland, A. 1991. Cue integration and surface completion. Invest. Ophthulmal. Visual Sci. 32(4), 1197. Poggio, T., Torre, V., and Koch, C. 1985. Computational vision and regularization theory. Nature (London) 317, 314-319. Simoncelli, E., and Adelson, E. 1990. Non-separable extensions of quadrature mirror filters to multiple dimensions. Proc. IEEE 78(4), 652-664. Terzopoulos, D. 1988. The computation of visible surface representations. IEEE Trans. PAMl10(4),417-439. Wilson, H., and Gelb, G. 1984. ModGed line-element theoyfor spatial-frequencyand width discrimination. J. Opt. SOC.Am.A 1(1), 124-131. Received 8 July 1991; accepted 4 November 1992.
This article has been cited by: 2. Katsuhisa Hirokawa, Kazuyoshi Itoh, Yoshiki Ichioka. 1997. Real-time Optical Wavelet-Transform with Positive and Negative Signals. Optical Review 4:3, 366-369. [CrossRef] 3. Y. Xu, R. N. Miles. 1996. Experimental determination of bending strain power spectra from vibration measurements. Experimental Mechanics 36:2, 166-172. [CrossRef]
Communicated by Steven J. Nowlan
Combining Exploratory Projection Pursuit and Projection Pursuit Regression with Application to Neural Networks Nathan Intrator' Institute for Brain and Neural Systems, Brown University, Box 1843, Providence, NO2912 USA
We present a novel classification and regression method that combines exploratory projection pursuit (unsupervised training) with projection pursuit regression (supervised training), to yield a new family of costlcomplexity penalty terms. Some improved generalization properties are demonstrated on real-world problems. 1 Introduction
Parameter estimation becomes difficult in high-dimensional spaces due to the increasing sparseness of the data. Therefore, when a low-dimensional representation is embedded in the data, dimensionality reduction methods become useful. One such method-projection pursuit regression (Friedman and Stuetzle 1981 (PPR)-is capable of performing dimensionality reduction by composition, namely, it constructs an approximation to the desired response function using a composition of lower dimensional smooth functions. These functions depend on low-dimensional projections through the data. When the dimensionality of the problem is in the thousands, even projection pursuit methods are almost always overparameterized, therefore, additional smoothing is needed for low variance estimation. Exploratory projection pursuit (Friedman and Tukey 1974; Friedman 1987) (EPP) may be useful in these cases. It searches in a high-dimensional space for structure in the form of (semillinear projections with constraints characterized by a projection index. The projection index may be considered as a universal prior for a large class of problems, or may be tailored to a specific problem based on prior knowledge. In this paper, the general form of exploratory projection pursuit is formulated to be an additional constraint for projection pursuit regression. In particular, a hybrid combination of supervised and unsupervised artificial neural network (ANN)is described as a special case. In addition, a specific projection index that is particularly useful for classification (Intrator 1990; Intrator and Cooper 1992) is introduced in this context. 'Present address: Computer Science Department, Tel-Aviv University, Ramat-Aviv, 69978 Israel. Neural Computation 5,443-455 (1993) @ 1993 Massachusetts Institute of Technology
Nathan Intrator
444
There have been many other attempts to combine unsupervised with supervised learning (Yamac 1969; Gutfinger and Sklansky 1991; Bridle and MacKay 1992). The formulation discussed below is based on projection pursuit ideas that generalize many of the classical statistical methods, and in our case, suggests a well-defined statistical framework, that allows formulation and comparison between these methods. 2 Brief Description of Projection Pursuit Regression
Let (X, Y) be a pair of random variables, X E Rd, and Y E R. The problem is to approximate the d-dimensional surface
f(x) = E[YI X
= X]
from n observations ( x ~ , y ~ .).!,.( x n , y n ) . PPR tries to approximate a function f by a sum of ridge functions (functions that are constant along lines) m
f(x) N CgjC$x) j=1
The fitting procedure alternates between an estimation of a direction Ci and an estimation of a smooth function g, such that at iteration j, the square average of the residuals
-r r-(xi) '1 = rij-l - gj(uj x i ) is minimized. This process is initialized by setting rio = yi. Usually, the initial values of uj are taken to be the first few principal components of the data. Estimation of the ridge functions can be achieved by various nonparametric smoothing techniques such as locally linear functions (Friedman and Stuetzle 1981), k-nearest neighbors (Hall 1989b), splines, or variable degree polynomials. The smoothness constraint imposed on g implies that the actual projection pursuit is achieved by minimizing at iteration j , the sum
2
$(xi)
+ c(gj)
i=l
for some smoothness measure C. Due to the fact that the estimation of the nonparametric ridge functions is not decoupled from the estimation of the projections, overfitting is very likely to occur in one of the low-order gj, thereby invalidating subsequent estimations. Obviously, if g is not well estimated, the search €or optimal projection direction will not yield good results.
Combining EPP and PPR
445
Several alternatives have been considered in addressing this problem: 0
0
Choose the ridge functions {gj} from a very small family of functions, for example, sigmoidals with a variable threshold. This eliminates the need to estimate the nonparametric ridge function, but increases the complexity of the architecture. This approach is widely used in artificial neural networks, and may partially explain their success. Estimate a fixed number of ridge functions and projections concurrently (as opposed to sequential estimation) provided that the ridge functions are taken from a very limited set of functions. Again this is used in the context of neural networks, due to the relatively small additional computational burden.
Additionally, one may attempt to 0
Partially decouple the estimation of the response function, or the estimation of each of the ridge regression functions from the estimation of the projections.
Ultimately, it is reasonable to combine all of the above. One such implementation is presented in the following sections. First, the issue of decoupling the estimation of the ridge functions from the estimation of the projections is discussed. 3 Estimating the Projections Using Exploratory Projection Pursuit -
Exploratory projection pursuit is based on seeking interesting projections of high-dimensional data points (Switzer 1970; Kruskall969,1972; Friedman and Tukey 1974; Friedman 1987; Jones and Sibson 1987; Hall 1988; Huber 1985, for review). The notion of interesting projections is motivated by an observation that for most high-dimensional data clouds, most low-dimensional projections are approximately normal (Diaconis and Freedman 1984). This finding suggests that the important information in the data is conveyed in those directions whose single dimensional projected distribution is far from gaussian. Various projection indices (measures for the goodness of a projection) differ on the assumptions about the nature of deviation from normality, and in their computational efficiency. They can be considered as different priors motivated by specific assumptions on the underlying model. To partially decouple the search for a projection vector from the search for a nonparametric ridge function, we propose to add a penalty term, which is based on a projection index, to the energy minimization associated with the estimation of the ridge functions and the projections. Specifically, let p(a) be a projection index that is minimized for projections with a certain deviation from normality. At the jth iteration, we
Nathan Intrator
446
When a concurrent minimization over several projections/ functions is practical, we get a penalty term of the form I
Since C and p may not be linear, the more general measure that does not assume a stepwise approach, but instead seeks I projections and ridge functions concurrently, is given by
In practice, p depends implicitly on the training data (the empirical density) and is therefore replaced by its empirical measure b.
3.1 Some Possible Measures. Some applicable projection indices have been discussed (Huber 1985; Jones and Sibson 1987; Friedman 1987; Hall 1989a; Intrator 1990). Probably, all the possible measures should emphasize some form of deviation from normality but the specific type may depend on the problem at hand. For example, a measure based on the Karhunen Ldve expansion (Mougeot et al. 1991) may be useful for image compression with autoassociative networks, since in this case one is interested in minimizing the L2 norm of the distance between the reconstructed image and the original one, and under mild conditions, the Karhunen Loeve expansion gives the optimal solution. A different type of prior knowledge is required for classification problems. The underlying assumption then is that the data are clustered (when projecting in the right directions) and that the classification may be achieved by some (nonlinear) mapping of these clusters. In such a case, the projection index should emphasize multimodality as a specific deviation from normality. A projection index that emphasizes multimodalities in the projected distribution (without relying on the class labels) has recently been introduced (Intrator 1990) and implemented efficiently using a variant of a biologically motivated unsupervised network (Intrator and Cooper 1992). Its integration into a backpropagation classifier will be discussed below. 4 A Variant of Projection Pursuit Regression: Backpropagation
Network In this section, we consider a parametric approach-the backpropagation network-as a variant of PPR. In this context the addition of an exploratory projection index is discussed.
Combining EPP and PPR
447
Backpropagation (Werbos 1974; Le Cun 1985; Rumelhart et al. 1986) has been chosen as a possible representative for the first two alternatives presented in Section 2, since it has become a useful tool for solving complicated pattern recognition tasks such as speech recognition (Lippmann 1989), and since the class of functions that can be approximated by a backpropagation type network is very large. This architecture (with an unlimited number of projections) can uniformly approximate arbitrary continuous functions on compact sets (Cybenko 1989; Hornik et al. 1989) as well as their derivatives (Hornik et al. 19901, and do so efficiently. Related results can be found (Carroll and Dickinson 1989; Funahashi 1989; Hecht-Nielsen 1989; Hornik 1991; Ito 1991). In this method, the error is efficiently propagated backward to the previous layer for modification of their synaptic weights (projections). The single hidden layer architecture is of the form
where u is an arbitrary (fixed) bounded monotone function. The form
is more suitable for classification tasks. Since this method can approximate any continuous function, great care should be taken so that the variance of the estimator is not large, namely, that the model does not "overfit" the training data (Wahba 1990; Geman et al. 1992, for discussion). This can be done using some form of complexity regularization (Barron and Barron 1988; Barron 1989; White 1990; 'Moody 1991) or by weight elimination penalties that aim to reduce the effective number of parameters in the model (Plaut et al. 1986; Mozer and Smolensky 1989; Le Cun et al. 1990; Weigend et al. 1991). The performance of the network is measured using a loss criterion, for example, mean squared error between the output and the target of the network (the class label). The estimation of the weights is done by minimizing the empirical average of the error via gradient descent of the form: h i j / a t = - d & / h , , where & = E,[&(x,w)],is the average contribution to the loss criterion of each of the random inputs x . 4.1 Adding EPP Constraintsto Backpropagation Network. One way of adding some prior knowledge into the architecture is by minimizing the effective number of parameters using weight sharing, in which a single weight is shared among many connections in the network (Waibel et al. 1989; Le Cun et al. 1989). An extension of this idea is the "soft weight sharing," which favors irregularities in the weight distribution in the form of multimodality (Nowlan and Hinton 1992). This penalty
Nathan Intrator
448
Figure 1: A hybrid EPP/PPR neural network (EPPNN).
improved generalization results obtained by weight elimination penalty. Both these methods make an explicit assumption about the structure of the weight space, but with no regard to the structure of the input space. As described in the context of projection pursuit regression, a penalty term may be added to the energy functional minimized by error backpropagation, for the purpose of measuring directly the goodness of the projections sought by the network. Since our main interest is in reducing overfitting for high-dimensional problems, our underlying assumption is that the surface function to be estimated can be faithfully represented using a low-dimensional composition of sigmoidal functions, namely, using a backpropagation network in which the number of hidden units is much smaller than the number of input units. Therefore, the penalty term may be added only to the hidden layer (see Fig. 1). The synaptic modification equations of the hidden units' weights become j -b i =
at
-€
aqw, [x>
b i j
+
aP(Wl7
' * *
7
WrI)
dWij
1
+(contribution of cost/complexity terms)
An approach of this type has been used in image compression, with a
penalty aimed at minimizhg the entropy of the projected distribution (Bichsel and Seitz 1989). This penalty certainly measures deviation from normality, since entropy is maximized for a gaussian distribution.
Combining EPP and PPR
449
5 Projection Index for Classification: The Unsupervised BCM Neuron
Intrator (1990) has recently shown that a variant of the Bienenstock, Cooper, and Munro neuron (BCM) (Bienenstock et al. 1982) performs exploratory projection pursuit using a projection index that measures multimodality. This neuron version allows theoretical analysis of some visual deprivation experiments (Intrator and Cooper 1992), and is in agreement with the vast experimental results on visual cortical plasticity (Clothiaux et al. 1991). A network implementation that can find several projections in parallel while retaining its computational efficiency, was found to be applicable for extracting features from very high-dimensional vector spaces (Intrator and Gold 1992; Intrator et al. 1991; Intrator 1992). The activity of neuron k in the network is Ck = Cixiwik wok. The inhibited activity and threshold of the kth neuron is given by
+
The threshold 6; is the point at which the modification function 4 changes sign (see Intrator and Cooper 1992 for further details). The function 4 is given by f#J(c,0,) = c(c - 0,)
The risk (projection index) for a single neuron is given by
The total risk is the sum of each local risk. The negative gradient of the risk that leads to the synaptic modification equations is given by
This last equation is an additional penalty to the energy minimization of the supervised network. Note that there is an interaction between adjacent neurons in the hidden layer. In practice, the stochastic version of the differential equation can be used as the learning rule.
5.1 Some Related Statistical and Computational Issues of This Projection Index. This section discusses some commonly asked questions regarding the connection of the above projection index to previous work in pattern recognition and statistics. Although the projection index is motivated by the desire to search for clusters in the high-dimensional data, the resulting feature extraction
450
Nathan Intrator
method is quite different from other pattern recognition methods that search for clusters. Since the class labels are not used in the search, the projection pursuit is not biased to the class labels. This is in contrast with classical methods such as discriminant analysis (Fisher 1936; Sebestyen 1962, and numerous recent publications). The projection index concentrates on projections that allow discrimination between clusters and not faithful representation of the data. This is in contrast to principal components analysis, or factor analysis, which tend to combine features that have high correlation (see review in Harman 1967). The method differs from cluster analysis by the fact that it searches for clusters in the low-dimensional projection space, thus avoiding the inherent sparsity of the high-dimensional space. The projection index uses low-order polynomial moments, which are computationally efficient, yet it does not suffer from the main drawback of polynomial moments-sensitivity to outliers. It naturally extends to multidimensional projection pursuit using the feedforward inhibition network. The number of calculations of the gradient grows linearly with the dimensionality and linearly with the number of projections sought. 6 Applications
We have applied this hybrid classification method to various speech and image recognition problems in high-dimensional space. In one speech application we used voiceless stop consonants extracted from the TIMIT database as training tokens (Intrator and Tajchman 1991). A detailed biologically motivated speech representation was produced by Lyon’s cochlear model (Lyon 1982; Slaney 1988). This representation produced 5040 dimensions (84 channels x 60 time slices). In addition to an initial voiceless stop, each token contained a final vowel from the set [aa, ao, er, iyl. Classification of the voiceless stop consonants using a test set that included 7 vowels [uh, ih, eh, ae, ah, uw, owl produced an average error of 18.8% while on the same task classification using backpropagation network produced an average error of 20.9% (a significant difference, p < 0.0013). Additional experiments on vowel tokens appear in Tajchman and Intrator (1992). Another application is in the area of face recognition from gray level pixels (Intrator et al. 1992). After aligning and normalizing the images, the input was set to 37 x 62 pixels (total of 2294 dimensions). The recognition performance was tested on a subset of the MIT Media Lab database of face images made available by Turk and Pentland (1991) which contained 27 face images of each of 16 different persons. The images were taken under varying illumination and camera location. Of the 27 images available, 17 randomly chosen ones served for training and the remaining 10 were used for testing. Using an ensemble average of hybrid networks (Lincoln and Skrzypek 1990; Pearlmutter and Rosenfeld 1991; Perrone
Combining EPP and PPR
451
and Cooper 1992) we obtained an error rate of 0.62% as opposed to 1.2% using a similar ensemble of backpropagation networks. A single backpropagation network achieves an error between 2.5 and 6% on these data. The experiments were done using 8 hidden units. 7 Summary
A penalty that allows the incorporation of additional prior information on the underlying model was presented. This prior was introduced in the context of projection pursuit regression, classification, and in the context of backpropagation network. It achieves partial decoupling of estimation of the ridge functions (in PPR) or the regression function in backpropagation net from the estimation of the projections. Thus it is potentially useful in reducing problems associated with overfitting, which are more pronounced in high-dimensional data. Some possible projection indices were discussed and a specific projection index that is particularly useful for classification was presented in this context. This measure that emphasizes multimodality in the projected distribution was found useful in several very high-dimensional problems. Acknowledgments
I wish to thank Leon Cooper, Stu Geman, and Michael Perrone for many fruitful conversations and the referee for helpful comments. The speech experiments were performed using the computational facilities of the Cognitive Science Department at Brown University. Research was sup ported by the National Science Foundation, the Army Research Office, and the Office of Naval Research. References Barron, A. R. 1989. Statistical properties of artificial neural networks. In Proc. IEEE Conf. on Decision and Control, pp. 280-285. IEEE Press, New York. Barron, A. R., and Barron, R. L. 1988. Statistical learning networks: A unifying view. In Computing Science and Statistics: Proc. 20th Symp. Interface, E. Wegman, ed., pp. 192-203. American Statistical Association, Washington, DC. Bichsel, M., and Seitz, P. 1989. Minimum class entropy: A maximum information approach to layered networks. Neural Networks 2,133-141. Bienenstock, E. L., Cooper, L. N., and Munro, P. W. 1982. Theory for the development of neuron selectivity: Orientation specificity and binocular interaction in visual cortex. 1.Neurosci. 2, 32-48. Bridle, J. S.,and MacKay, D. J. C. 1992. Unsupervised classifiers, mutual information and ‘Phantom Targets’. In Advances in Neural Information Processing
452
Nathan Intrator
Systems, Vol. 4,J. Moody, S.Hanson, and R. Lippmann, eds., pp. 1096-1101. Morgan Kaufmann, San Mateo, CA. Carroll, S. M., and Dickinson, 8. W. 1989. Construction of neural net using the radon transform. In International Joint Conference on Neural Networks, Vol. 1, pp. 607-611. IEEE Press, New York. Clothiaux, E. E., Cooper, L. N., and Bear, M. E 1991. Synaptic plasticity in visual cortex: Comparison of theory with experiment. Journal of Neurophysiology 66, 1785-1 804. Cybenko, G. 1989. Approximations by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems 2, 303-314. Diaconis, I?, and Freedman, D. 1984. Asymptotics of graphical projection pursuit. Ann. Statist. 12, 793-815. Fisher, R. A. 1936. The use of multiple measurements in taxonomic problems. Ann. Eugen. 7,179-188. Friedman, J. H. 1987. Exploratory projection pursuit. I. Am. Statist. Assoc. 82, 249-266. Friedman, J. H., and Stuetzle, W. 1981. Projection pursuit regression. J . Am. Statist. Assoc. 76, 817-823. Friedman, J. H., and Tukey, J. W. 1974. A projection pursuit algorithm for exploratory data analysis. I E E E Transact. Computers C(23), 881-889. Funahashi, K. 1989. On the approximate realization of continuous mappings by neural networks. Neural Networks 2, 183-192. Geman, S., Bienenstock, E., and Doursat, R. 1992. Neural networks and the bias-variance dilemma. Neural Comp. 4, 1-58. Gutfinger, D.and Sklansky, J. 1991. Robust classifiers by mixed adaptation. IEEE Transact. Pattern Anal. Machine Intelligence 13, 552-567. Hall, P. 1988. Estimating the direction in which data set is most interesting. Probab. Theory Rel. Fields 80, 51-78. Hall, P. 1989a. On polynomial-based projection indices for exploratory projection pursuit. Ann. Statist. 17, 589-605. Hall, P. 1989b.On projection pursuit regression. Ann. Statist. 17, 573-588. Harman, H. H. 1967. Modern Factor Analysis, 2nd ed. University of Chicago Press, Chicago. Hecht-Nielsen, R. 1989. Theory of the backpropagation neural network. In International Joint Conference on Neural Networks, Vol. 1, pp. 593-606. IEEE Press, New York. Hornik, K. 1991. Approximation capabilities of multilayer feedforward networks. Neural Networks 4,251-257. Hornik, K.,Stinchombe, M., and White, H. 1989. Multilayer feedforward networks are universal approximators. Neural Networks 2,359-366. Hornik, K., Stinchombe, M., and White, H. 1990. Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks. Neural Networks 3,551-560. Huber, P.J. 1985. Projection pursuit. (with discussion). Ann. Statist. 13,435-475. Intrator, N. 1990. Feature extraction using an unsupervised neural network. In Proceedings of the 1990 Connectionist Models Summer School, D. S. Touretzky,
Combining EPP and PPR
453
J. L. Ellman, T. J. Sejnowski, and G. E. Hinton, eds., pp. 310-318. Morgan Kaufmann, San Mateo, CA. Intrator, N. 1992. Feature extraction using an unsupervised neural network. Neural Comp. 4,98-107. Intrator, N., and Cooper, L. N. 1992. Objective function formulation of the BCM theory of visual cortical plasticity: Statistical connections, stability conditions. Neural Networks 5,3-17. Intrator, N., and Gold, J. I. 1992. Three-dimensional object recognition of gray level images: The usefulness of distinguishing features. Neural Comp. 5, 61-74. Intrator, N., and Tajchman, G. 1991. Supervised and unsupervised feature extraction from a cochlear model for speech recognition. In Neural Networks for Signal Processing - Proceedings of the 2992 IEEE Workshop, B. H. Juang, S. Y. Kung, and C. A. Kamm, eds., pp. 460-469. IEEE Press, New York. Intrator, N., Gold, J. I., Biilthoff, H. H., and Edelman, S. 1991. Three-dimensional object recognition using an unsupervised neural network: Understanding the distinguishingfeatures. In Proceedings of the 8th Israeli Conferenceon AICV, Y. Feldman and A. Bruckstein, eds., pp. 113-123. Elsevier, Amsterdam. Intrator, N., Reisfeld, D., and Yeshurun, Y. 1992. Face recognition using a hybrid supervised/unsupervised neural network. Preprint. Ito, Y. 1991. Representation of functions by superpositions of a step or sigmoid function and their applications to neural network theory. Neural Networks 4, 385-394. Jones, M. C. and Sibson, R. 1987. What is projection pursuit? (with discussion). J. R. Statist. SOC. Set. A(150), 1-36. Kruskal, J. B. 1969. Toward a practical method which helps uncover the structure of the set of multivariate observations by finding the linear transformation which optimizes a new ‘index of condensation’. In Statistical Computation, R. C. Milton and J. A. Nelder, eds. Academic Press, New York. Kruskal, J. 8. 1972. Linear transformation of multivariate data to reveal clustering. In Multidimensional Scaling: Theory and Application in the Behavioral Sciences, I, Theory, R. N. Shepard, A. K. Romney, and S. B. Nerlove, eds., pp. 179-191. Seminar Press, New York. Le Cun, Y. 1985. Une procedure d’apprentissage pour reseau B seuil assymetrique. In Cognitiva 85: A la Frontiere de I’lntelligence Artificielle des Sciences de la Connaissance des Neurosciences, pp. 599-604, Paris. (Paris 19851, CESTA. Le Cun, Y., Boser, B., Denker, J., Henderson, D., Howard, R., Hubbard, W., and Jackel, L. 1989. Backpropagation applied to handwritten zip code recognition. Neural Comp. 1, 541-551. Le Cun, Y., Denker, J., and Solla, S. 1990. Optimal brain damage. In Advances in Neural Information Processing Systems, Vol. 2, D. Touretzky, ed., pp. 598-605. Morgan Kaufmann, San Mateo, CA. Lincoln, W. P., and Skrzypek, J. 1990. Synergy of clustering multiple backpropagation networks. In Advances in Neural Information Processing Systems, Vol. 2, D. S. Touretzky and R. P. Lippmann, eds., pp. 650-657. Morgan Kaufmann, San Mateo, CA.
454
Nathan Intrator
Lippmann, R. P. 1989. Review of neural networks for speech recognition. Neural Comp. 1(1),1-38. Lyon, R. E 1982. A computational model of filtering, detection, and compression in the cochlea. In Proceedings l E E E International Conferenceon Acoustics, Speech, and Signal Processing, Paris, France. Moody, J. E. 1991. Note on generalization, regularization and architecture selection in nonlinear learning systems. In Neural Networks for Signal ProcessingProceedings of the 1991 IEEE Workshop, B. H. Juang, S. Y. Kung, and C. A. Kamm, eds., pp. 1-10. Mougeot, M., Azencott, R., and Angeniol, 8. 1991. Image compression with back propagation: Improvement of the visual restoration using different cost functions. Neural Networks 4, 467-476. Mozer, M. C., and Smolensky, P. 1989. Using relevance to reduce network size automatically. Connection Sci. Ul), 3-16. Nowlan, S. J. and Hinton, G. E. 1992. Simplifying neural networks by soft weight-sharing. Neural Comp. 4,473493. Pearlmutter, B. A., and Rosenfeld, R. 1991. Chaitin-Kolmogorov complexity and generalization in neural networks. In Advances in Neural lnfortnation Processing Systems, Vol. 3, R. P. Lippmann, J. E. Moody, and D. S. Touretzky, eds., pp. 925-931. Morgan Kaufmann, San Mateo, CA. Perrone, M. P.,and Cooper, L. N. 1992. Improving network performance: Using averaging to construct hybrid networks. Proceedings of the CAlP Conference, Rutgers University, October. Plaut, D. C., Nowlan, S. J., and Hinton, G. E. 1986. Experiments on learning by back-propagation. Tech. Rep. CMU-CS-86-126, Carnegie-Mellon University. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing, Vol. 1, D. E. Rumelhart and J. L. McClelland, eds., pp. 318-362. MIT Press, Cambridge, MA. Sebestyen, G. 1962. Decision Making Processes in Pattern Recognition. Macmillan, New York. Slaney, M. 1988. Lyon’s cochlear model. Tech. Rep., Apple Corporate Library, Cupertino, CA 95014. Switzer, P. 1970. Numerical classification. In Geostatistics, V.Bamett, ed. Plenum Press, New York. Tajchman, G. N., and Intrator, N. 1992. Phonetic classification of TIMIT segments preprocessed with Lyon’s cochlear model using a supervised/unsupervised hybrid neural network. In Proceedings International Conference on Spoken Language Processing, Banff, Alberta, Canada. Turk, M., and Pentland, A. 1991. Eigenfaces for recognition. J. Cog. Neurosc. 3, 71-86. Wahba, G . 1990. Splines Models for Obsmational Data. Series in Applied Mathematics, Vol. 59. SIAM, Philadelphia. Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., and Lang, K. 1989. Phoneme recognition using time-delay neural networks. l E E E Transact.ASSP 37,328339. Weigend, A. S., Rumelhart, D. E., and Huberman, 8. A. 1991. Generalization
Combining EPP and PPR
455
by weight-elimination with application to forecasting. In Advances in Neural lnforrnation Processing Systems, Vol. 3, R. I? Lippmann, J. E. Moody, and D. S. Touretzky, eds., pp. 875-882. Morgan Kaufmann, San Mateo, CA. Werbos, P. 1974. Beyond regression: New tools for prediction and analysis in the behavioral sciences. Ph.D. dissertation, Harvard University. White, H. 1990. Connectionists nonparametric regression: Multilayer feedforward networks can learn arbitrary mappings. Neural Networks 3, 535-549. Yamac, M. 1969. Can we do better by combining 'supervised' and 'nonsupervised' machine learning for pattern analysis. Ph.D. dissertation, Brown University. Received 26 June1992; accepted 26 October 1992.
This article has been cited by: 2. Shimon Edelman, Sharon Duvdevani-Bar. 1997. Similarity, Connectionism, and the Problem of Representation in VisionSimilarity, Connectionism, and the Problem of Representation in Vision. Neural Computation 9:4, 701-720. [Abstract] [PDF] [PDF Plus] 3. Tin-Yau Kwok, Dit-Yan Yeung. 1997. Constructive algorithms for structure learning in feedforward neural networks for regression problems. IEEE Transactions on Neural Networks 8:3, 630-645. [CrossRef] 4. David J. Field . 1994. What Is the Goal of Sensory Coding?What Is the Goal of Sensory Coding?. Neural Computation 6:4, 559-601. [Abstract] [PDF] [PDF Plus]
Communicated by Steven J. Nowlan
A Simplified Gradient Algorithm for IIR Synapse Multilayer Perceptrons Andrew D. Back Ah Chung Tsoi Department of Electrical Engineering, University of Queensland, St. Lucia 4072,Australia
A network architecture with a global feedforward local recurrent construction was presented recently as a new means of modeling nonlinear dynamic time series (Back and Tsoi 1991a). The training rule used was based on minimizing the least mean square (LMS)error and performed well, although the amount of memory required for large networks may become significant if a large number of feedback connections are used. In this note, a modified training algorithm based on a technique for linear filters is presented, simplifying the gradient calculations significantly. The memory requirements are reduced from O[n,(n, + nb)N,] to 0 [ ( 2 n , + nb)N,], where n, is the number of feedback delays, and N, is the total number of synapses. The new algorithm reduces the number of multiply-adds needed to train each synapse by n, at each time step. Simulations indicate that the algorithm has almost identical performance to the previous one.
1 Introduction
A solution to the problem of modeling nonlinear dynamic systems was proposed recently with the introduction of a global feedforward local recurrent network architecture (Back and Tsoi 1991a). This network is based on a multilayer perceptron structure, with synapses that have an infinite impulse response (IIR). Simulations have shown that the network is capable of better performance than a network with only finite impulse response (FIR) synapses. For large networks of this class, a significant amount of memory storage may be required for learning, due to the filtering required for the gradient term of each weight. The memory requirement for the network is O[n,(n, + #b)Ns] where n, is the number of feedback delays, nb is the number of feedforward delays (it is assumed that the order of every synapse is the same), and N, is the total number of synapses. Neurul Computation 5,456462 (1993) @ 1993 Massachusetts Institute of Technology
Algorithm for IIR Synapse Multilayer Perceptrons
457
One method of overcoming the memory and computational requirements of the network was presented by Back and Tsoi (1991b). In this case, a reduced complexity network was proposed that simplified the structure of the network for particular classes of problems. In this note, a modified gradient algorithm is presented that simplifies the learning algorithm, while retaining the same network architecture. The algorithm is presented in Section 2. Simulation results are given in Section 3, which indicate the performance of the algorithm is almost identical to the training rule presented in Back and Tsoi (1991a). 2 A Simplified Learning Algorithm
A description of the multilayer perceptron with global feedforward local recurrent structure is presented again here for convenience. Let the network have L 1 layers (I = 0,1,. . . ,L). Each layer has Nl neurons with outputs zf(t) (i = 1 , 2 , .. . ,N,)where I = 0 is the input layer and I = L is the output layer. An MLP with IIR synapses is defined by
+
(2.1)
$(t)
=
c
yf.#)
(2.2)
i=l
(2.3) where (2.4) (2.5) (2.6) and k = 1 , 2 , . . . ,NI where NI+Idenotes the output layer, q-Jx(t) = x ( t -j), n, and nb are, respectively, the delayed feedback and feedforward inputs to a neuron, and zk, is the bias. The polynomials B i k ( q - l ) and Aik(q-') are relatively prime with n, 2 nb. In the previous learning algorithm, the weights are adjusted by minimizing a performance criterion defined by (2.7)
Andrew D. Back and Ah Chung Tsoi
458
where = Y k ( 0 - zkL(t)
(2.8)
and yk(t) is the desired output at time t. The weights are adjusted according to (2.9) (2.10) where
Consider the calculation of the sensitivity components ai;,( t)/&!kj(t), and ai&(t)/ab$(t)from 2.9 and 2.10 (Back and Tsoi 1991a). We have, (2.12) (2.13) where the synaptic output y(t) is given by 2.3. In the linear adaptive filtering context, Hsia (1981)noted that a simplified version of this form of gradient filtering operation can be obtained by carrying out the filtering (via the 1lA(9-I) autoregressivefilter), and then delaying the output. This led to a reduced number of operations. In the current context, it is possible to employ the same technique in gradient computations. This results in only two filtering operations per synapse, as opposed to O(n,+ nb). This can be reduced further by performing the forward filtering operation B(q-')/A(q-') in two stages. This results in only one gradient filtering operation per synapse (for each learning step). Rewriting 2.3, we have (2.14)
Algorithm for IIR Synapse Multilayer Perceptrons
459
Hence the new values for 8i$(t)/a&(t), and d;6ikj(t)/ab&(t) can be found by delaying the terms in 2.14 and 2.15. Thus, (2.16)
= 9-j4(t)
(2.17)
Note that the filtering performed in 2.16 is done only for j = 0. The difference between the new regressor equations 2.16,2.17 and the original formulation in 2.12,2.13, is that in the simplified case, separate filters are not maintained for each regressor. Filtering is performed only at the output of the synapse (corresponding to j = 01, and then that filtered regressor is subjected to a delay (9-9 to obtain aiikj(t)/au:kj(t) for j = 1,2,.. . ,n,. Note that vik(t)= ~i-'(f)/A:~(9-~)is obtained in the forward pass. Equations 2.9, 2.10, 2.11, 2.16, and 2.17 form the new weight update equations. Previously, the memory requirements to train a synapse were o[n,(n, n b ) ] . With the new algorithm, this is reduced to O[(n, n b ) ] + O(n,) = 0[(2n, nb)]. The memory requirement for the network is 0[(2n, + nb)Ns]. The number of multiply-adds required during learning for each synaptic weight is reduced by n, operations at each time step. In the next section, simulation results are discussed and conclusions are made concerning the simplified gradient algorithm.
+
+
+
3 Discussion
To assess the performance of the new algorithm, simulations were carried out on a nonlinear system identification task. In the example considered here, the nonlinear system to be modeled is described by y(t) = sin
+
0.0154 O.O462q-' + 0.0462q-2 + 0.0154~j-~ ~ ( t ) ] (3.1) 1 - 1.999-l + 1.572q-' - 0.45839-3
where x( t) is a white noise input. An IIR MLP of 2 layers, 10 hidden units, and synaptic orders of (n,,n b ) = (7,6) (hidden layer) and (no,n b ) = (0,O) (output layer) was used to model the system. The network was trained using identical learning rates and initial weights for each algorithm. After convergence, each system was tested with a white gaussian input. It was observed that the mean square error in each case was very close to one another. The simplified gradient algorithm gave slightly better results,
460
Andrew D. Back and Ah Chung Tsoi
Figure 1: Mean square error performance of the initial IIR MLP algorithm and the simplified version. with an mse of 0.0013, as compared to 0.0018 for the previous algorithm (averaged over 200 data points after 250 x 103 iterations). The mean square error performance of each algorithm is shown in Figure 1, where it is evident that the algorithms have very similar learning characteristics. It is noted in Hsia (1981)that the transient performance of the algorithms are different due to the difference in the A(q-') weights between time t and t - 1. This is observed in the IIR MLP also as indicated in Figure 2. The final values of the corresponding weights learned by each algorithm are similar, but not identical. This is expected considering the differences in 2.12 and 2.13 compared with 2.16 and 2.17. A more important aspect than the differences in weights are the final pole-zero positions of the synapses due to the simplified algorithm. To ensure the dynamic performance of each corresponding synapse is the same when using the simplified algorithm in place of the initial version of the algorithm, the pole-zero positions should coincide in each case for the respective synapses. In the experiments performed, it is observed that this is indeed the case.
Algorithm for IIR Synapse Multilayer Perceptrons
461
. . . . Init. Alg.
.
8oooO
1OOOOO
3
E
0
2oooO
4-0
6oooO Samples
Figure 2 Parameter convergence of the initial IIR MLP algorithm and the simplified version. In conclusion, this technique offers a substantial savings on memory and computation during learning yet gives equivalent performance.
Acknowledgments The first author acknowledges support through a Research Fellowship with the Electronics Research Laboratory, DSTO, Australia. The second author acknowledges partial support from the Australian Research Council.
References Back, A. D., and Tsoi, A. C. 1991a. FIR and IIR synapses, a new neural network architecture for time series modeling. Neural Comp. 3(3), 375-385. Back, A. D., and Tsoi, A. C. 1991b. Analysis of hidden layer weights in a dynamic locally recurrent network. Artificial Neural Networks, T. Kohonen,
462
Andrew D. Back and Ah Chung Tsoi
K. Makisara, D. Simula, and J. Kangas (eds.), pp. 961-966. Elsevier Science Publishers B.V., North-Holland. Hsia, T. C. 1981. A simplified adaptive recursive filter design. Proc. IEEE 69(9), 1153-1155. White, S.A. 1975. An adaptive recursive digital filter. Proc. Ninth Asilomar Conf. Circuits, Systems and Computers, Pacific Grove, CA, pp. 21-25. Received 29 October 1991; accepted 19 August 1992.
This article has been cited by: 2. Stefan C. Kremer . 2001. Spatiotemporal Connectionist Networks: A Taxonomy and ReviewSpatiotemporal Connectionist Networks: A Taxonomy and Review. Neural Computation 13:2, 249-306. [Abstract] [PDF] [PDF Plus] 3. P. Campolucci, F. Piazza. 2000. Intrinsic stability-control method for recursive filters and neural networks. IEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing 47:8, 797-802. [CrossRef] 4. P. Campolucci, A. Uncini, F. Piazza, B.D. Rao. 1999. On-line learning algorithms for locally recurrent neural networks. IEEE Transactions on Neural Networks 10:2, 253-271. [CrossRef] 5. B. Cannas, S. Cincotti, A. Fanni, M. Marchesi, F. Pilo, M. Usai. 1998. Performance analysis of locally recurrent neural networks. COMPEL: The International Journal for Computation and Mathematics in Electrical and Electronic Engineering 17:6, 708-716. [CrossRef]
Communicated by Shun-ichi Am&
The Characteristics of the Convergence Time of Associative Neural Networks Toshiaki Tanaka Miki Yamada Advanced Research Laboratory, Toshiba R b D Center, 1 Komukai Toshiba-cho, Saiwai-ku, Kawasaki 210, Japan
The authors have analyzed the dynamics of associative neural networks based on macroscopic state equations and have shown that both a layered associative net and an autocorrelation type net have the same convergence property: If a recalling process succeeds, the network converges very fast to one of the memorized patterns. But if a recalling process fails, it converges very slowly to a spurious state or does not converge. This property was also checked by computer simulations on a large scale (N = 1000) neural network. Moreover, it is shown that the convergence time for a successful recall is of order log(N). If this convergence time difference is used, execution time and memory can be saved and it can be determined whether a recalling process succeeds or fails without any additional procedure.
1 Introduction The associative memory was proposed by Kohonen and Nakano independently (Kohonen 1972; Nakano 1972). Its dynamic behaviors have been well studied by many researchers (Amari 1972; Amari and Maginu 1988; Amit 1989; Cottrell 1988; Domany et al. 1989; Hopfield 1982; Meir and Domany 1987). The recalling process of associative networks is divided into two groups, successful recall and failed recall. If the network converges to one of the memorized patterns or near them, this process is called successful recall. If the network is trapped to a spurious state as an equilibrium or does not converge to a fixed point, it is called failed recall. From the information processing point of view, the distinction between successful recall and failed recall is required, especially in case of hierarchical associative networks because the generated meaningless output of the first layer becomes the input pattern of the second layer and meaningless activations spread. The distinction between successful recall and failed recall is equivalent to the distinction between a memorized pattern and a spurious state Neural Computation
5,463-472 (1993) @ 1993 Massachusetts Institute of Technology
Toshiaki Tanaka and Miki Yamada
464
as an equilibrium. So, it is an interesting problem on the dynamics of associative networks. Several ideas for such a distinction have been suggested. Amit (1989) plotted the average final overlaps and average retrieval times vs. the distance of stimulus from a memorized pattern. He discussed the relation between the basins of attraction and retrieval times. Parisi (1986) proposed asymmetric neural nets, in which only the time-independent outputs can be considered as meaningful outputs. Gutfreund (1988) introduced these ideas and discussed the importance of the problem of a hierarchical associative network. The convergence time is estimated in case of successful recalls (Komliis and Paturi 1988) and in worst cases (Florben 1991). But this problem has not been well analyzed. This paper deals with this problem on both a layered associative net and an autocorrelation type net. Macroscopic state equations that describe the dynamic behavior of these two types of nets have been obtained by Meir and Domany (1987) and Amari and Maginu (19881,respectively. Based on the former equation, the authors have analyzed the convergence time of the associative network in detail and have shown some characteristics on the convergence time are useful to make distinction beween successful recall and failed recall. 2 Macroscopic State Equations of Layered Associative Net
A layered associative net and its macroscopic state equation will be introduced in this section. The state Sf of a neuron i on a layer I (i = 1,.. . ,N)(I = 1,.. . ,L) takes binary values f l where N is the number of neurons in each layer and L is the number of layers. The state Sff' is determined by the following law
s!+'
=
sgn(hff')
(2.1)
where sgn(x) =
1, x 2 0 -1, x < o
Here, J:j is the connection weight value from neuron j on layer I to neuron i on layer I 1. And hi+' is the total input to neuron i on layer 1 + 1. The connection weight is defined as follows:
+
]k
where = (&)(v = 1,. . . ,K) is an N-dimensional memory vector defined in each layer 1. Each component in a memory vector takes binary
<:
Convergence Time of Associative Neural Networks
465
values f l randomly with equal probability 1/2. So, memory vectors are orthogonal in a stochastic sense. Let m1 be the average direction cosine between the state S,! and the memory pattern t,!,on layer I defined by
The dynamics of m1 can be obtained (Meir and Domany 1987; Domany et al. 1989) as follows: (2.4) 2
(A'+l)' = a + - e~p[-(m'/A')~] lr
(2.5)
where
(A')'
= a, =
KIN
AIL0
(K,N-, +OO)
Equations 2.4 and 2.5 are called the macroscopic state equations. The main results obtained by Meir & Domany will be listed (Meir and Domany 1987). 0
0
0
0
Equations 2.4 and 2.5 have three fixed points (two are stable and one is unstable) if a < ac(= 0.27) and one stable fixed point if a > Qc.
mE(a)exists so that mr converges to m* 2 1 if m' > mt(a)is satisfied and that m' converges to m' = 0 otherwise. rnE(a)is a monotonically increasing function of a(< a,) and tends to infinity if a > a,. For a << 1, the upper stable fixed point rn' has the form
This equation is an exact solution of a layered associative net. But it does not hold in autocorrelation type net. The approximated macroscopic state equations in this type of net is obtained by Amari and Maginu (1988).
Toshiaki Tanaka and Miki Yamada
466
Y
y=x
Figure 1: Schematicgraph of f ( x ) ment of the network dynamics.
(Q
< QJ. The arrows show the time develop-
3 Analysis of the Macroscopic State Equations
The macroscopic state equations 2.4 and 2.5 will be analyzed in detail in this section. Let HI+'be d+l
=
mffl/A/fl
Equations 2.4 and 2.5 are rewritten using this new variable to the following simple equation: d+l=f($)
(3.1)
where
From equation 3.2,f(x) ( x 2 0) is an S-shape function satisfyingf(0) = 0 and f(+w) = l/@. Schematic shapes of the function f ( x ) are shown in Figure 1 for a < at. The function f(x) has three fixed points 0, H:, H: for a: < ac(G0.27). It is clear that {$} is a monotonic decreasing (or increasing) sequence if 0 < & < M: or H: < id hold (or iii,' < i?i' < Hj).This property directly leads the monotonicity of the sequence {m'} for I2 2.
Convergence Time of Associative Neural Networks
467
2oo
150
0.20
L,(ml; a)100
0.25
50
0 .2
0
.6
.4
.8
1
m1
Figure 2: Numerical simulation results of L,(rnl;a)in using macroscopic state equation. Each value in the graph indicates the storage level a (e = 0.001). Let L,(m';a) be the minimum layer number at which the difference between m' and an equilibrium m* is less than E:
L,(ml;a) = min(l1
I m' - m* I<
E}
The sequence {m'} is monotonic and two different sequences {m'}, {m"} do not change the sign of m' - m" if these sequences converge to the same fixed point. So L,(ml; a) is a monotonically increasing function if 0 < m' < rn, or rn, < m1 and is a monotonically decreasing one if rn, < m' < m, where 0, m,, m, are the fixed points of equations 2.4 and 2.5 and satisfy rn, = Ei,,/Z, m, = Tiis&', and m' = E i ' f i . The convergence times L,(ml; a) are shown in Figure 2 for some a's (E = 0.001),which are obtained by calculating the macroscopic state equations 2.4 and 2.5 until it converges to an equilibrium. These numerical simulations show the monotonicity of LJm'; a) obtained. The increasing curve L,(m';a) for m1 > rn, cannot be seen when a << 1 because rn, N 1. Next, we estimate Lc(ml;a) quantitatively. When Eil < Ei,, Ei' converges to zero. Since f(B') is convex in the range of (0 Eic), the linear approximation of f(Ei') around Ei' = 0 gives a good approximation for Lc(rn';a). In general, the convergence time step T, at which a difference equation x,,+~ = px,, q (0 < p < 1) satisfies I x,, - x, I< E is estimated by
+
T -
log€
log I x1 - x*
logp
log P
I
(3.3)
Toshiaki Tanaka and Miki Yamada
468
where XI is the initial value and X. = q/(1 - p) is an equilibrium of the equation. When p > 1, the time needed to go away a distance E from the equilibrium x* is given by -Tc. Using equation 3.3, the convergence time step for Tii' is given by log€ log%? T,=--loga
where a =f'(O)
(3.4)
loga So L,(m';a) is estimated by
= 1/,-.
-
log[l + 2 / m ] 2 log u
log m1 loga
log E loga
(3.5)
4-
where E is replaced by E / , , / m since A' converges to when Ti7' becomes zero. For example, when a = 0 . 1 , ~= 0.001, and m1 = 0.1, the approximated convergence time Le(m';a)= 77. This approximation shows a good agreement with numerical simulation results of Figure 2. When Tii' is near Tii,, 5' converges to Tii, and the convergence time
T,
log E log b
= --
log I Tii' - Tii, log b
I
(3.6)
is obtained in the same manner where b = f'(Tii,). Putting ifi, N I/@', Tii' = m l / f i and replacing E by c/,/E, we get L,(m';a) =
log€ log b
--
log(1 - m') log b
(3.7)
and 1 =
-/,
1 +
+
a [ ( T a / ~ / a11
Hereafter, the recalling process will be called successful recall when the network converges to a fixed point m, and failed recall when the network converges to zero. The convergence time difference between successful recall (equation 3.7) and failed recall (equation 3.5) mainly depends on coefficients u and b. Numerical calculations show logallogb x 10 if a < 0.2. So the convergence times of successful recall are much smaller than those of failed recall. For example, when m1 = 0.9, a = 0.1, and E = 0.001, we obtain b = 0.02 and L,(m';a) = 1. When Tii' is near Mc, FI' converges to Tii, or zero. The convergence time Lc(ml;a ) is estimated by the sum of the time needed to move away from Tiic and the time needed to converge to the equilibrium from near
Convergence Time of Associative Neural Networks
469
it. The former time takes large values like equation 3.5 but the region where Lc(m';a ) is large at around 5' > E, is very small. So the "spike phenomena" is observed at around m' = m, shown in Figure 2. The convergence time L,(m'; a ) depends on the accuracy E . The problem is how to choose E to estimate the convergence time of finite size neural networks. The direction cosine m rtakes N + 1 discrete values 1, ( N - 2 ) / N , (N- 4)/N,.. . , -1 when N is finite. So it is sufficient to put E = 1/N. Then, it is easy to see that L,(m';a) is of order log(N) from equations 3.5 and 3.7. Numerical simulations of L,( m'; a ) based on the approximated macroscopic state equations obtained by Amari and Maginu are very similar to those of the layered associative net except for the critical value of a,. In this case, a, is about 0.16 (Amari and Maginu 1988). 4 Simulation Results of Autocorrelation Type Net
In the previous section, it is proved that the convergence time of a failed recall is much larger than that of a successful recall and that the convergence time is of order lo#) where N is the number of neurons of a layered associative net. In this section, it is examined by neural network simulation that these properties still hold even for an autocorrelation type of net, which is the discrete synchronous Hopfield net with Jii = 0 (i = 1,. . . ,N). Figure 3a and b shows the results of convergence times for a = O.OB,O.lO, respectively. The network output is synchronously updated until it converges to a fixed pattern. This criterion of convergence is different from that of the theoretical convergence time discussed. But there is little difference between them. In the figures, 0 indicates a successful recall (m*> 0.9) and + indicates a failed recall. We limit the iteration time to 100 and do not include cases in which the convergence time exceeds this upper limit. We often observe such cases for failed recall of m 1 < 0.4. In other words, the network always converges within an iteration time 100 for successful recall. From these simulation results, the convergence time distinction between successful recall and failed recall holds for this type of nets. And this tendency becomes clearer as a increases. It is concluded that in correlation-based associative networks the convergence time of failed recall is larger than that of successful recall. Figure 4 shows the average convergence times vs. the network sizes (the number of neurons) for a = 0.08. In the figure, 0, +, and 0 are the average convergence times over 100 successful recalls for 0.2 < m 1 < 0.3, 0.4 < m' < 0.5, and 0.8 < m1 < 0.9, respectively. The convergence time is shown to be of order log(N) for large values of m'. But for small values of m', it is not clear that the order estimate still holds. A more precise analysis and simulation for larger size networks are needed.
470
Toshiaki Tanaka and Miki Yamada
Figure 3: (a,b) Computer simulation of convergence time using neural network (N= 1OOO). 0 indicates a successful recall and + a failed recall. The horizontal bar is the initial direction cosine M I , and the vertical bar is the convergence time (iteration time). It is clear that the convergence time of successful recall is smaller than that of failed recall. 5 Discussion
An autocorrelation type associative net has two problems from the information processing point of view. One is that the convergence time depends deeply on the initial values. In some cases they need more than 10 times as much as those of successful recall. The other is that an automatic distinction between successful recall and failed recall needs to store each memorized pattern separately in order to calculate the direction co-
Convergence T i e of Associative Neural Networks
20
471
I
0.2-0.3 0 15
-
0.4-0.5
+
0
0.8-0.9 0
0
T
10
-
-
0
0
0
OO
-
0
0
OO
00
0 0
+++++
5 0 11
0
+0
+
0
+0 +oooo@
+
+ + + + o
0
0
-
on@
I
Figure 4: Average convergence time vs. network size. 0, +, and 0 are the average convergence times over 100 successful recalls for 0.2< m' < 0.3,0.4 < m' > 0.5, and 0.8 < m1 < 0.9,respectively. The convergence time is shown to be of order log(N)for large values of m'. sine between each memorized pattern and an obtained output pattern. Roughly speaking, the first problem is on the execution time, and the second one is on memory. These time and memory problems can be solved using the convergence time property that we have shown. If a network does not converge within a time limit, you can stop the execution of the network and ignore its output. This is because the network is shown to converge to a spurious state when the convergence time is larger than the time limit. On the other hand, if a network converges within the time limit, you can regard its process as successful recall and its output as a meaningful pattern. Acknowledgments We thank Professor S. Yoshizawa of Tokyo University for useful discussions and comments. References Amari, S. 1972. Learning patterns and pattern sequences by self-organizing nets of threshold elements. IEEE Transact. Computers C-21,1197-1206.
472
Toshiaki Tanaka and Miki Yamada
Amari, S., and Maginu, K. 1988. Statistical neurodynamics of associative memory. Neural Networks 1,63-73. Amit, D. J. 1989. Modeling Brain Function-The World of Attractor Neural Networks. Cambridge University Press, Cambridge. Cottrell, M. 1988. Stability and attractivity in associative memory networks. Biolog. Cybernet. 58, 129-139. Domany, E., Kinzel, W.,and Meir, R. 1989. Layered neural networks. J. Phys. A: Math. Gen. 22, 2081-2102. Florkn, P 1991. Worst-case convergence time for Hopfield memories. IEEE Transact. Neural Networks 2(5), 533-535. Gutfreund, H. 1988. Neural networks with hierarchically correlated patterns. Phys. Rev. A37,570-577. Hopfield, J. J. 1982. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. U.S.A. 79, 2554-2558. Kohonen, T. 1972. Correlation matrix memories. IEEE Transact. Computers C-21, 353-359. Koml6s, J., and Paturi, R. 1988. Convergence results in an associative memory model. Neural Networks 1,239-250. Meir, R., and Domany, E. 1987. Exact solution of a layered neural network model. Phys. Rev.Lett. 59,359-362. Nakano, K. 1972. Associatron-A model of associative memory. IEEE Transact. Syst., Man. Cybern. SMC-2, 381-388. Parisi, G. 1986. Asymmetric neural networks and the process of learning. J. Phys. A: Math. Gen. 19, L675L680. Received 28 May 1992; accepted 11 November 1992.
This article has been cited by: 2. Zhang Chengxiang , Chandan Dasgupta , Manoranjan P. Singh . 2000. Retrieval Properties of a Hopfield Model with Random Asymmetric InteractionsRetrieval Properties of a Hopfield Model with Random Asymmetric Interactions. Neural Computation 12:4, 865-880. [Abstract] [PDF] [PDF Plus]
Communicated by Gerald Tesauro
Robustness in Multilayer Perceptrons P. Kerlirzin F. Vallet Laborutoire Central de Recherches, Thornson-CSF 92404 Orsuy fcedex),France
In this paper, we study the robustness of multilayer networks versus the destruction of neurons. We show that the classical backpropagation algorithm does not lead to optimal robustness and we propose a modified algorithm that improves this capability.
1 Introduction
The distributed nature of information in neural networks suggests that they might be capable of "robust" computation, that is, a "graceful degradation" of performance with respect to damaged units and connections (Hinton et al. 1986;Le Cun 1987;DARPA 1988).In fact, such robustness is often observed empirically, but is usually uncontrolled. In this paper! we propose a method for optimizing during learning the ability of a network to compute after damage. Robustness to the destruction of neurons can be an interesting property for hardware implementation. One can note also that the problem of missing data corresponds to the destruction of input cells of the network. The degradation of a network due to a loss of precision of the synaptic weights ("clipping") has been studied (Hopfield 1982; Amit et al. 1985; Cottrell et al. 1987;Wong and Sherrington 1989)for associative memories (moderated drop of the capacity) and (Vallet and Cailton 1990)for linear classifiers. Here we study the case of the destruction of hidden and input neurons in a one-hidden-layer perceptron. The destruction of synapses can be tackled with similar solutions. The generalization to the case of a perceptron with more than one hidden layer is easy. Previously we have partially studied the case of linear networks (Kerlirzin 1990).We showed that the robustness optimizing algorithm behaves fairly, confirming theoretical results, and that it optimizes the robustness by distributing information on all the connections of the network. We now address the case of nonlinear networks. Neural Computation 5,473482 (1993) @ 1993 Massachusetts Institute of Technology
P. Kerlirzin and F. Vallet
474
2 Background
The network considered here performs a transformation F from R”to R”, through a hidden layer with p neurons (the output is linear for sake of clarity): P
X + F(X) = z~(’wl,X)W’, =Y i=l
where X is the input vector, Y the output one, o the neural transfer function [tanh(x) for our example), W, the input weight vector linking the input vector to the ith hidden cell, and Wi the output weight vector linking the ith hidden cell to the output vector (Fig. 1). Such a network tries to learn a desired function D (known on pattern examples: the learning set) with as few errors as possible. To achieve this, a cost function E, measuring the distance between the desired function D and the function F performed by the network (on the learning set) is minimized: N
E = C 11 D ( X ” ) - F(X”) [I2 ’”=I
the learning set being composed of N examples Xp. A method widely used to minimize this energy is a stochastic gradient descent procedure (Le Cun 1987; Rumelhart etal. 1986).Each convergence step (elementary learning) consists in modifying the weights W, and Wi in a direction opposite to the gradient, for the elementary contribution E(Xp) of the pattern Xfi to the total cost function E:
E ( X p ) =)I D(Xp) - F(X’”)[I2 We are interested here in partially damaged versions of the network, which implement the functions Fn: for which only a subset K of hidden cells is active:
F K ( X ) = z~(’wl,X)W’, iEE
K being a subset of {1,2,. . . p } , which represents the hidden neurons that are not destroyed. The measure of distance between the desired function D and the damaged one FK: is then N
[I D(Xp)- FK(X’”)It2
EK = p=l
The final goal is to minimize the average cost function, taking into account the probability P ( K ) of each configuration K occurring:
Robustness in Multilayer Perceptrons
input layer
475
hidden layer
output layer
Figure 1: Representation of the contribution of the ith hidden cells. We study here the simple and general example in which each cell has the same probability 7r to be damaged. The probability P ( K ) is thus given by
P ( K ) = (1 - 7r)lK17r(P-lKl)
(2.1)
1x1being the cardinal of K. It is now convenient to minimize the cost function N Ewer =
11 D ( x / ” -) F K ( X / ” )1)’
P(K)
(2.2)
K p=l
In order to minimize it, a stochastic gradient algorithm is used. It consists here in choosing one configuration K [randomly chosen according to the distribution given by P ( K ) l and one example X” (randomly chosen or chosen in a predefined order). A partial cost is defined for this configuration and this example:
Then the weights are modified in the opposite direction to the gradient direction of the elementary contribution E ( K , Xfi). One can note that the Probability P ( K ) is now implicitly taken into account by the distribution of the choice of the configuration K. The proposed algorithm is thus a doubly stochastic algorithm, that is, on examples and on damaged configurations.
476
P. Kerlirzin and F. Vallet
3 Simulations and Results
The efficiency of the proposed learning algorithm is studied on a realworld problem: the identification of radio-transmitters characterized by relevant features. This a (supervised) classification problem with 9 classes and vectors (patterns) of dimension 23. The learning set contains 810 examples and the test set (for evaluating the generalization rates) contains 400 examples. The network used is totally connected with 3 layers (23 x 16 x 9). This is the one-hidden-layer network that provides the best generalization rate (86.9%), whereas a network without hidden layer provides a generalization rate of 79%. We present here the generalization results (the only interesting ones in practice), the learning results have a similar behavior with higher values. We compare the results between the classical network that has learned without destruction of cells (called "classical") and the "optimized" network for which learning was made by a stochastic gradient with cost function (2.2). The probability of destruction of neurons during the learning phase is 10% (so P ( K ) = 0.91n10.1'6-lKI = 91K1/10'6). The robustness of each network when d hidden cells (d = 0,1,. . .) are destroyed is then measured. For each case we present two generalization rates: the average rate, which is calculated for a given d in taking into account all the possible combinations of d destroyed hidden cells, and the worst rate, which is the smallest rate found in testing all the configurations with d destroyed hidden cells. The proposed algorithm has been studied first with randomly initialized weights. In that case, the proposed algorithm is not efficient and the classical one seems to be robust. Another strategy was studied to obtain an optimized network that is efficient even without destroyed cells. This strategy consists in learning in two steps. The network is first trained with the classical backpropagation algorithm and, second, the robustness optimizing algorithm is used to continue its training. It has been observed that the supplementary learning time is of the same order as the fore-learning time. The obtained results are summarized in Table 1. The conclusions are now clear. First, the classical backpropagation algorithm is not optimal with respect to the destruction of neurons. The proposed algorithm hardly degrades the results from the classical one without destruction. When one considers the destruction of neurons, the worst and average cases are better for the optimized network. Further, given a number of killed cells, the optimized network exhibits a smaller performance standard (the standard deviation of the performance of the network for each possible configuration). This general solution can also be used in the case of a network with several hidden layers: learning is still made with random destruction
Robustness in Multilayer Perceptrons
477
Table 1: Comparison of Classical and Optimized Networks.
Worst case (%)
0 cell killed (0%) 1 cell killed (6%) 2 cells killed (12%) 3 cells killed (19%) 4 cells killed (25%) 8 cells killed (50%)
Average (%)
Classical
Optimized
Classical
Optimized
86.9 74.1 60.8 45.5
86.4 82.9 74.6 74.3 68.0 36.9
86.9 83.8 79.3 72.8 66.0 43.6
86.4 85.6 84.6 83.2 81.2 64.5
30.1 3.2
of neurons according to destruction laws, given for each layer. This approach should also be valid for the destruction of synaptic weights. It is interesting to study the effect of the mortality rate (probability of hidden neuron destruction during learning, A in equation 2.1) on the robustness of the network. The results are shown in Figure 2: average, worst cases and standard deviation of the generalization rates have been plotted versus the mortality rate A introduced during learning. These curves are shown for 6 different cases of the number d of destroyed cells in test phase: d = 0, 1, 2, 3, 4, and 8. It is interesting to notice that the recognition rates (worst and average cases) are roughly increasing functions of A and that their asymptotic values are reached for a value corresponding to d . Thus, if one wants to optimize a network for a destruction rate of A’, it is necessary to use a learning destruction rate A slightly greater than A’. The value of A need not be very precise, since we have shown that the generalization rate varies slowly with A. Another remarkable point is the decrease of the deviation of the results while A increases: the result with a given number d of destroyed hidden cells hardly depends on the choice of these cells. Finally, the proposed algorithm is able to solve the robustness problem under discussion. For d = 8 destroyed neurons during test, for example, the usual algorithm has an average test rate of 45% versus 75% for the proposed optimized version ( A = 50%). The previously proposed method can easily be applied to the case of the destruction of input cells. From an operational point of view, this is the problem of missing input data. Some data may arrive in an intermittent way or even completely disappear because of the failure of the upper processing step or of an out-of-order sensor. In this case, the aim is to have the most robust network with respect to this type of destruction. We have applied the learning algorithm described, in the case of input cells, to the problem previously described, and we show the results as previously. Figures 3 and 4 represent the generalization performances for the average and worst cases and the standard deviation for five values of the
I? Kerlirzin and F. Vallet
478
Averue nte on tat Ict
3 cells killed 2 cells killed
.....
I cell killed
- 0 cell killed
10
0
20
M
40
so
Word rate on teat r t
I,
. . I
10
.
..-. 8 cells kllled .--.4 calls kllled
-- .'*. 'b.
,
'. \.
.-.I.
.....
3 cells klUd 2 cells kllled I cell kUled
---..-.._.._..-.._. LO
m
M
40
Figure 2: Average and worst case and standard deviation of the generalization rate versus T probability of destruction of a hidden cell during learning. Six cases are represented, corresponding to a different number d of destroyed neurons during test: d = 0, 1, 2, 3,4, and 8.
479
Robustness in Multilayer Perceptrons
Average rate on test wt (%)
3 cells kllled 2 cells killed
- 0 cell killed 0
10
20
30
40
Worst rate on test set (96)
~
..
-
4 cells kllled (17%) 3 cells killed (t3%) 2 cells killed (9% ) t cell kllled (4%) 0 cell killed ( 0 % )
Learning mortality rate (%) 3 0 , 10
0
20
30
40
Standard deviation (%)
10.0
-
8.0
--'*
2.0
--
4 cells killed 3 cells kllled 2 cells kllled
-. '.
, Learning mortglity rate (%) ...
0.0 0
10
20
30
Figure 3 Missing input data. Average case, worst case, and standard deviation for the generalization rate versus s probability of destruction of an input cell during learning. Five cases are shown, corresponding to a different number d of destroyed neurons during test: d = 0, 1, 2, 3, and 4. Curves correspond to the randomly initialized network.
P. Kerlirzin and F. Vallet
480
Average rate on tesl set
90
..............................................................................................
86
82
...
--.
........... ............................................................................................ ...........*...... ................................ . .,...' ~
~
78 74
.,..-. -6
.,,,.
./
*.,'
,.*'
-*..
..*.
-.--
.__.4 cells kllled (17%)
....
--,/'
..........
3 cells kllled (13%) 2 cells killed ( 9 % )
-
I cell kllled (4%) 0 cell kllled ( 0 % )
.....
Learnlnp mortallty rate
70
-- n
p
Worst rate on test set .
I ............. ....... ...........................
1
........... .............................. ....... ................... ..-.'.". .................................................... , _...... ................. /,' _._._........ .. //.. ,,......... ..... ._-. 4 cells kllled ..... 3 cells kllled .... *........... 2 cells killed
8 O - t
70 60
so 40
~
~
~
~
--,.., '
.....
.a..
Learnlnp mortallty rate
3 0 , 0
2
--
10
20
........................................... .........
-
1 cell kllled 0 cell kllled
30
.............................................
.................... Learnlop mortallty rate
0
3
10
20
30
40
Figure 4 Same as Figure 3 with the fore-learning initialized network. number d of destroyed input cells during test. The figures on the left side correspond to the case of a randomly initialized network and those of the right side correspond to the case of an initialization by fore-learning. In opposition to the previous case, the fore-learning case does not seem to give notably better results than the randomly initialized case. For each d in {0,1,2,3,4,8},the value of the generalization rate for 7r = 10% of the
Robustness in Multilayer Perceptrons
481
randomly initialized case is higher than the corresponding value of the fore-learning case. 4 Conclusions
In this article, we have examined the problem of improving the robustness of a multilayer perceptron versus the destructions of hidden or input cells. We have shown that the classical backpropagation is not optimal when one considers robustness. We have proposed a learning algorithm that takes into account the potential destructions of cells (destructions during learning) and that was evaluated on a real example. The case of destroyed input cells seems different from the one of hidden cells. In the first case, a random initialization and a weak destruction of neurons during learning seem to give better results. In the second case a fore-learning initialization and a strongly damaged learning seem to be better. Our first results are very encouraging, but other experiments on real data have to be done, especially when the network size is large.
Acknowledgment The authors want to thank P. Gallinari for his support of this work.
References Amit, D. J., Gutfreund, H., and Sompolinsky,H. 1985. Storing infinite numbers of patterns in a spin-glass model of neural networks. Phys. Rev. Lett. 55(14), 1530-1 533.
Cottrell, D. M., Munro, P., and Zipser, D. 1987. Image compression by back-propagation: An example of extensional programming. In Advances in Cognitive Science, Vol. 3, N. E. Sharkey, ed. Ablex, Norwood, NJ. DARPA. 1988. Neural Network Study. AFCEA International Press, Fairfax. Hinton, G. E., McClelland, J. L., and Rumelhart, D. E. 1986. Distributed representations. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. I. Bradford Books, Cambridge, MA. Hopfield, J. J. 1982. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. U.S.A. 79, 2554-2558. Kerlirzin, P. 1990. Robustesse et capacitk des rkseaux multicouches. Rapport de stage (DEA Paris XI Orsay), LCR ASRF/90-8. Responsable: F. Vallet. Le Cun, Y. 1987. Modiles Connexionnistes de I'Apprentissage. Ph.D. thesis, Universite Pierre et Marie Curie, Paris, France. Rumelhart, D. E., McClelland, J. L., and the PDP research group. 1986. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. II. Bradford Books, Cambridge, MA.
482
I? Kerlirzin and F. Vallet
1990. Recognition rates of the Hebb rule for learning Vallet, F., and Cailton, J-G. boolean functions. Phys. Rev. A 41(6), 30593065, Wong, K. Y. M., and Sherrington, D. 1989. Theory of associative memory in randomly connected boolean neural networks. I. Phys. A 22, 2233-2263.
Received 28 October 1991; accepted 6 August 1992.
This article has been cited by: 2. Gaetano Perchiazzi, Rocco Giuliani, Loreta Ruggiero, Tommaso Fiore, G??ran Hedenstierna. 2003. Estimating Respiratory System Compliance During Mechanical Ventilation Using Artificial Neural Networks. Anesthesia & Analgesia 1143-1148. [CrossRef] 3. Yu G. Smetanin. 1998. Neural networks as systems for recognizing patterns. Journal of Mathematical Sciences 89:4, 1406-1457. [CrossRef] 4. Sepp Hochreiter, Jürgen Schmidhuber. 1997. Flat MinimaFlat Minima. Neural Computation 9:1, 1-42. [Abstract] [PDF] [PDF Plus] 5. Terrence L. FineFeedforward Neural Nets . [CrossRef]
Communicated by Richard Lippmann
Pattern Discrimination Using Feedforward Networks: A Benchmark Study of Scaling Behavior Thorsteinn Rognvaldsson Department of Theoretical Physics, University of Lund, Solvegatan 14 A, $223 62 Lund, Sweden
The discrimination powers of multilayer perceptron (MLP) and learning vector quantization (LVQ) networks are compared for overlapping gaussian distributions. It is shown, both analytically and with Monte Carlo studies, that the MLP network handles high-dimensional problems in a more efficient way than LVQ. This is mainly due to the sigmoidal form of the MLP transfer function, but also to the fact that the MLP uses hyperplanes more efficiently. Both algorithms are equally robust to limited training sets and the learning curves fall off like UM, where M is the training set size, which is compared to theoretical predictions from statistical estimates and Vapnik-Chervonenkis bounds. 1 Introduction
The task of discriminating between different classes of input patterns has proven to be well suited for artificial neural networks (A"). Standard methods, like making cuts or discriminant analysis, are repeatedly being outperformed by nonlinear ANN algorithms, where the most extensively used algorithms are the feedforward multilayer perceptron (MLP) (Rumelhart and McClelland 1986) and the learning vector quantization (LVQ) (Kohonen 1990). Both algorithms have shown good discrimination and generalization ability, although some confusion prevails on their performance on realistic large sized problems, especially concerning their parsimony in parameters to fit data-an important issue when algorithms are transferred to hardware. This paper compares, analytically and with Monte Carlo simulations, the discrimination power of the MLP and LVQ algorithms on separating two gaussian distributions. Two classes are sufficient since the results carry over to problems with more classes. The problem is designed to resemble "real-life" situations with many input nodes and overlapping distributions, making the classification fuzzy. Discrimination is thus only possible down to a minimum error; the Bayes limit (Duda and Hart 1973). It is found, in contrast to previous results (Kohonen et al. 1988; Barna and Kaski 1990), that the MLP is more efficient than the LVQ algorithm on heavily overlapping distributions. The sensitivities of the two algorithms Neural Computation 5,483-491 (1993) @ 1993 Massachusetts Institute of Technology
Thorsteinn Rognvaldsson
484
Table 1: Bayes Limit for the “Hard” and “Easy” Cases for Dimensions 2 5 d 5 8 Dimension
Bayes limit (%)
d
“Hard case ”Easy” case
2 3 4
5 6
7 8
26.4 21.4 17.6 14.8 12.4 10.6 9.0
16.4 13.8 11.6 9.8 8.4 7.2 6.2
to limited training data are also examined and compared to theoretical predictions. 2
T h e Problem
The problem (Kohonen et al. 1988) consists of two overlapping gaussian distributions, PI and P2r of dimensionality d and normalized to unity with standard deviations (TI = 1.0 and 0 2 = 2.0. Two versions are generated; one where the distributions have the same mean, referred to as the “hard” case, and one where their means are separated by a distance El referred to as the “easy” case (notation follows Kohonen et al. 1988):
I &I
Pl(r) = (qd%)-dexp --
(2.1)
(2.2)
<
where = 0 for the “hard” case, and = (2.32,0,0,.. . ,0) for the “easy” case. Bayes limit equals Jmin[P1,P2Irwhich is easily calculated for gaussian distributions (see Table 1). 3 Analytical Results
The optimal classification boundary between the two gaussian distributions is in both the “hard and “easy” cases the surface of a d-dimensional hypersphere, defined by Pl(r) = Pz(r). This is the surface the networks try to reproduce. The MLP uses ”hyperplanes” (see Fig. la) and is able to “cut corners” if a sigmoidal transfer function is used. The LVQ network divides the space by means of reference vectors, so-called “tessellation,”
Pattern Discrimination Using Feedforward Networks
485
Figure 1: (a) The MLP approximation to a sphere and (b)the LVQ tessellation for the same sphere (dots correspond to reference vectors). (c) The resulting polyhedron and quantities used in the text. and if the number of reference vectors is moderate the LVQ algorithm will result in one reference vector being inside the sphere and the others outside (Fig. lb). Both algorithms thus produce a . polyhedral-like reproduction of the sphere (Fig. lc) and the generalization error E can be expressed as
E x B + AVAP
(3.1)
where B is Bayes limit, AV is the deficit volume of the polyhedron as compared to the hypersphere, and AP is the average absolute difference between the two distributions inside that volume. This error can be estimated for the "hard" case, where the polyhedron is assumed to be spherically symmetric: If the number of planes bounding the polyhedron is N, it will consist of N conical sectors with opening angle a (see Fig. lc). If N >> d the end-cap of the cone is a (d - 1)dimensional hexagon, approximately a (d - 1)-dimensional hypersphere, and a will be given by
2 ) , r is the gamma function.' It is assumed where A d = 2 ~ ~ / ~ / I ? ( d /and in the last step that N is so large that a is small. 'Ad comes from the surface area A = Ad#-' radius r.
of a hypersphere of dimension d and
Thorsteinn Rognvaldsson
486
The volumes of the cone and the sector are given by (3.3) (3.4)
J2d
where R = ln[q/u1]u:u;/(u; - u:) is the radius of the optimal classification hypersphere. The deficit volume is approximated by expanding in power series
where equation 3.2 has been used in the last step. If AV is not too large, the density of patterns inside can be assumed constant. For u1 < a2 and AR << R one gets
where PR is the value of the distributions at the border. Inserting equations 3.5 and 3.6 into 3.1 gives
E - B % C(d)dN-4/(d-I)o< dN-4/(d-') B
(3.7)
where
is approximately constant for 4 < d < 10 (see Fig. 2). The true value of C(d) is not exactly given by equation 3.8. Equation 3.7 should instead be considered as a general scaling relation for heavily overlapping distributions of dimension d < 10, under the condition that N >> d and o can be considered small. For comparison with Monte Carlo simulations a value of C(d) = 0.5 was used, Deriving a similar expression for the "easy" case would be arduous; the polyhedron is not spherically symmetric and the simplifications above are not possible. The generalization error is, however, bounded from above by expression 3.7 If + 00 the distributions are perfectly separated by one hyperplane, with E equal to zero, and if ( -, 0 the "hard case is recovered. Hence, the "easy" problem never scales worse than the corresponding "hard problem.
Pattern Discrimination Using Feedforward Networks
2
3
4
5
6
7
8
487
9
1
10
d
Figure 2: The factor C ( d ) for 2 < d < 10 when 01 = 1.0 and a;!= 2.0. Each reference vector in an LVQ network with NLVQ computational units has at most NLVQ- 1 nearest neighbors. For moderate NLVQ,with a setup similar to Figure lb, the number of hyperplanes bounding the polyhedron is N = (NLVQ - 1). On the other hand, more than one reference vector will be inside the sphere if NLVQis large and the number of - 1). The LVQ hyperplanes bounding the polyhedron will be N < (NLVQ generalization error will thus scale like equation 3.7 for moderate NLVQ and worse for large NLVQ. Each hidden unit in an MLP with a Heaviside transfer function corresponds to one hyperplane. Hence, an MLP with NMLP hidden units has N = NMLP. A smooth sigmoidal transfer function allows the MLP to “cut corners” in the polyhedron, and the generalization error subsequently scales better than 3.7. How much better is difficult to predict, but analytical results (Sontag 1992) imply that a sigmoid allows a decrease of the hidden units with at least a factor of two, compared to a Heaviside. 4 Monte Car10 Studies
The dimension d of the gaussian distributions varied between 2 and 8, and 12 different architectures of MLP and LVQ networks were set up for each value of d. An ensemble of 100 networks was trained for each configurationto measure the worst, best, and average discrimination performances. The LVQ networks had d input units and NLVQ processing units, where NLVQ E {d,d+4,d+8,. . . ,d+4.11}. The weights were initialized with the on-line k-means clustering algorithm (MacQueen 1967) for a duration of 150 epochs, with randomly selected starting points. LVQ updating was
Thorsteinn Rognvaldsson
488
then applied for an additional 150 epochs, while the learning rate was lowered geometrically from 77 = 0.1 down to 77 = 0.001. One epoch corresponded to 1000 patterns. This choice of parameters and initialization was made to match those of Kohonen et al. (1988). The MLP networks had d input units and one hidden layer with NMLP hidden units, where NMLP E {d,d 2,d 4 , . . . , d 2 . 11). The initial weights were randomly picked from a flat distribution w E [-wo,WO], with wo = 0.1/(maximum “fan-in”)? On-line backpropagation with summed square error was used to train the networks (in the ”hard” case a Langevin form was used; Aw = -vVE+”noise,” to avoid local minima). The learning rates were scaled in inverse proportion to the “fan-in” of the layer, 7 o< l/“fan-in,” and dynamically changed with the “bold driver” method; increasing 77 if the network is improving and decreasing it otherwise. The momentum parameter was kept constant at cr = 0.5. The largest architectures were also trained with 7 different training lo4}, to estimate the sets, with sizes M E {lo, 103/2,lo2,lO5l2,lo3, learning curve. For each value of M the number of epochs was changed to keep the total number of presentations constant at 4 x lo5 for the MLP and 3 x lo5 for the LVQ networks. All simulations were performed with the network program package JETNET 2.0 (Lonnblad et al. 1992). Figure 3 shows the E(N)-behavior for d = 8, training on continuously generated data (figures for d < 8 are similar). Indicated for the “hard” case is the predicted scaling relation of equation 3.7 with C = 0.5. The LVQ errors follow the predicted behavior well, whereas the MLP errors scale better, as ex ected. For the “easy” case the LVQ error scales like (E - B ) / B oc dN-’ 2d. The worst case upper bound generalization error for this kind of binary classification problem is expected to be E I dvc/M, where dvc is the Vapnik-Chervonenkis dimension of the network (Cohn and Tesauro 1992). Figure 4 shows that this bound is indeed well above the actual learning curves if the number of weights is used as an approximate value of dvc. The learning curves are also well described by E c( 1/(M + Mo), predicted by statistical learning theories for problems with continuous generalization spectrum (Schwartz et al. 1990), and the algorithms are equally robust to limited training sets. An attempt was also made to estimate the prior generalization spectrum. The generalization ability of 10,000 untrained networks was tested and the learning curves numerically calculated from
+
+
+
P
where @ ) o is the Mth moment of the prior generalization distribution (Richard and Lippmann 1991). The resulting curves (dotted lines in %e “fan-in” of a unit is the number of units feeding to it.
Pattern Discrimination Using Feedforward Networks
489
Figure 3: Minimum, maximum and average values of the scaled generalization error for d = 8 using (a) MLP and (b) LVQ on the “hard task, and (c) MLP and (d) LVQ on the “easy” task. N is the number of hyperplanes bounding the polyhedron. For the MLP N equals the number of hidden units, whereas for the LVQ N equals the number of reference vectors minus one. The dashed line in (a) and (b) is the predicted (E - B ) / B 0: dN-4/(d-1),whereas in (c) and (d) it is a fitted (E - B ) / B 0: dN-=Iz. Fig. 4) do not follow the true learning curves. Statistical errors, however, are large already for logM x 1.5 and higher statistics were not pursued due to the CPU consumption involved.
5 Conclusions and Discussion The results demonstrate that the MLP architecture, with sigmoidal transfer functions, is superior to LVQ for discriminating between heavily overlapping distributions with convex borders. This is in contrast to previous results comparing LVQ and MLP architectures (Kohonen etal. 1988; Barna and Kaski 1990), but in agreement with results for Boltzmann (Kohonen et al. 1988) and mean field machines (Peterson and Hartman 1989). The MLP method is highly efficient and the more viable alternative for problems with many inputs. Furthermore, both algorithms are equally robust to limited training data and their learning curves follow a 1/M behavior.
490
Thorsteinn Rognvaldsson
Figure 4: The "hard task learning Curves for (a) MLP and (b) LVQ, and the "easy" task learning curves for (c) MLP and (d) LVQ. M is the number of training patterns used. The solid line is a fitted (E - B ) / B 0: 1/(M + constant) and the dashed line shows W / M where W is the number of weights. The dotted lines are the curves numerically calculated from the prior generalization spectrum. These results are also valid for binary problems where more than two classes of input patterns are present. An MLP with n output units can be used for an n-class problem. The output values will then correspond to the Bayesian a posteriori likelihoods that the input pattern belongs to the specific class (Richard and Lippmann 1991). A discrimination decision can therefore be made from these output values and the number of hidden units will in the worst case just be multiplied by n. The number ~ scale multiplied of units in the LVQ network will for moderate N L valso by n and the relative performances of MLP and LVQ will hence be approximately unchanged. LVQ may of course be more efficient than MLP for extreme problems with very large n >> d and where each class only needs one or two reference vectors. The reason the MLP performs well already for low Nhrl~pis clearly its sigmoidal transfer function, making it possible to smooth the corners of the polyhedron. Allowing direct input to output connections in the MLP would increase the discrimination power further (Sontag 1992; Peterson and Hartman 1989). The LVQ algorithm can be augmented with more advanced evalu-
Pattern Discrimination Using Feedforward Networks
491
ations of the output signals (Barna and Kaski 1990; Scabai et al. 1992) and thus be made to perform better. With such superstructures one can achieve discrimination performances comparable to the MLP, but at the price of many more parameters.
Acknowledgments
This work has benefited from discussions with Bo Saderberg. References Barna, G., and Kaski, K. 1990. Stochastic vs. deterministic neural networks for pattern recognition. Phys. Scripta T33, 110-115. Cohn, D., and Tesauro, G. 1992. How tight are the Vapnik-Chervonenkis bounds? Neural Comp. 4,249-269. Duda, R., and Hart, P. E. 1973. Pattern Classification and Scene Analysis. John Wiley, New York. Kohonen, T. 1990. Self-organization and Associative Memory, 3rd ed. SpringerVerlag, Heidelberg. Kohonen, T., Barna, G., and Chrisley, R. 1988. Statistical pattern recognition with neural networks: Benchmarking studies. IEEE Second Int. Conf. Neural Networks I, 61-68.SOS Printing, San Diego, CA. Lonnblad, L., Peterson, C., and Rognvaldsson, T. 1992. Pattern recognition in high energy physics with artificial neural networks-JETNET 2.0. Computer Phys. Comrnun. 70, 167-182. MaQueen, J. 1967. Some methods for classification and analysis of multivariate observations. In Proc. 5th Berkeley Symposium on Math. Stat. and Prob., L. M. LeCam and J. Neyman, eds. University of California Press, Berkeley. Peterson, C., and Hartman, E. 1989. Explorations of the mean field theory learning algorithm. Neural Networks 2,475-494. Richard, M., and Lippmann, R. 1991.Neural network classifiers estimate Bayesian a posteriori probabilities. Neural Comp. 3,461-483. Rumelhart, D., and McClelland, J. (eds.) 1986. Parallel Distributed Processing, \ Vol. 1. MIT Press, Cambridge, MA. Scabai, I., Czak6, F., and Fodor, Z.1992. Combined neural network-QCD classifier for quark and gluon jet separation. Nuclear Phys. B374,288-308. Schwartz, D., Samalam, V., Solla, S., and Denker, J. 1990. Exhaustive learning. Neural Comp. 2, 374-385. Sontag, E. 1992. Feedforward nets for interpolation and classification. 1. Computer Syst. Sci., in press. Received 6 July 1992;accepted 20 October 1992.
This article has been cited by: 2. Thorsteinn Rögnvaldsson . 1994. On Langevin Updating in Multilayer PerceptronsOn Langevin Updating in Multilayer Perceptrons. Neural Computation 6:5, 916-926. [Abstract] [PDF] [PDF Plus]
Communicated by Richard Lippmann
A Neural Network That Learns to Interpret Myocardial Planar Thallium Scintigrams Charles Rosenberg Geriatrics, Research, Education and Clinical Center, VA Medical Center, Salt Lake City, UT 84148 U S A
Jacob Ere1 Department of Cardiology, Sapir Medical Center-Meir General Hospital, Kfar Saba, Israel
Henri Atlan Department of Biophysics and Nuclear Medicine, Hadassah Medical Center, lerusalem, Israel
The planar thallium-201 (201T1)myocardial perfusion scintigram is a widely used diagnostic technique for detecting and estimating the risk of coronary artery disease. Interpretation is currently based on visual scoring of myocardial defects combined with image quantitation and is known to have a significant subjective component. Neural networks learned to interpret thallium scintigrams as determined by both individual and multiple (consensus) expert ratings. Four different types of networks were explored: single-layer, two-layer backpropagation (BPI, BP with weight smoothing, and two-layer radial basis function (RBF). The RBF network was found to yield the best performance (94.8% generalization by region) and compares favorably with human experts. We conclude that this network is a valuable clinical tool that can be used as a reference ”diagnostic support system” to help reduce inter- and intraobserver variability. This system is now being further developed to include other variables that are expected to improve the final clinical diagnosis. 1 Introduction Coronary artery disease (CAD) is one of the leading causes of death in the Western world. The planar thallium-201 is considered to be a reliable diagnostic tool in the detection of CAD. Thallium is a radioactive isotope that distributes in mammalian tissues after intervenous administration and is imaged by a gamma camera. The resulting scintigram is visually interpreted by the physician for the presence or absence of Neural Computation
5,492-502 (1993) @ 1993 Massachusetts Institute of Technology
Myocardial Planar Thallium Scintigrams
493
defects-areas with relatively lower perfusion levels. In myocardial applications, thallium is used to measure myocardial ischemia and to differentiate between viable and nonviable (infarcted) heart muscle (Pohost and Henzlova 1990). Diagnosis of CAD is based on the comparison of two sets of images, one set acquired immediately after a standard effort test (BRUCE protocol), and the second following a delay period of four hours. During this delay, the thallium redistributes in the heart muscle and spontaneously decays. Defects caused by scar tissue are relatively unchanged over the delay period (fixed defect), while those caused by ischemia are partially or completely filled-in (reversible defect) (Beller 19911. Image interpretation is difficult for a number of reasons: the inherent variability in biological systems that makes each case essentially unique, the vast amount of irrelevant and noisy information in an image, and the "context dependency" of the interpretation on data from many other tests and clinical history. Interpretation can also be significantly affected by attentional shifts, perceptual abilities, and mental state (Cuar6n et al. 1980; Franken and Berbaum 1991). Neural networks have been applied to several problems in cardiology including the detection of stenosis (Porenta et al. 1990; Lee 1990; Cios et al. 1989; Cianflone et al. 1990; Porenta et al. 1988). These studies encouraged us to explore the problem of visual interpretation using neural networks, which, to our knowledge, has not been previously addressed.' 2 Data
Scintigraphic images were acquired for each of three views: anterior (ANT), left lateral oblique (LAO 45)) and left lateral (LAT)2for each patient case. Each image was first preprocessed3 and presented to the networks as circumferential profiles (Franciscoet al. 1982; Garcia et al. 1981); in which maximum pixel counts within each of 60,6" contiguous segmental regions are plotted as a function of angle (Garcia 1991) (see Fig. 1A). Cases were preselected based on the following criteria (Beller 1991):
Insufficient exercise. Cases in which the heart rate was less than 130 bpm were eliminated, as this level of stress is generally deemed 'Since acceptance of this paper for publication, we learned that another project has developed along similar lines (Dorffner et al. 1992). *Also sometimes referred to as LAO 70 or the "steep LAO view. 3Preprocessing involved positioning of the region of interest (ROD, interpolative background subtraction, smoothing and rotational alignment to the heart's apex (Garcia 1991): 4Theprofileswere generated using the Elscint CTL software package for planar quantitative thallium-201 based on the Cedars-Sinai technique (Garcia et al. 1981; Maddahi et al. 1981; Areeda et al. 1982).
C. Rosenberg, J. Erel, and H. Atlan
494
60
40 Degrees
0
360
Regional Scores
B.
0
0
0
0
0
0
0
0
01
0
Severe Moderate Mild Normal
ttt
- - n ANT
LAO 45
LAT 70
VIEWS
Figure 1: (A) Circumferentialprofiles for stress (top) and delayed (bottom)distributions for the left lateral view (LAT). The relatively lower levels of perfusion in the center of both graphs is evidence of a mild-to-moderate defect in the apical region. The defect remains after the delay period (bottom curve), suggesting that the defect is "fixed and likely due to the presence of scar tissue. (8) The general two-layer network architecture. See text for explanation.
Myocardial Planar Thallium Scintigrams
495
insufficient to accurately distinguish normal from abnormal conditions.
Positional abnormalities. In a few cases, the “region of interest” was not positioned or aligned correctly by the technician. 0 Increased lung uptake. Typically in cases of multivessel disease, a significant proportion of the perfusion occurs in the lungs as well as in the heart, making it more difficult to determine the condition of the heart due to the partially overlapping positions of the heart and lungs. Twenty percent of the cases were eliminated due to insufficient heart rate, and an additional 5% due to either positional abnormalities or increased lung uptake. A set of 100 usable cases resulted. Each case was visually scored for each of nine anatomical regions generally accepted as those that best relate to the coronary circulation: septal, proximal and distal; anterior, proximal and distal; apex, inferior, proximal and distal; posterior-lateral, proximal and distal. Scoring for each region was from normal (1) to severe (41, indicating the level of the observed perfusion deficit. Approximately 80%of the cases were visually interpreted by a single expert alone. Ten percent were read by all three experts, and 10%by two experts, with agreement in the scoring reached by consensus. 0
2.1 Methods. Four different types of networks were explored: a onelayer (sigmoidal) network trained using the delta rule, a two-layer network trained using backpropagation (BP), a two-layer network trained using BP combined with weight smoothing (to be described), and a twolayer network combining one layer of competitive radial-basis function (RBF)units and a second layer of (sigmoidal) units trained using the delta rule. The general two-layer network architecture is depicted in Figure 1B. All networks consisted of 180 input units and nine output units. The input units were divided into three groups. Each group of 60 units encoded the circumferential profile for each view. A relative coding scheme was used in which the 60 ordered segments constituting each view were divided by the segment with the highest scintigraphic count. Each input pattern was then normalized to unit vector length. Target numerical values were assigned to the categorical visual scores to make the data suitable for network learning: normal = 0.0, mild defect = 0.3, moderate defect = 0.7, and severe defect = 1.0. 2.1.1 Weight Smoothing. Weight smoothing “penalized adjacent fanin weights to the hidden units in proportion to their difference:
C11. .-- ( w 1.1 . - w .#,,-I) . + (wij - wi,j+l) (2.1) where j - 1 and j 1 are the left and the right “neighbors” of j . This rule
+
C. Rosenberg, J. Erel, and H. Atlan
496
is a special case of one proposed elsewhere (Lang and Hinton 1990). The value of Cv was subtracted from Awij on each weight update:
w.. 11 - w..9 + Aw.. 11 - PC.. ‘J In the present experiments, the smoothing parameter
(2.2)
P was set to 0.01.
2.1.2 Radial Basis Function Netzuorks. RBF networks (Moody and Darken 1989; Poggio and Girosi 1990; Broomhead and Lowe 1988) typically consist of two layers of units: one layer of radial units, typically gaussian, and a second layer of semilinear units. In our experiments, the activation value of a gaussian unit, 0,,is given by
(2.4)
where j is an index to a gaussian unit, i is an input unit index, and 1 1 ~ 1 1 is the length of the current input vector v (for a single view). The width of the gaussian, given by w, was fixed at 0.25 for all units. The gaussian units were trained using a competitive learning scheme that moves the center of the unit closest to the current input pattern (Om,,,i.e., the “winner”) closer to the input pattern: (2.5) In addition, the other units were also pulled toward the input pattern, although to a much smaller extent? We used a ratio of 1 in 100. The second layer was trained using the delta-rule (Widrow and Hoff 1960). Input to the second layer consisted of the activations of the entire set of nine gaussian units plus an additional unit per view to encode the scaling factor of the input patterns lost as a result of input vector normalization. This additional unit was clamped to the minimum value of the input pattern for that view? In our experiments, first the competitive layer was trained and then the supervised. 2.2.3 Training. Because of the limited number of cases available, we employed the “leave-one-out” or “jackknife” method of training. We trained 100 separate networks; each network was trained on a subset of 5Referredto as the leaky learning model in Rumelhart and Zipser (1986). 6The scaling factor was given as additional input to the output units in the other networks as well to ensure that the comparisons were fair.
Myocardial Planar Thallium Scintigrams
497
99 of the 100 cases and tested on the remaining one. This procedure enabled us to determine the generalization performance for each case. Each of the 100 networks was initialized in the normal way: weights were initialized to small random values between 0.5 and -0.5 and trained with learning rate (q)= 0.05 and momentum ( a )= 0.9. Training was terminated based on cross-validation testing on the remaining case. An optimal level of training was observed for all network types. This was found to be around 40 epochs for the backpropagation and backpropagation with smoothing networks, 100 epochs for the delta rule networks, and 300 epochs for the RBF networks (supervised training portion). Further training always led to overtraining and poorer generalization. The RBF competitive layer was trained for 20 epochs at a learning rate (v) of 0.1, and then for 80 epochs at 0.01. Momentum ( a ) was not used for competitive learning. 2.2 Results. Performance was examined as a function of the number of hidden (or RBF) units (Fig. 2). For all networks, an optimal number of units was found. For the RBF and the BI' networks, three hidden units per input group was optimal (Fig. 3). For the BP with smoothing networks, five was optimal.' To compare the performance more directly to the target consensus scores, we found decision thresholds that best separated normal and mild defects from moderate and severe, as measured by percent agreement with the consensus scores. In all cases (except for the single-layered network), this decision threshold value turned out to be 0.50 (k0.01).' Based on these decision thresholds, the performances of the networks in separating normal and mild defects from moderate and severe defects were computed (Fig. 4). The best performance was attained by the RBF network. This network agreed with the target consensus scores on 94.8% (853/900) of the regions, sensitivity 57.0%(47/82) and specificity 98.5% (806/818). Differences among the networks were tested using the nonparametric Friedman test with further evaluation using the two-tailed Wilcoxon signed-rank test for significance ( p < 0.05). Analysis was based on mean squared differences between outputs and targets (control). ) 11.244,p < 0.0251. The Network type was a significant factor [ ~ ; ( 3= single-layer network was significantly inferior to all the other networks ( p < 0.001). However no significant differences were found among the RBF, BP, and BP with smoothing networks. The BP with smoothing network was more accurate than the BP network, although this difference was not significant ( p < 0.1). 7Althoughwe will not pursue this here, the optimal number of hidden units for the smoothing networks seemed to interact somewhat with smoothing level p. With larger values of p, larger numbers of hidden units were found to be optimal. *The threshold for the single-layered network was found to be 0.78.
C. Rosenberg, J. Erel, and H. Atlan
498
Comparison of 4 Network Types ,019 '02
f .ole
0 LMS
'
A BP 0smoothed BP 0 REFS
.017
,012 n i l. .-.
Y
.
b
0
.
. - . . . - . .. . . . . . . . 2
4
6
8
10
12
14
16
. - . .. 18
20
22
Hlddrn Units
Figure 2 Average mean squared error (MSE) of the four network types as a function of the number of hidden units.
Figure 3: ROC curve (sensitivity vs. specificity) for the RBF network with three hidden units.
Myocardial Planar Thallium Scintigrams
499
Figure 4: Percent agreement (by region) with the target consensus scores of the best network of each type. There were nine regions for each of 100 cases or 900 regions in total. Indicates a statistically significant difference, based on MSE, at the p < 0.05 level. lY
2.2.1 Comparison with an Individual Expert. A randomly selected set of 52 cases was reinterpreted by one of the participating physicians. %oring was performed without regard to the original scores. Nine of the cases were previously rated by this physician, either individually or in consensus. In normal or mild versus moderate or severe defect discrimination, this physician’s scores agreed with the consensus scores on 97.6% (457/468) of the regions Isens: 89.5% (34/38), spec: 98.4% (423/430)1. On the same set of cases, the RBF network’s scores agreed with the consensus scores on 95.9% (449/468) of the regions [sens: 63.2% (24/38), spec: 98.8%(425/430)1, and agreed with the physician’s scores for 94.9% (444/468) of the regions [sens: 56.1% (23/41), spec: 98.6% (421/427)1. These differences were not found to be significant in a paired sign test. 3 Conclusions and Future Directions
A network with one hidden layer of RBF units was compared with networks trained with backpropagation, backpropagation with weight smoothing, and single-layer networks trained with the delta-rule. The best performance (94.8%generalization) was attained by an RBF network
C. Rosenberg, J. Erel, and H. Atlan
500
with three hidden units per view (nine in total). This level of performance did not differ significantly from that of an expert. Performance of the single-layer delta-rule network was significantly poorer than that of the other networks. The BP with weight smoothing yielded a moderate, though not statistically significant, improvement over standard BP. Improvements and extensions include the following:
Standardize visual scores. The data currently are a mixture of single expert and consensus scores. Given the likelihood of individual differences in interpretation (Franken and Berbaum 1991; Cuarbn et al. 1980),improved performance should be attained if single expert or standardized group data are used. Elicit confidence ratings. Expert visual interpretations could be augmented by degree of confidence ratings. Highly ambiguous cases could be reduced in importance or eliminated. The ratings could also be used as additional targets for the network9 cases indicated by the network with low levels of confidence would require closer inspection by a physician. Provide additional information. We have not yet incorporated the delayed distribution profiles, clinical history, gender, and examination EKG. The inclusion of these variables should allow the network to approximate more closeJy a complete diagnosis, and boost the utility of the network in the clinical setting. Add constraints. Currently we do not utilize the angles that relate the three views. It may be possible to build these angles in as constraints and thereby cut down on the number of free network parameters. Expand application. Besides planar thallium, our approach may also be applied to nonplanar 3-D imaging technologies and other nuclear agents or stress-inducing modalities. Preliminary results are promising in this regard. We believe that our study is the first to successfully apply neural networks to the prediction of the human expert’s visual interpretation of myocardial scintigraphy. Our network can be used today as a standard reference tool to help the physician reduce response bias and other types of scoring variability and may, with the improvements suggested, eventually equal or surpass the interpretive abilities of the physician.
Acknowledgments The authors wish to thank Prof. Benny Shanon for much kind assistance and support throughout this research project, Mr. Haim Karger for tech9See Tesauro and Sejnowski (1988) for a related idea.
Myocardial Planar Thallium Scintigrams
501
nical assistance, and the Departments of Computer Science and Psychology at the Hebrew University for computational support. We would also like to thank Drs. David Shechter, Moshe Bocher, Roland Chisin, and the staff of the Department of Medical Biophysics and Nuclear Medicine for their help, both large and small. This paper has also benefitted from the comments of two anonymous reviewers. Terry Sejnowski suggested our usage of RBF units. Charles Rosenberg was supported by grants from the Ministry of Science and Technology, Israel and from the Golda Meir
Fund.
References Areeda, J., Van Train, K., Garcia, E. V., Maddahi, J., Rosanki, A., Waxman, A., and Berman, D. 1982. Improved analysis of segmental thallium-201 myocardial scintigrams: Quantitation of distribution, washout, and redistribution. In Digital Imaging, P. D. Esser, ed. Society of Nuclear Medicine, New York. Beller, G. A. 1991. Myocardial perfusion imaging with thallium-201. In Cardiac Imaging, M. L. Marcus, H. R. Schelbert, D. J. Skorton, and G. L. Wolf, eds. W. 8. Saunders, Philadelphia. Broomhead, D. S., and Lowe, D. 1988. Multivariable functional interpolation and adaptive networks. Complex Syst. 2,321-355. Cianflone, D., Carandente, O., Fragasso, G., Margononato, A., Meloni, C., Rossetti, E., Gerundini, P., and Chiechia, s. L. 1990. A neural network based model of predicting the probability of coronary lesion from myocardial perfusion sped data. In Proceedings of the 37th Annual Meeting of the Society of Nuclear Medicine, p. 797, May. Cios, K. J., Goodenday, L. S., Merhi, M., and Langenderfer, R. 1989. Neural networks in detection of coronary artery disease. Computers in Cardiology Conference, September, 33-37. Cuarh, A., Acero, A., Cbrdena, M., Huerta, D., Roddguez, A., and de Garay, R. 1980. Interobserver variability in the interpretation of myocardial images with Tc-99m-labeled diphosphonate and pyrophosphate. J . Nucl. Med. 21(1), 1-9. Dorffner, G., Prem, E., Mackinger, M., Kundrat, S., Petta, P., Porenta, G., and Sochor, H. 1992. Experiences with neural networks as a diagonostic tool in medical image processing. Unpublished manuscript. Francisco, D. A., Collins, S. M., and Go, R. T., Ehrhardt, J. C., Van Kirk, 0. C., and Marcus, M. L. 1982. Tomographic thallium-201 myocardial perfusion scintigrams after maximal coronary artery vasodilation with intravenous dipyridamole: Comparison of qualitative and quantitative approaches. Circulation 66(2), 370-379. Garcia, E. V. 1991. Physics and instrumentation of radionuclide imaging. In Cardiuclmuging, M. L. Marcus, H. R. Schelbert, D. J. Skorton, and G. L. Wolf, eds. W. B. Saunders, Philadelphia. Garcia, E. V., Maddahi, J., Berman, D. S., and Waxman, A. 1981. Space-time
502
C. Rosenberg, J. Erel, and H. Atlan
quantitation of thallium-201 myocardial scintigraphy. 1.Nucl. Med. 22,309317. Franken Jr., E. A., and Berbaum, K. S. 1991. Perceptual aspects of cardiac imaging. In Cardiac Imaging, M. L. Marcus, H. R. Schelbert, D. J. Skorton, and G. L. Wolf, eds. W. B. Saunden, Philadelphia. Lang, K. J., and Hinton, G. E. 1990. Dimensionalityreduction and prior knowledge in e-set recognition. In Advances in Neural Information Processing Systems 2, D. S . Touretzky, ed., pp. 178-185. Morgan Kaufmann, San Mateo, CA. Lee, S. C. 1990. Using a translation-invariant neural network to diagnose heart arrhythmia. In Advances in Neural Information Processing Systems 2, David S. Touretzky, ed., pp. 240-247. Morgan Kaufmann, San Mateo. Maddahi, J., Garcia, E. V., Berman, D. S., Waxman, A,, Swan, H. J. C., and Forrester, J. 1981. Improved noninvasive assessment of coronary artery disease by quantitative analysis of regional stress myocardial distribution and washout of thallium-201. Circulation 64, 924-935. Moody, J., and Darken, C. 1989. Fast learning in networks of locally-tuned processing units. Neural Comp. 1, 281-294. Poggio, T., and Girosi, F. 1990. Regularization algorithms for learning that are equivalent to multilayer networks. Science 247, 978-982. Pohost, G. M., and Henzlova, M. J. 1990. The value of thallium-201 imaging. N . Engl. 1.Med. 323(3), 190-192. Porenta, G., Dorffner, G., Schedlmayer, J., and Sochor, H. 1988. Parallel distributed processing as a decision support approach in the analysis of thallium-201 scintigrams. Computers in Cardiology Conference, September. Porenta, G., Kundrat, S., Dorffner, G., Petta, P., Duit, J., and Socher, H. 1990. Computer based image interpretationsof thallium-201 scintigrams: Assessment of coronary artery disease using the parallel distributed processing approach. Proceedings of the 37th Annual Meeting of the Society ofNuclear Medicine, p. 825, May. Rumelhart, D. E., and Zipser, D. 1986. Feature discovery by competitive learning. In Parallel Distributed Processing, D. E. Rumelhart and J. L. McClelland, eds., Vol. 1, Chapter 5, pp. 151-193. MIT Press, Cambridge, MA. Tesauro, G., and Sejnowski, T. J. 1988. Aparallel network that learns toplay backgammon. Tech. Rep. CCSR-88-2, University of Illinois at Urbana-Champaign Center for Complex Systems Research, February. Widrow, B., and Hoff, M. E. 1960. Adaptive switching circuits. In 2960 IRE WESCON Convention Record, vol. 4, pp. 96-104. IRE, New York. Received 13 January 1992;accepted 28 September 1992.
REVIEW
Communicated by Franqoise Fogelman
The Use of Neural Networks in High-Energy Physics Bruce Denby Fermi National Accelerator Laboratory, M.S. 318, Batavia, IL 60520 USA In the past few years a wide variety of applications of neural networks to pattern recognition in experimental high-energy physics has appeared. The neural network solutions are in general of high quality, and, in a number of cases, are superior to those obtained using "traditional'' methods. But neural networks are of particular interest in high-energy physics for another reason as well: much of the pattern recognition must be performed online, that is, in a few microseconds or less. The inherent parallelism of neural network algorithms, and the ability to implement them as very fast hardware devices, may make them an ideal technology for this application. 1 Introduction
High-energy physics (HEP) is the field that studies the basic constituents of matter and the fundamental forces through which they interact. Recently, high-energy physicists have become interested in neural networks as HEP data analysis tools. It has been only a few years since the first investigations of neural networks for HEP were undertaken (Denby 1988; Peterson 1989), and much of today's work is still exploratory; however, the growth in applications to HEP is quite striking. At the Second International AIHEP Workshop (AIHEP 1992) at La Londe-les-Maures, France, in January 1992, 25 applications of neural networks in highenergy physics were presented. For comparison, at the first workshop in this series, in Lyon, France in March 1990, there were only two such presentations. In applications to date, neural networks have proven themselves to be more efficient classifiers than the simple cuts normally used in HEP, have allowed certain measurements to be made with smaller uncertainties due to their superior ability at function approximation, and have permitted analyses to be made even from heavily overlapping distributions due to their good approximation to Bayes probabilities. There have been some extremely interesting results using hardware neural networks: it appears possible to make rather sophisticated pattern analyses directly in the readout hardware of HEP experiments rather than in the standard, time-consuming offline analysis. Neural Computation 5,505-549 (1993) @ 1993 Massachusetts Institute of Technology
506
Bruce Denby
interaction region
accelerator ring
Figure 1: An accelerator with six interaction regions. Particles in bunches circulate in opposite directions, being brought together for collisions within the interaction regions. Normally only one or two particles within the bunches will actually collide during the crossing of two bunches. The Fermilab Tevatron has a diameter of about 1 mile. The SSC to be built in Texas will be about seventeen times larger. 1.1 HEP Accelerators. HEP data are produced in experiments at the large accelerator centers worldwide as detailed in Table 1. Each site features a “ring” in which opposing beams of particles are made to collide at one or more “interaction regions” (Fig. l).’ In the collisions, daughter particles of many kinds are produced, and these are detected in arrays of particle detectors surrounding the interaction region (see Fig. 2). The data from these detectors constitute the HEP data sets from which physics results must be extracted. As more powerful particle accelerators are built, the accompanying experiments grow tremendously, both in physical size and in the demands they place upon their data readout systems. Figure 2, detailing
Figure 2: Facing page. Elevation view of the CDF experiment at the Fermilab Tevatron. Only half of the apparatus is shown; it is symmetric about the point marked “interaction point.” In the text, applications of neural networks to track reconstruction in a central tracking chamber, vertex finding in a vertex chamber, electron finding in an endplug calorimeter, and muon identification in a muon chamber are presented. ‘There are also experiments-in which the extracted beam is directed onto a fixed target; for simplicity we shall not discuss these here.
Batavia,IL Geneva, Switzerland Hamburg, Germany Stanford, CA Tsukuba, Japan Ellis cty., Tx Geneva, Switzerland
Location p,p e+,ee-,p e+,ee+,ep,p p,p
Beams 0.9 x 0.9 TeV 50 x 50 GeV 27 x 820 GeV 5Ox50GeV 30 x 30 GeV 20 x 20TeV 10 x 10 TeV
Energy 26 psec 96 nsec 7msec 5 psec 16nsec 16 nsec
4 psec
Period 1986 1988 1992 1988 1986 1999 1999
Startup
CDF, DO Delphi, Aleph, Opal, L.3 H1, Zeus SLD,MarkII Amy, Topaz, Venus SDC,GEM Under discussion
Major Experiments
'e- stands for electron, e+ for positron, p for proton, and p for antiproton. These particles are discussed in more detail in the following section. The unit of energy is the giga- or teraelectron volt (GeV or TeV), and time is measured in microseconds (psec) or nanoseconds (nsec). The LHC and SSC are two large new machines scheduled to turn on before the end of the decade.
LHC
ssc
Fermilab CERN DESY SLAC KEK SSC CERN
Tevatron LEP Hera SLC Tristan
~~
Lab
Accel.
~
Table 1: Names and Locations of the Major World Accelerator Centers with the Type and Energy of Beam Used,Time Between Collisions of Particle Bunches, First Date of Operation, and the Names of the Major Experiments at the Site!
1 U Y
w
m
0
ul
Neural Networks in High-Energy Physics
509
the CDF [Collider Detector at Fermilab (CDF 198811 experiment at Fermilab, gives an idea of the scale and complexity of the detectors used in a current experiment. Detectors at LHC and SSC will be larger again by a factor of two or so. The volume of data produced in these detectors and the rate at which it must be analyzed are daunting. A typical experiment may record hundreds of thousands of individual detector channels, corresponding to about 1 million bits of information, for each collision, or “event,” as they are usually called, and it is not uncommon to record many millions of events during a data taking run. The particles within a beam are stored in “bunches.” The rate of collisions varies considerably from machine to machine, and is determined by the spacing between the bunches stored in the machine, since typically only one or two particles will actually collide in each bunch crossing. In all cases, the rates are rather challenging from the standpoint of real-time processing: at the Tevatron, bunch crossings currently occur every 4 psec; at the SSC and LHC, they will occur about every 16 nsec. The growth in data set size and complexity and the unprecedented data rates at today’s and future colliders have been the major motivating factors in the search for more powerful data analysis tools for HER In the discussion of HEP neural network application that will follow, it will be necessary to have some familiarity with the terminology associated with high-energy particle collisions and the detectors that record them. Sections 1.2 and 1.3 provide an introduction.
1.2 Areas of HEP Research-the Standard Model. Much of current research in HEP is involved with the completion and verification of the so-called “Standard Model” of particle physics. In this model, the basic constituents of matter are quarks and leptons as described in Table 2. The constituents interact with each other by ”exchanging”’ particles called “bosons,” as described in Table 3. Both quarks and leptons can interact via the electroweak force, carried by the W, Z, and y bosons. This force combines the electric force, responsible for such phenomena as electricity and magnetism, with the weak force that is responsible for radioactivity. Quarks can also interact through the strong force, which is carried by bosons called gluons, usually represented as g. Individual quarks and gluons are not observable. The naturally occurring particles are either single leptons, or a “composite” of two or more quarks as in Table 4. A proton, for example, is composed of two ”u” quarks and a “ d quark, which are bound together by exchanging gluons. Composites containing quarks are also referred to as “hadrons.” Leptons and hadrons interact differently in matter, as described in Section 1.3. 2The word, “exchanging” is used figuratively. The true interaction is a quantum process that defies classical explanation.
Down Strange charm Bottom Top Positron, electron Muon Tau Electron neutrino Mu neutrino Tau neutrino
v,, fi,
vp,fip
v,, fie
r+, T-
p+. p-
t,t e+.e-
b,b
c,c
s,%
up
d,d
Name
u,u
Symbol
0
0
+2/3, -213 -113, +1/3 -113, +1/3 +2/3, -213 -113, +1/3 +2/3, -2/3 +l.-1 +1, -1 +1, -1 0
Charge
O? O? O?
100 MeV 100 MeV 500 MeV 1.5 GeV 5 GeV 130 GeV? 511 keV 106 MeV 1.8 GeV
---
Mass
Ordinary matter composed of up and down quarks Strange matter exists in stars Discovered in 1973 Current area of study Not yet seen; much sought Causes chemical bonds Exist naturally in cosmic rays First heavy lepton discovered Not visible in detectors, except as 'missing' energy; masses thought to be zero
Comments
'The equivalence of matter and energy allows us to write masses in energy units of eV
Neutral leptons
Charged leptons
Heavy quarks
Light quarks
Type
Table 2 Quarks and Leptons and Their Properties,Including Mass and Electric Charge."
Neural Networks in High-Energy Physics
511
Table 3: The Force Carrying Bosons and Their Properties.“ Force
Symbol
Name
Charge
Weak
Wf, W-
W+, W-
+1, -1
Z
Z
Photon
0 0
91 GeV
Y g
Gluon
0
0
Weak Electromagnetic Strong
Mass
Comments
81 GeV Discovered at CERN in 1983; carriers of weak
0
force Light is composed of photons Binds quarks in composites
“The first column tells the type of interaction the boson mediates: weak, electromagnetic, or strong.
Table 4: The Composites Most Commonly Encountered in HEP Detector Systems. Type
Symbol Quark content
Charge
Mass Comments 939 MeV Atomic nuclei made of 940 MeV protons and neutrons 135 MeV Most commonly 500 MeV produced composites
Proton
P
uud
+1
Neutron
n
udd
0
Pion Kaon
A
K
ud, uu + dd us, ds
+1, -1,0 +1, -1,0
--
A collision between particles is, in the Standard Model theoryj an interaction between two of the elementary constituents that they contain. For example, in a collision between a proton and an antiproton, the “true” collision may be between a quark and an antiquark, between a quark or antiquark and a gluon, or between two gluons. When physicists examine the debris of such a collision, they are seeking information on the constituents and force carriers that are produced in the collision. Quarks and gluons emerging from a collision are not directly observable in the detector; they are said to “fragment” into “jets” containing many particles as they emerge from a collision. Jets from quarks and from gluons are slightly different in their properties, as will be discussed in Section 4.2. The most “fashionable” areas of research in HEP today are the study of the production and decay properties of the “heavy” (i.e., massive) quarks, c and b, the search for the heaviest quark, called “top,” or simply, “t,” which is postulated but as yet undiscovered, studies of the bosons W and Z, the search for the Higgs particle (Table 51, an essential but as
512
Bruce Denby Table 5: The Higgs Particle.
Symbol Name Charge Mass
I-F‘
Higgs
0
?
Comments Essential to theory; not yet seen; gives mass to particles
yet unobserved element of the Standard Model believed to be the origin of the masses of all particles, and the study of the characteristics of jets. 1.3 HEP Measurement Tools. Although there are quite a number of different types of measurement tools used in high-energy physics, most can be classified as one of two main types, tracking chambers and calorimeters. Figure 3 shows a generic HEP detector system with a central tracking chamber and a vertex tracking chamber, calorimeter with sections called “electromagnetic” and “hadronic,” muon shielding iron, followed by another set of tracking chambers called muon chambers. The figure illustrates the behavior of the detectors for the four most commonly encountered types of particles and for a jet. Tracking chambers are used to detect the trajectories of electrically charged particles emerging from a collision. Usually the tracking chamber volume is within a magnetic field. This causes the path of the charged particle to curve, enabling a measurement of the momentum3 of the particle. A knowledge of the momenta of all charged particles allows a complete study of the underlying dynamics of the collision to be made. When a charged particle passes through the chamber, gas molecules along its trajectory are ionized (there are also tracking chambers which do not use gas as an active medium, but we shall not discuss them here). Highvoltage wires spaced regularly throughout the tracking volume collect this ionization in the form of electrical pulses, which can then be passed on to the data acquisition system for analysis and reconstruction of the tracks. Position resolution finer than the wire spacing is obtained by using an electronic device to measure the time it takes for the ionization to drift to the wire. This is referred to as the “drift time.” In Figure 3 only wires closest to the trajectories, called “hit” wires, are shown. There are many different types of calorimeters but all have the same basic principle of operation. Calorimeters are normally built from many layers of metal interleaved with layers of a plastic or gas active medium. Quite the opposite of the tracking chambers, through which the particles pass uninterrupted, a calorimeter is designed to cause most particles incident on it to interact and deposit all of their energy within its volume. The energy may be in the form of ionization or of light, but will 3The momentum P of a particle is defined as P = Ev/8 where E is its energy, v is its velocity, and c is the speed of light. For nonrelativistic particles E m s , where m is the mass, giving p E mu.
Neural Networks in High-Energy Physics
513
ultimately be converted into an electrical impulse with a size proportional to the energy of the particle. Most calorimeters have two sections, called "electromagnetic" and "hadronic" of different composition. The electromagnetic section is designed to absorb almost all of the energy of the electromagnetically interacting particles, i.e., electrons and photons, while hadrons will deposit the largest fraction of their energy in the hadronic section. Calorimeters are usually highly segmented in order to give information on the spatial extent of the energy deposit from the particle, as shown in Figure 3, where the energy in each cell is represented by the height of the tower drawn at each cell. Note that the segmentation in the electromagnetic section is twice as fine as in the hadronic section. Calorimeters are particularly useful for identification of electrons. An electron will deposit almost all of its energy in a highly localized region
Figure 3: Behavior of a muon, electron, pion, neutrino, and jet in a HEP detector system. The beam pipe is perpendicular to the plane of the page. The muon passes completely through the calorimeters, depositing only a small amount of energy in each section, and through the shielding iron, to be finally detected in the muon tracking chambers. The electron deposits all of its energy in a localized region of the electromagnetic calorimeter. The pion deposits its energy over a region of both electromagnetic and hadronic calorimeters. The jet is composed of many particles of different types, mostly pions, and deposits energy both in electromagnetic and hadronic sections of the calorimeter over a broad region. The neutrino does not interact at all and passes undetected through the apparatus.
Bruce Denby
514
of the electromagnetic calorimeter. By looking for a charged track that points at this localized region, and matching the calorimeter energy to the track momentum, an electron can be reliably identified. Muons are charged particles that are capable of penetrating through great thicknesses of material with only minimal energy loss. For this reason, special muon tracking chambers are placed outside the calorimeter and a thickness of uninstrumented shielding iron in order to detect possible tracks from muons produced in a collision. The energy of other types of particles will be completely absorbed in the calorimeters and the shielding iron. The muon can be identified by measuring its momentum in the central tracking chamber and seeing if its projection through the calorimeter and iron matches well with a track ”stub” found in the muon chambers. The detectors’ response to pions, neutrinos, and jets is described in the caption of Figure 3. 1.4 Pattern Recognition in HEP-Standard
Methods.
1.4.1 Introduction. The only particles that are directly observable are those that have a natural lifetime long enough to allow them to be detected in the apparatus, that is, photons, muons, electrons, and some of the low mass composites such as pions and kaons. Neutrinos normally leave no trace in the apparatus and are detectable only by their “missing” energy. Most of the constituents produced in a collision quickly decay into these observable particles, or, in the case of quarks and gluons, fragment into jets containing many particles. The properties of the constituents must therefore be inferred from patterns in the “visible” particles into which they decay or fragment. Reconstructing an event involves two types of pattern recognition. The first, which we shall call low level pattern recognition, consists of such things as finding tracks in the tracking chambers or identifying a candidate electron in the calorimeter (Fig. 3). The second type, which we shall call physics process determination, uses more sophisticated features, for example, the angular distribution of the jets in the event, to try to identify the underlying physics of the interaction that took place. Note that this nomenclature is not the same as typically found in classical pattern recognition, since classification, normally considered “high-level,” can occur both in our low-level and high-level pattern re~ognition.~ In HEP, the distinction between high-level and low-level pattern recognition is based on the complexity of the features used to perform the classification. Examples of the two types will be given in the sections to follow. We shall see that neural networks have found application to both. 4Segmentation of the data into events is performed trivially using timing information that correlates a block of data with the time of a particular bunch crossing.
Neural Networks in High-Energy Physics
515
In HEP it is also necessary to distinguish whether the pattern recognition is to be performed “on-line,” that is, in real time, or “off-line.” On-line pattern recognition is performed on the data before it is logged, in a part of the experiment referred to as the “trigger.” Off-line pattern recognition is done with conventional computers operating on the data after it has been logged to permanent storage media. These two areas will be discussed in more detail below. 1.4.2 Triggering. New HEP experiments study increasingly rare physical processes. The implications of this for data acquisition systems are best illustrated by an example. One of the main motivations for the construction of LHC and SSC is the search for the Higgs particle. The probability of producing a Higgs particle when two protons interact is so small that such interactions would have to occur loM times per second5 in order to produce a reasonable sample of detectable Higgs particles, say 1000, during a 1-year run. The probability for other processes however, not involving the Higgs, is higher by a factor of about lOI3. This implies that, during this 1-year run, events containing background processes will be continuously produced at a rate of about 1 billion per second. It is neither desirable nor feasible to log all of these events to permanent storage media such as magnetic tape. On-line pattern recognition, called “triggering,” is required to reject background events and retain the rare interesting events. Although LHC and SSC represent an extreme case in high-rate HEP data acquisition, the problems are common to all HEP experiments. Figure 4 shows a typical multilevel HEP trigger system. The data from the detectors pass into the trigger as a stream of events, each containing all the detector data produced in a single collision. Each level of trigger rejects most of the events it receives and passes the remainder on to the higher level triggers. Levels 1 and 2 are typically implemented as fast specialized analog or digital hardware, while level 3 is a “farm” of conventional processors. The processing times and event rates shown at each level are generic, but typical of those encountered at current protonantiproton collider experiments such as the CDF experiment at Fermilab (CDF 1988); rates will be one to two orders of magnitude higher at LHC and SSC. In level 1, simple tests on global event information are performed, for example: (1) comparing to a threshold the summed transverse energy, Et = C,Ei sin Hi, where Ei is the energy in calorimeter cell i and Hi is the angle with respect to the beam axis of a line from the collision point to the calorimeter cell, (2) looking for the presence of a charged track with transverse momentum, Pt = Psin H, where P is the track mo5The lp per second is technically the accelerator ”luminosity” required to produce the lo00 Higgs particles. Luminosity is defined as the square of the number of particles per bunch, times the number of bunches per beam, times the revolution frequency of the bunches within the ring, divided by the cross-sectional area of the beams.
Bruce Denby
516
f
\
Detectors
specialized hardware neural nets?
(effective time,
'conventional' CPUS
Figure 4 Generic HEP multilevel trigger system. mentum, above a threshold, and (3) looking for the presence of one or more track segments in the muon chambers. The first and second cuts eliminate "soft" interactions. Most interesting physics processes involve "hard" scatters of two constituents in the beam particles, which produce particles at large angles to the beam direction and thus deposit in the
Neural Networks in High-Energy Physics
517
calorimeter substantial energy transverse to the beam direction. "Soft,'' glancing collisions of beam particles are much more copiously produced than hard scatters, and most must be rejected. The third cut is useful since high Pt muons are produced in many of the interesting processes currently under study, but are produced only with low probability in background processes. Level 1 triggers have a typical processing time of about 1 psec and reduce the rate due to backgrounds by about two orders of magnitude. In the level 2 trigger, somewhat more sophisticated tests can be done, for example: (1) looking for a match between a high-Pt track and an energy cluster in the electromagnetic calorimeter, indicating the presence of a candidate electron, or between a high?, track and a track segment in the muon counters, indicating a candidate muon, and (2) looking for the presence of localized clusters of energy in the calorimeter, which will correspond to jets, with Et above some threshold. Validating the presence of leptons and jets as in (1) and (2) above ensures that the event is more likely to have come from an interesting physics process. Ten to twenty microseconds are available for level 2 decisions. Level 3 triggers are executed using algorithms written in standard high level computer codes running on a "farm" of conventional processors that operate in parallel on separate events. As each event comes into level 3, it is immediately sent to an available processor. The processing done by level 3 can be quite sophisticated, in some cases being identical to the code used in offline analyses. Some of the typical analyses performed in level 3 are (1) reconstruction of charged tracks, (2) accurate calculation of the position of the collision point in order to reject events too far from the detector center, to allow more accurate calculation of Et of calorimeter cells, and to detect multiple vertices, (3) high-quality electron and muon identification using accurate Pt measurements of the tracks, (4) imposition of isolation cuts, that is, requiring that an electron or muon has very little energy surrounding it in the calorimeter, and (5)formation of composite triggers, for example, electron plus missing transverse energy plus one or more jets would be a good trigger for top quark production. Such calculations as these are too complicated to be performed in level 2. The time to process a single event in level 3 may be of the order of a second, however as there are many processors operating in parallel, the effective processing time is a few milliseconds per event. 2.4.3 Ofline Reconstruction. Offline reconstruction is the final event reconstruction in which all available information is processed using whatever data analysis techniques may be available. Normally all the data from a run will be processed in a single reconstruction pass in which data sets of special interest are created, for example, one for the physics of b and c quarks, one for the search for the top quark, one for W and Z physics, etc. These are often analyzed many times over with ever more
518
Bruce Denby
refined sets of selection cuts. Analysis usually proceeds with the definition of several feature variables on which one-dimensional cuts are placed. The use of likelihood techniques is also common. The offline analysis does not have the same real-time constraint as online reconstruction; however, the codes used to process high-energy physics data are normally tens of thousands of lines long and require substantial computing resources in order to complete the processing in a reasonable amount of time. It is not uncommon for a complete offline reconstruction of a particular physics process to take 1 or 2 years. 2 The Need for Neural Networks In high-energy physics, neural networks have been used both in real-time and offline applications. Most applications to date have used MLP's trained with backpropagation, although a few instances of the use of learning vector quantization (LVQ) and feature maps have also appeared. Recurrent networks have been applied to the problem of charged track reconstruction as discussed in Section 5.1. For the offline applications, the advantage to HEP is the same as that for other fields: near optimal classification with a minimum of computational overhead. In the real-time applications, neural networks present an advantage because of their parallel architecture, which allows for faster processing. We now discuss these two areas in more detail. 2.1 Neural Networks for Triggering. It is interesting to note that some of the functions performed by standard level 1 and 2 triggers as discussed above, that is, thresholding performed upon a linear combination of inputs, already resemble those performed by an artificial neuron. High-energy physicists building fast trigger electronics have for decades been making use of electronic devices called "discriminators" for performing this function. The idea of applying true neural network technology in HEP triggering, however, is quite new (Denby 1988; Denby et al. 19901, and it is far from being accepted as a standard tool. Research is currently under way to evaluate computing technologies for the trigger systems of experiments to take place at the new accelerators. At the SSC and LHC, the time between bunch crossings is so short, about 15 nsec, that no known technology can keep pace with the events as they come in. For this reason, a technique called pipelining is envisioned in most level 1 trigger schemes for SSC and LHC. The data from the events enter consecutively into a shift register. Data are clocked from one location in the shift register to the next at a time interval identical to the bunch crossing interval. At each location, part of the level 1 triggering algorithm is executed, so that by the time an event is ready to exit the pipeline, the entire level 1 algorithm will have been performed on
Neural Networks in High-Energy Physics
519
it. Thus, although an initial startup time is needed to ”fill the pipeline,” once it is full, a level 1 decision will be made every 15 nsec. It has not yet been decided how the trigger processing will be apportioned among the level 1, level 2, and level 3 triggers, nor have the exact technologies for the triggers been chosen. The construction of the new accelerators will take several years, and there is a strong tendency to avoid freezing technology choices too early. Some scenarios prefer a very sophisticated level 1 trigger which reduces the rate sufficiently to pass events directly to the level 3 processor farm (Farber et al. 1991; Crosetto and Love 1992), while others prefer a simpler level 1 coupled with high speed arithmetic processors in level 2, which reduce the requirements placed on the level 3 processor farm (SDC 1992). Algorithms making use of recurrency are not well amenable to pipelining and may require compromises to be implemented in level 1. Present day silicon neural networks, with settling times of the order of several hundred nanoseconds, are not appropriate for level 1 triggers; however, with advances in technology, much faster chips may be possible (see, e.g., Hansen 1992). But even currently available neural networks are sufficiently fast to give competitive performance for level 2 triggers. Technologies currently under study for the level 2 processors for the SSC are associativememories (Amendolia et al. 19901, neural networks (Denby et al. 1990), associative string processors (Lea 19881, and image processors (Bock et al. 1990). Although trigger systems using conventional electronics can probably be made to handle the rates to be found at SSC and LHC, neural networks can make the triggers far more efficient and less costly by moving to level 2 the complex pattern recognition normally done in level 3. In Section 3 we shall show some specific examples of this: accurate muon Pt measurement in a few microseconds, application of an isolation cut at level 2, a possible scheme for determining the position of the collision point online, etc. This will reduce the requirements placed on the level 3 processor farm and significantly reduce the amount of data that must be recorded on tape for later analysis. Another attractive feature of neural nets for triggering is their programmability. In the past, many level 2 triggers have been built as hardwired special purpose electronic devices. To change the algorithm in such a device implies rebuilding it or rewiring it. In a neural network, the algorithm can be changed simply by downloading a different set of weights, which will make neural network triggers much more flexible than their predecessors. 2.2 Offline Applications. Historically, high-energy physicists have eschewed “complicated data analyses in favor of simple one-dimensional cuts. In HEP, such problems as incomplete understanding of detector response, and heavy dependence on Monte Carlo models render the extraction of a final physics result from the experimental data an
520
Bruce Denby
extremely difficult and time-consuming task, sometimes requiring hundreds of man years of effort. There was a strong tendency to try to keep the analyses as simple as possible. However, over the years in HEP, considerable experience in detector construction techniques and in software generation has been gained, and detector simulation packages that model instrumental effects have become extremely sophisticated. Too, with the growth of collaboration size, particular groups of researchers within an experiment have been able to devote themselves exclusively to data analysis problems. The key to the value of neural networks in offline HEP analyses is in creating efficient cuts to retain events from rare physics processes while rejecting as many as possible of the background events. A further advantage is that neural networks may make possible certain analyses which previously were considered hopeless precisely because simple one-dimensional cuts were known to be ineffective discriminators. An example of this is the classification of quark and gluon jets, which we shall discuss in Section 4. It has been argued that although a series of one-dimensional cuts is less efficient than a multidimensional cut, this can be compensated for by taking more data. As the interesting physics processes to study become more rare, however, this reliance on increased statistics becomes impossible: it becomes necessary to extract as much information as possible from the data at hand.
2.3 The Problem of Training Data. One of the major goals of HEP is to identify and characterize the properties of as yet unseen constituents in the standard model. This, however, presents a problem for classification schemes involving supervised learning since there are no existing real data containing these particles. It follows that Monte Carlo data must be generated according to some model. In some cases, there are a number of rather different models to choose from. Any classification based upon these models will therefore be biased towards the model chosen. This is of course a problem for any type of classifier, however, a number of high-energy physicists are concerned that it will be more difficult to understand model dependence using neural networks than using a simpler type of classifier. This is used as an argument against using neural networks in HEP analyses. Although it is true that model dependence in a nonlinear classifier is somewhat more difficult to characterize than in a linear classifier, the superior performance of nonlinear classifiers has led some researchers to expend the additional effort necessary to characterize the model dependence. This will be seen in some of the applications described in Section 4. This effect is particularly important in triggering. Events rejected by a trigger will not be recorded, and so can never be used to check what the trigger was doing. For this reason, there has been a tendency in the past to keep trigger cuts as simple as possible to facilitate understanding of
Neural Networks in High-Energy Physics
521
the trigger efficiency. This “validation” problem is not important for triggers based on low level pattern recognition such as track segment finding or electron identification since modern detector simulations can quite reliably simulate such simple entities as tracks and electrons. However, because of possible biases from model dependence, there is still work to be done in HEP to show convincingly that unbiased information can be extracted from data taken with triggers that select specific physics processes, whether they use neural networks or more conventional technology. 3 Applications to Low Level Pattern Recognition
These applications, as well as those in later sections, are summarized in Table 6. 3.1 Trigger Applications. We will treat in this section only those trigger applications which have already been realized or have been seriously proposed. Some of the other low level pattern recognition applications which follow are also intended for triggering but are still just studies.
3.2 .2 First Real-time Application: Muon Trigger. The first real-time application of a neural network in HEP was accomplished recently at Fermilab (Lindsey et al. 1992). 3.2.2.1 Conventional method. Identification of a muon with a transverse momentum Pt above a threshold is a useful trigger for detecting decays of Ws, Zs, and b quarks since each will decay about 10% of the time to a muon. The cut on Pt is necessary since background processes produce many low Pt muons. A measurement of the Pt of a muon in the trigger requires a knowledge of the angle of the muon track at the muon chamber. Although, offline, the wire drift times can be used to calculate the track angle quite accurately, in the trigger, only the information on which wires were hit is available, resulting in an inaccurate measurement of Pt in the trigger. It is therefore necessary to set the Pt trigger threshold quite low in order to avoid discarding high Pt events that have been poorly measured. This introduces a large amount of background. 3.2.1.2 Test beam results. In a simple test beam experiment at the Fermilab Tevatron, slopes and intercepts of muon tracks traversing a small prototype drift chamber were calculated accurately, in real-time, using a commercial VLSI neural network chip incorporated into the standard drift chamber data acquisition system. This was a test experiment carried out in an auxiliary particle beam; in a full scale collider experiment, the drift chamber would be duplicated many times over to cover an area
522
Bruce Denby
Table 6 Summary of the Main HEP Neural Network Applications Covered in This Paper. Continued next page. Problem
Training set
Test set
Network
Results and comments
Low level-triggering Muon trigger test beam experiment Fermilab
15-64-64 MLP ETA”
Monte Carlo tracks
Real online data
Isolation and Real and b trigger for Monte CDF calorim. Carlo Fermilab data
Real and Monte Carlo data
Level 2 trigger for H1 expt. at HEM
Monte Carlo data
Monte Carlo data
19-??-1 MLP+silicon
Proposed: simulations show a 10-fold background rejection and 10 psec execution time, suitable for level 2
Electron i.d. for LHC at CERN
Monte Carlo data
Monte Carlo data
192-96-1 MLP 92-92-1 silicon
Simulation studies show good rejection; prototype chip had propagation time of 15 nsec, suitable for LHC or SSC
Find primary vertex at E735 expt. Fermilab
Real collider data
Low level-offline Real 18-128-62 collider MLP data
Kink finding for charged particle tracks
Monte Carlo data
Z decay probabilities into b, c, and (uds)
Monte Carlo data
data
Physics process Real data from Delphi expt.
50-4-1 50-10-1 (b) MLP+ETANN
14-7-1 14-14-1 (res.) 42-6-1
determination 19-25-3 MLP oneoutput node for each of b, c, (uds)
Fifty-fold improvement in position resolution over conventional trigger; to be applied to DO expt. muon upgrade Currently installed in CDF experiment; 50-1 net is 100%efficient and triggers on real data; other 2 in tagging mode
Overlapping 18 wire sections summed; 3 times better resolution than TOF; finds multiple vertices naturally Both parameter and residual neural net methods exceed performance of standard chi-squared technique; parameter method gives 20x speedup
Z decay probabilities into b, c, and (uds) measured more accurately than with standard method
Neural Networks in High-Energy Physics
523
Table 6 Continued. Problem
Training set
Test set
Network
Results and comments
Quark/gluon discr. at CDF Fermilab
Monte Carlo data
Real data from CDF expt.
8-6-1MLP (feature map in new analys.)
Heavy overlap of quark/gluon distrib; first evidence for quark fraction increase with E,
B tagging
Monte Carlo data
Monte Carlo and real data
MLP MLP
Numerous references on b tagging, mostly at LEP
Track recons. with DenbyPeterson net
Deformable templates/ elastic arms
+ LVQ
Recurrent nets and track reconstr. Hand wired Real data Fully from connected Aleph recurrent expt. field network None
Monte Carlo data
Dynamic system
Neurons are links between hits; links form tracks as system settles; tested on Aleph expt. Inspired by DenbyPeterson net and elastic net methods; not really a neural network
of many square meters surrounding the other measuring devices, as in Figure 3. The drift chamber sense wires signals appeared on Time to Voltage Converters (TVCs)that convert the drift time of the ionization to the wire into a voltage. The setup is shown in Figure 5. The beam dump in the figure simulates the shielding iron of Figure 3. The small circles in the drift chamber volume represent the wires and the small horizontal lines above and below represent the TVC values interpreted as a drift distance. Note that there is an ambiguity as to on which side of the wire the particle passed. The neural net must resolve this ambiguity. The wires in Figure 5 are paired vertically. For each of the three pairs, two signals are produced: a drift time and a latch, indicating whether the lower or upper member of the pair was hit. The drift time signals had to be duplicated four times in order to achieve sufficient fanout for the analog neural net chip. These 12 signals were coupled with the three latch signals to form the 15 inputs to the neural network chip, configured as an MLP. Sixty-four hidden units in a single layer were used. The output layer consisted of 64 units divided in a group of 32 to encode slope and a group of 32 for intercept [this type of readout has been used in several previous studies of tracking with neural networks (Denby et al. 1990b; Lindsey and Denby 1991; Lindsey 199111. Each output unit covers .625 centimeters in intercept or .05 radians in slope. The network was trained
Bruce Denby
524
trigger counters
Drift chamber trigger . electr. -c readout computer
4-
readout motherboard
* * TVcsiADC's
T
V
ETANN output
'for
T ETA" board
digitisation'
Figure 5: Setup for the drift chamber neural net trigger test. on 10,000 tracks generated with a simple Monte Carlo, using gradient backpropagation. Target patterns consisted of gaussian histograms with means equal to the target slope and intercept and rms width of one bin. Architectures with fewer hidden units were also tried, but these resulted in degraded performance. (In an analog hardware network such as this, extra hidden units may be needed simply to increase fanout.) The weights obtained were downloaded into an Intel Electronically Trainable Analog Neural Network chip (ETANN) after performing emulation and chip-in-the-loop training using the Intel E T A " Development System (Intel 1991). The intercept position resolution available using the conventional trigger technique, which does not make use of the drift times, is 5 cm.The neural network trigger was found to have a position resolution of 1.2 mm. This resolution is only about a factor of two worse than the best obtainable offline using the complete reconstruction algorithm, but is available in about 8 psec. The neural network result, as shown in Figure 5, can be passed back to the readout motherboard for readout with the rest of the event information, without introducing dead time in the data acquisition system. 3.1.1.3 Future plans. The drift chamber used in the above tests was a prototype of chambers that are currently installed in the DO experiment at Fermilab (DO1983). A group on the DO experiment is currently installing an E T A " chip on one of their chambers to take test data during the 1992 run (Haggerty 1992). They also plan to incorporate the E T A "
Neural Networks in High-Energy Physics
525
readout into the trigger of the upgraded DO detector in the 1994 run of that experiment (Fortner 1992). This will allow a more accurate determination of the muon Pt, which will allow the threshold to be lowered and sigruficantly reduce the amount of background data recorded.
3.2.2 Test Case: the CDF Experiment. Neural network trigger hardware is installed for the current run of the CDF experiment. We describe below the conventional CDF calorimeter trigger and the neural network improvements to it. 3.2.2.2 Conventional techniques. The trigger for the CDF experiment at Fermilab has been in operation since the first experimental run in 1987 (CDF 1988). In this trigger, signals from the calorimeter cells appear as analog levels (i.e., voltages) at the ends of special 200-foot cables, where they are received by the trigger receiver boards. From this point on, the trigger can be thought of as operating on an array of voltages of size 24 (azimuthal angle) by 42 (pseudorapidity, related to polar angle) by 2 (electromagnetic/hadronic compartment), which represent the energies in the calorimeter. Analog processing is used in level 1 and level 2 for the cluster analysis, in which the total ET of the cluster, the number of towers in the cluster, and the cluster width are computed. Once the cluster analysis is finished, additional digital processing is performed, operating on the cluster quantities using the level 2 processors and special function modules, for example, calculation of the ratio of the cluster's energies in the electromagnetic and hadronic calorimeters. 3.1.2.2 CDF neural network triggers. The existing CDF calorimeter trigger is very powerful, but is based on the philosophy that clusters can be adequately described by their position, their width, the number of towers they contain, and the ratio of hadronic to electromagnetic energy they contain. Indeed, this information is adequate for a great many triggers. However, there are instances when a more sophisticated cluster analysis would be fruitful. A neural network trigger is currently installed at the CDF experiment (Wu et al. 1990; Denby et al. 1991; Badgett et al. 1992). For every cluster found by the cluster finder, the new trigger selects 5 by 5 trigger tower region of interest (in hadronic and in electromagnetic compartments) centered on the cluster and passes the 50 analog signals to analog neural network chips (Intel 1991). The chips are programmed to execute three different cluster algorithms: (1) determine if the cluster could be an isolated photon in the central calorimeter, (2) determine if the cluster could be an isolated electromagnetic shower in the endplug6 calorimeter, and (3) determine if the cluster could have come from the 6The endplug is a name given to calorimeters or other detectors which fit into the end openings of the cylindrical central detectors (Fig. 2).
526
Bruce Denby
Figure 6 Isolation templates for plug electron trigger. semileptonic decay of a b quark.7 None of these analyses would be possible using the existing calorimeter trigger without extensive hardware modifications. We choose the isolated endplug electron/photon trigger as a simple illustrative example. There is a very high rate of clusters in the endplug, which pass the conventional electron trigger but are in fact due not to electrons but to background processes. In the past, a high-energy threshold was used in the endplug in order to reduce the false positive rate. This, however, is undesirable since it rejects a significant number of real electrons and photons along with the background. Good electrons and photons are normally isolated in the calorimeter (i.e., have very little energy surrounding them). In 1992, an isolation requirement, implemented by a neural network, was tried in the level 2 trigger to allow the same trigger rate but with a lower energy threshold. Normally such a cut would have been made in the level 3 trigger. The conventional level 2 trigger cannot implement this cut since it no longer has access to the individual tower energies after cluster finding. The neural net endplug isolation trigger operates upon 5 by 5 tower regions of the electromagnetic and hadronic calorimeters as shown in Figure 6 (only the electromagnetic part is shown in the figure). The dark central region is meant to contain the electromagnetic shower, which normally produces a narrow cluster in one or two towers. Four templates are necessary since some of the shower's energy may spill over into 2 to 4 towers and since the center of the tower as found by the cluster finder may not perfectly center it in the 5 by 5 array in all cases. Each template will be represented as a hidden unit in the neural network, and each tower has a weight connecting it to one of these hidden units. Cells in the central region have a weight of F, and cells in the outer region have a weight of -1. Thus, the quantity presented to the hidden units, which are used as comparators, is
'A semileptonic decay is one in which a quark decays to a lepton plus other particles. In a purely leptonic decay, the quark decays to a charged lepton and a neutrino.
Neural Networks in High-Energy Physics
527
If this quantity is negative, the hidden unit will not “fire”: the energy outside the central region was greater than some fixed fraction of the central region energy and the cluster is thus not isolated. If the quantity is positive, the neuron fires, indicating an isolated cluster. If any of the templates fires, the cluster is isolated, that is, the output unit simply sums up the outputs of the hidden units. The value F = 0.16 was found to be optimum in the present application. (Since the network is very simple, and essentially “hand wired,” it was not necessary to train the network using, e.g., backpropagation.) Using this value, in a simulation of the trigger operating on real data from a previous CDF run, it was possible to lower the energy threshold for endplug electrons and photons from 23 to 15 GeV, while reducing background by a factor of 4 and retaining 95% of the signal. The trigger has been checked out in the current CDF run and appears to function as designed; the efficiency and background rejection are still being evaluated. The isolated central photon trigger operates in an analogous way, except that it operates in the central region of the calorimeter rather than the endplug, and in this case has only one template with a single tower in the central region of the 5 by 5 grid. This trigger provides access to a class of physics events containing so called “direct” photons, which tend to be isolated in the calorimeter. The trigger has been measured to be 100 percent efficient, and without the isolation cut, the high rate of background would severely limit the amount of good data which could be taken. In the case of the semileptonic b trigger, a Monte Carlo program was used to generate events containing the semileptonic b jets and background events not containing b jets. The semileptonic b jets will contain an electron as well as other particles, while the background jets will not contain electrons. A full detector simulation was used in order to model as closely as possible any instrumental effects. A training set was made from 5 by 5 regions centered on the b jets extracted from the signal and background events. This was used to train a feedforward neural network with 50 inputs, one hidden layer of 10 units and a single output unit to discriminate between b jets and non-b jets. This is the only one of the three CDF neural network triggers which uses a network trained with backpropagation. The other two are ”hand wired” nets. A simulation of the trigger showed a reduction of background of a factor of about 100 while retaining 30% efficiency for bs. The weights found in the simulation will be loaded into the neural network chip in order to allow online identification of the b jets. The performance of this trigger on real data in the current CDF run is still being evaluated. It would be impossible to carry out a discrimination such as this using conventional computer hardware within the time limits of the level 2 trigger (i.e., about 20 psec). The hardware for these triggers is installed. The central photon trigger is actually triggering the detector; the other two triggers are still ”tag-
Bruce Denby
528
ging" data taken on other triggers pending full checkout. All three of the triggers are implemented with identical hardware. It is remarkable that such different algorithms can be implemented with the same hardware simply by downloading different weights. Future modifications to any of the algorithms will also be easy because of the programmability of the neural net. 3.2.3 Other Trigger Applications.
3.2.3.2 The H2 experiment. The Hera accelerator, which collides electrons upon protons, is just coming on line at the time of writing. The experiments H1 and Zeus there will study the momentum distribution of constitutents within the proton and measure the coupling strength of the gluon to the different quarks. At Hera, the rate of produced events due to background processes such as interaction of a beam article with a residual gas molecule in the vacuum system is some 10 larger than the rate due to physics processes of interest. In the H1 experiment, a 4 level trigger system is envisioned in order to reduce this high rate to a manageable level of about 100 Hz. Level 1 is a digital pipeline which reduces the rate by about a factor of 100. An additional reduction of a factor of 10 is required in level 2 in order to provide an acceptable rate into levels 3 and 4, which are implemented in software on conventional computers. The level 2 trigger must complete its processing within 20 microseconds. A hardware neural network has been proposed as a solution to this problem (Ribarics et al. 1991; Ribarics 1992a,b). We describe the approach below. In level 1, 16 simple trigger quantities, such as total summed energy, total summed transverse energy, total energy in the central region of the calorimeter, etc., are compared to thresholds. Level 1 however ignores correlations among the input variables. More sophisticated cuts will be made in level 2 by augmenting the level 1 quantities with additional information which becomes available after the level 1 decision time and feeding the resulting list of variables to a feedforward neural network. At present 19 input variables, including energy sums in subsets of the calorimeter, information on the vertex position, number of charged tracks, etc. are used. The neural network will use these 19 vanables to determine whether the energy patterns in the event have come from an electron/proton collision or from a beam-gas collision or other background. The detailed architecture of the neural network is still under development, however typical results using Monte Carlo data with an MLP show retention of 98% of events from interesting physics processes and rejection of 90% of background events, that is, the reduction factor of 10 is achieved while maintaining excellent efficiency. The algorithm is planned to be executed by a Siemens MA16 neural network chip (Ra-
s
Neural Networks in High-Energy Physics
529
macher et al. 1991), which should be able to finish processing in 10 psec, well within the allocated time.
3.1.3.2 Trigger R&D at CERN. Some of the research and development projects at CERN are investigating neural networks for triggering applications for the LHC accelerator. In one project, a type of detector called a "transition radiation detector," TRD, was designed to tell electrons from pions in an online trigger (Hansen 1992). The TRD will have 192 input wires, embedded in a special substrate, which sense the passage of the electron. The analog values from these wires will be fed into an MLP with 96 hidden units, and one output unit which signals whether or not an electron was present. In a simulation, the TRD rejected 92% of pions, and accepted 90% of electrons. These results were better than the 89% rejection, 90% acceptance obtained with a more traditional analysis. Ultimately the neural network will be implemented in silicon with fixed weights. A prototype chip has already been built which has 32 input units and 32 hidden units. The propagation time through the chip is 15 nsec; thus, the processing is sufficiently fast for incorporation into a first level trigger for LHC or SSC. A group at the Dutch lab NIKHEF is investigating a calorimetry-based neural network trigger for the LHC accelerator as part of a research collaboration at CERN (Vermeulen 1992). The approach is similar to the CDF trigger in that it will perform simple pattern matching upon energy patterns in local regions of the calorimeter. This is a 2-year pilot project which will compare the neural net solution to other techniques. The exact hardware implementation is still under development but will probably use a fast digital signal processor to implement the neural network algorithm. 3.2 Other Low Level Pattern Recognition Applications.
3.2.1 Track Segment and Vertex Finding. This discussion is from Lindsey and Denby (1991) in which data from a proton antiproton collider experiment were fed to an MLP trained to find the primary vertex of the event, based upon drift times in the z-chamber, a drift chamber with three layers of wires placed near the beam pipe. The primary vertex is the point from which the tracks in the event emanate, and marks the location of the collision. Figure 7 shows the hits in the chamber for a typical event; here, only the hit wires are shown, not the drift times. The hits appear to emerge from a point on or near the beam line. The vertex position in collider experiments is normally not available online. This would, however, be very useful since it could be used to improve trigger calculations which assume a nominal vertex position at the center of the apparatus, and to flag or reject events that contain multiple interactions (i.e./ more than one primary vertex). Vertex calculations
Bruce Denby
530
Z Chamber Event
... Sense
Z Chamber
I
.
:
-5Ocrn
1
~
.....
Beam Line
Wires Hit Sense Wire
~
~ ,
.
.
u
..
. . . ... (
~
~
.
... . . .. .. I.... 4
-'g
Ocm
~
+50cm
0 Track Fit Vertex
+TOF
Vertex
...-'*...Neural Net
Ouput
x Max Net Output
Figure 7 A typical proton/antiproton collision viewed in the z-chamber of the E735 experiment. are normally not performed until the offline analysis. A cross check of the offline analysis is provided by the time-of-flight (TOF) system, which crudely measures the vertex position using timing information. The 288 sense wires of the chamber were broken up into 18 wire subsections (3 layers of 6 wires each) for processing by the network. The sets of 18 drift times became inputs to identical MLPs each with a single hidden layer of 128 units. Each output layer had 62 units, 60 representing 1.0-cm bins from -30 to 30 cm and 2 "overflow" units. The 18 input subnetworks were trained to represent the vertex position by a gaussian histogram in the output units, which gives good vertex position resolution with relatively few output units. Training was done using real data recorded in a previous run of the E735 experiment at Fermilab (E735 1991). Targets were obtained using the Z position of the vertex calculated using the standard offline algorithm. The 18 wire subsections were chosen so as to overlap in order not to miss tracks which may span subsections. The outputs of the subnets are then simply added. This is illustrated in Figure 8. Figure 9 compares the distribution of Zofnine- Z" to that of Z0mneZmF, where Z is the position along the direction of the beam particles. The neural network Z resolution is about 3 times better than TOF, and its performance can probably be even further improved by using additional wire layers in the chamber. TOF is currently analyzed offline. It might be possible to implement it online, but its resolution can probably not be improved because it is a technology which has already been pushed to its limits. Also, the TOF technique cannot handle cases of multiple vertices. The neural net treats these in a natural way: each vertex appears as a bump in the summed net output.
~
~
~
Neural Networks in High-Energy Physics
531
Figure 8: The z-chamber sits near the beam pipe to detect outgoing charged particles whose trajectories can be used to determine the vertex position. It is divided into 18 wire subsections, each with its own MLP, whose outputs are summed to give a distribution whose peak indicates the most probable vertex position.
3.2.2 Kink Recognition. A high-energy pion or kaon will sometimes decay in a tracking chamber volume into a muon and a neutrino. The neutrino is neutral and is not seen in the tracking chamber. The muon is charged and is seen, however has a different momentum from the original particle. The result is a track that appears to have a "kink" in it (Fig. 10). In this work (Stimpfl-Abele and Garrido 1991; Stimpfl 19921, simulated pion tracks of 3, 5, and 10 GeV momentum were generated and transported through a chamber modeled on that of the Aleph experiment at CERN. A detailed detector simulation was used to model noise hits and other instrumental effects. Two approaches were tried. In the first, helical track segments are fit to the hit positions in an inner region, 1, and an outer region, 2 (Fig. 10). The 5 helix parameters* in the two regions are then used as input to an MLP, which tells whether or not this track is due to a decay. In the second approach, a single fit is done to the track across both regions, and the residuals of the fit are used as input to the neural network. There will be 42 residuals, one for each sThe helix parameters are the Z position of the vertex, the polar and azimuthal angles of the axis of the helix, the radius, and the pitch.
532
Bruce Denby
measurement along the trajectory. As a variant to this second approach, groups of three residuals were averaged to give 14 residuals as input to the network. The results, are summarized in Table 7, which also shows the network architectures tried. Also given in the table is the result obtained with the standard method for kink identification, called the analytical x2 method, in which again the track is fit in two regions and a xz is calculated from the helix parameters in the two regions to determine the probability of the nonkink hypothesis. Both of the neural net methods are found to have higher efficiency than the standard chi-squared method. The neural network residual method is about 20 times faster to calculate than the analytical x2 method, assuming that the residuals are already available from the standard track fit. 3.2.3 Other Applications. A variety of other applications of neural networks to low level pattern recognition in high-energy physics have appeared, which we mention only briefly. The interested reader may consult the references. In an application to a Cherenkov’ detector, MLPs were used to find a set of dots forming a ring pattern in a noisy image (Altherr et al. 1992; deGroot and Los 1991). In another hardware application (Haggerty 1992), a discrete component hardware MLP was used to measure, in real time, the position of a muon track in a tracking chamber using charges induced on electrodes placed below the sense wire. MLPs have been used to perform electron/pion discrimination in a calorimeter (Garlatti Costa et al. 1992; Teykal 1992)and identification of heavy quarks using the presence of multiple vertices in a vertex tracking chamber (Gupta et al. 1991; Denby 1992). Applications to charged track reconstruction will be discussed in Section 5. 4 Physics Process Determination 4.1 B Tagging. Numerous groups have used neural networks for identifylng reactions containing b quarks. This is usually referred to as ”b tagging.” Typically this has been done at the four experiments at the LEI’ electron positron colliders (Proriol et al. 1991; Prorioll992; Bortolotto et al. 1991; deGroot and Los 1991; Gottschalk and Nolty 1991; Bellantoni et al. 1991; Seidel et al. 1992; Branchini et al. 1992; Brand1 19921, although some work with simulated jets at proton/antiproton colliders has also been reported (Denby et al. 1990). B tagging is of considerable interest since the properties of many particles containing b quarks have to date not been well studied. In the LEP work, the approach is typically to choose an ensemble of feature variables which describe the spatial distribution of energy within each jet and of the event as a whole. Additional 9 ACherenkov detector measures the mass of certain types of particles using the light the particle produces in passing through a transparent medium.
Neural Networks in High-Energy Physics
- 10
-6
533
5
0
10
cm
Figure 9 (a) difference between Zvertex as measured by the neural net and by the standard offline program, in centimeters. (b) Difference between Zvertex as measured by TOF counters and standard offline program. The neural net resolution is much better. Table 7 Efficiencies (in Percent) for Correctly Identifying Kinks (Defined in Text) in Pion Tracks of 3,5, and 10 GeV Momentum! Method
3GeV
5GeV
10GeV
5-5-1 (par) 5-10-1 (par)
78.9 79.1
67.0 67.2
53.5 53.6
14-7-1 (res) 14-14-1 (res) 42-6-1 (red
79.9 80.5 80.3
65.5 65.7 67.7
51.5 53.9 54.5
Analytical x2
76.0
62.0
40.2
'Two MLP architectureswere hied for the case of track parameters as net inputs, and three for the case of fit residuals as net inputs. The results for the standard method, analytical x2,are also given.
Bruce Denby
534
kink
Figure 10: A pion decays to a muon and a neutrino to produce what appears as a track with a "kink." The kink is recognized by comparing found track parameters in region 1 and region 2. information such as from vertex tracking chambers may also be included. We choose as an example of this type of study the analysis performed by members of the Delphi experiment, which extends the analysis to charm quarks and undifferentiated light quarks in order to extract the decay probabilities into these quarks of the Z boson. This analysis is described in the next section. 4.2 Decay Probabilities of the Z. The neutral boson Z can decay into any constituent plus its anticonstituent (e.g., electron plus positron, u quark plus Ci quark). The standard model dictates the types of interactions that the constituents can undergo, but the relative strengths of the various interactions must be verified experimentally A group from the DELPHI collaboration (one of the 4 major experiments at the LEP accelerator at CERN) has recently used a feedforward neural network to classify decays of the Z into three classes: CC pairs, bb pairs, or light quark (u, d, or +antiquark pairs (Cosmo et al. 1992; De Angelis 1992; Eerola 1992; see also Bortolotto et al. 1991). This classification has permitted a measurement of the probabilities of the Z to decay into these particles to be made with higher precision than was previously possible. The probability of the Z to decay into the leptons electron, muon, and tau has been well established. That measurement is "easy" to make since these particles are relatively easy to identify in the apparatus. The case of the decay of the Z into quarks is considerably more difficult since the final state quarks fragment immediately into jets. The problem then becomes
Neural Networks in High-Energy Physics
535
deducing the type of quark involved in the decay from the properties of the jets themselves and from their distribution within the apparatus. The standard technique for distinguishing heavy quarks from light quarks is through their so-called semileptonic decays, in which a particle containing a heavy quark decays to a lepton plus other particles. This technique has two disadvantages: (1) semileptonic decays account for only 20% of heavy quark decays, therefore with this technique it will be more difficult to obtain a sample large enough to assure small statistical errors; (2) in a semileptonic decay a neutrino is also emitted; these escape detection, making it impossible to completely reconstruct the event, leading to uncertainty in quark species in some cases. A technique that allows the use of all types of heavy quark decays is thus desirable. In the DELPHI work, 19 jet and event-shape variables were created as inputs to an MLP. The variables describe the spatial distribution of energy in the jets and in the event as a whole, various kinematical combinations of the momenta of the particles in the jets, as well as information about the presence of leptons in the event. An exact description of the 19 variables is not very illuminating to the nonspecialist; the interested reader is referred to the original works. The network architecture chosen had 25 hidden units and 3 output units to encode the three classes. The training data for the network were generated with a standard physics Monte Carlo program and a program that simulates the response of the DELPHI apparatus to particle collisions. A total of 6000 training events were used. An independent set of 200,000 events was used for testing the network. The trained network was then used to determine the relative fractions of b, c, and light quark decays in a sample of 123,475 real events from the DELPHI experiment. To do this, a two-dimensional representation of the network output was devised as follows. The values of the 3 output nodes were normalized to sum to 1. Each event can then be represented as a point within an equilateral triangle where the perpendicular distances of the point to the sides of the triangle represent the values of the output nodes. This type of representation is referred to as a Dalitz plot. Figure 11 shows the distribution within this plane of Monte Carlo events for b, c, and light quark decays, as well as for the real data. The fractions were obtained by fitting the real data distribution to a linear combination of the Monte Carlo distributions for the three classes:
R ( u , v ) = (1- F , - F b ) U i ( u , v ) + F , U p ( u , v ) + F b 1 1 ~ ( U , v ) where u,v are the variables defining the plane, R is the distribution of the real data, F, and F b are the fractions of decays containing c and b quarks, respectively, and U I , 112, and 113 are the distributions of the Monte Carlo data. The results of the fit are
Fc = 0.151 f0.008stat. f0.04lSyst, F b = 0.232 fO.OOSS,t, rt 0.017,yst.
536
Bruce Denby
Figure 11: Dalitz plots used to measure the relative fractions of b, c, and light (uds) quarks in the decays of the Z".The activation of the network output node corresponding to each class is represented as the perpendicular distance from the side of the triangle opposite the corner labeled with that class. The outputs of the three nodes always sum to 1. (a,b,c) The distribution of network outputs for Monte Carlo (uds), c, and b quarks, respectively. (d) The distribution for real data from Delphi. To extract the fractions of b, c, and (uds), the distribution in (d) is fit as a linear combination of the distributionsof (a), (b), and (c),where the coefficients in the linear combination are the desired fractions. where the first error is due to statistics, the second to an incomplete knowledge of certain parameters in the Monte Carlos, and the dependence of the result on which Monte Carlo model is used. For comparison, the best result to date for Fb (Abreu et al. 1992)using semileptonic decays is F b = 0.215 f0.017stat+systematic where the systematic error contains effects due to parameter and model
Neural Networks in High-Energy Physics
537
dependence. For the charm quarks, the best result to date (Abreu et al. 1990) is obtained by identifying a characteristic low energy pion from the decay of a particle containing a charm quark. The result is Fc = 0.162
+ -0.030stat + -0.050syst.
In the case of Fc, both statistical and systematic errors are better for the neural net result than for the semileptonic decay result. In the case of Fb, although the neural network result has a very slightly larger overall error, it is obtained with significantly lower statistics. This is because the neural net analysis allows all the data to be used in the analysis, while in the standard analysis, only the rarer semileptonic decays can be used. The neural network approach also has the advantage of providing a heavy quark probability on an event by event basis, whereas the semileptonic decay technique relies on global distributions for the entire data set. 4.3 QuarWGluon Separation. The ability to distinguish quark jets from gluon jets is clearly very desirable. The W and Z decay 80% of the time to two quarks, but normally these decays are unusable since it is not possible to distinguish these jets from the more copiously produced gluon jets. Furthermore, the most probable decay mode of the much sought top quark is into three quark jets, but this channel has long been considered unusable due to high backgrounds from multigluon final states. The ability to venfy three quark jets would dramatically reduce the background. Distinguishing quark jets from gluon jets has been thought by many high-energy physicists to be impossible due to the high degree of similarity between the two types of jets. Separation of quark and gluon jets using neural networks has been treated in a number of references (Lonnblad et al. 1990, 1991a; Bhat et al. 1990; Csabai et al. 1991; Baer et al. 1991; Barbagli et al. 1992). These results have been almost exclusively based upon data generated by Monte Carlo. Recently a new result from the Fermilab Tevatron collider has appeared (Bianchin et al. 1992a,b), which for the first time appears to give evidence of quark and gluon components in real jets produced in proton/antiproton collisions. In the Fermilab result, jets identified in the apparatus are represented by a set of 8 feature variables which describe the spatial distribution of energy within the jets, for example, the amount of energy contained within each of three concentric cones centered on the centroid of the jet, the rms width of the jet, etc. A backpropagation neural network with these 8 variables as inputs was trained to separate quark jets from gluon jets based on examples generated by Monte Carlo. It is necessary to use Monte Carlo since pure samples of quarks and gluons do not exist. The real data will always contain a mixture of quark and gluon jets, and in fact the relative ratio of quarks and gluons in various kinematical regions is one of the sought after results. For this reason, this problem too will suffer from the fact that the results will depend on which model of quark and gluon fragmentation has been used.
Bruce Denby
538
350
240
300
200
250 1 60
200 120
150
80
100
50
40
0
0
0.25
0.5
0.75
Pythia CC Et.gt.60
Pythio 00 Et.gt.60
900
0
C
Jet40 Et.gt.60
Figure 12: Output of 8-6-1 MLP for (a) Monte Carlo (the Pythia Monte Carlo was used in these studies) quarks, (b) Monte Carlo gluons, and (c) Real data from the CDF experiment (labeled “Jet 4 0 ) . All the jets are required to have Er greater than 60 GeV. The real data appear to be predominantly gluon-like with a small admixture of quarks, as expected from theoT.
There is considerable overlap of the two classes in all of the feature variables, and none is adequate to provide a useful classification of the jets. Figure 12 a and b shows the output of the trained neural network on independent test samples of Monte Carlo quarks and gluons. The quark and gluon distributions overlap substantially: quark and gluon jets are indeed very similar! However the separation achieved is useful because quark or gluon enriched samples can now be produced by placing cuts on the output of the neural network.
Neural Networks in High-Energy Physics
539
A study was made of the efficiency of the network, defined as the fraction of quark jets with network output above 0.5, as a function of the number of nodes in the hidden layer. Performance did not improve beyond the results with two hidden units, and in fact a simple perceptron (no hidden units) was only a few percentage points worse using this measure. However, the network output distributions for the zero and two hidden unit cases was much more gaussian in shape and did not include any regions in which the quark-to-gluon ratio was high. Such regions may prove valuable for placing cuts which enrich quark-to-gluon ratio at the price of reduced quark efficiency. For this reason, the results from the 6 hidden unit network were retained for the final analysis. The maximum efficiency achieved on the Monte Carlo data was 70%. Figure 12c shows the result of applying the trained net to a sample of real data from the CDF experiment. The real data distribution appears to be predominantly gluon-like with a non-zero admixture of quarks, which is consistent with the result expected on theoretical grounds for events in the kinematical regime in which the data was taken. A fit to the real data as a linear combination of the Monte Carlo quark and gluon distributions gives a good x2, but because of model dependence and some subtleties in the Monte Carlo programs, it has not yet been possible to extract the exact quark fraction from this distribution in an unambiguous way. However, the results are encouraging and work is continuing. More recently, another analysis was performed (Bianchin et al. 1992a) in which a feature map was trained on a sample of mixed Monte Carlo quarks and gluons and then used to identify quarks and gluons in an independent sample. A somewhat higher efficiency, about 72 percent was obtained. The feature map trained on Monte Carlo is also being applied to the real data, and, conversely, a feature map trained on real data is being applied to labeled Monte Carlo data. Training using only real data is very attractive since it avoids the problem of model dependence, although it may be necessary to use the Monte Carlo data to label the nodes in the topological map. These analyses are still in progress. 4.4 Additional Physics Process Applications. The use of learning vector quantization (LVQ) and topological maps is relatively new in HER An interesting application of topological maps appears in (Lonnblad et al. 1991b) in which a map is used to discover the b, c, and light quark classes in a sample of mixed Monte Carlo data. A similar application is being attempted for data at the Tevatron (Bianchin et al. 1992b). LVQ has been used for b tagging (Proriol et al. 1991; Proriol 1992) and discrimination of Tf events from background (Odorico 1991). Other MLP offline applications include resonance searches'O (Alexopoulos 1991), calculation ~~
''A resonance is a bound state of two or more particles and appears as a peak in a mass distribution.
540
Bruce Denby
Figure 13: Neuron links in the Denby-Peterson net. of the total mass of the particles in an event (Lonnblad et al. 1991b1, determination of the charge of the initial quark which produced a jet (Varela and Silva 1991), and identification of jet cascades with muons (Los 1992). 5 Neural Nets and Charged Track Reconstruction
5.1 Tracking with Recurrent Nets. Recurrent networks have been used in HEP for track reconstruction, using an algorithm developed by Denby and independently by Peterson (Denby 1988; Peterson 1989; Stimpfl-Abele and Garrido 1990; Denby and Linn 1990; Barbagli 1992). In this application a neuron is defined to be a directed link between two hits in a tracking detector. The approach resembles qualitatively the encoding used by Hopfield (Hopfield and Tank 1986) for solving the Traveling Salesman Problem with a recurrent net. The weight connecting two neurons i and j is determined by the angle Bi, between them (Fig. 13):
where li and l, are the lengths of the neurons (i.e./ distance between hits), wij = - B if i and j are head-to-head or tail-to-tail. An energy function is defined, E = -1 /2 C WijOiOj, where oi is the output of neuron i. The energy function will be smallest when the angles between close together neurons sharing points are small. This favors neurons lying along smooth trajectories such as those of particles moving in a magnetic field. The constraint term -B ensures a unique direction to the tracks to avoid a degeneracy that prevents settling of the network. The evolution of the system is if i and j do not both point into or out of the same point, and
Neural Networks in High-Energy Physics
541
Figure 1 4 Charged track reconstruction on real data in the Aleph central tracking chamber, using a recurrent neural network algorithm. In the top figures the beam pipe is perpendicular to the plane of the page, in the bottom figures, horizontal. The left-hand frames show the neuron links before evolution; at right are the found tracks at the end of evolution.
obtained by iteratively solving the update equations:
rduildt =
wVoi - ui;
oi = sigmoid(u;)
i
On each iteration, dt is kept much smaller than
7,the time constant of the system. This method has been used on real data at the ALEPH experiment at LEP (Stimpfl-Abele and Garrido 1990). Figure 14 shows r-phi (i.e., looking down the beam line) and r-z (side) views of an event in which a Z boson decays to hadrons, with all links defined before network evolution (left side of figure), and the event after settling of the network, with tracks found (right side). The efficiency is as good as the conventional
542
Bruce Denby
track reconstruction program but the neural net algorithm is somewhat faster. In this work, a study was made of execution time for the neural net and conventional algorithms as a function of track multiplicity (number of charged tracks in the event). The advantage of the neural algorithm is shown to increase with multiplicity. Although this type of algorithm has not yet been accepted as a standard track recognition algorithm, it may prove to be important in the future when track multiplicities will be larger. There is not a straightforward way to implement this algorithm in the fast hardware that would be needed to make it applicable at the trigger level, since the number of neurons and weights is high, and the weights must be recalculated for each event. In addition, the algorithm does not take advantage of all the available information, such as that tracks in a uniform magnetic field are known to be nearly perfect helices. This makes the algorithm more susceptible to noise since it will be less able to reject noise hits that happen to lie near the tracks. 5.2 Elastic Tracking. Improvement to the neural tracking are the socalled elastic tracking (Gyulassy and Harlander 1991) or deformable templates (Ohlsson et a/. 1991) approaches. In these approaches, a track is a helical object that settles into a shape which best fits the hits. The helix can be thought of as electrically charged and attracted to the hits which have opposite charge. Although these algorithms map the tracking problem onto dynamical systems, and are at least in principle parallelizable, they have lost some of the “neural” flavor of the original DenbyPeterson net. Nonetheless, the efficiency and robustness to noise of the elastic methods are excellent. One interesting study (Gyulassy and Harlander 1991) compared the robustness to noise of the standard method, the Denby-Peterson net, and the elastic tracking method. The standard method of track reconstruction is called the “roadfinder” since it starts with two nearby hits and then searches for additional hits on a ”road” in the direction of the segment joining them. Figure 15 from this study shows the efficacy of each method as a function of number of tracks. All data have 20% noise and 3% error on position measurement. The roadfinder breaks down between 5 and 10 tracks, the Denby-Peterson net at 10-15 tracks, but the elastic tracking always finds the correct answer in this study.
6 Conclusion
Five years ago there was no explicit mention of neural network techniques in HEP literature. A current bibliography of applications in HEP includes almost a hundred papers and reports. Much of the work is still exploratory and uses only the simplest techniques such as the MLP
Neural Networks in High-Energy Physics
543
Figure 15: Comparison of track reconstruction performance for the standard method (roadfinder),Denby-Peterson net, and elastic tracking.
trained with backpropagation, although some interesting results using learning vector quantization and feature maps have also appeared. In HER historically, data analysis has been done using simple onedimensional cuts; consequently the HEP community at large has yet to fully accept neural network techniques as standard tools. Nevertheless the neural network methods are beginning to show their worth. The decay probabilities of the Z boson into b, c and light quarks has been measured with higher precision than ever before using a technique based on an MLP. A neural network technique has given higher kink finding efficiency and faster execution speed than the standard method. Results consistent with identification of quark and gluon components in jets pro-
544
Bruce Denby
duced at a proton/antiproton collider have appeared for the first time using a feedforward neural network. Recurrent networks have provided a faster way of performing charged track reconstruction. One of the most exciting promises of neural network technology is in the realm of triggering for HEP. One test has already been completed: a VLSI neural network used in the data acquistion system of a drift chamber has provided, in only a few microseconds, track intercept resolution 50 times more accurate than that previously obtainable online. Neural network triggers for three large collider experiments are either currently installed or have been proposed for future installation. The neural network triggers will permit experiments to reject more background events earlier in the data stream, resulting in more efficient and cost-effective data acquisition systems and reduced data storage requirements. It is intellectually quite stimulating to witness a marriage between such seemingly disparate domains as high-energy physics and neural networks. Given the growth of applications and their success to date, HEP may turn out to be one of the driving forces in the integration of neural networks into science as data analysis tools.
References Abreu, P., et al. 1990. Phys. Lett. 2528, 140-148. Abreu, P.,et al. 1992. Measurement of the partial width of the Z" into bb final states using their semileptonic decays. CERN-PPE/92-89. Zeit. Phys. C 56, 47-62. AIHEP 1992. Proceedings of the Second International Workshopon Software Engineering, Artificial Intelligence, and Expert Systems for Nuclear and High Energy Physics, La Londe les Maures, France, January, World Scientific. Alexopoulos, T. 1991. Resonance searches using a neural network technique. Talk at DPF 91, Vancouver, Canada, August 1991, submitted to proceedings. Alexopoulos, T. 1991. Ph.D. Thesis, University of Wisconsin, unpublished. Altherr, T. 1992. Cerenkov ring recognition using adaptable and non-adaptable networks. New Computing Techniques in Physics Research, 11, D. Perret-Gallix, ed.,World Scientific. Amendolia, S. R., et al. 1990. Study of a fast trigger system on beauty events at fixed target and colliders. Nucl. Instrum. Methods A289, 539. Badgett, W., Burkett, K., Campbell, M., Wu, D. Y., Denby, B., Lindsey, C., Blair, R., Kuhlmann, S., and Romano, J. 1992. A neural network calorimeter trigger used in CDF. Proceedings of the 2992 IEEE Nuclear Science Symposium, Orlando, Florida. Baer, H., Karatas, D., and Giudice, G. 1991. Snagging the top quark with a neural network. FSU HEP 911130, Florida State University, Tallahassee, November. Barbagli, G., DAgostini, G., and Monaldi, D. 1992. Quark/gluon separation in
Neural Networks in High-Energy Physics
545
the photoproduction region with a neural network algorithm. Universita di Roma 'La Sapienza', Internal note N.992, February. Bellantoni, L., Conway, J. S., Jacobsen, J. E., Pan, Y. B., and Wu, Sau Lan, 1991. Using neural networks with jet shapes to identify b jets in e+e- interactions. CERN-PPE/91-80,24 May. Nucl. Instrum. Methods, A310: 618-622. Bhat, P., Lonnblad, L., Meier, K., and Sugano, K. 1990. Using neural networks to identify jets in Hadron Hadron collisions. Proceedings of the 1990 Summer Study on High Energy Physics-Research Directions for the Decade, Snowmass, Colorado, June 25-July 13. Bianchin, S., Denardi, M., Denby, B., Dickson, M., Pauletta, G., Santi, L., and Wainer, N. 1992a. Classification of jets from PPbar collisions at Tevatron energies. New Computing Techniques in Physics Research, 11, D. Perret-Gallix, ed., World Scientific. Bianchin, S., Dall'Agata, M., De Nardi, M., Pauletta, G., Santi, L., Denby, B., Wainer, N., and Dickson, M. 1992b. Jet classification at CDF. Proceedings of the Second Workshop, Neural Networks: from Biology to High Energy Physics, Elba International Physics Center, Isola dElba, Italy, 18-26 June. Int. I. Neural Syst., in press. Bock, R. K., et al. 1990. Feature extraction in future detectors. Nucl. Instrum. Methods A289, 534. Bortolotto C., Cosmo, G., DeAngelis, A., Linussio, A,, Eerola, P., and Kalkinnen, J. 1991. A measurement of the partial Hadronic widths of the Zo using neural networks. Proceedings of the Workshop Neural Networks: From Biology to High Energy Physics, Elba International Physics Center, Isola d'Elba, Italy, June 5-14, ETS Editrice, Pisa. Branchini, I?, Ciuchini, M., and Del Giudice, P. 1992. B tagging with neural networks: An alternative use of single particle information for discriminating jet events. N a o Computing Techniques in Physics Research, 11, D. Perret-Gallix, ed., World Scientific. Brandl, B., Falvard, A., Henrard, P., Jousset, J., and Proriol, J. 1992. Tagging of Z decays into heavy quarks in the Aleph detector using multivariate analysis methods: Neural networks, discriminant analysis, clustering. New Computing Techniques in Physics Research, 11, D. Perret-Gallix, ed., World Scientific. CDF 1988. The Collider Detector at Fermilab, a compilation of articles reprinted from Nucl. Instrum. Methods A, North Holland, Amsterdam. Cosmo, G., De Angelis, A., De Groot, N., Del Giudice, P., Eerola, P., Kalkkinen, J., Lyons, L., Los, M., Torassa, E., and Vallazza, E. 1992. Delphi Collaboration. Classification of the Hadronic decays of the Zo into b and c quark pairs using a neural network. XXVl Znternational Conference on High Energy Physics, Dallas, TX, August 5-12, submitted. Crosetto, D., and Love, L. 1992. Fully pipelined and programmable level 1 trigger. SSCL-576, SSC Laboratory, Dallas, Texas, July. Csabai, I., Czako, F., and Fodor, Z. 1991. Combined neural network-QCD classifier for quark and gluon jet separation. CERN Preprint CERN-TH.6038/91 and Eotvos University (Budapest) Institute for Theoretical Physics preprint ITP-Rep. Budapest 483, March.
546
Bruce Denby
De Angelis, A. 1992. Heavy flavour identification in Delphi. Proceedings of the Second Workshop, Neural Networks: From Biology to High Energy Physics, Elba International Physics Center, Isola d’Elba, Italy, 18-26 June, 1992. Int. 1. Neural Syst., in press. De Groot, N., and Los, M. 1991. B-tagging in Delphi with a feed-forward neural network. Proceedings of the Workshop Neural Networks: From Biology to High Energy Physics, Elba International Physics Center, Isola d’Elba, Italy, June 5-14, ETS Editrice, Pisa. Denby, B. 1988. Neural networks and cellular automata in experimental highenergy physics. Computer Phys. Commun. 49,429-448. Also, Denby, B. 1988. Neural network and cellular automata algorithms. Florida State University preprint FSU-SCRI-88-141, June. Tallahassee, Florida. Denby, B., Campbell, M., Bedesch, F., Chriss, N., Bowers, C., and Nesti, F. 1990a. Neural networks for triggering. IEEE Trans. Nucl. Sci. 37(2), 248-254. Denby, B., Lessner, E., and Lindsey, C. S. 1990b. Proceedings 2990 Conference on Computing in High Energy Physics, Santa Fe, NM, AIP Conf. Proc. 209, 211. Denby, B., and Linn, S. 1990. Computer Phys. Commun. 56, 293-297. Denby, B., Franklin, M., Kim, S. H., Konigsberg, J., and Timko, M. 1991. CDF Internal Note 1538. Proposal for a level-2 isolated plug electron trigger for the 1991/1992 run. CDF Collaboration, Fermi National Accelerator Laboratory, Batavia, Illinois. Denby, B. 1992. Quark flavor sensitivity of the mammalian cortex: Theoretical foundations. Proceedings of the Second Workshop,Neural Networks: From Biology to High Energy Physics, Elba International Physics Center, Isola d’Elba, Italy, 18-26 June. Int. J. Neural Syst., in press. DO Design Report. 1983. Fermilab, December; DO Upgrade. 1991. Fermilab P-823, April. Turkot, F., et al. 1991. Nucl. Phys. A525, 165-170. Eerola, P. 1992. Classification of the Hadronic decays of the Zo into b and c quark pairs using a neural network. Proceedings of the Second Workshop, Neural Networks: From Biology to High Energy Physics, Elba International Physics Center, Isola d’Elba, Italy, 18-26 June. Int. J. Neural Syst., in press. Farber, R. M., Kinnison, W., and Lapedes, A. S. 1991. A parallel non-neural trigger tracker for the SSC. LA-UR-91-607, Los Alamos National Laboratory, Los Alamos, NM. Fortner, M. 1992. Analog neural networks in an upgraded muon trigger for the DO detector. New Computing Techniques in Physics Research, II, D. Perret-Gallix, ed.,World Scientific. Garlatti Costa, P., De Angelis, A., Lanceri, L., Santi, L., Vignaduzzo, C., and Zoppolato, E. 1992. A neural network for e / p classification in a calorimeter. INFN Sezione di Trieste, Italy, technical note INFN/AE-92/14,27 April. Gottschalk, T. D., and Nolty, R. 1991. Identification of physics processes using neural network classifiers. Caltech Report CALT-68-1680. Gupta, L., Upadhye, A., Denby, B., and Amendolia, S. R. 1992. Neural network trigger algorithms for heavy quark selection in a fixed target high-energy physics experiment. Pattern Recog. 25, 413-421.
Neural Networks in High-Energy Physics
547
Gyulassy, M., and Harlander, M. 1991. Elastic tracking and neural network algorithms for complex pattern recognition. Computer Phys. Commun. 66, 31-46. Haggerty, H. 1992. Fermilab, private communication. Hansen, J. R. 1992. The need for neural networks at LHC and SSC. Proceedingsof the Second International Workshop on Software Engineering, Artificial Intelligence, and Expert Systems for Nuclear and High Energy Physics, La Londe les Maures, France, January, World Scientific, in press. Hopfield, J., and Tank, D. W. 1986. Science 233, 625. Intel. 1991. 80170NX Electrically Trainable Analog Neural Network, Intel Corporation, Santa Clara, California. Lea, R. M. 1988. ASP A cost effective parallel microcomputer. IEEE Micro, October. Lindsey, C. S. 1992. Drift chamber tracking with a VLSI neural network. Proceedings of the Second Workshop, Neural Networks: From Biology to High Energy Physics, Elba International Physics Center, Isola d’Elba, Italy, 18-26 June. Int. J. Neural Syst., in press. Lindsey, C. S. 1991. Tracking and vertex finding in drift chambers with neural networks. Proceedings of the Workshop Neural Networks: From Biology to High Energy Physics, Elba International Physics Center, Isola d’Elba, Italy, June 5-14, ETS Editrice, Pisa. Lindsey, C. S., and Denby, B. 1991. Nucl. Instrum. Methods A302, 217. Lindsey, C. S., Denby, B., Haggerty, H., and Johns, K. 1992. Nucl. Instrum. Methods A317, 346-356. Lonnblad, L., Peterson, C., and Rognvaldsson, T. 1900. Finding gluon jets with a neural trigger. Phys. Rev.Lett. 65,1321-1324. Lonnblad, L., Peterson, C., and Rognvaldsson, T. 1991a. Using neural networks to identify jets. Nucl. Phys. 8349, 675-702. Lonnblad, L., Peterson, C., and Rognvaldsson, T. 1991b. Mass reconstruction with a neural network. Lund University preprint LU TP 91-25, October. Phys. Lett. B, submitted. Lonnblad, L., Peterson, C., Pi, H., and Rognvaldsson, T. 1991~.Self organizing networks for extracting jet features. Computer Phys. Commun. 67,193-209. Los, M.1992. Using a neural network for classifymg jet cascades with a muon. Proceedings of the Second Workshop,Neural Networks: From Biology to High Energy Physics, Elba International Physics Center, Isola d’Elba, Italy, 18-26 June. Int. J. Neural Syst., in press. Cherubini, A., and Odorico, R. 1991. Identification by neural networks and statistical discrimination of new physics events at high energy colliders. Proceedings of the WorkshopNeural Networks: From Biology to High Energy Physics, Elba International Physics Center, Isola d’Elba, Italy, June 5-14, ETS Editrice, Pisa. Ohlsson, M., Peterson, C., and Yuille, A. 1991. Track finding with deformable templates-The elastic arms approach. Lund University Preprint LU TP 9127, November, Lund, Sweden. Track finding with neural networks. Computer Phys. Commun., submitted.
548
Bruce Denby
Peterson, C. 1989. Nucl. Instrum. Methods A279, 537. Proriol, J., et al. 1991. Tagging B quark events in Aleph with neural networks. Proceedingsof the Workshop Neural Networks: From Biology to High Energy Physics, Elba International Physics Center, Isola d’Elba, Italy, June 5-14, ETS Editrice, Pisa. Proriol, J. 1992. Tagging B quark events in e+e- colliders with neural networks. Comparisons of different sets of variables and different methods. Proceedings of the Second Workshop, Neural Networks: From Biology to High Energy Physics, Elba International Physics Center, Isola d’Elba, Italy, 18-26 June. Int. J. Neural Syst., in press. Ramacher, U., Beichter, J., Raab, W., Anlauf, J., Bruls, N., Hachmann, U., and Wesseling, M. 1991. Design of a first generation neurocomputer. In VLSZ Design of Neural Networks, U . Ramacher and U. Riickert, eds., pp. 271-310. Kluwer Academic Publishers. Ribarics, P., et al. 1991. Neural network trigger in the H1 experiment. Proceedings of the Workshop Neural Networks: From Biology to High Energy Physics, Elba International Physics Center, Isola d’Elba, Italy, June 5-14, ETS Editrice, Pisa. Ribarics, P. 1992a. Neural network level 2 trigger in the H1 experiment. New Computing Techniques in Physics Research, ZI, D. Perret-Gallix, ed., World Scientific. Ribarics, P. 1992b. Neural network trigger in the Hl experiment. Proceedings of the Second Workshop, Neural Networks: From Biology to High Energy Physics, Elba International Physics Center, Isola d’Elba, Italy, 18-26 June. Znt. J. Neural Syst., in press. SDC. 1992. The SDC Tech. Design Rept., SSCL-SR-1215, SSC Laboratory, Dallas, TX, 1 April. Seidel, F., et al. 1992. Extensive studies on a neural networks for b tagging and comparisons with a classical method. New Computing Techniques in Physics Research, 11, D. Perret-Gallix, ed., World Scientific. Stimpfl-Abele, G., and Garrido, L. 1991. Fast track finding with neural nets. Computer Phys. Commun. 64,46-56. Stimpfl-Abele, G., and Garrido, L. 1991. Recognition of decays of charged tracks with neural network techniques. Computer Phys. Commun. 67, 183-192. Stimpfle-Abele, G. 1992. Neural nets for kink finding. Proceedings of the Second Workshop, Neural Networks: From Biology to High Energy Physics, Elba International Physics Center, Isola d’Elba, Italy, 18-26 June. Int. J. Neural Syst., in press. Teykal, H. 1992. Using neural networks for the identification of electrons and pions in a calorimeter for high energy physics. Proceedings ofthe Second Workshop, Neural Networks: From Biology to High Energy Physics, Elba International Physics Center, Isola d’Elba, Italy, 18-26 June. Int. 1.Neural Syst., in press. Silva, l?, and Varela, J. 1991. Identification of the quark jet charge using neural networks. Proceedings of the Workshop Neural Networks: From Biology to High Energy Physics, Elba International Physics Center, Isola d’Elba, Italy, June 5-14, ETS Editrice, Pisa.
Neural Networks in High-Energy Physics
549
Vermeulen, J. 1992. A study of the feasibility of using neural networks for second level triggering at LHC. Neul Computing Techniques in Physics Research, 11, D. Perret-Gallix, ed., World Scientific. Wu, D., et al. 1990. CDF Internal Note 1310, A pattern recognition level-2 B trigger at CDF in 1991. CDF Collaboration, F e d National Accelerator Laboratory, Batavia, Illinois; and private communication. Received 5 August 1992; accepted 15 December 1992.
ARTlCLE
Communicated by Christof Koch
Stimulus-Dependent Synchronization of Neuronal Assemblies E. R. Grannan. D. Kleinfeld A T 6 T Bell Laboratories, Murray Hill, N1 07974 U S A
H.Sompolinsky Racah Institute of Physics and the Center for Neural Computation, Hebrew University, lerusalem, 91904 Israel, and A T 6 T Bell Laboratories, Murray Hill, NJ07974 U S A
We study theoretically how an interaction between assemblies of neuronal oscillators can be modulated by the pattern of external stimuli. It is shown that spatial variations in the stimuli can control the magnitude and phase of the synchronization between the output of neurons with different receptive fields. This modulation emerges from cooperative dynamics in the network, without the need for specialized, activitydependent synapses. Our results further suggest that the modulation of neuronal interactions by extended features of a stimulus may give rise to complex spatiotemporal fluctuations in the phases of neuronal oscillations. 1 Introduction A ubiquitous feature of the brain is the presence of widespread, rhythmic patterns of neuronal activity (Ketchum and Haberly 1991). One aspect of this activity is gamma oscillations in the visual cortex, with a frequency near 40 Hz, that are evoked by an external stimulus (Bouyer et al. 1981; Freeman and van Dijk 1987; articles in Schuster 1991). Singer and Gray and co-workers (Gray et al. 1989; Engel et al. 1991a,b) and Eckhorn et al. (1988) showed that the timing of these oscillations in a region of cortex can, under certain circumstances, be influenced by visual stimuli that lie outside its receptive field. In particular, the oscillations in regions with nonoverlapping receptive fields are synchronized when the direction of motion and orientation of stimuli presented to the individual fields are similar. Conversely, dissimilarity in these features results in a failure to synchronize. These results suggest that temporal coherence may be used to encode features of objects in multiple receptive fields. 'Present address: Department of Physics, Physical Sciences 2, University of California, Irvine, CA 92717.
Neural Computation 5, 550-569 (1993) @ 1993 Massachusetts Institute of Technology
Synchronizationof Neuronal Assemblies
551
Temporal synchrony across the visual cortex (Engel et al. 1991a) is most likely mediated by long-range axonal projections within the cortex. These long-range axonal projections appear to connect neurons with different receptive fields but similar orientation preference (Gilbert and Wiesel 1989). However, the mechanism by which the stimulus gates the influence of the long-range connections, and thereby modulates the synchrony of the oscillations, is unclear. Here we consider theoretically how spatial variations in an extended stimulus can modulate the interactions in a network of coupled neuronal assemblies. Each assembly consists of analog neurons that are extensively interconnected and produce oscillatory output as a result of inhibitory feedback. Neurons in different assemblies are coupled by relatively weak, feature-specific excitatory connections. The strength of all connections are fixed. Our work was motivated by the results of computer simulations (Sporns et al. 1989; Schillen and Konig 1991; Konig and Schillen 1991; Wilson and Bower 1991) and analytical studies (Schuster and Wagner 1991a,b; see also Aertsen ef al. 1989) that suggest that the magnitude and possibly the phase of the interaction between assemblies can be modulated by the stimulus. Other studies, however, suggested that these effects are insufficiently strong (Sporns ef al. 1991). Stimulus-dependent synchronizing interactions were postulated ad hoc in a previous theoretical study of a network of phase oscillators (Sompolinsky et al. 1990, 1991). It was shown that the emergent synchronization in the network was modulated by the extended properties of the stimulus in a manner similar to that found in experiments. In the present study we derive the form of the synchronizing interactions between the assemblies, and their dependence on spatial variations in the stimulus, from the full dynamics of the network. 2 Model
Our model describes the cooperative behavior of a network of weakly coupled clusters of neurons. Each cluster is analogous to a hypercolumn in primary visual cortex (e.g., Douglas and Martin 1990). It consists of neurons that respond to the presence of a stimulus within its receptive field. For simplicity, we limit ourselves to neurons whose response depends only on a single feature of the stimulus, namely the orientation of an edge. The time averaged value of this response has a pronounced peak at a particular orientation, referred to as the preferred orientation of the neuron. We assume that the preferred orientations are uniformly distributed among different neurons within each cluster. Each cluster contains two types of neurons. One, the excitatory cells, makes only excitatory connections on its postsynaptic target while the second, inhibitory cells, make only inhibitory connections. In our architecture, all
552
E. R. Grannan, D. Kleinfeld, and H. Sompolinsky
Figure 1: Schematic of the architecture of a network with two clusters. The open circles represent neurons that form only excitatory connections and filled circles represent neurons that make solely inhibitory connections. Only a representative fraction of the total number of connections is drawn. of the inhibitory neurons are equivalent and can be replaced by a “global” inhibitory cell. There are extensive connections between neurons in the same cluster but only sparse connections between neurons in different clusters. The architecture of a network with two clusters is shown in Figure 1. The dynamics of the network is described by circuit equations (Wilson and Cowan 1972;Amari 1972;Hopfield 1984):
Synchronizationof Neuronal Assemblies
553
where Z)R(B, t ) and vR(e,t ) are the potential and output, respectively, of the excitatory neuron with orientation preference 0 in the Rth cluster, N is the number of excitatory neurons in a cluster, U R ( f ) and &(t) are the potential and output of the inhibitory neuron, and time, t, is normalized by the neuronal time constant. The output of each neuron may be interpreted as its instantaneous rate of firing. It depends on the value of the potential through a nondecreasing function, which we assume is the same for all neurons:
vR(e,t ) = g[vR(d,t)]
and
UR(f)= g [ U R ( f ) ]
(2.2)
The form of g ( x ) is taken to be the logistic function g(x) = 1/{1 + exp [-4p(x - X O ) ] } , where P and xo correspond to the gain and threshold parameters of the neurons, respectively. The parameters JE and 11 denote the strength of the synapses from excitatory and inhibitory presynaptic neurons, respectively, to excitatory postsynaptic neurons within the same cluster. The parameter YE denotes the strength of the synapses from excitatory neurons to the inhibitory one. We have assumed that these parameters do not depend on the orientation preference of the the pre- and postsynaptic neuron. On the other hand, synapses between the excitatory neurons of two different clusters occur only between neurons with similar orientation preference. These synapses have strength E and their spatial dependence is specified by the function K(R - R’) = K(IR - R’l). The external input consists of two components. A time-independent part, I R ( e ) , encodes the orientation of a single edge within the receptive field of the Rth cluster. It is of the form I R ( e ) = I(le - eO(R)I)
(2.3)
where Bo(R)denotes the orientation of the stimulus and both 0 and 00 lie between 0 and K. For simplicity, only the excitatory neurons are taken to have external input. Temporal fluctuations in the input are denoted by a noise term, &(t). We have assumed that the noise is uniform within each cluster but varies between clusters (Sompolinsky et al. 1991). This noise competes with the interactions between different clusters and tends to destroy their relative synchrony. It is taken to be a gaussian variable with zero mean and variance (&(t)[Ri(f‘)) = ~ E &R’ T 6(t - f’)
(2.4)
where T is the strength of the noise relative to that of the intercluster connections. A basic assumption in our model is that the neurons within a cluster interact strongly among each other while the interaction between neurons in different clusters is weak. This implies that the value of E is small compared to that of J E , JL, and -1,. For this condition, we expect that the activity of a neuron is determined primarily by its connections to
554
E. R. Grannan, D. Kleinfeld, and H. Sompolinsky
neurons in the same cluster and by the stimulus within its receptive field. The dominant effect of the connections between clusters is to modulate the synchrony, that is, the relative phase, between neuronal activities in separate clusters. In our analysis of the model, we consider first the dynamics of a single cluster and ignore the interactions between clusters as well as the noise. We then derive the effective interaction between pairs of clusters, including the effects of noise, that results from their long-range connections. 3 Single Cluster Dynamics
The equations that describe a single cluster are (the label R is suppressed)
where V ( t )= (V(0,t ) ) e is the excitatory output averaged over all orientations, with (. . .)o = 10" dO/n. It is also useful to define the average value of the excitatory potential, u ( t ) = (v(0,t ) ) O , and the average value of the external stimulus, I = ( I ( 0 ) ) e . The potential of each excitatory neuron depends explicitly on its orientation preference only through the external stimulus (equation 3.1). Hence, neglecting transients, it can be expressed by
v(e,t ) = v(t) + q e ) - I
(3.2)
The mean output of the excitatory neurons can then be related to the mean excitatory potential by an instantaneous gain function, that is, V ( t )= G[v(t)], where G(u) = (g(u + I ( @ - I ) ) e
(3.3)
The equations for the average potentials v( t) and u(t) are found by averaging the equations for the cluster over all orientations (equation 3.1): (3.4) The dynamics of an isolated cluster is particularly simple in that all of the excitatory neurons have the same input except for a stationary, external contribution that depends on the orientation preference of the neuron (equation 3.2). Figure 2a shows the state diagram for the output (equation 3.4) in terms of the values of two parameters, the inhibitory synaptic strength JI and the average external input I, with all other parameters held constant. The value of the neuronal gain parameter /Iwas chosen to be large. In this limit the stable fixed points, whenever they exist, correspond to either
Synchronization of Neuronal Assemblies
555
an "OFF" state or an " O N state. In the "ON state all of the neurons are firing near their saturation level, that is, v(0,t) = v(0) x 1, while in the "OFF" state all of the neurons are essentially quiescent. In the region marked "ON + OFF", the network is stable in either state and the behavior depends on the initial condition. In the "OSC" region, almost all initial conditions will lead to oscillatory outputs, while in the "OSC + O N region, depending on the initial condition, the outputs will either oscillate or remain constantly active. An example of the output for an oscillatory state is displayed in Figure 2b. All the neurons oscillate with the same frequency, as implied by equation 3.2, but their average firing rates differ. The excitatory neurons with the greatest external input, that is, 0 x 60, are the first to fire within a cycle. They are followed by the neurons with weaker external input and the inhibitory neuron. The output of the inhibitory neuron gradually quenches the activity in the network until the external input again charges the excitatory neurons to a potential above their threshold level. Then the cycle begins anew. An important characteristic of a cluster is the tuning curve, that is, the average firing rate of an excitatory neuron, V ( 0 )= (V(0,t))t,where . ) t denotes an average over time. This quantity is a function of the difference between 0 and the orientation of the stimulus, 00. An evaluation of V ( 0 ) for different values of 60 yields the tuning curve shown in Figure 2c. For the particular form of the input we chose, the activity of the neuron is essentially zero for 10 - 001 > 40". (a
4 Phase Description of Interacting Clusters 4.1 Phase Equations. The dynamics of a network of interacting clusters can be greatly simplified in the limit that the intercluster coupling strength is small, that is, 6 << 1. First, equation 2.1 can be reduced to a set of closed mean-field equations that involve only the average excitatory and inhibitory potentials of the clusters, vR(t) and U R ( ~ ) respectively, , and the average input, I (Appendix A). Second, it can be shown (Winfree 1980; Kuramoto 1984) that the average potentials are of the form
(4.1)
where G(t)and u i ( t ) are the limit cycle solutions for the unperturbed cluster (equation 3.41, w is the frequency of the neuronal oscillation, and $ R ( t ) is the phase of the oscillation. In the absence of intercluster couplings, each of the $ R ( t ) are arbitrary constants whose values lies between 0 and 2n. The presence of a small coupling between the clusters induces temporal variations in the phases that are slow compared with the period
E. R. Grannan, D. Kleinfeld, and H. Sompolinsky
556
of the unperturbed limit cycle, that is, &(t) described by phase equations of the form
0
I
I
M
O(6). These variations are
U
7
I
I
a
ON
ON
+
i
OFF
G z W
0: I-
-10
u,
> U
!m?
I
\
+
ON
\
OFF
z
-20
n
t
osc
-2 0 AVERAGE EXTERNAL INPUT, I
1.0
m-
Y
' 0t
3
0
2
0.4
u I
Q
Y
>
0 1 2 -goo TIME,t (Units of 2 7 ~ 1 ~ )
O0 e-8,
goo
Synchronization of Neuronal Assemblies
557
where rRp($R - ll,~,)represents the pair-wise interaction between the phases $R(f) and ll,Rt(f), and UR(f) is a gaussian noise that originates from &(t) and has a variance with an amplitude of = T (see equation 2.4). The dynamics of the phases depends on the intercluster interaction Fm,(ll,R - l l , ~ ~ ) . 4.2 The Form of the Interaction between Phases. The interaction l?mt(ll,R- ll,p) is a periodic function, with period 2.rr/w. Its form depends on the structures of the unperturbed limit cycles of the Rth and R'th clusters. In our model the clusters are identical except for the orientation of their respective stimuli, &(R) and d0(R'). Thus I'm,(ll,R - ll,~,)depends on R and R' only through the relative orientation A00 = I&(R) - Bo(R')I. This implies that rmt($R - l l , ~ t ) = r(All,;A&), where All, = l l , ~- l l , ~ ~ . We have derived numerically, using the method of Kuramoto (19841, the form of F(All,;A&) from the mean-field equations for the potentials of the clusters (Appendix A). The results are shown in Figure 3a. Several features of r(All,; AOo) are apparent. First, it vanishes for AB0 > &, where
~
Figure 2 Facing page. Aspects of the dynamics of a single cluster. (a) A state diagram of the output of the network. The fixed parameters for the network were JE = 15, J E ~= 12, xo = 1.1, /3 = 3, T = 0, and E = 0 and the stimulus profile was 1(0 - 00) = Il + (Ih - II)l0 - e01/(~/2)with - It = 3.5. The value of Jr and the average input I = 1/2(4 + 11) were varied. The boundaries of existence of the different states were determined from numerical simulations of equations 3.3 and 3.4. However, since 0 is large, the boundaries for the " O N or "OFF"states are approximately straight lines that can be determined from equation 3.1 by a consistency analysis. In an "OFT" state the maximum potential of the excitatory neurons must be less than XO. Since v ( 0 ) = I ( 0 ) in this state, we require maximum[I(0)]< XO, which yields the vertical line. In the " O N state the minimal potential has to be larger than XO. In this state v(0) = JE Jr I(0) i, = 0 and V = U = 1, which leads to JE JI minimum[I(0)] 21 xo and yields the oblique line. The asterisk corresponds to the values J, = -7 and I = -0.25 used in the simulations for b-c. (b) The average firing rate of two excitatory neurons (upper panel) and the inhibitory neuron (lower panel) found from a simulation of the equations (equation 2.1) for a network with 60 neurons using the above parameters, except that we include noise amplitude T = 0.0006 (equation 2.4). We chose the initial conditions such that the neurons did not get stuck in an " O N state. The heavy line for the excitatory neurons refers to one with an orientation preference 0 - 00 = 3' while the thin line refers to one with 0 - 00 = 27". The period is 2x/w = 3.4. (c) The tuning curve, or time-averaged output of an excitatory neuron as a function of its orientation preference relative to the orientation of the stimulus. The average was calculated from simulations of the network with an averaging time of approximately 20 periods. The dots indicate the orientation preferences of the excitatory neurons featured in b.
+ +
+ +
E. R. Grannan, D. Kleinfeld, and H. Sompolinsky
558
0, is the full extent of the tuning curve (0, = 80" in Figure 2c). This can be understood by recalling that only neurons with the same orientation preference are connected by the intercluster couplings. Thus for AOo > 0, there are no pairs of active neurons that have the same orientation preference and the effective interaction between the clusters must vanish. Second, r(A$; Ado) is not monotonic in A&. Third, r(A$; A&,) is not an odd function of A$. This implies that the phase equations cannot be described in terms of a potential, that is, the interaction terms in equation 4.2 cannot be written in the form SW/B$R. This has important consequences on the dynamics of a network with many stimulated clusters (Discussion). The form of r(A$; Ado) indicates that it contains significant contributions from high harmonics in A$ (Figure 3a). We find that a good approximation is F(A$; A0,)
x
r0+ rl sin(A$ + a l )+ I'2sin(2A$ + (12)
(4.3)
as illustrated for A& = 15" in Figure 3a. The zeroth harmonic, with amplitude ro, represents the shift in the period of the oscillations and is presently irrelevant. The amplitudes rl and I'2 decrease monotonically with AOo and are zero for A00 > 0, (Fig. 3b). An unexpected result is the presence of large phase parameters, 01 and 02. They are nonzero even at A00 = 0" and increase with increasing values of A&. The nonzero value of the phase parameters appears to originate from the inhibitory feedback within each cluster. Specifically, the long-range excitatory synaptic input to weakly active neurons, that is, those stimulated away from their preferred orientation, increases the activity of these neurons and, in turn, increases the activity of the inhibitory neuron. This indirect activation of the inhibitory neuron contributes an "inhibitory" component to the interaction between the clusters. For sufficiently large values of AOo, all connections between a pair of clusters involve weakly active neurons and the inhibitory contributions dominate r(A$; A&). 4.3 Dynamics of 'Itvo Interacting Clusters. We consider in detail the case of only two interacting clusters, for which one can subtract the equation of $ R ' ( t ) from that for $R(t) (equation 4.2) to obtain an equation for the phase difference A$(t):
A$(t)
=
-F(A$;AOo)
+ V(t)
(4.4)
where f'(A.ri,;A00)= r(A$;A&) - F(-A$;A&) and V ( t ) = &(f). In the absence of noise, the steady-state solution of equation 4.4 is a fixed point and the phase difference between the two clusters will approach is determined from the constraints a constant, A&. The value of f'(A$o;A60) = 0 and Sl='(A$o;A8~)/8(A$)> 0. Note that f'(A$;A&)
Synchronization of Neuronal Assemblies
559
Kl i;m
0.4 W
n 3 k 1
a r a
a4
0
0"
45" RELATIVE ORIENTATION, As, 45"
90"
0"
90"
Figure 3 The interaction in a network of weakly coupled clusters. (a) The form of the effective long-range interaction between pairs of phase variables as a function of the relative difference in their phase (equation 4.2). Each curve corresponds to a different value of A&, the relative orientation of the stimuli. Note that the curves for A80 = 75" and 90" are essentially flat. The open symbols correspond to the form of F(A$J;15') given by its first three harmonics (equation 4.3).(b) The dependence of the amplitude and phase (radians) parameters for the first two harmonic of the interaction (a) on 0, (equation 4.3).
560
E. R. Grannan, D. Kleinfeld, and H. Sompolinsky
is an odd function of A$. Thus A$ = 0 and x are always zeros of f'( A$; A&), although they are not necessarily stable solutions. Further, the fixed points form degenerate pairs, &A$,,, that, except for the cases A,~J, = 0 or x , correspond to two different states. The shape of F(A$; A&) is shown in Figure 4a for several values of A&,. For small values the stable state is A& = 0. As A00 is increased (beyond 6" for our parameters) the fixed point moves to a nonzero, intermediate value of A$,,. As A00 is further increased (beyond 36") the stable fixed point becomes A&, = x and remains so until the force vanishes (beyond BOO). This behavior can be qualitatively understood by approximating F(A$J;A&)in terms of its first two harmonics (equation 4.31, that is, F(A$;AOo) x 2r1COSNI sin(A$) 2r2 cosa2sin(2A$). When the first harmonic dominates the interaction, as occurs when the value of aI is substantially smaller than x/2 and is substantially larger than r2,the phase difference is zero. This situation corresponds to small values of A0, (Fig. 3b and c). Similarly, a value of a1 near x leads to a phase difference of x , as occurs for large values of A&. When the value of a1 is near x / 2 , corresponding to intermediate values of Aso, the contribution from the first harmonic is of the same magnitude of that from the second. This gives rise to the pronounced anharmonic shape of f'( A$; A8) (Fig. 4a) and to an intermediate phase shift A$,, w cos-'( 4'1 cos 4 2 r 2 cos 0 2 ) . In the presence of noise, ij(t) in equation 4.4, the phase difference A$ fluctuates in time rather than approach a fixed value. The average phase coherence between the two clusters can be expressed by the intercluster correlation function
+
(4.5) with bVR(f)= V R ( ~ )(VR(t))fand, as before, denotes an average over time. The correlation function can be calculated from the unperturbed limit cycle of a single cluster and the phase dynamics (Appendix B). Since the clusters are identical, and the interaction between them is symmetric, an extremum will always occur at 7 = 0. The correlation function for several values of A00 are shown in Figure 4b-d. When both stimuli are aligned, that is, A& = 0", the Figure 4 Facing page. Aspects of the dynamics in a network with two clusters. (a) The force that acts on the difference between two clusters as a function of their relative phase difference (equation 4.2). Each curve corresponds to a different value of A&; those for A& = 75" and 90"are essentially flat. (b-d) The intercluster correlation function of the phase differencebetween two clusters as a function of time and different values of A00 (equations4.5 and B.4). The thin line refers to a low level of noise, 1/T = 33, and the thick line refers to an intermediate level, 1/T = 3.3. The network was equilibrated for approximately 150 periods and the correlation functions were averaged over an additional 150 periods.
Synchronizationof Neuronal Assemblies
561
correlation has a prominent peak at T = 0 (Fig. 4b). As the relative angle between the stimuli is increased, C M ~ ( Tdevelops ) a double peak that reflects the fluctuation of the network between two stable intermediate phase shifts. The presence of these phase-shifts also causes a minimum to occur at T = 0. These features are seen at A& = 30", for which
m0
I
I
I
-2a 0.4
I
I
'".
z 0
6 g z
o
a
W
$ I
3
j -0.4 a W
I-
z
a
8
0
21T
lT PHASE DIFFERENCE,A$
1.0
a W
k 3
cn
J
0 a
0
W
I-
I -1
0
1
-1
0
TI ME, T (Units of 27r/w
1
1
562
E. R. Grannan, D. Kleinfeld, and H. Sompolinsky
A$o N f1.3 (Fig. 4c). Note that, in practice, the intermediate phase shift may show up as a single peak at either a positive or a negative value of T if the activity of the network is averaged for only a short time. Lastly, the peak amplitude of CRR,(T)is not a monotonic function of A00 (cf. Fig. 4b-e). Further, while noise suppresses the amplitude of C,,(T) for any value of Ado, the suppression is greatest for intermediate values (cf. thick versus thin line in Fig. 4c). These features reflect the nonmonotonic behavior of I?(A$; A&) with respect to A00 (Fig. 3a). We calculated the equal-time correlation coefficient, CRR~(O), as a function of the relative orientation of two stimuli, A&, and for two levels of noise, ? (Fig. 5). The value of the coefficient rapidly decreases as a function of the relative orientation for either case. Beyond approximately the full-width at half-maximum of the tuning curve, 22" for our parameters (Fig. 2c), the coefficient becomes negative as a consequence of the substantial phase-shifts that occur for large values of A&. However, as the magnitude of the interaction is reduced for these angles, the corresponding magnitude of the coefficientsis also significantlyreduced, particularly at high levels of noise. 5 Discussion
Our main result is that a weak, fixed synaptic coupling between clusters of neurons can generate an effective interaction between the phases of their oscillatory response that depends strongly on the distribution of activity within each cluster. Thus the interaction is sensitive to the dissimilarity of the external inputs that stimulate the clusters. This result implies that stimulus-dependent synchronizing connections, postulated ad hoc in a previous network model of phase oscillators (Sompolinsky ef al. 1990,1991), can originate from neuronal dynamics without the need to invoke mechanisms of fast synaptic modification. This conclusion is consistent with the results of Konig and Schillen (19911, who simulated a network with time delayed connections and with the initial reports of Sporns et al. (1989). Our phase description is strictly valid only in the limit of weak intercluster coupling. In practice, the results of numerical calculations of the full equations for the model (equation 2.1) indicate that the phase model qualitatively describes the dynamics of the clusters even when the synaptic input a neuron receives via intercluster connections is about 5% of its total input ( E = 0.021~;data not shown). The time it takes to synchronize the output of two clusters from an initial, unsynchronized state is relatively short, about three cycles for this strength of interaction (insert to Fig. 5). In contrast to the ad hoc assumption in a previous work (Sompolinsky et al. 1990, 19911, the present analysis shows that dissimilarity in the external stimuli for each of the two clusters not only reduces the amplitude
Synchronization of Neuronal Assemblies
h
O
v
1.0
..h
7 A
I
U
563
h \
0.5
A
Ct-A--LA
ae, oo
A
0 v
i
0
a
0.5 TIME, t (Units of 2 r / w )
1
a
3
8 -0.5-I
I
I
RELATIVE ORIENTATION, Atlo
Figure 5: The equal-time intercluster correlation coefficient for the phases difference between two clusters as a function of A00 (equations 4.5 and B.4 with T = 0). This coefficient is a measure of the discrimination capability of the network. The thin line is for 1/T = 33, while the thick line is for 1/T = 3.3. The inset shows the amplitude of the coefficient during consecutive periods following the presentation of stimuli. Equations 2.1-2.4 were simulated numerically with the parameters used in the phase model (legend to Fig. 2a), l / T = 3.3 and E = 0.021E. Each datum reflects an average over 64 random initial conditions of the network. of their effective interaction but also induces a tendency to form phaseshifts. When only two clusters are stimulated, the phase-shifts appear in the intercluster correlation function (equation 4.5; Fig. 4c-e). Large differences in orientation between the two stimuli result in a phase shift of 7r (Fig. 4d and e). The phase shifts are less than 7r for intermediate differences in orientation and disappear for small differences. Our results with regard to the occurrence of phase shifts are in apparent contradiction to those of Schuster and Wagner (1990a). These authors studied the phase interaction between weakly coupled clusters of neu-
564
E. R. Grannan, D. Kleinfeld, and H. Sompolinsky
rons and claim that significant phase shifts do not occur. In contrast to the present work, the clusters in the model of Schuster and Wagner (1990a) had uniform external inputs and, further, their analysis was restricted to parameters near a Hopf bifurcation where the nonlinearities in the dynamics are weak. Our results are consistent with the simulations of Schillen and Konig (19911, where phase-shifts in the correlation between the output of two clusters are evident (see their Fig. 4). There is currently little experimental evidence for phase shifts among the oscillatory responses of neurons in visual cortex [but note Fig. Ig in Engel et al. (1991b)l. This is in apparent disagreement with the predictions of our model. One possibility is that the limit of weak, long-range coupling is inappropriate. Yet this limit is suggested from the experimental evidence on stimulus dependent synchronization across visual cortex (Eckhorn et al. 1988; Gray et al. 1989). In brief, stimuli outside the receptive field of a neuron may affect the cross-correlogram between it and other cells but these stimuli do not significantly perturb the magnitude or form of its autocorrelogram. This suggests that the effective interaction between distant neurons affects only their timing and not their rate of firing. A second possibility is that phase-shifts are particular to our choice of local architecture (Fig. 1). The numerical studies of Konig and Schillen (1991) make use of an architecture with solely excitatory connections plus synaptic delays, rather than inhibitory feedback. As mentioned above, the output of different clusters in their model exhibits phase-shifts. Further, Hansel et al. (1992) recently derived the form of the phase interaction between two Hodgkin-Huxley neurons. They show that shifts occur for a range of inputs with neurons coupled either by synapses or electrotonic junctions. Thus a body of evidence suggests that phase-shifts are a generic feature of the interaction between weakly coupled neuronal oscillators. There are a number of experimental issues that relate to the observation of phase shifts. The fully averaged cross-correlogramis symmetric in the presence of shifts. However, the cross-correlogram is likely to appear asymmetric when the averaging is incomplete so that only one of the two possible phases, A,$ = ~ c A(Fig. ~ J 4a), ~ dominates the interaction. Thus asymmetric cross-correlograms, traditionally interpreted as the signature of monosynaptic connections (Perkel etal. 19671, may in some cases reflect phase-shifted correlograms that have been averaged for too short a time. A second issue is that fluctuations in cortical activity may make shifts difficult to detect. The amplitude of the phase-shifted correlograms is expected to be reduced compared with correlograms without phase shifts (cf. Fig. 4a and c-e). This may significantly lower the signal-to-noise ratio of shifted cross-correlograms. However, even in the presence of noise stimulus-dependent phase-shifts should lead to a change in the shape of the cross-correlogram that depends on the form of the stimuli. Indeed, cross-correlogramswhose shape depends on the orientation of the stimulus have been observed (Ts’o et al. 1986). Lastly, both noise and variations
Synchronization of Neuronal Assemblies
565
in the intrinsic frequency of the oscillation will broaden the phase-shifted peaks in the correlogram. This may cause a shifted correlogram to appear as one with a relatively broad central peak. Such correlograms have been reported in recent studies (Nelson et al. 19921, although it is unclear if they result from the mechanism we propose. We suggest that the existence of phase shifts in the oscillatory part of neuronal responses to dissimilar stimuli deserves further experimental scrutiny. The presence of phase parameters can lead to dramatic changes to the phase dynamics (equation 4.2) when more than two clusters of neurons are stimulated. While the detailed behavior depends on the form of the intercluster interaction, r R R ' ( & - $ J R ~ )qualitative , aspects of the behavior may be accounted for by the simplified model E-'&(f)
=
-
C K ( R - R') J(At90)sin ($R
- $R!
+ a(A00))
R'#R
-k
VR(f)
(5.1)
Here At90 = OO(R) - Oo(R') is the relative orientation of the particular two of stimuli that act on a pair of clusters. The interaction parameter, J( A@,),measures the average overlap of the activities in a pair of clusters. It decreases monotonically with increasing values of At90 and vanishes for A& > &, where 0, is the full width of the tuning curve. Conversely, the phase parameter .(A&) increases monotonically with Ado. As before, K(R - R') specifies the spatial extent of the long-range connections (equation 2.1), and V R ( f ) is a gaussian noise (equation 4.2). The above model explicitly expresses the dependence of the amplitudes and phases of the interaction between the clusters on relative phases of each cluster on the spatial distribution of gradients in the orientation of the stimuli. When the phase parameters a(At90)are zero, as assumed in a previous work (Sompolinsky et al. 1990,1991), the network is unfrustrated. In the absence of noise the stimulated clusters will synchronize with zero phase shifts. In contrast, nonzero values of .(A&) may induce substantial frustration into the network and lead to a stable state with a complicated pattern of phase shifts. Further, the dynamics of the network is not governed by an energy function and thus the phases may not converge to fixed values. In cases where the values of phase parameters are large, such as when the stimulus contains sufficiently large spatial gradients, it is likely that the phases of each cluster, @ R ( t ) , will fluctuate chaotically in time. The phase model proposed here (equation 5.1)is likely to have validity beyond the specific architecture and dynamics of the circuit in the present work (Fig. 1). In fact, the simulation results of other circuits proposed for the 40 Hz oscillations in visual cortex (Sporns et al. 1991; Buhmann and von der Malsburg 1991;Konig and Schillen 1991; Wilson and Bower 1991) can be interpreted by this phase model. Thus, the model may provide a useful framework to probe the nature of spatiotemporal patterns of neuronal responses and their role in sensory processing.
566
E. R. Grannan, D. Kleinfeld, and H. Sompolinsky
Appendix A Here we sketch our derivation of the phase description from the full dynamics of the network. The equations for the dynamics of the full network (equation 2.1) are first reduced to a set of equations that involve only the potentials U R ( ~ )and , U R ( t ) , the input I, and the noise & ( t ) within the clusters. This is accomplished by averaging equation 2.1 over all orientations 8, so that
To close these equations, one must obtain a relationship between the average output V Rand the average potentials, U R ( ~ and ) U R ( f ) . Subtracting equation 2.1 from equation A.l and expanding all terms to first order in E yields
where
with g‘(x) = dg/dx and & ( x ) f g ( x ) - (g(x))e. Substitution of equations A.2 and A.3 into equation A.l gives
Synchronizationof Neuronal Assemblies
567
where
+
Y R R ! ( V RU, p ; f ) = K ( R - R’)G[UR(~)] YRR~(VR U .p ;
(A.5)
t)
and G(x) is defined by equation 3.3. The intercluster interaction term in equation A.4 is nonlocal in time. However, for small values of E the potentials can be approximated by vR(t) = %[wt $ R ( f ) ] and vRl(t) = G , [ w t $RJ(t)],where we have used equation 4.4 and the fact that the phases vary slowly on the time scale of the period for the oscillations, 27r/w-’. Substituting this form into equation A.4 results in equations that are now local in time. These equations represent a system of weakly coupled, two-dimensional limit cycles. By an appropriate average over the fast variables, they are further reduced to a set of equations (equation 4.2) that involves only the slow, phase variables, $ R ( f ) (Fig. 3). For details of this reduction, see Kuramoto (1984).
+
+
Appendix B Here we sketch our calculations of the intercluster correlation function (equation 4.5) in terms of the phase dynamics. The correlation can be expressed as
where (. . .), denotes averaging over time and over the noise in the phase equations (equation 4.2) and GV;;[wt $‘R(f)] = V { [ w t $ R ( f ) ] - ( V R ( f ) ) l where V;;[wt+ $ R ( t ) ] x G { G [ w f+ $ R ( f ) ] } is the solution of the equations for the unperturbed cycle (equations 2.2, 3.3, and 3.4). If we restrict ourselves to values of T that are on the order of w-I, we can make the approximation $R, ( t T ) x $,Q(t). , In this limit C R R( T~) depends only on fluctuations in the phase difference A$(t) = $Rf(t) - @ R ( t ) . For the case when only two clusters are stimulated, the form of equation 4.4 implies that the equilibrium distribution of the phase difference, a stochastic variable, is of a Gibbs’ form, that is,
+
+
+
D(A$)
(8.2)
o< ,-W(A@)/ZT
where the potential W(Ali,) is given in terms of the interaction f’(A$; Ado) (equation 4.4) by
1
A$
W(A$)
=
0
d$r($; Ado)
(8.3)
We thus arrive at
C R R ~ ( T=)
+ +
Jp(d$/27r)D($) Jo2””-’(wdt/27r)GV;;(wt)GV;;,(wt WT qj)
([W3t)12)r
Note that this result for C , ~ ( T ) is valid only for values
T
x
0 (w-’).
(8.4)
568
E. R. Grannan, D. Kleinfeld, and H. Sompolinsky
Acknowledgments
We thank B. Friedman, C. D. Gilbert, D. Hansel, J. A. Hirsch, P. C. Hohenberg, P. Konig, 0. Sporns, and D. Y. Ts'o for useful discussions. D. K. and H. S. thank the Aspen Center for Physics for its hospitality. This work was supported, in part, by the Fund for Basic Research administered by the Israeli Academy of Arts and Sciences and by the U.S.-Israel Binational Science Foundation.
References Aertsen, A. M. H. J., Gerstein, G. L., Habib, M. K., and Palm, G. 1989. 9namics of neuronal firing correlation: Modulation of effective connectivity. I. Neurophys. 61,900-917. Amari, S.-I. 1972. Learning patterns and patterns of sequences by self-organizing nets of threshold elements. IEEE Trans. Comp. 21, 1197-1206. Bouyer, J. J., Montaron, M. F., and Rouged, A. 1981. Fast fronto-parietal rhythms during combined focused attentive behaviour and immobility in cat: Cortical and thalamic localization. Electroenceph. Clim. 51, 244-252. Buhmann, J., and von der Malsburg, Ch. 1991. Sensory segmentation by neural oscillators. In Proceedings of the Znternational Conference on Neural Nefworks, pp. 603-607, Vol. 11. Douglas, R. J., and Martin, K. A. C. 1990. Neocortex. In Synaptic Organization ofthe Brain, 3rd ed., G. M. Shepherd, ed., pp. 356-438. Oxford University Press, New York. Eckhorn, R., Bauer, R., Jordan, W., Brosch, M., Kruse, W., Munk, M., and Reiboeck, R. J. 1988. Coherent oscillations: A mechanism of feature linking in the visual cortex? Multiple electrode and correlation analysis in the cat. Biol. Cybern. 60, 121-130. Engel, A. K., Konig, P., Kreiter, A. K., and Singer, W. 1991a. Interhemispheric synchronization of oscillatory neuronal responses in cat visual cortex. Science 252, 1177-1180. Engel, A. K., Kreiter, A. K., Konig, P.,and Singer, W.1991b. Synchronization of oscillatory neuronal responses between striate and extrastriate visual cortical areas of the cat. Proc. Natl. Acad. Sci. U.S.A. 88, 6048-6052. Freeman, W., and van Dijk, B. W. 1987. Spatial patterns of visual cortical fast EEG during conditioned reflex in a rhesus monkey. Brain Res. 422,267-276. Gilbert, C. D., and Wiesel, T. N. 1989. Columnar specificity of intrinsic horizontal and corticocortical connections in cat visual cortex. j . Neurosci. 9, 2432-2442. Gray, C. M., Konig, P.,Engel, A. K., and Singer, W. 1989. Oscillatory responses in cat visual cortex exhibit intercolumnar synchronizationwhich reflects global stimulus properties. Nature (London) 338,334-337. Hansel, D., Mato, G., and Meunier, C. 1992. Phase dynamics for weakly coupled model neurons. CNRS preprint. Hopfield, J. J. 1984. Neurons with graded response have collective computational properties like those of two-state neurons. Proc. Natl. Acad. Sci. U.S.A. 8,3088-3092.
Synchronizationof Neuronal Assemblies
569
Ketchum, K. L., and Haberly, L. B. 1991. Fast oscillationsand dispersive propagation in olfactory cortex and other cortical areas: A functional hypothesis. In Olfaction: A Model System for Computational Neuroscience, J. L. Davis and H. Eichenbaum, eds., pp. 69-100. MIT Press, Cambridge, MA. Konig, P., and Schillen, T. B. 1991. Stimulus-dependent assembly formation of oscillatory responses: I. Synchronization. Neural Comp. 3, 155-166. Kuramoto, Y. 1984. Chemical Oscillations, Waves, and Turbulence. Springer-Verlag, New York. Nelson, J. I., Salin, P. A., Munk, M. H., Arzi, M., and Bullier, J. 1992. Spatial and temporal coherence in cortico-cortical connections: A cross-correlation study in areas 17 and 18 in the cat. Visual Neurosci. 9,21-37. Perkel, D. H., Gerstein, G. L., and Moore, G. P. 1967. Neuronal spike trains and stochastic point processes. 11. Simultaneous spike trains. Biophys. J. 7, 419-440. Schillen, T. B., and Konig, P. 1991. Stimulus-dependentassembly formation of oscillatory responses: 11. Desynchronization. Neural Comp. 3, 167-178. Schuster, H. G. 1991. Nonlinear Dynamics and Neuronal Networks: Proceedings of the 63rd W. E . Heraues Seminar, Friedrichsdorf 1990. VCH, New York. Schuster, H. G., and Wagner, P. 1990a. A model for neuronal oscillations in the visual cortex: I. Mean-field theory and the derivation of the phase equations. Biol. Cybern. 64,77432. Schuster, H. G., and Wagner, P. 1990b. A model for neuronal oscillations in the visual cortex: II. Phase description of the feature dependent synchronization. Biol. Cybern. 64, 83-85. Sompolinsky, H., Golomb, D., and Kleinfeld, D. 1990. Global processing of visual stimuli in a network of coupled oscillators. Proc. Natl. Acad. Sci. U.S.A. 87,7200-7204. Sompolinsky,H., Golomb, D., and Kleinfeld, D. 1991. Cooperative dynamics in visual processing. Phys. Rev. A15 43,6990-7011. Sporns, O., Gally, J. A,, Reeke, G. A., and Edelman, G. M. 1989. Reentrant signaling among simulated neuronal groups leads to coherency in their oscillatory activity. Proc. Natl. Acad. Sci. U.S.A. 86, 7265-7269. Sporns, O., Tononi, G., and Edelman, G. M. 1991. Modeling perceptual grouping and figure-ground segregation by means of active reentrant connections. Proc. Natl. Acad. Sci. U.S.A. 88, 129-133. Ts’o, D. Y., Gilbert, C. D., and Wiesel, T. N. 1986. Relationshipsbetween horizontal interactions and functional architecture in cat striate cortex as revealed by cross-correlation analysis. J. Neurosci. 6, 1160-1170. Wilson, W. A., and Bower, J. A. 1991. A computer simulation of oscillatory behavior in primary visual cortex. Neural Comp. 3, 498-509. Wilson, H. R., and Cowan, J. D. 1972. Excitatory and inhibitory interactions in localized populations of model neurons. Biophys. ].12,1-24. Winfree, A. T. 1980. The Geometry of Biological Time. Springer-Verlag,New York. Received 4 February 1992; accepted 25 January 1993.
This article has been cited by: 1. B. Pesaran, D. Kleinfeld. 2009. Enter the ratrix. Proceedings of the National Academy of Sciences 106:46, 19209-19210. [CrossRef] 2. Demian Battaglia, Nicolas Brunel, David Hansel. 2007. Temporal Decorrelation of Collective Oscillations in Neural Networks with Local Inhibition and Long-Range Excitation. Physical Review Letters 99:23. . [CrossRef] 3. K. Pyragas, O. V. Popovych, P. A. Tass. 2007. Controlling synchrony in oscillatory networks with a separate stimulation-registration setup. Europhysics Letters (EPL) 80:4, 40002. [CrossRef] 4. Sitabhra Sinha, Jayanta Basak. 2002. Dynamical response of an excitatory-inhibitory neural network to external stimulation: An application to image segmentation. Physical Review E 65:4. . [CrossRef] 5. P. A Tass. 2002. Effective desynchronization with a stimulation technique based on soft phase resetting. Europhysics Letters (EPL) 57:2, 164-170. [CrossRef] 6. M.B.H. Rhouma, H. Frigui. 2001. Self-organization of pulse-coupled oscillators with application to clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence 23:2, 180-195. [CrossRef] 7. Bard Ermentrout. 1998. Reports on Progress in Physics 61:4, 353-430. [CrossRef] 8. U. Ernst, K. Pawelzik, T. Geisel. 1998. Delay-induced multistable synchronization of biological oscillators. Physical Review E 57:2, 2150-2162. [CrossRef] 9. Roger Rodriguez, Henry Tuckwell. 1996. Statistical properties of stochastic nonlinear dynamical models of single spiking neurons and neural networks. Physical Review E 54:5, 5585-5590. [CrossRef] 10. S. Campbell, DeLiang Wang. 1996. Synchronization and desynchronization in a network of locally coupled Wilson-Cowan oscillators. IEEE Transactions on Neural Networks 7:3, 541-554. [CrossRef] 11. H. Sompolinsky , M. Tsodyks . 1994. Segmentation by a Network of Oscillators with Stored MemoriesSegmentation by a Network of Oscillators with Stored Memories. Neural Computation 6:4, 642-657. [Abstract] [PDF] [PDF Plus] 12. D. Hansel, C. Meunier. 1993. Clustering and slow switching in globally coupled phase oscillators. Physical Review E 48:5, 3470-3477. [CrossRef]
Communicated by Laurence Abbott
Dynamics of Populations of Integrate-and-Fire Neurons, Partial Synchronization and Memory Marius Usher Computation and Neural Systems 226-76, Caltech, Pasadena, C A 91125 USA
Heinz Georg Schuster Institut fiir Theoretische Physik, Universitat Kiel, 0-2300 Kiel 2, Germany
Ernst Niebur Computation and Neural Systems 226-76, Caltech, Pasadena, C A 91125 USA
We study the dynamics of completely connected populations of refractory integrate-and-fire neurons in the presence of noise. Solving the master equation based on a mean-field approach, and by computer simulations, we find sustained states of activity that correspond to fixed points and show that for the same value of external input, the system has one or two attractors. The dynamic behavior of the population under the influence of external input and noise manifests hysteresis effects that might have a functional role for memory. The temporal dynamics at higher temporal resolution, finer than the transmission delay times and the refractory period, are characterized by synchronized activity of subpopulations. The global activity of the population shows aperiodic oscillations analogous to experimentally found field potentials. 1 Introduction
Most artificial neural networks are based on binary McCulloch-I'ittG neurons that have no intrinsic temporal characteristics. As opposed to these simplified units, real neurons perform temporal integration over their inputs with some specific decay constant and have refractory periods. Model neurons satisfying these constraints are often called integrate-andfire neurons. Although the dynamic behavior of a single such neuron is straightforward, their population dynamics are highly complex. Recently, several studies have shown that under specific assumptions analytical results characterizing such populations of neurons can be obtained. For example, Amit and Tsodyks (1992) have shown that on a longer time scale the dynamic behavior of integrate-and-fire neurons can be averaged Neural Computation 5, 570-586 (1993) @ 1993 Massachusetts Institute of Technology
Populations of Integrate-and-FireNeurons
571
out and characterized by continuous variables (firing rates or currents). However, this transformation is based on the assumption that there is no synchronicity among inputs and once made, the question of dynamics at higher temporal resolution cannot be addressed. Gerstner and Van Hemmen (1992) proposed a stochastic model for spiking neurons that incorporates refractory periods and transmission delays, but which disregards membrane decay properties. The issue of temporal modulations has been investigated by Mirollo and Strogatz (1990), who showed that a population of integrate-and-fire neurons in a strong external field (receiving a strong external input and acting as pulse oscillators) will phaselock and reach synchronicity if they have excitatory all-to-all connections. Van Vreeswijk and Abbott (1992) analyzed the behavior of completely connected populations of excitatory neurons and found that even in the absence of external input, such populations of neurons can exhibit sustained activity due to self-interaction. In particular, they showed that, in the absence of noise, the population may lock in one of several patterns of cyclic activity. The purpose of this work is to further analyze the dynamic characteristics of such populations in the presence of noise and small external inputs, and to find the specific way that a population responds to different inputs. In the first part we develop a discrete-time mean-field model, using a master equation that takes explicitly into account the stochastic behavior due to noise. Using the master equation and numerical simulations we show that for some values of the external field the population is dominated by two attractors, a self-sustained activity state and a silent state. These states are fixed points of the dynamics, and we discuss their basins of attraction. We also show that the network exhibits hysteresis and thus can function as a memory system that reacts to external inputs in a different way than systems consisting of McCulloch-Pitts neurons. In the last section, we study the temporal dynamics at a finer temporal resolution. Taking explicitly into account axonal and synaptic delays and refractory periods, we obtain synchronized activity of subpopulations and aperiodic oscillations of the whole population's activity.
2 The Model
Consider a fully connected population of N excitatory neurons, each one characterized by a continuous variable t i that represents the cell's potential (1 5 i 5 N). Each neuron integrates over its inputs, and once it reaches a threshold (chosen without loss of generality to be l),the neuron fires, sends its output to the other neurons in the population, and resets its potential to zero. For simplicity, we will use a synchronous updating rule where each iteration step corresponds to a typical time for the spike dynamics (including the refractory period) and transmission delays of
M. Usher, H. G. Schuster, and E. Niebur
572
about 2-3 msec. Thus for neuron i,
where X is a decay constant (0 5 X 5 l ) , I > 0 is the coupling constant, 8 is the unit step function, and the external input is characterized by a mean value h and a gaussian noise term (i (of zero mean and standard deviation (5). The first step function adds the contributions from other cells that are firing at time t, and the second step function enforces refractoriness (a neuron cannot fire at two consecutive time steps). In order to test the validity of our synchronous discretization scheme, we have to test whether its dynamic properties depend upon the length of the iteration steps (in the limit of infinitesimal time steps, our scheme should reduce to a continuous system of differential equations representing currents in the cells). As it stands, equation 2.1 implies that both the delay time and the refractory period are equal to the discretization steps. Thus in order to increase the temporal resolution (without changing the actual values of the biological parameters, i.e., refractoriness and delay time), equation 2.1 should be modified, so that the refractory period and the delay time extend over more than one (now smaller) time bin. We shall show in the last section that the results obtained from equation 2.1 (at low time resolution) are compatible with those obtained from the modified delayed equation. However, at the higher resolution new phenomena are revealed. As shown in Van Vreeswijk and Abbott (1992) for zero external input and noise (h = ( i = O), the population may either remain silent or segregate into a cyclic pattern of activity of length M. At time t a subgroup of n' cells fire and subsequently, at time t 1, a different subgroup of n'+' cells will fire etc., satisfying Ci+'nr = N. For X = 1 the length of the cycle M is limited (Van Vreeswijk and Abbott 1992) by: IN < M < - 2IN (2.2)
+
IN - 1
NI - 1
Although the length of the cycle is limited by this equation, this does ~, denot determine the sequence of activations nr,n'+', . . . ,t ~ ' + which pends on the initial conditions zi(0). As we shall show in the next section, under the influence of noise the temporal behavior is asymptotically characterized by homogeneous sequences [i.e., n ( t ) = n(t 1 ) .. . = n(t M ) ] . Thus the system converges to one of two possible fixed points, a homogeneous sequence of activations or the silent state. This situation is somewhat analogous to a ferromagnetic system.
+
+
3 Master Equation
The stochastic system in equation 2.1 is completely characterized by the . . . ,ZN). Since our system is fully full probability distribution P'(ZI,Z~,
Populations of Integrate-and-FireNeurons
573
connected, a mean-field approach yields a lower dimensional-probability distribution P'(z), 1 P'(z) = -
c J 6(z
- Zi)P'(Zl
N i
*
..
3
where Pf(z) is the probability that neuron i will have the synaptic potential z at time t, and P'(z) is the mean-field average probability of having a postsynaptic potential z in the system at time t. The time dependence of P'(z) is obtained from equation 2.1:
P'+'(z) = m'6(z)+
/
1
-m
dz' < 6[z- (Xz'
+ Jm'+ h + t)]> P'(z')
(3.1)
where m', the normalized fraction of active cells at time t is related to P'(z) via m' = JyP'(z)dz, and <> denotes the average over the noise probability distribution that is chosen to be gaussian: (3.2)
Inserting this into equation 3.1 we obtain,
P'+'(z) = m'6(z)+ J
1
dz'f(z - XZ' - Jm' - h)P'(z')
(3.3)
-W
from which mtfl can be calculated
m'"
W
=
dzP'+'(z) =
p
-w
dz'JW dzf(z - Xz' 1
- Jm' - h)P'(z')
(3.4)
3.1 Strong Decay. For X = 0, the integral in equation 3.4 factorizes and we obtain a nonlinear one-dimensional map:
m'+' = (1- m') Jw
l-\m'-h
P
d
Ap roximating the gaussian by IJ 7~12,we obtain
{
f(t) =
m'+' = _-_ 1 + tanh 1 m ' S h - l T 2
',
(3.5)
{2Tc0sh*(
(3.6)
The dynamic behavior of m' is governed by the parameters J, h, and T. In Figure la-c (solid lines) we plot the term on the right-hand side of equation 3.6 together with the identity function, for T = 0.2, 1 = 1.5, and different values of the external field h. For high external field h = 0.8 (Fig. la), the map has a single fixed point characterized by an activation value close to 0.5, which implies that for such high input the population segregates in two subgroups that fire consecutively (0.5 is the maximal
M. Usher, H. G. Schuster, and E. Niebur
574
0
0.5 mltl
0
1
0.5 mlt)
1
Id) 1
1
,. d
0.5 v
E
0
5
0.5
E
0 0
10
20
L
Figure 1: The one-dimensional map (equations 3.4 and 4.4) for J = 1.5, T = 0.2 and different external fields: (a) h = 0.8, equation 3.4 solid line, and equation 4.4 with nd = n, = 3 dashed line; (b) h = 0.6, equation 3.4 solid line and equation 4.4 with nd = n, = 2 long dashed line, nd = n, = 3 short dashed line; (c) h = 0.4, equation 3.4; (d) trajectories beginning from different initial conditions (mo= 0.1 or m0 = 0.7) for parameters corresponding to (b). Lines, solutions of equation 3.4; symbols, simulation with a population of N = 1000 neurons for the two initial conditions shown.
Populations of Integrate-and-FireNeurons
575
firing rate due to the refractoriness). The fixed point is stable and thus the temporal sequence of activations will manifest damped oscillations toward the stable fixed point. For intermediate values of the external field, the system has two stable fixed points and an unstable one (the intermediate one). Therefore, for such external fields the system has two attractors and can function as a memory system. The domains of attraction of each fixed point are unlike those in a McCulloch-Pitts system; the initial states that are attracted toward the high m value have intermediate m(O), while the very low and very high values m(0) < 0.14 or m(0) > 0.8 lead to the decay of the activation. This is illustrated in Figure Id, where we display the time sequence obtained from two different initial conditions using the one-dimensional map (lines) and a simulation with N = 1000 (symbols). A further decrease of the external field leads to the situation displayed in Figure lc, where the activity is always decaying. For 0 < X << 1 a two-dimensional map can be obtained, using a first order Taylor expansion of equations 3.3 and 3.4. For moderate values of X (e.g., X = 0.4),the system behaves qualitatively similar to a system with X = 0, that is, it has a single attractor with high m for a large external field, two attractors for intermediate external fields, and again a single attractor (with small rn) for small external fields. Due to the finite value of A, the transition points between these regimes occur at smaller field values than for X = 0 (e.g., for X = 0.4,the system has 2 attractors already for h = 0.4;data not shown).
3.2 Weak Decay. We have numerically solved the master equation for an initial distribution P ( z )corresponding to a fraction of active neurons mo, uniformly distributed otherwise. For X = 1, we found that the system always reaches a fixed point corresponding to a homogeneous sequence m' = 1/M (where M is compatible with equation 2.21, even for vanishing external input. This is illustrated in Figure 2a where beginning with a very high initial activation (which would lead to decay in the absence of noise), the system gradually recovers due to small gaussian noise (with CT corresponding to T = 0.05 in equation 3.6).This phenomenon is explained by the fact that noise causes diffusion of the probability distribution P(z),whose tail eventually reaches the threshold and turns the system on. Also due to noise, the system equalizes the size of the subgroups. Figure 2b shows the multimodal probability distribution (with four components corresponding to the M = 4 subgroups) reached after 20 iterations. The same stationary distribution is reached from all initial conditions. For X < 1, the noise-induced diffusion is balanced by the decay, and the system is not able to self-activate for all initial conditions. As in the small X case, the system will reach one of two attractors, depending on the initial conditions. In Figure 2c we display the time sequence obtained for X = 0.98, beginning with initial condition m0 = 0.28, and
M. Usher, H. G. Schuster, and E. Niebur
576
0
10
20 t
z
ICI
n
N
" PI
0
20 40 60 80 100 120 t
0 2
Figure 2: Solution m' of the master equation, equation 3.4, for @ ( z ) = m06(z 1)+ (I - mo)[O(z)- O(z - l)], = 1.5,T = 0.05,h = 0. (a) mtfor X = 1,mo = 0.44, (b) P'(z) for t = 30. (c) m tfor X = 0.98,m0= 0.28. (d) Probability distribution after t = 120 iterations, parameters as in (b). In both cases, the system reaches a fixed point of self-sustained activity.
P ( z )uniform otherwise. We observe damped oscillations that reach a fixed point of rn = 0.2, implying segregation into five equal subgroups. The corresponding multimodal probability distribution P(z) is displayed in Figure 2d. Noise tends to homogenize the initially nonhomogeneous sequence of activation by transporting probability from the larger to the smaller subgroups. Begining with a "too high" activation rnO= 0.44 and
Populations of Integrate-and-Fire Neurons
577
(a) 0.3
0.2-
0.1-
0
I
0
,I
0.2
I
I
I
0.4
0.6
0.8
1
z
0.3
--
0.2-
N
nl
0.1-
0
1
(C)
0.3
--
0.2-
N
D4
0.1-
0 0
0.2
0.4
I
I
1
0.6
0.8
1
z
Figure 3: P'(z), same parameters as in Figure 2, beginning with a high initial activation ma = o.M,PO(Z) = m06(z - I) + (1 - mo)[s(z) - e(z - I)]. (a) t = 3, (b) t = 9, (c) t = 120 iterations. The probability shrinks due to the decay, although, of course, JdzP'(z) = 1 for all t. After 5 iterations, m' M 0.
P(z)uniform otherwise, the activation decays to zero. In Figure 3 we display the probability distribution P'(z) at t = 3, t = 9, and t = 120. We observe that in spite of the diffusion, P'(z) shrinks and the populations remain inactive.
M. Usher, H. G. Schuster, and E. Niebur
578
The influence of the noise on the system dynamics can be summarized as follows: For very low noise (small T ) , the system reaches a limit cycle in which the system segregates into subpopulations of unequal sizes in each of which all neurons fire synchronously. The length of the limit cycle depends on the initial conditions. When the noise exceeds a certain value, a transition toward a fix-point representing segregation into equal subpopulations occurs, and the dependence on initial conditions reduces to the choice between 2 attractors (the silent and the active state). The noise value at which the transition occurs depends on the number of subpopulations in the cycle; for ] larger than, but near 1, cycles of long length are obtained, as shown in Figure 4a. In this case the different subpopulations are separated by small potential differences, and very low noise will transport neurons between the different subpopulations and contribute to their homogenization (Fig. 4b). As ] increases, the cycle length (and hence the number of subpopulations) decreases while the potential differences between them increase. Thus a higher noise level is required in order to transport neurons among the populations. A systematic simulation study of the I dependency shows the following: a
For ] = 1.9, X = 1, a cycle of order 3 is obtained in the absence of noise, and the temperature required for inducing the transition is T = 0.07.
a
For ] = 1.5, X = 1, a cycle of order 4 is obtained, and the temperature required for inducing the transition was T = 0.04.
a
For ] = 1.1 a cycle of length 14 was obtained, and a noise characterized by T = 0.01 was enough to induce the transition (Fig. 4b).
3.3 Inputs and Hysteresis. There are two ways by which a neural population can receive external inputs. The first one is via the initial state mo (i.e., a subgroup of the population is turned on), and the second one is by sustained external input (in form of postsynaptic potentials), that is modeled as an external field h. For h = 0, we have shown that the system has two attractors. This leads to the hysteresis phenomenon illustrated in Figure 5, where we show the average activation of the population, as we increase (or decrease) adiabatically the external field. We observe that for h = 0 the activation may have two values depending on history. Therefore, absence of excitation (or presence of small inhibition) will leave the population inactive. However, once positive excitation has reached the assembly causing it to fire, the population "remembers" and continues self-sustained firing even after the external input no longer reaches the population.
Populations of Integrate-and-FireNeurons
579
0.1
A
2
0.05
E
0
2
0
10 t
20
0
10 t
20
0.05
E
Figure 4 Long cycles for J = 1.1. (a) Without noise (T = 01, a cycle of length 14 is obtained, (b) a small level of noise (T= 0.01) leads to stochastic fluctuations around a homogeneous steady state. 4 Dynamics at Higher Time Resolution; Synchronicity and Field Po-
tentials
As we have already mentioned, equation 2.1 has to be modified in order to investigate the population dynamics on a time scale characterized by a finer discretization. This is accomplished by considering a bin size that equals a fraction, say l/n, of the characteristic delay time and refractory
M. Usher, H. G. Schuster, and E. Niebur
580
0.3
5
0.2
E
0.1
0 -0.1
0
-0.05
0.05 h
0.1
0.15
0.2
(b)
0.3
0.2
/
0.1
I
,+'
i A 0
0 -0.1
/I
-0.05
0
0 1
I
1
0.05
0.1
0.15
0.2
h
Figure 5: Hysteresis. The average mt is plotted as a function of the adiabatic field h. (The field was increased by 0.01 every 20 iterations, and the average activation m was plotted.) Lines represent numerical solutions of the master equation 3.4; symbols represent simulations of a population of N = 1000 cells. (a) Low noise (T = 0.05). Note the jumps caused by quanta1 transitionsbetween cyclic patterns of length M = 4 and M = 3. (b) Same for higher noise level T = 0.2. Parameters X = 0.98,J = 1.5 for (a) and (b).
period. We assume in the following that all pairs of cells are characterized by a single communication delay time (composed of axonal and synaptic delay), but we shall discuss the case when the delay is different from the refractory time. Each neuron integrates over its inputs, and once it
Populations of Integrate-and-FireNeurons
581
reaches the threshold, the neuron fires, sending its output to the other neurons in the population where it will arrive after a delay of length nd. After emitting the spike the neuron is reset to zero, and remains refractory for nr time steps. The modified equation is
where nd and n, stand for the delay time and the refractory period in units of the discretization, respectively. Increasing the temporal resolution corresponds to using larger values for these variables. We notice that the transformation from equation 2.1 to equation 4.1 implies a corresponding scaling of the decay constant X = X'J"n, and of the-temperature T = T/fi,,. The behavior of the neural population in the presence of noise can be characterized by a Markov process with memory of order n d ,
where the probability P'(z') is replaced by P(z',z", . . .), the probability that a neuron had value z' at time t, z" at time t - 1, and so on. We should notice that this probability is not multiplicative due to temporal correlations, for example, for the case nd = 2, Prob[z'(t) < l,z''(t - 1) < 11 = 1 - m(t)- m(t - 1) # [1 - m(t)l[1- m(t - 1)1 From equation 4.2, a map analogous to equation 3.6 can be obtained for X = 0:
The solutions of this delay equation are more complex than the anal-
ogous solutions of equation 3.6. However, special solutions that also satisfy equation 3.6 can be obtained easily. Consider the following 2 cases: 0
The activation for each subpopulation is spread homogeneously at = ... = m'+. the subrefractory time resolution, that is, m' =
M. Usher, H. G. Schuster, and E. Niebur
582
In this case, and for n d
m'+' = 1 - n,m' ~
2
= n,,
+
equation 4.3 transforms into:
1 tanh
Jm' + h - 1 T
(4.4)
where the time bin equals the delay and the refractory period. 0
Perfect synchronization of subpopulations, that is, mt # O,m'-' = . . . = mt-"d = 0 repeating cyclically modulo nd. Such periodic solutions can be studied easily, since they also satisfy equation 3.6.
We have shown that for X = 0, J = 1.5, T = 0.2 the population may converge to an attractor of high activity for high field values (e.g., h = 0.8, Fig. la, solid line), to the silent state for low field values (h = 0.4, Fig. lc, solid line), or to two attractors, for intermediate field values (h = 0.6, Fig. lb, solid line), depending on the initial activation. When the high resolution dynamics are taken into account (equation 4.3), the following characteristics are revealed: For h = 0.8 both the homogeneous case (equation 4.4 and Fig. la, dashed line) and the synchronized case (Fig. la, solid line) have nontrivial solutions. For intermediate field values h = 0.6, only the synchronized solution is nontrivial; Fig. l b (dashed lines) displays the map in equation 4.4 showing that for n d = n, 2 2, the only solution is the silent state. Numerical iteration of the map, equation 4.3, and simulations confirm that for high external field either a homogeneous or a modulated solution is obtained (depending on initial conditions), while for intermediate fields (that are essential for memory preservation) only the synchronized solutions remain. Finally, we present simulation results that display the high resolution dynamics, equation 4.1, for weak decay (X close to one). In Figure 6a, the system parameters were chosen identical to those used in Figure 2c (subject to the scaling of X and T)and n d = n, = 4. We see that the population's activity manifests rapid oscillations (of period equal to the delay time), with amplitude m = 0.2, reflecting a segregation into five populations (as in Fig. 2c). Figure 6b displays a close-up into Figure 6a showing that inside each such population, there is a relatively high but variable synchronicity that leads to slower and irregular waves superimposed on rapid oscillations (the periodicity of the signal in this synchronized state is imposed by the transmission delay). If the delay time is larger than the refractory period, a different behavior is obtained. In Figure 6c we illustrate the population dynamics for the same parameters as in Figure 6a, except that nd = 5, while n, = 4 as before. We observe that the activity oscillates with the period of the delay time, but the amplitude is increased to m = 0.5,implying segregation into 2 subpopulations. The reason for this change is that unlike in the previous case ( n d 5 n,) no activation is lost, that is, cells can send activation to themselves, since after the delay they are not refractory any
Populations of Integrate-and-Fire Neurons
583
Figure 6 Simulations with higher temporal resolution (equation 4.1), N = lo00 neurons. (a) Parameters as in Figure 2b, except nd = n, = 4; (b) close-up showing that the periodicity is determined by the transmission delay; (c) fld = 5, nr = 4; (d) parameters as in Figure 4a, nd = n, = 4, T = 0; (e) same with noise T = 0.01. more. In such a case, complete synchronization of the population is obtained for higher values of J in which all neurons fire with the period of the delay. In Figure 6d, we display the high resolution simulation ( n d = n, = 4),
584
M. Usher, H. G. Schuster, and E. Niebur
corresponding to small J values leading to long cycles (parameters as in Fig. 4). In the absence of noise, a periodic solution (periodicity 15nd) analogous to Figure 4a is obtained. Adding a noise factor equivalent to the one used in Figure 4b, we observe that the activity manifests irregular oscillations (Fig. 6e) whose maximal amplitude is rn = 0.05, implying segregation into approximately 20 subpopulations. The synchronization of each subpopulation varies stochastically, and thus the total activity is characterized by aperiodic spindles of high oscillations. Such spindles are highly suggestive of experimental field potentials or EEG recordings. Since these are also considered to reflect a global activity in some neural population, as it is the case in our model, we think that our simple model provides a simple explanation of possible mechanisms underlying these phenomena. 5 Discussion
Most neural network models are built of simplified units that do not take into account inherent properties of real neurons, such as refractory period and integration time. Such properties are essential for any computational process dealing with the temporal structure of neural activity. For example, in the absence of an integration constant, small jitter in the transmission of signals would wash out synchronicity effects. Thus networks based on refractory integrate-and-fire neurons are among the simplest models that may explain how the brain processes temporally structured information. However, despite the widespread use of integrate-and-fire neurons as models for single cells, only recently systematic studies of the population dynamics of such networks have been carried out (Amit and Tsodyks 1992; Ahlstrom and Levinsen 1992; Kuramoto 1992; Van Vreeswijk and Abbott 1992). In the present work, we have studied the behavior of large, completely connected networks of refractory integrate-and-fire neurons in the presence of homogeneous input and noise. Two results seem particularly noteworthy. One is the finding that at the time resolution of the synaptic delay, the presence of noise will always lead to a segregation of the system in neuronal groups of equal size. Thus on time scales larger than the transmission delay, our results support an interpretation of steady state firing rate as in Amit and Tsodyks (1992). However, at a finer time resolution, firing may be highly synchronized and temporally modulated with the periodicity of the delay time, as in the model proposed in Gerstner and Van Hemmen (1992). The second important result is the observation of hysteresis in the system. It is tempting to speculate that this is a mechanism employed for the neural implementation of short-term memory. The idea of memory being implemented in the brain by reverberating activity dates back at least to Caianiello and has received new vigor by recent observations showing
Populations of Integrate-and-Fire Neurons
585
that the activation of short-term memory is correlated with increased neural activity in neocortex (Miyashita and Chang 1988; Fuster 1990). Our results indicate that self-excitatory networks of integrate-and-fire neurons are able to manifest a basic form of this behavior, especially when they operate in the synchronized mode. As we have shown, when the system is in its memory mode, characterized by two attractors, only the synchronous (and not the homogeneously spread-out solution) is nontrivial. The main reason for this is that synchronized activity is more effective in activating other cells than randomly spread activity. This may also shed light on the widely argued discussion about the role of cortical oscillations in information processing. According to our model, aperiodic oscillations and synchronicity are emergent properties of neural populations with recurrent excitation, and contribute to its memory function. The dynamic properties of such systems are different from the memory systems based on McCulloch-Pitts neurons. For example, when transmission delays are equal or smaller than the refractory period, a transient external input is more effective in inducing sustained activity in populations of integrate-and-fire neurons if the input is at an intermediate value, rather than at a relatively high value. The significance of such dynamic characteristics to the actual physiological process underlying short-term memory, and the use of integrate-and-fire neurons for modeling physiological time dependent processes should be the subject of further research.
Acknowledgments EN is supported by the Office of Naval Research and wishes to thank HGS and the Institute for Theoretical Physics of the University of Kiel for their hospitality. HGS thanks Christof Koch for his kind hospitality during his stay at Caltech. MU is supported by a Bantrell fellowship. This work was supported by NATO Grant 911034COP.
References Ahlstrom, P., and Levinsen, M. T. 1992. Phase-locking structure of integrateand-fire models with threshold modulation. Phys. Lett. A 128(34), 187-192. Amit, D.J.,and Tsodyks, M. V. 1992. Quantitative study of attractor neural network retrieving at low spike rates. Network 2(3), 259-273. Fuster, J. M.1990. Inferotemporal units in selective visual attention and shortterm memory. J. Neurophysiol. 64,681-697. Gerstner, W.,and Van Hemmen, J. L. 1992. Associative memory in a network of ‘spiking’ neurons. Network 3, 139-164. Kuramoto, Y. 1992. Collective synchronization of pulse-coupled oscillators and excitable unit. Physica D 50(1), 15-30.
586
M. Usher, H. G . Schuster, and E. Niebur
Mirollo, R. E., and Strogatz, S. H. 1990. Synchronization of pulse-coupled biological oscillators. S I A M 1.Appl. Math. 50(6), 1645-1662. Miyashita, Y., and Chang, H. S. 1988. Neural correlate of pictorial short term memory in primate temporal cortex. Nature (London) 331, 68-70. Van Vreeswijk, C., and Abbott, L. F. 1992. Self-sustained firing in populations of integrate and fire neurons. S I A M I. Appl. Math. Received 14 September 1992;accepted 17 November 1992.
This article has been cited by: 2. A. N. Burkitt. 2006. A review of the integrate-and-fire neuron model: II. Inhomogeneous synaptic input and network properties. Biological Cybernetics 95:2, 97-112. [CrossRef] 3. Z. Yang, A. Murray, F. Worgotter, K. Cameron, V. Boonsobhak. 2006. A Neuromorphic Depth-From-Motion Vision Model With STDP Adaptation. IEEE Transactions on Neural Networks 17:2, 482-495. [CrossRef] 4. E. Del Moral Hernandez, Geehyuk Lee, N.H. Farhat. 2003. Analog realization of arbitrary one-dimensional maps. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications 50:12, 1538-1547. [CrossRef] 5. Carsten Meyer , Carl van Vreeswijk . 2002. Temporal Correlations in Stochastic Networks of Spiking NeuronsTemporal Correlations in Stochastic Networks of Spiking Neurons. Neural Computation 14:2, 369-404. [Abstract] [PDF] [PDF Plus] 6. A. N. Burkitt , G. M. Clark . 2001. Synchronization of the Neural Response to Noisy Periodic Synaptic InputSynchronization of the Neural Response to Noisy Periodic Synaptic Input. Neural Computation 13:12, 2639-2672. [Abstract] [PDF] [PDF Plus] 7. Grigory Osipov, Jürgen Kurths. 2001. Regular and chaotic phase synchronization of coupled circle maps. Physical Review E 65:1. . [CrossRef] 8. Naoki Masuda, Kazuyuki Aihara. 2001. Synchronization of pulse-coupled excitable neurons. Physical Review E 64:5. . [CrossRef] 9. Henry C. Tuckwell . 2000. Cortical Potential Distributions and Information ProcessingCortical Potential Distributions and Information Processing. Neural Computation 12:12, 2777-2795. [Abstract] [PDF] [PDF Plus] 10. A. N. Burkitt , G. M. Clark . 1999. Analysis of Integrate-and-Fire Neurons: Synchronization of Synaptic Input and Spike OutputAnalysis of Integrate-and-Fire Neurons: Synchronization of Synaptic Input and Spike Output. Neural Computation 11:4, 871-901. [Abstract] [PDF] [PDF Plus] 11. R. Mueller, A. Herz. 1999. Content-addressable memory with spiking neurons. Physical Review E 59:3, 3330-3338. [CrossRef] 12. Bard Ermentrout. 1998. Reports on Progress in Physics 61:4, 353-430. [CrossRef] 13. U. Ernst, K. Pawelzik, T. Geisel. 1998. Delay-induced multistable synchronization of biological oscillators. Physical Review E 57:2, 2150-2162. [CrossRef] 14. Doron Tal , Eric L. Schwartz . 1997. Computing with the Leaky Integrate-and-Fire Neuron: Logarithmic Computation and MultiplicationComputing with the Leaky Integrate-and-Fire Neuron: Logarithmic Computation and Multiplication. Neural Computation 9:2, 305-318. [Abstract] [PDF] [PDF Plus]
15. Andreas Herz, John Hopfield. 1995. Earthquake Cycles and Neural Reverberations: Collective Oscillations in Systems with Pulse-Coupled Threshold Elements. Physical Review Letters 75:6, 1222-1225. [CrossRef] 16. Wulfram Gerstner. 1995. Time structure of the activity in neural network models. Physical Review E 51:1, 738-758. [CrossRef] 17. Öjvind Bernander , Christof Koch , Marius Usher . 1994. The Effect of Synchronized Inputs at the Single Neuron LevelThe Effect of Synchronized Inputs at the Single Neuron Level. Neural Computation 6:4, 622-641. [Abstract] [PDF] [PDF Plus]
Communicated by Christof Koch
The Effects of Cell Duplication and Noise in a Pattern Generating Network Catherine H.Ferrar Thelma L. Williams Department of Physiology, St. George’s Hospital Medical School, University of London, London, U.K.
Graham Bowtell Department of Mathematics, The City University, London, U.K.
Stability against stochastic variation is an important property for biological systems. This paper investigates the robustness of the rhythmic activity produced by a model of the segmental rhythm generator for locomotion in the lamprey, by introducing stochastic properties into the network. In addition, since neuronal models for vertebrate systems often use a single neuron to represent a large class of cells, this paper explores one of the consequences of such reduction, by investigating the effects of duplicating all the cells of the network on its stability against stochastic variation. We have found the basic model network to be very stable, and have found that this stability is increased by doubling the number of cells in the network. 1 Introduction It is well known that rhythmic behavior in many animals is produced by the activity of autonomously oscillating groups of neurons, known as central pattern generators (CPGs), which require no sensory feedback to provide the basic timing for muscle activation (Delcomyn 1980). CPGs have been described in many invertebrate systems, typically containing a relatively small number of individually identifiable cells. In vertebrates, by contrast, the central nervous system generally consists of classes of cells with similar properties and connections, and in consequence, vertebrate CPG models often use single neurons to represent large classes of neurons. To gain insight into some of the consequences of such simplification, we have experimentally investigated the behavior of a model of the segmental CPG for locomotion in the lamprey. Although the detailed structure of the lamprey locomotor CPG is not known, a small network model consisting of neurons of identified classes known to be rhythmically active during fictive locomotion (Buchanan and Neural Computation 5, 587-5% (1993) @ 1993 Massachusetts Institute of Technology
588
Catherine H. Ferrar, Thelma L. Williams, and Graham Bowtell
Figure 1: Proposed basic network for the lamprey central pattern generator for swimming (Buchanan and Grillner 1987), with driving cells added to each side. C, crossed caudal interneuron (CCIN); E, excitatory interneuron (EIN); I, lateral interneuron (LIN); D, driving cell. Filled circles indicate inhibition and open triangles excitation. Grillner 1987) has been shown to produce oscillations with phase relationships among the three cell types that are similar to those seen in the lamprey spinal cord in vitvo (Grillner et al. 1988; Buchanan 1992; Wallen et al. 1993). In this paper we investigate the stability of this network by randomly varying the strengths of the interneuronal connections. In addition, we have investigated the effects on such stability of doubling the number of neurons in the rhythm generator. 2 Network Model
The model depicted in Figure 1 is based on that of Buchanan and Grillner (1987) containing three types of interneuron (EIN, excitatory interneuron; CCIN, crossed caudal interneuron; LIN, lateral interneuron) forming a network that is symmetric about a center line corresponding to the leftright axis. The EINs provide excitation to motor neurons, and the CCINs
Cell Duplication and Noise in a Pattern Generating Network
589
provide midcycle inhibition. In our simulations, tonic stimulation was increasing the provided to each side of the network by driver cells (D); activity of the driver cells increases the frequency of the oscillations. The initial conditions for the simulations had left/right symmetry, which leads to an unstable equilibrium if no stochastic noise is present. For this reason, a small left/right imbalance (1 in 109 was introduced into the driver cell connections in those simulations that had no noise. This allows the network to begin oscillating; after oscillation has been established such an imbalance can be discarded with no effect on the ensuing output. For networks with even quite low levels of introduced noise (< lo-*), a simulation goes spontaneously into oscillatory behavior with no requirement for imbalance. 3 The Doubled Network
To investigate the effects of doubling the number of neurons in the network, simulations were also run on a network containing twice as many EINs, CCINs, and LINs. The pattern of synaptic connections was the same as in the single network each CCIN made inhibitory synapses on all six contralateral cells (excluding the D cell), each EIN made excitatory synapses on the ipsilateral pairs of CCINs and LINs, and each LIN made inhibitory synapses only on the ipsilateral pair of CCINs. Since the total number of synapses on these cells was thus doubled, the levels of synaptic strengths was halved to allow direct comparison between the two types of network. All cells in the doubled network received the same tonic stimuli from the driver cells as in the single network. 4 Simulation
The basic algorithm for the simulation was that proposed by Grossberg (1978) and embodied in a commercial software package by McClelland and Rumelhart (1988). Each neuron is assumed to have an activation level corresponding to its membrane potential, and synaptic links between neurons are such that one neuron can only affect another neuron if its activation level is above a certain threshold. Although action potentials are not modeled explicitly the level of activation above the threshold can be thought of as representing the frequency of action potentials. The threshhold has been set, without loss of generality, to zero. Mathematically we define [ ~ i ]=
0,
=
Ui,
if if
Ui
~i
50 >0
where ai is the activation level of the ith neuron. The input to the ith postsynaptic neuron from the jth presynaptic neuron, which can be thought
Catherine H. Ferrar, Thelma L. Williams, and Graham Bowtell
590
of as representing a change in conductance, can be expressed as wi,[ai], where wij represents the synaptic strength of the synapse from the jth to the ith neuron. The discrete time course of the activation of the system is now computed from
Aai
=
( M a x - ai)€'xi + (a; - Min)lnhi - Decay(ai - Rest) + di(Max - ai)
where Aai is the change in the activation over one time increment. Exi and lnhi are the sums of all inputs to the ith cell from all presynaptic excitatory and inhibitory cells, respectively, within the network; these are given explicitly as
Exi = C w i j [ ~ ,i ] wij 2 0 i
Inhi = C w i j [ ~ i,l wij < o 1
Max and Min are the limits of depolarization and hyperpolarization,
playing the role of a reversal potential for the excitatory and inhibitory synapses, respectively. Decay is a rate constant controlling the rate of return of the activation to the resting potential Rest, and di is the tonic stimulus received by the ith cell from its driving cell. This network produces oscillations over a large parameter space. For simplicity, in this investigation Max and Min were set equal to +1 and -1, respectively, Decay to 0.1, and Rest to zero. The standard synaptic strengths wi, within the single network were set equal to 1 (positive for excitatory synapses, negative for inhibitory ones). The strengths of the synapses from the D cells to the CCINs was four times higher, at 0.12, than to EINs and LINs, at 0.03. 5 Numerical Investigation
Multiple simulations were run for both the single and double networks, using two different types of variation about the set of standard synaptic strengths. The first was a random variation in the synaptic strengths from the driver cells to all the cells of the network. In this case, each was multiplied by a factor chosen beforehand from a random distribution, to give a range of synaptic strengths from 50 to 100% of the standard values; variations greater than 100% frequently resulted in loss of oscillatory behavior. The synaptic strengths then remained constant during an entire run, and this variation will be referred to as the long time scale type. These simulations were run using a commercially available software package by McClelland and Rumelhart (1988). For the second type of variation, we wrote new software with identical algorithms except that any given synaptic strength could be selected in each time step from a
Cell Duplication and Noise in a Pattern Generating Network
591
pseudorandom distribution. Since a time step represents about 3%of the cycle duration, these variations occur on a time scale closer to that of real synaptic noise. The variations were applied to all the synaptic strengths within the network. With this short time scale variation, it was possible to increase the range of the random distribution of synaptic strengths to 0-200% of the standard values without destroying the stability of the system. In each case, a simulation was run for at least 300 time steps (about 10 cycles of oscillatory activity) or until the system appeared to be in a steady state, before measurements were made. In all runs (except the two runs of the single network in which oscillatory behavior failed, see below), all cells had oscillating membrane potentials (activation values) with the same average cycle durations. For comparison of different runs, the following measurements were made: cycle duration (measured from the times at which the activation level of a particular EIN cell crossed threshold), burst proportion (the fraction of the cycle duration during which the activation of the EIN was above threshold), and left-right phase lag (phase delay between EIN cells on opposite sides of the midline). For the short time scale runs, mean values were calculated from measurements made in 20 consecutive cycles. For the long time scale variation, the means were calculated from twenty independent runs.
6 Results The time course of the activation variables of the left and right EIN cells in one run of the single network without noise is shown in Figure 2A. The oscillator exhibits strict left-right oscillation with a burst proportion of 32%. The cycle duration is approximately 35 time steps. (If a time step is taken arbitrarily to represent 10 msec, the frequency of oscillation is near 3 Hz, a reasonable swimming frequency for a lamprey.) The double network without noise gave identical results. Examples of the effects of noise with a short time scale variation are shown in Figure 2B for the single network and in Figure 2C for the double network. For both these cases the cycle duration, burst proportion, and phase difference between left and right cells are no longer constant. Plots of cycle duration against consecutive run number (long time scale variation) or cycle number (short time scale variation) are shown in Figure 3 for both the single and double networks. It can be seen that the system is less affected by noise on the short time scale; the effects tend to average out over a cycle. For both time scales, doubling the network made the system less variable under the influence of noise. More precisely, Figure 4A shows the mean and standard deviation of these four data sets, from which it is seen that for both short time scale and long time scale variation, doubling the network approximately halves the standard deviations. F tests indicate that the variances of cycle durations
592
Catherine H. Ferrar, Thelma L. Williams, and Graham Bowtell
time
i
time n
-I
time
Figure 2 Outputs from a left (solid line) and a right (dotted line) EIN. (A) Standard single oscillator with no noise. (B) Single oscillator with 100% noise on the short time scale. (C) Double oscillator with 100% noise on the short time scale. The broken horizontal line represents the threshold level.
Cell Duplication and Noise in a Pattern Generating Network
-d
400
3 8
350 300
5 .-”
250 200
6
0
-U, 0
450
150 100 50 0
593
..
1 -1
L 5
10
15
20
15
20
cycle no.
-d 1 450
400
0
5
10
run no.
15
20
0
5
10
cycle no.
Figure 3 (A) Cycle duration plotted against run number for a single network with long time scale variation. (B) As in (A) but for the double network. (C) Cycle duration plotted against cycle number for the single network with short time scale variation. (D) As in (C) but for the double network. The solid horizontal line indicates the mean cycle duration. One computational time step taken arbitrarily as representing 10 msec. are significantly different in the single and double networks with p < 0.01 for the long time scale variation, and p < 0.05 for the short time scale variation. The means and standard deviations for the left-right phase difference are shown in Figure 4B; the mean was almost equal for each of the four cases. Statistically, it was found that doubling the network makes a significant change in the variability only when considering the long time scale cases. Figure 4C shows that doubling the network significantly reduces the variation of the burst proportion for long time scale variation, but not for short time scale variation. The standard deviations shown in Figure 4 for the single network with the long time scale (open squares) underestimate the variability in the following sense: in 2 of 22 runs with this configuration, the behavior of the network was oscillatory for only a few cycles, after which a nonpe-
Catherine H. Ferrar, Thelma L. Williams, and Graham Bowtell
594
A
C
B
-E -
400
0
0.5
0.75
e
2 aI
rc
c
.-0 U
200
a
Q)
T1
-
z
c
0.50
(II
c
a
0
0.25
0
bn 2
0.25
c
cn L
a
L
r
C
a
C
.-
0
.-0
n
I .U c
0
2
0
0
0 mean of single network, variation over the short timescale. 0 mean of single network, variation over the long timescale. 0 mean of double network, short timescale.
mean of double network, long timescale.
Figure 4 Means and standard deviations of (A) cycle durations in the EINs (one time step represents 10 msec), (B)phase differences between left and right Ems, and (C) burst proportions of Ems. riodic solution was reached, in which some cells were silent while others were tonically active. In these cases, in other words, the set of synaptic strengths was not within the parameter space in which oscillatory behavior is supported. No measurements were made on these runs. To test whether further duplication of the network would reduce variability, the standard deviation of cycle duration was calculated for 4-, 8-, and 16-unit networks (Fig. 5). It can be seen that the variability continues to decrease. 7 Discussion
To appreciate the extent of the simplification made in this model of the lamprey CPG for locomotion, it should be noted that in each segment of the lamprey spinal cord there are approximately 60 CCINs (Buchanan 1982) and at least 40 EINs (Buchanan et al. 1989). On the other hand, only 50-150 LINs are estimated to occur in the entire spinal cord (approximately 100 segments), and to be restricted to the rostra1 half of the cord. This allows for fewer than 2 LINs per segment, but the role of LINs
Cell Duplication and Noise in a Pattern Generating Network
1
2
4
8
595
16
Number of replicates
Figure 5: Effects of further network duplication on variability. Ordinate: standard deviation of the cycle duration, expressed as fraction of the mean.
in this circuit may actually be filled by the small, numerous inhibitory interneurons reported by Buchanan and Grillner (1988)to have a strong influence on the locomotor activity produced by the spinal cord in vitro. In spite of such uncertainty, the robustness of this oscillatory network in the face of even quite large random variations in synaptic current indicates a strongly attracting limit cycle, making the circuit a worthy candidate for the role of unit oscillator in the lamprey CPG. The finding that the behavior of the model network became even more stable when the number of neurons in each class was increased confirmed our intuitive prediction that it would benefit a network to have many neurons of the same class, since the random variation that occurs in a biological system would be less disturbing. From a modeling point of view it would appear that representing a class of cells by a single cell may not affect the behavior of the network as long as the reduced model is stable enough.
596
Catherine H. Ferrar, Thelma L. Williams, and Graham Bowtell
Acknowledgments This work was supported by the SERC with additional funding (for CHF) from Glaxo and from the William Rushton Fund of the Physiological Society. We are grateful to Jim Buchanan for introducing us to the McClelland and Rumelhart software and for generously sending us his unpublished results.
References Buchanan, J. T. 1982. Identification of interneurons with contralateral caudal axons in the lamprey spinal cord: Synaptic interactions and morphology. J . Neurophysiol. 47,961-975. Buchanan, J. T. 1992. Neural network simulations of coupled locomotor oscillators in the lamprey spinal cord. Biol. Cybernetics 66,367-374. Buchanan, J. T., and Grillner, S. 1987. Newly identified "glutamate interneurons" and their role in locomotion in the lamprey spinal cord. Science 236, 312-314. Buchanan, J. T., and Grillner, S. 1988. A new class of small inhibitory interneurons in the lamprey spinal cord. Brain Res. 438,404407. Buchanan, J. T., Grillner S., Cullheim, S., and Risling, M. 1989. Identification of excitatory interneurons contributing to generation of locomotion in lamprey: Structure, pharmacology and function. I. Neurophysiol. 62(1),59-69. Delcomyn, F. 1980. Neural basis of rhythmic behaviour in animals. Science 210, 492498. Crillner, S.,Buchanan, J. T., and Lanser, A. 1988. Simulation of the segmental burst generating network for locomotion in the lamprey. Neurosci. Lett. 89, 31-35. Crossberg, S. 1978. A theory of visual coding, memory, and development. In Formal Theories of Visual Perception, E. L. J. Leeuwenberg and H. F. J. M. Buffart, eds., pp. 7-26. Wiley, New York. McClelland, J. L., and Rumelhart, D. E. 1988. Explorations in Parallel Distributed Processing: A Handbook of Models, Programs and Exercises, 1st ed. MIT Press, Cambridge, MA. Rovainen, C. M. 1979. Neurobiology of lampreys. Physiol. Rev. 59, 1007-1077. Wallen, P., Ekeberg, O., Lansner, A., Brodin, L., Traven, H., and Grillner, S. 1993. A computer based model for realistic simulations of neural networks 11: The segmental network generating locomotor rhythmicity in the lamprey. I. Neurophysiol. 68,1939-1950. Received 15 April 1992; accepted 6 January1993.
This article has been cited by: 2. Jimmy Or. 2007. Robustness of Connectionist Swimming Controllers Against Random Variation in Neural ConnectionsRobustness of Connectionist Swimming Controllers Against Random Variation in Neural Connections. Neural Computation 19:6, 1568-1588. [Abstract] [PDF] [PDF Plus]
Communicated by Richard Andersen
Emergence of Position-Independent Detectors of Sense of Rotation and Dilation with Hebbian Learning: An Analysis Kechen Zhang Martin I. Sereno Margaret E. Sereno* Department of Cognitive Science, University of California,Sun Diego,
La Jolla,CA 92093-0515 USA We previously demonstrated that it is possible to learn position-independent responses to rotation and dilation by filtering rotations and dilations with different centers through an input layer with MT-like speed and direction tuning curves and connecting them to an MSTlike layer with simple Hebbian synapses (Sereno and Sereno 1991). By analyzing an idealized version of the network with broader, sinusoidal direction-tuning and linear speed-tuning, we show analytically that a Hebb rule trained with arbitrary rotation, dilatiodcontraction, and translation velocity fields yields units with weight fields that are a rotation plus a dilation or contraction field, and whose responses to a rotating or dilatinglcontracting disk are exactly position independent. Differences between the performance of this idealized model and our original model (and real MST neurons) are discussed. 1 Introduction A major stream of motion information processing in the primate visual system goes from layer 48 in primary visual cortex (Vl) to the middle temporal area (MT) and then to the medial superior temporal area (MST) (for reviews see Sereno and Allman 1991; Felleman and Van Essen 1991). Most neurons in area MT have moderate-sized receptive fields, and a subset is tuned to the local pattern velocity (Movshon et al. 1985). Neurons in the dorsal part of MST, by contrast, have much larger receptive fields and some are selective to higher order motion features-for example, rotation (either clockwise or counterclockwise, but not both), and dilation or contraction (but not both) on the frontoparallel plane (Saito et al. 1986; Sakata et al. 1986; Tanaka and Saito 1989; Tanaka et al. 1989; Duffy and Wurtz 1991a,b). Detecting rotation, dilation, and contraction provides useful information about an animal's motion relative to the environment or about the intrinsic motion of an object (Koenderink and 'Present address: Department of Psychology, University of Oregon, Eugene, OR 97403.
Neural Computation 5,597-612 (1993) @ 1993 Massachusetts Institute of Technology
598
Kechen Zhang, Martin I. Sereno, and Margaret E. Sereno
van Doorn 1975, 1976; Longuet-Higgins and Prazdny 1980; Koenderink 1986). An interesting property is that some dorsal MST neurons give nearly identical responses to a rotation, or dilation or contraction, no matter where the center of the velocity flow is located. We sought to find a neural mechanism for this position invariance. To be selective to a rotation or dilation/contraction with a fixed center, the receptive field of an MST neuron need just be composed of the MT neurons whose preferred directions are arranged circularly or radially around that center (Saito et al. 1986). At first glance, this simple mechanism would not seem to be able to support an invariant response when the position of the center changes (Saito et al. 1986; Tanaka et al. 1989; Duffy and Wurtz 1991b). Two previous proposals for a position-independent mechanism assume a homogeneous organization for an MST receptive field to ensure that all its subfields have identical structure and function. In one model, the local rotation and dilation of the velocity field is first derived and then summed up across space to get invariant responses (Duffy and Wurtz 1991b). This algorithm requires that MT neurons be selective to local rotation and dilation/contraction, which is generally not the case (Tanaka et al. 1986). Another model makes use of partially overlapping compartments in an MST receptive field (Saito et al. 1986). But this model needs a special surround effect in MT neurons to prevent many compartments from being activated simultaneously, the exact mechanism of which awaits further experimental proof. A simpler yet counterintuitive solution was discovered in a computer simulation experiment using a feedforward network and unsupervised learning (Sereno and Sereno 1990,1991). That work was based on a previous study in which Hebbian learning was used to find a solution to the aperture problem in a two-layer feed forward network corresponding to the connections from V1 -+ MT (Sereno 1989). When a similar network (with a larger interlayer divergence) representing MT MST connections is trained with rotation, dilation, and contraction using a Hebb rule and input-layer units with MT-like tuning curves, MST-like units with position-independent responses emerge. Surprisingly, such rotation or dilation/contraction detectors turned out to have inhomogeneous receptive fields with a circular, spiral, or radial arrangement of local direction selectivity, just as in the simple mechanism mentioned before. In this letter we analyze a modified version of the original model in Sereno and Sereno (1991). The input layer of the modified model has broader (cosine) tuning curves than in the original model (and broader than those of real MT neurons), but it allows us to derive explicit expressions for the course of learning empirically observed in the original model. The modified model gives rise to MST-like units that linearly decompose the flow field into flow field components-for example, a clockwise rotation-preferring unit will respond as well to the rotation in a clockwise spiral as to a pure clockwise rotation, ignoring any added di-+
Position-Independent Detectors of Rotation and Dilation
599
lation/contraction. By contrast, in our original model, the sharper tuning curves for the MT-like units result in MST-like units whose response falls off as other optic flow components are added. This smooth fall-off has also been observed with real MST neurons (Graziano et al. 1990; Orban et al. 1992). It is important to note, however, that the basic mechanism of position-invariant response to flow field stimuli (a position-variant direction-tuning template) is identical in both the idealized model with cosine tuning curves as well as the original model with narrower tuning curves. Linear decomposition might yet be found in areas beyond MST. It would be useful for filtering out certain movement components (e.g., translation) while exactly signaling the magnitude of others of interest (e.g., dilation). On the other hand, tighter input-layer tuning curves allow individual output layer units to code more information about a flow field (see Discussion).
2 A Mechanism for Position Independence First, we show the basic principles for the position-independent responses, as initially revealed by computer simulation (and recently independently derived in similar form by Poggio et al. 1990, 1991). Let v = v(r) be the velocity field on the image plane (frontoparallel plane), where the vector r = xi + yj denotes the position with i and j being the unit vectors of x and y axes. Consider MT-like units that are sensitive to the local stimulus velocities v. Each MT-like unit has a preferred direction. Given v as the stimulus velocity at a fixed position, the response or activation a of the MT-like unit at that position is assumed to be proportional to the velocity component in the preferred direction, or
a = cv cos(8 - 4) = cd, . v
(2.1)
where z, = IvI is the stimulus speed, c is a constant coefficient representing the slope of the (linear) speed tuning, 0 is the direction angle of v, and d# is the unit vector for the preferred direction angle 4. In other words, the unit has linear response to the speed v and a sinusoidal direction tuning curve with the maximum at the preferred direction 4 (Fig. 1). The artificial MT-like units resemble the real neurons in area MT of monkey in certain respects (Rodman and Albright 1987). For most MT neurons, speed does not alter the shape of direction tuning curves, which implies a multiplicative interaction of the speed tuning and the direction tuning as used in expression 2.1. The linear speed tuning is a reasonable approximation for small speeds, although the response is sometimes reduced when the speed exceeds an optimum value. The sinusoidal direction tuning is broader than a typical real MT neuron (Maunsell and Van Essen 1983). Also, the responses of real MT neurons to the antipreferred direction are usually smaller. We retain expression 2.1 for its simplicity and ease of analysis.
Kechen Zhang, Martin I. Sereno, and Margaret E. Sereno
600
Figure 1: A family of direction tuning curves for different speeds. Now consider an MST-like unit that receives inputs from many MTlike units. It is convenient to define the weight vector field of an MST-like unit. For any position on the image plane, we define the weight vector w at that point as
w = cwd4
(2.2)
where d4 is the preferred direction of the MT-like unit at that position, w is the scalar weight for its connection to the MST-like unit, and c is the same constant coefficient as in 2.1. The total input I to the MST-like unit is assumed to be the weighted sum of the inputs from all the MT-like units within the receptive field of the MST-like unit:
I = C wa = C w . v r
(2.3)
r
where the simple relation (see equations 2.1 and 2.2) wa = wcd4 .v= w ’ v has been used. The output of the MST-like unit is simply 0 = o(1)
where o( ) is a sigmoid function. If the weight vector field of an MST-like unit is itself a rotational field, namely, w = n x r = -Oyi+Rxj
(2.4)
where vector n = nk can be regarded as the “angular velocity” for the weight vector field w, with k being the unit vector of the t axis (perpendicular to the image plane), then we can prove that the MST-like unit’s response 0 to a rotating disk of angular velocity w = wk depends only on the angular speed w of the stimulus but not on the location of the stimulus disk.
Position-IndependentDetectors of Rotation and Dilation
601
Figure 2: Response elicited by a rotating (a) or dilating (b) ring in a receptive field with circular or radial distribution of direction selectivity is independent of the position of the ring (see text).
To show this, decompose the stimulus disk into many concentric rings and calculate the response elicited by a single rotating ring of radius R and width AR (Fig. 2a). Let v be the velocity field of the stimulus ring, and AZ = C v be the increment to the total input to the MST-like unit contributed by all the MT-like units within the area covered by the ring. We treat the weight vector field w = w(r) as depending continuously on the position r. The number of units for a unit area on the image plane is assumed to be a constant, and is taken as unity for simplicity. Replacing the sum by the integration along the ring, we get we
where dl = (v/u)dl with v/v being the unit vector in the circular direction, and S is the area enclosed by the ring. The last equality is a direct application of Stokes' theorem, where dS is the area element. Since the weight vector field 2.4 has a constant curl V x w = (&w,,/i3x-&wx/dy)k = 2Qk, the last integral in equation 2.5 is equal to Js 2R dS = 27rRR2. Since the rotational speed D of the stimulus ring is proportional to its radius (u = wR), we finally obtain
Al = 2rwRR3AR
Kechen Zhang, Martin I. Sereno, and Margaret E. Sereno
602
which is independent of the position of the stimulus ring. The total response to the disk of radius p is therefore
o = o(i)=
(nwnp4/2)
which is also position independent. If the angular speed w changes sign, that is, the disk rotates in the opposite direction, the total input I also changes sign. To get position-independent responses to dilation or contraction, just let the weight vector field be
w = Ar = Axi + Ayj
(2.6)
which is itself a "dilation" when constant A > 0 and a "contraction" when A < 0. This vector field has constant divergence V .w = aw,/ax + awy/ay = 2A (cf. the expression for constant curl above, except that div is a scalar). The response elicited by a dilating ring (Fig. 2b) is independent of the position of its center. The proof is similar, but Gauss' theorem is used to evaluate the integral:
fw.vdl=v
f w . n d l = v J, V.wdS=2nXAR3
where n = v/v is the unit vector in the radial direction of the ring, and v = XR, where X specifies the rate of dilation (as w specifies the rotation speed). It follows that the response to a dilating disk is also position independent. In the case that the stimulus is a contraction, the input just changes sign. This result can be intuitively appreciated by considering Figure 3. In (a), the stimulus is centered. Since the local stimulus direction v(r) always agrees with the weighted local preferred direction w4(r) in the receptive field, the dot product between each pair is positive, though small. In (b), with the stimulus center situated to the right of the receptive field center, local direction selectivity and local stimulus direction clash near the center of the receptive field-the dot products there are actually negative; but the negative terms are compensated, exactly, as we have seen, by the larger positive dot products in the periphery of the receptive field. 3 Development of Weight Vector Field under a Hebb Rule
Now we consider the general manner in which weight vector field changes during Hebbian learning. At each position on the image plane we use a set of MT-like units with different preferred directions. Let w4(r) denote the weight of the MT-like unit at position r with preferred direction 4. As before, the response or activation of the MT-like unit to the velocity field v(r) is
a&)
= c4(r)d4*
v(r)
(3.1)
Position-Independent Detectors of Rotation and Dilation
-
L
603
v(r) local stimulus direction w(r) local wei hted direction-selectivity in M!h receptive field
Figure 3: An intuitive interpretation of the mechanism for position independence. Negative dot products near the center of the receptive field in (b) are compensated by larger ones peripherally to give the same sum as in (a).
Figure 4 Each unit in the MST layer receives inputs from MT-like units at different positions and with different preferred directions (indicated by arrows). and the weight vector is defined as w4(r) = cg(r)w+(r)dg,with dg again being the unit vector for the preferred direction of an MT-like unit. The total input Z to the MST-like unit is the weighted s u m of the responses from all MT-like units within the receptive field (Fig. 41, i.e., summing
Kechen Zhang, Martin I. Sereno, and Margaret E. Sereno
604
over different positions in the receptive field as well as different preferred directions: (3.2)
where the identity w4(r)a4(r)= w4(r)c4(r)d4v(r) = w4(r). v(r) is used. We can treat the system as if there were only one MT-like unit specified by the weight vector at each position (call this the equivalent weight vector) w(r) :=
cwdr) c c4(r)w4(r)d4 =
4
4
so we can write equation 3.2 as
I = C w(r) v(r)
(3.3)
r
which is exactly the same as expression 2.3 in the previous section. As before, the output of the MST-like unit is 0 = u(1)
Suppose the increment of the weight in each training step follows a simple Hebb rule Aw4(r)= cu+(r)O
(3.4)
In the present model no explicit distinction has been made between excitatory and inhibitory synaptic connections, and the weights are allowed to change sign. Since w(r) = C0c4(r)w4(r)d4,the corresponding increment of the equivalent weight vector field is Aw(r) = Cc+(r)Aw+,(r)d4=
C [a$(r)d4 v(r)O] d4
4
4
where the last equality is obtained by substituting equation 3.1 into 3.4. The coefficient c4(r) is assumed to be 5 random variable with uniform distribution across the angle 4. Let c2 be the average of c$(r). It is assumed to be a constant across the image plane. As an approximation for large number of units, we have Aw(r) = c?* C [ d +. cOv(r)]d+
(3.5)
4
To simphfy this expression, first note that if there are n(> 3) unit vectors (d4) distributed evenly around the unit circle, then C ( d 4 V)d+ = nV 2 4
(3.6)
Position-IndependentDetectors of Rotation and Dilation
605
holds for all vector V. For a proof, write each vector as a complex number, namely, V = @" and d+ = e@, where 8 is the direction angle of V and p = )V1is the radius. Because d6 . V = pcos(8- 4) = &I (ei(@-6)+ e-@-6) is a real number, we have
1
Since is evenly distributed around the circle, C6e2'@ = 0. This proves equation 3.6. Assuming that the preferred directions of the MT-like units at each spatial position arepvenly distributed, we can employ formula 3.6 by identifying V with c2dv(r)so that equation 3.5 can be rewritten as 1 Aw(r) = -nec20v(r) (3.7) 2 where n is the number of the MT-like units at each position. This increment is caused by a single training step with the velocity field v(r). After training with a sequence of velocity fields, the equivalent weight vector field adds up to
w(r) = wo(r) + - n d Ov(r) 2 f I
-
C
(3.8)
where t(= 0,1,2,.. .) stands for all time steps in the training and wo(r) is the initial weight vector. In conclusion, the final equivalent weight vector field is just proportional to the sum of the training velocity fields weighted by the corresponding responses of the MST-like unit. 4 Training with Translation, Rotation, Dilation, and Contraction __
We are now ready to consider the training with translation, rotation, dilation, and contraction velocity fields. To begin with, suppose for a single training step the velocity field is a rotation centered at I, v(r) = w x (r - rc)
(4.1)
and in different steps both the angular velocity w and the center r, vary randomly. Substituting equation 4.1 into 3.8 and ignoring the initial weight vector for its smallness, we obtain the final weight vector field w(r) = qCOv(r) = q t
where v field
=
na?/2 is a constant. This can be identified with the rotational
w(r) = n x (r - ro)
(4.2)
606
Kechen Zhang, Martin I. Sereno, and Margaret E. Sereno
+
Figure 5: The final weight vector field is generally composed of a rotation field (a) and a dilation or contraction field (b). The result (c) is a spiral field. where the weight field "angular velocity" n and the weight field center ro are defined by n := r] Et Ow and n x ro := r] Ct(Ow x rc). The latter equation has a unique solution of ro as long as n # 0. In the special case n = 0, w(r) is a constant vector field (translation). Similarly, training with dilation or contraction v(r) = X(r - rc)
with rate X and center r, varying in time will lead to the final weight vector field
w(r) = A(r - ro)
(4.3)
where A := r] E, OX and Aro := 17 C, OXr,. This is either a dilation (A > 0) or a contraction (A < 0). In the special case A = 0, w(r) is a constant (translation). Note that expressions 4.2 and 4.3 are just what are required for position-independent responses (cf. equations 2.4 and 2.6). It should be realized that the center ro does not affect the curl and divergence of a vector field, and thus does not affect our previous conclusions. For training with a mixture of translations, rotations, dilations, and contractions, it is readily shown by similar argument that the final weight vector field takes the form
w(r) = n x (r - a)
+ A(r - b) + c
It can always be written equivalently as
w(r) = n x (r - ro)
+ A(r - ro)
which is a spiral centered at ro (Fig. 5). An MST-like unit with a spiral
Position-IndependentDetectors of Rotation and Dilation
607
weight vector field has position-independent responses to a particular sense of rotation as well as to either a dilation or contraction. Even if the training velocity fields have a zero average, for example, clockwise and counterclockwise rotations have an equal chance of appearing, the weight vector field is still expected to grow with time. We consider the simple case where all rotations and dilationslcontractions are centered at the same point so that the development of the two corresponding components is strictly independent. Consider the initial stage of development for training with, say, rotation fields alone. Now we need consider only the linear range of the sigmoid function c, and for simplicity we assume 0 = I. According to equations 3.7 and 3.3, at time step t 1
+
where the subscripts refer to time. Thus
It can be expressed as Ot+l = AfWt+l
+ 77(AQ,wt)(Awtw*+1)
(4.4)
where at and wf are the angular speeds for the vector fields wt and vf at time t, respectively, and A is a constant depending on the size and shape of the receptive field as well as the position of the rotation center. Imagine an ensemble of parallel training sessions starting from different initial weights and using different rotation sequences of random angular speeds, which are independent of each other while having identical statistics. We take the ensemble average on both sides of equation 4.4 to get (Ot+l)= A (a,) (u)+qA2 (Q,) (w’) (w), where the subscript for the angular speed w is dropped because the statistics of w does not change over time. If (w)= 0, then (Ot+~) = 0 for all t. However, taking the ensemble average after squaring equation 4.4 and using (q) = A2 (0:) (w2), we can obtain (4.5)
where & := A ( w 2 ) = ( Z r v.v) Q
:=
(w4)/(3>’
are constants. When w is drawn from a gaussian distribution of zero
Kechen Zhang, Martin I. Sereno, and Margaret E. Sereno
608
mean, for instance, (r = 3. Applying equation 4.5 iteratively yields (q+l) = e’/r (q), where constant T := I /
ln(1 + 27/f + tr$€*)
+
Because 0 1 = C, w1 . v1 and w1 = wo qOovo= ;rlOovo= r/(C wo . vo)vo, by similar arguments as above we can get (@) = (rr?€’ (0;): Hence (0:)= f$C*e(‘-1)/‘
(03
(4.6)
Consequently, when many MST-like units develop in parallel starting from random initial weights, the responses (either positive or negative) to rotation and to dilation/contraction are expected to grow exponentially in the initial stage of development. The variety of the initial responses leads to a continuous spectrum of selectivity to rotation and dilation/contraction, which is what has actually been found in the neurophysiological experiments (Duffy and Wurtz 1991a; Andersen et al. 1991). 5 Discussion
The model provides a unified, albeit simplified, account for several essential properties of MST neurons and how they might develop. These properties include selectivity to rotation, dilation, and contraction, the position independence of the responses (Saito et al. 1986; Tanaka and Saito 1989; Tanaka et al. 1989; Duffy and Wurtz 1991a,b), the selectivity to spiral velocity fields (Graziano et al. 1990; Andersen et al. 1991), and the continuous spectrum of selectivity (Duffy and Wurtz 1991a; Andersen et al. 1991). The model’s response saturates at higher speeds (as a result of the sigmoid function) as does the response of real neurons (Orban et al. 1992). In addition to rotation and dilation/contraction, shear also naturally arises in the optic flow (Koenderink 1986). Since the linear combination of shear fields is still a shear, according to equation 3.8 the weight vector field itself will also have a shear component. Consistent with the model, neurons selective to shear components were also found in the cortical areas including MST (Lagae et al. 1991). This model differs somewhat from the original model in Sereno and Sereno (1991)and from real MST neurons in that it “linearly decomposes’’ the velocity field-that is, an MST-like unit will respond exclusively to the, say, rotational component of a flow field, regardless of the magnitude of the radial component. Since a cosine tuning curve means that the input unit sees exactly the vector component of the local stimulus movement in the preferred (here rotational) direction, it leads to linear decomposition. With narrower tuning curves, the response of individual MST-like units provides more information about the exact composition of the flow field-for example, the extent to which it approximates a pure
Position-Independent Detectors of Rotation and Dilation
609
rotation; nevertheless, approximate position independence with narrower tuning curves is still explained by a direction-template mechanism like that described above. Roughly speaking, learning with a simple Hebb rule tends to maximize the total response by gradient ascent and thus tune the net to the input patterns that frequently occur. Consider the output 0 = o(Z) = u ( F w ; l ; )
The Hebb rule Awi a Z;O is always of the same sign as the gradient of the function E
aE
=
:02:
ao aw;= Z;Oo’(l)
- = 0-
aw;
because the derivative o‘ is always positive. As a consequence, there should be a general tendency for local direction selectivity to be aligned with the direction of the stimulus velocity. Recently, it was demonstrated that although dilation-sensitive MSTd neurons are basically position invariant in their responses, they often respond best to dilations centered at a particular location in the receptive field (often not the receptive field center) (Duffyand Wurtz 1991~).Similar results were obtained in the simulations in Sereno and Sereno (1991) using MT-like (narrower) input-layer tuning curves. It may be advantageous to retain information about combinations of flow field componentshere, dilation and translation-in single units since these combinations can have particular behavioral relevancefor example, in signaling direction of heading (Perrone 1992). More realistic peaked (instead of linear) speed tuning curves (Maunsell and Van Essen 1983) in the MT-like input layer could potentially sharpen the response to particular flow components since local speeds may be changed from the optimum as flow field components are added. Cross-direction inhibition (known to occur in MT: Snowden et al. 1991) could also be incorporated, effectively deleting portions of the flow field containing conflicting local motion signals. This could improve performance with more complex, real-world motion arrays. The rotation, dilation, and contraction velocity fields required for training are readily produced when an animal is moving around in a rigid environment. Exposure to such velocity fields may be crucial in order for a young animal to develop rotation and dilation cells in its visual system. Human babies, for instance, can distinguish a rotation field from a random velocity field only after several months of visual experience (Spitz et al. 1988). This could be tested by recording from MST in infant monkeys.
610
Kechen Zhang, Martin I. Sereno, and Margaret E. Sereno
Feedforward networks using Hebb rules have been shown to be capable of producing detectors selective to a hierarchy of features like those found in the successive stages of visual processing: center-surround units like those in the LGN (Linsker 1986a),orientation-selective units like simple cells in V1 (Linsker 1986b), pattern motion units like some cells in MT (Sereno 1989), and finally position-independent rotation and dilation units like cells in dorsal MST (Sereno and Sereno 1991). The visual system may use simple local learning rules and a richly textured environment to build up complex filters in stages. This strategy could drastically reduce the amount of supervision that is required later on (cf. Geman et al. 1992) as the visual system learns to recognize objects and direct navigation and manipulation.
Note Added in Proof Recently, Gallant et al. (1993) found that neurons in V4 respond selectively, and in a position-invariant way to static patterns containing concentric, radiating, shearing, or spiral contours. The main outlines of our analysis could be extended to explain the selectivity and development of these neurons by substituting an orientation-selective input layer for the direction-selective input layer considered here.
Acknowledgments
M. E. S. was supported by a postdoctoral fellowship, K. Z. by a graduate fellowship, and M. I. S. by a research award from the McDonnell-Pew Center for Cognitive Neuroscience at San Diego. We thank an anonymous reviewer for helpful comments.
References Andersen, R., Graziano, M., and Snowden, R. 1991. Selectivity of area MST neurons for expansion/contraction and rotation motions. Invest. Opkthal. Vis.Sci., Abstr. 32, 823. Duffy, C. J., and Wurtz, R. H. 1991a. Sensitivity of MST neurons to optic flow stimuli. I. A continuum of response selectivity to large-field stimuli. J. Neurophysiol. 65, 1329-1345. Duffy, C. J., and Wurtz, R. H. 1991b. Sensitivity of MST neurons to optic flow stimuli. 11. Mechanisms of response selectivity revealed by small-field stimuli. J. Neurophysiol. 65, 1346-1359. Duffy, C. J., and Wurtz, R. H. 1991~.MSTd neuronal sensitivity to heading of motion in optic flow fields. SOC. Neurosci. Abstr. 17, 441. Felleman, D., and Van Essen, D. C. 1991. Distributed hierarchical processing in primate cerebral cortex. Cerebral Cortex 1, 147.
Position-Independent Detectors of Rotation and Dilation
611
Gallant, J. L., Braun, J., and Van Essen, D. C. 1993. Selectivity for polar, hyperbolic, and Cartesian gratings in macaque visual cortex. Science 259, 100-103. Geman, S., Bienenstock, E., and Doursat, R. 1992. Neural networks and the bias/variance dilemma. Neural Comp. 4, 1-58. Graziano, M. S. A,, Andersen, R. A., and Snowden, R. 1990. Stimulus selectivity of neurons in macaque MST. SOC.Neurosci. Abstr. 16, 7. Koenderink, J. J. 1986. Optic flow. Vision Res. 26, 161-180. Koenderink, J. J., and van Doorn, A. J. 1975. Invariant properties of the motion parallax field due to the movement of rigid bodies relative to an observer. Opt. Acta 22, 773-791. Koenderink, J. J., and van Doorn, A. J. 1976. Local structure of movement parallax of the plane. J . Opt. SOC.Am. 66, 717-723. Lagae, L., Xiao, D., Ralguel, S., Maes, H., and Orban, G. A. 1991. Position invariance of optic flow component selectivity differentiates monkey MST and FST cells from MT cells. Invest. Ophthal. Vis. Sci., Abstr. 32, 823. Linsker, R. 1986a. From basic network principles to neural architecture: emergence of spatial-opponent cells. Proc. Natl. Acud. Sci. U.S.A. 83, 7508-7512. Linsker, R. 1986b. From basic network principles to neural architecture: emergence of orientation-selective cells. Proc. Natl. Acad. Sci. U.S.A.83,8390-8394. Longuet-Higgins, H. C., and Prazdny, K. 1980. The interpretation of a moving retinal image. Proc. R. SOC.London B 208,385-397. Maunsell, J. H. R., and Van Essen, D.C. 1983. Functional properties of neurons in middle temporal visual area (MT) of the macaque monkey: I. Selectivity for stimulus direction, speed and orientation. 1.Neurophysiol. 49, 1127-1147. Movshon, J. A., Adelson, E. H., Gizzi, M. S., and Newsome, W. T. 1985. Analysis of moving visual patterns. In Pattern Recognition Mechanisms, C. Chagas, R. Gattass, and C. Gross, eds., pp. 117-151. Springer-Verlag, New York. Orban, G. A., Lagae, L., Verri, A., Raiguel, S., Xiao, D., Maes, H., and Torre, V. 1992. First-order analysis of optical flow in monkey brain. Proc. Natl. Acad. Sci. U.S.A. 89, 2595-2599. Perrone, J. A. 1992. Model for the computation of self-motion in biological systems. J . Opt. SOC.Am. A 9, 177-194. Poggio, T., Verri, A., and Torre, V. 1990. Does cortical area MST know Green theorems? Instituto per la Ricerca Scientifica e Technologica Tech. Rep. No. 900807,143. Poggio, T., Verri, A., and Torre, V. 1991. Green theorems and qualitative p r o p erties of the optical flow. MIT A.I. Memo, No. 1289, 1-6. Rodman, H. R., and Albright, T. D. 1987. Coding of visual stimulus velocity in area MT of the macaque. Vision Res. 27,2035-2048. Saito, H., Yukie, M., Tanaka, K., Hikosaka, K., Fukada, Y., and Iwai, E. 1986. Integration of direction signals of image motion in the superior temporal sulcus of the macaque monkey. 1.Neurosci. 6, 145-157. Sakata, H., Shibutani, H., Ito, Y., and Tsurugai, K. 1986. Parietal cortical neurons responding to rotary movement of visual stimulus in space. Exp. Brain Res. 61, 658-663. Sereno, M. I. 1989. Learning the solution to the aperture problem for pattern
612
Kechen Zhang, Martin I. Sereno, and Margaret E. Sereno
motion with a Hebb rule. In Advances in Neural Information Processing System I, D. S. Touretzky, ed., pp. 468-476. Morgan Kaufmann, San Mateo, CA. Sereno, M. I., and Allman, J. M. 1991. Cortical visual areas in mammals. In The Neural Basis ofvisuul Function, A. G . Leventhal, ed., pp. 160-172. Macmillan, London. Sereno, M. I., and Sereno, M. E. 1990. Learning to discriminate senses of rotation and dilation with a Hebb rule. Invest. Ophthl. Vis. Sci., Abstr. 31, 528. Sereno, M. I., and Sereno, M. E. 1991. Learning to see rotation and dilation with a Hebb rule. In Advances in Neural Information Processing Systems 3, R. P. Lippmann, J. Moody, and D. S. Touretzky, eds., pp. 320-326. Morgan Kaufmann, San Mateo, CA. Snowden, R. J., Treue, S., Erickson, R. G., and Andersen, R. A. 1991. The response of area MT and V1 neurons to transparent motion. J. Neurosci. 11, 2768-2785. Spitz, R. V., Stiles-Davis, J., and Siegel, R. M. 1988. Infant perception of rotation from rigid structure-from-motion displays. SOC.Neurosci. Abstr. 14, 1244. Tanaka, K., and Saito, H.-A. 1989. Analysis of motion of the visual field by direction, expansion/contraction, and rotation cells clustered in the dorsal part of the medial superior temporal area of the macaque monkey. 1. Neurophysiol. 62, 626-641. Tanaka, K., Hikosaka, K., Saito, H.-A., Yukie, M., Fukada, Y., and Iwai, E. 1986. Analysis of local and widefield movements in the superior temporal visual areas of the macaque monkey. J.Neurosci. 6, 134-144. Tanaka, K., Fukada, Y., and Saito, H.-A. 1989. Underlying mechanisms of the response specificity of expansion/contraction and rotation cells in the dorsal part of the medial superior temporal area of the macaque monkey. J. Neurophysiol. 62, 642-656. Received 11 October 1991; accepted 11 January 1993.
This article has been cited by: 2. S. Pitzalis, M.I. Sereno, G. Committeri, P. Fattori, G. Galati, F. Patria, C. Galletti. 2010. Human V6: The Medial Motion Area. Cerebral Cortex 20:2, 411-424. [CrossRef] 3. Peter J. Bex, Andrew B. Metha, Walter Makous. 1998. Psychophysical evidence for a functional hierarchy of motion processing mechanisms. Journal of the Optical Society of America A 15:4, 769. [CrossRef] 4. Ruye Wang . 1995. A Simple Competitive Account of Some Response Properties of Visual Neurons in Area MSTdA Simple Competitive Account of Some Response Properties of Visual Neurons in Area MSTd. Neural Computation 7:2, 290-306. [Abstract] [PDF] [PDF Plus]
Communicated by Andrew Barto
Improving Generalization for Temporal Difference Learning: The Successor Representation Peter Dayan Computational Neurobiology Laboratory, The Salk Institute, P.0.Box 85800, Sun Diego, CA 92186-5800 USA Estimation of returns over time, the focus of temporal difference (TD) algorithms, imposes particular constraints on good function approximators or representations. Appropriate generalization between states is determined by how similar their successors are, and representations should follow suit. This paper shows how TD machinery can be used to learn such representations, and illustrates, using a navigation task, the appropriately distributed nature of the result. 1 Introduction
The method of temporal differences (TD, Samuel 1959; Sutton 1984,1988) is a way of estimating future outcomes in problems whose temporal structure is paramount. A paradigmatic example is predicting the long-term discounted value of executing a particular policy in a finite Markovian decision task. The information gathered by TD can be used to improve policies in a form of asynchronous dynamic programming (DP; Watkins 1989; Barto et al. 1989; Barto et al. 1991). As briefly reviewed in the next section, TD methods apply to a learning framework, which specifies the goal for learning and precisely how the system fails to attain this goal in particular circumstances. Just like a proposal to minimize mean square error, TD methods lie at the heart of different mechanisms operating over diverse representations. Representation is key-difficult problems can be rendered trivial if looked at in the correct way. It is particularly important for systems to be able to learn appropriate representations, since it is rarely obvious from the outset exactly what they should be. For static tasks, generalization is typically sought by awarding similar representations to states that are nearby in some space. This concept extends to tasks involving prediction over time, except that adjacency is defined in terms of similarity of the future course of the behavior of a dynamic system. Section 3 suggests a way, based on this notion of adjacency, of learning representations that should be particularly appropriate for problems to which TD techniques have been applied. Learning these representations can be viewed as a task itself amenable to TD methods, and so requires Neural Computation 5, 613-624 (1993)
@ 1993 Massachusetts Institute of Technology
Peter Dayan
614
no extra machinery. Section 4 shows the nature of the resulting representation for a simple navigation task. Part of this work was reported in Dayan (1991a,b). 2
TD Learning
Consider the problem of estimating expected terminal rewards, or returns, in a finite absorbing Markov chain; this was studied in the context of TD methods by Sutton (1988). An agent makes a transition between nonabsorbing states i and j E N according to the ijth element of the Markov matrix Q, or to absorbing state k E 7 with probability Sik, with a stochastic reinforcement or return whose mean is t k and whose variance is finite. In this and the next section, the returns and transition probabilities are assumed to be fixed. The immediate expected return from state i E N, represented as the ith element of a vector h, is the sum of the probabilities of making immediate transitions to absorbing states times the expected returns from those states:
The overall expected returns, taking account of the possibility of making transitions to nonabsorbing states first, are [f]i
+
+
+
=
[h]; [Qh]; [Q2hIi . . .
=
[ ( I - Q)-'h];
(2.1)
where 1 is the identity matrix. The agent estimates the overall expected return from each state (compiled into a vector r) with a vector-valued function i(w), which depends on a set of parameters w whose values are determined during the course of learning. If the agent makes the transition from state if to iltl in one observed sequence, TD(0) specifies that w should be changed to reduce the error: frti =
[i(w)li,+l- ['(w)li,
(2.2)
where, for convenience, [i(w)] is taken to be the delivered return ~ i , + if if+l is absorbing. This enforces a kind of consistency in the estimates of the overall returns from successive states, which is the whole basis of TD learning. More generally, information about the estimates from later states [i(w)] for s > 1 can also be used, and Sutton (1988) defined the TD(A) algorithm, which weighs their contributions exponentially less according to As. With the TD algorithm specifying how the estimates should be manipulated in the light of experience, the remaining task is one of function approximation. How w should change to minimize the error fI+l +% ,;
~
Improving Generalization for Temporal Difference Learning
615
in equation 2.2 depends on exactly how w determines [i(w)],,.Sutton (1988) represented the nonabsorbing states with real-valued vectors {xi}, [r(w)lias the dot product w xi of the state vector with w taken as a vector of weights, and changed w in proportion to
-
-(w . xi,+, - w . Xi,)Xi, using z;,+~instead of w . xi,+l if it+l is absorbing. This is that part of the gradient -Vwct+l that comes from the error at step xi,, ignoring the contribution from xi,+l (Werbos 1990; Dayan 1992). In the "batch-learning" case for which the weights are updated only after absorption, Sutton showed that if the learning rate is sufficiently small and the vectors representing the states are linearly independent, then the expected values of the estimates converge appropriately. Dayan (1992) extended this proof to show the same was true of TD(X) for 0 < x < 1. 3 Time-Based Representations
One of the key problems with TD estimation, and equivalently with TD based control (Barto et al. 19891, is the speed of learning. Choosing a good method of function approximation, which amounts in the linear case to choosing good representations for the states, should make a substantial difference. For prediction problems such as the one above, the estimated expected overall return of one state is a biased sum of the estimated expected overall returns of its potential successors. This implies that for approximation schemes that are linear in the weights w, a good representation for a state would be one that resembles the representations of its successors, being only a small Euclidean distance away from them (with the degrees of resemblance being determined by the biases). In this way, the estimated value of each state can be partially based on the estimated values of those that succeed it, in a way made more formal below. For conventional, static, problems, received wisdom holds that distributed representations perform best, so long as the nature of the distribution somehow conforms with the task-nearby points have nearby solutions. The argument above suggests that the same is true for dynamic tasks, except that neighborliness is defined in terms of temporal succession. If the transition matrix of the chain is initially unknown, this representation will have to be learned directly through experience. Starting at state i E N, imagine trying to predict the expected future occupancy of all other states. For the jth state, j E N, this should be [Xilj
=
+ [Qlij + [Q'lij +
[llij
= [ ( I - Q)-']ij.
.
'
(3.1)
where [MI;, is the ijth element of matrix M and 1 is the identity matrix. Representing state i using Xi is called the successor representation (SR).
Peter Dayan
616
A TD algorithm itself is one way of learning SR. Consider a punctate representation that devotes one dimension to each state and has the Ith element of the vector representing state k, [x& equal to [Ilk/. Starting from i, = i, the prediction of how often [xi,], = 1 for s 2 t is exactly the prediction of how often the agent will visit state j in the future starting from state i, and should correctly be [x,lj. To learn this, the future values of [xi,], for s 2 t can be used in just the same way that the future delivery of reinforcement or return is used in standard TD learning. For a linear function approximator, it turns out that SR makes easy the resulting problem of setting the optimal weights w*, which are defined as those making i = r(w*). If X is the matrix of vectors representing the states in the SR, [Xli, = [x,Ii, then W*is determined as
XTw* = f which implies, from equations 2.1 and 3.1, that W* = h
But h is just the expected immediate return from each state-it is insensitive to all the temporal dependencies that result from transitions to nonabsorbing states. The SR therefore effectively factors out the entire temporal component of the task, leaving a straightforward estimation problem for which TD methods would not be required. This can be seen in the way that the transition matrix Q disappears from the update equation, just as would happen for a nontemporal task without a transition matrix at all. For instance, for the case of an absorbing Markov chain with batch-learning updates, Sutton showed that the TD(0) update equation for the mean value of the weights W,l satisfies Wn+l = W,l
+ nXD(h + QX'W,
- XTW,,)
where X is the representation, a is the learning rate, and, since the updates are made after observing a whole sequence of transitions from start to absorption rather than just a single one, D is the diagonal matrix whose diagonal elements are the average number of times each state is visited on each sequence. Alternatively, directly from the estimates of the values of the states, (XTW,I+l- i) =
[I - aXTXD(Z- Q)](XTW,- i)
Using X instead, the update becomes W,l+l = W,
+ aXD(h - WIl)
or (Wtf+l
- h) = (I - a X D ) ( W , - h)
Improving Generalizationfor Temporal Difference Learning
617
Since X is invertible, Sutton’s proof that XTW, + r, and therefore that W,,+ h as n .+ 00, still holds. I conjecture that the variance of these estimates will be lower than those for other representations X (e.g., X = 0 because of the exclusion of the temporal component. For control problems it is often convenient to weigh future returns exponentially less according to how late they arrive-this effectively employs a discount factor. In this case the occupancy of future states in equation 3.1 should be weighed exponentially less by exactly the same amount. A possible objection to using TD learning for SR is that it turns the original temporal learning problem-that of predicting future reinforcement-into a whole set of temporal learning problems-those of predicting the future occupancy of all the states. This objection is weakened in two cases: 0
0
The learned predictions can be used merely to augment a standard representation such as the punctate one. An approximately appropriate representation can be advantageous even before all the predictions are quite accurate. Unfortunately this case is hard to analyze because of the interaction between the learning of the predictions and the learning of the returns. Such a system is used in the navigation example below. The agent could be allowed to learn the predictions by exploring its environment before it is first rewarded or punished. This can be viewed as a form of latent learning and works because the representation does not depend on the returns.
One could regard these predictions as analogous to the hidden representations in Anderson’s (1986) multilayer backpropagation TD network in that they are fashioned to be appropriate for learning TD predictions but are not directly observable and so have to be learned. Whereas Anderson‘s scheme uses a completely general technique that makes no explicit reference to states’ successors, SR is based precisely on what should comprise a good representation for temporal tasks. 4 Navigation Illustration
Learning the shortest paths to a goal in a maze such as the one in Figure 1 was chosen by Watkins (1989) and Barto et al. (1989) as a good example of how TD control works. For a given policy, that is, mapping from positions in the grid to directions of motion, a TD algorithm is used to estimate the distance of each state from the goal. The agent is provided with a return of -1 for every step that does not take it to the goal and future returns, that is, future steps, are weighed exponentially less using a discount factor. The policy is improved in an asynchronous form of
Peter Dayan
618
Agent Goal
Barrier ’ Figure 1: The grid task. The agent can move one step in any of the four directions except where limited by the barrier or by the walls. dynamic programming’s policy iteration by making more likely those actions whose consequences are better than expected. Issues of representation are made particularly clear in such a simple example. For the punctate case, there can be no generalization between states. Distributed representations can perform better, but there are different methods with different qualities. Watkins (1989), for a similar task, used a representation inspired by Albus’ CMAC (1975). In this case, CMAC squares which cover patches of 3 x 3 grid points are placed regularly over the grid such that each interior grid point is included in 9 squares. The output of the units corresponding to the squares is 0 if the agent is outside their receptive fields, and otherwise, like a radial basis function, is modulated by the distance of the agent from the tenter of the relevant square. Over most of the maze this is an excellent representation-locations that are close in the Manhattan metric on the grid are generally similar distances from the goal, and are also covered by many of the same CMAC squares. Near the barrier, however, the distribution of the CMACs actually hinders learning-locations close in the grid but on opposite sides of the barrier are very different distances from the goal, and yet still share a similar CMAC square representation. By contrast, the successor representation, which was developed in the previous section, produces a CMAC-like representation that adapts correctly to the barrier. If the agent explores the maze with a completely random policy before being forced to find the goal, the learned SR would closely resemble the example shown in Figure 2. Rather like a CMAC square, the representation decays exponentially away from the starting state (5,6)in a spatially ordered fashion-however, note SRs recognition
Improving Generalization for Temporal Difference Learning
619
Figure 2: The predictions of future occupancy starting from (5,6) after exploration in the absence of the goal. The z-coordinate shows the (normalized) predictions, and the barrier and the goal are overlaid. The predictions decay away exponentially from the starting location, except across the bamer. that states on the distant side of the bamer are actually very far away in terms of the task (and so the predictions are too small to be visible). Simulations confirm that using the SR in conjunction with a punctate representation leads to faster learning for this simple task (see Fig. 31, even if the agent does not have the chance to explore the maze before being forced to find the goal. This example actually violates the stationarity assumption made in Section 2 that transition probabilities and returns are fixed. As the agent improves its policy, the mean number of steps it takes to go from one state to another changes, and so the SR should change too. Once the agent moves consistently along the optimal path to the goal, locations that are not on it are never visited, and so the prediction of future occupancy of those should be 0. Figure 4 shows the difference between the final and initial sets of predictions of future occupancy starting from the same location ( 5 , 6 ) as before. The exponential decay along the path is caused by the discount factor, and the path taken by the agent is clear. If the task for the agent were changed such that it had to move from anywhere
Peter Dayan
620
.-- Rpunctate
- - --
-
-0
-
b
8 b .
&MAC
RSR, no latent learning RsR,
latent learning
'\'\
- \', *'
'Ib 8 '8
10
100
loo0 Learning iterations
Figure 3 Learning curves comparing punctate representation (Rpunctate),CMACsquares (&MAC) and a punctate representation augmented with the SR (RsR), in the latter case both with and without an initial, unrewarded, latent learning phase. TD control learning as in Barto et al. (1989) is temporarily switched off after the number of trials shown in the x-axis, and the y-axis shows the average number of excess steps the agent makes on the way to the goal starting from every location in the grid. Parameters are in Dayan (1991b). on the grid to a different goal location, this new form of the SR would actually hinder the course of learning, since its distributed character no longer correctly reflects the actual nature of the space. This demise is a function of the linked estimation and control, and would not be true for pure estimation tasks. 5 Discussion
This paper has considered some characteristics of how representation determines the performance of TD learning in simple Markovian environments. It suggests that what amounts to a local kernel for the Markov
Improving Generalization for Temporal Difference Learning
621
Figure 4: The degradation of the predictions. Both graphs show the differences between the predictions after 2000 steps and those initially-the top graph as a surface, with the barrier and the goal overlaid, and the bottom graph as a density plot. That the final predictions just give the path to the goal is particularly clear from the white (positive) area of the density plot-the black (negative) area delineates those positions on the grid that are close to the start point (5,6), and therefore featured in the initial predictions, but are not part of this ultimate path.
622
Peter Dayan
chain is an appropriate distributed representation, because it captures all the necessary temporal dependencies. This representation can be constructed during a period of latent learning and is shown to be superior in a simple navigation task, even over others that also share information between similar locations. Designing appropriate representations is a key issue for many of the sophisticated learning control systems that have recently been proposed. However, as Barto et al. (1991) pointed out, a major concern is that the proofs of convergence of TD learning have not been very extensively generalized to different approximation methods. Both Moore (1990) and Chapman and Kaelbling (1991) sought to exorcise the daemon of dimensionality by using better function approximation schemes, which is an equivalent step to using a simple linear scheme with more sophisticated input representations. Moore used kd trees (see Omohundro 1987, for an excellent review), which have the added advantage of preserving the integrity of the actual values they are required to store, and so preserve the proofs of the convergence of Q-learning (Barto et al. 1991; Watkins and Dayan 1992). However just like the CMAC representation described above, the quality of the resulting representation depends on an a priori metric, and so is not malleable to the task. Chapman and Kaelbling also used a tree-like representation for Qlearning, but their trees were based on logical formulas satisfied by their binary-valued input variables. If these variables do not have the appropriate characteristics, the resulting representation can turn out to be unhelpful. It would probably not afford great advantage in the present case. Sutton (1990),Thrun etal. (19911, and others have suggested the utility of learning the complete transition matrix of the Markov chain, or, for the case of control, the mapping from states and actions to next states. Sutton used this information to allow the agent to learn while it is disconnected from the world. Thrun, Moller and Linden used it implicitly to calculate the cost of and then improve a projected sequence of actions. The SR is less powerful in the sense that it provides only an appropriately distributed representation and not a veridical map of the world. A real map has the added advantage that its information is independent of the goals and policies of the agent; however, it is more difficult to learn. Sutton’s scheme could equally well be used to improve a system based on the learned representation. Sutton and Pinette (1985)discussed a method for control in Markovian domains that is closely related to the SR and that uses the complete transition matrix implicitly defined by a policy. In the notation of this paper, they considered a recurrent network effectively implementing the iterative scheme
where xi is the punctate representation of the current state i and Q is the
Improving Generalization for Temporal Difference Learning
623
transition matrix. x, converges to Xi from equation 3.1, the SR of state i. Rather than use this for representational purposes, however, Sutton and Pinette augmented Q so that the sum of future returns is directly predicted through this iterative process. This can be seen as an alternative method of eliminating the temporal component of the task, although the use of the recurrence implies that the final predictions are very sensitive to errors in the estimate of Q. The augmented Q matrix is learned using the discrepancies between the predictions at adjacent time steps-however, the iterative scheme complicates the analysis of the convergence of this learning algorithm. A particular advantage of their method is that a small change in the model (e.g., a slight extension to the barrier) can instantaneously lead to dramatic changes in the predictions. Correcting the SR would require relearning all the affected predictions explicitly. Issues of representation and function approximation are just as key for sophisticated as unsophisticated navigation schemes. Having a representation that can learn to conform to the structure of a task has been shown to offer advantages-but any loss of the guarantee of convergence of the approximation and dynamic programming methods is, of course, a significant concern.
Acknowledgments
I am very grateful to Read Montague, Steve Nowlan, Rich Sutton, Terry Sejnowski, Chris Watkins, David Willshaw, the connectionist groups at Edinburgh and Amherst, and the large number of people who read drafts of my thesis for their help and comments. I am especially grateful to Andy Barto for his extensive and detailed criticism and for pointers to relevant literature. Support was from the SERC.
References Albus, J. S. 1975. A new approach to manipulator control: The CerebellarModel Articulation Controller (CMAC). Transact. ASME: J. Dynam. Syst. Measure. Control 97, 220-227. Anderson, C. W. 1986. Learning and problem solving with multilayer connectionist systems. Ph.D. Thesis, University of Massachusetts, Amherst, MA. Barto, A. G., Sutton, R. S., and Watkins, C. J. C. H. 1989. Learning and sequential decision making. Tech. Rep. 89-95, Computer and Information Science, University of Massachusetts, Amherst, MA. Barto, A. G., Bradtke, S. J., and Singh, S. P. 1991. Real-time learning and control using asynchronous dynamic programming. TR 91-57, Department of Computer Science, University of Amherst, MA.
Peter Dayan
624
Chapman, D., and Kaelbling, L. P. 1991. Input generalization in delayed reinforcement learning: An algorithm and performance comparisons. Proceedings of the 1991 International Joint Conference on Artificial Intelligence, 726-731. Dayan, P. 1991a. Navigating through temporal difference. In Advances in Neural Information Processing, Vol. 3, R. P. Lippmann, J. E. Moody, and D. S. Touretzky, eds., pp. 464-470. Morgan Kaufmann, San Mateo, CA. Dayan, P. 1991b. Reinforcing connectionism: Learning the statistical way. Ph.D. Thesis, University of Edinburgh, Scotland. Dayan, P. 1992. The convergence of TD(A) for general A. Machine Learn. 8, 341-362. Moore, A. W. 1990. EfFcient memory-based learning for robot control. Ph.D. Thesis, University of Cambridge Computer Laboratory, Cambridge, England. Omohundro, S. 1987. Efficient algorithms with neural network behaviour. Complex Syst. 1, 273-347. Samuel, A. L. 1959. Some studies in machine learning using the game of checkers. Reprinted in Computers and Thought, E. A. Feigenbaum and J. Feldman, eds. McGraw-Hill, New York, 1963. Sutton, R. S. 1984. Temporal credit assignment in reinforcement learning. Ph.D. Thesis, University of Massachusetts, Amherst, MA. Sutton, R. S. 1988. Learning to predict by the methods of temporal difference. Machine Learn. 3, 9 4 . Sutton, R. S. 1990. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Proceedings of the Seventh lnternational Conference on Machine Learning. Morgan Kaufmann, San Mateo, CA.
Sutton, R. S., and Pinette, 8. 1985. The learning of world models by connectionist networks. In Proceedings of the Seventh Annual Conference of the Cognitive Science Society, pp. 54-64. Lawrence Erlbaum, Irvine, CA. Thrun, S. B., Moller, K., and Linden, A. 1991. Active exploration in dynamic environments. In Advances in Neural lnformation Processing, Vol. 3, R. P. Lippmann, J. E. Moody, and D. S. Touretzky, eds., pp. 450-456. Morgan Kaufmann, San Mateo, CA. Watkins, C. J. C. H. 1989. Learningfrom delayed rewards. Ph.D. Thesis, University of Cambridge, England. Watkins, C. J. C. H., and Dayan, P. 1992. &-learning. Machine Learn. 8,279-292. Werbos, P. J. 1990. Consistency of HDP applied to a simple reinforcement learning problem. Neural Networks 3, 179-189. Received 20 January 1992; accepted 20 October 1992.
This article has been cited by: 2. Sridhar Mahadevan. 2008. Representation Discovery using Harmonic Analysis. Synthesis Lectures on Artificial Intelligence and Machine Learning 2:1, 1-147. [CrossRef] 3. P. Read Montague, Brooks King-Casas, Jonathan D. Cohen. 2006. IMAGING VALUATION MODELS IN HUMAN CHOICE. Annual Review of Neuroscience 29:1, 417-448. [CrossRef] 4. Roland E. Suri , Wolfram Schultz . 2001. Temporal Difference Model Reproduces Anticipatory Neural ActivityTemporal Difference Model Reproduces Anticipatory Neural Activity. Neural Computation 13:4, 841-862. [Abstract] [PDF] [PDF Plus] 5. D.J. Foster, R.G.M. Morris, Peter Dayan. 2000. A model of hippocampally dependent navigation, using the temporal difference learning rule. Hippocampus 10:1, 1-16. [CrossRef] 6. Peter DayanReinforcement Learning . [CrossRef]
Communicated by Geoffrey Hinton
Discovering Predictable Classifications Jiirgen Schmidhuber’ Department of Computer Science, University of Colorado, Boulder, CO 80309, USA
Daniel Prelinger Institut fiir Informatik, Technische Universitat Miinchen, Arcisstrasse 21,8000 Miinchen 2, Germany
Prediction problems are among the most common learning problems for neural networks (e.g., in the context of time series prediction, control, etc.). With many such problems, however, perfect prediction is inherently impossible. For such cases we present novel unsupervised systems that learn to classih patterns such that the classifications are predictable while still being as specific as possible. The approach can be related to the IMAX method of Becker and Hinton (1989)and Zemel and Hinton (1991). Experiments include a binary stereo task proposed by Becker and Hinton, which can be solved more readily by our system.
1 Motivation and Basic Approach
Many neural net systems (e.g., for control, time series prediction) rely on adaptive submodules for learning to predict patterns from other patterns. Perfect prediction, however, is often inherently impossible. In this paper we study the problem of finding pattern class@cations such that the classes are predictable, while still being as specific as possible. To grasp the basic idea, let us discuss several examples. Example 1: Hearing the first two words of a sentence “Henrietta eats. .. ” allows you to infer that the third word probably indicates something to eat but you cannot tell what. The class of the third word is predictable from the previous words-the particular instance of the class is not. The class ”food” is not only predictable but also nontrivial and specific in the sense that it does not include aterything-”John,” for instance, is not an instance of “food.” *Current address: Institut fiir Informatik, Technische Universit5t Miinchen, Arcisstrasse 21,8000 Miinchen 2, Germany.
Neural Computation 5,625-635 (1993) @ 1993 Massachusetts Institute of Technology
626
Jiirgen Schmidhuber and Daniel Prelinger
The problem is to class+ patterns from a set of training examples such that the classes are both predictable and not too general. A general solution to this problem would be useful for discovering higher level structure in sentences generated by unknown grammars, for instance. Another application would be the unsupervised classification of different pattern instances belonging to the same class, as will be seen in the next example.
Example 2 (stereo task; due to Becker and Hinton 1989): There are two binary images called the "left" image and the "right" image. Each image consists of two "strips"+ach strip being a binary vector. The right image is purely random. The left image is generated from the right image by choosing, at random, a single global shift to be applied to each strip of the right image. An input pattern is generated by concatenating a strip from the right image with the corresponding strip from the left image. "So the input can be interpreted as a fronto-parallel surface at an integer depth. The only local property that is invariant across space is the depth (i.e. shift)'' (Becker and Hinton 1989). With a given pair of different input patterns, the task is to extract a nontrivial classification of whatever is common to both patterns-which happens to be the stereoscopic shift. Example 1 is an instance of the so-called asymmetric case: There we are interested in a predictable nontrivial classification of one pattern (the third word), given some other patterns (the previous words). Example 2 is an instance of the so-called symmetric case: There we are interested in the nontrivial common properties of two patterns from the same class. In its simplest form, our basic approach to unsupervised discovery of predictable classifications is based on two neural networks called TI and T2. Both can be implemented as standard backpropagation networks (Werbos 1974; LeCun 1985; Parker 1985; Rumelhart et al. 1986). With a given pair of input patterns, TI sees the first pattern and T2 sees the second pattern. Let us first focus on the asymmetric case. For instance, with the example 1 above TImay see a representation of the words "Henrietta eats," while T2 may see a representation of the word "vegetables." TZ'S task is to classify its input. Ti's task is not to predict T2's raw environmental input but to predict T2's output instead. Both networks have q output units. Let p E (1,. . . m } index the input patterns. T2 produces as an ou ut the classification yPv2 E (0,.. . 114 in response to an input vector xp? TI'S output in response to its input vector xPJ is the prediction ypJ E [0,. . . 114 of the current classification ypy2 emitted by T2. We have two conflicting goals which in general are not simultaneously satisfiable: (1)All predictions yp~' should match the corresponding classifications yPi2. (2) The ypt2 should be discriminative-different inputs xPv2 should lead to different classifications ypi2. We express the trade-off between (1) and (2) by means of two opposing costs.
Discovering Predictable Classifications
627
(1)is expressed by an error term M (for "Match): m
M=
Ilyp,' - yp*2112
(1.1)
p=l
Here llzlll denotes the euclidean norm. (2) is enforced by an additional error term 0 2 (for "Discrimination") to be minimized by T2 only. 0 2 will be designed to encourage significant euclidean distance between classifications of different input patterns. 02 can be defined in more than one reasonable way. The next section will list four alternative possibilities with mutual advantages and disadvantages. These alternatives include (1) a novel method for constrained variance maximization, (2) autoencoders, and (3) a recent technique called "predictability minimization" (Schmidhuber 1992). The total error to be minimized by T2 is EM+ (1 - E)D2
(1.2)
where 0 < E < 1 determines the relative weighting of the opposing error terms. In the asymmetric case, the total error to be minimized by TI is just €M
(1.3)
The error functions are minimized by gradient descent. This forces the predictions and classifications to be more like each other, while at the same time forcing the classifications not to be too general but to tell something about the current input. The procedure is unsupervised in the sense that no teacher is required to tell T2 how to classify its inputs. With the symmetric case (see example 2 above), both TI and T2 are naturally treated in a symmetric manner. They share the goal of uniquely representing as many of their input patterns as possible-under the constraint of emitting equal (and therefore mutually predictable) classifications in response to a pair of input patterns. Such classifications represent whatever abstract properties are common to both patterns of a typical pair. For handling such symmetric tasks in a natural manner, we only slightly modify TI'S error function for the asymmetric case, by iqroducing an extra "discriminating" error term D1 for TI. Now both T I ,1 = 1,2 minimize EM+ (1- E)D,
(1.4)
where alternative possibilities for defining the DIwill be defined in the next section. Figure 1 shows a system based on (equation 1.4) and a particular implementation of DI(to be explained in Section 2.4). The assumption behind our basic approach is that a prediction that closely (in the euclidean sense) matches the corresponding classification is a nearly accurate prediction. Likewise, two very similar (in the euclidean
JiirgenSchmidhuber and Daniel Prelinger
628
prediction
pndiction
prediction
pediction
prediction
prediction
prediction
pdictim
Figure 1: Two networks try to transform their different inputs to obtain the same representation. Each network is encouraged to tell something about its input by means of the recent technique for “predictability minimization.”This technique requires additional intrurepresentationalpredictors (8 of them shown above) for detecting redundancies among the output units of the networks. Alternatives are provided in the text.
sense) classifications emitted by a particular network are assumed to have very similar ”meaning.” It should be mentioned that in theory, even the slightest differences between classifications of different patterns are sufficient to convey all (maximal) Shannon information about the patterns (assuming noise-free data). But then close matches between predictions and classifications could not necessarily be interpreted as accurate predictions. The alternative designs of DJ (to be described below), however, will have the tendency to emphasize differences between different classifications by increasing the euclidean distance between them (sometimes under certain constraints, see Section 2). There is another reason why this is a reasonable thing to do: In a typical application, a classifier will function as a preprocessor for some higher level network. We usually do not want higher level input representations with different “meaning” to be separated by tiny euclidean distance. Weight sharing. If both TIand T2 are supposed to provide the same outputs in response to the same inputs (this holds for the stereo task but does not hold in the general case) then we need only one set of weights
Discovering Predictable Classifications
629
for both classifiers. This reduces the number of free parameters (and may improve generalization performance). Outline. The current section motivated and explained our basic approach. Section 2 presents various instances of the basic approach (based on various possibilities for defining 01). Section 3 mentions previous related work. Section 4 presents illustrative experiments and experimentally demonstrates advantages of our approach. 2 Alternative Definitions of
DI
This section lists four differentapproaches for defining DI, the term which enforces nontrivial discriminative classifications. Section 2.1 presents a novel method that encourages locally represented classes (like with winner-take-all networks). The advantage of this method is that the class representations are orthogonal to each other and easy to understand; its disadvantage is the low representation capacity. In contrast, the remaining methods can generate distributed class representations. Section 2.2 defines DIwith the help of autoencoders. One advantage of this straightforward method is that it is easy to implement. A disadvantage is that predictable information conveyed by some input pattern does not necessarily help to minimize the reconstruction error of an autoencoder (this holds for the stereo task, for instance). Section 2.3 mentions the Infomax approach for defining DIand explains why we do not pursue this approach. Section 2.4 finally defines DIby the recent method for predicfubility minimization (Schmidhuber 1992). An advantage of this method is its potential for creating distributed class representations with statistically independent components. 2.1 Maximizing Constrained Output Variance. We write
(2.1) and minimize DIsubject to the constraint vp:
C$'='
(2.2)
i
Here, as well as throughout the remainder of this paper, subscripts of symbols denoting vectors denote vector components: vi denotes the ith element of some vector D. is a positive constant, and y; denotes the mean of the ith output unit of TI. It is possible to show that the first term on the right-hand side of equation 2.1 is maximized subject to equation 2.2 if each input pattern is locally represented (just like with winnertake-all networks) by exactly one comer of the q-dimensional hypercube spanned by the possible output vectors, if there are sufficient output units
630
Jurgen Schmidhuber and Daniel Prelinger
(Prelinger 19921.' Maximizing the second negative term encourages each local class representation to become active in response to only ith of all possible input patterns. Constraint 2.2 is enforced by setting
where up?' is the activation vector (in response to x p i ) of a q-dimensional layer of hidden units of Ti,which can be considered as its unnormalized output layer. This novel method is easy to implement-it achieves an effect similar to the one of the recent entropy-based method by Bridle and MacKay (1992). 2.2 Autoencoders. With pattern p and classifier TI a reconstructor module A1 (another backpropagation network) receives yp,' as an input. The combination of TIand A1 functions as an autoencoder. The autoencoder is trained to emit the reconstruction bpi' of TI'Sexternal input xp,', thus forcing yp-' to tell something about xpg'. DIis defined as
2.3 Infomax. Following Linsker's Infomax approach (Linsker 1988), we might think of defining -DI explicitly as the mutual information between the inputs and the outputs of TI. We did not use Infomax methods in our experiments for the following reasons: (1)There is no efficient and general method for maximizing mutual information. (2)With our basic approach from Section 1, Infomax makes sense only in situations where it automatically enforces high variance of the outputs of the TI (possibly under certain constraints). This holds for the simplifying gaussian noise models studied by Linsker, but it does not hold for the general case. (3) Even under appropriate gaussian assumptions, with more than one-dimensional representations, Infomax implies maximization of functions of the determinant DET of the covariance matrix of the output activations (Shannon 1948). In a small application, Linsker explicitly calculated DET's derivatives. In general, however, this is clumsy. 'Simply maximizing the variance of the output units without obeying constraint 2.2 will not necessarily maximize the number of different classifications. Example: Consider a set of four different four-dimensional input patterns 1O00, 0100, 0010, Oool. Suppose the classifier maps the first two input patterns to the four-dimensional output pattern 1100 and the other two to 0011. This will yield a variance of 4. A "more discriminative" response would map each pattern to itself, but this will yield a lower variance of 3.
Discovering Predictable Classifications
631
2.4 Predictability Minimization. Schmidhuber (1992) shows how DI can be defined with the help of intrarepresentational adaptive predictors that try to predict each output unit of some TI from its remaining output units, while each output unit in turn tries to extract properties of the environment that allow it to escape predictability. This was called the principle of predictability minimization. This principle encourages each output unit of TI to represent environmental properties that are statistically independent from environmental properties represented by the remaining output units. The procedure aims at generating binary “factorial codes” (Barlow et al. 1989). It is our preferred method, because [unlike the methods used by Linsker (19881, Becker and Hinton (1989), and Zemel and Hinton (1991)l it has a potential for removing even nonlinear statistical dependencies2 among the output units of some classifier. Let us define D’ = --1 - $92 (2.4) 2 i
c(sy
where the sr’ are the outputs of Sl, the ith additional so-called intrarepresentational predictor network of TI (one such additional predictor network is required for each output unit of 7’1). The goal of Si is to emit the conditioned expectation of given {$”, k # i } . This goal is achieved by simply training Sl to predict 8’ from {I&’, k # i } (see Fig. 1). To encourage even distributions in output space, we slightly modify DI by introducing a term similar to the one in equation 2.1 and obtain
e’
(2.5) 3 Relation to Previous Work
Becker and Hinton (1989) solve symmetric problems (like the one of example 2, see Section 1) by maximizing the mutual information between the outputs of TI and T2 (IMAX). This corresponds to the notion of finding mutually predictable yet informative input transformations. One variation of the IMAX approach assumes that TI and T2 have single binary probabilistic output units. In another variation, TI and T2 have single real-valued output units. The latter case, however, requires certain (not always realistic) gaussian assumptions about the input and output signals (see also Section 2.3 on Infomax). In the case of vector-valued output representations, Zemel and Hinton (1991) again make simplifymg gaussian assumptions and maximize functions of the determinant D of the 9 x 9-covariance matrices ( D E T M A X ) of the output activations (Shannon 1948) (see Section 2.3). DETMAX *SteveNowlan has described an alternative nonpredictor-based approach for finding nonredundant codes (Nowlan 1988).
632
Jurgen Schmidhuber and Daniel Prelinger
can remove only linear redundancy among the output units. (It should be mentioned, however, that with Zemel’s and Hinton’s approach the outputs may be nonlinear functions of the inputs.) The nice thing about IMAX is that it expresses the goal of finding mutually predictable yet informative input transformations in a principled way (in terms of a single objective function). In contrast, our approach involves two separate objective functions that have to be combined using a relative weight factor. An interesting feature of our approach is that it conceptually separates two issues: (1) the desire for discriminating mappings from input to representation, and (2) the desire for mutually predictable representations. There are many different approaches (with mutual advantages and disadvantages) for satisfying (1). In the context of a given problem, the most appropriate alternative approach can be “plugged into” our basic architecture. Another difference between IMAX and our approach is that our approach enforces not only mutual predictability but also equality of yp,’ and yPs2. This does not affect the generality of the approach. Note that one could introduce additional “predictor networks”4ne for learning to predict yP,2from yp,’and another one for learning to predict yp~’from ypv2. Then one could design error functions enforcing mutual predictability (instead of using the essentially equivalent error function M used in this paper). However, this would not increase the power of the approach but would only introduce unnecessary additional complexity. In fact, one advantage of our simple approach is that it makes it trivial to decide whether the outputs of both networks essentially represent the same thing. The following section includes an experiment that compares IMAX to our approach. 4 Illustrative Experiments
The following experiments were conducted using an online backpropagation method with constant step size. With each experiment, positive training examples were randomly drawn from the set of legal pairs of input patterns. Details can be found in Schmidhuber and Prelinger (1992). 4.1 Finding Predictable Local Class Representations. This experiment was motivated by example 1 (see Section 1). At a given time, the “next” symbol emitted by a very simple stochastic “language generator” was not precisely predictable from the “previous” symbol but belonged to a certain class defined by the previous symbol. During training, at a given time TI saw the previous symbol while T2 saw the next symbol. T1minimized equation 1.3,T2 minimized equation 1.2 with D2 defined according to equations 2.1 and 2.2. Ten test runs with 15,000 training it-
Discovering Predictable Classifications
633
erations were conducted. T2 always learned to emit different localized representations in response to members of predictable classes, while superfluous output units remained switched off. 4.2 Stereo Task. The binary stereo experiment described in Becker and Hinton (1989)(see also example 2 in Section 1) served to compare IMAX to our approach. Becker and Hinton report that their system (based on binary probabilistic units) was able to extract the “shift” between two simple stereoscopic binary images only if IMAX was applied in successive ”layer by layer” bootstrap stages. In addition, they heuristically tuned the learning rate during learning. Finally they introduced a maximal weight change for each weight during gradient ascent. In contrast, the method described herein (based on continuous-valued units) does not rely on successive bootstrap stages or any other heuristic considerations. We minimized equation 1.4 with DIdefined by predictability minimization according to equation 2.5. With a first experiment, we employed a different set of weights for each network. With 10 test runs involving 100,000 training patterns the networks always learned to extract the stereoscopic shift. This performance of our nonbootstrupped system is comparable to the performance of Becker and Hinton’s bootstrapped system. With a second experiment, we used only one set of weights for both networks (this leads to a reduction of free parameters). The result was a significant decrease of learning tim-with 10 test runs the system needed between 20,000 and 50,000 training patterns to learn to extract the shift.
4.3 Finding Predictable Distributed Representations. Two properties of some binary input vector are the truth values of the following expressions:
1. There are more “ones” on the “right” side of the input vector than on the “left” side. 2. The input vector consists of more “ones” than “zeros.”
During one learning cycle, a randomly chosen legal input vector was presented to TI#another input vector randomly chosen among those with the feature combination of the first one was presented to Tz.TI and TZ were constrained to have the same weights. Input vectors with equal numbers of ones and zeros as well as input vectors with equal numbers of ones on both sides were excluded. We minimized equation 1.4 with DIdefined by an autoencoder (equation 2.3). Ten test runs involving 15,000 pattern presentations were conducted. The system always came up with a distributed near-binary representation of the possible feature combinations.
634
Jurgen Schmidhuber and Daniel Prelinger
With D, defined by modified predictability minimization (equation 2.5), with simultaneous training of both predictors and classifiers, 10 test runs involving 10,000 pattern presentations were conducted. Again, the system always learned to extract the two features. 5 Conclusion In contrast to the principled approach embodied by IMAX, our methods (1)tend to be simpler (e.g., do not require sequential layer by layer "bootstrapping" or learning rate adjustments-the stereo task can be solved more readily by our system), (2) do not require gaussian assumptions about the input or output signals, (3) do not require something like DETMAX, and (4) partly have (unlike DETMAX) a potential for creating classifications with statistically independent components (this holds for DI defined according to Section 2.4). In addition, our approach makes it easier to decide whether the outputs of both networks essentially represent the same thing. The experiments above show that the alternative methods of Section 2 can be useful for implementing the D,terms in equation 1.4 to obtain predictable informative input transformations. More experiments are needed, however, to become clear about their mutual advantages and disadvantages. It also remains to be seen how well the methods of this paper scale to larger problems.
Acknowledgments We thank Mike Mozer for fruitful discussions and Mike Mozer, Sue Becker, Rich Zemel, and an unknown referee for helpful comments on drafts of this paper. This research was supported in part by a DFG fellowship to J. Schmidhuber, as well as by NSF Award IN-9058450, Grant 90-21 from the James S. McDonnell Foundation.
References Barlow, H. B., Kaushal, T. P., and Mitchison, G. J. 1989. Finding minimum entropy codes. Neural Comp., 1(3), 412-423. Becker, S., and Hinton, G. E. 1989. Spatial coherence as an internal teacher for a neural network. Tech. Rep. CRG-TR-89-7, Department of Computer Science, University of Toronto, Ontario. Bridle, J. S., and MacKay, D. J. C. 1992. Unsupervised classifiers, mutual information and 'phantom' targets. In Advances in Neural Information Processing Systems 4, D. S. Lippman, J. E. Moody, and D. S. Touretzky, eds., pp. 10961101. Morgan Kaufmann, San Mateo, CA. LeCun, Y. 1985. Une procedure d'apprentissage pour rbseau A seuil asymbtrique. Proceedings of Cognitiva 85, Paris, 599-604.
Discovering Predictable Classifications
635
Linsker, R. 1988. Self-organization in a perceptual network. IEEE Computer 21, 105-117. Nowlan, S. J. 1988. Auto-encoding with entropy constraints. In Proceedings of ZNNS First Annual Meeting, Boston, M A . Also published in special supplement to Neural Networks. Parker, D. 8. 1985. Learning-logic. Tech. Rep. TR-47, Center for Comparative Research in Economics and Management Science, MIT. Prelinger, D. 1992. Diploma thesis. Institut fiir Informatik, Technische Universitat Miinchen. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing, Vol. 1, pp. 318-362. MIT Press, Cambridge, MA. Schmidhuber, J. H. 1992. Learning factorial codes by predictability minimization. Neural Comp. 4(6), 863-879. Schmidhuber, J. H., and Prelinger, D. 1992. Discovering predictable classifications. Tech. Rep. CU-CS-626-92, Department of Computer Science, University of Colorado at Boulder. Shannon, C. E. 1948. A mathematical theory of communication (parts I and 11). BelI System Tech. 1. XXVII, 379-423. Werbos, P. J. 1974. Beyond regression: N a o tools for prediction and analysis in the behavioral sciences. Ph.D. thesis, Harvard University. Zemel, R. S., and Hinton, G. E. 1991. Discovering viewpoint-invariant relationships that characterize objects. In Advances in Neural Information Processing Systems3, D. S. Lippman, J. E. Moody, and D. S. Touretzky, eds., pp. 299-305. Morgan Kaufmann, San Mateo, CA. Received 2 June 1992; accepted 5 January 1993.
This article has been cited by: 2. Sepp Hochreiter , Jürgen Schmidhuber . 1999. Feature Extraction Through LOCOCODEFeature Extraction Through LOCOCODE. Neural Computation 11:3, 679-714. [Abstract] [PDF] [PDF Plus] 3. Virginia R. de Sa , Dana H. Ballard . 1998. Category Learning Through Multimodality SensingCategory Learning Through Multimodality Sensing. Neural Computation 10:5, 1097-1117. [Abstract] [PDF] [PDF Plus] 4. Jim Kay, W. A. Phillips. 1997. Activation Functions, Computational Goals, and Learning Rules for Local Processors with Contextual GuidanceActivation Functions, Computational Goals, and Learning Rules for Local Processors with Contextual Guidance. Neural Computation 9:4, 895-910. [Abstract] [PDF] [PDF Plus] 5. J. Schmidhuber, S. Heil. 1996. Sequential neural text compression. IEEE Transactions on Neural Networks 7:1, 142-146. [CrossRef]
Communicated by Thomas Bart01
A Kinetic Model of Short- and Long-Term Potentiation M. Migliore lnstitute for lnterdisciplinary Applications of Physics, National Research Council, Via Arckirafi 36,l-90123 Palermo, Italy
G . F. Ayala Department of Psyckopkysiology, University of Palermo, Via Pascoli 8,l-90143 Palermo, ltaly
We present a kinetic model that can account for several experimental findings on short- and long-term potentiation (STP and LTP) and their pharmacological modulation. The model, which is consistent with Hebb’s postulate, uses the hypothesis that part of the origin of LTP may be a consequence of an increased release of neurotransmitter due to a retrograde signal. The operation of the model is expressed by a set of irreversible reactions, each of which should be thought of as equivalent to a set of more complex reactions. We show that a retrograde signal alone is not sufficient to maintain LTP unless long-term change of the rate constant of some of the reactions is caused by high-frequency stimulation. Pharmacological manipulation of LTP is interpreted as modifications of the rate constants of one or more of the reactions that express a given mechanism. The model, because of its simplicity, can be useful to test more specific mechanisms by expanding one or more reactions as suggested by new experimental evidence. 1 Introduction STP ( C o h o etal. 1992)and LTP (Bliss and Lomo 1973; Bliss and GardnerMedwin 1973)are the short- or long-lasting increases of synaptic coupling that follow a train of conditioning stimuli. These phenomena are highly reproducible, nevertheless the molecular mechanisms involved in their induction and maintenance are still unclear (Edwards 1991). They are perhaps the most elementary step toward higher brain functions such as memory, learning, associative recalling, and the process of cognition. At this time, only a key role of calcium is well accepted and experimentally demonstrated. Although considerable efforts have been made (Byrne and Berry 1988; Koch and Segev 1989), using a variety of computer models (e.g., Gamble and Koch 1987; Segev and Rall1988; Holmes and Levy 1990), simulations have failed so far to reproduce their central feature, that is the short- or long-lasting enhancement of the excitatory Neural Computation 5, 636-647 (1993) @ 1993 Massachusetts Institute of Technology
A Kinetic Model of Short- and Long-Term Potentiation
637
I
retrograde
presynaptic signal
a
bouton
neurotransmitter level
postsynaptic signal
spine
Figure 1: Schematic representation of the model. Note that the equations derived from this model are a set of simultaneous differentialequations. Each step should be thought of as a representation of a more complex set of events. The two pathways of the presynaptic signal for the production of V are independent in this model. postsynaptic potentials (EPSPs) obtained after high-frequency stimulation (HE) of the afferent pathway. The model presented in this paper mimics this experimental observation. It is based on the growing evidence of a determining role of a retrograde signal (Bredt and Snyder 1992) for the induction of the potentiation, and indicates further that, in order to maintain the potentiation, the rate constant of some reactions should change during the HFS. 2
T h e Model
Our model (Fig. 1)represents a simplified view of the events of the synaptic transmission, where I is an intensive independent variable that represents the input stimulus, and V, C, and K are, respectively, the level of released neurotransmitter molecules, the postsynaptic signal, and the retrograde signal. The operation of the model is expressed by a set of irreversible events, symbolically represented by the following reactions (and corresponding simultaneous differential equations), each of which should be thought of as equivalent to a set of more complex reactions:
I - % V V P C C A K I + K a, V
c’.
(a)
(b) (C)
(d) (e)
The release of neurotransmitter on the synaptic cleft and the production of an output signal are represented, respectively, by reactions (a) and
M. Migliore and G.F. Ayala
638
(b). Reaction (e) represents the degradation of C that could be thought of as the EPSP. Processes represented in these reactions are the classic synaptic events having ample experimental support. Reactions (c) and (d), instead, model (in a way that shall prove productive) the current controversial view (Edwards 1991) that the origin of LTP is due to an increased release of neurotransmitter secondary to a retrograde signal [e.g., nitric oxide (Bredt and Snyder 199211 acting in the feedback loop. In these reactions the variable K is the level of this retrograde signal or plasticity factor (Bliss and Lynch 1988). The system of differential equations derived from this model can be solved analytically. Linear stability analysis (Nicolis and Prigogine 1977) shows that the system is always stable. The only stationary solution that it admits, for 1 > 0, is
Let us now consider the behavior of the system as a consequence of a short pulse of I. While the pulse lasts, the values of V, C, and K will tend to those in equations 1, reaching final values VO, CO,and &, which will depend on the pulse length. From the end of the pulse on, the behavior of the system is described by
v
= Voe-flr
(2.1)
e-Pt)
(2.2)
An interesting consequence of equation 2.2 is that for CO= 0 it reduces to the widely used phenomenological formula describing the time course of EPSPs in terms of empirical times of onset, TO and decay, TD, that is, V E E P = COnSt[TD/(TD-TD)](eWf/m- e d T O ) , The phenomenological formula is thus derived from our model with TO = I/@and TO = l / ( y E ) . Integration of the system of differential equations derived from the model has been carried out with a fourth order Runge-Kutta method (Press et al. 1987)with a fixed time step of 1.O x W4. It should be stressed that the model is too simplified to justify a quantitative comparison with specific biophysical mechanisms that, at the time being, are thought to have a major role in the induction and maintenance of LTP. At this stage, for example, y could represent all those mechanisms that may result in the production of a retrograde signal (e.g., from calcium buffering by calmodulin to NO synthesis by nitric oxide synthase), p the kinetics of activation of ionic currents and E might include leakage, kinetics of
+
A Kinetic Model of Short- and Long-Term Potentiation
639
inactivation of ionic currents and Ca2+pumping, diffusion, and buffering. As we will see, to be qualitatively consistent with experiments, it is not necessary to specify in more details any of these processes, at least within the limits of the scope of this paper. 3 Results and Discussion
3.1 Simulations. Although the values of the rate constants used in our simulations were set arbitrarily, they were chosen to give a relationship among them that is physiologically reasonable and to be convenient from a computational point of view. In fact, E , the rate constant of the “postsynaptic signal degradation,” is the smallest of all the rate constants in order to use most of the postsynaptic signal ”for LTP purposes”; 0 > (y + e), that is, the rate constant of ”channel activation” for the production of C is larger than the sum of the rate constants of the “production of the retrograde signal“ K, and of the “postsynaptic signal degradation,” so that C is always > 0; 6, the rate constant for “the production of neurotransmitter, V, by the retrograde signal,” has a value to yield an amplitude of STP roughly comparable with experimental observations, and y is such that the 1/(7 E ) time constant is not too long, to avoid unnecessarily long simulation times. It turns out that all values are within one order of magnitude, and different values have been tried with no essential differences in the results. The actual model parameters are reported in the legend of Figure 2. The time course of quantities shown in Figure 2 [as simulated according to the equations derived from reactions (a)-(e)] starts from equilibrium with 1 = 0.01, V = VOZ V,, C = CO C, and K = KO K,. If there is enough time between stimuli for all the variables to return to equilibrium, each new stimulus will find the system in unperturbed conditions. This, however, will not be true in the HFS case, that is when the interstimulus interval is shorter than any of the time constants involved in equations 2 in EPSP decaying, that is y and E. In such case, and since > (y E), the average values of V and C will increase. As shown in Figure 2, at the end of HFS, the released neurotransmitter, V, and the postsynaptic signal, C, will return to the equilibrium values but K will be K H s > KO. Thus, the net result of a HFS, is to increase the final value of K to KHB, that will be the new initial, nonequilibrium, condition for the following stimuli. The larger K produces, at each pulse, Vs larger than the ones before HFS, resulting in higher C peaks and initial slopes. This phase can be considered as the equivalent of the STP in the experiments (Colino et al. 19921, and its time scale is essentially dependent on the value of the current in the presynaptic cell [i.e., the 1 independent variable in reaction d ) ] . This prediction of the model might be experimentally tested using different presynaptic holding potentials. It should be noted that still another form of short-lasting potentiation has been
+
+
M. Migliore and G. F. Ayala
640
2
I
1
I
C 1 -
-0 2
K 1 -
I
I
I
0
50
100
150
200
0.0 0
50
100
150
200
0
v 0.5
HFS
Time (a.u.)
Figure 2: Time course of the C, V, and K signals from simulation withlow- and high- (HFS) frequency stimuli. Amplitudes and times are given in arbitrary units. Values for the rate constants are a = 2, p = 5, y = 1, 6 = 10, E = 0.5. Pulses of I, not shown, are of amplitude 1 (arbitrary units) and duration 50 time steps. Interstimulus interval is 4950 and 50 time steps for low and high frequency, respectively. The raising time of K after the HFS as well as after each I pulse is mainly determined by the l / ( y + E ) time constant (see equation 2.3). The duration of the STP phase will also depend from l / ( y + E ) , but it is mainly determined by 6, as consequenceof reaction (d) and by the background current I (in this case I = 0.01).
A Kinetic Model of Short- and Long-Term Potentiation
HFS
time (a.u.)
641
HFS -
time (a.u.)
Figure 3: Left: Time course of C, V, and K from a simulation with reaction (d) changed to K 5 V and KO = 0. Right: Time course of C (solid)and V (dashed) from a simulation with y = 0 and & = 0. Other rate constants, parameters, and protocol of stimulation as in Figure 2. described, the posttetanic potentiation (PTP),and it is usually assumed that it involves an entirely presynaptic mechanism. Our model does not take into account this kind of potentiation because we are assuming that the mechanisms involved in STP and LTP are different from those of I ", namely the role of the retrograde signal. The role of the interaction between I and K in inducing SlT and LTP is also evident from the model. The schematic picture in Figure 1 shows, in its presynaptic portion, two separate pathways for I to produce the neurotransmitter V the first, directly, with a rate constant a, and, the second, interacting with the retrograde signal K, with a rate constant 6. Let us for a moment suppose that the interaction of I with the retrograde signal, K, is impaired, but the first pathway is left intact. In the simulations this is accomplished by suppressing I in reaction ( d ) , changing it to K6'V In such conditions, Figure 3 (left) shows that, using the same protocol of stimulation as before, the synaptic transmission is not impaired, and yet, there is no STP.This example illustrates the role of I in the interaction. The role of K is seen from Figure 3 (right), obtained for y = 0 that is K = 0. The requirement for the simultaneous presence of both I and K is of course consistent with Hebbs postulate (Hebb 1949). This is because the postsynaptic depolarization produces the retrograde signal,
642
M. Migliore and G . F. Ayala
K, capable of enhancing the production of V in response to presynaptic signal. The simulation reported in Figure 4a, where all reaction rate constants are kept fixed, is conveniently compared with the experimental data of Figure 4b. The comparison shows that the peak amplitudes of C (see also Figure 21, in response to stimuli that are qualitatively similar to the standard experimental protocol used to obtain LTP, are not similar to the typical experimental results on LTP but to those on STP. We tested several different simple models, but all of them consistently failed to show LTP. In fact, the system is always stable and it admits only one stable solution. Since the basic idea is to use the retrograde signal, to obtain LTP we can (1) consider alternative and more complex kinetic pathways (e.g., autocatalytic processes) using the retrograde signal or (2) assume that H E triggers the change of some rate constant. Both ways can make explicit predictions and can be used as useful tools to help the interpretation of experiments. Since our major objective in this paper is to keep the model as simple as possible to be mathematically tractable and easily modifiable to include more detailed specific processes, we follow (2). From equations 1 and 2 it follows that the peak value and the initial slope of the output signal C, after HFS, are determined by the equilibrium value of the retrograde signal, K,, and from the rate at which the neurotransmitter produces an output signal, 0.Thus an increase of or any change in some other rate constant that increases K, will give LTP as shown, for example, in Figure 4 c 4 , as obtained by mere alterations of the y and E rate constants [an increase and a decrease, respectively, to keep TD = l / ( y + E ) constant]. These changes appear to be physiologically reasonable and expected if one consider that they correspond to a "more efficient" production of retrograde signal, y, and a change in the processes involved in the postsynaptic signal degradation, E . More important, however, these changes are consistent only with the experimental fact (Gustafsson and Wigstrom 1990) that the time evolution of potentiated EPSPs does not change with LTP, after normalization. It should also be stressed that these changes are to be considered as the net results of changes to kinetic parameters of specific reactions underlying our simple model. The agreement with actual LTP experiments is clearly evident from Figure 4c and d. The transitory increase of C (Fig. 2) during H E has already been simulated and explained in terms of several reactions that involve calcium pumping, diffusion, and buffering in the spine head and neck (Gamble and Koch 1987; Holmes and Levy 1990) as well as the electrotonic characteristics of the membrane. The simple hypothesis at the basis of our model is that the increase of the postsynaptic signal C produces a large retrograde signal that triggers the induction of LTP. Its maintenance is obtained by a change in the rate constant involved in the production of the retrograde signal, and we propose that this change re-
A Kinetic Model of Short- and Long-Term Potentiation
643
0.6
b) 0.4 -
.. .. ..... ......... .:. ...... 0.2 :,:;:': *:., ..$ ;*
.:*,;:t
'
..
0.8
4
*
0.6 ;**:
. . :.. .-.
.:.a
; a : *
0.4 .
....."..'-: "*:. .... ..... .. I
0.2 ;
I time (a.u.)
time (min)
Figure 4: Peak amplitudes of C ( a ) compared with typical experimental findings on STP (b). The arrow on the time axis indicate the time of application of HFS. Parameters and protocol of stimulation as in Figure 1. Peak amplitudes of C (c), from a simulation where at the end of HFS (arrow) the y rate constant is increased to 1.2 and E decreased to 0.3, compared with typical experimental findings on LTP (d). Other rate constants and protocol of stimulation as in Figure 1. (Experimental data taken and redrawn from Colin0 et al. 1992.)
M. Migliore and G. F. Ayala
644
quires a short-term increase in the concentration of the retrograde signal itself.
3.2 Comparison with Experimental Data. The model can account for several experimental observations on the mechanisms involved in LTP, as shown by work currently in progress at our laboratory. For example, the transient depression followed by short-term potentiation after N-methylBaspartate (NMDA)application (Kauer et al. 1988) can be reproduced by assuming an initial reduction of & and the transitory increase of y and B. The presence of extracellular hemoglobin or inhibitors of NO production such as nitro-L-arginine methyl ester (L-NAME)has been shown (Haley et al. 1992) to prevent the maintenance of LTP. In terms of our model this observation suggests that the triggering cause required to produce LTP is not, or not only, the high postsynaptic calcium concentration, but the increase of the retrograde signal. Any pharmacological manipulation that interferes with this transient increase during HFS may prevent LTP. However, at this point the model cannot account for those experiments where the application of L-NAME after LTP has been established fails to inhibit the maintenance of the potentiation. The model easily accounts also for the finding that LTP cannot be induced by an HFS without a conjunctive depolarization of the postsynaptic cell, for example, when the postsynaptic cell is voltage clamped (Kelso et al. 1986). In fact, a voltage clamp corresponds, in our model, to force C to a low value during HFS.This has the effect to prevent the increase in the level of the retrograde signal K during HFS, and thus inhibiting the changes of the rate constants, that we assume to be consequent to the high K. It has been shown that calcium chelators ethylene-bis(oxyethy1enenitrite) tetra-acetic acid (EGTA) (Lynch et al. 1983) and Nitr-5 (Malenka et al. 1988) prevent that the Ca2+ influx through NMDA channels contributes to the EPSP potentiation. Since it is known that ionic flux through NMDA channels mediates the late component of an EPSP, we can simulate the effect of the chelators with an increase of the rate constant that represents all the processes involved with the EPSP decay, that is E. The model indicates that because of increase of E, the depolarization during a HFS is not sufficiently large to increase the retrograde signal to a level necessary to obtain LTP. Finally, two different types of potentiation have been reported in r e cent experiments (Gustafsson and Wigstrom 1990). They differ in the changes in the time course of the recorded EPSPs. In one case, only the peak value increases, without changes in their time evolution characteristics. In the other case, the potentiation is expressed as a prolongation of the onset time and a larger peak amplitude. In terms of equation 2.2, both findings can be explained as changes of y and E , such to maintain TD = l / ( y E ) constant or not, respectively.
+
A Kinetic Model of Short- and Long-Term Potentiation
645
4 Conclusions
Each of the (a)-(e) reactions should be considered (as already remarked) as representative of an equivalent set of more complex reactions. The purpose of the present work is to assess a simplified environment, qualitatively consistent with the available experimental data, on which one can expand one or more reactions in order to test a specific mechanism as suggested by experimental evidence. From this point of view, our model does not takes explicitly into account, for example, the kinetics of ionic fluxes through NMDA or other channels, or calcium-dependent protein kinases (Kitajima and Hara 1990). The detailed modeling of these or other processes is, of course, needed to define more precisely the still unknown biophysical mechanisms involved with LTP and to obtain quantitative agreement with experimental data. On the backbone of our set of reactions, any additional process can further modulate the induction or the expression of LTl? We believe that our model can be useful to stimulate discussion and experimentation in the field. The present model uses the hypothesis that part of the origin of LTP may be in the (presynaptic) increased release of neurotransmitter provoked by a retrograde signal produced by a postsynaptic mechanism (depolarization and/ or high calcium concentration). The simulations, using the model presented, further support this hypothesis because the results obtained are (1) in very good qualitative agreement with experimental data on STP and LTP, (2) consistent with Hebb’s postulate, (3) consistent with the expected time course for EPSPs in terms of molecular rate constants rather than empirically, and (4) consistent with and capable of reproducing the effects of several LTP modulators. In this case the model can also predict the possible kinetics of the retrograde signal itself (Fig. 2). Moreover, the model has shown (Fig. 4) that although the presence of a retrograde signal is enough to induce a potentiation very similar to STP, it is not sufficient to induce the maintenance of LTP, and other postsynaptic events at a molecular level (such as the change of the rate constant of its production) are also necessary.
Acknowledgments We thank Prof. M. U. Palma for a critical reading of the manuscript and making valuable suggestions, Prof. S. L. Fornili and Prof. A. Messina for useful discussions, and Mr. S. Pappalardo for technical assistance. This work has been carried out at IAIF-CNR and supported also by CRRN-SM local funds.
646
M. Migliore and G. F. Ayala
References
Bliss, T. V. l?, and Gardner-Medwin, A. R. 1973. Long-lasting potentiation of synaptic transmission in the dentate area of the unanaesthetized rabbit following stimulation of the perforant path. J. Physiol. 232,357-374. Bliss, T. V. P., and Lomo T. 1973. Long-lasting potentiation of synaptic transmission in the dentate area of the anaesthetized rabbit following stimulation of the perforant path. J. Physiol. 232, 331-356. Bliss, T. V. P.,and Lynch, M. A. 1988. Long-term potentiation of synaptic transmission in the hippocampus: Properties and mechanisms. In Long-Term Potentiation: From Biophysics to Behavior, l? w. Landfield and S. A. Deadwyler, eds., pp. 3-72. Alan R. Liss, New York. Bredt, D. S., and Snyder, S. H. 1992. Nitric Oxide, A novel neuronal messenger. Neuron 8, 3-11. Byrne, J. H., and Berry, W. O., eds. 1988. Neural Models ofPlasticity. Experimental and Theoretical Approaches. Academic Press, New York. Colino, A., Huang, Y.-You, and Malenka, R. C. 1992. Characterization of the integration time for the stabilization of long-term potentiation in area CAI of the hippocampus. 1. Neurosci. 12, 180-187. Edwards, E. 1991. LTP is a long term problem. Nature (London) 350, 271. Gamble, E., and Koch, C. 1987. The dynamics of free calcium in dendritic spines in response to repetitive synaptic input. Science 236, 1311-1315. Gustafsson, B., and Wigstrom, H. 1990. Basic features of long-term potentiation in the hippocampus. Semi. Neurosci. 2, 321-333. Haley, J. E., Wilcox, G. L., and Chapman, P. F. 1992. The role of nitric oxide in hippocampal long-term potentiation. Neuron 8, 211-216. Hebb, D. 0. 1949. The Organization of Behavior. Wiley, New York. Holmes, W. R., and Levy, W. B. 1990. Insights into associative long-term potentiation from computational models of NMDA receptor-mediated calcium influx and intracellular calcium concentration changes. J. Neurophysiol. 63, 1148-11 68. Kauer, J. A., Malenka, R. C., and Nicoll, R. A. 1988. NMDA application potentiates synaptic transmission in the hippocampus. Nature (London) 334, 250-252. Kelso, S. R., Ganong, A. H., and Brown, T. H. 1986. H&bian synapses in hippocampus. Proc. Natl. Acad. Sci. U.S.A. 83, 5326-5330. Kitajima, T., and Hara, K. 1990. A model of the mechanisms of long-term potentiation in the hippocampus. Biol. Cybern. 64, 33-39. Koch, C., and Segev, I., eds. 1989. Methods in Neuronal Modeling: From Synapses to Networks. MIT Press, Cambridge, MA. Lynch, G., Larson, J., Kelso, S., Barrionuevo, G., and Schottler, F. 1983. Intracellular injections of EGTA block induction of hippocampal long-term potentiation. Nature (London) 305, 719-721. Malenka, R. C., Kauer, J. A,, Zucker, R. J., and Nicoll, R. A. 1988. Postsynaptic calcium is sufficient for potentiation of hippocampal synaptic transmission. Science 242, 81-84.
A Kinetic Model of Short- and Long-Term Potentiation
647
Nicolis, G., and Prigogine, I. 1977. Self-Organization in Nonequilibrium Systems. From Dissipative Structures to Order through Fluctuations. Wiley, New York. Press, W. H., Flannery, 8. P., Teukolsky, S. A., and Vetterling, W. T. 1987. Numerical Recipes. The Art of Scientific Computing. Cambridge Univ. Press, Cambridge. Segev, I., and Rall, W. 1988. Computational study of an excitable dendritic spine. I. Neurophysiol. 60, 499. Received 5 June 1992; accepted 24 November 1992.
This article has been cited by: 2. Alain Destexhe, Zachary F. Mainen, Terrence J. Sejnowski. 1994. Synthesis of models for excitable membranes, synaptic transmission and neuromodulation using a common kinetic formalism. Journal of Computational Neuroscience 1:3, 195-230. [CrossRef]
Communicated by Anthony Zador and Christof Koch
Artificial Dendritic Trees John G . Elias Department of Electrical Engineering, University of Delaware, N m r k , DE 19716 U S A
The electronic architecture and dynamic signal processing capabilities of an artificial dendritic tree that can be used to process and classify dynamic signals is described. The electrical circuit architecture is modeled after neurons that have spatially extensive dendritic trees. The artificial dendritic tree is a hybrid VLSI circuit and is sensitive to both temporal and spatial signal characteristics. It does not use the conventional neural network concept of weights, and as such it does not use multipliers, adders, look-up-tables, microprocessors, or other complex computational units to process signals. The weights of conventional neural networks, which take the form of numerical, resistive, voltage, or current values, but do not have any spatial or temporal content, are replaced with connections whose spatial location have both a temporal and scaling significance. 1 Introduction
Interest in using artificial neural networks for the identification and control of dynamic systems is growing (e.g., Narendra and Parthasarathy 1990). However, most neural network models do not include spatiotemporal dynamic signal processing capabilities. In these models, the neuron is treated as a point entity that receives and processes inputs at the soma, which makes spatial signal processing difficult or impossible. In our modeling approach, we have looked beyond the soma to the extensive dendritic tree structure of neurons, which not only forms most of the cell's surface area but provides spatiotemporal signal processing capabilities not present in models that assume a point-entity neuron. The artificial dendritic tree described in this paper is a hybrid circuit and is sensitive to both temporal and spatial signal characteristics, but it does not use the conventional neural network concept of weights, and as such it does not require multipliers, adders, look-up-tables, or other complex computational units to process signals. The weights of conventional neural networks, which take the form of numerical, resistive, voltage, or current values, but do not have any spatial or temporal content, are replaced in our system with connections whose spatial location have both a temporal and scaling significance. Neural Computation 5, 648-664 (1993) @ 1993 Massachusetts Institute of Technology
Artificial Dendritic Trees
649
We have only recently begun to experiment with networks of artificial dendritic trees (Elias 1992a). We have fabricated and tested artificial dendritic branches in CMOS analog VLSI (Elias et al. 1992), and we have used a genetic algorithm to train simple networks to follow a maneuvering target that moves in one dimension (Elias 1992b). The research described here attempts to capture neurocomputational principles by applying structure and behavior modeled after synaptic and dendritic levels of biological implementation. We hope to demonstrate that electronic analogs of biological computational devices, that include the properties of spatially extensive dendritic trees and the impulse response of chemical synapses, can form the basis for powerful artificial neurosystems. 2 Artificial Dendrite and Chemical Synapse
In this section, we describe electronic circuits that (1) emulate the electrical behavior of passive dendritic trees and chemical synapses and (2) are simple and robust enough to ensure that networks, which ultimately need to support huge numbers of synapses, can be constructed with standard VLSI processing. Electronic analogs of active dendrite behavior (e.g., Llinas and Sugimori 1980; Shepherd et al. 1985, 1989; Hounsgaard and Midtgaard 1988) will not be treated in this paper. 2.1 Artificial Dendrite. Passive artificial dendrites are formed by a series of standard compartments, where each compartment has a capacitor, Cm,that represents the membrane capacitance, a resistor, Rm, that represents the membrane resistance, and an axial resistor, &, that represents the cytoplasmic resistance (e.g., Rall 1989). Figure la shows a section of artificial dendrite with five standard compartments that is part of a much longer branch like that shown in Figure lb. The transient response of the artificial dendrite is of primary importance. Figure l c shows the impulse response measured at point S due to inward impulse current at four different locations, A, B, C, and D on a passive artificial dendrite as represented in Figure lb. The location S represents the position of the soma. Therefore, the voltages measured at S are those that would affect somatic voltage-sensitive circuits and perhaps cause the generation of an efferent impulse. As with biological passive dendrites, the peak voltage amplitude is largest for transmembrane current nearest the soma and gets rapidly smaller for sites farther away. The time for the voltage to peak shows a similar behavior: time-to-peakvoltage increases with distance from S (e.g., Rall 1989). The behavior shown in Figure 1 illustrates how the concept of weight is an inherent property of the dendritic physical structure. It is clear that position along the artificial dendrite can be used to produce an effective weighting, in both time and amplitude, of afferent signals that are in the form of a transient inward or outward current.
650
John G. Elias
Figure 1: (a) Compartmental model of passive dendrite. Each RC section, Rm, &, and C,, is a standard compartment that simplifies VLSI layout. (b) Standard compartments are abutted on substrate to form silicon dendritic branches. (c) Measured impulse response of single artificial dendritic branch due to transient transmembrane current at indicated locations on branch. 2.2 Artificial Chemical Synapse. The means for enabling inward or outward impulsive current at a specific artificial dendrite location is accomplished by using a single MOS field effect transistor. A p-channel transistor enables inward current, which produces an excitatory type response, and an n-channel transistor enables outward current, which results in an inhibitory type response. The complete artificial dendrite circuit is shown in Figure 2a, where both p-channel (upper) and n-channel (lower) transistors are placed at uniform positions along the branch. The transistors are turned on by an impulse signal applied to their gate terminals. Both transistor types operate in the triode region (e.g., Allen and Holberg 1987). Therefore, the amount of transmembrane current depends on the conductance of the transistor in the on state, the duration of the gate terminal impulse signal, and the potential difference across the transistor, which is dependent on the state of the dendrite at the point of the synapse. All excitatory transistors have identical drawn dimensions (as do inhibitory transistors), and both excitatory and inhibitory artificial synapses are placed at the same locations in the current chip implementation. It is possible that after further system level experimentation, we may find that a different synapse distribution is preferred.
2.3 Electrical Response of Artificial Dendritic Trees and Synapses. In Figure 1, we illustrated the behavior of the impulse response amplitude as a function of the synapse position on the artificial dendritic tree, thus demonstrating the effective weighing of inputs that are mapped onto the tree structure. The impulse response amplitude as a function of the afferent impulse signal width is shown in Figure 3a, which represents measured responses from one of our VLSI circuits for four different
Artificial Dendritic Trees
651
S v
Figure 2: (a)A five compartmentsection of artificialdendrite with five excitatory and five inhibitory artificial synapses. V,,,, is the resting voltage, Vtop is the maximum membrane voltage. (b)A multibranched artificial dendritic tree that is constructed by piecing together artificial dendrite sections like that in (a). impulse widths. A similar postsynaptic behavior is found in biological preparations under presynaptic voltage-clamp: presynaptic depolarization produces a nearly linear increase in postsynaptic voltage (e.g., Angstadt and Calabrese 1991). This behavior may be due to a lengthening of the time over which transmitter is released, thereby increasing transmembrane current in the postsynaptic terminal. In any event, the efficacy of existing connections in our system can be changed by altering the impulse width. We are investigating how this may be done on a local basis, perhaps consistent with Hebbs postulate (Hebb 19491, such that both local synaptic strength and the location of the synapse on the branch combine to produce an effective synaptic weight for a given connection. The artificial dendrite's voltage response to closely spaced impulses is shown in Figure 3b. The response due to each synaptic event is added to the resultant branch point voltage from past events until the voltage reaches a maximum value. This behavior is the expected impulse response of an Nth-order system and is solely due to the effective postsynaptic membrane. The same behavior would be observed if the phasing of multiple, transiently conducting, artificial synapses was short compared to the effective membrane constant. An example that utilizes this behavior is discussed in Section 4. Multiple, simultaneously conducting synapses that are electrically close together produce a voltage at the soma that is less than the sum of their individual responses (e.g., Shepherd and Koch 1990). This sublinear effect is due to the shunting load seen by each synaptic site when electrically nearby synapses open their channels. In contrast, multiple, simultaneously conducting synapses that are electrically far apart produce a nearly linear resultant voltage at the soma. Both of these behaviors, as measured at the trunk of a two-branched artificial dendritic tree (e.g., point S in Fig. 2b), are shown in Figure 3c. The smaller voltage transient
John G. Elias
652
................ ......
op.
Figure 3: Experimental results from artificial dendrite-synapse circuit to afferent stimulation. (a) Graded response: amplitude of voltage peak at soma is linearly related to afferent impulse width over wide range. (b) Tetanus response: closely spaced impulses cause voltage response to saturate if impulse rate is faster than membrane decay time. (c) Nonlinear and nearly linear response: curve 1 is the resultant somatic voltage for simultaneous stimulation of two adjacent synapses on same branch (see Fig. 2b). Curve 2 is somatic voltage for simultaneous stimulation of two synapses on different branches. Positions of synapses for curves 1 and 2 were equidistant from soma.
(curve 1) was measured when two adjacent artificial synapses were simultaneously active on the distal end of one of the branches. The larger voltage transient (curve 2) shows the resultant voltage when two artificial synapses on separate branches were simultaneously active. In this case, the resultant is nearly twice that of the previous example. In both cases, the artificial synapses were equal distance from the point of measurement. This type of behavior clearly enriches the signal processing capabilities of systems comprised of spatially extensive dendritic trees (Koch and Poggio 1987). 3 Silicon Dendritic Trees
If artificial dendrites are to be used in real systems then they must be implemented via a process that can make huge numbers of them in a small area inexpensively. The only feasible path for doing this currently is with standard silicon processing methods (e.g., Mead 1989). In this section, we discuss briefly the implementation of a dendritic system in silicon. 3.1 Convergent, Divergent, and Recurrent Connections. Networks that are built with artificial dendrites and synapses process signals that have a spatiotemporal significance by mapping afferent signal pathways to specific locations on the dendritic trees. The connections between synapses and the outputs of sensors and neurons determine the overall system response for a. given dendritic dynamic behavior. The number of
Artificial Dendritic Trees
653
different connection patterns is quite large and is a factorial function of the number of synapses and sensor elements. If we limit, for the moment, the number of divergent connections to one, then the total number of different connection patterns is given by
N! ( N - M)! where N is the number of artificial synapses and M is the number of sensor and neuron outputs. Artificial systems may have thousands of afferents and many times more synapses, resulting in an extremely large number of possible connection patterns. In our system, we allow each sensor element or artificial neuron to make unrestricted divergent connections and each synapse to receive multiple convergent connections from both sensor elements and artificial neurons. This tends to make the number of possible connection patterns much larger than that indicated by equation 1.
3.2 Virtual Wires. In the implementation of an electronic system, the number of data pathways in or out of modules is limited by the available technology. Integrated circuit packages rarely exceed 500 pins; our current artificial dendrite chip is in a 40 pin package. This limitation in pin count is of special concern with dynamic artificial neuronal systems because of the analog nature of the computation. Each sensor or neuron output must be able to connect to any one of the artificial synapses in the system, and the spiking outputs from sensors and neurons must arrive at their artificial synapse destinations in a parallel fashion. In order to overcome I/O limitations and to meet connectivity and timing requirements, we make use of a multiplexing scheme that we refer to as virtual wires. In this scheme, the outputs of active neurons and sensors (i.e., those that are currently producing a spike or impulse) cause the synapses that they connect with to become activated after a delay that is programmable for each efferent connection. The process of reading an active output causes that output to return to the inactive (i.e., nonspiking) state. After all sensor and neuron output states have been sampled, the activated synapses throughout the system are turned on transiently by a global impulse stimulus signal. The process of determining active sensors and artificial neurons, delayed activation of synapses that connect to active sensors and neurons, and transiently turning on active artificial synapses continues indefinitely with a period that depends on system dynamics. Virtual wires are formed using four digital circuits: Stimulus Memory, which is closely associated with each synapse, Address Decoding, which serves all on-chip synapses, State Machine, which determines sensor and neuron output states, and Connection List, which specifies the locations of synapses and the axonal delay associated with each connection. Stim-
654
John G. Elias
CUA.'
Figure 4: Circuit diagram of one of the excitatory Stimulus Memory registers shown with its p-channel excitatory synapse transistor. When SET* is asserted the synapse is activated (i.e., excsyn*[nl is set to logic 0) and the artificial synapse will turn on when STIMULATE*is asserted. CLEAR* inactivates all Stimulus Memory locations throughout the system. Both STIMULATE* and CLEAR* are global signals in the system.
ulus Memory and Address Decoding are on-chip circuits; the Connection List and State Machine are off-chip. The circuit diagram of an excitatory Stimulus Memory register connected to its p-channel artificial synapse transistor is shown in Figure 4. In the current implementation, virtual wires add nine transistors to each artificial synapse. A synapse is activated when its excsyn*[n] is set to logic 0 by asserting SET' while CLEAR' is unasserted. The SET' signal is asserted by the on-chip address decoder when the proper combination of external address lines and control signal is asserted. An activated synapse will turn on when the global impulse signal, STIMULATE*, is asserted. The global signal CLEAR* is asserted after every STIMULATE* assertion to inactivate synapses in preparation for the next round of sampling and activation. The Connection List is a multiple-bit-wide memory that holds thg synapse addresses and axonal delay of each efferent connection in its domain. For large systems, we plan to divide the network into domains that will permit a certain level of parallel sampling of neuron and sensor outputs, which should enhance system scalability. The Connection Lists across all domains hold the pattern of connectivity for the system and thus their contents determine system behavior. A connection pattern can be fixed in ROM,or as in our present system, loaded via computer for experimentation. Figure 5 illustrates a simplified single-domain system comprising Connection List, sensor, State Machine, and four neuromorphic chips, each of which contains a number of artificial neurons. The outputs of the artificial neurons on each chip are sampled via a multiplexer, which is
Artificial Dendritic Trees
655
Figure 5 Simplified block diagram for single domain system showing basic operation. All sensory and neuronal outputs simultaneously activate the artificial synapses that they connect to through the virtual wires.
selected by the on-chip address decoder. Each neuron is in one of two states, so only a single output pin is needed to read all of them. In operation, the State Machine reads the state of each sensor element and every neuron in its domain. A spiking neuron or sensor output is detected by the State Machine, which then activates all of the synapses that connect to that particular sensor or neuron. After reading all outputs, STIMULATE* is asserted transiently, thus briefly turning on activated synapses. This is followed by asserting CLEAR*, which inactivates all synapses. As with Mahowald’s method of connecting neuron outputs to synapses (Mahowald 1992) addresses of synapses and neurons are used rather than direct connections carrying spikes.
3.3 Standard Dendrite Compartment. Figure 6 illustrates the basic integrated circuit layout of our standard dendrite compartment. Each compartment has a capacitor, C,,, that represents the membrane capacitance, a resistor, R,, that represents the membrane resistance, and an axial resistor, R,, that represents the cytoplasmic resistance. For the results
656
John G. Elias
Figure 6 Basic VLSI layout for standard dendrite compartment. Control lines for resistors permit adjustment of resistance over a limited range. Vmt establishes the resting voltage (typically 1 V). reported here, the size of the artificial dendrite standard compartment was 18 by 180 pm with most of this area being taken up by the capacitor. The capacitor is the largest element in the standard dendrite compartment and is implemented using conventional silicon processing methods (e.g., Allen and Holberg 1987). The capacitor was fabricated with two layers of polysilicon separated by a thin oxide layer. The top plate of the capacitor is polysilicon layer 2 (poly2) and connects to a ground bus that runs perpendicular to the long axis of the capacitor. The bottom plate is polysilicon layer 1 (polyl), which connects directly to the resistors, R, and R,,, and to the synapse transistors in the stimulus memory (see Figs. 4 and 6). The capacitance for the current size standard compartment capacitor is approximately 1 pF, which is based on an oxide thickness of approximately 700 A. There are many techniques to reduce the footprint of the capacitor while keeping the same capacitance: thinner dielectric, use material with a higher dielectric constant (e.g., silicon nitride), employ three-dimensional capacitors (e.g., trench or tower capacitors), but we will not explore these further here. The compartmental resistors may be implemented by a number of standard silicon fabrication techniques: well, pinched, active, and SC (e.g., Allen and Sanchez-Sinencio 1984). The resistor footprint for a particular resistance depends not only on the resistance value but also on the implementation technique. Well resistors are made by n- or p- diffusion and have a footprint advantage over the other techniques because the well resistor can be put under the capacitor. Therefore, a well resistor does not take up any silicon real estate but it has the disadvantage of relatively small resistance (measured as 5 kR per square for our chips). Pinched resistors have a higher resistance but can not be placed under the capacitor. Active and SC resistors are made from MOS transistor circuits that are designed to emulate resistor behavior over a certain range of terminal voltages and resistance values.
Artificial Dendritic Trees
657
We have implemented n-well, active, and SC resistors on different chips and will report on the details of their design and relative behavior later (Elias and Meshreki 1993). In our standard dendrite compartment, n-well resistors go under the capacitor and active or SC resistors are placed at the ends of the capacitor as shown in Figure 6. Independent control signals for changing the resistance of the SC or active R, and & pass along both sides of the compartment. For chips with active resistors, the control signals are DC voltages that permit a certain range of adjustment. With SC resistors, the control signals are AC voltages in which the frequency determines the resistance. Presently, the R,n resistors in all of the compartments share the same control signals and the R, resistors share a different set of control signals. Therefore, all compartments have nominally the same R, and & resistances. The standard dendrite compartment was designed to abut with adjacent compartments and was pitch-matched to the on-chip virtual wire circuitry. This method makes the construction of artificial dendritic trees a relatively simple task: to make a branch, standard compartments are placed side by side until the desired branch length is reached. Branches are then connected via metal or poly wires to form trees. The spacing between compartments is the minimum distance between capacitors (2 pm). The compartments are aligned such that the inputs of one compartment connect to the outputs of the previous compartment.
3.4 A Simple Silicon Implementation. The artificial dendritic tree circuit and its on-chip virtual wiring was fabricated using a 2-pm CMOS double-poly n-well process on a 2 by 2 mm Mosis Tiny Chip (e.g., Mead 1989). The artificial somata and output multiplexer were left off the chips to permit experimentation with different soma circuits. Four artificial dendritic branches each having 15 excitatory and 15 inhibitory artificial synapses were implemented on the chip. The number of synapses was kept low in order to leave open silicon areas on the chip for other analog test circuits. Figure 7 shows the complete artificial dendrite chip layout. In the current implementation, the four branches are in-line, with a gap in between each branch, and centered on the die. The ends of each branch are taken out of the chip through package pins to allow experimentation with different tree structures. Multiple chips can be combined as well to produce tree structures with more branches, longer branches, or higher order branching. The remaining circuitry makes up the virtual wires. We are currently working on several new chip designs that we expect will reduce the silicon area needed for the on-chip virtual wires. We are also investigating the addition of shunting inhibition as well as local synaptic weight storage that could be used to modify the effective weights of existing connections.
658
John G. Elias
Figure 7 Chip layout of artificialdendrites fabricated using a MOSIS 2-pm double polysilicon standard CMOS process. The four artificial dendritic branches can be seen in the center of the die. The ends of each branch are connected to pads, which allows experimentation with different branching structures. Each branch has 30 synapses (15 excitatory and 15 inhibitory),which are uniformly spaced along the branch.
4 Simple Test Circuits
Three simple, artificial dendritic tree experiments, two of which do not make use of temporal aspects of dendritic trees, are described next. In each experiment, the output shown represents the measured branch node voltage from one of our circuits for a period of time after the tree received impulsive afferent sensory signals. The sensory signals for each experiment were generated by a computer and were applied to the artificial synapses through a parallel interface. The sensor elements were set to a logic one if the sensor field was above a fixed threshold voltage and a logic zero otherwise. In each experiment, a sequence of sensor data over time was presented to the tree and the resultant waveform was captured with an &bit digitizer. Although binary level, one-dimensional sensor data were used, each test circuit would produce similar results with multilevel, two-dimensional sensor data, albeit with different connection patterns and branching structure. Sensor elements that are a logic one at time, t, cause an impulse signal to be applied to the gate terminals of the artificial synapses that they connect to through the virtual wires. Therefore, active sensor elements (i.e., those that hold a logic 1 at time t ) cause their respective artificial synapses to turn on transiently. Inactive sensor elements (i.e., those that hold a logic zero at time t ) do not cause any artificial synapses to turn on. Figure 8 .illustrates how a two-branched artificial dendritic tree can detect asymmetric patterns in the sensor field. In this experiment, the
Artificial Dendritic Trees
659
Figure 8: Input sensor with six elements connected to dual-branch artificial dendritic tree. The connection pattern shown classifiesinput patterns into symmetric and asymmetric classes. The sensor, with its data field, is shown at four different times, the last having a symmetric data pattern. Symmetric data fields result in a null output. Asymmetric sensor data fields produce either a positive or negative voltage trajectory. The output for the four sample times is shown at the right, which represents measured data from one of our circuits. Excitatory artificial synapses are top connections on each branch. Inhibitory artificial synapses are bottom connections.
sensor had 32 elements, but only six elements are shown here for simplicity. The same results would be obtained with virtually any size linear sensor array. The center of the sensor array represents the plane of symmetry, and the connections for a particular branch go only to sensor elements on one side of this plane. In this example, the top branch has one inhibitory and two excitatory connections. The bottom branch has one excitatory and two inhibitory connections. This connection pattern is not the only one that produces acceptable output. In general, there may exist a large number of connection patterns that produce acceptably good results. The sensor contents are shown at four different times, one of which contains a symmetric pattern. Not shown in the figure are the two trivial cases in which all elements are either logic one or logic zero. Both of these are symmetric and produce no output transient voltage. The case in which sensor elements are all ones activates all artificial synapses that are connected to these sensor elements, and the resultant signals sum approximately to zero at the branch point. Each connection from one side of the sensor plane of symmetry is mirrored with an opposite polarity connection on the other side that is equidistant from the branch point. When there is an asymmetric sensory pattern, as for sample numbers 0-2, there exists an imbalance between the activated artificial synapses, which results in a transient voltage at the branch point.
660
John G. Elias
Figure 9: Experimentsshowing the target-direction capabilities of dendrites (after Rall 1964). (a) Target (shaded sensor element)moves from right to left across sensor array. Resultant waveform is lower in peak voltage than in (b) where target moves from left to right. A simple thresholding device classifies target direction.
Figure 9 shows an experiment that follows an analysis of temporal processing in dendrites by Rall (1964). In this example, adjacent excitatory artificial synapses on one branch are connected to adjacent sensor elements. In this experiment, the sensor had 15 elements, but only eight elements are shown for simplicity. This simple connection pattern produces an output that is sensitive to the direction and speed of a moving target. Figure 9a shows the resultant branch point voltage for an eight segment time series in which a target moves across the sensor array from right to left. The target, in this case, is a logic one. The branch point voltage transients occur as the target moves across the sensor field, as can be seen in the plot of voltage vs. time. In Figure 9a, the artificial synapse nearest to the branch point is stimulated first and later followed by the stimulation of the next more distal synapse. The effect is that the resultant voltage transients arrive at the branch point well separated in time and therefore do not overlap significantly. In Figure 9b, the target moves from left to right. In this case, the most distal artificial synapse is stimulated first and followed later by stimulation of the more proximal synapses. The resultant voltage at the branch point is larger than it was for the case when the target moved in the opposite direction because the arrival times of the individual transients are more closely aligned. The classification of target direction is then completed by a simple comparator. Figure 10 shows a simplified circuit diagram that uses a single branch of artificial dendrite to provide a control signal for a maneuvering-target tracking application (Elias 1992~).The sensor, in this example, is a threshold device with seventeen elements, each of which outputs a one or a
Artificial Dendritic Trees
661
Figure 10: Input sensor with 17 elements connected to a branch of an artificial dendritic tree that responds to the position of a target in the sensor array. The connection pattern shown produces a dynamic response that depends nonlinearly on how far off the target is from the center. When the target is centered, the output (at S) is a null. As the target moves off center, the resultant voltage increases rapidly with separation distance between target and center. The sensor, with its data field, is shown at nine different times. Each sample time shows the target (in this case a 0) going off center and the resultant output (offset for clarity) from the artificial dendritic branch. The top half of the sensor array connects to only excitatory artificial synapses. The bottom half connects only to inhibitory synapses. The resultant voltage transients at S were captured using an 8-bit digitizer.
zero. In Figure 10, the one-dimensional sensor pattern over time is that of a simple maneuvering target, which, in this example, is a logic zero on a background of logic ones. In general, the target could be of any shape as long as it was distinguishable from the background. When the target is on center, the output of the dendritic branch is approximately zero. This is because all excitatory and inhibitory artificial synapses are simultaneously conducting, thereby cancelling each other. Small variations in target position around the center produce relatively small output voltage transitions which can be used for low gain system control. If the target moves below center, as shown in Figure 10, the resultant voltage transients are positive. If, however, the target moves above center, the transients are negative. As the target moves farther off center, either above or below, the resultant branch output peak voltage rapidly increases. This occurs because more proximal artificial synapses turn on, which, in effect, shifts the system control to higher gain. The relative amplitudes of the branch output voltage transients as a function
662
John G. Elias
of the distance between target and sensor center can be arbitrarily set by moving connections of particular sensor elements to either more distal or more proximal artificial synapses (Elias 1992b).
5 Summary and Discussion
In our research program, we have adopted Mead’s methodology (Mead 1989) for implementing neuromorphic systems: (1) study the relevant biological implementation; (2) extract the important computational principles; (3) make optimum use of the inherent properties of electronic devices; and (4) implement, using standard silicon processing techniques. In the work reported here, we have studied the properties of both active and passive biological dendritic trees as well as the dynamic and static behavior of chemical synapses. We have extracted principles of computation exhibited by passive dendrites with chemical synapses and have translated these principles to a simple and scalable electronic form implemented in standard CMOS technology. Although our electronic models of chemical synapse and passive dendritic tree are, in many respects, extreme simplifications of biological structures, their dynamic electrical behavior appears to satisfactorily follow that of their biological paragons. The artificial dendritic tree structure is based on a current understanding of passive dendritic trees, which results in an extremely simple circuit implementation that is highly scalable. Artificial neurons with extensive dendritic trees have the capability to process signals that have both temporal and spatial significance. In our networks, weights are replaced with connections that, when combined with the sublinear behavior of electrically close synapses and the nearly linear behavior of widely separated synapses, provide a rich computational substrate for signal processing.
Acknowledgments The author wishes to thank Peter Warter for several useful suggestions on chip architecture, Hsu Hua Chu, Samer Meshreki, and Sheela Sastry for assisting with chip layout, design, and testing, and the reviewers for many useful comments.
References Allen, I? E., and Holberg, D. R. 1987. CMOSAnalogCircuit Design. Holt, Rinehart & Winston, New York.
Artificial Dendritic Trees
663
Allen, P. E., and Sanchez-Sinencio, E. 1984. Switched Capacitor Circuits. Van Nostrand Reinhold, New York. Angstadt, J. D., and Calabrese, R. L. 1991. Calcium currents and graded synaptic transmission between heart intemeurons of the leech. J. Neurosci. 11(3),746759. Elias, J. G. 1992a. Spatiotemporal properties of artificial dendritic trees. Proc. Int. Joint Conf. Neural Networks, Baltimore 2, 19-26. Elias, J. G. 1992b. Genetic generation of connection patterns for a dynamic artificial neural network. IEEE Computer Society Press ProceedingsofCOGANN92, a workshop on combinations of genetic algorithms and neural networks. Elias, J. G. 1992c. Target tracking using impulsive analog circuits. In Applications of Artificial Neural Networks Ill, S. K. Rogers, ed. Proc. SPIE 1709, 338-350. Elias, J. G., Chu, H. H., and Meshreki, S. 1992. Silicon implementation of an artificial dendritic tree. Proc. Int. Joint Conf. Neural Networks, Baltimore 1, 154159. Elias, J. G., and Meshreki, S. 1993. Wide-range variable dynamics using switchedcapacitor neuromorphs. In preparation. Hebb, D. 0. 1949. The Organization of Behavior. Wiley, New York. Hounsgaard, J., and Midtgaard, J. 1988. Intrinsic determinants of firing pattern in Purkinje cells of the turtle cerebellum in vitro. J. Physiol. 402, 731-749. Koch, C., and Poggio, T. 1987. Biophysics of computation: Neurons, synapses and membranes. In Synaptic Function, G. M. Edelman, W. E. Gall, and W. M. Cowan, eds., Chap. 23. Wiley, New York. Koch, C., Poggio, T., and Torre, V. 1983. Nonlinear interactions in a dendritic tree: Localization, timing and role in information processing. Proc. Natl. Acad. Sci. U.S.A. 80, 2799-2802. Llinas, R., and Sugimori, M. 1980. Electrophysical properties of in vitro purkinje cell dendrites in mammalian cerebellar slices. J. Physiol. 305, 197-213. Mahowald, M. A. 1992. Evolving analog VLSI neurons. In Single Neuron Computation, T. McKenna, J. Davis, and S. F. Zornetzer, eds., Chap. 15. Academic Press, New York. Mahowald, M. A., and Douglas, R. 1991. A silicon neuron. Nature (London) 354, 515-518. Mead, C. 1989. Analog VLSI and Neural Systems. Addison-Wesley, Reading, MA. Narendra, K. S., and Parthasarathy, K. 1990. Identification and control of dynamical systems using neural networks. IEEE Transact. Neural Networks 1, 4-27. Rall, W. 1964. Theoretical significance of dendritic trees for neuronal inputoutput relations. In Neural Theory and Modeling, R. E Reiss, ed., p. 73. Stanford University Press, Stanford, CA. Rall, W. 1989. Cable theory for dendritic neurons. In Methods in Neuronal Modeling: From Synapses to Networks, Chap. 2. C. Koch and I. Segev, eds. MIT Press, Cambridge, MA. Shepherd, G. M., Brayton, R. K., Miller, J. F., Segev, I., Rinzel, J., and Rall, W. 1985. Signal enhancement in distal cortical dendrites by means of interactions between active dendritic spines. Proc. Natl. Acad. Sci. U.S.A. 82, 2192-2195.
664
JohnG. Elias
Shepherd, G. M., and Koch, C. 1990. Dendritic electrotonus and synaptic integration. In The Synaptic Organization of the Bruin, G . M. Shepherd, ed., appendix. Oxford University Press, New York. Shepherd, G. M., Woolf, T. B., and Carnevale, N. T. 1989. Comparisons between active properties of distal dendritic branches and spines: Implications for neuronal computations. 1. Cognit. Neurosci. 1, 273-286. Received 8 May 1992; accepted 11 November 1992.
This article has been cited by: 2. Christoph Rasche. 2007. Neuromorphic Excitable Maps for Visual Processing. IEEE Transactions on Neural Networks 18:2, 520-529. [CrossRef] 3. R. Jacob Vogelstein, Udayan Mallik, Joshua T. Vogelstein, Gert Cauwenberghs. 2007. Dynamically Reconfigurable Silicon Array of Spiking Neurons With Conductance-Based Synapses. IEEE Transactions on Neural Networks 18:1, 253-265. [CrossRef] 4. K.A. Boahen. 2004. A Burst-Mode Word-Serial Address-Event Link—II: Receiver Design. IEEE Transactions on Circuits and Systems I: Regular Papers 51:7, 1281-1291. [CrossRef] 5. K.A. Boahen. 2004. A Burst-Mode Word-Serial Address-Event Link—I: Transmitter Design. IEEE Transactions on Circuits and Systems I: Regular Papers 51:7, 1269-1280. [CrossRef] 6. C.M. Higgins, S.A. Shams. 2002. A biologically inspired modular VLSI system for visual measurement of self-motion. IEEE Sensors Journal 2:6, 508-528. [CrossRef] 7. C. Rasche, R.J. Douglas. 2001. Forward- and backpropagation in a silicon dendrite. IEEE Transactions on Neural Networks 12:2, 386-393. [CrossRef] 8. B.D. Brown, H.C. Card. 2001. Stochastic neural computation. I. Computational elements. IEEE Transactions on Computers 50:9, 891-905. [CrossRef] 9. K.A. Boahen. 2000. Point-to-point connectivity between neuromorphic chips using address events. IEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing 47:5, 416-434. [CrossRef] 10. Timothy K. Horiuchi , Christof Koch . 1999. Analog VLSI-Based Modeling of the Primate Oculomotor SystemAnalog VLSI-Based Modeling of the Primate Oculomotor System. Neural Computation 11:1, 243-265. [Abstract] [PDF] [PDF Plus] 11. John G. Elias , David P. M. Northmore , Wayne Westerman . 1997. An Analog Memory Circuit for Spiking Silicon NeuronsAn Analog Memory Circuit for Spiking Silicon Neurons. Neural Computation 9:2, 419-440. [Abstract] [PDF] [PDF Plus] 12. David P. M. Northmore, John G. Elias. 1996. Spike Train Processing by a Silicon Neuromorph: The Role of Sublinear Summation in DendritesSpike Train Processing by a Silicon Neuromorph: The Role of Sublinear Summation in Dendrites. Neural Computation 8:6, 1245-1265. [Abstract] [PDF] [PDF Plus]
Communicated by A. B. Bonds
Patterns of Local Connectivity in the Neocortex Andrew Nicoll Department of Physiology, School of Medical Sciences, University Walk, Bristol BS8 ZTD,UK
Colin Blakemore University Laboratory of Physiology, Parks Road, Oxford OX2 3PT, UK
Dual intracellular recording of nearby pairs of pyramidal cells in slices of rat visual cortex has shown that there are significant differences in functional connectivity between the superficial and deep layers (Mason et al. 1991; Nicoll and Blakemore 1993). For pairs of cells no farther than 300 pm apart, synaptic connections between layer 2/3 pyramidal neurons were individually weaker (median peak amplitude, A, of single-fiber excitatory postsynaptic potentials, EPSPs, = 0.4 mV) but more frequent (connection probability, p = 0.087) than those between layer 5 pyramidal neurons (mean A = 0.8 mV, p < 0.015). Taken in combination with plausible estimates of the density of pyramidal cells, the total numbers of synapses on them and the number of synapses formed on their intracortical axons, the present analysis of the above data suggests that roughly 70% of the excitatory synapses on any layer 2/3 pyramid, but fewer than 1% of those on a layer 5 pyramidal neuron, are derived from neighboring pyramidal neurons in its near vicinity. Even assuming very extreme values for some parameters, chosen to erode this difference, the calculated proportion of ”local synapses” for layer 5 pyramids was always markedly lower than for layer 2/3 pyramidal neurons. These results imply that local excitatory connections are much more likely to provide significant ”intracortical amplification” of af
