Mathematical Approaches to Neural Networks

(B.26)

whence the replica-symmetric ansatz and limit PA---t 00 yield the result (B.23). In the corresponding problems associated with learning a rule, to in (B.2) is replaced by the teacher output 9, while certain aspects of noise may be incorporated in the form of g(hP). Minimization of the corresponding training error may then be effectuated analagously to the above treatment of random patterns, but with 7 related to the { I j } ; j = 1,... by the teacher rule (76).

REFERENCES Aleksander I.; 1988, in “Neural Computing Architectures” ed. I. Aleksander (North Oxford Academic), 133 Amit D.J.; 1989, “Modelling Brain Function” (Cambridge University Press) Amit D.J., Gutfreund H. and Sompolinsky H.; 1985, Ann. Phys. 173, 30 de Almeida J.R.L. and Thouless D.J.; 1978, J. Phys. A l l , 983 Binder K. and Young A.P.; 1986, Rev. Mod. Phys. 58, 801 Coolen A.C.C. and Sherrington D.; 1992 “Dynamics of Attractor Neural Networks”, this volume Derrida B., Gardner E. and Zippelius A.; 1987, Europhys. Lett. 4, 167 Edwards S.F. and Anderson P.W.; 1975, J. Phys F5, 965 Fischer K.H. and Hertz J.A.; 1991 “Spin Glasses” (Cambridge University Press) Gardner E.; 1988, J. Phys A21, 257 Gardner E. and Derrida B.; 1988, J. Phys A21, 271 Glauber R.; 1963, J. Math. Phys. 4, 294 Gyorgy G. and Tishby N.; 1990, in “Neural Networks and Spin Glasses”, eds. W.K. Theumann and R. Koberle (World Scientific) Hertz J.A., Krogh A. and Palmer R.G.; 1991, “Introduction to the Theory of Neural Computation” ( Addison-Wesley) Hopfield J.J.; 1982, Proc. Natl. Acad. Sci. USA 79,2554 Kanter I. and Sompolinsky H.; 1987, Phys. Rev. Lett. 58,164 Kirkpatrick S., Gelatt C.D. and Vecchi M.P., 1983, Science 220, 671 Kirkpatrick S. and Sherrington D.; 1978, Phys. Rev. B17,4384 Krauth W. and MCzard M.; J. Physique (France) 5 0 , 3057 (1989)

291

Metropolis N., Rosenbluth A.W., Rosenbluth M.N., Teller A.H. and Teller E.; 1953, J. Chem. Phys. 21, 1087 Mizard M. and Parisi G.; 1986, J. Physique (Paris) 47, 1285 Mdaard M., Parisi G. and Virasoro M.A.; 1987, “Spin Glass Theory and Beyond” (World Scientific) Minsky M.L. and Papert, S.A.; 1969 “Perceptrons” (MIT Univ. Press) Muller B. and Reinhardt J.; 1990, “Neural Networks: an Introduction” (Springer-Verlag) Opper M., Kinael W., Kleina J. and Nehl R.; 1990, J. Phys A23, L581 Parisi G.; 1979, Phys. Rev. Lett. 43, 1754 Parisi G.; 1983, Phys. Rev. Lett. 5 0 , 1946 Seung H.S., Sompolinsky H. and Tishby N.; 1992, Phys. Rev. A45,6056 Sherrington D.; 1990, in “1989 Lectures on Complex Systems” ed. E. Jen (Addison-Wesley) p.415 Sherrington D.; 1992, in “Electronic Phase Transitions” ed. W. Hanke and Yu.V. Kopaev (North-Holland) p.79 Sherrington D. and Kirkpatrick S.; 1975, Phys. Rev. Lett 35, 1972 Thouless D.J., Anderson P.W. and Palmer R.; 1977, Phil. Mag. 35, 1972 Watkin T.L.H.; 1992 “Optimal Learning with a Neural Network”, to be published in Europhys. Lett. Watkin T.L.H., Rau A. and Biehl M.; 1992 “The Statistical Mechanics of Learning a Rule”, to be published in Revs. Mod. Phys. Watkin T.L.H., Sherrington D.; 1991, Europhys. Lett. 14,791 Wong K.Y.M. and Sherrington D.; 1987, J. Phys. A20, L793 Wong K.Y.M. and Sherrington D.; 1988, Europhys. Lett. 7 , 197 Wong K.Y.M. and Sherrington D.; 1989, J. Phys A22, 2233 Wong K.Y.M. and Sherrington D.; 1990a, J. Phys. A23, L175 Wong K.Y.M. and Sherrington D.; 1990b, J. Phys. A23, 4659 Wong K.Y.M. and Sherrington D.; 199Oc, Europhys. Lett. 10,419

This Page Intentionally Left Blank

Mathematical Approaches to Neural Networks J.G. Taylor (Editor) 8 1993 Elsevier Science Publishers B.V. All rights reserved.

293

Dynamics of Attractor Neural Networks

Ton Coolen and David Sherrington Department of Physics, University of Oxford, Theoretical Physics, 1 Keble Road, Oxford, OX1 3NP Abstract We illustrate the use of techniques from non-equilibrium statistical mechanics for studying dynamical processes in symmetric and non-symmetric attractor neural networks.

1. INTRODUCTION Although techniques from equilibrium statistical mechanics can provide much detailed quantitative information on the behaviour of large interconnected networks of neurons, they also have some serious restrictions. The first (obvious) one is that, by definition, they will only provide information on equilibrium properties. For associative memories, for instance, it is not clear how one can calculate quantities like sizes of domains of attraction without studying dynamics. The second (more serious) restriction is that for equilibrium statistical mechanics to apply the dynamics of the system under study must obey a property called detailed balance. For Ising spin neural networks in which the dynamics is a stochastic alignment to local fields (or post-synaptic potentials) which are linear ih the neural state variables, this requirement implies immediately symmetry of the interaction matrix. From a physiological point of view this is clearly unacceptable. The dynamics of symmetric systems can be understood in terms of the minimisation of some scalar quantitity (in equilibrium to be identified with the free energy). For nonsymmetric systems, although the microscopic probability distribution will again evolve in time to some equilibrium, it will no longer be possible to think in terms of some scalar quantity being minimised. Ergodicity breaking in the thermodynamic limit (i.e. on finite timescales) may now manifest itself in the form of limit-cycle attractors or even in chaotic trajectories. One must study the dynamics directly. The common strategy of all non-equilibrium statistical mechanical studies is rather simple: try to derive and solve the dynamical laws for a suitable smaller set of relevant macroscopic quantities from the dynamical laws of the underlying microscopic system. This can be done in two ways. The first route consists of calculating from the microscopic stochastic equations a differential equation for the macroscopic probability distribution, which subsequently is to be solved. The second route consists of solving the macroscopic stochastic equations directly; from this solution one then calculates the values of the macroscopic quantities. For such programmes to work the interaction matrix must either have a suitable structure of some sort, or contain (frozen) disorder, over which suitable averages can be performed (or a combination of both). A common feature of many sta-

294

tistical mechanical models for neural networks is separability of the interaction matrix, which naturally leads to a convenient description in terms of macroscopic order parameters.

2. THE MACROSCOPIC PROBABILITY DISTRIBUTION

In this section we will show how one can calculate from the microscopic stochastic evolution equations (at the level of individual neurons) differential equations for the probability distribution of suitably defined macroscopic state variables. We will investigate which are the conditions for the evolution of these macroscopic state variables to (a) become deterministic in the limit of infinitely large networks and, in addition, (b) be governed by a closed set of dynamic equations. For simplicity in illustrating the techniques we will restrict ourselves to systems of McCulloch-Pitts/Ising spin neurons a;E {-1, 1) (a;= 1 indicates that neuron i is firing with maximum frequency, u; = -1 indicates that it is at rest). The N-neuron network state will be denoted by the vector u E {-1, l}N;the probability to find the system at time t in state u by pt(u). The evolution in time of this microscopic state probability is governed by a stochastic process in the form of a master equation:

in which the Fj are ‘spin-flip’ operators: F j f ( u ) E f(a1,. . .,-uj,. . .,U N ) . The process (1) becomes a stochastic local field alignment if for the transition rates wj(u) of the transitions u -+ Fju we make the usual choice: 1 w ~ ( u ) - [l - t a n h ( p ~ j h j ( ~ ) ) ] 2

hj(U)

JjkUk

-

w;

(2)

k

where p = 1/T (the ‘temperature’ T being a measure of the amount of stochastic noise) and the quantities hj(u) are the local alignment fields (or post-synaptic potentials). There are clearly many alternative ways of defining a stochastic dynamics for such systems, most of which are Markov processes (i.e. with discrete time steps). The advantage of the above choice is simply that we are now dealing with differential equations (as opposed to discrete mappings) from the very beginning.

2.1 A Toy Model

Let us first illustrate the basic ideas with the help of a simple toy model: J.. =

-J7 . tr . 3

1 3 - N

w;= 0

(the variables q; and & are arbitrary, but may not depend on N). For 77; = [ i = 1 we recover the infinite range ferromagnet (J > 0) or anti-ferromagnet (J < 0); for q; = [; E {-1,l) (random) and J > 0 we recover the Liittinger (1976) or Mattis (1976) model

295

(equivalently: the Hopfield (1982) model with only one stored pattern). Note, however, that the interaction matrix is non-symmetric as soon as a pair ( i j ) exists, such that v;(j # qjt; (in general, therefore, equilibrium statistical mechanics does not apply). The local fields become h;(u) = Jv;m(u) with m ( u ) E & x k (kuk. Since they depend on the microscopic state u only through the value of m , the latter quantity appears to constitute a natural macroscopic level of description. The ensemble probability of finding the macroscopic state m ( u ) = m is given by Pt[ml

= C P t ( U ) 6 [m-m(u)l U

Its time derivative is obtained by inserting (1):

Inserting the expressions (2) for the transition rates and the local fields gives:

In the thermodynamic limit N + 00 only the first term survives. The solution of the resulting differential equation for Pt [m] is:

Pt[m]= / d m o %[m016 [m-m'(t)l

This solution describes deterministic evolution, the only uncertainty in the value of m is due to uncertainty in initial conditions. If at t = 0 the quantity m is known exactly, this will remain the case for finite timescales; m turns out to evolve in time according to (3).

2.2 Arbitrary S y n a p t i c Interactions

We will again define our macroscopic dynamics according to the master equation (l),but we will now allow for less trivial choices of the interaction matrix. We want to calculate the evolution in time of a given set of macroscopic state variables n(u)f (CiI(u), . ..,Cin(u))in the thermodynamic limit N + 00. At this stage there are no restrictions yet on the form or the number n of these state variables nk.0); such conditions, however, naturally arise if we require the evolution of the variables 0 to obey a closed set of deterministic laws, as we will show below. The ensemble probability of finding the system in macroscopic state f2 is given by:

296

The time derivative of this distribution is obtained by inserting (1)and can be written as

This expansion (to be interpreted in a distributional sense, i.e. only to be used in expressions of the form JdnPt(n)G(f?) with sufficiently smooth functions G ( n ) , so that all derivatives are well-defined and finite) w i l l only make sense if the single spin-flip shifts Ajk in the state variables n~, are sufficiently small. This is to be expected from a physical point of view: for finite N any state variable &(c)can only assume a finite number of possible values; only in the limit N + 00 may we expect smooth probability distributions for our macroscopic quantities (the probability distribution of state variables which only depend on a small number of spins, however, will not become smooth, whatever the system size). The first (I = 1)term in the series (4) is the flow term; retaining only this term leads us to a Liouville equation which describes deterministic flow in n space, driven by Including the second ( I = 2) term as well leads us to a Fokker-Planck the flow field dl). equation which (in addition to the flow) describes diffusion in 0 space of the macroscopic probability density Pt [n],generated by the diffusion matrix F$). According t o (4) a sufficient condition for a given set of state variables n(u)to evolve in time deterministically in the limit N + 00 is:

(since now for N + 00 only the 1 = 1 term in (4) is retained). In the simple case where the state variables n k are of the same type in the sense that $ shifts Ajk are of the same order in the system size N (i.e. there is a monotonic function AN such that Ajk = AN) for all jk), for instance, the above criterion becomes:

If for a given set of macroscopic quantities the condition (5) is satisfied we can for large N describe the evolution of the macroscopic probability density by the Liouville equation:

the solution of which describes deterministic flow:

291

d -6)*(t) = F(l)[f?*(t);t] dt

n*(o) = no

(7)

In taking the limit N + 00, however, we have to keep in mind that the resulting deterministic theory is obtained by taking this limit for finite t. According to (4) the 1 > 1 terms do come into play for sufficiently large times t ; for N + 00, however, these times diverge by virtue of (5). The equation (7) governing the (deterministic) evolution in time of the macroscopic state variables 6) on finite timescales w i l l in general not be autonomous; tracing back the origin of the explicit time dependence in the right-hand side of (7) one finds that in order to calculate F(') one needs to know the microscopic probability density p * ( u ) . This, in turn, requires solving the master equation (1) (which is exactly what one tries to avoid). However, there are elegant ways of avoiding this pitfall. We will now discuss two constructions that allow for the elimination of the explicit time dependence in the right-hand side of (7) and thereby turn the state variables 6) and their dynamic equations (7) into an autonomous level of description. The first way out is to choose the macroscopic state variables $2 in such a way that there is no explicit time dependence in the flow field F(') [f?;t ] (if possible). According to the definition of the flow field this implies making sure that there exists a vector field # [n]such that N

(with Aj ( h a , . ..,Aj,,)) in which case the time dependence of F ( l )drops out and the macroscopic state variables f? evolve in time according to:

This is the construction underlying the approach in papers like Buhmann and Schulten (1988), Riedel et al (1988), Coolen and Ruijgrok (1988). The advantage is that no restrictions need to be imposed on the initial microscopic configuration; the disadvantage is that for the method to apply, a suitable separable structure of the interaction matrix is required. If, for instance, the macroscopic state variables S l k depend linearly on the microscopic state variables u (i.e. n ( m ) $ Cj"=,,fjuj), we obtain (with the transition rates (2)):

in which case it turns out that the only further condition necessary for (8) to hold is that all local fields hk must (in leading order in N) depend on the microscopic state u only through the values of the macroscopic state variables n (since the local fields depend linearly on u this, in turn, implies that the interaction matrix must be separable). If it is not possible to find a set of macroscopic state variables that satisfies both conditions (5,8), additional assumptions or restrictions are needed. One natural assumption that allows us to close the hierarchy of dynamical equations and obtain an autonomous

298 flow for the state variables n is to assume equipartitioning of probability in the subshells of the ensemble, which allows us to make the replacement:

n-

with the result

Whether or not the above way of closing the set of equations is allowed will depend on w j ( u ) A j ( u )is constant within the extent t o which the relevant stochastic vector the 0-subshells of the ensemble. At t = 0 there is no problem, since one can always choose the initial microscopic distribution n(a)to obey equipartioning. In the case of extremely diluted networks, introduced by Derrida et al (1987), this situation is subsequently maintained by assuring that, due t o the extreme dilution, no correlations can build up in finite time and equipartitioning will be sustained (see also the review paper by Kree and Zippelius 1991). The advantage of extreme dilution is that less strict requirements on the structure of the interaction matrix are involved; the disadvantage is that the required sparseness of the interactions (compared to the system size) does not correspond t o biological reality.

c:,

3. SEPARABLE MODELS

In this section we will show how the formalism described in the previous section can be applied t o networks for which the matrix of interactions Jij has a separable form (which includes most symmetric and non-symmetric Hebbian type attractor models). We will restrict ourselves t o models with Wi = 0; the introduction of non-zero thresholds is straightforward and does not pose new problems.

3.1 Description at the Level of Sublattice Magnetisations The following type of models was introduced by van Hemmen and Kiihn (1986) (for symmetric choices of the kernel Q). The dynamical properties (for arbitrary choices of the kernel Q) were studied by Riedel et al (1988):

1

J'3. . = - -Q N

(€..<.) 3 1'

€a

= (e, .'a

(9)

*.

C

The components are assumed to be drawn from a finite discrete set A which contains elements (again the variables .$ are not allowed t o depend on N). They typically represent/contain the information ('patterns') t o be stored or processed. The Hopfield model, for instance, corresponds to choosing Q(z; y) E 2 . y and A E {-1,l). One can now introduce a partition of the system {I,. . . ,N} into nz so-called sublattices 19: ng

19 s {;I

& = 9)

{1,..., N }

EUI~ 9

9 E Ap

299

The number of spins in sublattice 19 will be denoted by 1191 (this number will have t o be large). If we choose as our macroscopic state variables the magnetisations within these sublattices, we are able to express the alignment fields hk solely in terms of macroscopic quantities:

If all relative sublattice occupation numbers p 9 are of the same order in N (which, for example, is the case if the vectors & have been drawn at random from the set AP) we may write Aj9 = O(nXN-') and use ( 6 ) . The evolution in time of the sublattice magnetisations is then found t o be deterministic in the thermodynamic limit if lim N-+w

P=o log N

Furthermore, condition (8) holds, since

We may conclude that the evolution in time of the sublattice magnetisations is governed by the following autonomous set of differential equations:

A nice overview of applications and analysis of these equations (and generalisations thereof) for various choices of the kernel Q(.; .) can be found in review papers like van Hemmen and Kiihn (1991) and Kiihn and van Hemmen (1991). The above procedure does not require symmetry of the interaction matrix. In the symmetric case Q(a; y ) = Q(y; a) one can easily verify that the system will approach equilibrium; if the kernel Q is positive definite, for instance, by inspection of the Liapunov function L { m q } :

which is bounded and obeys:

Also for some very specific non-symmetric cases Liapunov functions do exist, as shown by Herz et al (1991).

300 Note that from the sublattice magnetisations one can easily calculate the so called 'overlap' order parameters (which can be written as averages over the nz sublattice magnetisations):

Whether our not, in turn, there exists an autonomous set of laws at the level of overlaps depends on the form of the kernel Q(.; .).

3.2 Description at the Level of Overlaps Equation (10) suggests that at the level of overlaps there will be, in turn, an autonomous set of dynamical laws if the kernel Q is bilinear, i.e. Q(z;y) = C,,,,z,APy,, or: r

v

Symmetric and non-symmetric models of this type were introduced and studied by Coolen and Ruijgrok (1988) and Shiino et al (1989); also the pseudo-inverse (Kohonen 1984) is of the form (12). Now the components need not be drawn from a finite discrete set, as we will see (as long as they do not depend on N). The Hopfield model corresponds to choosing A,, ,6 and E {-1,l). The alignment fields h k can now be written in terms of the overlap order parameters m,:

(r

(r

hk(u) = t

k

.Am(u)

m

= (ml,. . . ,mp)

Since for the present choice of macroscopic variables we find A,, = O(N-'), the evolution in time of the overlap vector m becomes deterministic in the thermodynamic limit if (according to ( 6 ) ) : lim - P =O N-w

rn

Again condition (8) holds, since N

In the thermodynamic limit the evolution in time of the overlap vector m is governed are drawn at random by an autonomous set of differential equations; if the vectors according to some distribution p ( ( ) these dynamical laws become:

tk

As in the case of the more familiar methodology of equilibrium statistical mechanics, the macroscopic laws one obtains are coupled and non-linear. Only for very special cases

30 1

T=O. 1

-1

0

T=0.6

-1

1

0

T=1.1

1

-1

1

1

1

0

0

0

-1

-1

-1

0

1

0

1

mz

-1 -1

m1

0 m 1

1

-1

0

1

m1

Figure 1. Flow diagrams obtained by numerically solving eqns. (13) for p = 2. Upper row: A,, = ,6 (the Hopfield model); lower row: A =

(ll )

(for both models the critical temperature is T,= 1).

will one be able to calculate all solutions analytically (Coolen et al 1989, Coolen 1991, Jonker and Coolen 1991). Generalisations of (13) to systems with special structure (where the order parameters will also be position dependent: m(t)+ m ( e , t ) )can be found in Coolen (1990) and Coolen and Lenders (1992). Figure 1 shows in the ml,mz-plane the result of solving the macroscopic laws (13) numerically for p = 2 and two choices of the matrix A. The first choice (upper row) corresponds to the Hopfield model; as the temperature increases the amplitudes of the four attractors (corresponding to the two patterns (’’ and their mirror imagesf’) continuously decrease, until at the critical temperature T, = 1 they merge into the trivial attractor m = (0,O). The second choice corresponds to a non-symmetric model (i.e. without detailed balance); at the macroscopic level of desciption (at finite timescales) the system clearly does not approach equilibrium; macroscopic order now manifests itself in the form of a limit-cycle (provided the temperature T is below the critical temperature T, = 1 where this limit-cycle is destroyed). To what extent the laws (13) are in agreement with the result of performing the actual simulations in finite systems is illustrated in figure 2. Again symmetry of the interaction matrix is not required. For specific non-symmetric choices for the matrix A limit-cycle solutions exist (Coolen and Ruijgrok 1988, Nishimori et al 1990); competition between such limit-cycles and ordinary fixed-point attractors

302

N= 1 000

Analytical

N=2000

1

1

1

0

0

0

-1

-1

0

-1

1

m2

-1 -1

0

m1

m 1

1

-1

0

1

m1

Figure 2. Comparison between simulation results for finite systems ( N = 1000 and N = 2000) and the analytical prediction (13) with respect to the evolution of the order parameters m ( p = 2, T = 0.8 and A=

(

gives rise to interesting phase transitions (Coolen and Sherrington 1992). In the symmetric case A,, = A , the system will approach equilibrium; the Liapunov function (11) now becomes:

L{m,}

1

.

1

= -m A m - -(log cosh [/3(

2

P

+

Am])t

If one is willing to pay the price of restricting oneself to the limited class of models (12) (as opposed to the more general class (9)) and t o the more global level of description in terms of p overlap parameters m, instead of nx sublattice magnetisations mv, then (for there are two rewards. Firstly there will be no restrictions on the stored quantities instance, they are allowed to be real-valued); secondly the number p of patterns stored can be much larger for the deterministic autonomous dynamical laws to hold (p << f l instead of p << log N,which from a biological point of view is too restrictive.

(r

4. PATH INTEGRAL FORMALISM

A different approach is based on a formal solution of the microscopic dynamical laws (1) developed by Sommers (1987), in the spirit of a dynamical method for Langevin spins, based on the calculation of a generating functional (De Dominicis, 1978) as applied to spin-glasses by Sompolinsky and Zippelius (1982) and Crisanti and Sompolinsky (1987). Now the step from the microscopic level of description in terms of spin states u t o the macroscopic level of description in terms of dynamic order parameters is taken after having (formally) solved the microscopic master equation. The local fields hk are defined in a more general way h i( u ;t ) =

c N

j=1

Jijc7j

+4(t)

J 11 .. - 0

303

in order to enable the calculation of time dependent response functions and correlation functions by differentiation with respect to the time-dependent external fields O k ( t ) . The solution of the master equation as given by Sommers holds for a specific type of initial conditions:

and takes the form of a path integral:

in which the complex functions m;(t;8,s’) are the solutions of:

d -m;

dt

= tanh phk(s(t)jt ) ] - m;

+ is; [l - mi2]

m;(O)

= mi

This expression is obtained by performing manipulations on the formal solution of the master equation:

in which ? is the time-ordering operator. We will not discuss the details of the derivation of (15) from (16), since firstly it is somewhat involved and, secondly, although there appears to be agreement in the literature on the final result (15), there is still discussion about which is the proper derivation (see Sommers,l987, and Lusakowski, 1991). If, finally, we insert the unit operator

and introduce additional variables hi(.r)for representing the 6 functions as integrals, we obtain

d -m; dt

= tanh p h k ( t ) ] - m;

+ is; [l - mi2]

m;(O)

= mi

(17)

The advantage of this (rather complicated) expression (17) for the solution p t ( u ) of the master equation is that the interaction variables {J;j}appear linearly in the exponent, which, in cases where the interactions contain quenched disorder, enables one to average easily expectation values of self-averaging physical quantities. Application of Sommer’s

304

result to the dynamics of the Hopfield model by Rieger et al(1988) led to a dynamical verification of the equilibrium picture obtained by Amit et al. (1985). The order parameters in this description are not only the familiar overlaps m,, but, in addition, time-dependent correlation and response functions. More recently Shim et al (1991) applied a modified version of (15) to symmetric Hopfield-type systems with transmission delays. Note, finally, that the restriction to initial conditions of the type (14) is not dramatic, since any initial distribution (and corresponding t > 0 solution) can be written as an integral over distributions of the type (14). A similar method was used by Horner et al (1989) who obtained approximate dynamic mean field equations for symmetric Hopfield-type models. Here the key object is a suitably defined generating functional 8,rather than the formal solution of the master equation. From 5, which is defined as an average over all trajectories generated by the microscopic stochastic process and all initial conditions, one can calculate observables and time-dependent response functions by functional differentiation. The path integral methods described in this section are powerfull tools, in that they generate exact dynamical expressions at the microscopic level of description. Therefore they can in principle be applied in notorious cases, where no alternative dynamical method is yet available (like having an extensive number of p = a N stored patterns in an N-neuron attractor model). The common ingredient of the applications of these methods to neural network dynamics that have appeared in literature so far, however, is that in the aforementioned notorious cases, in order to arrive at analytical results it appeared to be necessary to either restrict oneself to semi-static situations (Rieger et al, 1988) or to resort to interpolations between such semi-static situations and exact short-time solutions (Horner et al, 1989).

5 . OUTLOOK

In this contribution we have restricted ourselves to McCulloch-Pitts/Ising spin models of neural networks. Most of the techniques we have described, however, can also be applied to dynamical models which employ different types of neural variables, different types of interactions (or interaction architectures) or different types of stochastic dynamical rules. We will briefly discuss some references in which such extensions can be found. Dynamical studies of diluted and fully connected models with non-Ising neural variables have been performed, for instance, for Langevin spins (Kree and Zippelius, 1987), phase variables (Noest, 1988), Potts variables (Boil6 and Mallezie, 1989), continuous (‘graded response’) neurons (Mertens, 1991; Jedrzejewski and Komoda, 1992) and discrete multi-state spins (Bolle et al, 1992). Applications of dynamical methods to systems with multi-spin interactions (where the local alignment fields are no longer linear in the neural state variables) can be found in, for instance, Kanter (1988), Kuhn et al (1989) and Tamarit et al (1991). The dynamics of models with layered feed-forward interaction architectures (where the analysis is simplified by the architecture, which allows for iteratively solving the dynamics layer by layer) has been studied by Domany et al (1989) (for Ising spin neurons) and Shim et al (1992) (for Potts spins). The macroscopic analysis in terms of dynamical order parameters for separable fully connected Ising spin models with

305 parallel stochastic dynamics has been performed by Bernier (1991) (the coupled non-linear differential equations for the macroscopic state variables now become coupled non-linear discrete mappings).

Fortunately, many interesting unsolved problems remain. There are, for instance, the as yet unexplained remanence effects in attractor networks where numerical simulations show that above the critical storage capacity line (in contrast to what the equilibrium theory predicts) the retrieval order parameter is found not to vanish (Amit and Gutfreund, 1987).

REFERENCES Amit D.J., Gutfreund H. and Sompolinsky H.; 1985, Phys. Rev. A32,1007 Amit D.J., Gutfreund H. and Sompolinsky H.; 1985, Phys. Rev. Lett. 55, 1530 Amit D.J. and Gutfreund H.; 1987, Annals of Physics 173, 30 Bernier 0.; 1991, Europhys. Lett. 16, 531 Buhmann J. and Schulten K.; 1987, Europhys. Lett. 4, 1205 Coolen A.C.C.; 1990, in “Statistical Mechanics of Neural Networks” ed. L. Garrido (Berlin, Springer), 381 Coolen A.C.C.; 1991, Europhys. Lett. 16, 73 Coolen A.C.C., Jonker H.J.J. and Ruijgrok Th.W.; 1989, Phys. Rev. A40, 5295 Coolen A.C.C. and Ruijgrok Th.W.; 1988, Phys. Rev. A38, 4253 Coolen A.C.C. and Lenders L.G.V.M.; 1992, J. Phys. A25,2577 Coolen A.C.C. and Sherrington D.; 1992, to be published in J. Phys. A Crisanti A. and Sompolinsky H.; 1987, Phys. Rev. A36,4922 De Dominicis C.; 1978, Phys. Rev. B18, 4913 Derrida B., Gardner E. and Zippelius A.; 1987, Europhys. Lett. 4, 167 Domany E., Kinzel W. and Meir R.; 1989, J. Phys. A22, 2081 van Hemmen J.L. and Kiihn R.; 1986, Phys. Rev. Lett. 57, 913 van Hemmen J.L. and Kiihn R.; 1991, in “Models of Neural Networks” Hers A.V.M., Li 2. and van Hemmen J.L.; 1991, Phys. Rev. Lett. 66, 1370 ed. E. Domany, J.L. van Hemmen and K. Schulten (Berlin, Spinger), 1 Hopfield J.J.; 1982. Proc. Natl. Acad. Sci. USA 79, 2554 Horner H., Bormann D., Frick M., Kinzelbach H. and Schmidt A.; 1989, Z. Phys. B76, 381 Jedrzejewski J. and Komoda A.; 1992, Europhys. Lett. 18, 275 Jonker H.J.J. and Coolen A.C.C.; 1991, J. Phys. A24,4219 Kanter I.; 1988, Phys. Rev. A38, 5972 Kree R. and Zippelius A.; 1987, Phys. Rev. A36,4421 Kree R. and Zippelius A.; 1991, in “Models of Neural Networks” ed. E. Domany J.L. van Hemmen and K. Schulten (Berlin, Spinger), 193 Kiihn R. and van Hemmen J.L.; 1991, in “Models of Neural Networks” ed. E. Domany, J.L. van Hemmen and K. Schulten (Berlin, Spinger), 213 Kiihn R., van Hemmen J.L. and Riedel U.; 1989, J. Phys. A22, 3123 Lusakowski A.; 1991, Phys. Rev. Lett. 66, 2543

306

Luttinger J.M.; 1976, Phys. Rev. Lett. 37, 778 Mattis D.C.; 1976, Phys. Lett. A56, 421 Mertens S.; 1991, J. Phys. A24, 337 Nishimori H., Nakamura T. and Shiino M.; 1990, Phys. Rev. A41, 3346 Noest A.J.; 1988, Europhys. Lett. 6, 469 Noest A.J.; 1988, Phys. Rev. A38, 2196 Riedel U., K i h n R. and van Hemmen J.L.; 1988, Phys. Rev. A38, 1105 Rieger H., Schreckenberg M. and Zittartz J.; 1988, Z. Phys. B72, 523 Shiino M., Nishimori H. and Ono M.; 1989, J. Phys. SOC.Japan 58, 763 Shim G.M., Choi M.Y. and Kim D.; 1991, Phys. Rev. A43, 1079 Shim G.M., Kim D. and Choi M.Y.; 1992, Phys. Rev. A45, 1238 Sommers H.J.; 1987, Phys. Rev. Lett. 58, 1268 Sommers H.J.; 1987, Habilitationsschrift, Essen Sompolinsky H. and Zippelius A.; 1982, Phys. Rev. B25,6860 Tamarit F.A., Stariolo D.A. and Curado E.M.F.; 1991, Phys. Rev. A43, 7083

Mathematical Approaches to Neural Networks

J.G.Taylor (Editor) 0

1993 Elsevier Science Publishers B.V. All rights reserved.

307

Information Theory and Neural Networks John G . Taylor” and Mark D. Plumbley” ”Centrefor Neural Networks, Department of Mathematics, King’s College London, Strand, London WC2R 2LS, UK

1

Introduction

Ever since Shannon’s “Mathematical Theory of Communication” [40] first appeared, information theory has been of interest to psychologists and physiologists, t o try to provide an explanation for the process of perception. Attneave [S] proposed that visual perception is the construction of an economical description of a scene from a very redundant initial representation. Barlow [9] suggested that lateral inhibition in the visual pathway may reduce the redundancy of an image, so information can be represented more efficiently. More recently, Linsker with his ‘Infomax’ principle [21, 221, Atick and Redlich (5, 61, and Plumbley and Fallside [30, 321 have continued with this approach with considerable success. There have also been important advances in data compression techniques associated with principal component analysis. The original work of Oja [23] has now been extended to the analysis of higher-order statistics by Taylor and Coombes [45], and these techniques are presently attracting a great amount of interest. They have implications for the rapid feature analysis of patterns which are not linearly separable, and for higher order curve and surface fitting. Finally there is a very general approach to the information-theoretic analysis of neural networks, using differential geometric ideas, by Amari [2]. This appears to be the deepest usage of information theory in neural networks, although only very preliminary results are available at present.

2 2.1

Information Theory and Learning Information Theory

Let us briefly introduce some concepts from information theory. In particular, we are interested in the entropy of a random variable, and the mutual information between two random variables [40].

308

Consider an experiment with n possible outcomes z, 1 5 z 5 n with respective probabilities p,. The entropy, H ( X ) , of this system is defined by the formula

Entropy gives us a measure of the uncertainty in a system. If the probabilities of the various outcomes are equal, then the entropy will be maximised. On the other hand, if one of the outcomes is certain to happen, so its probability is one and the other outcomes have probability zero, then the entropy will be zero. (For this to work properly in the limit, we take p , logp, + 0 in the limit p, + 0.) As an example, for a fair coin toss X , with n = 2 and Phead = pt,il = 1/2, we have 1

1

1

'>

H(R) = - (-log5+ 2 -log2 2 = log2.

However, suppose we have seen the coin land with e.g. heads uppermost: call this observation Y . We now know what the outcome of the toss is, so we have Phed = 1 and ptail = 0. If we measure the entropy now, we get

H ( X 1 Y ) = - ( l l o g l +OlogO) =

o+o=o

i.e. no uncertainty, since we know that the coin landed with heads uppermost. We use H ( X I Y ) to denote the entropy in X given that we know Y . We shall see shortly that the information in the observation X about the coin toss Y is the difference between these two entropies. More formally, if we have two random variables X and Y which take values z and y in the range 1 5 5 5 n and 1 5 y 5 m, and have joint probabilities pZy,their joint entropy is defined to be H ( X ,Y ) = lOgP,y (2)

CP", 2.Y

and the conditional entropy of X given Y is

H ( X 1 Y ) = - c p z y 1 0 g ~= H ( X , Y ) - H ( Y ) 2,Y PY

(3)

If S and Y are independent, then p,, = p,py, so

H ( X 1 Y ) = -Cp,,log- PzPy = - C p , l o g p , 2.Y

PY

=H(X).

2.Y

The mutual information I ( X ,Y) between X and I' is defined by the formula

I ( X , Y ) = H ( X ) - H ( X 1 Y ) = ~ p , y l o g P- X Y 2.Y PZP,

(4)

309

which is zero if X and Y are independent. Note that Z ( X , Y ) = I ( Y , X ) , so the information in Y about X is the same as the information in X about Y . Another information-theoretic measure is the I-divergence or cross-entropy distortion measure DCE from one probability distribution to another. The cross-entropy from a 'true' probability distribution p to an 'estimated' probability distribution q is defined by the expression Pi DCE(P, q ) = C P i log. ; t

which is zero if p and q are identical, but positive otherwise. Cross-entropy is asymmetric, so that in general D c ~ ( p , q# ) DcE(q,p). By substituting pi = p,, and qi = p,p, in the expression for D c E above, it can be seen that mutual information between X and Y is the cross-entropy from the true probability distribution p,, governing X and Y , to the probability distribution p,p, which would govern them if they were independent (but had the same marginal distributions). If the random variables X and Y are continuous, we use probability density p(x,y) instead of probability distribution p,, an integral instead of a sum in our expressions for entropy and mutual information:

H(X) = -

Jm

p(x) logp(x) dx

-m

For example, if X has an N-dimensional Gaussian probability density with covariance matrix C x , its entropy will be given by

H ( X ) = log ( ( 2 ~ e ) ~ / ' ( d eCX)"') t

.

( 7)

For the special case where Y is a function of X with some additive noise

Y =f

a, i.e.

( x ) +@

then the mutual information between Y and X is given by

I ( Y , X )= H ( Y )- H ( @ )

(8)

i.e. the entropy of the 'signal plus noise' less the entropy of the 'noise'. If both Y and are Gaussian with covariance matrices CY and C, respectively, we get

I()', S)= log ((det C\,)'/') - log ((det C,)'/')

2.2

Supervised Learning

2.2.1

BackProp with Mean Squared Error Distortion

.

One of the major uses for information theory has been in interpretation and guidance for unsupervised neural networks: networks which are not provided with a teacher or target output which they are to emulate. However, we can see how information relates

310

Figure 1: Supervised neural network with distortion measure D to the more familiar supervised learning schemes, and in particular the use of Error Back Propagation (‘BackProp’) to mininize mean squared error (MSE) in a multi-layer perceptron (MLP). Given a particular input X , supervised learning attempts to produce an output Y as ‘close’ as possible to some target output R, where closeness is measure with some distortion measure D (Fig. 1). Typically, the MSE distortion measure E = D M ~ E ( R , Y=) E (IY - 01’) is used. BackProp calculates the derivative of E with respect to each of the weights (the adjustable parameters) in the network, and adapts each weight to move ‘downhill’ in this error landscape towards the minimum distortion position [37]. Consider for the moment an alternative, information-theoretic, approach. Suppose we would like the output I’ to contain as much information as possible about the target R. We can show that t,he two approaches are related, in that using BackProp to minimise MSE represents an approximation to maximising the information I(R,Y) in Y about R. If we write ip = Y - R, minimising MSE amounts to minimising the term

E (1ip1’) = Trace(C@) where Ce = E(ipipT) is the correlation matrix of ip. On the other hand, maximising the information I ( R , Y )= H(R) - H(R(Y) is equivalent to minimising the entropy of R given Y , H(RlY), since H(R) is outside our control, and is independent of the changes made in the network. It is simple to show [28] that

H(R1Y) = H(ipIY) 5 H ( @ ) and

H ( @ )5 log ( ( 2 ~ e ) ~ / * ( dCe)”’) et where N is the dimensionality of the output. Now, since (det Ce)’IN5 Trace(Ce)/N by the arithmet,ic/geometric mean inequality, minimising the MSE Trace(Ce) will mininiise an upper bound on H(RlY), and thus maximise a lower bound on the information I(R, IT)in Y about R. Due to the number of inequalit,ies appearing in the process, this bound may not be particularly tight, especially when the error are ‘unbalanced’ in some way. See [28] for a discussion of some of the implications of this.

31 1

2.2.2

Cross-Entropy Distortion

Other distortion measures are possible in place of MSE. In particular, the informationtheoretic cross-entropy distortion [43] has been suggested. The use of this distortion measure relies on the both the output and the target representing probability distributions directly. Thus a single output must be restricted to lie between 0 and 1: the sigmoid exp(-u)) as the final stage in the output unit will conveniently function a(.) = 1/(1 achieve this. With more than one output unit, they can either be treated as independent features, in which case the total distortion measure is given by

+

or they can be treated as asingle probability distribition, in which case the total distortion

In the latter case, the additional restriction that the outputs must sum to one must be enforced, to ensure the outputs form a true probability distribution. Bridle’s Softmar function [13]

can be used to enforce this condition.

2.3 2.3.1

Unsupervised Learning: Direct Approaches G-Max: Maximising Cross-Entropy

An early unsupervised learning algorithm derived from an information-theoretic approach was the G-Maximization algorithm suggested by Pearlmutter and Hinton [27]. They used single binary neuron with N binary inputs xi and corresponding weights w,,i = 1,. . . ,N and a single output y, which can be either ‘0’or ‘1’. The probability that the output state is ‘1’is

where the n ( . ) is the logistic sigmoid function

+

1

u(x)= 1 exp(-z)’

(10)

They wanted the unit to discover regularities in the input patterns, i.e. patterns that occur more often than would be expected if the activities of the individual input lines were assumed to be independent.

312

To this end, they chose as their target function the cross-entropy distortion

between the true output probability distribution p, and an ‘independent input xtivity’ probability distribution q of the output. The algorithm is run in two phases. The first phase is run with with the real input data to accumulate the output statistics which define the output probability distribution p,. The probabilities of each of the inputs is also measured for use in the second phase. In the second phase, the inputs xi are set randomly to ‘1’or ‘0’ with the same probabilities that they had in the first phase, but independently of each other. The output statistics are measured in this second phase: this defines the probability distribution qy. Since the cross-entropy measure used can be differentiated with respect to each of the weights wi, a steepest descent search can then be used to modify the weights to maximise the G measure. When this algorithm was tested on a 10 x 10 ‘retina’ exposed to randomly positioned and oriented edges, the unit typically developed a centre-on surround-off response (or vice versa). This type of response is typical of retinal ganglion cells. Unfortunately, it was rather awkward to extend the G-max algorithm to more than one output unit, since some additional mechanism has to be employed to prevent all the units learning to respond to the same features of the input. 2.3.2

I-Max: Maximising Mutual Information between Output Units

More recently, Becker and Hinton [ll]suggested that the information betmeenoutput units could be used as the objective function for an unsupervised learning technique (Fig. 2). In a visual system, this scheme would attempt to extract higher-order features of the visual scene which are coherent over space (or time). For example, if two networks each produce a single output from two separate but neighbouring patches of a retina, the objective of their algorithm is to maximise the mutual information I(Y1,Yz) between these two outputs. A steepest-ascent procedure can be used to find the maximum of this mutual information function, both for binary- and real-valued units. One application of this principle is the extraction of depth from random-dot stereograms. Nearby patches in an image usually view objects of a similar depth, so if the mutual information between neighbouring patches is to be maximised, the outputs from both output units y1 and y2 should correspond to the information which is common between the patches, rather than that which is different. In other words the outputs should both learn to extract the common depth information rather than any other property of the random dot patterns. For binary-valued units, with each unit similar to that used by the G-max scheme described above, the mutual information I(Y1, Y2) between the two output units is

so if the probability distributions P(yl), P(y2) and P(ylry2) are measured, this mutual information can be calculated directly. Of course, it is sufficient to measure P(y1, yz) only,

313

Yl

t

Y2

Maximise mutual information

t

Retina

Figure 2: Maximising Mutual Information between Output Units since

1

P(Y1) =

1P(Yl,?/Z) yz=o

and similarly for P(y2). The derivative of (11) can be taken with respect to the weights in the network for each different input pattern, so enabling the steepest-ascent procedure to be used. For real-valued outputs it would be impossible to measure the entire probability distribution P(Y1, Y z ) , so instead it is assumed that the two outputs have a Gaussian probability distribution, and that one of the outputs is a noisy version of the other, with independent additive Gaussian noise. In this case, the information I(Y1,1’2) between the two outputs can be calculated from the variances of one of the outputs (the ‘signal’) and the variance of the difference between the outputs (the ‘noise’) as

where u;, is the variance of the output of the first unit, and u ; , - ~is the variance of the difference between the two outputs. If we accumulate the mean and variance of both Y1 and Y1 - Y2, it is possible to find the derivative of (12) for each input pattern, with respect to each weight value. Thus the weights in the network can be updated in a steepest-ascent procedure to maximise I(Yl,Y2), or at least the approximation to I(Yl,1’2) given by (12).

314

Becker and Hinton found that unsupervised networks using this principle could learn to extract depth information from random-dot stereograms with either binary- or continuousvalued shifts, as appropriate for the type of outputs used, although in some cases it helped to force the units to share weight values, further enforcing the idea that the units should calculate the same function of the input. They generalised their scheme t o allow networks with hidden layers, and also to allow multiple output units, with each unit maximising the mutual information between itself and a value predicted from its neighbouring units. This latter scheme allowed the system to discover an interpolation for curved surfaces. Subsequently, Zemel and Hinton [49] generalised this procedure t o allow for more than one output per module. In their network, they use 4 outputs per unit to attempt identify four degrees of freedom in 2-dimensional objects: horizontal and vertical position, orientation, and size. The mutual information measure is now

where C,;+V,is the covariance matrix of the sum of Y I and 12’ (now vectors, of course), and Cy,-l,, is the covariance matrix of their difference. By measuring the degree of mismatch between the two representations, the model can tell roughly how much one half of an object, is perturbed away from the position and orientation which is consistent with the other half of the object.

3 3.1

Predictive Coding Redundancy and Visual Scenes

Soon after the introduction of his Information Theory, Shannon [42] used a prediction ‘game’ to investigate the information content of English text. He showed that the entropy of text is just over 1 bit per letter, much less than the 4.7 bits (= logz26) which would be needed if the 26 letters of english text were independent and occurred with equal frequency. This is itself much less than the 8 bits per letter used to represent text in a typical computer. This shows that there is a significant amount of redundancy in a typical passage of text: redundancy is the ratio of the information capacity used to represent a signal to the information actually present in the signal. If a communication system could take advantage of this redundancy, text could be transmitted more economically than before, since each bit used costs a certain amount of resources to transmit it. Attneave [S] suggested that the process of perception may be creating an economical description of sensory inputs, by reducing the redundancy in the stimulus. As a concrete example, he considered a picture of an ink bottle 011 a desk, similar to Fig. 3. If a subject is asked to predict the colours in the picture, scanning along the rows from left to right, taking successive rows down the picture, most mistakes are made a t the edges and corners in the picture. Applying this ‘Shannon game’ on the scene suggests that most information is concentrated around the edges and corners of a scene, which fits well with psychological suggestions that these features are important in a visual scene. It is also strengthened by the later observations by Hubel and Wiesel [20] of ‘line detector’ cells in the visual cortex.

315

Figure 3: A redundant visual stimulus (After Attneave [ S ] )

Figure 4: Network output signal Y is corrupted by noise CJ. Using this redundancy reduction hypothesis, Barlow [9] suggested that lateral inhibition may be producing an economical description of the scene. By subtracting a weighted sum of surrounding sensor rcsponses, the resulting activations are more independent (less redundant) than the original stimulus, while the total information content remains unchanged. Using lateral inhibition, the visual scene can be represented more economically.

3.2

Whitening filters and predictive coding

The original justification for the redundancy reduction hypothesis considered for example signals with a finite number of levels, such as just noticeable diflerences (JNDs). However, a related approach was formulated by Shannon for economical transmission of signals through a noisy channel [41]. Consider a syst,em (a neural network in this case) which is transmitting its real-valued output signal Y through a channel where it will be corrupted by additive noise 0 (Fig. 4). If there were no restrictions on Y , we could simply amplify it until we had overcome as much of the noise as we like. However, suppose that there is a power cost

associated with transmitting the signal 1' t,lirough the channel, where S ( f ) is the power

316

spectral density of Y (the ‘signal’) at frequency f , and B is a bandwidth limit. Then we wish to maximise the transmitted information (assuming both signal and noise are Gaussian)

where N ( f ) is the power spectral density of ip (the ‘noise’), for a given power cost ST. Using the Lagrange multiplyer technique, we attempt to maximise

as a function of

S(f)for every 0 5 f 5 B. This is the case when S(f)+ N(f) = constant

(17)

so if N ( f ) is white noise, i.e. the power spectral density is uniform (or flat), the power

spectral density S(f)should also be uniform [41].A filter which performs this flattening is called a whitening filter. It is well known that a signal with flat power spectral density has an autocorrelation function R Z I y ( ~=) E ( Y ( t ) , Y ( t T ) ) which is proportional to a delta funct,ion b ( ~ ) In . other words, the time-varying output signal Y(t1) at any time t l should be uncorrelated with the signal Y ( t 2 ) at any other time t 2 # tl. This approach leads to an equivalent condition where the signal is represented over a regular grid of units in space instead of time, such as a signal from a grid of visual receptors. In this case the signal should be transformed so that the outputs of the transform are uncorrelated from each other. One way of achieving this decorrelation of outputs is to use the technique of linear predictive coding (See e.g [26]).To see how this works, consider a time-sequence of input values I ; , 1 i - 1 , . . . where zi is the most recent value. We can form the least mean square (LMS) linear prediction ?i of I ; from the previous values as follows:

+

2; = u15,-1

where the coefficients

uj

+ U 2 2 i - 2 + . ..

(18)

are chosen to minimise the expected squared error E

=E

(I;-2;)2.

(19)

Taking the derivative of this with respect, to each coefficient u j , the condition for this minimum is E [ ( T ; - 2;)1i-j] = 0 (20) for all j > 0. If we take the residual yi = I ; - ?i to be the output of our transform, the LMS linear prediction gives us E [ y i ~ i - j ]= 0 (21) for all j > 0, and therefore E [Yiyk] = 0 (22) for all k < i, since Y k = x k - (ulzk-1 a2xk-2 ..’). Thus linear predictive coding has given us the uncorrelated outputs we need.

+

+

317

Figure 5: Linear decorrelating networks ( M = 2).

3.3

Local Decorrelating Algorithms

One of the early suggestions for learning in neural networks was Hebb's (191 principle, that the effectiveness of the connection between two units should be increased when they are both active a t the same time. This has been used as the basis of a number of artificial neural network learning algorithms, so-called Hebbian algorithms, which increase a connection weight in proportion to the product of the unit activations at each end of the connection. If the connection weight decreases (or increases its inhibition) in proportion to the product of the unit activations, this is called anti-Hebbian learning. A number of anti-Hebbian algorithms have been proposed to perform decorrelation of output units. For example, Barlow and Foldigk [lo] have suggested a network with linear recurrent lateral inhibitory connections (Fig. 5(a)) with an anti-Hebbian local learning algorithm. In vector notation, we have an M-dimensional input vector z, an M-dimensional output vector y, and an M x M lateral connection matrix V. For a fixed input, the lateral connections cause the output values to evolve according to the expression (yt)t+l

= 2, - Cut3(y3)t

i.e.

&+l

=X-V&

(23)

3

at time step t , which settles to an equilibrium when 2 = .r - Vy, which we can write as

+

provided ( I M V) is positive definite. We assume that this settling happens virtually instantaneously. The matrix V is assumed to be symmetrical so that the inhibition from unit i to unit j is the same as the inhibition from j t o i, and for the moment we assume that there are no connections from a unit back to itself, so the diagonal entries of V are zero. Barlow and FoldiAk [lo] suggested that for each input z, the weights uj3 between different units should altered by a small change i#j Avij = V Y i Y j where 7 is a small update factor. In vector notation this is A V = Fffdiag(ygT)

(25)

318

since the diagonal entries of V remain fixed at zero. This algorithm converges when E(y,y,) = 0 for all i # j , and thus causes the outputs to become decorrelated [lo]. Atick and Redlich (71 considered a similar network, but with an integrating output d y / d t = c - Vy leading to y = V-'a when it has settled. They show that a similar algorithm for the lateral inhziitory connections between different output units leads to decorrelated outputs, while reducing a information-theoretic redundancy measure. The algorithms considered so far simply decorrelate their outputs, but ignore what happens to the diagonal entries of the covariance matrix. For a signal with statistics which are position-independent, such as images on a regularly-spaced grid of receptors, we can consider the problem in the spatial frequency domain. Decorrelation is optimal, as we have seen above, and the variance of all the outputs will happen to be equal. If we do not have position-independent statistics, we can go back to the power-limited noisy channel argument, but use the actual output covariance matrix instead of working in the frequency domain. For small output noise, we can express the transmitted information as

I ( q , X ) = 1/2logdetC,, - 1/210gdetCo

(27)

and the power cost as S, = Trace(Cy).

Using the Lagrange multiplier technique again, we wish to maximise

J = I ( @, X ) - 1/2XST which leads to the condition [30]

(29)

cy = l/XI&f.

In other words, not only should the outputs be decorrelated, but they should all have the same variance, E(y:) = 1 / X . The Barlow and FoldiAk [lo] algorithm can be modified to achieve this, if self-inhibitory connections from each unit back to itself are allowed [30]. The algorithm becomes A V ; ~= vyiyj - (l/X)S,j

i.e.

A V = v(g/gT- (l/X)I&f)

(31)

which monotonically increases J as it progresses. This is perhaps a little awkward, since the self-inhibitory connections have a different update algorithm to the normal lateral inhibitory connections. As an alternative, a linear network wit.11 inhibitory interneurons (Fig. 5(b)) can he used. After an initial transient, this network settles to y=c-Vg and z = V T y(32) i.e. y = ( I + VVT)-'c (33) where v i j is now the weight of the excitatory (positive) connection from yi to z j , and also the weight of the inhibitory (negative) connection back from to yi. Suppose that the weights in this network are updated according to the algorithm Avij = ? ~ ( y i-~ l/Xlfij) j

(34)

319

r - - - - -

I - - - - - - ,

sh

*

G(f) -

which is a Hebbian (or anti-Hebbian) algorithm with weight decay, and is

in vector notation. Then the algorithm will converge when Cy = l / A I n 4 , which is precisely what we need to maximise J . In fact, this algorithm will also monotonically increase J as it progresses. This network suggests that inhibitory interneurons, which are found in many places in sensory systems, may be performing some sort of decorrelation task. Not only does the condition of decorrelated equal variance output optimize information transmission for a given power cost, but it can be achieved by various biologically-plausible Hebb-like algorithms.

3.4

Optimal filtering

Srinivasan, Laughlin and Dubs [44]suggested that predictive coding is used in the fly’s visual system to perform decorrelation. They compared measurements from the fly with theoretical results based on predictive coding of typical scenes, and found reasonably good agreement at both high and low light levels. However, they did find a slight mismatch, in that the surrounding inhibition was a little more diffuse than t,he theory predicted. A possible problem with the original predictive coding approach is that only the output noise is considered in the calculation of information: the input noise is assumed to be part of the signal. At low light levels, where the input noise is a significant proportion of the input, the noise is simply considered t o change the input power spectrum, making it flatter [44].This assumption means that the predictive coding is an approximation to a true optimal filter: the approximation is likely to be worse for either high frequency components, where the original signal power spectral density is small, or for low light conditions, where all signal compoiient,s are small. In fact, it is possible to analyse the system for both input and output noise (Fig. 6). We can take a similar Lagrange multiplier approach as before, and attempt to maximise transmitted information for a fixed power cost. Omitting the details, we get the following quadratic equation to solve for this optimal filter a t every frequency f [33]

320

Figure 7: Typical optimal filter solution, for equal white receptor and channel noise. where R, is the channel signal to noise power spectral density ratio &IN,, and R, is the receptor signal to noise power spectral density ratio S,/N,, and y is a Lagrange multiplier which determines the particular optimal curve to be used. This leads to a non-zero filter gain Gh whenever R, > [(y/N,) - 11-l. For constant N, (corresponding to a flat channel noise spectrum) there is therefore a certain cut-off point below which noisy input signals will be suppressed. Fig. 7 shows a typical optimal solution, has been investigating modifications to the together with its asymptotes. Plumbley [29,31] decorrelating algorithms mentioned above which may learn to approximate this optimal filtering behaviour. Atick and Redlich [5] used a similar optimal filtering approach in their consideration of the mammalian visual system, minimising redundancy for fixed information rather than maximising information for fixed power. They compared their theory with the spatiotemporal response of the human visual system, and found a very good match [4]. These results suggest very strongly that economical transmission of information is a major factor in the organization of the visual system, and perhaps other sensory systems as well.

4

Principal Component Analysis and Infomax

Principal component analysis (PCA) is widely used for dimension reduction in data analysis and pre-processiiig, and is used under a variet,y of names such as the (discrete) Karhunen Lokve Transform (KLT), factor analysis, or the Hotelling Transform in image processing. Its primary use is to provide a reduction in the number of parameters used to represent a quantity, while minimising the error introduced by so doing. In the case

32 1

Figure 8: The Oja Neuron. of PCA, a purely linear transform is used to reduce the dimensionality of the data, and it is the transform which minimises the mean squared reconstruction error. This is the error which we get if we transform the output y back into the input domain to try to reconstruct the input g so that the error is minimised. Linsker’s principal of maximum information preservation, “Infomax” , can be applied to a number of different forms of neural network. The analysis, however, is much simpler when we are dealing with simple networks, such as binary or linear systems. It is instructive to look a t the linear case of PCA in some detail, since much effort in other fields has been directed at linear systems. We should not be too surprised to find a neural network system which can perform KLT and PCA. From one point of view, these conventional data processing methods let us know what to expect from a linear unsupervised neural network. However, the information theoretic approach to the neural network system can help us with the conventional data processing methods. In particular, we shall find that a dilemma in the use of PCA, known as the scaling problem, can be clarified with the help of information theory.

4.1

The Linear Neuron

Arguably the simplest form of unsupervised neural network is an N-input, single-output linear neuron (Fig. 8). Its output response y is simply the sum of the inputs zi multiplied by their respective weights wi,i.e. N

or, in vector notation, y =ZTZ

where u, = [wl, . . . ,w ~ and] = ~ [XI,.. . ,Z N ] ~are column vectors. The output y is thus the dot product -?:.u) of the input c with the weight vector u.If 0 is a unit vector, i.e. = 1, y is the component of .?: in the direction of u) (Fig. 9).

10

322

”?

x

Figure 9: Output y as a component of g,with unit weight vector.

We thus have a simple neuron which finds the component of the input g in a particular direction. We would now like to have a neural network learning rule for this system, which will modify the weight vector depending on the inputs which are presented to the neuron.

4.2

The Oja Principle Component Finder

A very simple form of Hebbian learning rule would be to update each weight by the product of the activations of the units at either end of the weight. For the single linear neuron (Fig. 8), this would result in a learning algorithm of the form AIUi

= qx,y

(39)

or in vector notation

A 0 = qgy. Unfort,unately, this learning algorithm alone would cause any weight to increase without bound, so some modificat,ion has to he used to prevent the weights from becoming too large. One possible solution is to limit t,he absolute vaJues that each weight 2ui can take 1461, while another is to renormalise the weight vector 0 to have unit length after each update [23]. ,4n alteruat,ive is to use a weight decay term which causes the weight vector to tend to haw unit length as the algorithm progresses, without explicitly normalising it. To see how t,liis works, consider the following weight update algorithm, due to Oja [23]:

A g = ~ ( gy w-y2) -

q(zzT w - w(OT g r T a)).

(41)

When t,he weight vector is small, the update algorithm is dominated by the first term on t,lie right hand side, which causes the weight to increase a.. for the unmodified Hebbian algorithm. However, as the weight vector increases, the second term (the ‘weight decay’

323

term) on the right hand side becomes more significant, and this tends t o keep the weight vector from becoming too large. To find the convergence conditions of the Oja algorithm, let us consider the average weight update over some number of input presentations. We shall assume the input vectors c have zero mean, and we shall also assume that the weight update factor is so small that the weight itself can be regarded as approximately constant over this number of presentations. Thus the mean update is given by

where X = gTC,u, and C, = E ( z g T )is the covariance matrix of the input data c. When the algorithm has converged, the average value of A 0 will be zero, so we have

C& = u x

(43)

i.e. the weight vector 0 is an eigenvector of the input covariance mat,rix C,. A perturbation analysis confirms that the only stable solution is for u to be the principal eigenvector of C,. To find the eventual length of 0 we simply substitute (43) into the expression for A , and we find that x = WT(C,W) = Z.T(aX) (44) i.e. provided X is non-zero, uTu,= 1 so the final weight vector has unit length. We have therefore seen that as the Oja algorithm progresses, the weight vector will converge to the normalised principal eigenvector of the input covariance matrix (or its negative) [23]. The component of the input which is extracted by this neuron, to be transmitted through its output y, is called t,he principal component of the input, and is the component with largest variance for any unit length weight vector.

4.3

Reconstruction Error

For out single-output syst,em, suppose we wish to find the best estimate 2 of the input g from the single output y = a T g . We form our reconstruction using the vector

as

follows: ?=gy

(45)

where 21 is to be adjusted to minimise the mean squared error

If we minimise 6 with respect to 14 for a given weight vector 0, we get a minimum for

E

at

324

where C, = E [ u T ]as before (assuming that g has zero mean). Our best estimate of g is then given by

where the matrix

is a projection operator, a matrix operator which has the property that Q2= Q. This means that the best estimate of the reconstruction vector &, from the output yx = I&, is 2, itself. Once this is established, it is possible to minimise E with respect to the original weight vector w. Provided the input covariance matrix C, is positive definite, this minimum occurs when the weight vector is the principal eigenvector of C,. Thus PCA minimises mean squared reconstruction error.

4.4

The Scaling Problem

Users of PCA are sometimes presented with a problem known as the scaling problem. The result of PCA, and related transforms such as KLT, is dependent on the scaling of the individual input components xi. When all of the input components come from a related source, such as light level receptors in an image processing system, then it is obvious that all the inputs should have the same scaling. However, when different inputs represent unrelated quantities, then the relative scaling which each input should be given is not so apparent. As an extreme example of this problem, consider two uncorrelated inputs which initially have equal variance. Whichever input has the largest scaling will become the principal component. While this extreme situation is unusual, the scaling problem does cause PCA to produce scaling-dependent results, which is rather unsatisfactory. Typically, this dilemma is solved by scaling each input to have the same variance as each other [47]. However, there is also a related problem which arises when multiple readings of the same quantity are available. These readings can either be averaged to form a single reading, or they can be used individually as separate inputs. If same-variance scaling is used, these two options again produce inconsistent results. Thus although PCA is used in many problem areas, these scaling problems may lead us not to trust it to give us a consistent result in an unsupervised learning system.

4.5

Information Maxmization

We have seen that the Oja neuron learns to perform a principal component analysis of its input, but that principal component analysis itself suffers from an inconsistency problem when the scaling of the input components is not well defined. In order to gain some insight to this problem, we shall apply Linsker’s Znfomax principle [21] to this situation. Consider a system with input X and output Y . Linsker’s Infomax principle states that a network should adjust itself so that the information I ( X ,Y ) transmitted to its output ’I’ about its input X should be maximised. This is equivalent to the information in the input S about the output ’I7, since Z(
325

However, if Y is a noiseless function of X, as is the case for our linear neuron

then there will be an infinite amount of information in the output Y about X, because Y represents X infinitely accurately. In order to proceed, we must assume that the input contains some noise 9 which prevents any of the input from being measured too accurately. Consider the case where the input signal g is zero mean Gaussian with covariance matrix C,, and the noise $ is also zero mean Gaussian, with covariance matrix C+= 021, so that the noise on each input component is uncorrelated with equal variance. The output of the neuron is then the weighted sum

Writing down the information in the output y about the input signal g, we get

I ( Y ; X )= ;log-

S+N N

where

s = E (IUTZl')

= WTC,u,

and T

N = E (lul

- 2 2 &I 2 ) 0 lull

Since (51) is monotonically increasing in S I N , I ( X , Y ) is maximised when u,is the principal eigenvector of C,, i.e. it is the principal component of the input. This is the same condition for minimising the mean squared reconstruction error considered above, but now we have an explicit condition on the noise on the input. The condition is that the noise on each input should be uncorrelated, and each input should have the same noise variance. The scaling problem of principal component analysis is now changed to one of guessing the noise on each of the inputs, and scaling them so that this noise is approximately equal. The standard approach of scaling all inputs so that their signal variance is equal is therefore equivalent to assuming that the signal to noise ratio of all inputs is equal [28]. Of course, the assumptions that the input signal and noise are Gaussian and zero mean are very strong, but can be relaxed somewhat if information loss is considered rather than transmitted information. However, the result in each case is the same: the Oja algorithm, which finds the principal component of the input, maximises information capacity on condition that the noise on the input components is equal.

4.6

Multi- dimensional P CA

There are a number of algorithms which extend Oja's algorithm to more than one output neuron. For these we need an output vector 2 and a weight matrix W , such that

326

If the Oja algorithm was used for each output neuron with no modification, each would find the same principal component of the input data. Some mechanism must be used to force the outputs to learn something different from each other. One possibility is to use a lateral inhibition network between the output neurons, which forces their outputs to be decorrelated [16]. An alternative is to modify the weight decay term of the Oja algorithm: this approach is used by William’s Symmetric Error Correction (SEC) algorithm (481, Oja and Karhunen’s M-output PCA algorithm (241, and Sanger’s Generalised Hebbian Algorithm (GHA) 1391. In fact, these algorithms have much in common. Although the weight vectors themselves have different algorithms, the subspace defined by the orthogonal projection

P = W(WTW)-’WT which is the subspace spanned by the weight vectors to each of the outputs, moves in exactly the same way for each of these algorithms [as].Since this subspace, rather than the weight vector itself, determines the change in information transmitted through the network, these three algorithms tend to increase the transmitted information in exactly the same way. All three lead to a set of weight vectors which spans the same space as the largest principal eigenvectors of the input covariance matrix, which is sufficient to maximise the transmitted information (under the equal noise conditions which we outlined above) [28]. The three algorithms differ only in the behaviour of the weight vectors themselves. In particular, the SEC algorithm [48] leads to weight vectors which are orthogonal and unit length, but have no particular relationship to the eigenvectors of the input covariance matrix. Oja and Karhunen’s algorithm [24] uses a Gramm-Schmidt Orthogonalisation (GSO) approach to find the principal components themselves, in order. Sanger’s algorithm (391 uses GSO in a slightly different way, but also finds the principal components in order.

4.7

Discussion

We have seen that linear neurons, with a modified form of Hebbian learning algorithm, can learn to find the principal component or principal subspace of its input data. We have also seen that this principal subspace maximises information capacity of the system, under the condition that the input components have uncorrelated, independent, equal-variance Gaussian noise on all of the components. When we perform principal component analysis in practice, we also tend to use a set of output which are decorrelated, or at the very least not highly correlated with each other. The algorithms considered here also do this, but this does not seem to be required to maximise information capacity. We should only have to find the principal subspace of the input data: correlation between the output components should be irrelevant. The puzzle here arises because we have neglected noise which may occur after the network which we are currently considering. As we have already seen in 53.2, decorrelated outputs tend to be better protected against later noise (which may be added noise or calcnlation errors) than outputs which are highly correlated [lo]. One of the authors has recently investigated combination algorithms which may be suitable with both input and

321

output noise (29, 311. It may be that real perceptual systems are able to take account of both noise sources at the same time.

Non-linear Principal Component Analysis

5 5.1

Introduction

In the previous section various algorithms were suggested so as to achieve principal component analysis on a set of input patterns and thereby allow for dimension reduction and data compression. At the same time it was pointed out in 94.5 that such analysis leads to maximisation of information capacity. All of this analysis was performed in the context of linear neurons, with 0utput.s given by (37) or (38). The statistics of the input data being used was only up to second order, whilst it is suspected that higher order statistics may be being analysed by higher layers in visual cortex, each of which has at least a quadratic output from complex cells [35]. As such, a t each layer visual cortex may be working on up to fourth order statistics. This could rapidly build up to quite high order analysis in several layers, as is clearly appropriate for an effective object recognition system.

5.2

Non-linear PCA

The extension of the Oja principal component analyser described in $4 to non-linear neurons has been performed in (451. This uses, in the case of the linear and quadratic neuron, the output rule y = w,xi W i j X i X j (53) (where the summation convention is being used, with summation on any twice repeated index) and extension of the Hebbian updat,e rule (41) to

+

AW;j = q(y/”;~j- y2W..) ‘3 (54) On averaging over the input patterns it is possible to write an extension of (42) which preserves its form. This if the two-component object R = (“) and the (N N 2 ) x w., (N N 2 ) matrix C = be introduced, where (C2)ij = E ( z , x j ) (C2 = C, of 54), ( C 3 ) i j k = E ( z ; x j z k ) , ( C 4 ) i j k e = E(zixjzkxe), then (41) and (54), with y given by (53), become no = V(CR - An) (55) where X = RTCR. The equation (55) has identical form to equation (42), but now involves up to fourth order statistics of the input patterns. It is evident that this process can be continued by adding higher order terms still into the right hand side of (53). If terms of order TI are included then the object RT is extended to RT = ( W i , W;,,. . . ,Wil..,jn),whilst C is enlarged to

+

+

(22)

c,,,

...

...

328

Equation (55) remains unchanged. Extensions may be given along the lines of 34.6 to learning eigenvectors corresponding to lower eigenvalues of C. These will be of importance in giving a better reconstructed image. However we will not discuss these in any detail here.

Pattern Non-linear Reconstruction In all of these cases the learning proceeds till R becomes the normalised eigenvector of 5.3

C with maximal eigenvalue. As in the linear case, principal component analysis may be shown to minimise the mean-squared reconstruction error, so leading again to important data compression. The reduction is not completely trivial, so we will give a brief demonstration here which extends that of 34.3 for the linear case [15]. The optimal reconstruction is expected to be a non-linear extension of (45) [34], so taking the form Py = yaGi + yay"? + . . . (56) where a is the pattern label. The value of f a can be expressed in vector form using the notation qa(YIT = (Y",ya'Yp, Y a Y v ,' . .) (57a) -T

G

=

-

(c,@,cPy ...)

(a,a3x;, a3-3kx;x;,. . .).

(57b)

In terms of (57), 2" may be written as P" = Z.?"(y)

(58)

where the dot product in (58) is over the vector indices square reconstruction error, extending (46), is E

= E(ll.- - 211*) = E(llz-

P,-y etc in (57a,b). The mean

G.?(Y)1l2)

(59)

where y is given by the non-linear expression, extending (53), as y = w,x,

+ WIJX,XJ + . . . .

If we denote combinations of indices 31 . . . j , by j , of patterns 71,. . . ,-ym by P xfj . . . xf; by x~ - then (59) may be written

Variations of

E

in (61) with respect to a,, leads to the matrix equation atLA,k =

(60)

2,and

329

where

Then a solution of (62) is

which is a non-linear extension of (47) (and reduces to it in the linear case). The minimum error for this solution may be obtained from (59) and (63) as

-1 [C(,)2 - (RTC20)/(RTCR)] 2 0

(64)

This is minimised on the original weight vector when R is chosen as the principle eigenvector of C. In fact there are more general solutions a;&= (Cn)if~J0C%) to equation (62), but they all lead to the same minimum error (64), and hence also to the principle component choice. As in the linear case, the scaling problem enters, albeit now in a nonlinear fashion. It is not as easy to resolve as in the linear case by considering additive input noise, and scaling so that input noise variances are equal. This is because it is no longer possible to write down the analogue of (51) in the non-linear case. It may be possible to give an approximation to this for small additive noise, although no definite results are available on this yet.

5.4

Non-linear Pattern Reconstruction

An important question to be resolved is as to the quantitative benefit of using higher order statistics in pattern reconstruction. This was analysed above using only the mean square error (MSE) E of equation (59). The MSE can be reduced to zero by using all of the eigenvectors in the linear case [38]. This case can only take account of C2, so cannot be expected to reproduce higher order statistics correctly. To assess these higher orders a suitable extension of the MSE must be introduced. One such is the Kullback-Liebler distance D(P,P) between the input probability distribution P ( g ) and the reconstruction distribution P ( 2 ) of $2.2.2, defined as

In the linear case one has, from (45)

P(&)=

S"(& - g ( u . g ) ) P ( gd"g )

330 Regularisation of the &function and use Of a Gaussian distribution for P(.) allows a Gaussian distribution to be obtained for P , and the result that d ( P , P ) is a minimum when 11 is the principal eigenvector of C2, as is g.The extension of (66) to be non-linear case is P ( 2 ) = /6"(- - G.Y(y(z))P(g) d"c (67)

--

where we are using the notation of the previous subsection. This approach appears to be the most natural one in the context of the differential geometric approach to estimation theory 121 t o be outlined in the next section; it has still to be pursued much further in this context. It is possible to indicate the power of the non-linear PCA approach outlined above if an extension is made to the error term (59). This modification is designed so error minimisation is achieved for all of the statistics of the input, and not just the quadratic part. In terms of the notation of s5.3 the new error function is

where xi = x,,x,, . . . x,", and 67 = (al, a,z!, a,kxfzz,. . .). On variations with respect to the free parameters of (57b),which are now u2 (where the subscripts i run over the same set of labels as in the summation in ( 6 8 ) but the indices J are multi-vectoral as described below), the natural extension of (62) is obtained as a@& _ _

The indices j-, b in (69) are now multi-vectoral, with j- = J1,. . . ,&),& = some P and m, but only denotes a single vector index, and Q&

= (O'Q)

rp), s

(kl,...

rpQ)kr 7

where

As in the case of ( 6 4 ) , the miiiimised error from (68) becomes

This has minimum when R is along the eigenvector direction with largest eigenvalue which is, exactly the principle component of C being learnt by the rule ( 5 5 ) . thus this

33 1

rule enables minimum reconstruction error (in the MSE sense) of the input patterns and their higher statistics (up to the order contained in C). The above analysis can be extended immediately to the case of learning lower components, by means of extending the single non-linear neuron (60) to a set of M of them. The resulting set of QT = (O(l),. . . ,O ( M ) )is a multi-tensor, and the learning rule, to one or other of the PCA-subspace rules, is of the form

Numerous extensions of ( 7 3 ) are possible to obtain orthogonal decompositions of the PCA subspace; they were briefly considered in 84.6, and will not be discussed here further.

5.5

Discussion

It is relevant to note here that neurons in general are non-linear, and the effect of a sigmoid non-linearity on the neuron output on determination of the principal components of the second order input statistics have already been studied [25]. In general improved stability of the PCA algorithms were found to result from this non-linearity. In this section we have not discussed this aspect of non-linear neurons in the PCA context, but focussed on the ability of such neurons, with suitably adaptive higher-order weights, to learn higher than second order statistics of their inputs. This was first presented in [45], and practical aspects will be discussed more fully in [15]. We have indicated here briefly how to achieve the non-linear extension of much of the linear PCA analysis of the previous section. MSE approach to the reconstruction problem indicates how to achieve better assessment of the importance of the higher order statistics in the pattern reconstruction process. It will be explored more detail elsewhere [15].

6 6.1

Differential Geometry of the Manifold of Networks Introduction

Any neural network of a given architecture is a parameterised form of mapping

z 4F ( u , c )= Y

(74)

from the space of inputs g to that of the outputs y, where the parameters u are a finite set of real valued quantities, usually the weights and thresholds of the separate neurons. As w varies, the set of such parameterised maps F forms a space of maps N characterising all neural networks with the given architecture. It is highly relevant to discover suitable tools for describing the intrinsic structure of N , and also the manner in which it is embedded in the space S of all mappings from a to y. Such structure is of importance in describing how training algorithms change the network along a trajectory on N , or more general

332

architecture optimisation strategies modify the network inside the more general space S of all maps. An important approach to these questions has been developed by Amari [2] in terms of differential geometric concepts associated with statistical estimation theory. This approach uses suitable parametrisations of N , so that will be considered first. We will then, in the following sub-sections discuss the appropriate differential geometry and consider neural networks in that framework.

6.2

Network Parametrizations

The simplest and most complete parametrization is for the case of stochastic neurons with n binary inputs 14 and a single output y. In that case the probability of emitting a one is a quantity which we can denote ag: proh(y = 11%)= ax

(75a)

prob(y = 01%)= 1 - a,-

(75b)

The 2“ set of quantities = {a,} give a complete description of the neuron. Such a neuron has been termed a PRAM [45] and discussed extensively in a series of papers (see [17] or [18] for a review). The PRAM has the particularly attractive features of (a) having a simple hardware realisability; (b) possessing continuously variable “connection weights” ax,which can be trained by reward learning in a hardware-realisable manner [14]; and (c) representing the stochastic response arising from noisy quanta1 synaptic transmission in living neurons. As such, PRAMS completely fill the space S,RAM of binary input and output stochastic neurons. Subspaces Nk of S can be formed by those PRAMS which only have non-zero memory contents a, for 1111 5 k , so that

Networks of PRAMS can be built [14], and are parametrised by the a ’ s of the respective PRAM components; they also will in general be subsets of .S ,,, Other subspaces of S,R,M exist, such as that formed by linear weighted summed neurons with

where 20, t are connection weights and threshold respectively. Parametrization of graded input or output neurons can be developed in a similar manner. The neurons of 54.1 are parametrised by a single vector, whilst the non-linear neuron of 55 by the multi-tensor quantity R = ( w ; ,w i j , w i j b , . . .). In general a neuron may have an input-output transform which is parametrised by any number of parameters, so that for such cases N can be infinite-dimensional. However it would be usual to have a bound on the number of adaptive parameters in a neural net, so only finite dimensional subspaces of S would arise.

333

6.3

Differential Geometry

The output of the stochastic neuron of the previous subsection can be interpreted as the random variable ru for binary input I,so that a 2"-component random variable 2: = { r g } ensues, with probability

The above expression gives a family of probability distributions for 11, co-ordinatised by of the stochastic neuron. It is possible to introduce an exponential the parameters family of co-ordinates Ox, with ax = (1 e-'=)-' (79)

+

in terms of which a dually-flat Riemannian structure can be defined. A dually flat manifold is defined in terms of two special sets of dually coupled coordinates @ and 4. Linear curves in @ or in t are geodesics which are dual to each other. This duality is defined by the tangent vectors gU along - along the co-ordinates 8% and the aprwith the condition &.g2 = 6; The structure of the manifold is determined by potential functions $(@) and 4(8) with

8" = (6/6au)4(a) = (6/68")$(8)

(8la) (81b)

An important result [l]is that a dually flat manifold admits a unique invariant divergent measure D(P , Q) between any two points P, Q with value

If the manifold is that of probability distributions then this divergence is identical with the Kullback-Liebler divergence which was mentioned briefly in the previous section. The divergence is itself an extension of the Euclidean distance to the case of the metric (82). It has the useful generalised Pythagorean property

if the @geodesicconnecting P and Q is orthogonal at Q to the dual g-geodesic connecting Q and R. Moreover it has the further valuable projection property that the point Q p in a sub-manifold M of S which minimises the divergence to any point P in S is given by the t-geodesic projection of P onto M .

334

6.4

Geometry of the Neuron Manifold

In the case of the PRAM of 55.2, the 0' and a, co-ordinates were already defined by equation (75) and (79). The corresponding metric and potential functions of 55.2 are then (21

$(e) =

c [8. U -

+ l n ( l + e-")]

(84a)

where guv -_ is the Fisher information metric and its inverse. For N observations, the estimation error of an output ru is given by

E((+% - .J2)

1 = -a (1 - a,) N L

(85)

The invariant divergence between two points P ( a ) and Q(Q) is given simply by the Kullback-Liebler information distance

It is possible to use the geometric theorems of the previous subsection to show 121 how the approximation error D ( P * ,p k ) , for a neuron p k of order k, so with a, = 0 for 1341 > k , can be decomposed into a sum of distances between approximations to neurons of successively higher orders

c qp;,,,

n-1

D ( P * ,Pk)=

p;)

+ D ( S ,Pk)

(87)

m=k

with P; = P'. The above approach has been applied to analyse learning in the Boltzman machine [3] and to obtain [2] the error between P and the maximum likelihood estimate P * , obtained when T is the number of training examples and d the number of free parameters (2" in this case), to be D( P, P') = (d/2T) This is an important result on generalisation, and is at the basis of the "Rule of Thumb" that there must be an order of magnitude more training patterns than free parameters in order to guarantee a generalisation error of less than 5%. More detailed analysis have been given of learning in particular situations 131, to which attention is directed. It is also possible to extend the structure to graded inputs and/or outputs, when the spaces of networks becomes an infinite-dimensional function space. The associated expression (78) becomes the weighted log likelihood

335

with p ( c ) the input probability distribution, a(c)is the output for c, and T ( E ) is the random variable with prob(r(c) = 1) = a@) (90) The above framework can evidently be extended to more than one output in an evident manner.

7 Discussion It should be clear from the foregoing that information-theoretic approaches are making important progress in analysing both artificial neural networks and possible processing strategies in early vision (at both retinal and visual cortical levels). This is particularly true for understanding preprocessing, in terms of predictive coding in the retina and PCA and decorrelation processing in early visual cortex. However there is a difficult question which must be considered before we can be satisfied with the explanations given in this chapter of neurobiological information processing, which arises because living neurons and their nets are far more complex than those considered in the present analyses. Even if we are including higher orders of non-linearity in the response function in 55 there is no inclusion of complex temporal features as might arise from channel openings and closings (alpha functions) or from neuronal geometry, or of the details of stochastic synaptic transmission, or of the many other features of living neurons. The question is therefore if the explanations of retinal and early visual cortical processing, given in terms of informationtheoretic principles, and on the basis of non-physiological realistic learning rules, will still be valid when more realistic neurons and nets are used? Moreover learning rules themselves must have a biological reality. Will there be realistic rules which will lead to the desired connection weights? It is clear that some of the discussion in the earlier sections contravenes known biological features. Thus the anti-Hebbian learning rules of (2.12) of ref [34], and of equation (25) here, use adaptivity of inhibitory synapses, which is a feature with no experimental validation. It is possible to avoid this problem by using fixed inhibitory feedback but variable lateral excitatory connections on the inhibitory neurons. This corresponds to having the adaptive connection weight matrix V for the lateral excitatory connections in fig 3(b), but the fixed inhibitory connection matrix-C for the feedback. Thus the equation (23) becomes y =x -

cz,

and (24) y = (1

2

= VTy

+ CVT)-'s

Then the learning algorithm of equations (25) or (26) will again give uncorrelated output when learning has been completed, but now only the excitatory synapses have been trained. The inhibitory interneurons will still perform decorrelation on the outputs. However care would have to be taken to ensure that the weights v,j do not go negative, so that equation (25) would have to be modified so as to be clipped when the w;j become zero. The resultant level of decorrelation has yet to be analysed.

336 Another aspect where biological realism could be introduced is associated with PCA in both sections 4 and 5. This in particular involves the presence of the decay term proportional to the square of the output in (41) or (54). Both of these expressions involve a global assessment of output, and its use in a local manner at each synapse. To avoid this one could take a decay term depending only on the weight value at the synapse [36], as

Awij = q ( ~ , y j w;)

(93)

This has been shown to result in asymptotic weight values proportional to a high root of the principal eigenvector components of the correlation function; its value for PCA is not yet clarified. These modifications only go a very short way towards responding to the equation raised earlier. There is no reason, however, why approximations to living neurons cannot be used to advance understanding in this area; the use of more realistic neurons will ultimately have to be faced up to. Use of non-trivial temporal features [12] and other properties are increasingly being modelled, so that this aspect of the program can be started. Finally we note there are many directions for future work. We have only scratched the surface of this very important approach to neural networks, leaving out numerous avenues being actively followed by others. However we hope that we have given in this chapter enough of a survey to indicate the nature of the approach.

Acknowledgements One of the authors (MDP) is supported by a Temporary Lectureship from the Academic Initiative of the University of London. The other author (JGT) would like to thank DRA for financial support to allow part of this work to be done.

References [l] S.-I. Amari. Differential geometry of a parametric family of invertible linear systems-

Riemannian metric, dual affine connections and divergence. Mathematical Systems Theory, 20:53-82, 1987. [2] S.-I. Amari. Dualistic geometry of the manifold of higher-order neurons. Neural Networks, 4:443-451, 1991. [3] S.-I. Amari, I<. Kwata, and H. Nagaoka. Geometry of Boltzmann machine manifolds. Mathematical Engineering Technical Report 90-19, Univeristy of Tokyo, Faculty of Engineering, 1990. [4] J. J. Atick and A. N. Redlich. Quantitative tests of a theory of retinal processing: Contrast sensitivity curves. Technical Report IASSNS-HEP-90/51, NYU-NN-90/2, School of Natural Sciences, Institute for Advanced Study, Princeton; Center for Neural Science, New York University, 1990.

337

[5] J. J. Atick and A. N. Redlich. Towards a theory of early visual processing. Neural Cornputation, 2:308-320, 1990. (61 J. J. Atick and A. N. Redlich. What does the retina know about natural scenes? Neural Computation, 4:196-210, 1992. [7] J. J. Atick and A. N. Redlich. Convergent algorithm for sensory receptive field development. Neural Computation, 5:45-60, 1993. (81 F. Attneave. Some informational aspects of visual perception. Psychological review, 61:183-193, 1954. [9] H. B. Barlow. Three points about lateral inhibition. In W. Rosenblith, editor, Sensory Communication, pages 782-786. MIT Press, 1961.

[lo] H. B. Barlow and P. FoldiAk. Adaptation and decorrelation in the cortex. In The Computing Neuron, pages 54-72. Addison-Wesley, Wokingham, England, 1989. [ll] S. Becker and G. E. Hinton. Spatial coherence as an internal teacher for a neural network. Technical Report CRG-TR-89-7, Department of Computer Science, University of Toronto, Dec. 1989. [12] P. C. Bressloff and J. G. Taylor. Spatio-temporal pattern processing in a compartmental model neuron. Physics Review E, 1993. (in press). [13] J. S. Bridle. Probabilistic interpretation of feedfonvard classification network outputs, with relationships to statistical pattern recognition. In F. F. SouliC and J. Herault, editors, Neurocomputing - Algwithms, Architectures and Applacations, pages 227236, Berlin, 1990. Springer-Verlag. (141 T. G. Clarkson, D. Gorse, and J. G. Taylor. Hardware realisable models of neural processing. In Proceedings of the IEE F m t International Conference on Artificial Neural Networks, pages 242-246, 1989. I151 S. Coombes, R. Petersen, J. G. Taylor, and S. Wright. Non-linear principal component analysis and pattern reconstruction. In preparation. (161 P. Foldikk. Adaptive network for optimal linear feature extraction. In Proceedings of the International Joint Conference on Neural Networks, IJCNN-89, pages 401-405, Washington DC, 18-22 June 1989. [17] D. Gorse and J. G. Taylor. On the equivalence and properties of noisy neural and probabilistic RAM nets. Physics Letters A , 131:326-332, 1988. (181 D. Gorse and J. G. Taylor. A review of the theory of PRAMS. In Proceedings of the Weightless Neural Networks Conference, York, UK, Apr. 1993. (To appear). [19] D. 0. Hebb. The Organization of Behavior. Wiley, New York, 1949.

338

[20] D. H. Hubel and T. N. Wiesel. Receptive fields of single neurones in the cat’s striate cortex. Journal of Physiology, 148:574-591, 1959. (211 R. Linsker. Self-organization in a perceptual network. IEEE Computer, 21(3):105117, Mar. 1988. [22] R. Linsker. Local synaptic learning rules suffice to maximize mutual information in a linear network. Neural Computation, 4:691-702, 1992. [23] E. Oja. A simplified neuron model as a principal component analyser. Journal of Mathematical Biology, 15267-273, 1982. [24] E. Oja and J. Karhunen. On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix. Journal of Mathematical Analysis and Applications, 106:69-84, 1985. [25] E. Oja, H. Ogawa, and J. Wanguiwattana. Learning in nonlinear constrained Hebbian networks. In T. Kohonen et al., editors, Artificial Neuml Networks, pages 385-390. Elsevier, 1991. [26] A. Papoulis. Probability, Random Variables and Stochastic Processes. McGraw-Hill, second edition, 1984. (271 B. A. Pearlmutter and G. E. Hinton. G-maximization: An unsupervised learning procedure for discovering regularities. In J. S. Denker, editor, Proceedings ef Neuml Networks for Computing, pages 333-338. American Institute of Physics, 1986. [28] M. D. Plumbley. On information theory and unsupervised neural networks. Technical

Report CUED/F-INFENG/TR.78, Cambridge University Engineering Department, Cambridge, UK, 1991. [29] M. D. Plumbley. Approximating optimal information transmission using local Hebbian algorithms in a double feedback loop. In Proceedings of the International Conference on Artificial Neural Networks, ICANN’93, Amsterdam, Sept. 1993. (To appear). [30] M. D. Plumbley. Efficient information transfer and anti-Hebbian neural networks. Neural Networks, 1993. (in press). [31] M. D. Plumbley. A Hebbian/anti-Hebbian network which optimizes information capacity by orthonormalizing the principal subspace. In Proceedings of the IEE Artificial Neural Networks Conference, ANN-93, Brighton, UK, May 1993. (To appear).

[32] M. D. Plumbley and F. Fallside. An information-theoretic approach to unsupervised connectionist models. In D. Touretzky, G. Hinton, and T. Sejnowski, editors, Proceedings of the 1988 Connectionist Models Summer School, pages 239-245, San Mateo, CA., 1988. Morgan-Kaufmann.

339

[33] M. D. Plumbley and F. Fallside. The effect of receptor signal-to-noise levels on optimal filtering in a sensory system. In Proceedings of the Internation Conference on Acoustics, Speech, and Signal Processing. ICASSP-91, volume 4, pages 2321-2324, Toronto, Canada, May 1991. [34] T. Poggio. On optimal nonlinear associative recall. Biol. Cybernetics, 19:201-209, 1975.

[35) D. A. Pollen, J . P. G a s h , and L. D. Jacobson. Physiological constraints on models of vision. In R. M. J. Cotterill, editor, Models of Brain Function, pages 115-136. Cambridge University Press, 1989.

[36] H. Riedel and D. Schild. The dynamics of Hebbian synapses can be stabilized by a nonlinear decay term. Neural Networks, 5:459-463, 1992. (371 D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning internal representations by error propagation. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Vol. 1: Foundations. Bradford BooksfMIT Press, Cambridge, MA, 1986. [38] T. D. Sanger. Optimal unsupervised learning in a single-layer linear feedforward neural network. Neural Networks, 2:459-473, 1989. [39] T. D. Sanger. An optimality principle for unsupervised learning. In D. S. Touretzky, editor, Advances i n Neural Information Processing Systems 1. pages 11-19. Morgan Kaufmann, San Mateo, CA, 1989. [40] C. E. Shannon. A mathematical theory of communication. Bell System Technical Journal, 27:379-423,623456, 1948. [41] C. E. Shannon. Communication in the presence of noise. Proceedings of the IRE, 37:lO-21, 1949. [42] C. E. Shannon. Prediction and entropy of printed english. Bell System Technical Journal, 30:50-64, 1951. [43] S. A. Solla, E. Levin, and M. Fleisher. Accelerated learning in layered neural networks. Complex Systems, 2, 1988. (441 M. V. Srinivasan, S. B. Laughlin, and A. Dubs. Predictive coding; a fresh view of inhibition in the retina. Proceedings of the Royal Society of London, Series B, 216:427-459,1982. [45] J . G. Taylor and S. Coombes. Learning higher order correlations. Neural Networks, 6(3):423-428, 1993. [46] C. von der Malsburg. Self-organization of orientation sensitive cells in the striate cortex. Kybernetik, 14:85-100, 1973.

340

(471 S. Watanabe. Pattern Recognition: Human and Mechanical. John Wiley & Sons, New York, 1985. [48] R. J. Williams. Feature discovery through error-correction learning. ICS Report 8501, University of California, San Diego, 1985. [49] R. S. Zemel and G. E. Hinton. Discovering viewpoint-invariant relationships that characterize objects. In Advances in Neural Information Processing Systems 3, San Mateo, CA, 1991. Morgan Kaufmann.

Mathematical Approaches to Neural Networks

J.G.Taylor (Editor) 0 1993 Elsevier Science Publishers B.V. All rights reserved.

34 1

Mathematical Analysis of a Competitive Network for Attention J. G. Taylor and F. N. Alavi Centre for Neural Networks, King’s College London, The Strand, London WC2R 2LS, United Kingdom

Abstract A neurobiologically based model of a competitive network is analysed mathematically. The network is shown to possess the property of sustaining global competition between inputs. Various qualitative aspects of the network activity are supported by simulations.

1

The Nature of the problem

Attention requires some sort of global competition to be occuring amongst the choices of objects to attend to in the external world or in the internal signals from the body. This competitive aspect has already been noted as a fundamental constituent of many papers on the subject, both from the general fields of psychology and physiology and from the more specific aspect of connectionist modelling. However, none of these references, and the many more using competitive nets to explain other aspects of neurobiological processing, appear to have any strong foundation in neurobiological fact. A non-trivial proportion of neurons in cortex, suggested at about 25%, are inhibitory, but long-range connections between cortical areas appear to be excitatory (Douglas and Martin 1991), although with feedback inhibition coming from local lateral inhibition. There is good evidence for inhibitory effects in orientation sensitivity in early cortex (Martin 1988) but this has at most been suggested as arising from half a hypercolumn distance, or about 0.5 mm, and giving fine-tuning of earlier sensitivity (Wortgotter et al. 1991). Another place in which competition may be searched for is in the thalamus. This is composed of numerous nuclei acting in some manner as way-stations for input going to primary cortical areas (vision through the lateral geniculate nucleus LGN, for example) and as transit stations for activity in later cortical areas. There is also a considerable amount of input to thalamus from the mid-brain and the systems in the cortex denoted as the limbic systems and occupying the rim around the cortex (including at least the

342

hippocampus, amygdala, hypothalamus and cingulate gyrus). There are inhibitory interneurons in the thalamus, but very little in the way of lateral connections between different thalamic nuclei. Even in a given nucleus, any inhibitory interneurons only have short-ranged ramifications, nor is there little in the way of long-range connections between the excitatory relay cells. It appears, therefore, that the absence of any long-range inhibitory connections leading to the production of competition between activity in the numerous areas (occipital parietal, temporal and frontal lobe) involved in guiding the various aspects of attention (Posner and Petersen 1990) requires that one search elsewhere than in the cortex. A particularly desirable feature of the competitive mechanism would be that it leads not only to efficient competition but also to global guidance. By this we mean the ability of a winning input to a portion of cortex to control activity in other parts of cortex at the same (or in very closely subsequent) time. This latter activity which loses the competition is not destroyed, but is allowed to exist only in certain forms, guided by the winning input. Global guidance is not a standard property of WTA-type networks'. Indeed it appears to be very different and not compatible with any known WTA architecture. One possible class of neural networks able to implement global guidance to some degree is that of nets with spatial instability. Such nets have been known since the time of the work on visual hallucinations (Ermentrout and Cowan 1978) add from the wealth of examples of instability and pattern formation in reaction-diffusion equations (Murray 1989). We present here results on a particular brain network, the nucleus reticularis thalami (NRT), which we propose functions by means of its special lateral inhibitory connectivity so as to achieve global guidance as well as a more local competitive form of action. The manner in which such global guidance can be used in attentive processing is still to be analysed.

1.1

The Nucleus Reticularis of the Thalamus (NRT)

The NRT, present in all mammals, is a thin, curved sheet of cells so situated between thalamic relay cells and their cortical target sites as to be highly suggestive of its possible controlling influence on cortical input and activity. It is pierced by thalamwcortical and returning cortico-thalamic fibres in a roughly topographic fashion, especially for inputs to and from primary sensory cortical areas. The main organisational principle (Jones 1975) is that the NRT can be considered as a series of overlapping sectors, each related to a particular dorsal thalamic nucleus (or nuclei). The axons penetrating NRT give off excitatory collaterals to it, whilst NRT cells themselves only feed back inhibitorily in a roughly topographic fashion to the thalamic relay cells from which they have received 'WTA = Winner Takes All

343

their collaterals. The main NRT cell neurotransmitter is GABA, which is a well-known inhibitor. The NRT structure eminently qualifies it to be some sort of integrative filter modulating thalamic and cortical activity, the filter itself controlled by cortical, mid-brain and some brain stem structures (which also have inputs to NRT). That NRT performs a control function has been remarked on briefly in numerous papers. For example, as noted by Schiebel (1980): "Situated like a thin nuclear sheet draped over the lateral and anterior surfaces of the thalamus, it has been likened to the screen grid interposed between cathode and anode in the triode or pentode vacuum tube". There are various features of the detailed circuitry and functionality which must be properly accounted for in any serious modelling:

1. the nature of the inter-connectivity on the NRT sheet is itself species-specific. Thus in the rat, fine structure analysis shows only ax-dendritic synapses of presumed excitatory or inhibitory type (Ohara and Lieberman 1985), but that of the cat and monkey have very clear dendro-dendritic synapses (Ohara 1988; Deschknes et al. 1989), some of these even being reciprocal (Deschknes et al. 1989). It should be added that these dendro-dendritic synapses occur near cell bodies, and not on their distal dendrites.

2. the exact nature of the inhibitory effect of the GABA-rgic

NRT cells is unclear. Thus evidence has been presented (McCormick and Prince 1986; Spreafico et al. 1988) that local application of GABA to NRT neurons causes depolarisation of cell membrane rather than hyperpolarisation. Direct evidence of lateral inhibition of NRT neurons on each other was earlier provided by Ahlskn and Lindstriim (1982). Moreover, the equilibrium potential Ecl of the chloride conductance (the ion channel by which GABA influences the membrane potential) is about -65mV (McCormick and Prince 1986) whilst the average membrane potential is about -56mV (Avanzini et al. 1989). Thus the effect of GABA on neurons at or above their resting potential will be expected to be inhibitory. It is interesting to note that this effect will become excitatory if the membrane potential goes below Eel. This may be a useful control mechanism to prevent NRT neurons becoming so hyperpolarised as to be in their bursting state. Throughout this paper we will assume the action of GABA on NRT is inhibitory, and that the neurons have membrane potentials always above E a (although this latter will not be explicitly discussed in the modelling). The effect of GABA-rgic feedback in thalamic relay cells and interneurons will be considered later.

3. the continuous nature of the NRT sheet is reasonably well supported by anatomical

344

and cytoarchitectonic evidence, except for that part of it adjacent to the visual input (through the lateral geniculate nucleus, LGN) termed the perigeniculate nucleus (PGN). Presently, the weight of opinion is in favour of complete connectivity between PGN and NRT. 4. there are at least two modes of action of thalamic relay and NRT neurons:

(i) relay-like behaviour, defined by tonic firing in response to inputs. (ii) a phasic bursting discharge behaviour, with the ability to maintain rhythmic burst discharges at about 6 to 8 Hz. There is great relevance of mode (ii) in thalamic and cerebral spindling (sequences of rhythmic bursting) activity in sleep and in the sleep-wake transition (Avanzini et al. 1989). Since we are mainly interested here in the waking state, only mode (i) of the NRT (and thalamic) neurons will be considered explicitly.

5. there is an important species-specific differencein the NRT feedback to thalamus. In the rat, only LGN has inhibitory interneurons, but other thalamic nuclei with output to NRT are known to possess very few non-relay cells (Barbaresi et al. 1986; Harris and Hendrickson 1987). In the cat and monkey, however, such interneurons are apparently widespread. Inhibitory NRT feedback on these latter interneurons was used as an important part of the model of Steriade, Domich and Oakson (1986) and later by La Berge (1990). This model assumed disinhibition of interneuronal input control by inhibitory NRT feedback. However, in rat, without such interneurons, one might expect NRT inhibition to feed back directly on to input thalamic relay cells, so having just the opposite effect on thalamic inputs. We will discuss this problem when we turn to the details of our model, and in particular consider it in the context of stability. 6. there are also interesting features of the global wiring diagram in which NRT is concerned. Thus NRT is claimed not to be connected at all to the anterior thalamic nuclei in cats (Jones 1975; Pard et al. 1987), but there is apparently such connectivity in rats (Ohara and Lieberman 1985). The question of how important NRT inputs are to conscious processing will be discussed later in the paper. 7. there is (Spreafico et al. 1991) a clear change of cellular type in NRT from cells with round dendritic fields (termed R-type), of about 200 pm across, at the anterior pole to more elongated large fusiform cells (F-type) moving posteriorly, to even more elongated small fusiform cells (f-type) on the medial and lateral borders of the sheet. F and f-type cells have dendrites running either vertically or horizontally

345

[ I Property of NRT 1 1. 1 dendrdendritic synapses .

-

2. inhibitory interneurons in

I Rat I No

No non-sensory thalamic nuclei 3. inhibitory interneurons in LGN Yes 4. NRT connected to anterior Yes thalamic nuclei

Cat, Primate Yes Yes Yes No

Table 1: Species differences between various structural properties of NRT. in the plane of the NRT sheet (the elongated dendritic fields of f-type usually being horizontal), the f-type extending up to 300400 pm in length from the cell body. The f-cells seem to be located especially in the region of the NRT related to sensory thalamic nuclei, and receive afferents from sensory cortex.

8. it has also been discovered in the rabbit (Montero et al. 1977; Crabtree and Killackey 1989; Crabtree 1992) and in the cat (Crabtree and Killackey 1989; Crabtree 1989, 1991, 1992) that inputs from local regions of cortex lie parallel to the plane of the nucleus creating maps perpendicular to the reticular sheet. Thus correct modelling of NRT should take account of this three-dimensional character of the nucleus. However, our modelling of the cortex to be presented herewith is only at the level of considering it as a tw-dimensional sheet; to be consistent, NRT can therefore only be regarded as two-dimensional. It is hoped that the NRT model presented here will be extended to a three-dimensional structure related to the cortical structure in the appropriate manner (Crabtree 1992). The most important aspects of these features of NRT are summarised in Table 1.

1.2

Neural Modelling

In order to deduce properties of neural models it is usual to do so either by general arguments or, alternatively and more precisely, to simulate them using simplified models of the neurons which they contain. The latter approach becomes difficult if there are many neurons and/or many coupled nets. Since we wish to attempt to consider both of these latter cases we appear to be forced to attempt to use the more general method. We will try to be more precise, however, by using the techniques of dynamical systems theory to allow us to deduce qualitative results for a class of models. This is a method which has now become well established and for which there are numerous texts and reviews. The main concepts are attractors (fixed points, cycles, strange attractors) and stability (bifurcations of various types). These have been discussed in the cortical context in a

346

general fashion in Ermentrout and Cowan (1978), and many more discussions are now appearing, which are too numerous to mention. In specific cases quantitative results are being obtained which permit even more precise descriptions of various biological neural processes. The main model of the neuron we will use is the leaky integrator neuron, with output, describing its mean firing rate, given by a suitable sigmoidal function of the membrane potential. The input is a linear sum of the outputs of other neurons. This is now standard in the field of artificial neural networks, and has proved increasingly effective in many industrial applications. The model is obviously an enormous simpzfication of any biologically realistic neuron. However, we argue in support of simplicity on two grounds. Firstly, general features of simulations of nets of highly complex neurons has been duplicated by using the simple neurons we are advocating (Lijenstrom 1991). Secondly, if it is possible to achieve insight into basic principles for new paradigms of information processing that might be used by the brain in terms of such simple model neurons, then it could give a valuable guidance to further research. One would then have to ascertain if the use of biologically more realistic neurons would make such processing as, if not more than, effective. It also might help in developing further new paradigms for ever more global information processing, again with the purpose of guiding understanding of additional styles of brain processing. These new paradigms could occur on using more global wiring diagrams to understand the functionality of an even larger range of brain regions than those under the initial detailed scrutiny, but involved with these latter in an important manner. We will describe such extensions later. Part of the important advances being made by artificial neural networks is in their ability to learn any suitably smooth function with simple sigmoidally responding neurons and with a one hidden layer feedforward architecture (Hornik, Stinchcombe and White 1989). This aspect of neural networks allows an adaptive feedforward net to be arbitrarily flexible. On the face of it, such powers would seem to make easier the problem of modelling attention. However, such a method appears difficult to make effective, since the criteria for an attentional net are not simple to specify. One might claim that a single competitive net will achieve effective results (Koch and Ullmann 1985). However, it is well known (Posner and Petersen 1990) that attention has several stages, containing at least the steps of disengagement, movement and reengagement. There is also the difference between exogenous and endogenous attention. Several nets therefore must be involved in the overall action of attending. This makes the desired input-output transforms of each net less obvious. Moreover, it is likely that not until after being attended to does recognition and memory storage of an object occur. Thus the on-going net learning should not

347

take place until after attention. Adaptive processes may be necessary to develop suitable feature detectors for preprocessing (Linsker 1988) but that need not be regarded as part of the activity we must consider most specifically during actual attentional processing; the study of the development of feature detectors has not led to any perceptible insight into attention. In the face of these difficulties, we will use the hints which can be gleaned from lesion studies and gross circuit diagrams (Posner and Petersen 1990) and from the details of the micro-neuroanatomy we have presented above, and the neurophysiologyof the appropriate neural circuits, as contained, for example, in Steriade et al. (1986) and Steriade et al. (1991).

2

TheModel

In order to carry out a competition between several neurons, it is necessary to have a mechanism for enhancing the activity of the most active one (or ones) in comparison to the remainder. Such a process could be achieved solely by increasing the winnning neurons’ activity over that of the remainder. However, such augmentation would lead to the danger of an epileptic-type of response ensuing, brought about by runaway feedback excitation. To avoid this, some measure of lateral inhibition appears to be essential, and has always been present in computational models of neuron processing. The standard mechanism for this has been a Mexican hat form for the dependence of the lateral connection weights, with a short-ranged excitation giving way at longer distances to inhibition. The former enhances activity in a local region, the latter achieves the reduction of lesser, distant, activity. It was argued earlier that the competition in neural networks in the brain necessary to implement attention can only arise from long-ranged inhibition. This particularly seems needed to achieve the focussing of attention on a single object. Moreover, the inability to attend to more than one single theme at a time, seems only to emphasise the property of attention as a very narrow filter. However, since various features of an object may be attended to at one and the same time, it would seem that the input to the attentional filter must come from different processing areas. Such an aspect is an intrinsic part, for example, of the feature integration theory of Treisman (Treisman and Gelade 1980). Moreover, different objects are expected to be represented in different parts of associative areas. The inhibition between different object representations competing for attention is then expected to require long-ranged inhibitory connections.

348 It is difficult to discover such long-ranged inhibitory effects in cortex or thalamus, whilst thc general properties of a lateral inhibitory sheet of neurons, the NRT, were presented in the previous section; NRT seems a natural candidate to fit our requirements. It is the NRT which we posit achieves the long-ranged lateral inhibition needed by attention. This is not a new hypothesis, at least as far as the involvement in NRT as a gate for controlling the input to cortex from thalamus is concerned. A quotation was already given from Schiebel (1980), and there are a host of others in the literature of neuroanatomy and neurophysiology which argue the same feature (Yingling and Skinner 1977). What is claimed to be new here is the possibility that the NRT functions not only as a gate but also as a global gate. In other words, by virtue of the very nature of the structure and action of NRT, astride the main thalamo-cortical and corticc-thalamic pathways, NRT acts so as to: (1) allow only a single object at a time to be processed by crucial brain structures during directed attention (2) help change the particular focus of attention from one to another of a set of inputs, in particular the reengage part of the directed attention process (Posner and Petersen 1990) (3) allow for rapid response to significant or novel objects outside the directed attention paradigm (in the exogenous or ‘bottom-up’ attentional phase). To show how these three different processing activities could be achieved by means of NRT may be regarded as premature. The recent discovertis of Crabtree et al. (1989, 1991, 1992) indicate that the NRT has a more complex structure then had originally been contemplated. The studies of Llinas and Ribary (1991) and many others indicate that the nature of the thalamic relay cell transmission, both on anatomical and physiological grounds, has still to be properly understood. In the remainder of this section, we will present an extremely simplified version of our model of the thalmus-NRT-cortex (TNC) complex. We expect that this still contains the basic principles of the more complete living system. Our stripped down model has to achieve three processing activities, the first a simpler version of the second: (a) to enable competition between different thalamic inputs or cortical activities to be carried out, (b) to enable global competition to arise between all thalamic inputs and cortical activities, (c) to respond so that a novel input rapidly wins the competition. Our first simplification will neglect cortical activity altogether, so that the basic structure we are considering is that of Figure 1. Input Ij enters the thalamic relay cells T,, whose excitatory output Oj feeds to the corresponding NRT cell Nj vertically above

349

Tj. The cell Nj sends excitatory activity solely back to Tj, but also sends inhibitory signals to other nearby cells in NRT. The NRT cells thereby violate Dales's Law (that all outputs from a given neuron have the same sign), but that can easily be corrected by including inhibitory interneurons in the thalamus for which the effect of NRT is disinhibition of thalamic cells; the inclusion of such interneurons is done in the more complete model in the next section, but will be neglected here initially for reasons of simplicity. That the interneurons do lead to a net disinhibitory effect of NRT on thalamus is supported be experimental evidence (Steriade et al. 1986). The neglect of lateral connections of thalamus to NRT or NRT feedback to thalamus may be justified on the basis that the neurons of Figure 1 are regarded as corresponding to the averaged activity of groups of neurons in thalamus or NRT. Alternatively, it may be claimed that lateral connections, say with a Gaussian spread, will tend to smear activity out somewhat; the model of Figure 1 contains the ideal, sharper, mode of information processing which may be satisfactory for a simplified set of inputs. Let us suppose that all cells have roughly the same thresholds and connection strengths in Figure 1. We may then argue that at the position of maximum of input I, the resultant excitation of the NRT cell N will be greater than that of the other neighbouring NRT cells. After transient activity, the membrane potential of the thalamic cells will stabilise at values for which that of the cell at the maximum of the input will be larger than those of the others by a margin greater than that by which the maximum input exceeds the lower, nearby, inputs. This can be worked out mathematically in the simple case of linear neurons; for suitable excitatory symmetric connection weights between thalamus and NRT, the magnification coefficient m, equal to the ratio of activity in T2to that in TI (for only two cells in Figure 1) is equal to (11- 1)/(1 - I),where Z is the ratio Il/ZZ, and I = (1 - a2 - A B ) / a A B ,where A and B are the connection weights from the thalamus to NRT and NRT to thalamus, respectively, and a is the lateral inhibitory weights from 1 to 2 or vice versa. For example, for A = B = a = f , m takes the value (41 - 1)/(4 - I),which gets arbitrarily large as I approaches 4 '(with saturation effects, neglected in the linear analysis, ultimately putting an upper bound on m). We conclude that symmetric inhibitory lateral connections between NRT cells, with excitatory NRT-thalamic feedback, sets up competition between thalamic inputs. There is enhancement of the direct input by the feedback, together with a reduced inhibitory contribution from nearby input. In the above example, with the numbers quoted, the and fZz - $11 respectively. potentials V T ~ V, T ~of the thalamic cells 1 and 2 are !I1 The magnification factor increases the excitatory input on TI relative to that on T2, whilst the inhibitory contributions will be larger at the weaker input than the stronger

350 one. The above features, of strengthening the stronger input, and larger inhibition from that input being fed back onto the positions of weaker input, can be seen in the more general case of linear neurons with general connection matrices A (NRT to thalamus), B (thalamus to NRT) and -C (NRT to NRT). The static membrane potential equations from Figure 1, are VT = I A .VT, (1)

+

VN

=B

'VT - C 'VN,

(2)

where VT and V N are the vectors of membrane potentials for thalamic and NRT cells with solution VT = [l - A(l C)-'B]-'I, (3) I ~ ( i C)-'BI + .. .,

+

+

+

+

where the third expression in (3) is valid for small A or B and (1 C)-' exists for positive definite C. For A and B close to diagonal matrices, we can discern in (3) the same effect as in the simpler model, with enhancement of inputs by roughly the factor [det(l C)]-', and inhibition by neighbouring inputs with coefficients of order the elements of the offdiagonal elements of adj(1 C) times [det(l C)]-' with further factors equal to the diagonal elements of A and B. Thus again the architecture of Figure 1 will lead to competition on inputs.

+

+

2.1

+

The Third Simplified Model

It is possible to enhance the sensitivity of the system by introducing non-linear neurons, both by means of the gain factor and more particularly by a thresholding effect. This is seen clearly for the thalamic cells, since the lateral inhibitory spread on NRT could reduce membrane potentials on those cells for which the input is non-maximal below their effective thresholds, and so cut off their outputs completely. This can be seen in detail for the case of the two sets of NRT-thalamus neurons treated earlier in the linear case, where the extension of (1) and (2) is the system (for no lateral spread of thalamusNRT connections) given by:

VTZ

- Af

B ( m ( f ( V T 2 )

- f(VT1))) = 12,

(5)

with f a non-linear sigmoid function with effective threshold t (so f(z)< 0 for z < t , and f ( m ) = +1, f ( - w ) = -1). The equations (4) and (5) are seen to have the consistent

35 1 solutions

-

1,

f ( ~ ~ 1 ) ~ ( v T Z )

(h

-

-1, provided that

2B

(m))1,

f

(12

+ Af ( 1- 5 -1, )) a2

hl

It is clear that there is a range of values of the parameters A, B, a,t , and inputs Il > 4 , for which (6) has a solution, in particular by choosing A and B large enough or a close enough to 1. These features are expected to extend to more general non-linear competitive implementation of the model.

2.2

A Simplified Global Competitive Model

The features of the model of Figure 1 we have explored so far lead to competition between thalamic neurons for which there is lateral inhibitory connection between corresponding cells on the NRT net. However, NRT cells have not been observed to have connections across the whole NRT sheet, in spite of their extensive dendrite trees (especially in the rostra1 part of NRT). Thus the range over which this competition can effectively take place will be limited. The localised properties of such competition have already been analysed over a decade ago (Ermentrout and Cowan 1978), in terms of the even more simplified model of a single sheet with excitatory input and a Mexican hat style of lateral coupling; this model results from that of Figure 1 by collapsing the thalamic cells onto the corresponding NRT ones, and further violating Dale’s Law. The range over which the competition can occur is of order that of the range of connections on the sheet. This feature of the sheet is not satisfactory for global control of the form of global guidance described earlier. It would seem that without considerable extensions of the lateral connections on NRT (or a small enough sheet, as might be the case in the rat) global guidance could not arise, and attentional control would be weak. A number of localised activities could then be supported on cortex, in disagreement to the ability to attend only to a single object at a time. The important feature of NRT, noted at least for more advanced mammals, above the rat, in Table 1, is the presence of dendrc-dendritic synapses. These latter allow the NRT to be considered as a totally connected net even without the presence of axon collaterals of the form considered in Figure 1. Such dendro-dendritic synapses arise between horizontal cells in the outer plexiform layer of the retina (Dowling 1987), in the form of electrical gap junctions. They were modelled as linear resistors in the mathematical model of the retina in Taylor (1990). On NRT, the dendro-dendritic synapses appear to be chemical ones (as high-magnification electromicrographs show vesicles on or on both sides of the synapses). Such synapses need to be modelled in a non-linear fashion. In general, the dendro-dendritic synaptic contributions to the membrane potential v T ( r ) at

352

a particular cell at the point r on the NRT sheet might be approximated as a sum of contributions from the nearby synapsing cells, each depending on the membrane potential differences between the cells. Thus, a typical form would be

in terms of some non-linear function F. Working with only small changes of potentials, F can be linearised, to give the contributions

where G = F’(0) is positive in the case of inhibitory action in the dendredendritic synapse. For values of r’ close to r, the continuum limit can be taken of the NRT system, and the expression (8) reduces to (Taylor 1990): -a2V2vT

+ O(a4),

(9)

where Vz is the two-dimensional Laplacian operator in Cartesian coordinates and a is a real constant determined by the average spacing between the neurons of the net. The resulting equations for the action of linearised cells is the extension of (2) by adding the dendredendritic synaptic contribution on the R.H.S. This leads to (1) and (2) modified by (9): VT

VN

=I

+A .

VN,

+ a Z V Z v N= B ’vT - c - v N .

(1’)

(2’)

Upon neglect of the lateral connection matrix C, and with A and B diagonal, we obtain the simpler equation V N 4- b 2 V Z V N = J, (10) where bZ = (1 - AB)-’aZ,J = (1 - AB)-’I, and we assume AB < 1 to prevent infinite gain in the thalamus-NRT feedback loop system. The expression (10) is the basis of the simple version of the global gating model. What can we deduce about the dependence of the response of the NRT cells’ potentials (and hence that of the thalamic cells by (1)) from (10). We claim that (10) instantiates a form of competition in the spatial wavelength parameters of incoming inputs J. Physical systems with this underlying description have been investigated in a number of cases: spatially inhomogeneous superconducting states on tunnel injection of quasparticles (Iguchi and Langenburg 1980); a Peierls insulator under strong dynamic excitation of electron-hole pairs or in the presence of electromagnetic radiation (Berggren and

353 Huberman 1978). These models, and related ones for growth and dispersal in a population (Cohen and Murrray 1981; Murray 1989). We may see precise forms of competition arising from the NRT modelled by (11) or (10) by looking at these equations for inputs made of sums of plane waves. Thus, if J is composed of a set of separate waves of wavenumbers kl, . . .,kn, so J = Cj cj sin kj . r, then for suitable coefficients kj, from (lo), n

'J sink, - r ,

j=1

Thus, NRT activity will also display the same spatial oscillation as the input, but now with amplification of those waves with 1 k? = -

b2'

3

Such augmentation corresponds to a process of competition on the space of the Fourier transform of inputs, where the Fourier transform f(k) for an input I(r) involves the global recombination 1 f(k) = - d2r eik'rI(r). 2* It is in this manner that we can see how NRT can exercise global control, by singling out those components f(k) of I(r) by (13) for which (12) is true. Other values of k do not have such amplification. In other words, it would appear that the NRT would oscillate spatially with wavelength 2rb, with net strength given by the component of the input with the same wavelength. The way in which global control arises now becomes clear. Only those inputs which have special spatial wavelength oscillations are allowed through to the cortex, or are allowed to persist in those regions of cortex strongly connected to the NRT: the thalamusNRT system acts as a spatial Fourier filter. There is evidence for this in that feature detectors occur in a regular manner across striate cortex (Hubel and Weisel 1962) as well as facecoding appears to have a spatial lattice structure (Harries and Perrett 1991). Other explanations of the spatial periodicity of striate cortex feature detectors have been proposed (Durbin and Mitchison 1990) but these are consistent with our present proposal which may only add a further spatial instability to that explored in those references by non-NRT processes. It would seem that the model of Figure 1 with dendro-dendritic synapses, as described by equations (1) and (2), can satisfy criterion (b) of $2, at least to a limited extent. The model can be extended immediately to allow activity (c) of $2 to be implemented by addition of a further direct input arising from a net assessment of peripheral visual inputs.

1

354

There are known diffuse neurochemically identifiable afferents (cholinergic, GABAergic, serotonergic, and noradregernic) which enter NRT. These are known to modify NRT cell firing. In some parts, these inputs may be used to switch off on-going NRT activity (by inhibition) or win any on-going competition (by excitation). This could be modelled by an additional input on the R.H.S. of (2). It is relevant to note that NRT activity is expected to be more sensitive to such inputs than that arising indirectly from T in (1). There is still the question of stability of the solutions. We also have to discuss how flexible is the competitive process. That will be in terms of the more complete model which we outline in the next section. This will indicate that global control may also depend on the input amplitude, as well as their wavelengths, in the non-linear regime.

3

Modelling the Global Competitive Gate

3.1

The Model

3.1.1

Formulation

We discussed in the previous section various simplified versions of the overall model to be presented in this chapter. There were also some unanswered questions raised by these models, and in particular, that of the stability and flexibility of the system. In this section, we wish to present the more complete model and discuss in general how it functions. Furthermore, we will show how it possesses features able to give a broad range of responses to inputs. Most specifically, we wish to show that parameter ranges can be varied by internal and external factors. The basis of the model is best seen in terms of its wiring diagram in Figure 2. The model is an extension of that in Figure 1 by means of : (i) addition of the cortical layer C, (ii) addition of interneurons IN in thalamus, (iii) extension of the input to IN cells as well as the T cells, and NRT feedback solely taken to IN cells. The latter conectivity is an approximation to the results of the analysis of Steriade and colleagues (1986), in which the effect of GABA on GABAergic cells (from N cell feedback to IN cell) is claimed to be an order of magnitude more effective than of GABA on excitatory cells. We note that the model may be regarded as the framework for more extensive modelling with layered cortex and NRT. However, at this stage, too much complexity would be introduced too rapidly by such structural richness. We now turn to the detail of the equations used to describe the thalamus-NRT-

355

cortex interacting system, whose general structure is shown in Figure 2. Excitatory neurons are assumed to be at a coordinate position r (using the same coordinate frame for thalamus, NRT and cortex) on the thalamic and cortical sheets and to have membrane potentials labelled uT(r) and uc(r), and outputs f,(u;), (i = T and C,respectively), where f;(z) = [l exp with threshold 0; and temperature 5";. Inhibitory neurons in the thalamus and on the NRT (with the same coordinate positions) have membrane potentials denoted by vT(r), v N ( r ) , with similar sigmoidal outputs g T , g N to the other neurons. Connection weights from the j'th excitatory (inhibitory) neuron at r' to the i'th excitatory (inhibitory) neuron at r are denoted by WtE(r, r'), W r ( r , r'), W4E(r,r'), W4r(r,r'). The resulting leaky integrator neurons (LINs) satisfy

+

,'-I)?(

1 G T ( r ) = --uT(r) 7T

+1c

[@g(r,r')fC(UC(r'))

1 + w%r, r ' ) g T ( v T ( r ' ) ) ] + -l(r) TT

(17)

r'

(where we are using the notation 4 = &/at). We have rescaled all the connection weights, and the input on the thalamic cells, so that the decay constants r;, r+ drop out for stationary activity. Moreover the dendredenritic contribution has yet to be included in (15), it being given by the analysis of Taylor (1990), as cited earlier, to be equal in the linearised limit to -Gz(NvN(r) -

vN(r')).

(18)

A purely rectangular net with four neighbours at horizontal and vertical distances a*, b* (so r' = (r (a*, b i ) ) ) gives, in the continuum limit (Taylor 1990), the dendro-dendritic term is 1 + G z ( v . V V N -A2 . V z v ~+) higher order terms (19) 2 where v = (a+ - a _ , b+ - b - ) T , A' = (a: + a!, b: b'_)=, V2 = ( 3 z , d i ) T , and G2 is a negatively-valued constant in (19). The equation (15) thus reduces to a negative Laplacian net, with a linear derivative term proportional to v . The inputs to the net depend, however, in a non-linear manner on ON as given by (14), (16), (17).

+

+

+

356 3.1.2

Analysis of the Model

The model of Figure 2, expressed mathematically by equations (14)-(19), is expected to have similar properties to those of the various models associated with Figure 1 and discussed in 52. That is clear for the asymptotic solutions satisfying the equations obtained by setting the left hand sides of (14)-(17) all to zero. The main differences between this static system and the earlier equations of 52 are: (i) the presence of the C-layer and its feedback, (ii) non-linearity of all neurons, (iii) the presence of the inhibitory interneurons IN, and NRT feedback to them rather than the T-cells (as in Figure 1). We will briefly discuss each of these features in turn. The cortical layer can be seen as a mechanism for achieving extra input amplification, by means of the thalamus-cortex-thalamus feedback loop, as has already been discussed by La Berge and colleagues (1992). Indeed, the thalamus-NRT feedback loop is functioning in a similar manner, where the factor 1/(1 - AB) arises as the gain factor in the model of 52. This amplification may make the competition on NRT more effective (La Berge et d.1992). This will be borne out by simulation of our complete system. Nontrivial transforms in C will be expected to modify this result, but will not be considered here. The non-linear functionality of neurons was already argued to be a means of increasing the sensitivity of the system in 52. We will appeal to this same argument here. The inhibitory interneurons have been included so as to be able to implement Dale’s Law properly. This is clearly satisfied in Figure 2 and equations (14)-(17). The effect of NRT activity on the IN cells will therefore be that of disinhibition, which is a mode of action used in numerous parts of the brain. Its net effect, for suitable parameter ranges of cell thresholds is similar to that of excitation of NRT activity directly on the thalamic relay cells, as observed experimentally for example by Steriade et al. (1986). In this range of parameters, then, the disinhibitory NRT-IN-T activity of Figure 2 reduces to the net excitatory NRT-T activity of Figure 1. It would appear that the model of Figure 2 should therefore have similar properties to that of Figure 1, in particular supporting both local and global competition and being able to account for endogenous activity by addition of direct input to NRT on the R.H.S. of (15). However, there is still the crucial question of stability and more detailed inputdependence, of plane waves excited globally across the NRT. Stability can be analysed by looking at higher order terms that might arise in (19), and also consider a linearised analysis of the lateral connection term involving the con-

351

nection weight W"(r,r') on the R.H.S. of (15). The former of these gives (assuming symmetry in the x and y directions)

ahd the latter, for a Gaussian spread of lateral inhibition, the convolution product

where

with b < a. The dispersion relation for the temporal dependence ex* of the NRT membrane potential in (15), neglecting all but the lateral connections as a first approximation (since the other terms do not play a crucial role) becomes for the Fourier mode k, A = -G2k. k - c2(k. k)2 - F ( k ) ,

(23)

with

In Figure 3(a), the general shape of F ( k ) is plotted, and in Figure 3(b), that of A(k). Instability arises for values of k for which X(k) 2 0. This is the interval (kl,k z ) in Figure 3(b). As argued with clarity in Murray (1989),inhomogeneous NRT activity, with wave numbers in the interval (kl,kz) will be expected to grow. The stability of the resulting globally inhomogeneous activity depends on the more detailed non-linear system (14)(18), and presently can only be ascertained by simulation. Results of the latter will be presented in $3.2; they will show the system is stable. The dependence of the winning activity on the maximum of the wavelength number being singled out by the competition on NRT will be discussed further in the next sub-section. There is finally the non-trivial dependence on the inputs of the inhomogeneous mode winning the competition on the NRT sheet. Thus, whilst equation (23) would seem to indicate lack of dependence of this mode on the input, the full non-linear system does indeed have such a dependence. There does not seem to be much mathematical analysis of this problem in the literature, although dependence on boundary and initial conditions for reaction-diffusion models has been studied (Arcuri and Murray 1986). We will turn to answer this question by simulation in the next sub-section; we propose to consider it mathematically elsewhere.

358

3.2 3.2.1

Simulations The Simulation Model

The simulation model we investigated is illustrated in Figure 2. It is essentially onedimensional, which corresponds to lines of thalamic, NRT and cortical neurons. In future work, it is intended to extend the simulation work to the more realistic case of tww dimensional sheets of these neurons. A simplified version of Figure 2 is presented in Figure 4. It is useful for simulation purposes to think of the boxed subsystem in Figure 4 as a single module that can be linearly replicated (with bidirectional lateral links among the NRT units). This allows the size of the simulation to be scaled to suit the available computing power. Within each module, we have inter-unit signal flow as specified by the arrow-tipped curves. Every tip labelled with a ‘+’ signifies an excitatory signal, while every ‘-’ corresponds to an inhibitory signal. Within every module, the time evolution of each neural unit’s voltage is determined by the solution of equations (14)-(17). Within a computing context, of course, a discretised version of these equations has to be integrated over some suitable small time step (we used the RungeKutta fourthmder integration routine). Such a scheme entails in practice that we replace the hard-limiting non-linearity by a smooth analytical function. We adopted the following form, as used by La Berge et al. (1992): f(.A)

= hAY(.A

- 0,)

[I - exp { - p A ( . A

- OA)}]

(25)

where h~ is a scaling parameter, Y ( o )is the step function, 6, is the threshold for unit A and pa is its inverse temperature. In order to proceed with the simulations, it was found necessary to obtain limits on the ranges of the large number of parameters involved. To obtain an idea of what constitutes ‘good’ parameter ranges, consider the equations (14)-(17) in the static limit (obtained by setting d u ; / d t = dvi fdt = 0, or, equivalently,by allowing the ri’s, i = T, N, C to go to zero): uc = w c c y ( U c VN

= wNIY(UT VT

UT

=Ik

+

-

- ec),

(14’)

eT)+ w N c y ( U c - ~ c ) ,

= w T N y ( v N - ON),

WTTY(VT

- 0,)

+WTCy(UC-

(15’) (16’)

@C),

(17‘)

Note that in these equations, we have taken the full non-linearity Y ( o )for the function in (25). To distinguish between zero and finite external inputs, we define further Z i = z k = 0 and 1; = I k # 0. In order to set up the subsystem so that we can achieve global control,

359

we require that units T, Nk and C be switched ‘off’ whenever unit I is ‘on’ in the case of zero external input, and vice versa in the case of finite external input2. Looking at (17‘) separately for the case when I&= I; and I&= I t , it is easy to show that the thresholds of each of the units have to lie in the ranges

3.2.2

Simulation Results

The series of simulation results we carried out reflect in part the major aim of establishing the existence of the global control model for the NRT. In Figures 5-15, we present the results obtained from actual simulation runs. The simulation code we developed was very modular in design, allowing for easy scaling to more powerful simulations, which could, with some reconfiguration, also be performed on parallel processing machines. It is hoped in future to extend the simulations to the more realistic case of two-dimensional sheets of thalamus, NRT and cortical tissue, and this can easily be accommodated. The z-axes in these plots represent the spatial positions of a line of 100 neurons. The y-axes represent OUT(UT) and OUT(VT) for the cases of the thalamic excitatory and inhibitory neurons respectively (where OUT has its usual ANN interpretation), and user-scaled3 raw voltage output from the UN and Uc excitatory neurons in the NRT and cortex respectively. Time delays for signal propagation between neurons were set to zero (although they could easily be incorporated in any more realistic future runs). The very simplest situation one can envisage with this system is that where there is no lateral coupling between NRT neurons. Such a system (Figure 5) is very useful for evaluating the effect of feedback strength between neurons of the three layers. We see that each vertical module acts in essence like an amplifier for its particular input signal. Since there exist a large number of free parameters in this system (between 2040, depending on how the simulations are configured), it is useful to determine suitable ranges for the allowed values these can take, in addition to the constraints in (26). We do this by introducing lateral connectivity (given by equations (18), (22)) and experimenting with different values of the amplitudes, thresholds and temperatures of the neurons, and the range of the spreads for the lateral terms. The last of these turns out to be a stability ’The ‘on’ state corresponds to OUT(unit)=l, while the ‘off’state is OUT(unit)=O, with OUT having its usual artificial neural network meaning. ’As a result, the vertical scales in any of the plots are not in proportion. This, however, is not a concern, since we are interested in the general behaviour of the system rather than particular values of output voltages.

360

parameter, in that excessively long-ranged influence of the lateral connectivity terms leads to unrealistically large output voltages in the NRT neurons. We identify such behaviour as the nonlinear regime of operation. This behaviour is illustrated in Figures 6-8, where we successively increase the spread from 5 to 50. Every positively valued segment of the NRT wave acts to allow an input getting through to the cortex, while every negatively valued segment acts to restrict it. This is best illustrated in Figure 7. For moderate values of the spread, we find that the spread has a second attribute as a mechanism for local control. For large values of the spread, the NRT activity begins to grow disproportionately (Figure 8). The spread also plays a part in the outcome of amplitude competition, as in Figure 9, for instance. Of the two plateaus representing strong and weak inputs, the stronger input dominates the weaker one. The spatial wavelengths set up in the NRT region are much smaller than the spread of the plateaus, yet exercise strong control over the allowed cortical response. There is partially global control over the allowed set of inputs propagating up to the cortex, determined principally by wavelike activity in the NRT, itself a function of the spread. It is interesting to note here the relative effects of the difference of Gaussian (DOG) and dendro-dendritic terms in the emergence of waves on the NRT sheet. Figure 10 illustrates the result we obtain be eliminating the dendro-dendritic term, while Figure 11 shows the activity upon removal of the DOG term. We see that the DOG term is clearly less influential in both spatial waveform generation and the development of strong patterns of activity in the cortical layer. This is to be expected, however, since axon collaterals in the DOG representation are not long-ranged. The global control mechanism that is predicted in $2 is illustrated in Figures 12-15. The spatially global wave of activity on the NRT exercises wavelength dependent control on the signals propagating forth to the cortex. The cortical activity persisting does not reflect the inputs very strongly (Figure 13), as it did for the partially global control in Figure 9, being influenced instead by the oscillatory character of the NRT activity A significant aspect of this system, when operating in a global control phase, is its sensitive dependence on classes of inputs. We have seen that during such a phase, the cortex only sees the winner of the competition taking place on the NRT. Equivalently, the NRT (according to the theory of $2) is acting like non-linear filter that allows one out of all the Fourier components presented to it to propagate through. We expect therefore that the selection of this particular Fourier component would be critically influenced by factors such as the wavelength and the amplitude of the input (we noted this towards the end of $2 as well). It is difficult to establish with certainty, however, which of these two

36 1 are the more significant, since our simulation results shows only trends in the output, and not actual magnitudes thereof, as mentioned earlier.

4

Discussion

The theoretical and simulation results presented in the previous section show that the simplified model of the thalamus-NRT-cortex complex does achieve the requirements (a), (b) and (c) outlined in $2, to allow rapid local and global competition to occur between thalamic inputs. There is still much analysis to be done on this model, both theoretically and by simulation. Thus the following questions, among many, need to be answered: 1. delineation of the domain of parameter space in which the competition between in-

puts occurs in momentum space or instead of on the separate coordinate-dependent amplitudes, due to non-linearities, or vice versa;

2. the crucial parameters on which the speed of the outcome of the competition depends on; 3. the effects of including more realistic properties of neurons, such as stochasticity, spikes, neuronal geometry, etc.; 4. the effect of the more complete structure of cortex and NRT, and of more realistic

modelling of thalamic glomeruli, on the nature of the system;

5. the effects of information transformation in cortex on the details of the operation of the competition. The answer to 2 is of relevance to ongoing experimental work on the bringing to sensory awareness of sensations of touch by direct cortical electrical stimulations (Libet et aJ. 1964), as has been pointed out recently by one of us (Taylor 1993a). The answer is expected to be in terms of an exponential increase in time, with a time-constant depending upon the injected current, as the stability analysis of $2.1 indicates. The answer to 3 is relevant to the level of neuro-biological realism of the modelling, but is not expected to make much difference to the principles of the system. Questions 4 and 5 may allow considerable extension of the model, especially if one takes account of cortical activity at working memory centres being injected as thalamic input; these working memory units may, along with primary cortical input areas, be considered as the source of sensory awareness (Taylor 1993b). Finally, question 1 requires further work

362

also involved with investigations of the other questions. W e hope to be able to consider answers to these questions in due course.

Acknoweledgements One of us (F.N.A.) would like to thank the Science and Engineering Research Council (SERC) of the U.K. for grant GR/F92251 which enabled part of this work to be carried out.

363

p’

; \i

.;, I2

P’

I3

Figure 1. The structure of the first model of the thalamus-NRT-cortex complex, in which the cortex is dropped, and competition occurs only between the inputs Ij to the thalamic relay cell Tj;the strengths Oj of the outputs indicate the winners and losers of the competition carried out between the corresponding NRT neurons Nj by means of their inhibitory lateral connections.

364

Figure 2. The wiring diagram of the main model of the thalamus-NRT-cortex complex. Input Ij is sent both the the thalamic relay cell Tj and the inhibitory interneuron I N j , which latter cell also feeds to Ti. Output from Tj goes up to the corresponding cortical cell Cj, which returns its output to Tj. Both the axons TjCj and CjTj send axon collaterals to the corresponding NRT cell N j . There is axonal output from Nj to I N j , as well as collaterals to neighbouring NRT cells; there are also dendro-dendritic synapses between the NRT cells.

365

Figure Sa. Dependence of the Fourier transform @(k) of the lateral NRT connection weights on the wave variable k. Initially positive, the value of @(k) becomes negative. so producing possible instability according to equation (23).

Figure Sb. Dispersion relation for U N in wavenumber space. The interval (kl,122) contains the wave numbers for which instability occurs, and spatially inhomogeneous activity is expected to arise.

367

r (k-- - - ~ r - - - (k+~ r - - Kth subsystem

1)'lh Subsystem

1)'th Subsystem

1

I I

I

I

I 1

_ - -

- - _ I

(k-1)'th Input

k'th Input

(k+ 1)'th Input

Figure 4. Schematic of the "modularised"version of Figure 2, as used in obtaining the results of the computer simulations.

368

Figure 5. Simulation run with no lateral coupling between NRT neurons. The input essentially feeds through to the cortex, as might be expected.

1

NRT(vo1ts) OUT(Tha1am.) OUT(lnhi6.) -

Input(v0lts) Figure 6. Simulation run with lateral connectivity (both dendro-dendtitic and DOG) intrclduced. The value chosen for the spread is small here. Wave activity is beginning to appear on NRT.

Figure 7. As for Figure 6, but with moderate values for the spread. The NRT is clearly influencing what is dowed to propagate through to the cortex.

37 1

Figure 8. As for Figure 6, but with large values for the spread. The activity on the NRT is beginning to take on a non-linear mode of operation. There is still, however, control over what is allowed to go through to the cortex.

312

Figure 9. Simulation run showing amplitude competition. Of the strong and weak inputs being fed in, only the strong survives the journey to the cortex. There is partially global control exercised over this by the activity on the NRT.

373

Figure 10. Simulation run showing the development of activity on NRT with only a DOG form for the lateral connectivity. (Compare with Figure 11).

374

T

Y

.c

0

S:_"I

P

IIIIIIIIII

IIIIIIIIII

IIIIIIIIIII

IIIIIIIIII

.r

Figure 11. Simulation run showing the development of activity on NRT with only a dendro-

dendritic form for the lateral connectivity. (Compare with Figure 10).

315

Figure 12. Simulation run showing full global control with a spatially constant input. The activity on the cortex reflects the activity on the NRT, and is not dependent on the form of the input.

316

Figure 13. Simulation run showing full global control with semi-constant spatial input. Again, the cortex activity is influenced by the NRT alone.

377

Figure 14. Simulation run showing full global control with short-wavelength periodic input.

378

Figure 15. Simulation run showing full global control with medium-wavelength periodic input.

319

5

References

AhlsBn, G. and Lindstijm, S. (1982). Mutual Inhibition between Perigeniculate Neurons, Brain Res., 236,482-486. Arcuri, P.and Murray, J. D. (1986). Pattern sensitivity to boundary and initial conditions in reaction-diffusion models, Math. Biol., 24, 141-165. Avanzini, G., de Curtis, M., Ferruccio, P. and Spreafico, R. (1989). Intrinsic Properties of Nucleus Reticularis Thalami Neurons of the Rat Studied in uitro, J. Physiol., 416, 111-122. Berggren, K. F. and Huberman, B. A. (1978). Peierls state far from equilibrium, Phys. Rev. B, 18,3369-3375. Barbaresi, P., Spreafico, R.,Frassoni, C. and Rustioni, A. (1986). GABA-ergic neurons are present in the dorsal column nuclei but not in the ventroposterior complex of rats, Brain Res., 382,305-326. Cohen, D. S. and Murray, J. (1981). A Generalised Diffusion Model for growth and dispersal in a population, Math. Biol., 12, 237-249. Crabtree, J. W. (1989). Evidence for topographic maps within the visual and somatosensory sectors of the thalamic reticular nucleus: A comparison of cat and rabbit, SOC. Neurosci. Abs., 15, 1393. Crabtree, J. W. (1991). Maps within the cat’s somatosensory thalamus, SOC. Neurosci. Abs., 17,623. Crabtree, J. W. (1992). The somatotopic organization within the rabbit’s thalamic reticular nucleus, Eur. J. Neurosci., 4, 1343-1351. Crabtree, J. W. and Killackey, H. P. (1989). The topographic organization and axis of projection within the visual sector of the rabbit’s thalamic reticular nucleus, Eur. J. Neurosci., 1, 94-109. Deschknes, M., Madariaga-Domich, A. and Steriade, M. (1989). Dendro-dendritic synapses in the cat reticularis thalami nucleus: A Structural basis for Thalamic Spindle Synchronisation, Brain Res., 334,165-168. Douglas, R. J. and Martin, K. A. (1991). A Functional Microcircuit for Cat Visual Cortex,

380

J. Physiol., 440, 735-769. Dowling, J. (1987). The Retina. Harvard University Press. Durbin, R. and Mitchison, G. (1990). A dimension reduction framework for understanding cortical maps, Nature, 343, 644-647. Ermentrout, G. B. and Cowan, J. D. (1978). Some Aspects of the ‘Eigenbehaviour of Neural Nets’, Studies in Mathematics: The Mathematical Assoc. of America, 15, 67-117. Harries, M. H. and Perret, D. I. (1991). Visual Processing of Faces in Temporal Cortex: Physiological Evidence for a Modular Organisation and Possible Anatomical Correlates, J. Cog. Neurosci., 3, 9-23. Harris, R. M. and Hendrickson, A. E. (1987). Local circuit neurons in the rat ventrobasal thalamus - A GABA immunocytochemical study, Neurosci., 21, 229-236. Hornik, K., Stinchcombe, M. and White, H. (1989). Multilayer feedforward networks are universal approximators, Neural Networks, 2, 359-368. Hubel, D. H. and Weisel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex, J. Phys., 160,106-154. Iguchi, 1. and Langenburg, D. N. (1980). Diffusive Quasiparticle Instability toward Multiple-Gap States in a Tunnel-Injected Nonequilibrium Superconductor, Phys. Rev. Lett., 44, 486-489. Jones, E. G. (1975). Some Aspects of the Organisation of the Thalamic Reticular Complex, J. Comp. Neurobiol., 162,285-308. Koch, C. and Ullman, S. (1985). Shifts in Selective Visual Attention: Towards the Underlying Circuitry, Human Neurobiol., 4, 219-227. La Berge, D. (1990). Thalamic and Cortical Mechanisms of Attention suggested by recent Positron Emission Tomographic Experiments, J. Cog. Neurosci., 2,358-373. La Berge, D., Carter, M. and Brown, V. (1992). A Network Simulation of Thalamic Circuit Operations in Selective Attention, Neural Computation, 4, 318-331. Libet, B., Alberts, W. W., Wright, E. W., Delattre, L. D., Levin, G. and Feinstein, B. (1964). Production of threshold levels of conscious sensation by electrical stimulation of human somatc-sensory cortex, J. Neurophys., 27, 546-578.

38 1

Lijenstr6m, H. (1991). Modelling the Dynamics of Olfactory Cortex using Simplified Network Units and Realistic Architectures, Int. J. Neurosci., 1-2, (to appear). Linsker, R. (1988). Self-organisation in a Perceptual Network, Computer, 21, 105-1 17. Llinas, R. and Ribary, U. (1991). Ch. 7 in Induced Rhythms in the Brain. Eds. E. Basar and T. Bullock, Birkhauser, Boston. Martin, K. A. (1988). From Single Cells to Single Circuits in the Cerebral Cortex, Quart. J. Exp. Physiol., 73,637-702. McCormick, D. A. and Prince, D. A. (1986). Acetylcholine induces burst firing in thalamic reticular neurons by activating a potassium conductance, Nature, 319,402-405. Montero, V. M., Guillery, R. W. and Woolsey, C. N. (1977). Retinotopic organization within the thalamic reticular nucleus demonstrated by a double label autoradiographic technique, Brain Res., 138,407-421. Murray, J. M. (1989). Mathematical Biology, Springer-Verlag. Ohara, P. T. and Lieberman, A. R. (1985). The Thalamic Reticular Nucleus of the Adult Rat: experimental anatomical studies, J. Neurocyt., 14,365-411. Ohara, P. T. (1988). Synaptic Organisation of the Thalamic Reticular Necleus, J. Elect. Mic. Tech., 10,283-292. Park, D., Steriade, M., Deschdnes, M. and Oakson, G. (1987). Physiological Characteristics of Anterior Thalamic Nuclei, a group devoid of inputs from Reticular Thalamic Nucleus, J. Neurophysiol., 57,1669-1 685. Posner, M. I. and Petersen, S. E. (1990). The Attention System of the Human Being, Ann. Rev. Neurosci., 13,25-42. Schiebel, A. B. (1980), in Reticular Formation Revisited, eds. J. A. Hobson and B. A. Brazier, Raven Press, New York. Spreafico, F., De Curtis, M., Frassoni, C. and Avanzini, G. (1988). Electrophysiological Characteristics of Morphologically identified Reticular Thalamic Neurons from Rat Slices, Neurosci., 27, 629-638. Spreafico, F., Battaglia, G. and Frassoni, C. (1991). The Reticular Thalamic Nucleus (RTN) of the Rat: Cytoarchitectural, Golgi, Immunocytochemical and Horseradish Per-

382

oxidase Study, J. Comp. Neuro., 304, 478-490. Steriade, M., Domich, L. and Oakson, G. (1986). Reticularis Thalami Neurons Revisited: Activity Changes During Shifts in States of Vigilance, J. Neurosci., 6,68-81. Steriade, M., Currc-Dossi, R. and Oakson, G. (1991). Fast Oscillations (20-40 Hz), in thalamocortical systems and their potentiation by Mesopontine Cholinergic Nuclei in the Cat, Proc. Nat. Acad. Sci., 88, 4396-4400. Taylor, J. G. (1990). A Silicon Model of Vertebrate Retinal Processing, Neural Networks, 3,171-178. Taylor, J. G. (1993a). A Competition for Sensory Awareness? King’s College London Preprint. Taylor, J. G. (1993b). Goals, Drives and Consciousness. King’s College London Preprint. Triesman, A. and Gelade, G. (1980). Cog. Sci. 12,99-136. WGrgiitter, F., Niebur, E. and Koch, C. (1991). Isotropic Connections Generate Functional Asymmetrical Behaviour in Visual Cortical Cells, J. Neurophysiol., (in press). Yingling, C. D. and Skinner, J. E. (1977). Gating of Thalamic Inputs to Cerebral Cortex by Nuclear Reticularis Thalami, Prog. Clin. Neurophysiol., 1, 70-96.